Implement batch operation for REST resources with Spring Batch
Background
Spring Batch allows us to perform large volumes of records from several resources such as File, Relational Database, and, JSON file to name a few.
In this article, we will explore how to implement batch operation that reads from REST resources with Spring Batch through JsonItemReader. We will retrieve a list of users from JSON Placeholder and save them into a database.
Job Configuration
Next is to implement the job that will be responsible to read from REST resource and save them into a database. Job consists of Step and Step
consists of ItemReader and ItemWriter. We will implement all of them in UserJobConfiguration.
@Configuration
class UserJobConfiguration {
private final JobRepository jobRepository;
private final PlatformTransactionManager transactionManager;
private final MongoOperations mongo;
UserJobConfiguration(JobRepository jobRepository, PlatformTransactionManager transactionManager, MongoOperations mongo) {
this.jobRepository = jobRepository;
this.transactionManager = transactionManager;
this.mongo = mongo;
}
@Bean
public Job userJob() throws MalformedURLException {
return new JobBuilder("userJob", jobRepository).start(step()).build();
}
private Step step() throws MalformedURLException {
return new StepBuilder("userStep", jobRepository)
.<User, User>chunk(10, transactionManager)
.reader(reader())
.writer(writer())
.build();
}
private JsonItemReader<User> reader() throws MalformedURLException {
JacksonJsonObjectReader<User> jsonObjectReader = new JacksonJsonObjectReader<>(User.class);
jsonObjectReader.setMapper(new ObjectMapper());
return new JsonItemReaderBuilder<User>()
.name("userReader")
.jsonObjectReader(jsonObjectReader)
.resource(new UrlResource("https://jsonplaceholder.typicode.com/users"))
.build();
}
private MongoItemWriter<User> writer() {
return new MongoItemWriterBuilder<User>()
.template(mongo)
.build();
}
}From the code above, we can see that a URL form of Resource is assigned to JsonItemReader. We will depend on JacksonJsonObjectRader to convert response from JSON Placeholder to User object.
@Configuration
class UserJobConfiguration {
private JsonItemReader<User> reader() throws MalformedURLException {
JacksonJsonObjectReader<User> jsonObjectReader = new JacksonJsonObjectReader<>(User.class);
jsonObjectReader.setMapper(new ObjectMapper());
return new JsonItemReaderBuilder<User>()
.name("userReader")
.jsonObjectReader(jsonObjectReader)
.resource(new UrlResource("https://jsonplaceholder.typicode.com/users"))
.build();
}
}Now that we have implemented the Job, we can verify that it is working by executing an integration test.
Verification
We will launch userJob which will retrieve list of User from JSON Placeholder and save them into a database.
Once completed then we will verify that the database contains the expected number of users.
@Testcontainers
@SpringBatchTest
@SpringJUnitConfig({ BatchTestConfiguration.class, MongoTestConfiguration.class, UserJobConfiguration.class })
class UserBatchJobTests {
@Container
@ServiceConnection
private final static MySQLContainer<?> MYSQL_CONTAINER = new MySQLContainer<>("mysql:latest")
.withInitScript("org/springframework/batch/core/schema-mysql.sql");
@Container
@ServiceConnection
private final static MongoDBContainer MONGO_DB_CONTAINER = new MongoDBContainer("mongo:latest");
@Autowired
private JobLauncherTestUtils launcher;
@Autowired
private MongoOperations mongoOperations;
@Test
@DisplayName("Given there are 10 users returned from REST Service When the job is COMPLETED Then all users should be saved to MongoDB")
void launch() {
await().atMost(ofSeconds(30)).untilAsserted(() -> {
var execution = launcher.launchJob();
assertThat(execution.getExitStatus()).isEqualTo(COMPLETED);
});
var persistedUsers = mongoOperations.findAll(User.class);
assertThat(persistedUsers).hasSize(10);
}
}Full implementation can be found in UserBatchJobTests.