1 year ago

#155083

test-img

vrajkuma

Spring Batch very slow when using 2 datasources - one for Spring Batch and another for the App

I modified this sample batch job provided by spring to use two custom datasources instead of the one autoconfigured by boot. Both datasources point to the same MySql DB server, but to different schemas. One schema for Batch/Task tables and another for the app tables. MySql is running locally. The performance was much much slower compared to the same job running with default boot configured datasource or with ONE custom datasource.

Here is the timing I got and can't figure out why #3 is taking a long time:

  1. Default Boot configured datasource - 1 second
  2. One custom datasource (for both Batch/Task and App) - 1 second
  3. Two custom datasources (one each for Batch/Task and App) - 90 seconds !!!

Do I need to set any CP settings for the custom datasources when using two of them? I tried a few, but didn't help.

Here is the properties file:

spring.application.name=fileIngest

spring.datasource.url=jdbc:mysql://localhost:3306/test-scdf?useSSL=false
spring.datasource.username=<user>
spring.datasource.password=<pwd>
spring.datasource.driverClassName=org.mariadb.jdbc.Driver

app.datasource.url=jdbc:mysql://localhost:3306/test?useSSL=false
app.datasource.username=<user>
app.datasource.password=<pwd>
app.datasource.driverClassName=org.mariadb.jdbc.Driver

Here are relevant portions of my datasource config as recommended here.

@Bean(name = "springDataSource") // for Batch/Task tables
public DataSource dataSource(@Qualifier("springDataSourceProperties")DataSourceProperties springDataSourceProperties) {
    return DataSourceBuilder.create().driverClassName(springDataSourceProperties.getDriverClassName()).
        url(springDataSourceProperties.getUrl()).
        password(springDataSourceProperties.getPassword()).
        username(springDataSourceProperties.getUsername()).
        build();
}

@Bean(name = "appDataSource") // for App tables
@Primary
public DataSource appDataSource(@Qualifier("appDataSourceProperties") DataSourceProperties appDataSourceProperties) {
    DataSource ds = DataSourceBuilder.create().driverClassName(appDataSourceProperties.getDriverClassName()).
        url(appDataSourceProperties.getUrl()).
        password(appDataSourceProperties.getPassword()).
        username(appDataSourceProperties.getUsername()).
        build();

I just inject the appropriate datasource into the BatchConfiguration as needed.

@Configuration
@EnableBatchProcessing
public class    BatchConfiguration extends DefaultBatchConfigurer {
...
    
@Override
    @Autowired
    public void setDataSource(@Qualifier("springDataSource") DataSource batchDataSource) {
        super.setDataSource(batchDataSource);
    }

    @Bean
    public BatchDataSourceInitializer batchDataSourceInitializer(@Qualifier("springDataSource") DataSource batchDataSource,
                                                                 ResourceLoader resourceLoader) {
        BatchProperties batchProperties = new BatchProperties();
        batchProperties.setInitializeSchema(DataSourceInitializationMode.ALWAYS);
        return new BatchDataSourceInitializer(batchDataSource, resourceLoader, batchProperties);
    }

spring-data

spring-batch

spring-jdbc

spring-cloud-dataflow

spring-cloud-task

0 Answers

Your Answer

Accepted video resources