AnsweredAssumed Answered

JDBC Query Consumer not fetching more than 1000 records

Question asked by muralik22 on Jun 20, 2018
Latest reply on Jun 21, 2018 by metadaddy

We are trying to fetch data from table in SAP HANA using JDBC Query Consumer in Streamsets, but we are only able to fetch 1000 records (with Maximum Pool Size in Advanced tab in JDBC Query Consumer set to 1) though there are more than 5 million records in source table.

Streamsets version 2.6.0.1

Configuration in Streamsets for max.ProductionBatchSize set to 1000.

But when we increase Maximum Pool Size in Advanced tab in JDBC Query Consumer to 5000, we are able to fetch 5 million records.

But Maximum Pool Size referring to number of database connections using which it will extract data from source table in external system.

Is it always recommended to increase the Maximum Pool Size irrespective of StreamSets version for this scenario, without which we can't fetch more than the records mentioned in max.ProductionBatchSize.

Else Is this already fixed in latest StreamSets version and are we able to fetch more than records mentioned in max.ProductionBatchSize with Maximum Pool Size in Advanced tab in JDBC Query Consumer set with 1 itself.

Outcomes