I've been running a spark streaming job for 5 days or so with varying load.
I've set the batch interval to be 7 minutes and set the Kafka rate-per-partition + # partitions to equate to 27.72 million records during that 7 minute interval.
I saw this work for literally days (the load falls to to 1-5 million on average and spikes up to 27.72 million every so often). All of the sudden it spiked up to 40 million and messed up my job though after 5 days.
Are there known conditions under which "spark.streaming.kafka.maxRatePerPartition" will fail to be honored by Spark Streaming + MapR Streams? I don't see any OOM errors or anything like that to have triggered the change.
The job seems to be catching up/recovering now but a slightly bigger spike could easily have killed the job which is why I'm concerned.
|2017/09/06 01:34:00||27720000 records||17 min||5.2 min||23 min|
|2017/09/06 01:27:00||27720000 records||13 min||5.3 min||18 min|
|2017/09/06 01:20:00||30923257 records||9.4 min||4.6 min||14 min|
|2017/09/06 01:13:00||40422828 records||5.2 min||4.5 min||9.7 min|
|2017/09/06 01:06:00||27720000 records||2 ms||6.1 min||6.1 min|
|2017/09/06 00:59:00||9344608 records||0 ms||3.1 min||3.1 min|
|2017/09/06 00:52:00||15178678 records||0 ms||3.8 min||3.8 min|