AnsweredAssumed Answered

Spark Streaming - Stop Batch Pile-Up?

Question asked by john.humphreys on Jul 26, 2017
Latest reply on Jul 28, 2017 by john.humphreys

I couldn't find an answer to this; though it's hard to think it hasn't been asked before.

 

If I have a very large batch get pulled into spark streaming (I know, that's bad in a perfect world), then spark still keeps scheduling a new batch every 15 seconds.

 

Eventually, I can end up with like 500 "pending" batches.

 

Is there a way to tell it to hold off on scheduling new intervals after, say, 10 have piled up?  The app seems to freeze at a certain point; so, I'm guessing that this pile-up is in fact causing problems.

Outcomes