I couldn't find an answer to this; though it's hard to think it hasn't been asked before.
If I have a very large batch get pulled into spark streaming (I know, that's bad in a perfect world), then spark still keeps scheduling a new batch every 15 seconds.
Eventually, I can end up with like 500 "pending" batches.
Is there a way to tell it to hold off on scheduling new intervals after, say, 10 have piled up? The app seems to freeze at a certain point; so, I'm guessing that this pile-up is in fact causing problems.