Can anyone explain how the offsets in MapR streams are managed/incremented?
One of my use cases is:
- Receive file from external source
- Put file name in MapR streams
- Have various apps consume stream of file names and process files
Given this, my monitoring usually looks at the offset lag (max offset - min offset) to determine how many files are back logged. This works 99% of the time, but every now and then the offsets in MapR Streams will just jump by 500,000 or so for a single message (which breaks the monitoring).
- Why does the offset sometimes jump?
- Is there a way to actually monitor pending # of messages rather than offsets if offsets are un-reliable?