AnsweredAssumed Answered

MapR Streams Offset Jump?

Question asked by john.humphreys on Mar 26, 2018
Latest reply on Mar 26, 2018 by john.humphreys

Can anyone explain how the offsets in MapR streams are managed/incremented?

 

One of my use cases is:

  • Receive file from external source
  • Put file name in MapR streams
  • Have various apps consume stream of file names and process files

 

Given this, my monitoring usually looks at the offset lag (max offset - min offset) to determine how many files are back logged.  This works 99% of the time, but every now and then the offsets in MapR Streams will just jump by 500,000 or so for a single message (which breaks the monitoring).

 

So:

  1. Why does the offset sometimes jump?
  2. Is there a way to actually monitor pending # of messages rather than offsets if offsets are un-reliable?

Outcomes