AnsweredAssumed Answered

MapR Streams - Only compaced/cleaned in MCS

Question asked by john.humphreys on Jan 5, 2018
Latest reply on Jan 12, 2018 by maprcommunity

I'm heavily using MapR streams (~10 billion messages a day), and I have a 4 day TTL set on the stream.

 

Multiple times now, I have filled up the volume the stream is on.


When looking in MCS at the volume page, I can see that it is in fact full.  When I go to "MapR Tables" and view the stream there, I see that it is taking a huge amount of space compared to what is expected after 4 days.

 

Here's the interesting part...  As I'm viewing the stream in MCS, I see the stream size start to fall down gradually like it's being compacted.  I just observed one go from 7TB to 800GB gradually and have observed others fall on multiple occasions now.  So, it seems like even though I'm constantly reading/writing to the topics in the stream, the stream is not being "cleaned" (for lack of a better word) until I view it in MCS.

 

Is this normal behavior in MapR?  Am I supposed to be calling some sort of "cleanse" or "purge" operation regularly that it doesn't do itself?  What does MCS implicitly do when you view a stream/table?

Outcomes