I'm trying to track down the reason behind the sudden drop in put rates to two Mapr-DB tables defined in the same volume. I was very surprised by the sharp drop and I would like to understand how I can trace the cause.
When I see that writes are slow in my application, I see logs indicating lots of pending flushes and HighLatency writes in mfs.log-5 in /opt/mapr/logs. I'm not sure what to make of the pending flushes since its not clear what's the underlying cause.
I did some tests on my cluster now and I'm not seeing any such pending flushes in the logs and the write rate is pretty good. (~32k puts per sec across the cluster). In this small test cluster (4 nodes), I've seen somewhere between 50 - 60k puts per sec across the cluster when I've a fire hose of events being written to the cluster.
Could you point me to what I should be looking for to track this done? I've attached the mfs log from the time of the issue. See logs for 2016-09-02. This drop in puts was the reason I raised What is the impact of using more than one Async HBase client?