AnsweredAssumed Answered

MapR issues

Question asked by mmercer on Jan 28, 2014
Latest reply on Jan 30, 2014 by mmercer
We have been seeing some strange issues with performance, and now, we are seeing errors we can't make sense of:
<pre>
"MapR RA" daemon prio=10 tid=0x00007f315082b800 nid=0xb3c in Object.wait() [0x00007f30a5e48000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f3481b45a40> (a com.mapr.fs.MapRFsInStream$RAThread)
        at java.lang.Object.wait(Object.java:485)
        at com.mapr.fs.MapRFsInStream$RAThread.run(MapRFsInStream.java:564)
        - locked <0x00007f3481b45a40> (a com.mapr.fs.MapRFsInStream$RAThread)
        at java.lang.Thread.run(Thread.java:662)

-----
"pool-3-thread-32" prio=10 tid=0x00007f322c95f000 nid=0x3e9e runnable [0x00007f319228b000]
   java.lang.Thread.State: RUNNABLE
        at com.mapr.fs.jni.MapRClient.writeRPC(Native Method)
        at com.mapr.fs.Inode.flushPages(Inode.java:317)
        at com.mapr.fs.Inode.releaseDirty(Inode.java:425)
        at com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:70)
        at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:81)
        - locked <0x00007f3af70952a8> (a com.mapr.fs.MapRFsOutStream)
        at com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:46)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
        - locked <0x00007f3af6f54170> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190)
        at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:111)
        - locked <0x00007f3af6f54170> (a java.io.OutputStreamWriter)
        at java.io.BufferedWriter.write(BufferedWriter.java:212)
        - locked <0x00007f3af6f54170> (a java.io.OutputStreamWriter)
        at com.qf.util.twitter.HDFSUtils$.writeStringToFile(HDFSUtils.scala:88)

---
</pre>
Seems like mapr is struggling to write, and we aren't sure why.  I can test the disks write ability and we are seeing 135MB/s on average, same for network.  Unfortunately, we are simply seeing many issues with MapR.

Would love to know where to start looking.

Thanks,

Outcomes