AnsweredAssumed Answered

MapR-Streams Java Client C++ Heap Error?

Question asked by john.humphreys on Mar 12, 2018
Latest reply on Mar 26, 2018 by john.humphreys

I got this error in a long-running Java process consuming off of MapR-Streams.  It took the process down in production.

 

It looks like it is a C++ library error in the MapR client.  Does anyone know what this is and how to mitigate it?


Community Manager - Can you recommend an expert to help get it debugged quickly please?

 

2018-03-10 22:41:57,6590 ERROR Client fs/client/fileclient/cc/client.cc:6206 Thread: 17132 rpc err Resource temporarily unavailable(11) 35.33 to 10.48.82.25:5660, fid 8672.45.131304, upd 1
2018-03-10 22:41:58,0798 ERROR Client fs/client/fileclient/cc/client.cc:6206 Thread: 17132 rpc err Resource temporarily unavailable(11) 35.33 to 10.48.82.25:5660, fid 8672.45.131304, upd 1
2018-03-10 22:41:58,3312 ERROR Client fs/client/fileclient/cc/client.cc:6206 Thread: 17132 rpc err Resource temporarily unavailable(11) 35.33 to 10.48.82.25:5660, fid 8672.45.131304, upd 1
*** glibc detected *** /usr/bin/java: malloc(): smallbin double linked list corrupted: 0x00007fff891dcab0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x371e675f4e]
/lib64/libc.so.6[0x371e67a528]
/lib64/libc.so.6(__libc_malloc+0x5c)[0x371e67ab1c]
/usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x37226bd09d]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs5Arena16AllocateNewBlockEm+0x32)[0x7fffd8bca542]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs5Arena16AllocateFallbackEm+0x2b)[0x7fffd8bca5bb]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs5Arena15AllocateAlignedEm+0x54)[0x7fffd8bca6a4]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs11DocKeyValue15ParseFromRowColEPNS0_10ByteReaderEPNS0_24DocumentSerializeContextE+0x34c)[0x7fffd8bc73cc]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs8Document12ParseDocBodyEPNS0_10ByteReaderEPNS0_24DocumentSerializeContextE+0x199)[0x7fffd8bc0259]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs11DocKeyValue15ParseFromRowColEPNS0_10ByteReaderEPNS0_24DocumentSerializeContextE+0x453)[0x7fffd8bc74d3]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs8Document12ParseDocBodyEPNS0_10ByteReaderEPNS0_24DocumentSerializeContextE+0x199)[0x7fffd8bc0259]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs8Document15ParseFromRowColEPNS0_10ByteReaderEPNS0_19JsonParserCallbacksEb+0xe8)[0x7fffd8bc0648]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs14MessageFetcher16DecodeScannedRowEPNS0_5InodeEPNS0_5ArenaEPNS0_10ScannedRowE+0x887)[0x7fffd8bf9057]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs7Scanner19PopulateFetchedRowsEPNS1_10BufferInfoEPNS0_5InodeEPFiS5_PNS0_5ArenaEPNS0_10ScannedRowEE+0x1b2)[0x7fffd874d392]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs7Scanner21UpdateScanAheadBufferEPKNS0_12ScanResponseEP5iovecPbPFvRKNS0_9StringRefEPS8_EPFiPNS0_5InodeEPNS0_5ArenaEPNS0_10ScannedRowEEPi+0x20e)[0x7fffd8751fae]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs14MessageFetcher16UpdateFeedBufferEPNS0_10FeedBufferEPKNS0_12ScanResponseEP5iovecPbPi+0x4f)[0x7fffd8bf7bdf]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs14MessageFetcher25UpdateFetcherWithResponseEPNS0_15FetcherWorkItemEPNS0_8IovArrayEPPNS0_10FeedBufferE+0xb12)[0x7fffd8bfc662]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs14MessageFetcher18FetchMessagesForWIEPNS0_15FetcherWorkItemE+0xe7)[0x7fffd8bfd8a7]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(+0xb08733)[0x7fffd8be5733]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs14AsyncReqBuffer10FlushWriteEPP7JNIEnv_+0x21)[0x7fffd87110d1]
/tmp/mapr-glpmpbp-libMapRClient.5.2.0-mapr.so(_ZN4mapr2fs10WriteQueue7FlusherEPv+0xce)[0x7fffd87112ee]
/lib64/libpthread.so.0[0x371ee07aa1]
/lib64/libc.so.6(clone+0x6d)[0x371e6e893d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:06 1050537 /local/0/sw/java/jdk1.8.0_121_64/jre/bin/java
00600000-00601000 rw-p 00000000 08:06 1050537 /local/0/sw/java/jdk1.8.0_121_64/jre/bin/java
00601000-026a3000 rw-p 00000000 00:00 0 [heap]

Outcomes