AnsweredAssumed Answered

hdfsWrite() bug in libMapRClient.so

Question asked by youngshin on Mar 11, 2013
Latest reply on Apr 10, 2013 by youngshin
Hello,

I frequently got seg-fault error when indexing some data into the maprfs.

During debugging it, I found hdfsWrite() bug.

When the COMPRESSION option is ON and specific pattern data are appended, hdfsWrite() produces a seg-fault error.

I'm using MapR 2.1.1 on centos 6.3. (gcc 4.4.6)

If it is known issue, please tell me whether it is resolved in MapR 2.1.2.

======

If not, I need to tell you more details.

I made simple test program and 2 data files which always produce a seg-fault error. (mail me if you need)

The test program read 2 data files and append those to maprfs file.

The call stack in gdb is as belows.
<pre>
$ gdb ./maprfs_append_test data.1 data.2 maprfs:///debug/data
(gdb) bt
   #0  0x000000399aa891ab in memcpy () from /lib64/libc.so.6
   #1  0x00007fd9e3eab0f2 in mapr::fs::MapClntWriteBuffer::AddCluster (this=0x21d8c00, mapClient=<value optimized out>, inode=<value optimized out>, offsetInPrimary=8257536, filelet=0x7fffa12ef340, niov=428, inIOV=0x7fffa12ef450,
    off=8257536, len=65536, lenLeft=121304) at fs/client/fileclient/cc/writebuf.cc:557
   #2  0x00007fd9e3eac102 in mapr::fs::Inode::AddClusterToBuffer (this=0x21ae450, client=0x2197d90, offsetInPrimary=8257536, filelet=0x7fffa12ef340, niov=428, iov=0x7fffa12ef450, clusterOff=8257536, clusterLen=65536, lenLeft=121304)
    at fs/client/fileclient/cc/writebuf.cc:641
   #3  0x00007fd9e3ea5ef0 in mapr::fs::MapClient::WriteBuffered (this=0x2197d90, inode=0x21ae450, startOff=8257536, length=<value optimized out>, numIov=438, iov=0x7fffa12ef3b0) at fs/client/fileclient/cc/client.cc:2704
   #4  0x00007fd9e3ecf129 in mapr::fs::InodePlus::write (this=0x21ad110, mapClient=0x2197d90, buf=0x7fd9e38c9010 "\365\341\063\016\267\206\002", length=3576924) at fs/client/fileclient/cc/libhdfs/api_support.cc:1528
   #5  0x0000000000400cd7 in AppendToMaprFile (path=0x7fffa12f28c2 "maprfs:///debug/data", buffer=0x7fd9e3649010 "\021", len=6198364) at maprfs_append_test.cc:53
   #6  0x0000000000400dc3 in main (argc=3, argv=0x7fffa12f1128) at maprfs_append_test.cc:74
</pre><br/><br/>

The valgrind result is as belows.
<pre><code>
$ valgrind --db-attach=yes --leak-check=full --track-origins=yes ./maprfs_append_test data.1 data.2 maprfs:///tm4p/debug/data
==18128==
==18128== Thread 2:
==18128== Conditional jump or move depends on uninitialised value(s)
==18128==    at 0x4E710D2: mapr::fs::RpcBinding::CallAllCBs(mapr::fs::RpcWorkArea*, int) (rpcbinding.cc:238)
==18128==    by 0x4E6D942: mapr::fs::RpcServer::DataArrived(int, bool) (rpcserver.cc:915)
==18128==    by 0x4E6ED07: mapr::fs::RpcServer::Runner() (rpcserver-epoll.cc:444)
==18128==    by 0x4E6F043: mapr::fs::RpcServer::Thread(void*) (rpcserver-epoll.cc:284)
==18128==    by 0x4E6F0A9: mapr::fs::RpcServer::Run(bool) (rpcserver-epoll.cc:273)
==18128==    by 0x4E755C6: mapr::fs::RpcThr::ServerThread(void*) (rpcthr.cc:410)
==18128==    by 0x399B207850: start_thread (in /lib64/libpthread-2.12.so)
==18128==    by 0x399AAE811C: clone (in /lib64/libc-2.12.so)
==18128==  Uninitialised value was created by a heap allocation
==18128==    at 0x4A075BC: operator new(unsigned long) (vg_replace_malloc.c:298)
==18128==    by 0x4E75921: mapr::fs::RpcThrWorkArea::AllocateWorkArea() (rpcthr.cc:82)
==18128==    by 0x4E75B89: mapr::fs::RpcThr::SendRequest(mapr::fs::RpcBinding*, unsigned short, unsigned short, google::protobuf::MessageLite*, int, iovec*, google::protobuf::MessageLite*, int, iovec*, void (*)(int, unsigned char*, void*), void*) (rpcthr.cc:188)
==18128==    by 0x4E9AFA8: mapr::fs::CidCache::GetCidForVolume(char const*, mapr::fs::VolLinkAttributeType, unsigned int*, mapr::fs::FidMsg*, bool) (rpcthr.h:40)
==18128==    by 0x4E9C195: mapr::fs::CidCache::Init(char const*, int, unsigned long*, unsigned int*, mapr::fs::MapClient*, bool) (cidcache.cc:789)
==18128==    by 0x4E78102: mapr::fs::MapClient::Init(unsigned long*, int, unsigned int, unsigned int, unsigned int, char const*, _jstring*, bool) (client.cc:396)
==18128==    by 0x4E7900B: mapr::fs::MapClient::GetClient(char const*, unsigned long*, int, unsigned int, unsigned int, unsigned int, char const*, _jstring*, bool) (client.cc:296)
==18128==    by 0x4EB2A0B: mapr::fs::GetClient(mapr::fs::FCClusterConfEntry*) (api_support.cc:293)
==18128==    by 0x400DA0: main (maprfs_append_test.cc:64)
==18128==
......
==18128== Thread 1:
==18128== Invalid write of size 8
==18128==    at 0x4A08D13: memcpy (mc_replace_strmem.c:882)
==18128==    by 0x4E910F1: mapr::fs::MapClntWriteBuffer::AddCluster(mapr::fs::MapClient*, mapr::fs::Inode*, unsigned long, mapr::fs::FidMsg const*, int, iovec const*, unsigned long, int, int) (writebuf.cc:557)
==18128==    by 0x4E92101: mapr::fs::Inode::AddClusterToBuffer(mapr::fs::MapClient*, unsigned long, mapr::fs::FidMsg*, int, iovec*, unsigned long, int, int) (writebuf.cc:641)
==18128==    by 0x4E8BEEF: mapr::fs::MapClient::WriteBuffered(mapr::fs::Inode*, unsigned long, int, int, iovec*) (client.cc:2704)
==18128==    by 0x4EB5128: mapr::fs::InodePlus::write(mapr::fs::MapClient*, unsigned char*, int) (api_support.cc:1528)
==18128==    by 0x400CE1: AppendToMaprFile(void*, char const*, char*, int) (maprfs_append_test.cc:46)
==18128==    by 0x400E47: main (maprfs_append_test.cc:79)
==18128==  Address 0x2000000002008 is not stack'd, malloc'd or (recently) free'd
==18128==
</code></pre>

Outcomes