AnsweredAssumed Answered

libMapRClient "Resource temporarily unavailable" and connection reset errors

Question asked by will_cannings on Sep 24, 2013
Latest reply on Sep 25, 2013 by will_cannings
Hi,

I'm running a MapR installation (3.0.1.21771.GA) on two machines, and a Hypertable installation on the same machines, using MapR as its DFS. Things have been running fine for around 2 months, but one of the Hypertable processes is now consistently encountering an error when writing a log file, meaning Hypertable is now unable to start.

Hypertable links to libMapRClient.so, and the relevant log lines (from the MapR library) are:
<pre>
2013-09-25 09:15:45,2434 ERROR Client fs/client/fileclient/cc/client.cc:2772 Thread: 140369484797696 rpc 28.21 to 129.78.110.186:5660, fid 2049.16.2, upd 0, failed err 104
2013-09-25 09:17:00,7301 ERROR Client fs/client/fileclient/cc/client.cc:417 Thread: 140369484797696 Create failed for file 3, error Resource temporarily unavailable(11)
2013-09-25 09:17:00,7302 ERROR LibHdfsApiSupport fs/client/fileclient/cc/libhdfs/api_support.cc:359 Thread: 140369484797696 Create failed for file /hypertable/servers/master/log/mml/3, error: Resource temporarily unavailable(11)
</pre>
I can't see any errors printed in any of the MapR logs after this happens, and changing the log level of 'LibHdfsApi', 'LibHdfsApiSupport', and 'LibHdfsFCClusterConf' to DEBUG didn't change what was logged.

Reading/writing the file (/hypertable/servers/master/log/mml/3) manually (hadoop fs) works, and writing a quick C app that links to libMapRClient and reads/writes to the file also works, but Hypertable consistently errors on this one file.

Has anyone seen "Resource temporarily unavailable" errors, or connection reset (104) errors before when using libMapRClient? And does anyone have any suggestions for how to see more log output from MapR to try and trace down what's happening? Thanks,

Will

Outcomes