AnsweredAssumed Answered

Cluster Down

Question asked by dannapureddy on Sep 9, 2013
Latest reply on Sep 9, 2013 by Ted Dunning
NODE_ALARM_METRICS_WRITE_PROBLEM

Cluster went down and observed below messages from error file and logs.
Error Message from /opt/mapr/logs$ vi hoststats.err

<code>

2013-02-28 15:47:24,7245 ERROR Client fs/client/fileclient/cc/client.cc:520 Thread: 140347603330816 Create failed for file clustermetrics.2013-02-28.15:00:00, error Resource temporarily unavailable(11)
2013-02-28 15:47:24,7246 ERROR LibHdfsApiSupport fs/client/fileclient/cc/libhdfs/api_support.cc:439 Thread: 140347603330816 Create failed for file /var/mapr/local/osblks-ch2-a05.sys.comcast.net/metrics/clusterm, error: Resource temporarily unavailable(11)

</code>
Log Message from /opt/mapr/logs$ vi hoststats.log

<code>
2013-08-22 00:09:44,9523 ERROR Hoststats hoststats.cc:2071 Thread: 140028535088896 Exception while trying to insert Task data: ERROR Description: Table has no partition for value 20130822
2013-08-22 00:09:45,2600 ERROR Hoststats hoststats.cc:2274 Thread: 140028535088896 Exception while trying to insert Task Attempt data: ERROR Description: Table has no partition for value 20130822
2013-08-22 00:09:49,8690 ERROR Hoststats hoststats.cc:2427 Thread: 140028535088896 Soci Error: Table has no partition for value 20130822
2013-08-23 01:26:08,7303 ERROR Hoststats hoststats.cc:2071 Thread: 140028535088896 Exception while trying to insert Task data: ERROR Description: Table has no partition for value 20130823
2013-08-23 01:26:09,0404 ERROR Hoststats hoststats.cc:2274 Thread: 140028535088896 Exception while trying to insert Task Attempt data: ERROR Description: Table has no partition for value 20130823
2013-08-23 01:26:12,9724 ERROR Hoststats hoststats.cc:2427 Thread: 140028535088896 Soci Error: Table has no partition for value 20130823
2013-08-24 18:07:29,7031 ERROR Hoststats hoststats.cc:2071 Thread: 140028535088896 Exception while trying to insert Task data: ERROR Description: Table has no partition for value 20130824
2013-08-24 18:07:29,7351 ERROR Hoststats hoststats.cc:2274 Thread: 140028535088896 Exception while trying to insert Task Attempt data: ERROR Description: Table has no partition for value 20130824
</code>

Outcomes