AnsweredAssumed Answered

HostStats Down Alarm

Question asked by myacoub on Aug 17, 2016
Latest reply on Aug 23, 2016 by myacoub

Hello Community Manager

 

Actually I am recently Facing this issue on one of the Cluster Members and i don't expect that it may be related to the topology as there were no change taking place at all when this alarm rise

 

Below you will find the logs taken from /opt/mapr/logs/hoststats.log

 

Hoststats is shutting down with signal: 15. No further requests will be served**** starting hoststats **** args: 5660 /opt/mapr/server/data/TaskTracker.stats -S 1

 

isGatherStats=true

 

Setting continuous mode

 

2016-03-17 20:02:16,8446 Program: hoststats on Host:  IP: 0.0.0.0, Port: 1111, PID: 42694

 

2016-03-17 20:02:21,1229 ERROR Hoststats hoststats.cc:260 Thread: 139734076618496 Oops! Failed to connect to maprfs!  with error count since last shown error: 0

 

2016-03-22 16:58:59,4596 ERROR Hoststats hoststats.cc:260 Thread: 139734215849760 Failed to 'write'. Trying to reopen file /var/mapr/local/slave39/metrics/clustermetrics.2016-03-22.14: with error count since last shown error: 9

 

2016-04-30 22:45:06,6427 ERROR Hoststats hoststats.cc:260 Thread: 139734215849760 Failed to 'write'. Trying to reopen file /var/mapr/local/slave39/metrics/clustermetrics.2016-04-30.20: with error count since last shown error: 0

 

Hoststats is shutting down with signal: 15. No further requests will be served**** starting hoststats **** args: 5660 /opt/mapr/server/data/TaskTracker.stats -S 1

 

isGatherStats=true

 

Setting continuous mode

 

2016-05-01 01:45:27,2670 Program: hoststats on Host:  IP: 0.0.0.0, Port: 1111, PID: 18962

 

2016-05-01 01:45:31,6933 ERROR Hoststats hoststats.cc:260 Thread: 140671004051200 Oops! Failed to connect to maprfs!  with error count since last shown error: 0

 

2016-06-03 11:36:10,3536 ERROR Hoststats hoststats.cc:260 Thread: 140671069665056 Failed to 'write'. Trying to reopen file /var/mapr/local/slave39/metrics/clustermetrics.2016-06-03.09: with error count since last shown error: 8

 

Hoststats is shutting down with signal: 15. No further requests will be served**** starting hoststats **** args: 5660 /opt/mapr/server/data/TaskTracker.stats -S 1

 

isGatherStats=true

 

Setting continuous mode

 

2016-06-11 08:32:27,6932 Program: hoststats on Host:  IP: 0.0.0.0, Port: 1111, PID: 20212

 

2016-06-11 08:32:35,4899 ERROR Hoststats hoststats.cc:260 Thread: 140474924979968 Metrics path does not exist. Will not process data until it is  /var/mapr/local/slave39/metrics with error count since last shown error: 0

 

2016-08-03 02:02:05,9596 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to 'write'. Trying to reopen file /var/mapr/local/slave39/metrics/clustermetrics.2016-08-02.23: with error count since last shown error: 4

 

2016-08-03 02:09:00,1644 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.00: with error count since last shown error: 0

 

2016-08-03 02:17:13,4563 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.00: with error count since last shown error: 1

 

2016-08-03 02:24:55,1928 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.00: with error count since last shown error: 1

 

2016-08-03 02:33:57,1866 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.00: with error count since last shown error: 1

 

2016-08-03 02:42:24,5849 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.00: with error count since last shown error: 1

 

2016-08-03 02:50:43,7491 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.00: with error count since last shown error: 1

 

2016-08-03 02:58:43,2559 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.00: with error count since last shown error: 1

 

2016-08-03 03:06:51,4054 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.01: with error count since last shown error: 3

 

2016-08-03 03:14:53,3342 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.01: with error count since last shown error: 3

 

2016-08-03 03:23:01,0327 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.01: with error count since last shown error: 3

 

2016-08-03 03:30:31,6589 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.01: with error count since last shown error: 3

 

2016-08-03 03:37:32,7840 ERROR Hoststats hoststats.cc:260 Thread: 140474990593824 Failed to open for writing! /var/mapr/local/slave39/metrics/clustermetrics.2016-08-03.01: with error count since last shown error: 3

 

**** starting hoststats **** args: 5660 /opt/mapr/server/data/TaskTracker.stats -S 1

 

isGatherStats=true

 

Setting continuous mode

 

2016-08-04 00:53:28,9665 Program: hoststats on Host:  IP: 0.0.0.0, Port: 1111, PID: 25728

 

**** starting hoststats **** args: 5660 /opt/mapr/server/data/TaskTracker.stats -S 1

 

isGatherStats=true

 

Setting continuous mode

 

2016-08-08 14:30:50,0810 Program: hoststats on Host:  IP: 0.0.0.0, Port: 1111, PID: 34034

 

Please need your support to know how to fix this issue and what is the exact impact of such Alarm and what is the exact functionality of this Hoststat

 

Thanks and Best Regards

Outcomes