AnsweredAssumed Answered

MCS can't see CLDB, JT; CLDB can't see JT

Question asked by thealy on Nov 5, 2014
Latest reply on Nov 7, 2014 by nabeel
Running M3 / 3.1.0.23703.GA
Nodes running REdHat release 6.6 (Santiago)

Cluster ran mostly fine for over a year; a config mistake updating RedHat loaded 3.1.1 on several nodes and recovery attempts trashed the whole cluster. One of three ZooKeepers was replaced by another node, and configure run on all nodes to reflect that change. Still trying to recover; Unfortunately the Ubuntu node running adminui was also toasted. So without MCS, it is difficult to see what is broken where. Days - now weeks of searching here and trying many full re-installs have not resulted in any progress.

* CLDB running, sees approx 50% of ~55 nodes.
* Running "maprcli node list -columns csvc" on JT shows no services for 22 nodes;
* Running same command on CLDB shows services for those 22 nodes and more;

These symptoms suggest some issue with services not being seen or propagated by the ZooKeepers, but I can't get a handle on it.

Any suggestions appreciated.

Outcomes