AnsweredAssumed Answered

M3 CLDB down. Replicas stored on CLDB server?

Question asked by stormcrow on Feb 23, 2014
A failing drive on our cldb server hosts /opt/mapr, swap, and a smaller mfs parition. The CLDB server will no longer start because it cannot load up its local mfs

    2014-02-23 11:21:15,297 FATAL CLDB [WaitForLocalKvstore Thread]: CLDBShutdown: CLDB had master lock and was waiting for its local mfs to become Master.Waited for 7 (minutes) but mfs did not become Master. Shutting down CLDB to release master lock.

When I restart the zookeepers, it appears the cldbnodes only reside on the cldb server (192.168.3.147):

    [root@job ~]# maprcli dump cldbnodes -cluster cluster -zkconnect 192.168.3.148:5181
    *** warn: MAPR_SUBNETS set to 192.168.3.0/24
    valid                invalid
    192.168.3.147:5660-  ...
    192.168.3.147:5660-  ...

Am I reading that correctly? Do we have two copies of cid 1 on the cldb server? How can I copy one of these volumes to another node and promote it to the cldb server? The CLDB server boots, but is clearly in rough shape.

Is it possible to rebuild the CLDB server without container 1? If so, how would I do this, and how long would the process take? We are running M3.

Outcomes