AnsweredAssumed Answered

CLDBShutdown: Currrent CLDB's KvStore Epoch is not latest

Question asked by frank on Jun 28, 2012
Latest reply on Apr 30, 2013 by nabeel
I am installing M3 on cluster with two 2 nodes. However, when i am starting CLDB on node1, I have error as following. How to resolve this?


    2012-06-29 15:38:38,102 INFO  com.mapr.fs.cldb.CLDB [main]: Initializing CLDB
    2012-06-29 15:38:38,103 INFO  com.mapr.fs.cldb.CLDB [main]: Loading properties file : /opt/mapr/conf/cldb.conf
    2012-06-29 15:38:38,298 INFO  com.mapr.fs.cldb.counters.CLDBMetrics [main]: Initializing CLDB Metrics with serviceName: cldbServer
    2012-06-29 15:38:38,313 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBInit: Using hostname file /opt/mapr/hostname
    2012-06-29 15:38:38,313 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBInit: Using hostid file /opt/mapr/hostid
    2012-06-29 15:38:38,313 INFO  com.mapr.fs.cldb.CLDB [main]: CLDB Startup
    2012-06-29 15:38:38,313 INFO  com.mapr.fs.cldb.CLDB [main]: CLDB Properties from configuration file: {cldb.jmxremote.port=7220, cldb.web.port=7221, cldb.port=7222, cldb.zookeeper.servers=mapr1:5181,mapr2:5181, cldb.numthreads=10, cldb.min.fileservers=1, hadoop.version=0.20.2}
    2012-06-29 15:38:38,313 INFO  com.mapr.fs.cldb.CLDB [main]: CLDB Command line args: /opt/mapr/conf/cldb.conf
    2012-06-29 15:38:38,313 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBInit: Initializing CLDB
    2012-06-29 15:38:38,314 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBInit: Starting RPCServer on port 7222 with num thread 10 and heap size of 123(MB)
    2012-06-29 15:38:38,349 INFO  com.mapr.fs.cldb.CLDB [main]: MapR BuildVersion: 1.2.7.14133.GA
    2012-06-29 15:38:38,349 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBInit: Start CLDBServer
    2012-06-29 15:38:38,375 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDBInit: HostName : mapr1
    2012-06-29 15:38:38,375 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDBInit: ServerId: 3154220128610019474
    2012-06-29 15:38:38,376 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDBInit: Cluster name : my.cluster.com
    2012-06-29 15:38:38,383 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB creds setting uid as 0
    2012-06-29 15:38:38,383 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB creds setting adding gid 0
    2012-06-29 15:38:38,384 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB creds setting adding gid 1
    2012-06-29 15:38:38,384 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB creds setting adding gid 2
    2012-06-29 15:38:38,384 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB creds setting adding gid 3
    2012-06-29 15:38:38,384 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB creds setting adding gid 4
    2012-06-29 15:38:38,384 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB creds setting adding gid 6
    2012-06-29 15:38:38,384 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB creds setting adding gid 10
    2012-06-29 15:38:38,384 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBState: CLDB State change : INITIAZING
    2012-06-29 15:38:38,388 INFO  com.mapr.fs.cldb.CLDBServer [main]: Read 2 nodes from the node list file
    2012-06-29 15:38:38,394 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [main]: ZooKeeperClient init: zk timeout = 30000 ms
    2012-06-29 15:38:38,450 INFO  com.mapr.fs.cldb.CLDBServer [main]: CLDB Init: ZooKeeper Servers configured. Connected to mapr1:5181,mapr2:5181
    2012-06-29 15:38:38,487 INFO  com.mapr.fs.cldb.CLDBServer [main-EventThread]: ZooKeeper event None on path null
    2012-06-29 15:38:38,496 INFO  com.mapr.fs.cldb.CLDBServer [main-EventThread]: onZKConnect: Received connect from ZooKeeper.
    2012-06-29 15:38:38,539 INFO  com.mapr.fs.cldb.ActiveContainersMap [main]: Initializing containers cache with 1000000 number of entires
    2012-06-29 15:38:39,142 INFO  com.mapr.fs.cldb.CLDBServer [main-EventThread]: onZKConnect: Established connection to ZooKeeper
    2012-06-29 15:38:39,235 INFO  com.mapr.volumemirror.VolumeMirror [main]: Initializing volume mirror thread ...
    2012-06-29 15:38:39,302 INFO  com.mapr.fs.cldb.http.HttpServer [main]: Creating listener for 0.0.0.0
    2012-06-29 15:38:39.321::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
    2012-06-29 15:38:39,381 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBState: CLDB State change : WAIT_FOR_FILESERVERS
    2012-06-29 15:38:39,382 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBInit: Exporting program 2345
    2012-06-29 15:38:39,382 INFO  com.mapr.fs.cldb.CLDB [main]: CLDBInit: Starting HTTP Server
    2012-06-29 15:38:39,382 INFO  com.mapr.fs.cldb.CLDBServer [main]: Init: Start HTTP Server
    2012-06-29 15:38:39,382 INFO  com.mapr.fs.cldb.http.HttpServer [main]: WebServer: Starting WebServer
    2012-06-29 15:38:39,384 INFO  com.mapr.fs.cldb.http.HttpServer [main]: Lisetner started on SelectChannelConnector@0.0.0.0:7221 port 7221
    2012-06-29 15:38:39,384 INFO  com.mapr.fs.cldb.http.HttpServer [main]: Starting Jetty WebServer
    2012-06-29 15:38:39.384::INFO:  jetty-6.1.14
    2012-06-29 15:38:40.155::INFO:  Started SelectChannelConnector@0.0.0.0:7221
    2012-06-29 15:38:40,156 INFO  com.mapr.fs.cldb.CLDBServer [main]: HTTP Server
    2012-06-29 15:38:48,700 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 31 from 16.158.83.183:5660 Generating reply with status: 30
    2012-06-29 15:38:48,930 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 31 from 16.158.82.1:5660 Generating reply with status: 30
    2012-06-29 15:38:59,173 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: ZooKeeperClient: CLDB has latest epoch. Checking cleanbit
    2012-06-29 15:38:59,173 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: ZooKeeperClient: KvStore is clean and of latest epoch CLDB trying to become Master
    2012-06-29 15:38:59,181 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: ZooKeeperClient: CLDB is current Master
    2012-06-29 15:38:59,181 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: CLDB became master. Initializing KvStoreContainer for cid: 1
    2012-06-29 15:38:59,183 FATAL com.mapr.fs.cldb.CLDB [Wait for ZooKeeper Connected thread]: CLDBShutdown: Currrent CLDB's KvStore Epoch is not latest. KvStore epoch: 2 latest epoch: 3 Shutting down CLDB
    2012-06-29 15:38:59,183 INFO  com.mapr.fs.cldb.CLDBServer [Wait for ZooKeeper Connected thread]: Shutdown: Stopping CLDB
    2012-06-29 15:38:59,184 INFO  com.mapr.fs.cldb.CLDB [Thread-10]: CLDB ShutDown Hook called
    2012-06-29 15:38:59,184 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Thread-10]: Zookeeper Client: Closing client connection:
    2012-06-29 15:38:59,207 INFO  com.mapr.fs.cldb.CLDBServer [main-EventThread]: ZooKeeper event NodeDeleted on path /datacenter/controlnodes/cldb/active/CLDBMaster
    2012-06-29 15:38:59,207 INFO  com.mapr.fs.cldb.CLDB [Thread-10]: CLDB shutdown

Outcomes