AnsweredAssumed Answered

CLDB error on M3

Question asked by humphrey on Aug 7, 2013
Latest reply on Aug 8, 2013 by humphrey
I have three CLDB node
two of them can not be started,it roll restart all the time .

#cldb.log
----------

     2013-08-08 14:24:56,665 INFO CLDB [main]: CLDB Command line args: /opt/mapr/conf/cldb.conf
        2013-08-08 14:24:56,665 INFO CLDB [main]: CLDBInit: Initializing CLDB
        2013-08-08 14:24:56,666 INFO CLDB [main]: CLDBInit: Starting RPCServer on port 7222 with num thread 10 and heap size of 3962(MB)
        2013-08-08 14:24:56,699 INFO CLDB [main]: MapR BuildVersion: 2.1.3.20987.GA
        2013-08-08 14:24:56,699 INFO CLDB [main]: CLDBInit: Start CLDBServer
        2013-08-08 14:24:56,747 INFO CLDBServer [main]: CLDBInit: HostName: hph01.hadoop ServerId: 697989251083029968
        2013-08-08 14:24:56,747 INFO CLDBServer [main]: CLDBInit: Cluster name : maprcluster
        2013-08-08 14:24:56,757 INFO CLDBServer [main]: CLDB creds setting uid as 2020
        2013-08-08 14:24:56,758 INFO CLDBServer [main]: CLDB creds setting adding gid 16
        2013-08-08 14:24:56,758 INFO CLDBServer [main]: CLDB creds setting adding gid 33
        2013-08-08 14:24:56,758 INFO CLDBServer [main]: CLDB creds setting adding gid 2020
        2013-08-08 14:24:56,780 INFO CLDB [main]: CLDBState: CLDB State change : INITIAZING
        2013-08-08 14:24:56,802 INFO ZooKeeperClient [main]: ZooKeeperClient init: zk timeout = 30000 ms
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:zookeeper.version=3.3.6--1, built on 09/07/2012 18:16 GMT
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:host.name=hph01.hadoop
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:java.version=1.6.0_32
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:java.vendor=Sun Microsystems Inc.
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:java.home=/usr/java/jdk1.6.0_32/jre
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:java.class.path=/opt/mapr:/opt/mapr/conf:/opt/mapr/lib/JPam-1.1.jar:/opt/mapr/lib/adminuiapp-0.1.jar:
        /opt/mapr/lib/ant-1.7.1.jar:/opt/mapr/lib/antlr-2.7.7.jar:/opt/mapr/lib/baseutils-0.1.jar:/opt/mapr/lib/c3p0-0.9.1.2.jar:/opt/mapr/lib/cldb-0.1.jar:/opt/mapr/lib/clifr
        amework-0.1.jar:/opt/mapr/lib/commons-codec-1.5.jar:/opt/mapr/lib/commons-collections-3.2.1.jar:/opt/mapr/lib/commons-el-1.0.jar:/opt/mapr/lib/commons-email-1.2.jar:/o
        pt/mapr/lib/commons-lang-2.5.jar:/opt/mapr/lib/commons-logging-1.0.4.jar:/opt/mapr/lib/commons-logging-api-1.0.4.jar:/opt/mapr/lib/dom4j-1.6.1.jar:/opt/mapr/lib/eval-0
        .5.jar:/opt/mapr/lib/flexjson-2.1.jar:/opt/mapr/lib/globalfsck-0.1.jar:/opt/mapr/lib/google-collect-1.0.jar:/opt/mapr/lib/gson-2.1.jar:/opt/mapr/lib/hadoop-metrics-0.2
        0.2-dev.jar:/opt/mapr/lib/hadoop-metrics2-0.20.2-dev.jar:/opt/mapr/lib/hibernate-c3p0-3.3.1.GA.jar:/opt/mapr/lib/hibernate-commons-annotations-3.2.0.Final.jar:/opt/map
        r/lib/hibernate-core-3.6.8.Final.jar:/opt/mapr/lib/httpclient-4.2.jar:/opt/mapr/lib/httpclient-cache-4.2.jar:/opt/mapr/lib/httpcore-4.2.jar:/opt/mapr/lib/jasper-compil
        er-5.5.12.jar:/opt/mapr/lib/jasper-runtime-5.5.12.jar:/opt/mapr/lib/javassist-3.12.1.GA.jar:/opt/mapr/lib/jetty-6.1.26.jar:/opt/mapr/lib/jetty-plus-6.1.26.jar:/opt/map
        r/lib/jetty-util-6.1.26.jar:/opt/mapr/lib/jobmngmnt-0.1.jar:/opt/mapr/lib/joda-time-2.0.jar:/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/jsp-2.1.jar:/opt/mapr/lib/jsp
        -api-2.1.jar:/opt/mapr/lib/jta-1.1.jar:/opt/mapr/lib/junit-3.8.1.jar:/opt/mapr/lib/junit-4.5.jar:/opt/mapr/lib/kvstore-0.1.jar:/opt/mapr/lib/libprotodefs.jar:/opt/mapr
        /lib/log4j-1.2.14.jar:/opt/mapr/lib/log4j-1.2.15.jar:/opt/mapr/lib/logging-0.1.jar:/opt/mapr/lib/mail.jar:/opt/mapr/lib/maprbuildversion.jar:/opt/mapr/lib/maprcli-0.1.
        jar:/opt/mapr/lib/maprfs-1.0.3-mapr-2.1.3.2.jar:/opt/mapr/lib/maprfs-diagnostic-tools-1.0.3-mapr-2.1.3.2.jar:/opt/mapr/lib/maprsecurity-0.1.jar:/opt/mapr/lib/maprutil-
        0.1.jar:/opt/mapr/lib/persistence-api-1.0.jar:/opt/mapr/lib/protobuf-java-2.4.1-lite.jar:/opt/mapr/lib/servlet-api-2.5-6.1.26.jar:/opt/mapr/lib/volumemirror-0.1.jar:/o
        pt/mapr/lib/warden-0.1.jar:/opt/mapr/lib/zookeeper-3.3.6.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.
        1.jar
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:java.library.path=/opt/mapr/lib
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:java.io.tmpdir=/tmp
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:java.compiler=<NA>
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:os.name=Linux
        2013-08-08 14:24:56,817 INFO ZooKeeper [main]: Client environment:os.arch=amd64
        2013-08-08 14:24:56,818 INFO ZooKeeper [main]: Client environment:os.version=3.0.13-0.27-default
        2013-08-08 14:24:56,818 INFO ZooKeeper [main]: Client environment:user.name=mapr
        2013-08-08 14:24:56,818 INFO ZooKeeper [main]: Client environment:user.home=/home/mapr
        2013-08-08 14:24:56,818 INFO ZooKeeper [main]: Client environment:user.dir=/etc/init.d
        2013-08-08 14:24:56,820 INFO ZooKeeper [main]: Initiating client connection, connectString=hph01:5181,hph02:5181,hph03:5181,hph04:5181 sessionTimeout=30000 watcher=com
        .mapr.fs.cldb.CLDBServer@1f8166e5
        2013-08-08 14:24:56,861 INFO CLDBServer [main]: CLDB configured with ZooKeeper ensemble with connection string hph01:5181,hph02:5181,hph03:5181,hph04:5181
        2013-08-08 14:24:56,861 INFO ClientCnxn [main-SendThread()]: Opening socket connection to server hph02/192.168.0.254:5181
        2013-08-08 14:24:56,876 INFO ClientCnxn [main-SendThread(hph02:5181)]: Socket connection established to hph02/192.168.0.254:5181, initiating session
        2013-08-08 14:24:56,898 INFO ClientCnxn [main-SendThread(hph02:5181)]: Session establishment complete on server hph02/192.168.0.254:5181, sessionid = 0x140580eb3e700f1
        , negotiated timeout = 30000
        2013-08-08 14:24:56,902 INFO CLDBServer [main-EventThread]: The CLDB received notification that a ZooKeeper event of type None occurred on path null
        2013-08-08 14:24:56,917 INFO CLDBServer [main-EventThread]: onZKConnect: The CLDB has successfully connected to the ZooKeeper server State:CONNECTED Timeout:30000 sess
        ionid:0x140580eb3e700f1 local:/192.168.0.253:55420 remoteserver:hph02/192.168.0.254:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0 in th
        e ZooKeeper ensemble with connection string hph01:5181,hph02:5181,hph03:5181,hph04:5181
        2013-08-08 14:24:57,108 INFO CLDBServer [ZK-Connect]: Previous CLDB was not a clean shutdown waiting for 20000ms before attempting to become master
        2013-08-08 14:24:57,418 INFO VolumeMirror [main]: Initializing volume mirror thread ...
        2013-08-08 14:24:57,421 INFO VolumeMirror [main]: Spawned 1 VolumeMirror Threads
        2013-08-08 14:24:57,554 INFO HttpServer [main]: Creating listener for 0.0.0.0
        2013-08-08 14:24:57.585:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
        2013-08-08 14:24:57,661 INFO CLDB [main]: CLDBState: CLDB State change : WAIT_FOR_FILESERVERS
        2013-08-08 14:24:57,661 INFO CLDB [main]: CLDBInit: Exporting program 2346
        2013-08-08 14:24:57,661 INFO CLDB [main]: CLDBInit: Exporting program 2345
        2013-08-08 14:24:57,662 INFO CLDB [main]: CLDBInit: Starting HTTP Server
        2013-08-08 14:24:57,662 INFO HttpServer [main]: WebServer: Starting WebServer
        2013-08-08 14:24:57,664 INFO HttpServer [main]: Listener started on SelectChannelConnector@0.0.0.0:7221 port 7221
        2013-08-08 14:24:57,664 INFO HttpServer [main]: Starting Jetty WebServer
        2013-08-08 14:24:57.664:INFO::jetty-6.1.26
        2013-08-08 14:24:58.116:INFO::Started SelectChannelConnector@0.0.0.0:7221
        2013-08-08 14:25:05,145 INFO CLDBServer [Lookup-1]: Rejecting RPC 2345.5 from 192.168.0.253:58104 with status 30 as CLDB is not yet initialized.
        2013-08-08 14:25:17,124 INFO ZooKeeperClient [ZK-Connect]: ZooKeeperClient: KvStore is of latest epoch CLDB trying to become Master
        2013-08-08 14:25:17,128 INFO ZooKeeperClient [ZK-Connect]: ZooKeeperClient createActiveEphemeralMasterZNode: /datacenter/controlnodes/cldb/active/CLDBMaster already ex
        ists
        2013-08-08 14:25:17,128 INFO ZooKeeperClient [ZK-Connect]: ZooKeeperClient: Some other CLDB become master. Current CLDB is Slave
        2013-08-08 14:25:17,129 INFO ZooKeeperClient [ZK-Connect]: CLDB got role of slave
        2013-08-08 14:25:17,129 INFO CLDBServer [ZK-Connect]: Starting thread to become slave CLDB
        2013-08-08 14:25:20,135 INFO ZooKeeperClient [Becoming Slave Thread]: Waiting for local KvStoreContainer to become valid. KvStore ContainerInfo  Container ID:1 Master:
        192.168.0.1(2)-22(7062963176682162160) Servers:  192.168.0.1(2)-22(7062963176682162160) 192.168.0.253(3)-22(697989251083029968) 192.168.0.254(3)-22(3262400746715083979
        ) Inactive:  Unused:  Epoch:22 SizeMB:0 CLDB ServerID : 697989251083029968
        2013-08-08 14:25:20,135 INFO ZooKeeperClient [Becoming Slave Thread]: Local KvStoreContainer became valid. KvStore ContainerInfo  Container ID:1 Master:192.168.0.1(2)-
        22(7062963176682162160) Servers:  192.168.0.1(2)-22(7062963176682162160) 192.168.0.253(3)-22(697989251083029968) 192.168.0.254(3)-22(3262400746715083979) Inactive:  Un
        used:  Epoch:22 SizeMB:0 CLDB ServerID : 697989251083029968
        2013-08-08 14:25:20,138 INFO BecomeSlaveThread [Becoming Slave Thread]: IPAddress of local kvstore is 192.168.0.253:5660
        2013-08-08 14:25:20,179 INFO Table [Becoming Slave Thread]: KvStore Init: Opening Tables
        2013-08-08 14:25:20,245 INFO LicenseManager [Becoming Slave Thread]: using default license validator
        2013-08-08 14:25:20,329 INFO LicenseManager [Becoming Slave Thread]: 0x26: unique id: d4ae529e6cb4-44454C4C-4300-1047-8036-C7C04F463358-0001835009-0004718593-001139514
        2
        2013-08-08 14:25:20,329 FATAL BecomeSlaveThread [Becoming Slave Thread]: license not found for CLDB HA: shutting down
        2013-08-08 14:25:20,329 FATAL CLDB [Becoming Slave Thread]: CLDBShutdown: license not found for CLDB HA: shutting down
        2013-08-08 14:25:20,329 INFO CLDBServer [Becoming Slave Thread]: Shutdown: Stopping CLDB
        2013-08-08 14:25:20,330 INFO CLDB [Thread-11]: CLDB ShutDown Hook called
        2013-08-08 14:25:20,330 INFO ZooKeeperClient [Thread-11]: Zookeeper Client: Closing client connection:
        2013-08-08 14:25:20,334 INFO ZooKeeper [Thread-11]: Session: 0x140580eb3e700f1 closed
        2013-08-08 14:25:20,334 INFO ClientCnxn [main-EventThread]: EventThread shut down
        2013-08-08 14:25:20,334 INFO CLDB [Thread-11]: CLDB shutdown
    


----------
# maprcli dump zkinfo -json
    /datacenter/controlnodes/cldb/epoch/1/KvStoreContainerInfo":" Container ID:1 VolumeId:1 Master:192.168.0.1:5660-192.168.1.1:5660--22-VALID Servers:  192.168.0.1:5660-192.168.1.1:5660--22-VALID 192.168.0.253:5660-192.168.1.253:5660-132.40.130.81:5660--22-VALID 192.168.0.254:5660-192.168.1.254:5660-132.40.130.82:5660--22-VALID Inactive Servers:  Unused Servers:  Latest epoch:22"

Outcomes