AnsweredAssumed Answered

CLDB Error after downgrade from M5 trial to M3

Question asked by littonpeng on Jun 30, 2013
Latest reply on Jul 1, 2013 by littonpeng
I have 3 nodes run under M5 trial. After the license expired, i just keep one nfs server and one cldb server. I reconfigured all nodes and reboot them. But the cldb servie can not run from that on. I'd like to know it's an expected action or someting wrong?
Thanks.

== The cldb log ==
<pre>
Header: hostName: s03-c1-p001-dgzmt.n01.cloudiaas.esin, Time Zone: China Standard Time, processName: cldb, processId: 30310, MapR Build Version: 2.1.3.19871.GA
2013-07-01 16:22:05,972 INFO CLDB [main]: Loading properties file : /opt/mapr/conf/cldb.conf
2013-07-01 16:22:06,091 INFO CLDBMetrics [main]: Initializing CLDB Metrics with serviceName: cldbServer
2013-07-01 16:22:06,094 INFO CLDB [main]: CLDBInit: Using hostname file /opt/mapr/hostname and hostid file /opt/mapr/hostid
2013-07-01 16:22:06,094 INFO CLDB [main]: CLDB Properties from configuration file: {cldb.web.port=7221, cldb.zookeeper.servers=s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181, cldb.numthreads=10, hadoop.version=0.20.2, cldb.port=7222, cldb.min.fileservers=1, cldb.detect.dup.hostid.enabled=false, num.volmirror.threads=1, cldb.jmxremote.port=7220}
2013-07-01 16:22:06,094 INFO CLDB [main]: CLDB Command line args: /opt/mapr/conf/cldb.conf
2013-07-01 16:22:06,094 INFO CLDB [main]: CLDBInit: Initializing CLDB
2013-07-01 16:22:06,095 INFO CLDB [main]: CLDBInit: Starting RPCServer on port 7222 with num thread 10 and heap size of 2545(MB)
*** warn: MAPR_SUBNETS set to 192.168.2.0/24
2013-07-01 16:22:06,108 INFO CLDB [main]: MapR BuildVersion: 2.1.3.19871.GA
2013-07-01 16:22:06,108 INFO CLDB [main]: CLDBInit: Start CLDBServer
2013-07-01 16:22:06,129 INFO CLDBServer [main]: CLDBInit: HostName: s03-c1-p001-dgzmt.n01.cloudiaas.esin ServerId: 209710224993071387
2013-07-01 16:22:06,129 INFO CLDBServer [main]: CLDBInit: Cluster name : c1-p001-dgzmt
2013-07-01 16:22:06,134 INFO CLDBServer [main]: CLDB creds setting uid as 888888
2013-07-01 16:22:06,134 INFO CLDBServer [main]: CLDB creds setting adding gid 500
2013-07-01 16:22:06,134 INFO CLDBServer [main]: CLDB creds setting adding gid 888888
prog: 2345, proc 31, RpcProgram not found
2013-07-01 16:22:06,146 INFO CLDB [main]: CLDBState: CLDB State change : INITIAZING
2013-07-01 16:22:06,158 INFO ZooKeeperClient [main]: ZooKeeperClient init: zk timeout = 30000 ms
2013-07-01 16:22:06,165 INFO ZooKeeper [main]: Client environment:zookeeper.version=3.3.6--1, built on 09/07/2012 18:16 GMT
2013-07-01 16:22:06,165 INFO ZooKeeper [main]: Client environment:host.name=s03-c1-p001-dgzmt.n01.cloudiaas.esin
2013-07-01 16:22:06,165 INFO ZooKeeper [main]: Client environment:java.version=1.6.0_24
2013-07-01 16:22:06,165 INFO ZooKeeper [main]: Client environment:java.vendor=Sun Microsystems Inc.
2013-07-01 16:22:06,165 INFO ZooKeeper [main]: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:java.class.path=/opt/mapr:/opt/mapr/conf:/opt/mapr/lib/adminuiapp-0.1.jar:/opt/mapr/lib/ant-1.7.1.jar:/opt/mapr/lib/antlr-2.7.7.jar:/opt/mapr/lib/baseutils-0.1.jar:/opt/mapr/lib/c3p0-0.9.1.2.jar:/opt/mapr/lib/cldb-0.1.jar:/opt/mapr/lib/cliframework-0.1.jar:/opt/mapr/lib/commons-codec-1.5.jar:/opt/mapr/lib/commons-collections-3.2.1.jar:/opt/mapr/lib/commons-el-1.0.jar:/opt/mapr/lib/commons-email-1.2.jar:/opt/mapr/lib/commons-lang-2.5.jar:/opt/mapr/lib/commons-logging-1.0.4.jar:/opt/mapr/lib/commons-logging-api-1.0.4.jar:/opt/mapr/lib/dom4j-1.6.1.jar:/opt/mapr/lib/eval-0.5.jar:/opt/mapr/lib/flexjson-2.1.jar:/opt/mapr/lib/globalfsck-0.1.jar:/opt/mapr/lib/google-collect-1.0.jar:/opt/mapr/lib/gson-2.1.jar:/opt/mapr/lib/hadoop-metrics-0.20.2-dev.jar:/opt/mapr/lib/hadoop-metrics2-0.20.2-dev.jar:/opt/mapr/lib/hibernate-c3p0-3.3.1.GA.jar:/opt/mapr/lib/hibernate-commons-annotations-3.2.0.Final.jar:/opt/mapr/lib/hibernate-core-3.6.8.Final.jar:/opt/mapr/lib/httpclient-4.2.jar:/opt/mapr/lib/httpclient-cache-4.2.jar:/opt/mapr/lib/httpcore-4.2.jar:/opt/mapr/lib/jasper-compiler-5.5.12.jar:/opt/mapr/lib/jasper-runtime-5.5.12.jar:/opt/mapr/lib/javassist-3.12.1.GA.jar:/opt/mapr/lib/jetty-6.1.26.jar:/opt/mapr/lib/jetty-plus-6.1.26.jar:/opt/mapr/lib/jetty-util-6.1.26.jar:/opt/mapr/lib/jobmngmnt-0.1.jar:/opt/mapr/lib/joda-time-2.0.jar:/opt/mapr/lib/JPam-1.1.jar:/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/jsp-2.1.jar:/opt/mapr/lib/jsp-api-2.1.jar:/opt/mapr/lib/jta-1.1.jar:/opt/mapr/lib/junit-3.8.1.jar:/opt/mapr/lib/junit-4.5.jar:/opt/mapr/lib/kvstore-0.1.jar:/opt/mapr/lib/libprotodefs.jar:/opt/mapr/lib/log4j-1.2.14.jar:/opt/mapr/lib/log4j-1.2.15.jar:/opt/mapr/lib/logging-0.1.jar:/opt/mapr/lib/mail.jar:/opt/mapr/lib/maprbuildversion.jar:/opt/mapr/lib/maprcli-0.1.jar:/opt/mapr/lib/maprfs-diagnostic-tools-0.20.2-2.1.3.jar:/opt/mapr/lib/maprfs-jni-0.20.2-2.1.3.jar:/opt/mapr/lib/maprfs-jni-0.20.2-2.1.3-tests.jar:/opt/mapr/lib/maprsecurity-0.1.jar:/opt/mapr/lib/maprutil-0.1.jar:/opt/mapr/lib/persistence-api-1.0.jar:/opt/mapr/lib/protobuf-java-2.4.1-lite.jar:/opt/mapr/lib/servlet-api-2.5-6.1.26.jar:/opt/mapr/lib/volumemirror-0.1.jar:/opt/mapr/lib/warden-0.1.jar:/opt/mapr/lib/zookeeper-3.3.6.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.1.jar
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:java.library.path=/opt/mapr/lib
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:java.io.tmpdir=/tmp
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:java.compiler=<NA>
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:os.name=Linux
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:os.arch=amd64
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:os.version=2.6.32-358.el6.x86_64
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:user.name=mapr
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:user.home=/home/mapr
2013-07-01 16:22:06,166 INFO ZooKeeper [main]: Client environment:user.dir=/etc/rc.d/init.d
2013-07-01 16:22:06,167 INFO ZooKeeper [main]: Initiating client connection, connectString=s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181 sessionTimeout=30000 watcher=com.mapr.fs.cldb.CLDBServer@3e018c74
2013-07-01 16:22:06,185 INFO CLDBServer [main]: CLDB configured with ZooKeeper ensemble with connection string s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181
2013-07-01 16:22:06,185 INFO ClientCnxn [main-SendThread()]: Opening socket connection to server s01-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.200:5181
2013-07-01 16:22:06,193 INFO ClientCnxn [main-SendThread(s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Socket connection established to s01-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.200:5181, initiating session
2013-07-01 16:22:06,203 INFO ClientCnxn [main-SendThread(s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Session establishment complete on server s01-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.200:5181, sessionid = 0x3f993870b70012, negotiated timeout = 30000
2013-07-01 16:22:06,205 INFO CLDBServer [main-EventThread]: The CLDB received notification that a ZooKeeper event of type None occurred on path null
2013-07-01 16:22:06,211 INFO CLDBServer [main-EventThread]: onZKConnect: The CLDB has successfully connected to the ZooKeeper server State:CONNECTED Timeout:30000 sessionid:0x3f993870b70012 local:/192.168.2.204:39683 remoteserver:s01-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.200:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0 in the ZooKeeper ensemble with connection string s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181
prog: 2345, proc 31, RpcProgram not found
2013-07-01 16:22:06,305 INFO VolumeMirror [main]: Initializing volume mirror thread ...
2013-07-01 16:22:06,308 INFO VolumeMirror [main]: Spawned 1 VolumeMirror Threads
2013-07-01 16:22:06,320 INFO ZooKeeperClient [ZK-Connect]: ZooKeeperClient: KvStore does not have latest epoch  CLDB trying to wait until it is Ready
2013-07-01 16:22:06,325 INFO HttpServer [main]: Creating listener for 0.0.0.0
2013-07-01 16:22:06.341:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
2013-07-01 16:22:06,388 INFO CLDB [main]: CLDBState: CLDB State change : WAIT_FOR_FILESERVERS
2013-07-01 16:22:06,388 INFO CLDB [main]: CLDBInit: Exporting program 2346
2013-07-01 16:22:06,388 INFO CLDB [main]: CLDBInit: Exporting program 2345
2013-07-01 16:22:06,388 INFO CLDB [main]: CLDBInit: Starting HTTP Server
2013-07-01 16:22:06,388 INFO HttpServer [main]: WebServer: Starting WebServer
2013-07-01 16:22:06,390 INFO HttpServer [main]: Listener started on SelectChannelConnector@0.0.0.0:7221 port 7221
2013-07-01 16:22:06,390 INFO HttpServer [main]: Starting Jetty WebServer
2013-07-01 16:22:06.390:INFO::jetty-6.1.26
2013-07-01 16:22:06,543 INFO CLDBServer [RPC-1]: Rejecting RPC 2345.89 from 192.168.2.204:56861 with status 30 as CLDB is not yet initialized.
2013-07-01 16:22:06.704:INFO::Started SelectChannelConnector@0.0.0.0:7221
2013-07-01 16:22:09,324 INFO ZooKeeperClient [ZK-Connect]: Waiting for local KvStoreContainer to become valid. KvStore ContainerInfo  Container ID:1 Master:192.168.2.202-33(1167715975377370690) Servers:  192.168.2.202-33(1167715975377370690) Inactive:  192.168.2.204-31(209710224993071387) 192.168.2.200-32(2149437202928348961) Unused:  Epoch:33 SizeMB:0 CLDB ServerID : 209710224993071387
2013-07-01 16:23:07,895 INFO CLDBServer [RPC-10]: Rejecting RPC 2345.103 from 192.168.2.202:46646 with status 30 as CLDB is not yet initialized.
</pre>

== the mfs log ==
<pre>
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:730 x.x.0.0:0 SP SP4:/dev/sdk Delete empty containers for cid-chain with cid 2566
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:759 x.x.0.0:0 SP SP4:/dev/sdk Cidmap Loaded cid 2566 rootblk: 0x550058
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:730 x.x.0.0:0 SP SP4:/dev/sdk Delete empty containers for cid-chain with cid 2570
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:759 x.x.0.0:0 SP SP4:/dev/sdk Cidmap Loaded cid 2570 rootblk: 0x550020
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:730 x.x.0.0:0 SP SP4:/dev/sdk Delete empty containers for cid-chain with cid 2577
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:759 x.x.0.0:0 SP SP4:/dev/sdk Cidmap Loaded cid 2577 rootblk: 0x550080
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:730 x.x.0.0:0 SP SP4:/dev/sdk Delete empty containers for cid-chain with cid 2581
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:759 x.x.0.0:0 SP SP4:/dev/sdk Cidmap Loaded cid 2581 rootblk: 0x550098
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:730 x.x.0.0:0 SP SP4:/dev/sdk Delete empty containers for cid-chain with cid 2588
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:759 x.x.0.0:0 SP SP4:/dev/sdk Cidmap Loaded cid 2588 rootblk: 0xd9b68
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:730 x.x.0.0:0 SP SP4:/dev/sdk Delete empty containers for cid-chain with cid 2592
2013-07-01 16:22:04,4911 INFO  loadcidmap.cc:759 x.x.0.0:0 SP SP4:/dev/sdk Cidmap Loaded cid 2592 rootblk: 0xd9f80
2013-07-01 16:22:04,4911 INFO  spinit.cc:767 x.x.0.0:0 SP SP4:/dev/sdk Containers loaded
2013-07-01 16:22:04,4911 INFO  spinit.cc:775 x.x.0.0:0 Deleting empty container done for sp SP4:/dev/sdk
2013-07-01 16:22:04,4911 INFO  spinit.cc:1002 x.x.0.0:0 SP SP4:/dev/sdk Initialized
2013-07-01 16:22:04,4911 INFO  loadsp.cc:166 x.x.0.0:0 Done loading disks from disktab
2013-07-01 16:22:04,4911 INFO  iomgr.cc:2531 x.x.0.0:0 Refresh disktab state: old state: 0 0, failed SPs: 0, failed disks: 0
2013-07-01 16:22:05,6147 ERROR  cldbha.cc:675 x.x.0.0:49 Got error Connection reset by peer (104) while trying to register with CLDB 192.168.2.204:7222
2013-07-01 16:22:06,6228 ERROR  cldbha.cc:675 x.x.0.0:0 Got error Read-only file system (30) while trying to register with CLDB 192.168.2.204:7222

Outcomes