AnsweredAssumed Answered

all cldb services shut down

Question asked by littonpeng on May 30, 2013
Latest reply on May 31, 2013 by nabeel
My Cluster run under M5 trial license. I installed the cldb service in two nodes. The cluster work well for days(tested the nfs HA). But the both cldb services shutdown themselves by unknown causes. I reboot all nodes, the cldb services on both nodes can run a while but then shutdown again. Please help to check . Following is the logs from cldb.log. Thanks.

Node1 :
<pre>
2013-05-31 18:53:49,594 INFO ClientCnxn [main-SendThread(s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Client session timed out, have not heard from server in 10032ms for sessionid 0x3efa384eac0001, closing socket connection and attempting reconnect
2013-05-31 18:53:50,123 INFO ClientCnxn [main-SendThread(s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Opening socket connection to server s01-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.200:5181
2013-05-31 18:54:00,133 INFO ClientCnxn [main-SendThread(s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Client session timed out, have not heard from server in 10439ms for sessionid 0x3efa384eac0001, closing socket connection and attempting reconnect
2013-05-31 18:54:01,731 INFO ClientCnxn [main-SendThread(s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Opening socket connection to server s02-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.202:5181
2013-05-31 18:54:01,732 INFO ClientCnxn [main-SendThread(s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Socket connection established to s02-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.202:5181, initiating session
2013-05-31 18:54:01,733 INFO ClientCnxn [main-SendThread(s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Unable to read additional data from server sessionid 0x3efa384eac0001, likely server has closed socket, closing socket connection and attempting reconnect
2013-05-31 18:54:02,632 INFO ClientCnxn [main-SendThread(s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Opening socket connection to server s03-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.204:5181
2013-05-31 18:54:11,836 INFO ClientCnxn [main-SendThread(s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Client session timed out, have not heard from server in 10003ms for sessionid 0x3efa384eac0001, closing socket connection and attempting reconnect
2013-05-31 18:54:11,937 FATAL CLDB [ZK-Connect]: CLDBShutdown: ZooKeeperClient : KvStoreContainerInfo read failed from ZooKeeper. Stopping CLDB
2013-05-31 18:54:11,937 INFO CLDBServer [ZK-Connect]: Shutdown: Stopping CLDB
2013-05-31 18:54:11,938 INFO CLDB [Thread-11]: CLDB ShutDown Hook called
2013-05-31 18:54:11,938 INFO ZooKeeperClient [Thread-11]: Zookeeper Client: Closing client connection:
2013-05-31 18:54:12,366 INFO ClientCnxn [main-SendThread(s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Opening socket connection to server s01-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.200:5181
2013-05-31 18:54:12,849 INFO CLDBServer [RPC-6]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:13,850 INFO CLDBServer [RPC-7]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:14,852 INFO CLDBServer [RPC-8]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:15,854 INFO CLDBServer [RPC-9]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:16,857 INFO CLDBServer [RPC-10]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:17,859 INFO CLDBServer [RPC-1]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:18,861 INFO CLDBServer [RPC-2]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:19,863 INFO CLDBServer [RPC-3]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:20,866 INFO CLDBServer [RPC-4]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:21,868 INFO CLDBServer [RPC-5]: Rejecting RPC 2345.31 from 192.168.2.202:5660 with status 30 as this CLDB is shutting down.
2013-05-31 18:54:22,046 INFO ZooKeeper [Thread-11]: Session: 0x3efa384eac0001 closed
2013-05-31 18:54:22,046 INFO CLDB [Thread-11]: CLDB shutdown
</pre>

Node2:
<pre>
2013-05-31 18:51:45,307 INFO ClientCnxn [main-SendThread(s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Socket connection established to s02-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.202:5181, initiating session
2013-05-31 18:51:45,309 INFO ClientCnxn [main-SendThread(s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Unable to read additional data from server sessionid 0x3efa3623840001, likely server has closed socket, closing socket connection and attempting reconnect
2013-05-31 18:51:46,303 INFO ClientCnxn [main-SendThread(s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Opening socket connection to server s01-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.200:5181
2013-05-31 18:51:46,304 INFO ClientCnxn [main-SendThread(s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Socket connection established to s01-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.200:5181, initiating session
2013-05-31 18:51:46,305 INFO ClientCnxn [main-SendThread(s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Unable to read additional data from server sessionid 0x3efa3623840001, likely server has closed socket, closing socket connection and attempting reconnect
2013-05-31 18:51:46,775 FATAL CLDB [ZK-Connect]: CLDBShutdown: The CLDB is going to shut down now because it could not connect to the ZooKeeper ensemble with connection string s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181,s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181 within 10 seconds
2013-05-31 18:51:46,775 INFO CLDBServer [ZK-Connect]: Shutdown: Stopping CLDB
2013-05-31 18:51:46,776 INFO CLDB [Thread-11]: CLDB ShutDown Hook called
2013-05-31 18:51:46,776 INFO ZooKeeperClient [Thread-11]: Setting the clean cldbshutdown flag to true
2013-05-31 18:51:46,987 INFO ClientCnxn [main-SendThread(s01-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Opening socket connection to server s03-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.204:5181
2013-05-31 18:51:46,987 INFO ClientCnxn [main-SendThread(s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Socket connection established to s03-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.204:5181, initiating session
2013-05-31 18:51:46,988 INFO ClientCnxn [main-SendThread(s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Unable to read additional data from server sessionid 0x3efa3623840001, likely server has closed socket, closing socket connection and attempting reconnect
2013-05-31 18:51:46,995 INFO CLDBServer [Lookup-8]: Rejecting RPC 2345.4 from 192.168.2.200:1111 with status 30 as this CLDB is shutting down.
2013-05-31 18:51:47,089 ERROR ZooKeeperClient [Thread-11]: setCldbCleanShutdown failed to update value on znode /datacenter/controlnodes/cldb/active/MasterInfofailed with exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /datacenter/controlnodes/cldb/active/MasterInfo
2013-05-31 18:51:47,089 INFO ZooKeeperClient [Thread-11]: Zookeeper Client: Closing client connection:
2013-05-31 18:51:47,099 INFO ClientCnxn [main-SendThread(s03-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Opening socket connection to server s02-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.202:5181
2013-05-31 18:51:47,100 INFO ClientCnxn [main-SendThread(s02-c1-p001-dgzmt.n01.cloudiaas.esin:5181)]: Socket connection established to s02-c1-p001-dgzmt.n01.cloudiaas.esin/192.168.2.202:5181, initiating session
2013-05-31 18:51:47,108 INFO CLDBServer [Lookup-1]: Rejecting RPC 2345.5 from 192.168.2.202:53020 with status 30 as this CLDB is shutting down.
2013-05-31 18:51:47,201 INFO ZooKeeper [Thread-11]: Session: 0x3efa3623840001 closed
2013-05-31 18:51:47,201 INFO ClientCnxn [main-EventThread]: EventThread shut down
2013-05-31 18:51:47,201 INFO CLDB [Thread-11]: CLDB shutdown
</pre>

Outcomes