AnsweredAssumed Answered

CLDB - not available

Question asked by gabi_k on Jan 28, 2013
Latest reply on Jul 15, 2013 by nabeel
Hi, 
I restarted the CLDB node, and now i cannot start CLDB.. 
please look at the following cldb.log:
<pre>
2013-01-29 16:45:13,443 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 40 from 106.103.1.85:38293 Generating reply with status: 30
2013-01-29 16:45:14,927 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: ZooKeeperClient: CLDB has latest epoch. Checking cleanbit
2013-01-29 16:45:14,927 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: ZooKeeperClient: KvStore is clean and of latest epoch CLDB trying to become Master
2013-01-29 16:45:14,931 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: ZooKeeperClient: CLDB is current Master
2013-01-29 16:45:14,931 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: CLDB became master. Initializing KvStoreContainer for cid: 1
2013-01-29 16:45:14,933 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Wait for ZooKeeper Connected thread]: Storing KvStoreContainerInfo to ZooKeeper  Container ID:1 Servers:  106.103.1.81:5660--10-VALID(1369973134422650278) Inactive Servers:  106.103.1.85:5660--10-VALID(957510465599090838) 106.103.1.84:5660--10-VALID(5308322330462534014) 106.103.1.82:5660--10-VALID(4093149274661164202) 106.103.1.83:5660--10-VALID(8628902522623566439) Unused Servers:  Latest epoch:10 SizeMB:0
2013-01-29 16:45:14,938 INFO  com.mapr.fs.cldb.CLDBServer [Wait for ZooKeeper Connected thread]: Starting thread to monitor waiting for local kvstore to become master
2013-01-29 16:45:15,718 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 103 from 106.103.1.81:37296 Generating reply with status: 3
2013-01-29 16:45:27,566 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 40 from 106.103.1.81:51244 Generating reply with status: 3
2013-01-29 16:45:27,643 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: FSRegister: Request  FSID: 8628902522623566439 FSNetworkLocation: / FSHost:Port: 106.103.1.83:5660- FSHostName: joya-03 StoragePools 95947cb955f9bba10050e04b9b08a536-ce0ab0edd6eb1a980050e04b9a07c042-e0b9f5af2cc995a20050e04b9c080eaf-e29bd9c38a10830e0050e04b99080531- Capacity: 18298924 Available: 11241035 Used: 7057889 Role: 0 isDCA: false Received registration request
2013-01-29 16:45:27,644 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: FSRegister: CLDB waiting for local mfs to register and become master, requesting fileserver 106.103.1.83:5660- FSID: 8628902522623566439 to try again by returning ESRCH
2013-01-29 16:45:28,203 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 103 from 106.103.1.82:45056 Generating reply with status: 3
2013-01-29 16:45:29,156 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 40 from 106.103.1.85:58306 Generating reply with status: 3
2013-01-29 16:45:29,731 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: FSRegister: Request  FSID: 5308322330462534014 FSNetworkLocation: / FSHost:Port: 106.103.1.84:5660- FSHostName: joya-04 StoragePools 1057cdf1df6aa2880050e04cf6055fdd-625668cee9ceb4ab0050e04cf704ecef-4229545d2fbc7db20050e04cf8057973-77f5b1541f2509370050e04cf9045b07- Capacity: 18298924 Available: 11220335 Used: 7078589 Role: 0 isDCA: false Received registration request
2013-01-29 16:45:29,731 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: FSRegister: CLDB waiting for local mfs to register and become master, requesting fileserver 106.103.1.84:5660- FSID: 5308322330462534014 to try again by returning ESRCH
2013-01-29 16:45:29,890 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 103 from 106.103.1.83:46582 Generating reply with status: 3
2013-01-29 16:45:29,916 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 40 from 106.103.1.83:46582 Generating reply with status: 3
2013-01-29 16:45:30,913 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: FSRegister: Request  FSID: 957510465599090838 FSNetworkLocation: / FSHost:Port: 106.103.1.85:5660- FSHostName: joya-05 StoragePools a4d8848f5f61ee970050e0547c0c9164-daf69cced0df98b50050e0547e0d7738-53bad5fd4c75e9f00050e0547d0c7d44-c47f19c73de8c2290050e0547f0cc110- Capacity: 18298924 Available: 11204069 Used: 7094855 Role: 0 isDCA: false Received registration request
2013-01-29 16:45:30,913 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: FSRegister: CLDB waiting for local mfs to register and become master, requesting fileserver 106.103.1.85:5660- FSID: 957510465599090838 to try again by returning ESRCH
2013-01-29 16:45:31,481 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 103 from 106.103.1.81:39926 Generating reply with status: 3
2013-01-29 16:45:33,154 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 40 from 106.103.1.85:49729 Generating reply with status: 3
2013-01-29 16:45:33,341 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: FSRegister: Request  FSID: 1369973134422650278 FSNetworkLocation: / FSHost:Port: 106.103.1.81:5660- FSHostName: joya-01 StoragePools 203451d68c1b7df300894d99780edc23-f26b97c53398ffdd00894d99790b6ecc-711393e74550b91900894d997b0ba3bc-c107a4c66cec22f400894d997a0bb629- Capacity: 18297256 Available: 9628832 Used: 8668423 Role: 0 isDCA: false Received registration request
2013-01-29 16:45:33,342 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: Cluster uuid is -7832729812637418072--764775730858407929
2013-01-29 16:45:33,366 INFO  com.mapr.fs.cldb.counters.FileServerMetrics [pool-1-thread-1]: Initializing File Server Metrics with hostName=joya-01
2013-01-29 16:45:33,366 INFO  com.mapr.fs.cldb.topology.FileServer [pool-1-thread-1]: Instantiating fileserver metrics with context:com.mapr.fs.cldb.counters.MapRGangliaContext31
2013-01-29 16:45:33,369 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: FSRegister: Registered FileServer: 106.103.1.81:5660-
2013-01-29 16:45:33,424 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-2]: Allocating WorkUnit type : NOCOMPRESS_LIST_UPDATED for container 0 with sequence number 0 to 106.103.1.81:5660-
2013-01-29 16:45:33,453 WARN  com.mapr.fs.cldb.Containers [pool-1-thread-1]: FileServer 106.103.1.81:5660- for container 1 on StoragePool 203451d68c1b7df300894d99780edc23 did not have any master  for container work. Marking it invalid
2013-01-29 16:45:33,456 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [pool-1-thread-1]: Storing KvStoreContainerInfo to ZooKeeper  Container ID:1 Servers:  Inactive Servers:  106.103.1.85:5660--10-VALID(957510465599090838) 106.103.1.84:5660--10-VALID(5308322330462534014) 106.103.1.82:5660--10-VALID(4093149274661164202) 106.103.1.83:5660--10-VALID(8628902522623566439) 106.103.1.81:5660--10-INVALID(1369973134422650278) Unused Servers:  Latest epoch:10 SizeMB:0
2013-01-29 16:45:33,463 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: No master for kvstore, waiting for local kvstore to become master
2013-01-29 16:45:33,463 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: Processing stale containers  on StoragePool 203451d68c1b7df300894d99780edc23 from FileServer 106.103.1.81:5660-
2013-01-29 16:45:33,463 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: Processing stale containers  on StoragePool f26b97c53398ffdd00894d99790b6ecc from FileServer 106.103.1.81:5660-
2013-01-29 16:45:33,464 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: Processing stale containers  on StoragePool 711393e74550b91900894d997b0ba3bc from FileServer 106.103.1.81:5660-
2013-01-29 16:45:33,464 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: Processing stale containers  on StoragePool c107a4c66cec22f400894d997a0bb629 from FileServer 106.103.1.81:5660-
2013-01-29 16:45:33,527 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: Allocating WorkUnit type : CONTAINER_INVALID for container 1 with sequence number 0 to 106.103.1.81:5660-
2013-01-29 16:45:33,739 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: Processing stale containers  on StoragePool 203451d68c1b7df300894d99780edc23 from FileServer 106.103.1.81:5660-
2013-01-29 16:45:33,740 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: Processing stale containers  on StoragePool f26b97c53398ffdd00894d99790b6ecc from FileServer 106.103.1.81:5660-
2013-01-29 16:45:33,740 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: Processing stale containers  on StoragePool 711393e74550b91900894d997b0ba3bc from FileServer 106.103.1.81:5660-
2013-01-29 16:45:33,740 INFO  com.mapr.fs.cldb.Containers [pool-1-thread-1]: Processing stale containers  on StoragePool c107a4c66cec22f400894d997a0bb629 from FileServer 106.103.1.81:5660-
2013-01-29 16:45:33,748 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [pool-1-thread-1]: Storing KvStoreContainerInfo to ZooKeeper  Container ID:1 Master:106.103.1.81:5660--10-BECOME_MASTER(1369973134422650278) Servers:  106.103.1.81:5660--10-BECOME_MASTER(1369973134422650278) Inactive Servers:  106.103.1.85:5660--10-VALID(957510465599090838) 106.103.1.84:5660--10-VALID(5308322330462534014) 106.103.1.82:5660--10-VALID(4093149274661164202) 106.103.1.83:5660--10-VALID(8628902522623566439) Unused Servers:  Latest epoch:10 SizeMB:0
2013-01-29 16:45:33,820 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 103 from 106.103.1.83:37529 Generating reply with status: 3
2013-01-29 16:45:33,835 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: Allocating WorkUnit type : SERVER_MASTER_FOR_CONTAINER for container 1 on StoragePool 203451d68c1b7df300894d99780edc23 to 106.103.1.81:5660- ifClean set to true
2013-01-29 16:45:33,846 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: RPC: PROGRAMID: 2345 PROCEDUREID: 40 from 106.103.1.83:37529 Generating reply with status: 3
2013-01-29 16:45:34,049 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [pool-1-thread-1]: Storing KvStoreContainerInfo to ZooKeeper  Container ID:1 Master:106.103.1.81:5660--10-VALID(1369973134422650278) Servers:  106.103.1.81:5660--10-VALID(1369973134422650278) Inactive Servers:  106.103.1.85:5660--10-VALID(957510465599090838) 106.103.1.84:5660--10-VALID(5308322330462534014) 106.103.1.82:5660--10-VALID(4093149274661164202) 106.103.1.83:5660--10-VALID(8628902522623566439) Unused Servers:  Latest epoch:10 SizeMB:0
2013-01-29 16:45:34,055 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: CLDB Mode: MASTER_REGISTER_READY CLDB HA Check
2013-01-29 16:45:34,076 INFO  com.mapr.fs.license.LicenseManager [pool-1-thread-1]: using default validator
2013-01-29 16:45:34,080 INFO  com.mapr.kvstore.Operation [pool-1-thread-1]: setNoDelete=true
2013-01-29 16:45:34,398 INFO  com.mapr.fs.license.LicenseManager [pool-1-thread-1]: 0x25: unique id: 782bcb1652aa-4C4C4544-0059-4710-8046-C8C04F31354A-0061865985-0037486593-0010616834
2013-01-29 16:45:34,398 FATAL com.mapr.fs.license.LicenseManager [pool-1-thread-1]: CLDB HA check failed: not licensed, failover denied: elapsed time since last failure=53 minutes
2013-01-29 16:45:34,398 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: CLDB HA check returned: false
2013-01-29 16:45:34,398 FATAL com.mapr.fs.cldb.CLDB [pool-1-thread-1]: CLDBShutdown: HA Check failed: Shutting down CLDB
2013-01-29 16:45:34,398 INFO  com.mapr.fs.cldb.CLDBServer [pool-1-thread-1]: Shutdown: Stopping CLDB
2013-01-29 16:45:34,399 INFO  com.mapr.fs.cldb.CLDB [Thread-10]: CLDB ShutDown Hook called
2013-01-29 16:45:34,400 INFO  com.mapr.fs.cldb.zookeeper.ZooKeeperClient [Thread-10]: Zookeeper Client: Closing client connection:
2013-01-29 16:45:34,405 INFO  com.mapr.fs.cldb.CLDBServer [main-EventThread]: ZooKeeper event NodeDeleted on path /datacenter/controlnodes/cldb/active/CLDBMaster
2013-01-29 16:45:34,405 INFO  org.apache.zookeeper.ZooKeeper [Thread-10]: Session: 0x3c86c02eff000b closed
2013-01-29 16:45:34,405 INFO  com.mapr.fs.cldb.CLDBServer [main-EventThread]: ZooKeeper event of type: NodeDeleted on path /datacenter/controlnodes/cldb/active/CLDBMaster
2013-01-29 16:45:34,405 INFO  com.mapr.fs.cldb.CLDB [Thread-10]: CLDB shutdown
</pre>

i am not sure why it search for HA.. it is the same server, i just restart it..

Thanks,

Outcomes