AnsweredAssumed Answered

Could not connect to CLDB

Question asked by vijaypk10 on Jul 21, 2016
Latest reply on Sep 24, 2016 by yuvarajgopal


Hi,

As i am installing and configuring my MapR 3 node cluster using community edition, looks like my packages got corrupt after completing the install and the nodes won't start. I had installed the clean packages since and When i tried to start the cluster, the CLDB is not coming up clean even though it says the CLDB is running. HEre is what i have checked:

 

root@BGLR-MAPR03:~# service mapr-zookeeper qstatus

JMX enabled by default

Using config: /opt/mapr/zookeeper/zookeeper-3.4.5/conf/zoo.cfg

Mode: leader

root@BGLR-MAPR03:~# service mapr-warden status

WARDEN running as process 6468.

root@BGLR-MAPR03:~# service mapr-warden status

WARDEN running as process 6468.

root@BGLR-MAPR03:~# service mapr-cldb status

CLDB running as process 7299.

root@BGLR-MAPR03:~# maprcli node cldbmaster

ERROR (10009) -  Couldn't connect to the CLDB service

root@BGLR-MAPR03:~# maprcli service list

ERROR (10009) -  Could not connect to CLDB and no Zookeeper connect string provided

 

 

maprcli-root-0.log:

==============

2016-07-21 16:24:57,136 INFO  com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils [main]: Bad CLDB credentials removed: CLDB Ips: 10.30.3.145-, Port: 7222

2016-07-21 16:24:57,137 ERROR com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils getDataForParticularCLDB [main]: No data returned in RPC: CLDB Ips: 10.30.3.146-, Port: 7222. Continue searching for correct CLDB

2016-07-21 16:24:57,137 INFO  com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils [main]: Bad CLDB credentials removed: CLDB Ips: 10.30.3.146-, Port: 7222

2016-07-21 16:24:57,137 ERROR com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils getZkConnect [main]: No data is received from any cldb

2016-07-21 16:24:57,138 ERROR com.mapr.cli.ServerCommands getServicesInfo [main]: zkConnectString is null/empty. Cannot proceed further.

2016-07-21 16:24:57,161 ERROR com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils getDataForParticularCLDB [main]: CLDB Ips: 10.30.3.147-, Port: 7222 is attempting to become a master. Retrying !

2016-07-21 16:25:32,166 INFO  com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils [main]: Bad CLDB credentials removed: CLDB Ips: 10.30.3.147-, Port: 7222

2016-07-21 16:25:32,167 ERROR com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils getDataForParticularCLDB [main]: No data returned in RPC: CLDB Ips: 10.30.3.145-, Port: 7222. Continue searching for correct CLDB

2016-07-21 16:25:32,167 INFO  com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils [main]: Bad CLDB credentials removed: CLDB Ips: 10.30.3.145-, Port: 7222

2016-07-21 16:25:32,168 ERROR com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils getDataForParticularCLDB [main]: No data returned in RPC: CLDB Ips: 10.30.3.146-, Port: 7222. Continue searching for correct CLDB

2016-07-21 16:25:32,168 INFO  com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils [main]: Bad CLDB credentials removed: CLDB Ips: 10.30.3.146-, Port: 7222

2016-07-21 16:25:32,168 ERROR com.mapr.cli.ServerCommands sendRequest [main]: RPC Request to list Nodes failed. No data returned

2016-07-21 16:25:32,184 INFO  com.mapr.cliframework.driver.CLIMainDriver [main]: ERROR (10009) -  Couldn't connect to the CLDB service. Check if at least one CLDB is running.

 

CLDB.log:

=========

2016-07-21 16:46:44,094 INFO CLDBServer [ZK-Connect]: Starting thread to monitor waiting for local kvstore to become master

2016-07-21 16:46:44,111 INFO CLDB [main]: CLDBState: CLDB State change : WAIT_FOR_FILESERVERS

2016-07-21 16:46:44,112 INFO CLDB [main]: CLDBInit: Starting RPCServer on port 7222 with num thread 10, heap size of 623(MB) and with startup options -Xms383m -Xmx639m -XX:ErrorFile=/opt/cores/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/cores -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=60 -XX:+UseCMSInitiatingOccupancyOnly -XX:ThreadStackSize=256

2016-07-21 16:46:44,114 INFO CLDB [main]: CLDBInit: Starting HTTP Server

2016-07-21 16:46:44,114 INFO HttpServer [main]: WebServer: Starting WebServer

2016-07-21 16:46:44,116 INFO HttpServer [main]: Listener started on SelectChannelConnector@0.0.0.0:7221 port 7221

2016-07-21 16:46:44,116 INFO HttpServer [main]: Starting Jetty WebServer

2016-07-21 16:46:44,116 INFO log [main]: jetty-6.1.26

2016-07-21 16:46:44,613 INFO log [main]: Started SelectChannelConnector@0.0.0.0:7221

2016-07-21 16:46:45,058 INFO FileServerHandler [RPC-1]: FSRegister: Request  FSID: 873509164464153678 Build: 5.1.0.37549.GA FSNetworkLocation:  FSHost:Port: 10.30.3.147- FSHostName: BGLR-MAPR03.quinnox.corp StoragePools 7fc0b535b1223f1400578fd2c50d2d37-edd32a9263b048ce00579046c700a6f4- Capacity: 46708 Available: 45904 Used: 804 Role: 0 isDCA: false uniq: 40a140d51f59664d-5790a730030f1f Received registration request

2016-07-21 16:46:45,121 INFO FileServerHandler [RPC-1]: FSRegister: Registered FileServer: 10.30.3.147- at topology /default-rack/BGLR-MAPR03.quinnox.corp/5660

2016-07-21 16:46:45,122 INFO FileServerHandler [RPC-1]: FileServer Registration Request: Node Configuration

2016-07-21 16:46:45,122 INFO FileServerHandler [RPC-1]: NumCpus: 2 Avail Memory: 3353 Num Sps: 2 Num Instances: 1

2016-07-21 16:47:19,163 INFO CLDBServer [Lookup-1]: Rejecting RPC 2345.5 from 10.30.3.147:45868 with status 3 as CLDB is waiting for local kvstore to become master.

2016-07-21 16:48:57,576 INFO CLDBServer [RPC-2]: Rejecting RPC 2345.205 from 10.30.3.147:48866 with status 3 as CLDB is waiting for local kvstore to become master.

2016-07-21 16:50:07,599 INFO CLDBServer [RPC-3]: Rejecting RPC 2345.205 from 10.30.3.147:48866 with status 3 as CLDB is waiting for local kvstore to become master.

 

Any help is appreciated.

 

Thanks,

Vijay

Outcomes