AnsweredAssumed Answered

CLDB corruption while copying files to HDFS

Question asked by asadali on Aug 29, 2016
Latest reply on Sep 7, 2016 by asadali

I'm back after having a hadoop cluster run successfully for over a month and I'm hoping someone can help me recover the cluster. Here is my setup:

 

Node1 - zookeeper, cldb, fileserver, webserver, nfs, jobtracker, tasktracker

Node2 - zookeeper, fileserver, jobtracker, tasktracker, resourcemanager, historyserver

Node3 - zookeeper, fileserver, tasktracker

Node4 - fileserver, jobtracker, tasktracker

Node5 - fileserver, jobtracker, tasktracker

Node6 - fileserver, jobtracker, tasktracker

 

I was copying files to my hadoop NFS when it suddenly stopped. I couldn't access the webserver either. I decided to restart the warden on Node1 but it still didn't work. Then I restarted Ubuntu on Node1, and I'm in the same place.

 

I think the zookeepers are online

>> sudo service mapr-zookeeper qstatus

Using config: /opt/mapr/zookeeper/zookeeper-3.4.5/conf/zoo.cfg

Mode: follower

 

There is a leader as well. However, I'm unable to dump zkinfo. I've tried with the -zkconnect option but it doesn't yield anything, which is very worrying.

>> sudo maprcli dump zkinfo -json

 

I checked the output of cldb.log, and there is something about a corruption:

 

CLDB.LOG

2016-08-29 16:38:32,584 INFO ZooKeeper [RPC-9]: Initiating client connection, connectString=hadoop10.nodes.com:5181,hadoop4.nodes.com:5181,hadoop5.nodes.com:5181 sessionTimeout=30000 watcher=com.mapr.util.zookeeper.ZKDataRetrieval@592c3f10

2016-08-29 16:38:32,585 INFO ZooKeeperSaslClient [RPC-9-SendThread(hadoop5.nodes.com:5181)]: Client will use SIMPLE-SECURITY as SASL mechanism.

2016-08-29 16:38:32,585 INFO ClientCnxn [RPC-9-SendThread(hadoop5.nodes.com:5181)]: Opening socket connection to server hadoop5.nodes.com/10.252.101.74:5181. Will attempt to SASL-authenticate using Login Context section 'Client_simple'

2016-08-29 16:38:32,586 INFO ClientCnxn [RPC-9-SendThread(hadoop5.nodes.com:5181)]: Socket connection established to hadoop5.nodes.com/10.252.101.74:5181, initiating session

2016-08-29 16:38:32,597 INFO ClientCnxn [RPC-9-SendThread(hadoop5.nodes.com:5181)]: Session establishment complete on server hadoop5.nodes.com/10.252.101.74:5181, sessionid = 0x256b329bc2f02ce, negotiated timeout = 30000

2016-08-29 16:38:32,597 INFO ZKDataRetrieval [RPC-9-EventThread]: Process path: null. Event state: SyncConnected. Event type: None

2016-08-29 16:38:32,597 INFO ZKDataRetrieval [RPC-9]: Connected to ZK: hadoop10.nodes.com:5181,hadoop4.nodes.com:5181,hadoop5.nodes.com:5181

2016-08-29 16:38:32,598 INFO ZKDataRetrieval [RPC-9-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None

2016-08-29 16:38:32,598 INFO ZKDataRetrieval [RPC-9]: Getting servicesMap

2016-08-29 16:38:32,650 INFO ZKDataRetrieval [RPC-9]: Getting configServicesMap

2016-08-29 17:38:01,131 INFO CLDBServer [RPC-2]: FileServer 10.252.101.73- reported a corrupt StoragePool 885c1975508e3af8005788200a015589

2016-08-29 17:38:01,131 FATAL CLDBServer [RPC-2]: The CLDB's local fileserver 10.252.101.73- reported a corrupt CLDB container. Stopping CLDB

2016-08-29 17:38:01,131 FATAL CLDB [RPC-2]: CLDBShutdown: The CLDB's local fileserver 10.252.101.73- reported a corrupt CLDB container. Stopping CLDB

2016-08-29 17:38:01,134 ERROR CLDB [RPC-2]: Thread: main-SendThread(hadoop10.nodes.com:5181) ID: 17

2016-08-29 17:38:01,134 ERROR CLDB [RPC-2]: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

2016-08-29 17:38:01,134 ERROR CLDB [RPC-2]: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

2016-08-29 17:38:01,134 ERROR CLDB [RPC-2]: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

2016-08-29 17:38:01,134 ERROR CLDB [RPC-2]: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1070)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: Thread: RPC-5 ID: 36

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: Thread: RPC-3 ID: 34

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: Thread: RBal ID: 71

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.lang.Thread.sleep(Native Method)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.replication.RoleBalancer$RoleBalancerThread.run(RoleBalancer.java:95)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: Thread: pool-6-thread-144 ID: 246

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: Thread: RMI TCP Accept-7220 ID: 11

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.net.PlainSocketImpl.socketAccept(Native Method)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.net.ServerSocket.implAccept(ServerSocket.java:530)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.net.ServerSocket.accept(ServerSocket.java:498)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:399)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:371)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: Thread: Lookup-5 ID: 41

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: Thread: HB-3 ID: 54

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,135 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: Thread: RPC-6 ID: 45

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: Thread: Reference Handler ID: 2

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.lang.Object.wait(Object.java:503)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: Thread: FCR-3 ID: 102

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: Thread: EmailManager ID: 22

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.util.EmailManager$EmailManagerThread.run(EmailManager.java:115)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: Thread: AssignVouchersPurger ID: 66

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.TimerThread.mainLoop(Timer.java:552)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.TimerThread.run(Timer.java:505)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: Thread: 10285724@qtp-1062526344-1 - Acceptor0 SelectChannelConnector@0.0.0.0:7221 ID: 28

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: Thread: VolumeMirrorThread0 ID: 24

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1979)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: com.mapr.volumemirror.VolumeMirror$VolumeMirrorThread.run(VolumeMirror.java:929)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: Thread: ACR-7 ID: 96

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,136 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: Thread: ACR-3 ID: 63

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: Thread: RPC-1 ID: 29

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: Thread: Signal Dispatcher ID: 5

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: Thread: ACR-5 ID: 89

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: Thread: kvwriter ID: 23

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: com.mapr.kvstore.KvStoreWriter$KvStoreWriterThread.run(KvStoreWriter.java:42)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: Thread: RPC-10 ID: 49

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: Thread: RPC-7 ID: 46

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,137 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: Thread: RPC-2 ID: 33

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.dumpThreads(Native Method)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.getAllStackTraces(Thread.java:1640)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.CLDB.shutdown(CLDB.java:180)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.CLDBServer.containerCorrupt(CLDBServer.java:8452)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.CLDBServer.processContainerOpRpc(CLDBServer.java:4375)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.CLDBServer.processRpc(CLDBServer.java:4554)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.CLDBServer.requestArrived(CLDBServer.java:3263)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: com.mapr.fs.Rpc$RpcExecutor.run(Rpc.java:160)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: Thread: Repl ID: 69

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.sleep(Native Method)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.replication.ReplicationHandlerThread.run(ReplicationHandlerThread.java:188)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: Thread: RPC-8 ID: 47

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: Thread: RPC-4 ID: 35

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: Thread: RPC-9 ID: 48

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: Thread: pool-4-thread-1 ID: 179

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: Thread: HB-6 ID: 58

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,138 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: Thread-6 ID: 21

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.sleep(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.ActiveContainersMap$WriteBackAtimeUpdater$IdleFlusher.run(ActiveContainersMap.java:801)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: RMI TCP Accept-0 ID: 10

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.net.PlainSocketImpl.socketAccept(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.net.ServerSocket.implAccept(ServerSocket.java:530)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.net.ServerSocket.accept(ServerSocket.java:498)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:399)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:371)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: Timer thread for monitoring cldb ID: 13

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.TimerThread.mainLoop(Timer.java:552)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.TimerThread.run(Timer.java:505)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: DBal ID: 70

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.sleep(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.replication.DiskBalancer$DiskBalancerThread.run(DiskBalancer.java:345)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: Health-Monitor ID: 15

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.sleep(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: com.mapr.baseutils.threadpool.HealthCheckMonitor.run(HealthCheckMonitor.java:31)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: Lookup-8 ID: 44

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: main-EventThread ID: 18

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: FCR-1 ID: 100

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: Lookup-4 ID: 40

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,139 ERROR CLDB [RPC-2]: Thread: Lookup-1 ID: 37

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: Thread: RPC-9-SendThread(hadoop5.nodes.com:5181) ID: 280

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1070)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: Thread: ACR-8 ID: 97

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: Thread: HB-2 ID: 53

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: Thread: RMI TCP Accept-0 ID: 12

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.net.PlainSocketImpl.socketAccept(Native Method)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.net.ServerSocket.implAccept(ServerSocket.java:530)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.net.ServerSocket.accept(ServerSocket.java:498)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:399)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:371)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: Thread: Lookup-3 ID: 39

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: Thread: Timer-1 ID: 90

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.TimerThread.mainLoop(Timer.java:552)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.TimerThread.run(Timer.java:505)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: Thread: Lookup-6 ID: 42

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,140 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: MetricsPollTimer ID: 91

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Object.wait(Object.java:503)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.TimerThread.mainLoop(Timer.java:526)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.TimerThread.run(Timer.java:505)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: DestroyJavaVM ID: 32

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: pool-6-thread-173 ID: 275

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: HB-Monitor ID: 72

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Thread.sleep(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.MonitorThread.run(MonitorThread.java:86)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: RScan ID: 68

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Thread.sleep(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.replication.ReplicationScannerThread.run(ReplicationScannerThread.java:87)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: TopoScriptWatcher ID: 19

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.TimerThread.mainLoop(Timer.java:552)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.TimerThread.run(Timer.java:505)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: pool-4-thread-2 ID: 200

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: RPC-9-EventThread ID: 281

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: Timer thread for monitoring fileserver ID: 51

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.TimerThread.mainLoop(Timer.java:552)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.TimerThread.run(Timer.java:505)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: Thread-9 ID: 26

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: com.mapr.fs.jni.RpcNative.waitForRequest(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: com.mapr.fs.Rpc.access$000(Rpc.java:13)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: com.mapr.fs.Rpc$RpcListenerThread.run(Rpc.java:188)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: Thread: Lookup-2 ID: 38

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,141 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: 483253443@qtp-1062526344-0 ID: 27

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: Thread-15 ID: 60

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: ACR-6 ID: 95

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: Lookup-7 ID: 43

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: ACR-4 ID: 64

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: MetricsStorageTimer ID: 92

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Object.wait(Object.java:503)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.TimerThread.mainLoop(Timer.java:526)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.TimerThread.run(Timer.java:505)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: Finalizer ID: 3

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: HB-5 ID: 57

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: Thread: ACR-1 ID: 56

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,142 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: Thread-5 ID: 20

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Thread.sleep(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.ActiveContainersMap$WriteBackSizeUpdater$IdleFlusher.run(ActiveContainersMap.java:928)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: Timer-0 ID: 30

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.TimerThread.mainLoop(Timer.java:552)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.TimerThread.run(Timer.java:505)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: Thread-38 ID: 282

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Object.wait(Object.java:503)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: com.mapr.util.zookeeper.ZKDataRetrieval$1.run(ZKDataRetrieval.java:124)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: HB-1 ID: 52

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: FCR-2 ID: 101

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: TimerEventHandlerThread ID: 67

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Object.wait(Object.java:503)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.timer.TimerEvent$TimerEventHandlerThread.run(TimerEvent.java:109)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: ZK-Connect ID: 16

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Object.wait(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Object.wait(Object.java:503)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.CLDBServer$WaitForZKReconnect.onMasterWakeup(CLDBServer.java:15277)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: com.mapr.fs.cldb.CLDBServer$WaitForZKReconnect.run(CLDBServer.java:15258)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: HB-4 ID: 55

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: Thread: ACR-2 ID: 59

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: sun.misc.Unsafe.park(Native Method)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

2016-08-29 17:38:01,143 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

2016-08-29 17:38:01,144 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

2016-08-29 17:38:01,144 ERROR CLDB [RPC-2]: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

2016-08-29 17:38:01,144 ERROR CLDB [RPC-2]: java.lang.Thread.run(Thread.java:745)

2016-08-29 17:38:01,144 INFO CLDBServer [RPC-2]: Shutdown: Stopping CLDB

2016-08-29 17:38:01,144 INFO CLDB [Thread-13]: CLDB ShutDown Hook called

2016-08-29 17:38:01,144 INFO ZooKeeperClient [Thread-13]: Setting the clean cldbshutdown flag to true

2016-08-29 17:38:01,171 INFO ZooKeeperClient [Thread-13]: Zookeeper Client: Closing client connection:

2016-08-29 17:38:01,189 INFO CLDBServer [main-EventThread]: The CLDB received notification that a ZooKeeper event of type NodeDeleted occurred on path /datacenter/controlnodes/cldb/active/CLDBMaster

2016-08-29 17:38:01,189 INFO ZKDataRetrieval [RPC-9-EventThread]: Process path: /datacenter/controlnodes/cldb/active/CLDBMaster. Event state: SyncConnected. Event type: NodeDeleted

2016-08-29 17:38:01,189 INFO ZooKeeper [Thread-13]: Session: 0x56b329bb34000a closed

2016-08-29 17:38:01,189 INFO CLDB [Thread-13]: CLDB shutdown

Loaded library '/opt/mapr/server/permissions/libmapr_roles_refimpl.so' to resolve role membershipRoleMap: found 0 users and 0 roles.

Header: hostName: hadoop4.nodes.com, Time Zone: Pacific Standard Time, processName: cldb, processId: 13724, MapR Build Version: 5.1.0.37549.GA

2016-08-29 17:38:04,760 INFO CLDB [main]: Loading properties file : /opt/mapr/conf/cldb.conf

2016-08-29 17:38:04,950 INFO CLDBMetrics [main]: Initializing CLDB Metrics with serviceName: cldbServer

2016-08-29 17:38:04,952 INFO CLDB [main]: CLDBInit: Using hostname file /opt/mapr/hostname and hostid file /opt/mapr/hostid

2016-08-29 17:38:04,952 INFO CLDB [main]: CLDB Properties from configuration file: cldb.web.port=7221cldb.zookeeper.servers=hadoop10.nodes.com:5181,hadoop4.nodes.com:5181,hadoop5.nodes.com:5181cldb.numthreads=10cldb.web.https.port=7443enable.replicas.invariant.check=falsehadoop.version=2.7.0cldb.port=7222cldb.min.fileservers=1cldb.detect.dup.hostid.enabled=falsenum.volmirror.threads=1cldb.jmxremote.port=7220

2016-08-29 17:38:04,952 INFO CLDB [main]: CLDB Command line args: /opt/mapr/conf/cldb.conf

2016-08-29 17:38:04,952 INFO CLDB [main]: CLDBInit: Initializing CLDB

2016-08-29 17:38:04,953 INFO CLDB [main]: MapR BuildVersion: 5.1.0.37549.GA

2016-08-29 17:38:04,953 INFO CLDB [main]: CLDBInit: Start CLDBServer

2016-08-29 17:38:04,977 INFO CLDBServer [main]: CLDBInit: HostName: hadoop4.nodes.com ServerId: 5194036052395192442

2016-08-29 17:38:04,988 INFO CLDB [main]: CLDBState: CLDB State change : INITIAZING

2016-08-29 17:38:05,000 INFO CLDBServer [main]: Initializing Cldb JNI with memory 255459328 bytes

2016-08-29 17:38:05,006 INFO ZooKeeperClient [main]: ZooKeeperClient init: zk timeout = 30000 ms

2016-08-29 17:38:05,017 INFO ZooKeeper [main]: Client environment:zookeeper.version=3.4.5-mapr-1503--1, built on 02/26/2016 02:13 GMT

2016-08-29 17:38:05,017 INFO ZooKeeper [main]: Client environment:host.name=hadoop4.nodes.com

2016-08-29 17:38:05,017 INFO ZooKeeper [main]: Client environment:java.version=1.7.0_80

2016-08-29 17:38:05,017 INFO ZooKeeper [main]: Client environment:java.vendor=Oracle Corporation

2016-08-29 17:38:05,017 INFO ZooKeeper [main]: Client environment:java.home=/usr/lib/jvm/java-7-oracle/jre

2016-08-29 17:38:05,017 INFO ZooKeeper [main]: Client environment:java.class.path=/opt/mapr:/opt/mapr/conf:/opt/mapr/lib/adminuiapp-5.1.0-mapr.jar:/opt/mapr/lib/ant-1.7.1.jar:/opt/mapr/lib/antlr-2.7.7.jar:/opt/mapr/lib/antlr4-runtime-4.5.jar:/opt/mapr/lib/aopalliance-1.0.jar:/opt/mapr/lib/asm-3.3.1.jar:/opt/mapr/lib/c3p0-0.9.1.2.jar:/opt/mapr/lib/central-logging-5.1.0-mapr.jar:/opt/mapr/lib/cglib-2.2.2.jar:/opt/mapr/lib/cldb-5.1.0-mapr.jar:/opt/mapr/lib/cliframework-5.1.0-mapr.jar:/opt/mapr/lib/cliframework-5.1.0-mapr-tests.jar:/opt/mapr/lib/commons-codec-1.5.jar:/opt/mapr/lib/commons-collections-3.2.2.jar:/opt/mapr/lib/commons-configuration-1.6.jar:/opt/mapr/lib/commons-el-1.0.jar:/opt/mapr/lib/commons-email-1.2.jar:/opt/mapr/lib/commons-io-2.4.jar:/opt/mapr/lib/commons-lang-2.5.jar:/opt/mapr/lib/commons-logging-1.1.3-api.jar:/opt/mapr/lib/commons-logging-1.1.3.jar:/opt/mapr/lib/dom4j-1.6.1.jar:/opt/mapr/lib/eval-0.5.jar:/opt/mapr/lib/flexjson-2.1.jar:/opt/mapr/lib/gateway-5.1.0-mapr.jar:/opt/mapr/lib/gateway-5.1.0-mapr-tests.jar:/opt/mapr/lib/globalfsck-5.1.0-mapr.jar:/opt/mapr/lib/guava-14.0.1.jar:/opt/mapr/lib/hadoop-0.20.2-dev-core.jar:/opt/mapr/lib/hadoop-auth-2.7.0.jar:/opt/mapr/lib/hadoop-common-2.7.0.jar:/opt/mapr/lib/hadoop-yarn-api-2.7.0-mapr-1602.jar:/opt/mapr/lib/hadoop-yarn-client-2.7.0-mapr-1602.jar:/opt/mapr/lib/hadoop-yarn-common-2.7.0-mapr-1602.jar:/opt/mapr/lib/hibernate-c3p0-3.6.8.Final.jar:/opt/mapr/lib/hibernate-commons-annotations-3.2.0.Final.jar:/opt/mapr/lib/hibernate-core-3.6.8.Final.jar:/opt/mapr/lib/httpclient-4.2.jar:/opt/mapr/lib/httpclient-cache-4.2.jar:/opt/mapr/lib/httpcore-4.2.jar:/opt/mapr/lib/jackson-annotations-2.7.1.jar:/opt/mapr/lib/jackson-core-2.7.1.jar:/opt/mapr/lib/jackson-databind-2.7.1.jar:/opt/mapr/lib/jasper-compiler-5.5.12.jar:/opt/mapr/lib/jasper-runtime-5.5.12.jar:/opt/mapr/lib/javassist-3.12.1.GA.jar:/opt/mapr/lib/jetty-6.1.26.jar:/opt/mapr/lib/jetty-plus-6.1.26.jar:/opt/mapr/lib/jetty-util-6.1.26.jar:/opt/mapr/lib/jline-2.11.jar:/opt/mapr/lib/jobmngmnt-5.1.0-mapr.jar:/opt/mapr/lib/joda-time-2.0.jar:/opt/mapr/lib/JPam-1.1.jar:/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/json-smart-1.2.jar:/opt/mapr/lib/jsp-2.1-6.1.14.jar:/opt/mapr/lib/jsp-api-2.1-6.1.14.jar:/opt/mapr/lib/jta-1.1.jar:/opt/mapr/lib/junit-3.8.1.jar:/opt/mapr/lib/junit-4.5.jar:/opt/mapr/lib/kvstore-5.1.0-mapr.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/mail.jar:/opt/mapr/lib/maprbuildversion-5.1.0-mapr.jar:/opt/mapr/lib/maprcli-5.1.0-mapr.jar:/opt/mapr/lib/maprcli-5.1.0-mapr-tests.jar:/opt/mapr/lib/maprdb-5.1.0-mapr.jar:/opt/mapr/lib/maprdb-5.1.0-mapr-tests.jar:/opt/mapr/lib/maprdb-mapreduce-5.1.0-mapr.jar:/opt/mapr/lib/maprdb-mapreduce-5.1.0-mapr-tests.jar:/opt/mapr/lib/maprdb-shell-5.1.0-mapr.jar:/opt/mapr/lib/maprfs-5.1.0-mapr.jar:/opt/mapr/lib/maprfs-diagnostic-tools-5.1.0-mapr.jar:/opt/mapr/lib/mapr-hbase-5.1.0-mapr.jar:/opt/mapr/lib/mapr-hbase-5.1.0-mapr-tests.jar:/opt/mapr/lib/mapr-java-utils-5.1.0-mapr-tests.jar:/opt/mapr/lib/mapr-streams-5.1.0-mapr.jar:/opt/mapr/lib/mapr-streams-5.1.0-mapr-tests.jar:/opt/mapr/lib/mapr-tools-5.1.0-mapr.jar:/opt/mapr/lib/mapr-tools-5.1.0-mapr-tests.jar:/opt/mapr/lib/maven-ant-tasks-2.1.3.jar:/opt/mapr/lib/mysql-connector-java-5.1.25.jar:/opt/mapr/lib/ojai-1.0.jar:/opt/mapr/lib/ojai-mapreduce-1.0.jar:/opt/mapr/lib/persistence-api-1.0.jar:/opt/mapr/lib/protobuf-java-2.5.0.jar:/opt/mapr/lib/servlet-api-2.5-6.1.26.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/snakeyaml-1.14.jar:/opt/mapr/lib/spring-asm-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-beans-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-context-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-core-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-expression-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-shell-1.1.0-mapr-1602.jar:/opt/mapr/lib/volumemirror-5.1.0-mapr.jar:/opt/mapr/lib/warden-5.1.0-mapr.jar:/opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/hadoop/hadoop-2.7.0/lib/hadoop-2.7.0-dev-core.jar:

Outcomes