AnsweredAssumed Answered

HBase jobs failing to connect to Zookeper (KeeperException: ConnectionLoss for /hbase/master))

Question asked by ravi on Mar 19, 2013
Latest reply on Mar 19, 2013 by ravi
Hi -

MapR Version: 2.0.1.15869.GA, M5 distribution with 3-Node HBase mini cluster.

I'm having issues with Zookeeper connection while running HBase jobs. This was a working cluster until one of the CLDB instance was removed from one of the nodes. After removing the CLDB, I reran the configure.sh on all nodes. Here is the error I'm receiving while running M/R job from client:
<pre><code>
13/03/19 12:27:07 INFO fs.JobTrackerWatcher: Current running JobTracker is: eng-poc03.archerlab.local/10.5.152.224:9001
13/03/19 12:27:07 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 24380@eng-poc06.archerlab.local
13/03/19 12:27:07 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
13/03/19 12:27:07 INFO util.RetryCounter: The 1 times to retry  after sleeping 2000 ms
13/03/19 12:27:10 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
13/03/19 12:27:10 INFO util.RetryCounter: The 2 times to retry  after sleeping 4000 ms
13/03/19 12:27:15 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
13/03/19 12:27:15 INFO util.RetryCounter: The 3 times to retry  after sleeping 8000 ms
13/03/19 12:27:23 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
13/03/19 12:27:23 ERROR zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries
13/03/19 12:27:23 WARN zookeeper.ZKUtil: hconnection Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:580)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:907)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
        at com.rsa.itsrm.poc.hadoop.hbase.mapreduce.MRTableSink.main(MRTableSink.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
13/03/19 12:27:23 ERROR zookeeper.ZooKeeperWatcher: hconnection Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:580)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:907)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
        at com.rsa.itsrm.poc.hadoop.hbase.mapreduce.MRTableSink.main(MRTableSink.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
13/03/19 12:27:23 FATAL client.HConnectionManager$HConnectionImplementation: Unexpected exception during initialization, aborting
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:580)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:907)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
        at com.rsa.itsrm.poc.hadoop.hbase.mapreduce.MRTableSink.main(MRTableSink.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
13/03/19 12:27:25 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
13/03/19 12:27:25 INFO util.RetryCounter: The 1 times to retry  after sleeping 2000 ms
13/03/19 12:27:28 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
13/03/19 12:27:28 INFO util.RetryCounter: The 2 times to retry  after sleeping 4000 ms
13/03/19 12:27:32 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
13/03/19 12:27:32 INFO util.RetryCounter: The 3 times to retry  after sleeping 8000 ms
13/03/19 12:27:40 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
13/03/19 12:27:40 ERROR zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries
13/03/19 12:27:40 WARN zookeeper.ZKUtil: hconnection Unable to set watcher on znode /hbase/root-region-server
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:583)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:907)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
        at com.rsa.itsrm.poc.hadoop.hbase.mapreduce.MRTableSink.main(MRTableSink.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
13/03/19 12:27:40 ERROR zookeeper.ZooKeeperWatcher: hconnection Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:583)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:907)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
        at com.rsa.itsrm.poc.hadoop.hbase.mapreduce.MRTableSink.main(MRTableSink.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
13/03/19 12:27:40 FATAL client.HConnectionManager$HConnectionImplementation: Unexpected exception during initialization, aborting
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:154)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:226)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:583)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:569)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:186)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:194)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:907)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
        at com.rsa.itsrm.poc.hadoop.hbase.mapreduce.MRTableSink.main(MRTableSink.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
13/03/19 12:27:40 ERROR mapreduce.TableOutputFormat: java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@4f675ff4 closed
13/03/19 12:27:40 INFO mapred.JobClient: Cleaning up the staging area maprfs:/var/mapr/cluster/mapred/jobTracker/staging/root/.staging/job_201303191157_0001
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@4f675ff4 closed
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:200)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:907)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
        at com.rsa.itsrm.poc.hadoop.hbase.mapreduce.MRTableSink.main(MRTableSink.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@4f675ff4 closed
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:794)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:782)
        at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:249)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:213)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:195)
        ... 16 more
</code> </pre>

And here is the ZK dump:
<pre><code>
HBase is rooted at /hbase
Active master address: eng-poc07.archerlab.local,60000,1363712434604
Backup master addresses:
 eng-poc02.archerlab.local,60000,1363712434795
Region server holding ROOT: eng-poc02.archerlab.local,60020,1363712450017
Region servers:
 eng-poc07.archerlab.local,60020,1363712456006
 eng-poc02.archerlab.local,60020,1363712450017
 eng-poc03.archerlab.local,60020,1363712453021
Quorum Server Statistics:
 eng-poc03.archerlab.local:5181
  Zookeeper version: 3.3.2--1, built on 06/05/2012 18:33 GMT
  Clients:
   /10.5.152.232:49170[0](queued=0,recved=1,sent=0)
   /10.5.152.222:35113[1](queued=0,recved=216,sent=222)
   /10.5.152.222:35129[1](queued=0,recved=1252,sent=1281)
   /10.5.152.232:49100[1](queued=0,recved=228,sent=228)
   /10.5.152.222:35117[1](queued=0,recved=216,sent=221)
   /10.5.152.224:36623[1](queued=0,recved=136,sent=136)
   /10.5.152.224:36598[1](queued=0,recved=234,sent=235)
   /10.5.152.224:36602[1](queued=0,recved=225,sent=231)
   /10.5.152.232:49143[1](queued=0,recved=1179,sent=1179)
   /10.5.152.224:36604[1](queued=0,recved=226,sent=235)
   /10.5.152.232:49103[1](queued=0,recved=217,sent=221)
   /10.5.152.232:49101[1](queued=0,recved=189,sent=190)
  
  Latency min/avg/max: 0/0/9
  Received: 4731
  Sent: 4791
  Outstanding: 0
  Zxid: 0x1100000c68
  Mode: leader
  Node count: 156
 eng-poc02.archerlab.local:5181
  Zookeeper version: 3.3.2--1, built on 06/05/2012 18:33 GMT
  Clients:
   /10.5.152.224:57393[1](queued=0,recved=210,sent=213)
   /10.5.152.222:35915[1](queued=0,recved=169,sent=169)
   /10.5.152.224:57441[1](queued=0,recved=51,sent=51)
   /10.5.152.232:48892[0](queued=0,recved=1,sent=0)
   /10.5.152.222:35906[1](queued=0,recved=230,sent=238)
   /10.5.152.222:35903[1](queued=0,recved=226,sent=233)
   /10.5.152.224:57390[1](queued=0,recved=226,sent=235)
   /10.5.152.222:35902[1](queued=0,recved=228,sent=237)
   /10.5.152.224:57395[1](queued=0,recved=221,sent=227)
   /10.5.152.224:57386[1](queued=0,recved=191,sent=193)
   /10.5.152.224:57409[1](queued=0,recved=2578,sent=2584)
   /10.5.152.232:48826[1](queued=0,recved=217,sent=221)
   /10.5.152.224:57407[1](queued=0,recved=170,sent=170)
   /10.5.152.232:48830[1](queued=0,recved=219,sent=224)
   /10.5.152.224:57388[1](queued=0,recved=224,sent=230)
   /10.5.152.232:48817[1](queued=0,recved=150,sent=154)
  
  Latency min/avg/max: 0/0/9
  Received: 6068
  Sent: 6163
  Outstanding: 0
  Zxid: 0x1100000c68
  Mode: follower
  Node count: 156
 eng-poc07.archerlab.local:5181
  Zookeeper version: 3.3.2--1, built on 06/05/2012 18:33 GMT
  Clients:
   /10.5.152.232:51143[1](queued=0,recved=227,sent=229)
   /10.5.152.222:36076[1](queued=0,recved=225,sent=232)
   /10.5.152.232:51132[1](queued=0,recved=213,sent=218)
   /10.5.152.222:36090[1](queued=0,recved=138,sent=140)
   /10.5.152.222:36079[1](queued=0,recved=226,sent=233)
   /10.5.152.232:51128[1](queued=0,recved=223,sent=229)
   /10.5.152.232:51142[1](queued=0,recved=26059,sent=28200)
   /10.5.152.222:36072[1](queued=0,recved=189,sent=189)
   /10.5.152.224:60011[1](queued=0,recved=187,sent=187)
   /10.5.152.222:36081[1](queued=0,recved=281,sent=323)
   /10.5.152.222:36071[1](queued=0,recved=194,sent=197)
   /10.5.152.222:36080[1](queued=0,recved=274,sent=274)
   /10.5.152.232:51196[0](queued=0,recved=1,sent=0)
   /10.5.152.222:36086[1](queued=0,recved=136,sent=136)
   /10.5.152.232:51130[1](queued=0,recved=225,sent=233)
   /10.5.152.232:51126[1](queued=0,recved=186,sent=186)
   /10.5.152.232:51140[1](queued=0,recved=170,sent=170)
   /10.5.152.224:60000[1](queued=0,recved=187,sent=187)
   /10.5.152.232:51135[1](queued=0,recved=186,sent=186)
   /10.5.152.222:36070[1](queued=0,recved=230,sent=231)
   /10.5.152.232:51168[1](queued=0,recved=135,sent=135)
  
  Latency min/avg/max: 0/0/39
  Received: 30553
  Sent: 32780
  Outstanding: 0
  Zxid: 0x1100000c68
  Mode: follower
  Node count: 156
</code></pre>

Any help in debugging this problem is appreciated.

FYI: Restarted the Zookeeper instances, still the same issue and restarted the cluster and didn't solve either. I can still connect to HBase from my client(Eclipse), using just HBase API.

Outcomes