AnsweredAssumed Answered

Problem Loading data to Hbase Table - ConnectionLoss for /hbase

Question asked by harish on Sep 29, 2012
Latest reply on Nov 15, 2012 by nabeel
Hello folks,

I am running a java program to load data into a hbase table. The job fails with the below zookeeper error. Tried increasing hbase.zookeeper.property.maxClientCnxns to 1000 in hbase_site.xml of client, master and regionservers as well as zoo.cfg of all zk nodes. Still getting the same error.
Any suggestions would be of great help.

P.S. : I can see around 2k records loaded in the hbase table every time even if the job actually fails

Thanks
Harish

Error :
<pre>
12/09/30 14:35:20 INFO fs.JobTrackerWatcher: Current running JobTracker is: ibm-r1-node2.mapr-nextgen.com/10.132.100.226:9001
12/09/30 14:35:20 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/09/30 14:35:20 INFO input.FileInputFormat: Total input paths to process : 1
12/09/30 14:35:20 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/09/30 14:35:20 WARN snappy.LoadSnappy: Snappy native library not loaded
12/09/30 14:35:20 INFO mapred.JobClient: Creating job's output directory at /pigoutput/out1
12/09/30 14:35:20 INFO mapred.JobClient: Creating job's user history location directory at /pigoutput/out1/_logs
12/09/30 14:35:20 INFO mapred.JobClient: Running job: job_201209292153_0004
12/09/30 14:35:21 INFO mapred.JobClient:  map 0% reduce 0%
12/09/30 14:35:42 INFO mapred.JobClient: Task Id : attempt_201209292153_0004_m_000001_0, Status : FAILED on node ibm-r1-node14.mapr-nextgen.com
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
at LoadDataToHbase$DataLoadMapper.map(LoadDataToHbase.java:38)
at LoadDataToHbase$DataLoadMapper.map(LoadDataToHbase.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:668)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 15 more

12/09/30 14:36:26 INFO mapred.JobClient: Task Id : attempt_201209292153_0004_m_000012_2, Status : FAILED on node ibm-r1-node4.mapr-nextgen.com
org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
at LoadDataToHbase$DataLoadMapper.map(LoadDataToHbase.java:38)
at LoadDataToHbase$DataLoadMapper.map(LoadDataToHbase.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:668)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 15 more

attempt_201209292153_0004_m_000012_2: log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapred.Task).
attempt_201209292153_0004_m_000012_2: log4j:WARN Please initialize the log4j system properly.
12/09/30 14:36:39 INFO mapred.JobClient: Job complete: job_201209292153_0004
12/09/30 14:36:39 INFO mapred.JobClient: Counters: 8
12/09/30 14:36:39 INFO mapred.JobClient:   Job Counters
12/09/30 14:36:39 INFO mapred.JobClient:     Aggregate execution time of mappers(ms)=1006178
12/09/30 14:36:39 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
12/09/30 14:36:39 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
12/09/30 14:36:39 INFO mapred.JobClient:     Rack-local map tasks=23
12/09/30 14:36:39 INFO mapred.JobClient:     Launched map tasks=36
12/09/30 14:36:39 INFO mapred.JobClient:     Data-local map tasks=13
12/09/30 14:36:39 INFO mapred.JobClient:     Aggregate execution time of reducers(ms)=0
12/09/30 14:36:39 INFO mapred.JobClient:     Failed map tasks=1

Outcomes