AnsweredAssumed Answered

HBase Connection refused

Question asked by spettinato on Nov 11, 2013
Latest reply on Oct 28, 2015 by bollaking
Hi, I am getting an error running a simple table scan using HTablePool. 

Configuration

 - MapR version 3.0.1.21771.GA
 - Local VM CentOS Single Node Cluster
 - The error occurs both when the HBase table exists, and when it does not exist.

I have boiled the code down to the following,

    import java.io.IOException;
    
    import org.apache.hadoop.hbase.HBaseConfiguration;
    import org.apache.hadoop.hbase.client.HTablePool;
    
    public class DeleteMe {

/**
 * @param args
 */
public static void main(String[] args) {
 
  if (args.length != 1) {
   System.out.println("First argument should be the table name");
   System.exit(1);
  } 
  String tableName = args[0];
 
  HTablePool htp = new HTablePool(HBaseConfiguration.create(), 1);
 
  htp.getTable(tableName);
 
  try {
   htp.close();
  } catch (IOException e) {
   e.printStackTrace();
  }

}
}

----------
When I run this I get the following output

    Fetching MapR default for: hbase.mappingrule.impl
    Fetching MapR default for: hbase.table.namespace.mappings
    Fetching MapR default for: hbase.available
    Fetching MapR default for: hbase.zookeeper.recoverable.waittime
    Fetching MapR default for: hbase.client.rpc.maxattempts
    Fetching MapR default for: hbase.rpc.timeout
    Fetching MapR default for: hbase.client.prefetch.limit
    Fetching MapR default for: hbase.meta.scanner.caching
    Fetching MapR default for: hbase.client.instance.id
    Fetching MapR default for: hadoop.spoof.user
    Fetching MapR default for: hadoop.security.authentication
     Creating new Groups object
    Fetching MapR default for: hadoop.security.group.mapping
    Fetching MapR default for: hadoop.native.lib
    Trying to load the custom-built native-hadoop library...
    Loaded the native-hadoop library
    Using JniBasedUnixGroupsMapping for Group resolution
    Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
    Fetching MapR default for: hadoop.security.groups.cache.secs
    Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
    Fetching MapR default for: hadoop.security.auth_to_local
    Fetching MapR default for: hbase.client.rpc.maxattempts
    Fetching MapR default for: hbase.rpc.timeout
    Fetching MapR default for: hbase.client.prefetch.limit
    Fetching MapR default for: hbase.htable.threads.max
    Fetching MapR default for: hbase.htable.threads.keepalivetime
    Fetching MapR default for: zookeeper.znode.rs
    Fetching MapR default for: zookeeper.znode.draining.rs
    Fetching MapR default for: zookeeper.znode.master
    Fetching MapR default for: zookeeper.znode.backup.masters
    Fetching MapR default for: zookeeper.znode.state
    Fetching MapR default for: zookeeper.znode.unassigned
    Fetching MapR default for: zookeeper.znode.masterTableEnableDisable
    Fetching MapR default for: zookeeper.znode.clientTableEnableDisable
    Fetching MapR default for: zookeeper.znode.masterTableEnableDisable92
    Fetching MapR default for: zookeeper.znode.clusterId
    Fetching MapR default for: zookeeper.znode.splitlog
    hconnection opening connection to ZooKeeper with ensemble (localhost:2181)
    Fetching MapR default for: zookeeper.recovery.retry
    Fetching MapR default for: zookeeper.recovery.retry.intervalmill
    Fetching MapR default for: zookeeper.dump.connection.timeout
    Client environment:zookeeper.version=3.3.6--1, built on 09/07/2012 18:16 GMT
    Client environment:host.name=localhost.localdomain
    Client environment:java.version=1.6.0_24
    Client environment:java.vendor=Sun Microsystems Inc.
    Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
    Client environment:java.class.path=/home/hadoopdev/psbatchinstall/psUpdater.jar:/opt/ps/hadoop_library/emr-metrics-1.0.jar:/opt/ps/hadoop_library/servlet-api-2.5-6.1.14.jar:/opt/ps/hadoop_library/commons-lang-2.6.jar:/opt/ps/hadoop_library/aspectjtools-1.6.5.jar:/opt/ps/hadoop_library/mapr-hbase-1.0.3-mapr-3.0.1.jar:/opt/ps/hadoop_library/asm-3.2.jar:/opt/ps/hadoop_library/jasper-compiler-5.5.12.jar:/opt/ps/hadoop_library/jetty-6.1.26.jar:/opt/ps/hadoop_library/hbase-0.94.9-mapr-1308.jar:/opt/ps/hadoop_library/commons-httpclient-3.1.jar:/opt/ps/hadoop_library/commons-logging-1.0.4.jar:/opt/ps/hadoop_library/commons-collections-3.2.1.jar:/opt/ps/hadoop_library/jetty-servlet-tester-6.1.26.jar:/opt/ps/hadoop_library/jersey-json-1.8.jar:/opt/ps/hadoop_library/hsqldb-1.8.0.10.jar:/opt/ps/hadoop_library/commons-daemon-1.0.1.jar:/opt/ps/hadoop_library/amazon-s3.jar:/opt/ps/hadoop_library/jersey-core-1.8.jar:/opt/ps/hadoop_library/commons-logging-api-1.0.4.jar:/opt/ps/hadoop_library/hbase-0.94.9-mapr-1308-tests.jar:/opt/ps/hadoop_library/slf4j-api-1.4.3.jar:/opt/ps/hadoop_library/xmlenc-0.52.jar:/opt/ps/hadoop_library/logging-0.1.jar:/opt/ps/hadoop_library/protobuf-java-2.4.1.jar:/opt/ps/hadoop_library/commons-math-2.1.jar:/opt/ps/hadoop_library/httpcore-4.1.jar:/opt/ps/hadoop_library/mockito-all-1.8.2.jar:/opt/ps/hadoop_library/jersey-server-1.8.jar:/opt/ps/hadoop_library/commons-logging-1.1.1.jar:/opt/ps/hadoop_library/servlet-api-2.5-20081211.jar:/opt/ps/hadoop_library/mockito-all-1.8.5.jar:/opt/ps/hadoop_library/commons-net-1.4.1.jar:/opt/ps/hadoop_library/jackson-mapper-asl-1.5.2.jar:/opt/ps/hadoop_library/hadoop-0.20.2-dev-capacity-scheduler.jar:/opt/ps/hadoop_library/commons-cli-1.2.jar:/opt/ps/hadoop_library/baseutils-0.1.jar:/opt/ps/hadoop_library/jackson-core-asl-1.5.2.jar:/opt/ps/hadoop_library/commons-codec-1.5.jar:/opt/ps/hadoop_library/mysql-connector-java-5.0.8-bin.jar:/opt/ps/hadoop_library/httpclient-4.1.1.jar:/opt/ps/hadoop_library/commons-el-1.0.jar:/opt/ps/hadoop_library/slf4j-log4j12-1.4.3.jar:/opt/ps/hadoop_library/aws-java-sdk-1.3.26.jar:/opt/ps/hadoop_library/core-3.1.1.jar:/opt/ps/hadoop_library/commons-configuration-1.8.jar:/opt/ps/hadoop_library/log4j-1.2.15.jar:/opt/ps/hadoop_library/kfs-0.2.2.jar:/opt/ps/hadoop_library/mapr-hbase-1.0.3-mapr-3.0.1-tests.jar:/opt/ps/hadoop_library/commons-net-3.1.jar:/opt/ps/hadoop_library/maprfs-1.0.3-mapr-3.0.1.jar:/opt/ps/hadoop_library/guava-13.0.1.jar:/opt/ps/hadoop_library/hadoop-0.20.2-dev-core.jar:/opt/ps/hadoop_library/hadoop-0.20.2-dev-fairscheduler.jar:/opt/ps/hadoop_library/libprotodefs.jar:/opt/ps/hadoop_library/jetty-util-6.1.26.jar:/opt/ps/hadoop_library/junit-4.5.jar:/opt/ps/hadoop_library/commons-httpclient-3.0.1.jar:/opt/ps/hadoop_library/maprsecurity-0.1.jar:/opt/ps/hadoop_library/oro-2.0.8.jar:/opt/ps/hadoop_library/jasper-runtime-5.5.12.jar:/opt/ps/hadoop_library/jets3t-0.6.1.jar:/opt/ps/hadoop_library/eval-0.5.jar:/opt/ps/hadoop_library/zookeeper-3.3.6.jar:/opt/ps/hadoop_library/aspectjrt-1.6.5.jar:/opt/ps/hadoop_library/maprfs-diagnostic-tools-1.0.3-mapr-3.0.1.jar
    Client environment:java.library.path=/opt/mapr/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64/
    Client environment:java.io.tmpdir=/tmp
    Client environment:java.compiler=<NA>
    Client environment:os.name=Linux
    Client environment:os.arch=amd64
    Client environment:os.version=2.6.32-358.23.2.el6.x86_64
    Client environment:user.name=root
    Client environment:user.home=/root
    Client environment:user.dir=/home/hadoopdev/psbatchinstall
    Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
    zookeeper.disableAutoWatchReset is false
    Opening socket connection to server localhost/127.0.0.1:2181
    The identifier of this process is 11767@localhost.localdomain
    Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
    java.net.ConnectException: Connection refused
            at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
            at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:597)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
    Ignoring exception during shutdown input
    java.nio.channels.ClosedChannelException
            at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:661)
            at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:378)
            at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1229)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1194)
    Ignoring exception during shutdown output
    java.nio.channels.ClosedChannelException
            at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:672)
            at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:386)
            at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1236)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1194)
    Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
    Sleeping 2000ms before retry #1...

----------
Then the program continues to try in an infinite loop,

    Sleeping 2000ms before retry #1...
    Opening socket connection to server localhost/127.0.0.1:2181
    Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
    java.net.ConnectException: Connection refused
            at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
            at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:597)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
    Ignoring exception during shutdown input
    java.nio.channels.ClosedChannelException
            at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:661)
            at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:378)
            at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1229)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1194)
    Ignoring exception during shutdown output
    java.nio.channels.ClosedChannelException
            at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:672)
            at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:386)
            at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1236)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1194)
    Opening socket connection to server localhost/127.0.0.1:2181
    Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
    java.net.ConnectException: Connection refused
            at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
            at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:597)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
    Ignoring exception during shutdown input
    java.nio.channels.ClosedChannelException
            at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:661)
            at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:378)
            at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1229)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1194)
    Ignoring exception during shutdown output
    java.nio.channels.ClosedChannelException
            at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:672)
            at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:386)
            at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1236)
            at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1194)
    Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
    Sleeping 4000ms before retry #2...

Outcomes