prakhar

Hive-Sentry Integration: Connection refused

Blog Post created by prakhar Employee on Dec 5, 2016

By default,zookeeper port is set to 2181 in hive-default.xml.template. And sometimes zookeeper services are not running on the same port(eg.  5181). We need to ensure this that the listening port for both sentry and zookeeper should be same in event of integrating hive with sentry.Sentry uses hive configuration to get the zookeeper client port details to integrate with  hive.

 

Below is the error you will get when you encounter such issue:

 

2013-10-15 11:47:03,180 WARN zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
2013-10-15 11:47:03,290 WARN ZooKeeperHiveLockManager (ZooKeeperHiveLockManager.java:setContext(136)) - Unexpected ZK exception when creating parent node /hive_zookeeper_namespace
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hive_zookeeper_namespace
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:643)
at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.setContext(ZooKeeperHiveLockManager.java:132)
at org.apache.hadoop.hive.ql.Driver.setLockManager(Driver.java:169)
at org.apache.hadoop.hive.ql.Driver.checkLockManager(Driver.java:142)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:910)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)

 

 

 

You can determine on which port zookeeper services are running by using below command

 

netstat -plant|grep 2181

 

If you havent get the output check on 5181 port using the same command
 netstat -plant|grep 5181
tcp        0      0 :::5181                     :::*                        LISTEN      6895/java
tcp        0      0 ::ffff:10.10.71.21:5181     ::ffff:10.10.71.21:41782    ESTABLISHED 6895/java
tcp        0      0 ::ffff:10.10.71.21:5181     ::ffff:10.10.71.21:42472    ESTABLISHED 6895/java
tcp        0      0 ::ffff:10.10.71.21:5181     ::ffff:10.10.71.20:54342    ESTABLISHED 6895/java
tcp        0      0 ::ffff:10.10.71.21:5181     ::ffff:10.10.71.21:41778    ESTABLISHED 6895/java
tcp        0      0 ::ffff:10.10.71.21:5181     ::ffff:10.10.71.20:54338    ESTABLISHED 6895/java

 

Check the java process 6895 using below command

 

ps -ef|grep 6895

mapr      6895     1  0 Nov23 ?        00:14:41 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/bin/java -Dzookeeper.log.dir=/opt/mapr/zookeeper/zookeeper-3.4.5/logs -Dzookeeper.root.logger=INFO, ROLLINGFILE -XX:ErrorFile=/opt/mapr/zookeeper/zookeeper-3.4.5/logs/hs_err_pid%p.log -cp /opt/mapr/zookeeper/zookeeper-3.4.5/bin/../build/classes:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../build/lib/*.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/slf4j-api-1.6.1.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/netty-3.2.2.Final.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/log4j-1.2.15.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/jline-0.9.94.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../zookeeper-3.4.5-mapr-1604.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../src/java/lib/*.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/conf::/opt/mapr/lib/maprfs-5.2.0-mapr.jar:/opt/mapr/lib/protobuf-java-2.5.0.jar:::/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/flexjson-2.1.jar:/opt/mapr/lib/commons-codec-1.5.jar -Djava.library.path=/opt/mapr/lib -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.saslprovider=com.mapr.security.maprsasl.MaprSaslProvider -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/mapr/zookeeper/zookeeper-3.4.5/conf/zoo.cfg

 


Configure below property accordingly and update the hive-site.xml


 <property>
    <name>hive.zookeeper.client.port</name>
    <value>5181</value>
    <description>
      The port of ZooKeeper servers to talk to.
      If the list of Zookeeper servers specified in hive.zookeeper.quorum
      does not contain port numbers, this value is used.
    </description>
  </property>

Outcomes