AnsweredAssumed Answered

Spark:  Failing to connect to hive metastore in yarn-cluster mode

Question asked by raju.bairishetti on Aug 31, 2016
Latest reply on Dec 6, 2016 by asukhenko

Hello,

    We are using mapr5 distribution. Currently running spark 1.5.2 and hive 1.2.0 in our cluster. We are able to run spark sql jobs in yarn-client mode. But when we submit jobs in yarn-cluster mode driver(which is in AM) is not able to connect to hive metastore.

 

We are seeing the following logs in the Application master (which has driver logs):

 

16/09/01 02:58:31 INFO hive.HiveContext: Initializing execution hive, version 1.2.1

16/09/01 02:58:31 INFO client.ClientWrapper: Inspected Hadoop version: 2.7.0-mapr-1506

16/09/01 02:58:31 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.7.0-mapr-1506

Trying to connect to metastore with URI thrift://ds-hadoop-0001:9083

16/09/01 03:00:32 WARN hive.metastore: set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.

org.apache.thrift.transport.TTransportException

  at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)

  at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

  at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)

  at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)

  at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)

  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3688)

  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3674)

  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:436)

  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236)

  at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)

  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

  at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)

  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)

  at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)

  at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3058)

  at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3077)

  at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)

  at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)

  at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166)

 

 

 

This is happening while fetching the databases from hive metastore.

   

16/09/01 03:00:32 ERROR hive.log: Got exception: org.apache.thrift.transport.TTransportException null

org.apache.thrift.transport.TTransportException

  at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)

  at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

  at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)

  at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)

  at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)

  at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)

  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_databases(ThriftHiveMetastore.java:739)

  at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_databases(ThriftHiveMetastore.java:727)

  at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1031)

 

 

 

We are using hive 1.2.0-mapr-1601  and spark sql is internally is using 1.2.1 hive. Looks like this is becuase of 1.2.1 client is not able to talk to 1.2.0 hive server.

 

From mapr hive: hive/HiveMetaStoreClient.java at branch-1.2 · mapr/hive · GitHub

 

from the code it says creates a client and failing while setting ugi

 

try {
  transport.open();
  isConnected = true;
  } catch (TTransportException e) {
  tte = e;
   if (LOG.isDebugEnabled()) {
   LOG.warn("Failed to connect to the MetaStore Server...", e);
  } else {
   // Don't print full exception trace if DEBUG is not on.
   LOG.warn("Failed to connect to the MetaStore Server...");
  }
  }
   if (isConnected && !useSasl && conf.getBoolVar(ConfVars.METASTORE_EXECUTE_SET_UGI)){
   // Call set_ugi, only in unsecure mode.
   try {
   UserGroupInformation ugi = Utils.getUGI();
  client.set_ugi(ugi.getUserName(), Arrays.asList(ugi.getGroupNames()));
  } catch (LoginException e) {
   LOG.warn("Failed to do login. set_ugi() is not successful, " +
   "Continuing without it.", e);
  } catch (IOException e) {
   LOG.warn("Failed to find ugi of client set_ugi() is not successful, " +
   "Continuing without it.", e);
  } catch (TException e) {
  LOG.warn("set_ugi() not successful, Likely cause: new client talking to old server. "
   + "Continuing without it.", e);
  }
  }

Outcomes