AnsweredAssumed Answered

Spark application cannot connect to Hive when security enabled

Question asked by sourcedelica on Jul 7, 2015
Latest reply on Jul 9, 2015 by sourcedelica
I'm running MapR 4.1, Spark 1.3.1.

I am trying to run a Spark application that works correctly on another MapR cluster with the same configuration - except for security being enabled.   The stacktrace is below.

`spark-shell` works fine on the cluster with security.   I can replicate the problem in `spark-shell` by commenting out these lines in `spark-env.sh`:

    #if [ "$MAPR_SECURITY_STATUS" = "true" ]; then
    #  SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dmapr_sec_enabled=true"
    #fi

But setting `-Dmapr_sec_enabled=true` in my application does not fix the problem.  I verified that the property is set by checking the system properties in the Spark application UI.

![alt text][1]

Here is the stacktrace:

    2015-07-08 00:19:34,557  INFO [Remote-akka.actor.default-dispatcher-2] (hive.metastore) - Trying to connect to metastore with URI thrift://redlxd00005.mycorp.com:9083
    2015-07-08 00:19:54,564  WARN [Remote-akka.actor.default-dispatcher-2] (hive.metastore) - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
    org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
            at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
            at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
            at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
            at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
            at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
            at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
            at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3266)
            at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3252)
            at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:352)
            at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:214)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
            at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
            at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
            at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
            at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
            at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
            at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
            at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
            at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
            at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:976)
            at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:950)
            at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:201)
            at org.apache.spark.sql.hive.HiveContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HiveContext.scala:262)
            at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply(Catalog.scala:161)


  [1]: /storage/temp/266-2015-07-07-19-45-08-notebook-environment.png

Outcomes