AnsweredAssumed Answered

File /user/mapr/tmp/hive does not exist

Question asked by thbeh on Mar 16, 2017
Latest reply on Mar 22, 2017 by maprcommunity

Hi, I am testing a spark app that will run in a mapr client node and getting the above. I have change the hive scratch dir on the MapR cluster and the hive logs showed that there's activities in the scratch dir. The logs & config shows the changes -

 

Config

[mapr@node1 mapr]$ hive -e 'set;' | grep scratch

Logging initialized using configuration in jar:file:/opt/mapr/hive/hive-1.2/lib/hive-common-1.2.0-mapr-1611.jar!/hive-log4j.properties
hive.exec.local.scratchdir=/tmp/mapr
hive.exec.scratchdir=/tmp/
hive.scratch.dir.permission=700
hive.scratchdir.in.dest=_scratchdir
hive.start.cleanup.scratchdir=false
[mapr@node1 mapr]$

 

hive.log

2017-03-16 20:32:18,088 INFO [main]: SessionState (SessionState.java:printInfo(954)) -
Logging initialized using configuration in jar:file:/opt/mapr/hive/hive-1.2/lib/hive-common-1.2.0-mapr-1611.jar!/hive-log4j.properties
2017-03-16 20:32:18,248 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(379)) - Trying to connect to metastore with URI thrift://node2:9083
2017-03-16 20:32:18,275 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(424)) - Opened a connection to metastore, current connections: 1
2017-03-16 20:32:18,301 INFO [main]: hive.metastore (HiveMetaStoreClient.java:open(476)) - Connected to metastore.
2017-03-16 20:32:18,602 INFO [main]: session.SessionState (SessionState.java:createPath(644)) - Created HDFS directory: /tmp/mapr
2017-03-16 20:32:18,640 INFO [main]: session.SessionState (SessionState.java:createPath(644)) - Created local directory: /tmp/c474306d-97f4-45d6-bfd7-6d79140d67f7_resources
2017-03-16 20:32:18,654 INFO [main]: session.SessionState (SessionState.java:createPath(644)) - Created HDFS directory: /tmp/mapr/c474306d-97f4-45d6-bfd7-6d79140d67f7
2017-03-16 20:32:18,656 INFO [main]: session.SessionState (SessionState.java:createPath(644)) - Created local directory: /tmp/mapr/c474306d-97f4-45d6-bfd7-6d79140d67f7
2017-03-16 20:32:18,659 INFO [main]: session.SessionState (SessionState.java:createPath(644)) - Created HDFS directory: /tmp/mapr/c474306d-97f4-45d6-bfd7-6d79140d67f7/_tmp_space.db

 

Spark-shell

java.lang.RuntimeException: java.io.FileNotFoundException: File /user/mapr/tmp/hive does not exist
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:189)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:247)
at org.apache.spark.sql.hive.HiveClientUtil.newClient(HiveClientUtil.scala:235)
at org.apache.spark.sql.hive.HiveClientUtil.<init>(HiveClientUtil.scala:129)
at org.apache.spark.sql.internal.SnappySharedState.metadataHive$lzycompute(SnappySharedState.scala:33)
at org.apache.spark.sql.internal.SnappySharedState.metadataHive(SnappySharedState.scala:33)
at org.apache.spark.sql.internal.SnappySharedState.externalCatalog$lzycompute(SnappySharedState.scala:37)
at org.apache.spark.sql.internal.SnappySharedState.externalCatalog(SnappySharedState.scala:36)
at org.apache.spark.sql.internal.SnappySessionState.catalog$lzycompute(SnappySessionState.scala:200)
at org.apache.spark.sql.internal.SnappySessionState.catalog(SnappySessionState.scala:199)
at org.apache.spark.sql.internal.SnappySessionState.catalog(SnappySessionState.scala:53)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:568)
at org.apache.spark.sql.SparkSession.table(SparkSession.scala:564)
... 48 elided
Caused by: java.io.FileNotFoundException: File /user/mapr/tmp/hive does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:607)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:877)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:597)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:602)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
... 61 more

 

Are there any change I need to make in the MaprR client node to use /tmp/mapr instead of /user/mapr/tmp/hive.

 

Thanks in advance

Outcomes