AnsweredAssumed Answered

Tez authentication error on MapR 6

Question asked by rbukarev on Feb 22, 2018
Latest reply on Feb 26, 2018 by maprcommunity

I guess it's a continuation of my security struggles (here part1 and part2 -- part1 is not yet fully solved btw, and I guess it's all somehow related).

 

So, thanks to John Humphreys I've been able to run a spark-submit job after doing the "export MAPR_TICKETFILE_LOCATION" trick. I've added that line to my .bashrc, to not think of that ever.. for 1 month. 

 

Then, using beeline, I created a Hive table in a database, to which I'm granted all privileges. Next, I tried to run a simple SQL statement (also in beeline): select count(*) from mydb.mytable;

 

Got an error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask (state=08S01,code=1)

 

Here's related part of the hive.log:

2018-02-22T22:47:06,831 INFO [HiveServer2-Background-Pool: Thread-37] client.TezClientUtils: Using tez.lib.uris.classpath value from configuration: null
2018-02-22T22:47:07,609 INFO [HiveServer2-Background-Pool: Thread-37] client.TezClient: Tez system stage directory maprfs:/user/mapr/tmp/hive/mapr/_tez_session_dir/a26e36b9-fa3a-4397-b899-d6f1e9ee753c/.tez/application_1519333629126_0001 doesn't exist and is created
2018-02-22T22:47:07,770 WARN [HiveServer2-Background-Pool: Thread-37] acls.ATSHistoryACLPolicyManager: Could not post timeline domain
com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 401, message: Authentication required
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter$1.run(TimelineClientImpl.java:233)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:182)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter.handle(TimelineClientImpl.java:244)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:533)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingObject(TimelineClientImpl.java:501)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$1.run(TimelineClientImpl.java:346)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$1.run(TimelineClientImpl.java:343)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1631)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPosting(TimelineClientImpl.java:343)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putDomain(TimelineClientImpl.java:336)
at org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.createTimelineDomain(ATSHistoryACLPolicyManager.java:127)
at org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.createSessionDomain(ATSHistoryACLPolicyManager.java:165)
at org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.setupSessionACLs(ATSHistoryACLPolicyManager.java:226)
at org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:573)
at org.apache.tez.client.TezClient.start(TezClient.java:462)

 

A similar thing occurs in Zeppelin, with hive interpreter configured (jdbc):

java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:349)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251)

 

Is there any other setting, except exporting the ticket location, that I haven't done? 

 

Worth mentioning, all the components I have/had trouble with (Spark, Hive, Tez) have been installed using MapR Installer, I'd expect this whole process to be smoother..

Outcomes