AnsweredAssumed Answered

MapR Ticket expiry in spark streaming?

Question asked by john.humphreys on Dec 1, 2017
Latest reply on Jan 16, 2018 by john.humphreys

I have a Spark Streaming job which was up for around a week.  The job was launched with spark-submit on YARN with a long-lived MapR ticket good until 2021 exported in MAPR_TICKETFILE_LOCATION.

 

The job failed after a week and that may or may not have been ticket related; but in any case, when it was re-attempted by YARN, it got the following ticket error immediately.

 

Why would the re-attempt not have the long-lived ticket?  Can we fix this?

 

Exception in thread "main" java.io.IOException: failure to login: Unable to obtain MapR credentials      at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:751)      at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:688)      at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:572)      at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:65)      at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:764)      at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) Caused by: javax.security.auth.login.LoginException: Unable to obtain MapR credentials      at com.mapr.security.maprsasl.MaprSecurityLoginModule.login(MaprSecurityLoginModule.java:227)      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)      at java.lang.reflect.Method.invoke(Method.java:483)      at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)      at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)      at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)      at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)      at java.security.AccessController.doPrivileged(Native Method)      at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)      at javax.security.auth.login.LoginContext.login(LoginContext.java:587)      at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:724)      ... 5 more Caused by: com.mapr.login.MapRLoginException: Found ticket for cluster 'AMPHADP21' but it has expired.      at com.mapr.login.client.MapRLoginHttpsClient.doesSecurityHaveGoodKey(MapRLoginHttpsClient.java:233)      at com.mapr.login.client.MapRLoginHttpsClient.authenticateIfNeeded(MapRLoginHttpsClient.java:154)      at com.mapr.login.client.MapRLoginHttpsClient.authenticateIfNeeded(MapRLoginHttpsClient.java:115)      at com.mapr.security.maprsasl.MaprSecurityLoginModule.login(MaprSecurityLoginModule.java:221)      ... 17 more

Outcomes