AnsweredAssumed Answered

Spark-on-Yarn jobs failing due to Invalid attribute value for hadoop.security.authentication of maprsasl  error

Question asked by mikeengland on Jan 6, 2015
Hi,

I am trying to set up Spark-on-YARN but I am encountering some issues relating to authentication. I have MapR security set up and a ticket generated for the user. The application gets accepted but then fails as below. I have also added output from stderr logs on the application managers.

15/01/06 08:56:07 INFO yarn.Client: Application report for application_1420476370372_0023 (state: ACCEPTED)
15/01/06 08:56:08 INFO yarn.Client: Application report for application_1420476370372_0023 (state: ACCEPTED)
15/01/06 08:56:09 INFO yarn.Client: Application report for application_1420476370372_0023 (state: FAILED)
15/01/06 08:56:09 INFO yarn.Client:
         client token: N/A
         diagnostics: Application application_1420476370372_0023 failed 2 times due to AM Container for appattempt_1420476370372_0023_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
        at org.apache.hadoop.util.Shell.run(Shell.java:418)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:295)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:314)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

____________________

STDERR from application masters:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-mapr/nm-local-dir/filecache/10/spark-assembly-1.2.0-hadoop2.3.0-mapr-4.0.0-FCS.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/filecache/54/spark-assembly-1.2.0-hadoop2.3.0-mapr-4.0.0-FCS.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/01/06 08:56:03 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:513)
        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: org.apache.spark.SparkException: Unable to load YARN support
        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:199)
        at org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:194)
        at org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
        ... 2 more
Caused by: java.lang.IllegalArgumentException: Invalid attribute value for hadoop.security.authentication of maprsasl
        at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:677)
        at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:227)
        at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:252)
        at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:43)
        at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.<init>(YarnSparkHadoopUtil.scala:45)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at java.lang.Class.newInstance(Class.java:374)
        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:196)


____________________

Has anyone encountered the error above (java.lang.IllegalArgumentException: Invalid attribute value for hadoop.security.authentication of maprsasl at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:677) ) ?

Thanks,
Mike

Outcomes