AnsweredAssumed Answered

Configuring Spark remote client that is not part of cluster - failing with error in hadoop userlogs

Question asked by rvelfre on May 12, 2016
Latest reply on Nov 15, 2016 by mvince


Hello,

 

I would like to execute a Spark application using spark submit from a remote machine that is not part of my cluster (I'm not using any MapR Client or Mapr Client posix).

 

I installed Spark and copied all Yarn configuration file from my cluster to the remote machine.

 

I'm able to execute the SparkPi application from my remote cluster and I can see that the application  has been submitted to my cluster on YARN.

 

However the spark application is failing with the following ERROR (found on hadoop userlogs directory):

 

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/tmp/hadoop-mapr/nm-local-dir/filecache/3477/spark-assembly-1.5.2-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

16/05/12 14:29:30 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]

16/05/12 14:29:31 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1460361889798_2237_000002

16/05/12 14:29:31 INFO spark.SecurityManager: Changing view acls to: mapr

16/05/12 14:29:31 INFO spark.SecurityManager: Changing modify acls to: mapr

16/05/12 14:29:31 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mapr); users with modify permissions: Set(mapr)

16/05/12 14:29:31 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread

16/05/12 14:29:31 INFO yarn.ApplicationMaster: Waiting for spark context initialization

16/05/12 14:29:31 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...

16/05/12 14:29:31 INFO spark.SparkContext: Running Spark version 1.5.2

16/05/12 14:29:31 WARN spark.SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).

16/05/12 14:29:31 INFO spark.SecurityManager: Changing view acls to: mapr

16/05/12 14:29:31 INFO spark.SecurityManager: Changing modify acls to: mapr

16/05/12 14:29:31 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mapr); users with modify permissions: Set(mapr)

16/05/12 14:29:32 INFO slf4j.Slf4jLogger: Slf4jLogger started

16/05/12 14:29:32 INFO Remoting: Starting remoting

16/05/12 14:29:32 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.132.20.108:48974]

16/05/12 14:29:32 INFO util.Utils: Successfully started service 'sparkDriver' on port 48974.

16/05/12 14:29:32 INFO spark.SparkEnv: Registering MapOutputTracker

16/05/12 14:29:32 INFO spark.SparkEnv: Registering BlockManagerMaster

16/05/12 14:29:32 INFO storage.DiskBlockManager: Created local directory at /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1460361889798_2237/blockmgr-6dff57d3-8595-42a3-925f-5d29ad8d1c59

16/05/12 14:29:32 INFO storage.MemoryStore: MemoryStore started with capacity 983.1 MB

16/05/12 14:29:32 INFO spark.HttpFileServer: HTTP File server directory is /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1460361889798_2237/spark-dfee233c-e72d-4ccf-a5d7-82ea5e2f2add/httpd-cefc129a-3b2f-451f-b1cf-dc43b5b387b9

16/05/12 14:29:32 INFO spark.HttpServer: Starting HTTP Server

16/05/12 14:29:32 INFO server.Server: jetty-8.y.z-SNAPSHOT

16/05/12 14:29:32 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:54140

16/05/12 14:29:32 INFO util.Utils: Successfully started service 'HTTP file server' on port 54140.

16/05/12 14:29:32 INFO spark.SparkEnv: Registering OutputCommitCoordinator

16/05/12 14:29:32 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter

16/05/12 14:29:32 INFO server.Server: jetty-8.y.z-SNAPSHOT

16/05/12 14:29:32 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:53969

16/05/12 14:29:32 INFO util.Utils: Successfully started service 'SparkUI' on port 53969.

16/05/12 14:29:32 INFO ui.SparkUI: Started SparkUI at http://10.132.20.108:53969

16/05/12 14:29:32 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler

16/05/12 14:29:32 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.

16/05/12 14:29:32 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57362.

16/05/12 14:29:32 INFO netty.NettyBlockTransferService: Server created on 57362

16/05/12 14:29:32 INFO storage.BlockManagerMaster: Trying to register BlockManager

16/05/12 14:29:32 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.132.20.108:57362 with 983.1 MB RAM, BlockManagerId(driver, 10.132.20.108, 57362)

16/05/12 14:29:32 INFO storage.BlockManagerMaster: Registered BlockManager

16/05/12 14:29:32 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka://sparkDriver/user/YarnAM#1394650146])

16/05/12 14:29:32 INFO client.RMProxy: Connecting to ResourceManager at hdpsrvrec1.intranet.darty.fr/10.132.20.107:8030

16/05/12 14:29:32 INFO yarn.YarnRMClient: Registering the ApplicationMaster

16/05/12 14:29:33 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): DIGEST-MD5: digest response format violation. Mismatched URI: default/; expecting: null/default

16/05/12 14:29:33 ERROR yarn.ApplicationMaster: Uncaught exception:

javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched URI: default/; expecting: null/default [Caused by org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): DIGEST-MD5: digest response format violation. Mismatched URI: default/; expecting: null/default]

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

        at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)

        at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)

        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

        at com.sun.proxy.$Proxy11.registerApplicationMaster(Unknown Source)

        at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:222)

        at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.registerApplicationMaster(AMRMClientImpl.java:214)

        at org.apache.spark.deploy.yarn.YarnRMClient.register(YarnRMClient.scala:75)

        at org.apache.spark.deploy.yarn.ApplicationMaster.registerAM(ApplicationMaster.scala:254)

        at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:307)

        at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:155)

        at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:625)

        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)

        at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

        at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)

        at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:623)

        at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): DIGEST-MD5: digest response format violation. Mismatched URI: default/; expecting: null/default

        at org.apache.hadoop.ipc.Client.call(Client.java:1468)

        at org.apache.hadoop.ipc.Client.call(Client.java:1399)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

        at com.sun.proxy.$Proxy10.registerApplicationMaster(Unknown Source)

        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)

        ... 22 more

16/05/12 14:29:33 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched URI: default/; expecting: null/default [Caused by org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): DIGEST-MD5: digest response format violation. Mismatched URI: default/; expecting: null/default])

16/05/12 14:29:33 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched URI: default/; expecting: null/default [Caused by org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): DIGEST-MD5: digest response format violation. Mismatched URI: default/; expecting: null/default])

16/05/12 14:29:33 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1460361889798_2237

...

 

Seems like my digest MD5 URI value is not the same on my MapR cluster and on my remote machine.

How could I re-configure this value (either on my remote machine or on the cluster) ?

 

 

Thanks for you help.

Raphaël

Outcomes