AnsweredAssumed Answered

Issue with spark-submit with YARN and deploy-mode cluster

Question asked by sagar.sonawane on Jul 31, 2017
Latest reply on Aug 2, 2017 by cathy

I am facing error given below, when I try to submit from Linux client machine,

 

[mapr@ip-xx-xxx-xx-xxx spark-2.1.0]$ /opt/mapr/spark/spark-2.1.0/bin/spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.1.0-mapr-1703.jar
Spark Command: /opt/jdk1.8.0_121/bin/java -cp /opt/mapr/spark/spark-2.1.0/conf/:/opt/mapr/spark/spark-2.1.0/jars/*:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/:/opt/mapr/conf/:/opt/mapr/lib/maprbuildversion-5.2.0-mapr.jar:/opt/mapr/lib/maprfs-5.2.0-mapr.jar:/opt/mapr/lib/maprfs-5.2.0-mapr-tests.jar:/opt/mapr/lib/maprfs-diagnostic-tools-5.2.0-mapr.jar:/opt/mapr/lib/maprdb-5.2.0-mapr.jar:/opt/mapr/lib/maprdb-5.2.0-mapr-tests.jar:/opt/mapr/lib/maprdb-mapreduce-5.2.0-mapr.jar:/opt/mapr/lib/maprdb-mapreduce-5.2.0-mapr-tests.jar:/opt/mapr/lib/maprdb-shell-5.2.0-mapr.jar:/opt/mapr/lib/mapr-hbase-5.2.0-mapr.jar:/opt/mapr/lib/mapr-hbase-5.2.0-mapr-tests.jar:/opt/mapr/lib/mapr-streams-5.2.0-mapr.jar:/opt/mapr/lib/mapr-streams-5.2.0-mapr-tests.jar:/opt/mapr/lib/mapr-tools-5.2.0-mapr.jar:/opt/mapr/lib/mapr-tools-5.2.0-mapr-tests.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/central-logging-5.2.0-mapr.jar:/opt/mapr/lib/antlr4-runtime-4.5.jar:/opt/mapr/lib/guava-14.0.1.jar:/opt/mapr/lib/jackson-annotations-2.7.2.jar:/opt/mapr/lib/jackson-core-2.7.2.jar:/opt/mapr/lib/jackson-databind-2.7.2.jar:/opt/mapr/lib/jline-2.11.jar:/opt/mapr/lib/ojai-1.1.jar:/opt/mapr/lib/ojai-mapreduce-1.1.jar:/opt/mapr/lib/protobuf-java-2.5.0.jar:/opt/mapr/lib/spring-asm-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-beans-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-context-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-core-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-expression-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-shell-1.2.0.M1-mapr-1607.jar:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/*:/opt/mapr/hadoop/hadoop-2.7.0/contrib/capacity-scheduler/*.jar:/opt/mapr/lib/kvstore*.jar:/opt/mapr/lib/libprotodefs*.jar:/opt/mapr/lib/baseutils*.jar:/opt/mapr/lib/maprutil*.jar:/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/flexjson-2.1.jar org.apache.spark.deploy.SparkSubmit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.1.0-mapr-1703.jar
========================================
17/07/31 10:29:30 ERROR MapRFileSystem: Failed to delete path maprfs:/user/mapr/.sparkStaging/application_1501065125932_0040, error: No such file or directory (2)

 

Whereas, it is successfull when executed in "client" mode,

 

[mapr@ip-xx-xxx-xx-xxx spark-2.1.0]$ /opt/mapr/spark/spark-2.1.0/bin/spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.1.0-mapr-1703.jar
Spark Command: /opt/jdk1.8.0_121/bin/java -cp /opt/mapr/spark/spark-2.1.0/conf/:/opt/mapr/spark/spark-2.1.0/jars/*:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/:/opt/mapr/conf/:/opt/mapr/lib/maprbuildversion-5.2.0-mapr.jar:/opt/mapr/lib/maprfs-5.2.0-mapr.jar:/opt/mapr/lib/maprfs-5.2.0-mapr-tests.jar:/opt/mapr/lib/maprfs-diagnostic-tools-5.2.0-mapr.jar:/opt/mapr/lib/maprdb-5.2.0-mapr.jar:/opt/mapr/lib/maprdb-5.2.0-mapr-tests.jar:/opt/mapr/lib/maprdb-mapreduce-5.2.0-mapr.jar:/opt/mapr/lib/maprdb-mapreduce-5.2.0-mapr-tests.jar:/opt/mapr/lib/maprdb-shell-5.2.0-mapr.jar:/opt/mapr/lib/mapr-hbase-5.2.0-mapr.jar:/opt/mapr/lib/mapr-hbase-5.2.0-mapr-tests.jar:/opt/mapr/lib/mapr-streams-5.2.0-mapr.jar:/opt/mapr/lib/mapr-streams-5.2.0-mapr-tests.jar:/opt/mapr/lib/mapr-tools-5.2.0-mapr.jar:/opt/mapr/lib/mapr-tools-5.2.0-mapr-tests.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/central-logging-5.2.0-mapr.jar:/opt/mapr/lib/antlr4-runtime-4.5.jar:/opt/mapr/lib/guava-14.0.1.jar:/opt/mapr/lib/jackson-annotations-2.7.2.jar:/opt/mapr/lib/jackson-core-2.7.2.jar:/opt/mapr/lib/jackson-databind-2.7.2.jar:/opt/mapr/lib/jline-2.11.jar:/opt/mapr/lib/ojai-1.1.jar:/opt/mapr/lib/ojai-mapreduce-1.1.jar:/opt/mapr/lib/protobuf-java-2.5.0.jar:/opt/mapr/lib/spring-asm-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-beans-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-context-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-core-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-expression-3.0.3.RELEASE.jar:/opt/mapr/lib/spring-shell-1.2.0.M1-mapr-1607.jar:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/hdfs/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/lib/*:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/*:/opt/mapr/hadoop/hadoop-2.7.0/contrib/capacity-scheduler/*.jar:/opt/mapr/lib/kvstore*.jar:/opt/mapr/lib/libprotodefs*.jar:/opt/mapr/lib/baseutils*.jar:/opt/mapr/lib/maprutil*.jar:/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/flexjson-2.1.jar -Xmx1g org.apache.spark.deploy.SparkSubmit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi examples/jars/spark-examples_2.11-2.1.0-mapr-1703.jar
========================================
Pi is roughly 3.1391756958784796                       

 

I am using MapR cluster version 5.2.0.39122.GA and only spark and drill packages are from MEP 3.0.0. Also, on client side I have installed spark-2.1.0 and mapr_client, check below:

[mapr@ip-xx-xxx-xx-xxx spark-2.1.0]$ rpm -qa | grep mapr
mapr-client-5.2.0.39122.GA-1.x86_64
mapr-spark-2.1.0.201703271134-1.noarch

 

Please help with the issue.

Outcomes