AnsweredAssumed Answered

Error executing SparkStreaming example on yarn

Question asked by sagar.sonawane on Aug 1, 2017
Latest reply on Aug 8, 2017 by maprcommunity

I have followed all the instructions given at https://github.com/mapr-demos/finserv-application-blueprint.

When I do spark-submit using "deploy-mode" as "cluster" to run job on MapR cluster(5.2.0) then it gives errors given below in attachment.

On console it gives error like below:

Exception in thread "main" org.apache.spark.SparkException: Application application_1501587885552_0007 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

However, if I am running same program with same spark-submit params on MapR Sandbox(5.2.1) it runs successfully.

 

I am running "spark-2.1.0-mapr-1703" on both sandbox and my 3-node dev cluster.

 

Please guide what am I missing here?

Outcomes