AnsweredAssumed Answered

ERROR SparkContext: Error initializing SparkContext - MapR with Spark 1.6.0

Question asked by a.mueller on May 9, 2016
Latest reply on Dec 7, 2016 by vmeghraj

Running my script on MAPR with Spark 1.6.0 I will get the following:

 

16/05/09 07:06:46 ERROR SparkContext: Error initializing SparkContext.

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)

        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)

        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)

        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)

        at py4j.Gateway.invoke(Gateway.java:214)

        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)

        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)

        at py4j.GatewayConnection.run(GatewayConnection.java:209)

        at java.lang.Thread.run(Thread.java:745)

16/05/09 07:06:46 WARN MetricsSystem: Stopping a MetricsSystem that is not running

Traceback (most recent call last):

  File "/home/dcmn/bigdata/bin/import-piwik-logs.py", line 44, in <module>

    sc         = app.hadoop.get_spark_context()

  File "/home/dcmn/bigdata/bin/../app/hadoop.py", line 37, in get_spark_context

    sc = SparkContext(conf=spark_conf)

  File "/opt/spark-1.6.1-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/context.py", line 115, in __init__

  File "/opt/spark-1.6.1-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/context.py", line 172, in _do_init

  File "/opt/spark-1.6.1-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/context.py", line 235, in _initialize_context

  File "/opt/spark-1.6.1-bin-hadoop2.6/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 1064, in __call__

  File "/opt/spark-1.6.1-bin-hadoop2.6/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.

: org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:124)

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:64)

        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)

        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)

        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)

        at py4j.Gateway.invoke(Gateway.java:214)

        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)

        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)

        at py4j.GatewayConnection.run(GatewayConnection.java:209)

        at java.lang.Thread.run(Thread.java:745)

 

 

On a second installation with Spark 1.5.2 My script is running...

I am running Python 3.5.1, which is also working with Spark 1.5.2.

What does it mean and how can I solve it?

Outcomes