AnsweredAssumed Answered

Error while creating spark session and loading csv file in Jupyterhub

Question asked by Arunav on Jun 7, 2017
Latest reply on Jun 20, 2017 by Arunav

Hi,

Mathieu Dumoulin helped me with a very similar issue. Thanks again. Could you please suggest on this one?

 

I have a 5 node MapR cluster running on RHEL 6.8 and MapR 5.2. I've Jupyterhub installed on one of the servers. 

I usually launch spark codes with the py2(python2.7) kernel of Jupyterhub by creating a SparkConf and SparkContext. I was trying to run the same kind of jobs by creating a SparkSession but the jobs are only running on local mode. When running on Yarn, there's a Stage failure error. 

Also, the same jobs run just fine when I submit through spark-submit.

I checked the resource manager logs and the containers are getting allocated.

Here's a small piece of code:

The error:

 

Thanks,

Arunava Addy

Outcomes