AnsweredAssumed Answered

Building Latest Spark for Mapr

Question asked by fivetentaylor on Jul 15, 2014
Latest reply on Jul 18, 2014 by mandoskippy

We've been running the spark-0.9.1 mapr package and it works great.  Recently we started exploring the graphx functionality and discovered some bugs in the 0.9.1 version. 

Our contact at mapr suggested we use the mapr maven target to build the latest spark, at least until mapr releases an updated package.

I'm able to successfully build, but then my jobs fail to see files on maprfs.  Here is my process:

 - fetch spark-1.0.1 tar ball
 - build
    - export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
    - mvn -Pmapr -DskipTests clean package
 - The build succeeds then I copy over the conf/spark-env.hs and conf/slaves from the mapr package and point to my new spark home
 - sbin/
     - The cluster starts up fine, the web ui is on, jobs that don't use maprfs work great
     - jobs that use maprfs can't find the files
     - I tried maprfs:///... and /mapr...

Any ideas on what I might be missing?  I'm using a machine that's not in the cluster as my spark driver, is that okay?