AnsweredAssumed Answered

HBase dependencies issue when running Spark with YARN client

Question asked by Velumani on May 27, 2016
Latest reply on Jun 2, 2016 by cmcdonald


Hi,

     I am facing HBase dependency issue when running a spark job using YARN client.  The same job running successfully if I run in local mode.

The problem occurs with Spark-1.6.1 version. I don't see this issue issue in 1.5.2 version

I tried the solution suggested in Spark-YARN: how to get rid of yarn-client NoClassDefFoundError  but no luck !

 

Below is the command which I ran

     /opt/mapr/spark/spark-1.6.1/bin/spark-submit --driver-class-path `hbase classpath` --class org.sample.test.App  --master yarn-client target/sparkstreamhbaseapp-1.0.jar

 

Exception Trace

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 5, AWS-DEV-MAPR-02): java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at java.lang.Class.forName0(Native Method)

        at java.lang.Class.forName(Class.java:348)

        at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)

        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)

        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)

        at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1484)

        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1334)

        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)

        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)

        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)

        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)

        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)

        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)

        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)

 

Any Suggestions are highly appreciable. 

 

Outcomes