AnsweredAssumed Answered

MapR-HBase on Spark: NoClassDefFoundError / ClassNotFoundException on BaseTableMappingRules / MapRTableMappingRules

Question asked by sumitsu on Nov 30, 2015
Latest reply on Dec 15, 2015 by sumitsu
I have a Spark job running in YARN-cluster mode which attempts to create an `HBaseAdmin` object to communicate with MapR's HBase implementation as follows:

    try {
        log.info("test create HBaseAdmin...");
        log.info("HBaseAdmin: " + new HBaseAdmin(HBaseConfiguration.create()));
    } catch (Throwable exc) {
        log.error("FAILURE to create HBaseAdmin: " + exc, exc);
    }

The code above fails with the following `ClassNotFoundException` / `NoClassDefFoundError`, whether it is run in the initial Spark flow (*outside* of operations performed on an RDD) or by an Executor (*within* a function submitted as part of a call on an RDD):

    15/11/30 14:45:21 ERROR SparkMainClass: FAILURE to create HBaseAdmin: java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: Error occurred while instantiating com.mapr.fs.hbase.MapRTableMappingRules.
    ==> org/apache/hadoop/hbase/client/mapr/BaseTableMappingRules.
    java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: Error occurred while instantiating com.mapr.fs.hbase.MapRTableMappingRules.
    ==> org/apache/hadoop/hbase/client/mapr/BaseTableMappingRules.
            at org.apache.hadoop.hbase.client.HBaseAdmin.commonInit(HBaseAdmin.java:432)
            at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:233)
            at test.SparkMainClass.process(SparkMainClass.java:243)
            at test.SparkMainClass.main(SparkMainClass.java:220)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:483)
    Caused by: java.io.IOException: java.lang.RuntimeException: Error occurred while instantiating com.mapr.fs.hbase.MapRTableMappingRules.
    ==> org/apache/hadoop/hbase/client/mapr/BaseTableMappingRules.
            at org.apache.hadoop.hbase.client.mapr.TableMappingRulesFactory.create(TableMappingRulesFactory.java:65)
            at org.apache.hadoop.hbase.client.HBaseAdmin.commonInit(HBaseAdmin.java:424)
            ... 8 more
    Caused by: java.lang.RuntimeException: Error occurred while instantiating com.mapr.fs.hbase.MapRTableMappingRules.
    ==> org/apache/hadoop/hbase/client/mapr/BaseTableMappingRules.
            at org.apache.hadoop.hbase.client.mapr.GenericHFactory.getImplementorInstance(GenericHFactory.java:40)
            at org.apache.hadoop.hbase.client.mapr.TableMappingRulesFactory.create(TableMappingRulesFactory.java:47)
            ... 9 more
    Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/mapr/BaseTableMappingRules
            at java.lang.ClassLoader.defineClass1(Native Method)
            at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
            at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
            at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
            at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
            at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
            at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
            at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            at java.lang.Class.forName0(Native Method)
            at java.lang.Class.forName(Class.java:264)
            at org.apache.hadoop.hbase.client.mapr.GenericHFactory.getImplementorInstance(GenericHFactory.java:30)
            ... 10 more
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.mapr.BaseTableMappingRules
            at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
            at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            ... 26 more

The `ClassNotFoundException` above is thrown in spite of the fact that both of the classes named in the stack trace ( `org.apache.hadoop.hbase.client.mapr.BaseTableMappingRules` and its superclass `com.mapr.fs.hbase.MapRTableMappingRules`) are included in the application uber-jar I am submitting via `spark-submit`.

I have also tried specifying the following JARs as dependencies via the `--jars` option to `spark-submit`, then adding them to the context as follows, but to no avail:

    context.addJar("file:///local/path/on/driver/node/hbase-client-0.98.12-mapr-1506.jar");
    context.addJar("file:///local/path/on/driver/node/mapr-hbase-5.0.9-mapr.jar");
    context.addJar("file:///local/path/on/driver/node/hbase-common-0.98.12-mapr-1506.jar");
    
    context.addJar("hbase-client-0.98.12-mapr-1506.jar");
    context.addJar("mapr-hbase-5.0.9-mapr.jar");
    context.addJar("hbase-common-0.98.12-mapr-1506.jar");

I am able to manually instantiate `BaseTableMappingRules` via `Class.forName`, using either the class ClassLoader or the thread-context ClassLoader, but the equivalent instantiation of `MapRTableMappingRules` fails on both counts.

What do I need to do to allow the Spark executors to use MapR HBase libraries?

Outcomes