AnsweredAssumed Answered

Spark 2.1 with MaprDB version mismatch

Question asked by dp on Sep 21, 2017
Latest reply on Sep 29, 2017 by cathy

I am trying to run spark and maprDB and getting into few issues:

1. Mismatch found for java and native libraries java build version  5.2.1.42646.GA, native build version BUILD_VERSION=5.2.1.42385RELEASE_APP=GA 

 

with following code:

SparkConf sparkConf = new SparkConf();
JavaSparkContext sc = new JavaSparkContext(sparkConf);
Configuration hbaseConfig = HBaseConfiguration.create();
hbaseConfig.set("mapr.hbase.default.db", "maprdb");
hbaseConfig.set("fs.mapr.bailout.on.library.mismatch", "false");
hbaseConfig.set("fs.mapr.working.dir", "maprfs:///user/mapr");
hbaseConfig.set(TableInputFormat.INPUT_TABLE, "/user/mapr/" + sourceTable);
JavaPairRDD<ImmutableBytesWritable, Result> hBaseRDD = sc.newAPIHadoopRDD(hbaseConfig, TableInputFormat.class, ImmutableBytesWritable.class, Result.class);

2. Another issue is that I cannot follow mapr example creating JavaHbaseContext as it is not found in hbase jar 1.1.8-mapr-1703, hbase/JavaHBaseBulkGetExample.java at 1.1.8-mapr-1703 · mapr/hbase · GitHub , I do find this class only in cloudera hbase-spark

JavaHBaseContext hbaseContext = new JavaHBaseContext(jsc, conf);

 

versions I have:

mapr cluster version: 5.2.1.42646.GA

mapr client on development machine  version: 5.2.1.42646.GA

poms:

<properties>
<hbase.version>1.1.8-mapr-1703</hbase.version>
<spark.version>2.1.0-mapr-1703</spark.version>
</properties>

<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>${hbase.version}</version>
</dependency>

<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>${hbase.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
</dependency>

Thanks

Outcomes