AnsweredAssumed Answered

Not able to get spark 1.5.2 talk to hive 1.0 ( M3)

Question asked by premg on Jan 20, 2016
Latest reply on Jan 26, 2016 by prabhujoseph
**$ rpm -qa | grep mapr**
  

   mapr-mapreduce2-2.7.0.32987.GA-1.x86_64 

   mapr-zookeeper-5.0.0.32987.GA-1.x86_64 

   mapr-nodemanager-2.7.0.32987.GA-1.x86_64 

   mapr-sqoop-1.4.5.201411051314-1.noarch 

   mapr-hivemetastore-1.0.201505191740-1.noarch 

   mapr-hadoop-core-2.7.0.32987.GA-1.x86_64 

   mapr-mapreduce1-0.20.2.32987.GA-1.x86_64 

   mapr-zk-internal-5.0.0.32987.GA.v3.4.5-1.x86_64 

   mapr-tasktracker-5.0.0.32987.GA-1.x86_64 

   mapr-resourcemanager-2.7.0.32987.GA-1.x86_64 

   mapr-historyserver-2.7.0.32987.GA-1.x86_64 

   mapr-hbase-0.98.9.201503251553-1.noarch 

   mapr-hive-1.0.201505191740-1.noarch 

   mapr-core-internal-5.0.0.32987.GA-1.x86_64 

   mapr-core-5.0.0.32987.GA-1.x86_64 

   mapr-fileserver-5.0.0.32987.GA-1.x86_64 

   mapr-hbasethrift-0.98.9.201503251553-1.noarch 

   mapr-spark-1.5.2.201512161339-1.noarch 


I can run simple spark examples like  

 MASTER=yarn-client /opt/mapr/spark/spark-1.5.2/bin/run-example org.apache.spark.examples.SparkPi 10 
  


But I'm not able to get spark talk to hive 

I'm following instructions per http://doc.mapr.com/display/MapR/Integrate+Spark  

Please suggest how to make it work  


**Error Msg:**

$ MASTER=yarn-client /opt/mapr/spark/spark-1.5.2/bin/run-example sql.hive.HiveFromSpark
16/01/20 01:16:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/01/20 01:16:01 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
Exception in thread "main" java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: org/apache/hadoop/util/VersionInfo when creating Hive client using classpath: file:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/activation-1.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/paranamer-2.3.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/servlet-api-2.5.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/api-util-1.0.0-M20.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/asm-3.2.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/avro-1.7.4.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/stax-api-1.0-2.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-6.1.26.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-cli-1.2.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xmlenc-0.52.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-codec-1.4.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jetty-util-6.1.26.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-collections-3.2.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.5-mapr-1503.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-compress-1.4.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsch-0.1.42.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-configuration-1.6.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-digester-1.8.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-httpclient-3.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-io-2.4.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-lang-2.6.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-5.0.0-mapr-tests.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-logging-1.1.3.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mapr-hbase-5.0.0-mapr.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-math3-3.1.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/commons-net-3.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mysql-connector-java-5.1.25-bin.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-client-2.7.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-core-5.0.0-mapr.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-framework-2.7.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-jni-5.0.0-mapr.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/curator-recipes-2.7.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/gson-2.2.4.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/guava-13.0.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jettison-1.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-annotations-2.7.0-mapr-1506.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/json-20080701.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1506.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprutil-5.0.0-mapr.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hamcrest-core-1.3.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsp-api-2.1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpclient-4.2.5.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/httpcore-4.2.5.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/libprotodefs-5.0.0-mapr.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/central-logging-5.0.0-mapr.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jsr305-3.0.0.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jackson-xc-1.9.13.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-core-1.9.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-json-1.9.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jersey-server-1.9.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/jets3t-0.9.0.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/junit-4.11.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/log4j-1.2.17.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-5.0.0-mapr.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-core-5.0.0-mapr-tests.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/mockito-all-1.8.5.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/netty-3.6.2.Final.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/xz-1.0.jar, file:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-diagnostic-tools-5.0.0-mapr.jar, file:/opt/mapr/spark/spark-1.5.2/conf/../, file:/opt/mapr/hive/hive-1.0/lib/accumulo-core-1.6.0.jar, file:/opt/mapr/hive/hive-1.0/lib/hive-contrib-1.0.0-mapr.jar
Please make sure that jars for your version of hive and hadoop are included in the paths passed to SQLConfEntry(key = spark.sql.hive.metastore.jars, defaultValue=builtin, doc=
 Location of the jars that should be used to instantiate the HiveMetastoreClient.
 This property can be one of three options: "
 1. "builtin"
   Use Hive 1.2.1, which is bundled with the Spark assembly jar when
   <code>-Phive</code> is enabled. When this option is chosen,
   <code>spark.sql.hive.metastore.version</code> must be either
   <code>1.2.1</code> or not defined.
 2. "maven"
   Use Hive jars of specified version downloaded from Maven repositories.
 3. A classpath in the standard format for both Hive and Hadoop.
    , isPublic = true).
    at org.apache.spark.sql.hive.client.IsolatedClientLoader.liftedTree1$1(IsolatedClientLoader.scala:189)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader.<init>(IsolatedClientLoader.scala:179)
    at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:263)
    at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
    at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:392)
    at org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:251)
    at org.apache.spark.sql.SQLContext$$anonfun$6.apply(SQLContext.scala:250)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:250)
    at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:72)
    at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:46)
    at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)




Outcomes