AnsweredAssumed Answered

Using HiveContext in spark sql throws an exception

Question asked by parvaz15 on Mar 11, 2017
Latest reply on Mar 17, 2017 by MichaelSegel


I have to use HiveContext instead of SQLContext because of using some window functions that are available only through HiveContext. I have added the following lines to my pom.xml:

 

   

<dependency>    <groupId>org.apache.spark</groupId>    <artifactId>spark-hive_2.10</artifactId>    <version>1.6.0</version> </dependency>

 

and the machine the machine that I run the code has cloudera on it and  the spark version on that is also 1.6.0. The hive version of the machine that I run the code is 1.1.0-cdh5.7.0. However, when I submit my code to spark, I get the following exception:

    

read "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Iface.get_all_functions()Lorg/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse;

and here is the stack trace:

      

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllFunctions(HiveMetaStoreClient.java:2060)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:105)     at com.sun.proxy.$Proxy27.getAllFunctions(Unknown Source)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1998)     at com.sun.proxy.$Proxy27.getAllFunctions(Unknown Source)     at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3179)     at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:210)     at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:197)     at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:307)     at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:268)     at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:243)     at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:512)     at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)     at java.lang.reflect.Constructor.newInstance(Constructor.java:526)     at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:249)     at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:329)     at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:239)     at org.apache.spark.sql.hive.HiveContext$$anon$2.<init>(HiveContext.scala:459)     at org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:459)     at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:458)     at org.apache.spark.sql.hive.HiveContext$$anon$3.<init>(HiveContext.scala:475)     at org.apache.spark.sql.hive.HiveContext.analyzer$lzycompute(HiveContext.scala:475)     at org.apache.spark.sql.hive.HiveContext.analyzer(HiveContext.scala:474)     at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:34)     at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:133)     at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)     at org.apache.spark.sql.SQLContext.baseRelationToDataFrame(SQLContext.scala:442)     at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:223)     at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:146)     at com.cloudera.sparkwordcount.FindServers.main(FindServers.java:74)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:606)     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


Outcomes