AnsweredAssumed Answered

Hive NoSuchFieldError: doubleTypeInfo

Question asked by bhabaniranjan on Sep 2, 2017
Latest reply on Sep 6, 2017 by bhabaniranjan

Hi Team,

 

Can you please help on this? Currently I am using Mapr Sandbox and trying to load data into Hive table stored as parquet. Its works fine as long as it is not stored in parquet. But when I load to parquet tables it gives me following error. In /opt/mapr/hive/hive-2.1/lib I can see only parquet-hadoop-bundle-1.8.1.jar. So I added parquet-hive-bundle-1.6.0.jar.

However I am still getting the same error. Not sure where else we need to put the jar.

 

hive> add jar /opt/mapr/hive/hive-2.1/lib/parquet-hive-bundle-1.6.0.jar;
Added [/opt/mapr/hive/hive-2.1/lib/parquet-hive-bundle-1.6.0.jar] to class path
Added resources: [/opt/mapr/hive/hive-2.1/lib/parquet-hive-bundle-1.6.0.jar]
hive> insert into stockdata_par select * from stockdata;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = mapr_20170902055408_9009d916-ff43-482e-8548-ec6a27adf20e
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1504343561128_0025, Tracking URL = http://maprdemo:8088/proxy/application_1504343561128_0025/
Kill Command = /opt/mapr/hadoop/hadoop-2.7.0/bin/hadoop job -kill job_1504343561128_0025
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2017-09-02 05:54:27,575 Stage-1 map = 0%, reduce = 0%
2017-09-02 05:55:05,523 Stage-1 map = 100%, reduce = 0%
Ended Job = job_1504343561128_0025 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1504343561128_0025_m_000000 (and more) from job job_1504343561128_0025

Task with the most failures(4):
-----
Task ID:
task_1504343561128_0025_m_000000

-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:454)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:137)
... 22 more
Caused by: java.lang.NoSuchFieldError: doubleTypeInfo
at org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.getObjectInspector(ArrayWritableObjectInspector.java:71)
at org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.<init>(ArrayWritableObjectInspector.java:64)
at org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:118)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:359)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:358)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:546)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:498)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:546)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:498)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:443)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)
... 22 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 MAPRFS Read: 0 MAPRFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

 

Table info:

create table stockdata(sno string, symbol string, open float, high float, low float, close float, volume bigint) row format delimited fields terminated by ',' stored as textfile;

create table stockdata_par(sno string,symbol string,open float,high float,low float,close float,volume bigint) stored as parquet;

Outcomes