AnsweredAssumed Answered

job failed with error java.lang.NoClassDefFoundError

Question asked by shiv on Jan 31, 2018
Latest reply on Feb 5, 2018 by maprcommunity

arjun kr Mathieu Dumoulin

Hi Guys,

I am facing another issue with webhcat.

MapR version: 5.2.1

 

Error

2018-02-01 13:58:07,382 INFO [main] org.apache.hive.hcatalog.templeton.tool.TrivialExecService: Starting cmd: [hive-2.1.tar.gz/hive-2.1/bin/hive, --service, cli, --hiveconf, mapreduce.job.credentials.binary=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1517455815041_0003/container_e05_1517455815041_0003_01_000002/container_tokens, --hiveconf, tez.credentials.path=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1517455815041_0003/container_e05_1517455815041_0003_01_000002/container_tokens, --hiveconf, mapreduce.job.tags=job_1517455815041_0003, --hiveconf, hive.metastore.sasl.enabled=false, --hiveconf, hive.metastore.execute.setugi=true, --hiveconf, hive.execution.engine=mr, --hiveconf, hive.conf.hidden.list=javax.jdo.option.ConnectionPassword,hive.server2.keystore.password, -e, select * from hive_test.shiv_test;]
 2018-02-01 13:58:07,382 INFO [main] org.apache.hive.hcatalog.templeton.tool.TrivialExecService: Removing env var: HADOOP_ROOT_LOGGER=INFO,console 2018-02-01 13:58:07,387 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/LoggerContextFactory at org.apache.hive.hcatalog.templeton.tool.TempletonUtils.dumpPropMap(TempletonUtils.java:490) at org.apache.hive.hcatalog.templeton.tool.TrivialExecService.logDebugInfo(TrivialExecService.java:69) at org.apache.hive.hcatalog.templeton.tool.TrivialExecService.run(TrivialExecService.java:64) at org.apache.hive.hcatalog.templeton.tool.LaunchMapper.startJob(LaunchMapper.java:177) at org.apache.hive.hcatalog.templeton.tool.LaunchMapper.run(LaunchMapper.java:410) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.spi.LoggerContextFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 12 more

 

Command used

curl -s -d execute="select * from hive_test.shiv_test;" \
-d statusdir="pokes.output" \
http://webhcathost:50111/templeton/v1/hive?user.name=mapr

 

I see, we have 

/opt/mapr/hive/hive-2.1/lib/log4j-1.2-api-2.4.1.jar

/opt/mapr/hive/hive-2.1/lib/log4j-api-2.4.1.jar

also, It has loaded(in webhcat-console.log):

[Loaded org.apache.logging.log4j.spi.LoggerContextFactory from file:/opt/mapr/hive/hive-2.1/lib/log4j-api-2.4.1.jar] also,

[Loaded org.apache.log4j.XXXX....... from file:/opt/mapr/hbase/hbase-1.1.8/lib/log4j-1.2.17.jar]  

in hive, we have following code:

for f in ${HIVE_LIB}/*.jar; 

do   CLASSPATH=${CLASSPATH}:$f;

done

# Add hbase conf, hbase jars and zookeeper jar to classpath
  

if [ ! -z "${MAPR_HBASE_CONF}" ]; then   

CLASSPATH=${CLASSPATH} fi if [ ! -z "${MAPR_HBASE_JAR}" ] ; then   

CLASSPATH="${CLASSPATH}"; fi if [ ! -z "${MAPR_HBASE_TEST_JAR}" ] ; then   

CLASSPATH=${CLASSPATH}; fi if [ -f "${MAPR_ZK_HOME}/zookeeper-*.jar" ] ; then   CLASSPATH=${CLASSPATH}:${MAPR_ZK_HOME}/zookeeper-*.jar;

fi

if [[ -n $SENTRY_HOME ]] ; then   for f in $SENTRY_HOME/lib/s*.jar; do     

CLASSPATH=${CLASSPATH}:${f}   done fi

It should be using hive lib jars for logging but it's loading log jar from hbase lib too.

does this has to do something with HBase jars?

Please help me to resolve this issue

Outcomes