AnsweredAssumed Answered

hive.aux.jar

Question asked by sriharsha on Oct 24, 2013
Latest reply on Oct 25, 2013 by sriharsha
Hi ,

 i am facing issue with the hive.aux.jar we have this property set in hive-site.xml and even though hive is not able to find the jar . i have all the jars on a shared location where all the jar files are readable in the jobtracke url i see this

<code>
java.lang.ClassNotFoundException: wsapi.twitter.tweet.hive.serde.JSONSerDe
Continuing ...
java.lang.ClassNotFoundException: wsapi.twitter.tweet.hive.serde.JSONSerDe
Continuing ...
java.lang.ClassNotFoundException: wsapi.twitter.tweet.hive.serde.JSONSerDe
Continuing ...
java.lang.ClassNotFoundException: wsapi.twitter.tweet.hive.serde.JSONSerDe
Continuing ...
</code>

syslog logs

<pre>
2013-10-24 17:12:13,218 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-10-24 17:12:13,219 INFO org.apache.hadoop.security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
2013-10-24 17:12:13,275 INFO org.apache.hadoop.mapred.Child: JVM: jvm_201306112107_22054_m_1620930191 pid: 30240
2013-10-24 17:12:13,484 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/distcache/5935090061763907868_-1508532961_1817129957/maprfs/user/user/tmp/hive/hive_2013-10-24_17-11-52_561_7189687461618393935/-mr-10003/efb84931-f994-49b8-9772-6038c4968aaf <- /tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/attempt_201306112107_22054_m_000001_2/work/HIVE_PLANefb84931-f994-49b8-9772-6038c4968aaf
2013-10-24 17:12:13,487 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/jars/.job.jar.crc <- /tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/attempt_201306112107_22054_m_000001_2/work/.job.jar.crc
2013-10-24 17:12:13,487 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/jars/job.jar <- /tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/attempt_201306112107_22054_m_000001_2/work/job.jar
2013-10-24 17:12:13,501 INFO org.apache.hadoop.mapred.Child: Starting task attempt_201306112107_22054_m_000001_2
2013-10-24 17:12:13,501 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2013-10-24 17:12:13,558 INFO org.apache.hadoop.mapreduce.util.ProcessTree: setsid exited with exit code 0
2013-10-24 17:12:13,564 WARN org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree: /proc/<pid>/status does not have information about swap space used(VmSwap). Can not track swap usage of a task.
2013-10-24 17:12:13,565 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@3ad3c6a3
2013-10-24 17:12:13,969 WARN org.apache.hadoop.hive.conf.HiveConf: hive-site.xml not found on CLASSPATH
2013-10-24 17:12:14,215 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-10-24 17:12:14,228 INFO org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file maprfs:/source/twitter/profiles/year=2013/month=10/day=01/profiles-030002
2013-10-24 17:12:14,236 INFO ExecMapper: maximum memory = 2031943680
2013-10-24 17:12:14,236 INFO ExecMapper: conf classpath = [file:/opt/mapr/hadoop/hadoop-0.20.2/conf/, file:/usr/java/jdk1.6.0_24/lib/tools.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/, file:/opt/mapr/hadoop/hadoop-0.20.2/hadoop*core*.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/amazon-s3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/asm-3.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/aspectjrt-1.6.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/aspectjtools-1.6.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/aws-java-sdk-1.3.26.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-cli-1.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-codec-1.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-configuration-1.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-daemon-1.0.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-el-1.0.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.0.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-lang-2.6.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.1.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-math-2.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-net-1.4.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-net-3.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/core-3.1.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/emr-metrics-1.0.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/eval-0.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/gson-1.4.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/guava-13.0.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-capacity-scheduler.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-fairscheduler.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/httpclient-4.1.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/httpcore-4.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jackson-core-asl-1.5.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jackson-mapper-asl-1.5.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jersey-core-1.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jersey-json-1.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jersey-server-1.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jets3t-0.6.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jetty-6.1.26.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jetty-servlet-tester-6.1.26.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jetty-util-6.1.26.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/junit-4.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/kfs-0.2.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/log4j-1.2.15.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/logging-0.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.20.2-2.1.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.20.2-2.1.3-tests.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-diagnostic-tools-0.20.2-2.1.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-jni-0.20.2-2.1.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-jni-0.20.2-2.1.3-tests.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/mysql-connector-java-5.0.8-bin.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/oro-2.0.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/protobuf-java-2.4.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-20081211.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/xmlenc-0.52.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/zookeeper-3.3.6.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-2.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-api-2.1.jar, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/jars/classes, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/jars/job.jar, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/distcache/4770048322580707444_-1547346296_1817130353/maprfs/var/mapr/cluster/mapred/jobTracker/staging/user/.staging/job_201306112107_22054/libjars/hive-builtins-0.9.0.jar, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/distcache/2736992359284884474_1041507340_1817130393/maprfs/var/mapr/cluster/mapred/jobTracker/staging/user/.staging/job_201306112107_22054/libjars/hive/, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/attempt_201306112107_22054_m_000001_2/work/]
2013-10-24 17:12:14,237 INFO ExecMapper: thread classpath = [file:/opt/mapr/hadoop/hadoop-0.20.2/conf/, file:/usr/java/jdk1.6.0_24/lib/tools.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/, file:/opt/mapr/hadoop/hadoop-0.20.2/hadoop*core*.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/amazon-s3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/asm-3.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/aspectjrt-1.6.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/aspectjtools-1.6.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/aws-java-sdk-1.3.26.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-cli-1.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-codec-1.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-configuration-1.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-daemon-1.0.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-el-1.0.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.0.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-lang-2.6.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-1.1.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-math-2.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-net-1.4.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/commons-net-3.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/core-3.1.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/emr-metrics-1.0.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/eval-0.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/gson-1.4.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/guava-13.0.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-capacity-scheduler.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-core.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/hadoop-0.20.2-dev-fairscheduler.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/httpclient-4.1.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/httpcore-4.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jackson-core-asl-1.5.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jackson-mapper-asl-1.5.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jersey-core-1.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jersey-json-1.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jersey-server-1.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jets3t-0.6.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jetty-6.1.26.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jetty-servlet-tester-6.1.26.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jetty-util-6.1.26.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/junit-4.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/kfs-0.2.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/log4j-1.2.15.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/logging-0.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.20.2-2.1.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-0.20.2-2.1.3-tests.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-diagnostic-tools-0.20.2-2.1.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-jni-0.20.2-2.1.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/maprfs-jni-0.20.2-2.1.3-tests.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.2.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.5.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/mysql-connector-java-5.0.8-bin.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/oro-2.0.8.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/protobuf-java-2.4.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-20081211.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/xmlenc-0.52.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/zookeeper-3.3.6.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-2.1.jar, file:/opt/mapr/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-api-2.1.jar, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/jars/classes, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/jars/job.jar, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/distcache/4770048322580707444_-1547346296_1817130353/maprfs/var/mapr/cluster/mapred/jobTracker/staging/user/.staging/job_201306112107_22054/libjars/hive-builtins-0.9.0.jar, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/distcache/2736992359284884474_1041507340_1817130393/maprfs/var/mapr/cluster/mapred/jobTracker/staging/user/.staging/job_201306112107_22054/libjars/hive/, file:/tmp/mapr-hadoop/mapred/local/taskTracker/user/jobcache/job_201306112107_22054/attempt_201306112107_22054_m_000001_2/work/]

</pre>

if i export the hive.aux.jar path it works fine how to solve this. any clue would be thankfull

Outcomes