AnsweredAssumed Answered

Hive Query Plan Distributed Cache Issue

Question asked by mandoskippy on Jan 26, 2013
Latest reply on Jan 27, 2013 by gera
I ran a query that failed, and I grabbed one of the failed tasks logs (below) I am on 2.1.0.16877 GA and gera mentioned in another thread about an issue prior to 2.1.1. That being said, I looked at the release notes of 2.1.1. and didn't see anything mentioned and wondering if this is the same issue: 

`
2013-01-26 08:26:14,707 INFO org.apache.hadoop.mapred.Child: JVM: jvm_201211291052_0589_m_-86179283 pid: 9859
2013-01-26 08:26:14,858 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/distcache/709937278984476727_-194202419_2000706296/maprfs/user/hive/scratch/hive-darkness/hive_2013-01-26_08-26-09_946_5435366776719987/-mr-10004/bb64c838-3dd8-438a-8f6f-b1877d1f0ca1 <- /tmp/mapr-hadoop/mapred/local/taskTracker/darkness/jobcache/job_201211291052_0589/attempt_201211291052_0589_m_000000_0/work/HIVE_PLANbb64c838-3dd8-438a-8f6f-b1877d1f0ca1
2013-01-26 08:26:14,861 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/darkness/jobcache/job_201211291052_0589/jars/.job.jar.crc <- /tmp/mapr-hadoop/mapred/local/taskTracker/darkness/jobcache/job_201211291052_0589/attempt_201211291052_0589_m_000000_0/work/.job.jar.crc
2013-01-26 08:26:14,861 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/darkness/jobcache/job_201211291052_0589/jars/job.jar <- /tmp/mapr-hadoop/mapred/local/taskTracker/darkness/jobcache/job_201211291052_0589/attempt_201211291052_0589_m_000000_0/work/job.jar
2013-01-26 08:26:14,871 INFO org.apache.hadoop.mapred.Child: Starting task attempt_201211291052_0589_m_000000_0
2013-01-26 08:26:14,872 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2013-01-26 08:26:14,882 INFO org.apache.hadoop.mapreduce.util.ProcessTree: setsid exited with exit code 0
2013-01-26 08:26:14,887 WARN org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree: /proc/<pid>/status does not have information about swap space used(VmSwap). Can not track swap usage of a task.
2013-01-26 08:26:14,888 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@ffdadcd
2013-01-26 08:26:14,949 WARN org.apache.hadoop.hive.conf.HiveConf: hive-site.xml not found on CLASSPATH
2013-01-26 08:26:14,959 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-01-26 08:26:14,964 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: java.io.FileNotFoundException: HIVE_PLANbb64c838-3dd8-438a-8f6f-b1877d1f0ca1 (No such file or directory)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:224)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:537)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.io.FileNotFoundException: HIVE_PLANbb64c838-3dd8-438a-8f6f-b1877d1f0ca1 (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at java.io.FileInputStream.<init>(FileInputStream.java:79)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
... 11 more
2013-01-26 08:26:14,966 WARN org.apache.hadoop.mapred.Task: Parent died.  Exiting attempt_201211291052_0589_m_000000_0
`

Outcomes