AnsweredAssumed Answered

The temporary job-output directory file:/XXX/output/_temporary doesn't exist!

Question asked by viadea on Feb 24, 2012
Latest reply on Feb 27, 2012 by viadea
I build a simple cluster with default parameters.
Doing the test with standard alone mapreduce job which should run locally:

hadoop jar hadoop-0.20.2-dev-examples.jar grep -Dmapred.job.tracker=local file:///opt/mapr/hadoop/hadoop-0.20.2/input  file:///opt/mapr/hadoop/hadoop-0.20.2/output 'dfs[a-z.]+'

The input and output directories are on local file.
This job will error out saying, there is no such _temp directories on all nodes.
For example:

    12/02/25 23:09:05 INFO mapred.JobClient: Task Id : attempt_201112191353_0029_r_000000_0, Status : FAILED on node sdw5
    java.io.IOException: The temporary job-output directory file:/opt/mapr/hadoop/hadoop-0.20.2/output/_temporary doesn't exist!

**Question is :
Since mapred.job.tracker=local is specified, why did this job try to output to all nodes?
If I change the output directory to mapR-FS instead of local directory, this job will run successfully.
EG:
hadoop jar hadoop-0.20.2-dev-examples.jar grep -Dmapred.job.tracker=local file:///opt/mapr/hadoop/hadoop-0.20.2/input /output 'dfs[a-z.]+'**


The complete error log is :

    [root@mdw hadoop-0.20.2]# hadoop jar hadoop-0.20.2-dev-examples.jar grep -Dmapred.job.tracker=local file:///opt/mapr/hadoop/hadoop-0.20.2/input  file:///opt/mapr/hadoop/hadoop-0.20.2/output 'dfs[a-z.]+'
    12/02/25 23:08:48 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    12/02/25 23:08:48 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    12/02/25 23:08:48 INFO mapred.FileInputFormat: Total input paths to process : 4
    12/02/25 23:08:49 INFO mapred.JobClient: Running job: job_local_0001
    12/02/25 23:08:49 INFO util.ProcessTree: setsid exited with exit code 0
    12/02/25 23:08:49 WARN util.ProcfsBasedProcessTree: /proc/<pid>/status does not have information about swap space used(VmSwap). Can not track swap usage of a task.
    12/02/25 23:08:49 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@5090d8ea
    12/02/25 23:08:49 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
    12/02/25 23:08:49 INFO mapred.LocalJobRunner: file:/opt/mapr/hadoop/hadoop-0.20.2/input/abcd:0+28
    12/02/25 23:08:49 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
    12/02/25 23:08:49 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@26659db7
    12/02/25 23:08:49 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
    12/02/25 23:08:49 INFO mapred.LocalJobRunner: file:/opt/mapr/hadoop/hadoop-0.20.2/input/dfsa:0+0
    12/02/25 23:08:49 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done.
    12/02/25 23:08:49 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@6a8046f4
    12/02/25 23:08:49 INFO mapred.Task: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
    12/02/25 23:08:49 INFO mapred.LocalJobRunner: file:/opt/mapr/hadoop/hadoop-0.20.2/input/123dfs:0+0
    12/02/25 23:08:49 INFO mapred.Task: Task 'attempt_local_0001_m_000002_0' done.
    12/02/25 23:08:49 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@132789d2
    12/02/25 23:08:49 INFO mapred.Task: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
    12/02/25 23:08:49 INFO mapred.LocalJobRunner: file:/opt/mapr/hadoop/hadoop-0.20.2/input/dfsb:0+0
    12/02/25 23:08:49 INFO mapred.Task: Task 'attempt_local_0001_m_000003_0' done.
    12/02/25 23:08:49 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@7162e295
    12/02/25 23:08:49 INFO mapred.LocalJobRunner:
    12/02/25 23:08:49 INFO mapred.Merger: Merging 4 sorted segments
    12/02/25 23:08:49 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 34 bytes
    12/02/25 23:08:49 INFO mapred.LocalJobRunner:
    12/02/25 23:08:49 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
    12/02/25 23:08:49 INFO mapred.LocalJobRunner:
    12/02/25 23:08:49 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
    12/02/25 23:08:49 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to maprfs://172.28.8.250:7222/grep-temp-434872713
    12/02/25 23:08:49 INFO mapred.LocalJobRunner: reduce > reduce
    12/02/25 23:08:49 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
    
    12/02/25 23:08:50 INFO mapred.JobClient:  map 100% reduce 100%
    12/02/25 23:08:50 INFO mapred.JobClient: Job complete: job_local_0001
    12/02/25 23:08:50 INFO mapred.JobClient: Counters: 18
    12/02/25 23:08:50 INFO mapred.JobClient:   FileSystemCounters
    12/02/25 23:08:50 INFO mapred.JobClient:     FILE_BYTES_READ=943899
    12/02/25 23:08:50 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=1225747
    12/02/25 23:08:50 INFO mapred.JobClient:   Map-Reduce Framework
    12/02/25 23:08:50 INFO mapred.JobClient:     Map input records=4
    12/02/25 23:08:50 INFO mapred.JobClient:     Reduce shuffle bytes=0
    12/02/25 23:08:50 INFO mapred.JobClient:     Spilled Records=4
    12/02/25 23:08:50 INFO mapred.JobClient:     Map output bytes=28
    12/02/25 23:08:50 INFO mapred.JobClient:     CPU_MILLISECONDS=0
    12/02/25 23:08:50 INFO mapred.JobClient:     Map input bytes=28
    12/02/25 23:08:50 INFO mapred.JobClient:     SPLIT_RAW_BYTES=398
    12/02/25 23:08:50 INFO mapred.JobClient:     Combine input records=2
    12/02/25 23:08:50 INFO mapred.JobClient:     Reduce input records=2
    12/02/25 23:08:50 INFO mapred.JobClient:     Reduce input groups=2
    12/02/25 23:08:50 INFO mapred.JobClient:     Combine output records=2
    12/02/25 23:08:50 INFO mapred.JobClient:     PHYSICAL_MEMORY_BYTES=0
    12/02/25 23:08:50 INFO mapred.JobClient:     Reduce output records=2
    12/02/25 23:08:50 INFO mapred.JobClient:     VIRTUAL_MEMORY_BYTES=0
    12/02/25 23:08:50 INFO mapred.JobClient:     Map output records=2
    12/02/25 23:08:50 INFO mapred.JobClient:     GC time elapsed (ms)=56
    12/02/25 23:08:50 INFO fs.JobTrackerWatcher: Current running JobTracker is: mdw/172.28.8.250:9001
    12/02/25 23:08:50 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    12/02/25 23:08:50 INFO mapred.FileInputFormat: Total input paths to process : 1
    12/02/25 23:08:50 INFO mapred.JobClient: Running job: job_201112191353_0029
    12/02/25 23:08:51 INFO mapred.JobClient:  map 0% reduce 0%
    12/02/25 23:08:56 INFO mapred.JobClient:  map 100% reduce 0%
    **12/02/25 23:09:05 INFO mapred.JobClient: Task Id : attempt_201112191353_0029_r_000000_0, Status : FAILED on node sdw5
    java.io.IOException: The temporary job-output directory file:/opt/mapr/hadoop/hadoop-0.20.2/output/_temporary doesn't exist!**
            at org.apache.hadoop.mapred.FileOutputCommitter.getWorkPath(FileOutputCommitter.java:250)
            at org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:240)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:116)
            at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:455)
            at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433)
            at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
            at org.apache.hadoop.mapred.Child.main(Child.java:262)
    
    **12/02/25 23:09:11 INFO mapred.JobClient: Task Id : attempt_201112191353_0029_r_000000_1, Status : FAILED on node sdw3
    java.io.IOException: The temporary job-output directory file:/opt/mapr/hadoop/hadoop-0.20.2/output/_temporary doesn't exist!**
            at org.apache.hadoop.mapred.FileOutputCommitter.getWorkPath(FileOutputCommitter.java:250)
            at org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:240)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:116)
            at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:455)
            at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433)
            at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
            at org.apache.hadoop.mapred.Child.main(Child.java:262)
    
    **12/02/25 23:09:19 INFO mapred.JobClient: Task Id : attempt_201112191353_0029_r_000000_2, Status : FAILED on node sdw1
    java.io.IOException: The temporary job-output directory file:/opt/mapr/hadoop/hadoop-0.20.2/output/_temporary doesn't exist!**
            at org.apache.hadoop.mapred.FileOutputCommitter.getWorkPath(FileOutputCommitter.java:250)
            at org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:240)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:116)
            at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:455)
            at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:433)
            at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1074)
            at org.apache.hadoop.mapred.Child.main(Child.java:262)
    
    12/02/25 23:09:27 INFO mapred.JobClient: Job complete: job_201112191353_0029
    12/02/25 23:09:27 INFO mapred.JobClient: Counters: 21
    12/02/25 23:09:27 INFO mapred.JobClient:   Job Counters
    12/02/25 23:09:27 INFO mapred.JobClient:     Launched reduce tasks=4
    12/02/25 23:09:27 INFO mapred.JobClient:     Aggregate execution time of mappers(ms)=1664
    12/02/25 23:09:27 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
    12/02/25 23:09:27 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
    12/02/25 23:09:27 INFO mapred.JobClient:     Launched map tasks=1
    12/02/25 23:09:27 INFO mapred.JobClient:     Data-local map tasks=1
    12/02/25 23:09:27 INFO mapred.JobClient:     Failed reduce tasks=1
    12/02/25 23:09:27 INFO mapred.JobClient:     Aggregate execution time of reducers(ms)=25803
    12/02/25 23:09:27 INFO mapred.JobClient:   FileSystemCounters
    12/02/25 23:09:27 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=43832
    12/02/25 23:09:27 INFO mapred.JobClient:   Map-Reduce Framework
    12/02/25 23:09:27 INFO mapred.JobClient:     Combine output records=0
    12/02/25 23:09:27 INFO mapred.JobClient:     Map input records=2
    12/02/25 23:09:27 INFO mapred.JobClient:     PHYSICAL_MEMORY_BYTES=247066624
    12/02/25 23:09:27 INFO mapred.JobClient:     Spilled Records=2
    12/02/25 23:09:27 INFO mapred.JobClient:     Map output bytes=28
    12/02/25 23:09:27 INFO mapred.JobClient:     CPU_MILLISECONDS=770
    12/02/25 23:09:27 INFO mapred.JobClient:     VIRTUAL_MEMORY_BYTES=1693446144
    12/02/25 23:09:27 INFO mapred.JobClient:     Map input bytes=44
    12/02/25 23:09:27 INFO mapred.JobClient:     Combine input records=0
    12/02/25 23:09:27 INFO mapred.JobClient:     Map output records=2
    12/02/25 23:09:27 INFO mapred.JobClient:     SPLIT_RAW_BYTES=110
    12/02/25 23:09:27 INFO mapred.JobClient:     GC time elapsed (ms)=6
    12/02/25 23:09:27 INFO mapred.JobClient: Job Failed: NA
    java.io.IOException: Job failed!
            at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1273)
            at org.apache.hadoop.examples.Grep.run(Grep.java:84)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
            at org.apache.hadoop.examples.Grep.main(Grep.java:93)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
            at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
            at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:186)



Outcomes