AnsweredAssumed Answered

Lots of:  java.io.IOException: Can not get the relative path: base, errors

Question asked by zackurey on Mar 19, 2012
Latest reply on Mar 19, 2012 by zackurey
Recently we've been getting a lot of errors on a 5 node M5 cluster.  On 3 of the cluster nodes map tasks consistently fail with  java.io.IOException: Can not get the relative path: base, when attempting to store intermediate data between map and reduce tasks.

Disabling those task tracker on those 3 nodes seems to get things running more smoothly, but we still experience random failures storing job output, even from the 2 more functional nodes:

Here is a more full stack trace of a failure for storing job output:

`java.io.IOException: Can not get the relative path: base =
maprfs://10.57.188.39:7222/tmp/temp-332809891/tmp379837418/_temporary/_atte
mpt_201203161742_0203_r_000000_1 child =
maprfs://10.57.188.40:7222/tmp/temp-332809891/tmp379837418/_temporary/_atte
mpt_201203161742_0203_r_000000_1/part-r-00000
    at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getFinalPath(Fil
eOutputCommitter.java:249)
    at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.moveTaskOutputs(
FileOutputCommitter.java:195)
    at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.moveTaskOutputs(
FileOutputCommitter.java:214)
    at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileO
utputCommitter.java:167)
    at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommi
tter.commitTask(PigOutputCommitter.java:252)
    at org.apache.hadoop.mapred.Task.commit(Task.java:1028)
    at org.apache.hadoop.mapred.Task.done(Task.java:892)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:437)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
ava:1109)
    at org.apache.hadoop.mapred.Child.main(Child.java:264)`

I've tried restarting the entire cluster and the errors haven't improved.  Any other steps that would be recommended to try to clear this up?

Outcomes