AnsweredAssumed Answered

Setting mapred.task.timeout?

Question asked by chriscurtin on Oct 23, 2012
Latest reply on Oct 23, 2012 by chriscurtin
Hi,

We've tried setting it in /opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml on all nodes and bouncing both the Jobtracker and all the TaskTrackers, but it always defaults back to 600 seconds.

We want to increase it since some jobs are taking about 12 - 15 minutes to complete the 'commit' part of a step and there is no counter updates during the 'commit'.

The only thing I can think of causing the 'commit' to run really long is moving the files created in the _temporary to the parent directory. This is using Cascading to do a 'binning' process so there are 10's of thousands of files being moved around.

This is on 1.2.3.12961

Last thing in the log before the task is killed:

    2012-10-23 12:44:18,752 INFO cascading.tap.hadoop.Hadoop18TapUtil: committing task: 'attempt_201210231233_0001_r_000000_0' - maprfs://10.120.41.15:7222/offlined/working/4/deliverability/_temporary/_attempt_201210231233_0001_r_000000_0

Thanks,

Chris

Outcomes