AnsweredAssumed Answered

MR jobs retired within a minute of completion

Question asked by ledbit on May 21, 2014
Latest reply on May 21, 2014 by ledbit
Is there any particular reason why MapR has chosen to force retiring of MR job (almost) immediately after they succeed (within 30sec)?

The reason why it is important for an MR job to remain in the "completed" rather than "retired" state is that the client that started the job does not get any more task completion events if the job gets retired. The default behavior is different from Apache Hadoop, which is fine, however what's **very concerning** is that the following config option in mapred-site.xml is not honored:

    mapred.jobtracker.completeuserjobs.maximum

Here's what the jobtracker logs show:

    2014-05-21 04:27:38,316 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201405190917_0024 has completed successfully.
    ...
    2014-05-21 04:27:50,362 INFO org.apache.hadoop.mapred.JobTracker: User limit exceeded. Marking job: job_201405190917_0024 for retire.
    2014-05-21 04:27:50,362 INFO org.apache.hadoop.mapred.JobTracker: Retired job with id: 'job_201405190917_0024' of user 'root'


Is there a list of known config options that are not honored by MapR?


Outcomes