MapReduce Tasks fails due to failed to create directory /tmp/mapr-hadoop/mapred/local/taskTracker

Document created by wade on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Jitendra Yadav, last modified by Hao Zhu on May 8, 2015

 

Original Publication Date: May 1, 2015

 

Environment
MapR 3.x

Symptom

MapReduce job fails with the following error:

Failed to create directory /tmp/mapr-hadoop/mapred/local/taskTracker/user - No such file or directory failed to initialize user directory

However following directory does exist: /tmp/mapr-hadoop/mapred/local/ttprivate/taskTracker/user.

Root Cause

TT Private is used by Task Tracker for its own usage(Permission 0700).Job's files are downloaded in private cache firstly and then job is initialized in public cache (/tmp/mapr-hadoop/mapred/local/taskTracker/).
The distributed log splitting feature is known to behave in a faulty manner at times.
In this case either the
/tmp/mapr-hadoop/mapred/local/taskTracker/ does not have the correct permission or it actually dose not exist.
One of the major culprit of this issue is linux tmpwatch package which clears the /tmp directory and removes old unused directory files.

Solution

TT private is created automatically when Task Tracker starts therefore it is recommended to restart the Task Tracker on affected nodes so directories can be created with proper permissions.If tmpwatch is causing the issue, please disbale tmpwatch or configure it to ignore /tmp/mapr-hadoop path while deleting. For example:

/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \ 
-x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
-X '/tmp/hadoop-*' -X '/tmp/hsperfdata_*' 10d /tmp
1 person found this helpful

Attachments

    Outcomes