AnsweredAssumed Answered

Cleaning up job and log files?

Question asked by chriscurtin on Aug 13, 2012
Latest reply on Sep 11, 2012 by chriscurtin
Hi,
 
We spent several hours last night cleaning up the cluster and restarting everything after filling up the disk where mapred.local.dir and $HADOOP_LOG_DIR reside.
 
Looking through the files that were there it doesn't appear the cluster is cleaning up after itself very well. We looked at configuration files and did some online searches, but we can't figure out where the settings are to clean these directories. < mapred.local.dir >/taskTracker/hadoop/jobcache had over 100,000 directories in it when we ran out of space.
 
Does the cluster cleanup after itself, or do we need cron jobs to clean up? If we have to do it, any pointers to scripts to use or specific directories to clean up and how often?
 
What we found:
 
<mapred.local.dir>/jobTracker
- running jobs, files are removed (moved?) when completed
 
<mapred.local.dir>/taskTracker/hadoop/jobcache
- current tasks
- but old files for tasks long completed are still here
- > 1000 directories which is the # of jobs set in mapred.jobtracker.retiredjobs.cache.size
 
<mapred.local.dir>/toBeDeleted/<date string>/hadoop/jobcache
- old jobs, moved here when the node was restarted?
- when are they deleted?
 
<mapred.local.dir>/ttprivate/tasktracker/hadoop/jobcache
- currently running jobs?
 
HADOOP_LOG_DIR
- job_*.xml are currently running jobs
 
HADOOP_LOG_DIR/history
- what are these? Look like job files?
- when are they removed?
 
HADOOP_LOG_DIR/userlogs
- directory per job, child directories per attempt
- never cleaned up? > 9000 job directories after being purged less than 24 hours ago
 
mapred-site.xml settings:
  <name>mapred.jobtracker.retiredjobs.cache.size</name>
  <value>1000</value>
 
Relevant log file settings:
 
hadoop-env.sh:export HADOOP_LOG_DIR="/nfs/mapr/hadoop/logs"
taskcontroller.cfg:hadoop.log.dir=/nfs/mapr/hadoop/logs
 
Thanks,
 
Chris

Outcomes