AnsweredAssumed Answered

Should there always be something in syslog?

Question asked by communityadmin on Jul 30, 2013
Latest reply on Aug 1, 2013 by aaron
For a standard and default install.
For the simplest word count M/R job.
Where I have not changed any log4j settings...
Should I always see something in the nodes syslog? I only seem to see it when there are various failures. (I can't determine a pattern yet)

For example it seems hadoop is always logging to INFO when the task starts.

2013-07-30 11:30:27,668 INFO org.apache.hadoop.mapred.Child: JVM: jvm_201307250849_7224_m_-588177888 pid: 3879
2013-07-30 11:30:28,077 INFO org.apache.hadoop.mapred.Child: Starting task attempt_201307250849_7224_m_000033_1
..etc...

Shouldn't I always see the above lines?

Here are the log4j settings extracted from a running reduce task.

//////////
log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender
log4j.appender.TLA.append=true
log4j.appender.TLA.bufferSize=8192
log4j.appender.TLA.bufferedIO=false
log4j.appender.TLA.file=/opt/mapr/hadoop/hadoop-0.20.2/logs/userlogs/job_201307250849_7456/attempt_201307250849_7456_r_000112_0/syslog
log4j.appender.TLA.immediateFlush=true
log4j.appender.TLA.isCleanup=false
log4j.appender.TLA.taskId=attempt_201307250849_7456_r_000112_0
log4j.appender.TLA.totalLogFileSize=204800000
log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
log4j.appender.TLA.layout.contentType=text/plain
log4j.appender.TLA.layout.conversionPattern=%d{ISO8601} %p %c: %m%n
log4j.appender.EventCounter=org.apache.hadoop.metrics.jvm.EventCounter
log4j.rootLogger=INFO, TLA, EventCounter
log4j.logger.org.apache.hadoop.mapred.MapTask=WARN
log4j.logger.org.apache.hadoop.mapred.JobTracker=INFO
log4j.logger.org.apache.hadoop.mapred.TaskTracker=INFO
log4j.logger.org.apache.hadoop.mapred.ReduceTask=WARN
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=WARN
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.follow=false
log4j.appender.console.immediateFlush=true
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.contentType=text/plain
log4j.appender.console.layout.conversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
log4j.logger.SecurityLogger=INFO, console
log4j.logger.org.apache.hadoop.mapred.JobClient=INFO
log4j.logger.org.apache.hadoop.fs.FSNamesystem.audit=WARN
log4j.logger.org.apache.hadoop.security.JniBasedUnixGroupsMapping=WARN
log4j.logger.org.apache.hadoop.mapred.JobInProgress$JobSummary=INFO, TLA
log4j.additivity.org.apache.hadoop.mapred.JobInProgress$JobSummary=false
log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR
log4j.logger.org.apache.zookeeper=ERROR

Outcomes