AnsweredAssumed Answered

Reducers fail with OutOfMemoryError while copying Map outputs

Question asked by lewis on Jan 5, 2014
Latest reply on Jan 7, 2014 by Ted Dunning
Hi ,
  I am using M7,Reducers fail while copying Map outputs with following exception:

  View Diagnostics:

    Error: java.lang.OutOfMemoryError:
    Java heap space at
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1774)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutputFromFile(ReduceTask.java:1487)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1361)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1278)

Log:
  Task Logs: 'attempt_201310300312_3086_r_000004_3'



stdout logs

stderr logs


syslog logs

    2014-01-06 05:58:19,833 INFO org.apache.hadoop.mapred.Child: JVM: jvm_201310300312_3086_r_-213461558 pid: 24238
    2014-01-06 05:58:20,168 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/distcache/-5171337186490676887_910871727_1712680517/maprfs/tmp/temp-339219909/tmp-563709662 <- /tmp/mapr-hadoop/mapred/local/taskTracker/psgadmin/jobcache/job_201310300312_3086/attempt_201310300312_3086_r_000004_3/work/pigsample_1604911155_1388987122994
    2014-01-06 05:58:20,172 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/psgadmin/jobcache/job_201310300312_3086/jars/job.jar <- /tmp/mapr-hadoop/mapred/local/taskTracker/psgadmin/jobcache/job_201310300312_3086/attempt_201310300312_3086_r_000004_3/work/job.jar
    2014-01-06 05:58:20,173 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/mapr-hadoop/mapred/local/taskTracker/psgadmin/jobcache/job_201310300312_3086/jars/.job.jar.crc <- /tmp/mapr-hadoop/mapred/local/taskTracker/psgadmin/jobcache/job_201310300312_3086/attempt_201310300312_3086_r_000004_3/work/.job.jar.crc
    2014-01-06 05:58:20,198 INFO org.apache.hadoop.mapred.Child: Starting task attempt_201310300312_3086_r_000004_3
    2014-01-06 05:58:20,199 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=SHUFFLE, sessionId=
    2014-01-06 05:58:20,628 INFO org.apache.hadoop.mapreduce.util.ProcessTree: setsid exited with exit code 0
    2014-01-06 05:58:20,675 WARN org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree: /proc/<pid>/status does not have information about swap space used(VmSwap). Can not track swap usage of a task.
    2014-01-06 05:58:20,675 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@6f9ec4ed
    2014-01-06 05:58:20,795 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
    2014-01-06 05:58:20,795 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
    2014-01-06 05:58:30,300 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
    2014-01-06 05:58:30,308 INFO org.apache.hadoop.mapred.Merger: Merging 65 sorted segments
    2014-01-06 05:58:30,308 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 4 segments left of total size: 335290939 bytes
    2014-01-06 05:58:30,406 INFO org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call- Usage threshold init = 85983232(83968K) used = 642506992(627448K) committed = 669253632(653568K) max = 754974720(737280K)
    2014-01-06 05:58:32,480 INFO org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call - Collection threshold init = 85983232(83968K) used = 645135144(630014K) committed = 713555968(696832K) max = 754974720(737280K)
    

Does anybody know this issue? It happens every time. Here are some values for following properties:

    SET job.name 'log processing for beacon on DEV on 1.7';
    SET default_parallel 8
    set job.priority HIGH;
    set io.sort.mb 200
    set io.sort.factor 100;
    set mapred.compress.map.output true;
    set mapred.job.shuffle.input.buffer.percent 0.7
    set mapred.job.shuffle.merge.percent 0.5;
    

the other values are MapR default settings, please help me to check it, thanks a lot!

Outcomes