AnsweredAssumed Answered

map reduce program container beyond physical memory limits and killed

Question asked by suxingfate on Feb 9, 2015
I was trying to import a 4G size tsv file to HBase using importtsv tool. It runs map reduce program, and during the process, there are several containers are killed because of memory usage limitiioon reached.
Which will slow down the overall progress.

Finnally there are 9 map tasks were killed. And yet the job is finished successfully.


How can I change the configuration for container memory allocation usage, or is there some way to avoid this ?


Thanks.

    hadoop jar /opt/mapr/hbase/hbase-0.94.21/hbase-0.94.21-mapr-1409.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,account:name,address:state,address:street,address:zipcode -Dimporttsv.separator='|' students /user/mapr/data



    15/02/09 21:27:39 INFO mapreduce.Job:  map 30% reduce 0%
    15/02/09 21:27:43 INFO mapreduce.Job:  map 31% reduce 0%
    15/02/09 21:27:44 INFO mapreduce.Job:  map 32% reduce 0%
    15/02/09 21:27:48 INFO mapreduce.Job:  map 33% reduce 0%
    15/02/09 21:27:51 INFO mapreduce.Job:  map 34% reduce 0%
    15/02/09 21:27:54 INFO mapreduce.Job:  map 35% reduce 0%
    15/02/09 21:27:58 INFO mapreduce.Job:  map 36% reduce 0%
    15/02/09 21:28:01 INFO mapreduce.Job:  map 37% reduce 0%
    15/02/09 21:28:04 INFO mapreduce.Job:  map 38% reduce 0%
    15/02/09 21:28:07 INFO mapreduce.Job:  map 39% reduce 0%
    15/02/09 21:28:12 INFO mapreduce.Job:  map 40% reduce 0%
    15/02/09 21:28:18 INFO mapreduce.Job:  map 41% reduce 0%
    15/02/09 21:28:21 INFO mapreduce.Job:  map 42% reduce 0%
    15/02/09 21:28:24 INFO mapreduce.Job:  map 43% reduce 0%
    15/02/09 21:28:25 INFO mapreduce.Job:  map 44% reduce 0%
    15/02/09 21:28:30 INFO mapreduce.Job:  map 45% reduce 0%
    15/02/09 21:28:33 INFO mapreduce.Job:  map 46% reduce 0%
    15/02/09 21:28:34 INFO mapreduce.Job: Task Id : attempt_1423458361522_0011_m_000006_0, Status : FAILED
    Container [pid=29384,containerID=container_1423458361522_0011_01_000008] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.5 GB of 2.1 GB virtual memory used. Killing container.
    Dump of the process-tree for container_1423458361522_0011_01_000008 :
            |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
            |- 29384 28473 29384 29384 (bash) 0 0 11784192 341 /bin/bash -c /usr/java/jdk1.7.0_75/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx900m -Djava.io.tmpdir=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1423458361522_0011/container_1423458361522_0011_01_000008/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000008 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 169.254.200.4 38256 attempt_1423458361522_0011_m_000006_0 8 1>/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000008/stdout 2>/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000008/stderr
            |- 29394 29384 29384 29384 (java) 24915 3481 1571201024 271777 /usr/java/jdk1.7.0_75/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx900m -Djava.io.tmpdir=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1423458361522_0011/container_1423458361522_0011_01_000008/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000008 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 169.254.200.4 38256 attempt_1423458361522_0011_m_000006_0 8
    
    Container killed on request. Exit code is 143
    Container exited with a non-zero exit code 143
    
    15/02/09 21:28:35 INFO mapreduce.Job:  map 41% reduce 0%
    15/02/09 21:28:37 INFO mapreduce.Job:  map 42% reduce 0%
    15/02/09 21:28:42 INFO mapreduce.Job:  map 43% reduce 0%
    15/02/09 21:28:43 INFO mapreduce.Job:  map 44% reduce 0%
    15/02/09 21:28:49 INFO mapreduce.Job:  map 45% reduce 0%
    15/02/09 21:28:55 INFO mapreduce.Job:  map 46% reduce 0%
    15/02/09 21:29:01 INFO mapreduce.Job:  map 47% reduce 0%
    15/02/09 21:29:04 INFO mapreduce.Job:  map 48% reduce 0%
    15/02/09 21:29:08 INFO mapreduce.Job:  map 49% reduce 0%
    15/02/09 21:29:13 INFO mapreduce.Job:  map 50% reduce 0%
    15/02/09 21:29:17 INFO mapreduce.Job:  map 51% reduce 0%
    15/02/09 21:29:19 INFO mapreduce.Job: Task Id : attempt_1423458361522_0011_m_000008_0, Status : FAILED
    Container [pid=29795,containerID=container_1423458361522_0011_01_000010] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.5 GB of 2.1 GB virtual memory used. Killing container.
    Dump of the process-tree for container_1423458361522_0011_01_000010 :
            |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
            |- 29795 28473 29795 29795 (bash) 0 0 11784192 342 /bin/bash -c /usr/java/jdk1.7.0_75/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx900m -Djava.io.tmpdir=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1423458361522_0011/container_1423458361522_0011_01_000010/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000010 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 169.254.200.4 38256 attempt_1423458361522_0011_m_000008_0 10 1>/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000010/stdout 2>/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000010/stderr
            |- 29805 29795 29795 29795 (java) 20776 1842 1564930048 270458 /usr/java/jdk1.7.0_75/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx900m -Djava.io.tmpdir=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1423458361522_0011/container_1423458361522_0011_01_000010/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000010 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 169.254.200.4 38256 attempt_1423458361522_0011_m_000008_0 10
    
    Container killed on request. Exit code is 143
    Container exited with a non-zero exit code 143
    
    15/02/09 21:29:20 INFO mapreduce.Job:  map 47% reduce 0%
    15/02/09 21:29:23 INFO mapreduce.Job:  map 48% reduce 0%
    15/02/09 21:29:26 INFO mapreduce.Job:  map 49% reduce 0%
    15/02/09 21:29:29 INFO mapreduce.Job:  map 50% reduce 0%
    15/02/09 21:29:32 INFO mapreduce.Job:  map 51% reduce 0%
    15/02/09 21:29:35 INFO mapreduce.Job:  map 52% reduce 0%
    15/02/09 21:29:38 INFO mapreduce.Job:  map 53% reduce 0%
    



    15/02/09 21:33:46 INFO mapreduce.Job:  map 96% reduce 0%
    15/02/09 21:33:49 INFO mapreduce.Job:  map 97% reduce 0%
    15/02/09 21:33:54 INFO mapreduce.Job: Task Id : attempt_1423458361522_0011_m_000015_2, Status : FAILED
    Container [pid=10184,containerID=container_1423458361522_0011_01_000030] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.5 GB of 2.1 GB virtual memory used. Killing container.
    Dump of the process-tree for container_1423458361522_0011_01_000030 :
            |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
            |- 10184 23653 10184 10184 (bash) 0 0 11583488 333 /bin/bash -c /usr/java/jdk1.7.0_75/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx900m -Djava.io.tmpdir=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1423458361522_0011/container_1423458361522_0011_01_000030/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000030 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 169.254.200.4 38256 attempt_1423458361522_0011_m_000015_2 30 1>/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000030/stdout 2>/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000030/stderr
            |- 10194 10184 10184 10184 (java) 7841 582 1570410496 263714 /usr/java/jdk1.7.0_75/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx900m -Djava.io.tmpdir=/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1423458361522_0011/container_1423458361522_0011_01_000030/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.4.1/logs/userlogs/application_1423458361522_0011/container_1423458361522_0011_01_000030 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 169.254.200.4 38256 attempt_1423458361522_0011_m_000015_2 30
    
    Container killed on request. Exit code is 143
    Container exited with a non-zero exit code 143
    
    15/02/09 21:33:55 INFO mapreduce.Job:  map 94% reduce 0%
    15/02/09 21:34:05 INFO mapreduce.Job:  map 95% reduce 0%
    15/02/09 21:34:12 INFO mapreduce.Job:  map 96% reduce 0%
    15/02/09 21:34:21 INFO mapreduce.Job:  map 97% reduce 0%
    15/02/09 21:34:30 INFO mapreduce.Job:  map 98% reduce 0%
    15/02/09 21:34:39 INFO mapreduce.Job:  map 99% reduce 0%
    15/02/09 21:34:48 INFO mapreduce.Job:  map 100% reduce 0%
    15/02/09 21:34:52 INFO mapreduce.Job: Job job_1423458361522_0011 completed successfully
    15/02/09 21:34:52 INFO mapreduce.Job: Counters: 34
            File System Counters
                    FILE: Number of bytes read=0
                    FILE: Number of bytes written=1713454
                    FILE: Number of read operations=0
                    FILE: Number of large read operations=0
                    FILE: Number of write operations=0
                    MAPRFS: Number of bytes read=4377935947
                    MAPRFS: Number of bytes written=0
                    MAPRFS: Number of read operations=534518
                    MAPRFS: Number of large read operations=0
                    MAPRFS: Number of write operations=0
            Job Counters
                    Failed map tasks=9
                    Launched map tasks=26
                    Other local map tasks=9
                    Data-local map tasks=17
                    Total time spent by all maps in occupied slots (ms)=1781111
                    Total time spent by all reduces in occupied slots (ms)=0
                    Total time spent by all map tasks (ms)=1781111
                    Total vcore-seconds taken by all map tasks=1781111
                    Total megabyte-seconds taken by all map tasks=1823857664
                    DISK_MILLIS_MAPS=890561
            Map-Reduce Framework
                    Map input records=100000000
                    Map output records=100000000
                    Input split bytes=1479
                    Spilled Records=0
                    Failed Shuffles=0
                    Merged Map outputs=0
                    GC time elapsed (ms)=47966
                    CPU time spent (ms)=2053700
                    Physical memory (bytes) snapshot=11902414848
                    Virtual memory (bytes) snapshot=26986983424
                    Total committed heap usage (bytes)=13773570048
            ImportTsv
                    Bad Lines=0
            File Input Format Counters
                    Bytes Read=4377934468
            File Output Format Counters
                    Bytes Written=0
    

Outcomes