AnsweredAssumed Answered

Container is running beyond physical memory limits!

Question asked by Karthee on Apr 28, 2017
Latest reply on May 2, 2017 by cathy

Hi There,

I have a 3 node cluster, each node is having a 256g of RAM.and i have changed the memory config's in 

mapred-site.xml,yarn-site.xml and hadoop-env.sh

please find the config's,

mapred.cluster.map.memory.mb=16g
mapred.cluster.reduce.memory.mb=16g

mapred.cluster.max.map.memory.mb=32g
mapred.cluster.max.reduce.memory.mb=32g

mapred.job.map.memory.mb=3g
mapred.job.reduce.memory.mb=9g

yarn.scheduler.maximum-allocation-mb=32g
yarn.scheduler.minimum-allocation-mb=8g

HADOOP_CLIENT_OPTS="-Xmx12288m

 

but after all these config's still my team is getting the "Container is running beyond physical memory limits"

here's the log,

 

Diagnostic Messages for this Task:

Container [pid=31336,containerID=container_e73_1492864090767_0240_01_000348] is running beyond physical memory limits. Current usage: 8.6 GB of 8 GB physical memory used; 34.1 GB of 16.8 GB virtual memory used. Killing container.

Dump of the process-tree for container_e73_1492864090767_0240_01_000348 :

     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

     |- 31340 31336 31336 31336 (java) 10136 351 36485783552 2250095 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131.x86_64/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx32768m -Djava.io.tmpdir=/tmp/hadoop-mapr/nm-local-dir/usercache/akundu/appcache/application_1492864090767_0240/container_e73_1492864090767_0240_01_000348/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1492864090767_0240/container_e73_1492864090767_0240_01_000348 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.107.80.83 39394 attempt_1492864090767_0240_m_000078_3 80264348827996

     |- 31336 31334 31336 31336 (bash) 0 0 108670976 326 /bin/bash -c /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131.x86_64/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx32768m -Djava.io.tmpdir=/tmp/hadoop-mapr/nm-local-dir/usercache/akundu/appcache/application_1492864090767_0240/container_e73_1492864090767_0240_01_000348/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1492864090767_0240/container_e73_1492864090767_0240_01_000348 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.107.80.83 39394 attempt_1492864090767_0240_m_000078_3 80264348827996 1>/opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1492864090767_0240/container_e73_1492864090767_0240_01_000348/stdout 2>/opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs/application_1492864090767_0240/container_e73_1492864090767_0240_01_000348/stderr 

 

Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

 FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

 

i am confused with, where does the 8g of physical memory gets picked up and virtual memory as well ????

where do i change those default vaules?

 

Really appreciate your guidance in this!##

Outcomes