AnsweredAssumed Answered

How to Increase HistoryServer memory (Heap) in MapR distribution?

Question asked by dafox on May 3, 2016
Latest reply on May 12, 2016 by dafox

Hello,

 

We need to increase the Xmx for HistoryServer service but it does not work as expected.

 

We increased memory for HistoryServer as described in hadoop documentatin, just add following line to yarn-env.sh and restart warden service:

export YARN_HISTORYSERVER_HEAPSIZE=20000

 

The problem is change is not taking effect after warden restart. See `ps aux` output:

mapr  19565  0.1  0.1 3079240 996208 ? Sl   Apr29   8:22 /usr/java/latest/bin/java -Dproc_historyserver -Xmx1000m 
 -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/opt/mapr/hadoop/hadoop-2.7.0/logs 
 -Dhadoop.log.file=hadoop.log <other -D...> -Dhadoop.security.logger=INFO,NullAppender 
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer

 

The reason why we need to increase Xmx is OutOfMemoryException in HistoryServer service (/opt/mapr/roles/historyserver) and we cannot see mapreduce-jobs logs in Hadoop UI. Probably we had huge log file for some job.

2016-04-15 14:33:42,281 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Start
ing scan to move intermediate done files
2016-04-15 14:44:15,386 WARN org.apache.hadoop.ipc.Server: IPC Server handler 4 
on 10020, call org.apache.hadoop.mapreduce.v2.api.HSClientProtocolPB.getJobRepor
t from 10.0.0.1:37709 Call#50610 Retry#0
java.lang.OutOfMemoryError: Java heap space

 

So what we are doing wrong? Is there some MapR specific step required to make this work?

 

Thanks.

Outcomes