AnsweredAssumed Answered

Reducing MFS memory utilization on EMR master node

Question asked by mdelaurentis on Nov 17, 2014
Latest reply on Nov 24, 2014 by mdelaurentis
We're running M3 on Amazon Elastic Map Reduce clusters, and I'm trying to figure out how to get the MFS process to use less memory on the master node. We're using an r3.4xlarge instance for the master node (with 122 GB of memory), and 12 m3.2xlarge instances for the core nodes (each with 30 GB of memory). I see the following settings in `/opt/mapr/conf/warden.conf`:

<pre>
service.command.mfs.heapsize.percent=20
service.command.mfs.heapsize.maxpercent=85
service.command.mfs.heapsize.min=512
</pre>

I would like to significantly reduce the amount of memory that MFS uses on the master node, in order to make more memory available for some other processes we run on that node. I would like to leave the settings as-is on the core nodes. Is there a recommended way to do that on an EMR cluster? I could add a bootstrap script that edits `/opt/mapr/conf/warden.conf` to change those settings, but I would need to make sure that it only changes it for the master node. Is there an option I can set when I start the EMR cluster that will change this setting?

Outcomes