System memory usage is significantly increased after upgrading from MapR v2.x/v3.0.x to MapR v3.1.x.

Document created by jbubier Employee on Feb 13, 2016
Version 1Show Document
  • View in full screen mode

Author: Jonathan Bubier

 

Original Publication Date: August 13, 2014

 

When MapR's warden service starts on a node it calculates the service memory allocation based on the configured roles and the configuration in /opt/mapr/conf/warden.conf.  For each role installed the memory allocation is determined based on the values for the following configuration parameters:

 

service.command.<service>.heapsize.percent

service.command.<service>.heapsize.max

service.command.<service>.heapsize.min

 

 

Each service is guaranteed to be allocated at least service.command.<service>.heapsize.min and capped by the lower of service.command.<service>.heapsize.percent and service.command.<service>.heapsize.max.

 

In v3.0.2 and earlier the memory allocation is calculated for all configured services and the remaining system memory is allocated to map-reduce and divided for map and reduce slots. By default the memory allocation for MFS is configured as 20% of the system memory if M7 is not used and 35% if M7 is used.  The memory allocated for map-reduce is fixed so if the number of map and/or reduce slots is reduced the new slots get more memory rather than providing memory back to other services. 

 

In v3.1 warden first allocates memory for all services except map-reduce and MFS.  If the node is configured as a TaskTracker the optimal number of map and reduce slots is calculated based on multiple factors including the number of MFS disks, the CPUs and the overall system memory. Once the optimal number of slots is calculated the total memory required for map-reduce based on the memory per slot is calculated.  All remaining memory is then allocated to MFS up to the lower of service.command.mfs.heapsize.max, unset by default, and service.command.mfs.heapsize.maxpercent, set by default to 85%.  If the node is not configured as a TaskTracker no slot calculation is needed and MFS is allocated the remaining system memory up to the lower of service.command.mfs.heapsize.max and service.command.mfs.heapsize.maxpercent.

 

If after upgrading to v3.1.x the system memory usage is noticeably higher than on v3.0.x or earlier the first item to verify is the current memory allocation for MFS.  As described above in some cases the memory allocation for MFS will be increased significantly per the memory allocation algorithm in v3.1.  To identify the current memory allocation for MFS review the command line arguments for the MFS process. 

 

Ex:

 

$ ps -ef | grep mfs

mapr     22819     1  0 Jun18 ?        01:12:53 /opt/mapr/server/mfs -b -p 5660 -n inode:10:log:6:meta:10:dir:15:small:15 -m 2756 -g -O /opt/mapr/conf/mapr-clusters.conf -

 

Note: The '-m' argument above which indicates MFS is allocated 2756MB.  Compare this value with the total system memory to determine the percentage used by MFS and compare against the 20% or 35% system memory usage in v3.0.x. 

 

If there remains memory unaccounted for after the upgrade review the output of standard system diagnostic tools such as 'free','vmstat' and 'ps' and the contents of /proc/meminfo.  These will be useful in determining the current system memory usage, the distribution of the system memory and the memory usage for all user processes.  If further assistance is needed to analyze the memory usage generate a support-dump from the host and contact MapR Support.

Attachments

    Outcomes