AnsweredAssumed Answered

Resource Manager (yarn) sees only 50% of the memory

Question asked by kaushal_jha on Aug 2, 2016
Latest reply on Sep 19, 2016 by kaushal_jha

Hi Team,

 

Need some help with Yarn ResourceManager not using all the available ram as per http:/RM-HOST:8088/cluster/

 

it sees about 97% of the CPU(which is good), but it sees only 50% of the RAM(which is bad)

other than hiveserver and sqoop there are no other ecosystem packages installed ,  I have removed JT, TT, drill, spark.

 

Could someone please help me figure out how to allow the RM to use the avaiable RAM (i.e not ignore the remaining 50% of RAM) please

or maybe I am missing something here.  I get it that the filserver, NFS server etc needs resources too, but 50% of RAM for the base system seems a bit too much

 

I have already done the configure.sh -R and restarted warden not seeing any improvements.

 

mapr@bdmapr03:/opt/mapr/conf$ maprcli node list -columns svc

hostname                 service                                                  ip

bdmapr01.quinstreet.net  cldb,hoststats,fileserver,nodemanager                    10.1.86.11

bdmapr02.quinstreet.net  cldb,hoststats,resourcemanager,fileserver,nodemanager    10.1.86.12

bdmapr03.quinstreet.net  hs2,hoststats,hivemeta,fileserver,webserver,nodemanager  10.1.86.13

bdmapr04.quinstreet.net  hoststats,fileserver,nfs,webserver,nodemanager           10.1.86.14

bdmapr05.quinstreet.net  hoststats,fileserver,nfs,nodemanager                     10.1.86.15

bdmapr06.quinstreet.net  hoststats,fileserver,historyserver,nfs,nodemanager       10.1.86.16

 

Here is the memory stats for every node in the cluster via free -g

 

 

bigtable@bdadmin01:~$ ./cluster.sh 'uname -n ; free -g '

bdmapr01.quinstreet.net

             total       used       free     shared    buffers     cached

Mem:            35         15         20          0          2          1

-/+ buffers/cache:         11         23

Swap:           68          0         68

bdmapr02.quinstreet.net

             total       used       free     shared    buffers     cached

Mem:            35         18         17          0          2          2

-/+ buffers/cache:         13         22

Swap:           68          0         68

bdmapr03.quinstreet.net

             total       used       free     shared    buffers     cached

Mem:            35         21         14          0          1          8

-/+ buffers/cache:         11         23

Swap:           68          0         68

bdmapr04.quinstreet.net

             total       used       free     shared    buffers     cached

Mem:            35         14         20          0          1          2

-/+ buffers/cache:         11         23

Swap:           68          0         68

bdmapr05.quinstreet.net

             total       used       free     shared    buffers     cached

Mem:            35         14         20          0          2          2

-/+ buffers/cache:          9         25

Swap:           68          0         68

bdmapr06.quinstreet.net

             total       used       free     shared    buffers     cached

Mem:            35         14         21          0          2          1

-/+ buffers/cache:         10         25

Swap:           68          0         68

Outcomes