AnsweredAssumed Answered

TaskTracker memory management

Question asked by bonchbruevich on Oct 2, 2014
Latest reply on Oct 8, 2014 by nabeel
I'm trying to figure out how heapsizes for map and reduce slots are being calculated by the TaskTracker.

Here is a piece of a log file from one of my cluster's nodes:

    2014-09-23 12:24:42,020 INFO mapred.TaskTracker [main]: CPUS: 24 
    2014-09-23 12:24:42,020 INFO mapred.TaskTracker [main]: mfsCPUS: 2 
    2014-09-23 12:24:42,020 INFO mapred.TaskTracker [main]: Total MEM: 62.882755GB 
    2014-09-23 12:24:42,020 INFO mapred.TaskTracker [main]: Reserved MEM: 41344MB 
    2014-09-23 12:24:42,020 INFO mapred.TaskTracker [main]: Reserved MEM for Ephemeral slots 200  
    2014-09-23 12:24:42,020 INFO mapred.TaskTracker [main]: DISKS: 0 
    2014-09-23 12:24:42,020 INFO mapred.TaskTracker [main]: mfsDISKS: 5 
    2014-09-23 12:24:42,081 INFO mapred.MapRedSlotUtil [main]: Before adjustment, maxMapSlots = 16, maxReduceSlots = 8  
    2014-09-23 12:24:42,081 INFO mapred.MapRedSlotUtil [main]: After CPU adjustment, maxMapSlots = 16, maxReduceSlots = 8 
    2014-09-23 12:24:42,081 INFO mapred.MapRedSlotUtil [main]: After Disk adjustment, maxMapSlots = 10, maxReduceSlots = 3 
    2014-09-23 12:24:42,081 INFO mapred.MapRedSlotUtil [main]: After adjustment, maxMapSlots = 10, maxReduceSlots = 3 
    2014-09-23 12:24:42,081 INFO mapred.MapRedSlotUtil [main]: mapTaskMem = 1024, reduceTaskMem = 3072 
    2014-09-23 12:24:42,081 INFO mapred.TaskTracker [main]: map and reduce slots have been computed based on requested 
    heap sizes for map and reduce slots  
    2014-09-23 12:24:42,081 INFO mapred.TaskTracker [main]: maptask heapsize: 1024 
    2014-09-23 12:24:42,081 INFO mapred.TaskTracker [main]: reducetask heapsize: 3072 


There are a couple of things that don't make sense to me.

According to the log file reserved memory for MapReduce is 41344MB.
When I sum up the total heap memory for all slots, I get: 1024*10 + 3072*3 = 19456MB

19456MB is less then a half of 41344MB. Which means that the other half of the memory is not going to be used to provide heap for map/reduce tasks. Where is the other half of the memory being used?
    

Outcomes