AnsweredAssumed Answered

attempt job task executing time on one node is  inversely with the number of map slots

Question asked by tony on Jan 9, 2014
Latest reply on Jan 12, 2014 by Ted Dunning
if we set more map slots nums and reduce slots nums on one srever< we found out it will slowly task execute speed。 So we we have to reduce the slots num。 I found out the Input Bytes/Sec and Input Records/Sec  is slow when we set more slots. But strangely that; the server io,cpu and mem is not high. wheat's the reason and how to optimize this?

for example:

     **one node server spec: 192G mem, 24 cpus, 11 disks.**
     map slots (72) = cpus *2  (24* 2+ 24*2*0.5)
     reduce slots(36) = cpu * 1.5

the result is :
**Input Bytes/Sec  = 100KB of attampt task and one attampt task cost 54min**
----------
if we reduce slots num like below:

     map slots (72) = cpus *1
     reduce slots(36) = cpu * 0.75

the results is:
**Input Bytes/Sec  = 860KB of attampt task and one attampt task cost 9min**

**and how to set slots nums like this server spec and what's reason for that?** anyone know it? it is not good use server utilities   if we set very little slots num. thanks in advance.

Outcomes