AnsweredAssumed Answered

ExpressLane seems not working correctly

Question asked by pablo on Jun 22, 2012
Latest reply on Jul 2, 2012 by amit
Currently we are trying to see how jobs get scheduled when enabling and disabling ExpressLane.

To try and "restrict" the scheduling possibilities of MapR we've setup the following configuration:

    mapred.fairscheduler.smalljob.max.maps=10
    mapred.fairscheduler.smalljob.max.reducers=10
    mapred.fairscheduler.smalljob.max.inputsize=188743680
    mapred.fairscheduler.smalljob.max.reducer.inputsize=419430400
    mapred.cluster.ephemeral.tasks.memory.limit.mb=300
    mapred.tasktracker.map.tasks.maximum=2
    mapreduce.tasktracker.prefetch.maptasks=10.0
    mapred.tasktracker.reduce.tasks.maximum=1

 1. Our cluster has a Map Capacity of 12 and a Reduce Capacity of 6.
 2. Our big job is being split into 126 Map tasks and 1 Reduce task. It processes 7 4GB files.
 3. Our 6 small jobs are being processed in one Map task and one Reduce task. They each process a single 100MB file.

For some reason the scheduler is not considering the map tasks in the small jobs to be "ephemeral tasks". As you can see from the fair scheduler event output below they are all considered to be normal mapr jobs.

We would appreciate your thoughts on why our small jobs are not going into the Express Lane.

2012-06-21 03:15:31,894 BEGIN_DUMP<br />
2012-06-21 03:15:31,894 JOB job_201206210310_0001 Keyword Count mapr NORMAL mapr 126 99 184 131.0 1.0 1 0 0 0.0 1.0<br />
2012-06-21 03:15:31,894 JOB job_201206210310_0002 Keyword Count mapr NORMAL mapr 1 0 0 0.0 1.0 1 0 0 0.0 1.0<br />
2012-06-21 03:15:31,894 JOB job_201206210310_0003 Keyword Count mapr NORMAL mapr 1 0 0 0.0 1.0 1 0 0 0.0 1.0<br />
2012-06-21 03:15:31,894 JOB job_201206210310_0004 Keyword Count mapr NORMAL mapr 1 0 0 0.0 1.0 1 0 0 0.0 1.0<br />
2012-06-21 03:15:31,894 JOB job_201206210310_0005 Keyword Count mapr NORMAL mapr 1 0 0 0.0 1.0 1 0 0 0.0 1.0<br />
2012-06-21 03:15:31,894 JOB job_201206210310_0006 Keyword Count mapr NORMAL mapr 1 0 0 0.0 1.0 1 0 0 0.0 1.0<br />
2012-06-21 03:15:31,894 JOB job_201206210310_0007 Keyword Count mapr NORMAL mapr 1 1 1 1.0 1.0 1 0 0 0.0 1.0<br />
**2012-06-21 03:15:31,894 POOL ExpressLane 1.0 0 0 0 0 0**<br />
**2012-06-21 03:15:31,894 POOL mapr 1.0 7 0 0 0 0**<br />
2012-06-21 03:15:31,894 POOL default 1.0 0 0 0 0 0<br />
2012-06-21 03:15:31,894 END_DUMP<br />
2012-06-21 03:15:31,928 SMALLJOB HEARTBEAT tracker_ip-10-92-242-193.ec2.internal:localhost/127.0.0.1:40194<br />
2012-06-21 03:15:31,928 SMALLJOB RUNNABLE_TASKS 185 100 0 0<br />
**2012-06-21 03:15:31,928 SMALLJOB RUNNABLE_EPHEMERAL_TASKS 0 0 0 0**<br />

Thansk in advance!

Outcomes