AnsweredAssumed Answered

Why does nested UserQueue (e.g. GPUqueue.dafox) not inherit label from parent queue (GPUqueue)? Queue label policy on Fair Scheduler NOT FUNCTIONAL

Question asked by dafox on Mar 15, 2016
Latest reply on May 12, 2016 by wade

Update on May 3, 2016: My question is why nested UserQueue (e.g. GPUqueue.dafox) does not inherit label from parent queue (GPUqueue)?

 

Original Question:

Hello,

 

We have heteregenous hadoop cluster and need to run some mapreduce jobs on specific nodes (with appropriate HW). We are using currently YARN with Fair Scheduler to provide each user with fair share of resources. Same way there should be one queue for run jobs on dedicated/labeled nodes only.

 

In MapR distribution this works when we have only one queue:

 

<allocations>
    <queue name="GPUqueue">
        <maxResources>1100000 mb,100 vcores, 100 disks</maxResources>
        <label>nodesWithGPU</label>
    </queue>
    <queuePlacementPolicy>
        <rule name="specified" create="false"/>
        <rule name="user" create="true"/>
    </queuePlacementPolicy>
</allocations>

 

But when we configure scheduler to create nested queues for each user, child-nested queues (e.g. GPUqueue.smith) does not have Queue Label in hadoop web UI and submitted jobs run on all nodes. When we run the command to refresh labels (rmadmin -refreshLabels and mapred job -refreshLabels) the label appears and jobs run on labeled nodes only. So it looks like nested child queue not inherits queue lable from parent on creation. See configuration:

 

<allocations>
    <queue name="GPUqueue" type="parent">
        <maxResources>111000 mb,100 vcores, 100 disks</maxResources>
        <label>nodesWithGPU</label>
    </queue>
    <queuePlacementPolicy>
        <rule name="nestedUserQueue" create="true">
            <rule name="specified" create="false"/>
        </rule>
        <rule name="user" create="true"/>
    </queuePlacementPolicy>
</allocations>

 

Are we doing something wrong?

Is this configuration supported by MapR distribution?Do you have any experiance with similar configuration?

 

EDIT: added the line with label "<label>nodesWithGPU</label>" to second code example. Actually I have this line in my production configuration so problem still there.

Outcomes