AnsweredAssumed Answered

Mapr YARN fair scheduler config has no apparent effect

Question asked by reedv on Feb 9, 2018
Latest reply on Feb 15, 2018 by awisawe

Running a test of the fair scheduler in YARN with

[myuser@mapr001 user]$ hadoop jar /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0-mapr-1710.jar pi 4 100

I see that the scheduler page in the YARN web UI shows that a new queue is created called "root.myuser", which (I think) is the default behavior for the scheduler. However, it should not be using the default scheduler rules from the initial mapr install. My /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/fair-scheduler.xml file looks like

[mapr@mapr001 hadoop]$ cat fair-scheduler.xml
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Note: all queues will be children of the root queue even if root queue not listed -->
<allocations>
        <queue name="mapr">
                <aclSubmitApps>mapr mapr</aclSubmitApps>
                <aclAdministerApps>mapr mapr</aclAdministerApps>
                <weight>30.0</weight>
        </queue>
        <queue name="sdcuser">
                <aclSubmitApps>sdcuser,mapr sdcuser,mapr</aclSubmitApps>
                <aclAdministerApps>mapr mapr</aclAdministerApps>
                <weight>30.0</weight>
        </queue>
        <queue name="public">
                <aclSubmitApps>*</aclSubmitApps>
                <aclAdministerApps>mapr mapr</aclAdministerApps>
        </queue>


        <queuePlacementPolicy>
                <rule name="specified" create="false"/>
                <rule name="primaryGroup" create="false"/>
                <rule name="default" queue="public"/>
        </queuePlacementPolicy>
</allocations>
[mapr@mapr001 hadoop]$ echo $MAPR_HOME
/opt/mapr

So, from this, I would think that the job submitted by "myuser" should be run in a queue called "public" and certainly not a queue named after this user. From the mapr docs (Configuring the Fair Scheduler), the fair-scheduler.xml file should have taken effect around 10 seconds after it was changed, which does not seem to be the case here. Restarting the cluster also does not seem to fix the problem. Does anyone know what is going on here? Thanks. 

Outcomes