AnsweredAssumed Answered

How to call fair scheduler and label scheduler together in Python scripts and Talend jobs ?

Question asked by sriharshads on Jan 24, 2017
Latest reply on Feb 13, 2017 by maprcommunity



We want to use Label Scheduler and Fair scheduler together so that we can direct specific jobs to specific nodes (using label scheduler) And also control the resources consumed and number of jobs that can concurrently in a particular job queue (using fair scheduler) To achieve this, we have tagged different Labels to different queues (see "fair-scheduler.xml" attachment in "Fair Scheduler" folder for label "Landing" tagged to "LandingQueue" and so on...)


Now that we want to use Python scripts and Talend ETL tool as our primary software to move data into MapR and within MapR, we are looking for ways to tag fair scheduler queues to Python scripts and Talend ETL jobs. (We want to avoid extra step of converting python scripts into jar file to use hadoop jar command.)  Is this possible ? Or we have to use user based queue labels for Python scripts and Talend ETL jobs? In case of user based queue labels, how do we use customized fair scheduler options ?


Attached intended fair scheduler and label scheduler configurations attached.



Sri Harsha D S