AnsweredAssumed Answered

Yarn MRV2 performance Tuning Number of mappers and Reducers MRV1 performance better

Question asked by madhureddy915 on Jun 25, 2015
Latest reply on Jul 3, 2015 by madhureddy915
I'm running MR java program with Yarn , even though Number of mappers are 24 , but actual running mappers are 10 , the remaining 14 maps are in pending state. how to make them also to running mode . we are running MAPR 6 node cluster . changed the below properties in mapred-site.xml and Yarn-site.xml . these values are overridden from the default values , still I'm not seeing any perfomance Bench mark .

Note . I'm running same Program with MRv1 performance wise it's better some how . So please suggest me how to utilize the cluster utilization resources well .

command Used : yarn jar /opt/cluster/bin/logmessage-1.0-SNAPSHOT.jar com.message.WordPreprocessDriver -Dmapreduce.input.fileinputformat.split.maxsize=33554432 /data/123.txt

In yarn-site



**yarn.nodemanager.resource.memory-mb
20960
yarn.scheduler.minimum-allocation-mb
2048**


In mapred-site


**mapreduce.map.memory.mb
4096
mapreduce.reduce.memory.mb
8192
mapreduce.map.java.opts
-Xmx3072m
mapreduce.reduce.java.opts
-Xmx6144m**



mapreduce.job.maps

** mapreduce.job.maps
4**
You see the image below .Actually the total containers were running as 11 . there was 40 vcores were available, among them 11 vcores were used while running MR program . can you please share what properties needs to change for that .?

enter image description here![alt text][1]


  [1]: /storage/temp/258-image.png




Outcomes