AnsweredAssumed Answered

mapred.reduce.tasks property cannot be set with HADOOP_OPTS

Question asked by andrescorrada on Oct 17, 2013
Latest reply on Oct 21, 2013 by gera
Hello,

I want to set the number of reduce tasks on the fly when I invoke "hadoop jar ..." on a MapR cluster. My attempt at this is to write a shell script that has:
export HADOOP_OPTS="-Dmapred.reduce.tasks=10"
By echoing the command before the end of the "hadoop" script, I can see that the property is correctly passed in the java command that call the RunJar class. But the number of reducers still ends being 1. Only when mapred-site.xml gets modified, does the number of reducers increase from 1 to the 10 the HADOOP_OPTS option is trying to set.

What am I missing.

Outcomes