AnsweredAssumed Answered

configuring hadoop.log.dir on EMR

Question asked by sorenmacbeth on Jun 5, 2013
Latest reply on Jun 6, 2013 by gera
Hello,

I'm running into problem running M3 version 2.1.2 on EMR. My root partitions are filling up. The main culprit are the logs under the `hadoop.log.dir` directory set in `taskcontrollers.cfg`. I would like to configure this to point to a different directory.

What is the recommended way to do with as part of my cluster startup? I'm assuming I'm going to have to write a custom bootstrap action. My first attempt was to use sed to change the directory using sed and then restart the tasktrackers via the `maprcli` using the following shell script:

<pre>
#!/bin/bash
set -x
MAPCLI=/usr/bin/maprcli
TASKCONTROLLER_CFG=/opt/mapr/hadoop/hadoop-0.20.2/conf/taskcontroller.cfg
sudo sed -i 's/hadoop.log.dir=.*/hadoop.log.dir=\/var\/tmp/' $TASKCONTROLLER_CFG
$MAPCLI node services -filter [ip=="*"] -tasktracker restart
</pre>

This didn't work. I got:

<code>ERROR (10002) -  Service: tasktracker is not configured on node: [nodename]</code>

How should I do this properly?

Outcomes