AnsweredAssumed Answered

Task Tracker Not Coming Up, Hadoop dfs with Intermittent Error

Question asked by parker on Feb 25, 2014
Latest reply on Feb 27, 2014 by parker
Used MAP Quick Intaller 3.1.0. CentOS 6.5.

The 3 control nodes appear to be working properly.

The data node task trackers never come up. I receive the following error in the nodes:

<code>
2014-02-25 14:10:21,878 INFO mapred.TaskTracker [main]: /tmp is not tmpfs or ramfs. Java Hotspot Instrumentation will be disabled by default
2014-02-25 14:10:21,880 INFO mapred.TaskTracker [main]: Cleaning up config files from the job history folder
2014-02-25 14:10:21,881 INFO mapred.TaskTracker [main]: TT local config  is /opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml
2014-02-25 14:10:21,881 INFO mapred.TaskTracker [main]: Loading resource properties file : /opt/mapr/logs/cpu_mem_disk
2014-02-25 14:10:21,882 INFO mapred.TaskTracker [main]: Physical memory reserved for mapreduce tasks = 1651507200 bytes
2014-02-25 14:10:21,882 INFO mapred.TaskTracker [main]: CPUS: 4
2014-02-25 14:10:21,882 INFO mapred.TaskTracker [main]: mfsCPUS: 2
2014-02-25 14:10:21,882 INFO mapred.TaskTracker [main]: Total MEM: 7.6889GB
2014-02-25 14:10:21,882 INFO mapred.TaskTracker [main]: Reserved MEM: 1375MB
2014-02-25 14:10:21,882 INFO mapred.TaskTracker [main]: Reserved MEM for Ephemeral slots 200
2014-02-25 14:10:21,882 INFO mapred.TaskTracker [main]: DISKS: 1
2014-02-25 14:10:21,882 INFO mapred.TaskTracker [main]: mfsDISKS: 1
2014-02-25 14:10:21,950 INFO mapred.MapRedSlotUtil [main]: Before adjustment, maxMapSlots = 1, maxReduceSlots = 0
2014-02-25 14:10:21,950 INFO mapred.MapRedSlotUtil [main]: After CPU adjustment, maxMapSlots = 1, maxReduceSlots = 0
2014-02-25 14:10:21,950 INFO mapred.MapRedSlotUtil [main]: After Disk adjustment, maxMapSlots = 1, maxReduceSlots = 0
2014-02-25 14:10:21,950 INFO mapred.MapRedSlotUtil [main]: After adjustment, maxMapSlots = 1, maxReduceSlots = 1
2014-02-25 14:10:21,950 INFO mapred.MapRedSlotUtil [main]: mapTaskMem = 1024, reduceTaskMem = 3072
2014-02-25 14:10:21,950 INFO mapred.TaskTracker [main]: map and reduce slots have been computed based on requested
heap sizes for map and reduce slots
2014-02-25 14:10:21,950 INFO mapred.TaskTracker [main]: maptask heapsize: 1024
2014-02-25 14:10:21,951 INFO mapred.TaskTracker [main]: reducetask heapsize: 3072
2014-02-25 14:10:21,951 INFO mapred.TaskTracker [main]: Map slots 1, Default heapsize for map task 1024 mb
2014-02-25 14:10:21,951 INFO mapred.TaskTracker [main]: Reduce slots 1, Default heapsize for reduce task 3072 mb
2014-02-25 14:10:21,951 INFO mapred.TaskTracker [main]: Ephemeral slots 1, memory given for each ephemeral slot 200 mb
2014-02-25 14:10:21,951 INFO mapred.TaskTracker [main]: Prefetch map slots 0
2014-02-25 14:10:22,032 INFO mortbay.log [main]: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2014-02-25 14:10:22,113 INFO http.HttpServer [main]: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2014-02-25 14:10:22,133 INFO mapred.TaskLogsTruncater [main]: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
<b>2014-02-25 14:10:22,141 INFO mapred.TaskTracker [main]: Checking for local volume. If volume is not present command will create and mount it. Command invoked is : /opt/mapr/server/createTTVolume.sh NER-FAA04.icenet.local /var/mapr/local/NER-FAA04.icenet.local/mapred/ /var/mapr/local/NER-FAA04.icenet.local/mapred/taskTracker/</b>

Another post mentioned that Hadoop may not be running correctly. I check to see if the file system was running, and I'll get an intermittent error from time to time.

[root@NER-FAA04 ~]# hadoop dfs -ls /
Found 3 items
-rwxr-xr-x   3 mapr mapr          8 2014-02-25 11:19 /hbase
-rwxr-xr-x   3 mapr mapr          1 2014-02-24 14:50 /user
drwxr-xr-x   - mapr mapr          1 2014-02-24 14:50 /var
[root@NER-FAA04 ~]# hadoop dfs -ls /
Found 3 items
-rwxr-xr-x   3 mapr mapr          8 2014-02-25 11:19 /hbase
-rwxr-xr-x   3 mapr mapr          1 2014-02-24 14:50 /user
drwxr-xr-x   - mapr mapr          1 2014-02-24 14:50 /var
[root@NER-FAA04 ~]# hadoop dfs -ls /
<b>2014-02-25 14:14:35,1997 ERROR Cidcache fs/client/fileclient/cc/cidcache.cc:1288 Thread: 140447619344128 Lookup of volume users failed, error Connection reset by peer(104), CLDB: 192.168.52.137:7222 backing off ...</b>
Found 3 items
-rwxr-xr-x   3 mapr mapr          8 2014-02-25 11:19 /hbase
-rwxr-xr-x   3 mapr mapr          1 2014-02-24 14:50 /user
drwxr-xr-x   - mapr mapr          1 2014-02-24 14:50 /var

Outcomes