AnsweredAssumed Answered

Problems with reducers on MapR cluster

Question asked by philipt on Sep 25, 2012
Latest reply on Oct 3, 2012 by philipt
Hi,

I'm currently planning our migration from Hadoop to MapR. I've spun up a single node MapR v2.0.1 M3 cluster, and everything worked fine. I'm now spinning up a three node cluster to do some benchmarking with, and having problems getting it to work. I'm trying to run one of the Hadoop examples to test the cluster, and the reducers are failing:

(in /opt/mapr/hadoop/hadoop-0.20.2):

    # hadoop jar hadoop-0.20.2-dev-examples.jar pi 10 10

    Number of Maps  = 10
    Samples per Map = 10
    Wrote input for Map #0
    Wrote input for Map #1
    Wrote input for Map #2
    Wrote input for Map #3
    Wrote input for Map #4
    Wrote input for Map #5
    Wrote input for Map #6
    Wrote input for Map #7
    Wrote input for Map #8
    Wrote input for Map #9
    Starting Job
    12/09/25 09:38:20 INFO fs.JobTrackerWatcher: Current running JobTracker is: test_hive02.[...]/10.70.48.58:9001
    12/09/25 09:38:23 INFO mapred.FileInputFormat: Total input paths to process : 10
    12/09/25 09:38:23 INFO mapred.JobClient: Creating job's output directory at maprfs:/user/root/PiEstimator_TMP_3_141592654/out
    12/09/25 09:38:23 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/user/root/PiEstimator_TMP_3_141592654/out/_logs
    12/09/25 09:38:24 INFO mapred.JobClient: Running job: job_201209250935_0001
    12/09/25 09:38:25 INFO mapred.JobClient:  map 0% reduce 0%
    12/09/25 09:38:39 INFO mapred.JobClient:  map 20% reduce 0%
    12/09/25 09:38:41 INFO mapred.JobClient:  map 50% reduce 0%
    12/09/25 09:38:42 INFO mapred.JobClient:  map 60% reduce 0%
    12/09/25 09:38:43 INFO mapred.JobClient:  map 90% reduce 0%
    12/09/25 09:38:45 INFO mapred.JobClient:  map 100% reduce 0%
    12/09/25 09:38:49 INFO mapred.JobClient: Task Id : attempt_201209250935_0001_r_000000_0, Status : FAILED on node Unknown
    Error: java.lang.NullPointerException
            at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:3022)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2931)
    
    12/09/25 09:38:52 INFO mapred.JobClient: Task Id : attempt_201209250935_0001_r_000000_1, Status : FAILED on node Unknown
    Error: java.lang.NullPointerException
            at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:3022)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2931)
    
    12/09/25 09:38:55 INFO mapred.JobClient: Task Id : attempt_201209250935_0001_r_000000_2, Status : FAILED on node Unknown
    Error: java.lang.NullPointerException
            at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:3022)
            at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2931)

I've had a look round some of the logs and I can't see any obvious culprits. I'm new to MapR though, so I'm probably not looking in all of the right places. It looks a little similar to this situation: [http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201007.mbox/%3C441D899DB0E264409344AFC3D1998EEA01229FE6@email4.us.syncsort.com%3E][1duce-user/201007.mbox/%3C441D899DB0E264409344AFC3D1998EEA01229FE6@email4.us.syncsort.com%3E][1]

Does anyone have any ideas?

  [1]: http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201007.mbox/%3C441D899DB0E264409344AFC3D1998EEA01229FE6@email4.us.syncsort.com%3Educe-user/201007.mbox/%3C441D899DB0E264409344AFC3D1998EEA01229FE6@email4.us.syncsort.com%3E

Outcomes