AnsweredAssumed Answered

Failure starting jobs

Question asked by heathbar on Sep 13, 2012
Latest reply on Sep 13, 2012 by nabeel
I find that after I run a few dozen map reduce jobs successfully, eventually something goes wrong and all jobs I start fail immediately with the same message (no matter what job).  If I restart the job and task trackers, the problem goes away for awhile but eventually returns after a few dozen more jobs. 

This is the error message from MapR v2.0.0.

     12/09/14 05:05:51 INFO util.NativeCodeLoader: Loaded the native-hadoop library
     12/09/14 05:05:51 WARN snappy.LoadSnappy: Snappy native library not loaded
     12/09/14 05:05:52 INFO fs.JobTrackerWatcher: Current running JobTracker is: ip-10-188-2-182.us-west-1.compute.internal/10.188.2.182:9001
     12/09/14 05:05:52 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
     12/09/14 05:05:52 INFO mapred.FileInputFormat: Total input paths to process : 100
     12/09/14 05:05:52 INFO mapred.JobClient: Creating job's output directory at /data/test/tide_v00/sift/100000/bow.pert
     12/09/14 05:05:52 INFO mapred.JobClient: Creating job's user history location directory at /data/test/tide_v00/sift/100000/bow.pert/_logs
     12/09/14 05:05:53 INFO mapred.JobClient: Running job: job_201209132255_0050
     12/09/14 05:05:54 INFO mapred.JobClient:  map 0% reduce 0%
     12/09/14 05:05:54 INFO mapred.JobClient: Task Id : attempt_201209132255_0050_m_000297_0, Status : FAILED on node ip-10-162-10-255.us-west-1.compute.internal
     Error initializing attempt_201209132255_0050_m_000297_0:
     java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskTracker$6.run(TaskTracker.java:3771)

     12/09/14 05:05:54 WARN mapred.JobClient: Error reading task outputhttp://ip-10-162-10-255.us-west-1.compute.internal:50060/tasklog?plaintext=true&attemptid=attempt_201209132255_0050_m_000297_0&filter=stdout
     12/09/14 05:05:54 WARN mapred.JobClient: Error reading task outputhttp://ip-10-162-10-255.us-west-1.compute.internal:50060/tasklog?plaintext=true&attemptid=attempt_201209132255_0050_m_000297_0&filter=stderr
     12/09/14 05:05:54 INFO mapred.JobClient: Task Id : attempt_201209132255_0050_r_000101_0, Status : FAILED on node ip-10-162-10-255.us-west-1.compute.internal
     Error initializing attempt_201209132255_0050_r_000101_0:
     java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskTracker$6.run(TaskTracker.java:3771)

     12/09/14 05:05:54 WARN mapred.JobClient: Error reading task outputhttp://ip-10-162-10-255.us-west-1.compute.internal:50060/tasklog?plaintext=true&attemptid=attempt_201209132255_0050_r_000101_0&filter=stdout
     12/09/14 05:05:54 WARN mapred.JobClient: Error reading task outputhttp://ip-10-162-10-255.us-west-1.compute.internal:50060/tasklog?plaintext=true&attemptid=attempt_201209132255_0050_r_000101_0&filter=stderr
     12/09/14 05:05:55 INFO mapred.JobClient: Task Id : attempt_201209132255_0050_m_000297_1, Status : FAILED on node ip-10-162-171-111.us-west-1.compute.internal

Outcomes