Job fails with ClassNotFoundException DirectFileOutputCommitter

Document created by wade on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Nabeel Moidu, last modified by Sanjamala Nayeem on May 7, 2015

 

Original Publication Date: May 2, 2015

 

Environment

Affects all versions of mapr software.This issue is seen typically in developer machines, where one tries to submit a job that accesses MaprDB tables using the org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil API but does not have the correct classes in his classpath.

 

Symptom

The exception stack normally looks like :

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/lib/output/DirectFileOutputCommitter
at com.aexp.ims.cmph4.RowCounter.run(RowCounter.java:84)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
........ ........
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.lib.output.DirectFileOutputCommitter
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)

Root Cause

The DirectFileOutputCommitter class is part of the amazon-s3.jar shipped with the mapr distribution. This is missing from the classpath of the job submitted above. Hence the issue.

 

Solution

Include the jar /opt/mapr/hadoop/hadoop-0.20.2/lib/amazon-s3.jar in your job classpath and issue will be resolved.

Attachments

    Outcomes