ClassNotFoundException DirectFileOutputCommitter

Document created by wade on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Narsi Subramanian

Original Publication Date: May 3, 2015

 

Environment

Affects all versions of mapr software.
This issue is seen typically in developer machines, where one tries to submit a job that accesses MaprDB tables using the org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil API but does not have the correct classes in it's classpath.

 

Symptom

The exception stack normally looks like :

 

Exception in thread "main" java.lang.NoClassDefFoundError:

org/apache/hadoop/mapreduce/lib/output/DirectFileOutputCommitter

at com.aexp.ims.cmph4.RowCounter.run(RowCounter.java:84)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

........ ........

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.lib.output.DirectFileOutputCommitter

at java.net.URLClassLoader$1.run(URLClassLoader.java:202)

Root Cause

The DirectFileOutputCommitter class is part of the amazon-s3.jar shipped with the mapr distribution. This is missing from the classpath of the job submitted above and hence the issue.

 

Solution

Include the jar /opt/mapr/hadoop/hadoop-0.20.2/lib/amazon-s3.jar in your job classpath and that will resolve the issue.

Attachments

    Outcomes