Unable to run MapReduce jobs due to ClassNotFoundException

Document created by wade on Feb 27, 2016
Version 1Show Document
  • View in full screen mode

Author: Jitendra Yadav, last modified by Sanjamala Nayeem on May 7, 2015

 

Original Publication Date: May 1, 2015

 

Environment
MapR 3.1.1

Symptom

When running map-reduce jobs there is an issue that hadoop is unable to find a method in the jar which is a custom program packaged in a jar file.Although even after setting export HADOOP_CLASSPATH=mycustom.jar:$HADOOPCLASSPATH it didn't picked customer jar.

 

Root Cause
The Custom jar is not available on all the cluster nodes and causing "Class NotFound Exception" therefore either we need to manually add it on hadoop class path on all the nodes or we can add it in distributed cache while running application.

Solution

Adding in distributed cache in your program .

Try using DistributedCache.addFileToClassPath(

JobConf job = new JobConf();

DistributedCache.addFileToClassPath(new URI("/myapp/mycustom.jar"), job);

Attachments

    Outcomes