AnsweredAssumed Answered

How do you pass external jars to map reduce job from application

Question asked by charanthota on Mar 28, 2016
Latest reply on Apr 19, 2016 by maprcommunity

I want to use DataFactory jar in my mapper class to generate some random data and load it in to hbase. So I came up with the below two classes:

 

BulkLoadCreditData.java

MyMapper.java

 

I tried the Distributed Cache to load the jars in to classpath, but it still gives the same below error.

 

I am running the application below way.

 

java -cp `hbase classpath`:./myjar.jar:./datafactory-0.8.jar  BulkLoadCreditData /projects/hbase/Card /projects/poc

 

and I am getting the  below class not found exception.

 

2016-03-28 11:15:59,861 INFO  [main] mapreduce.Job: Task Id : attempt_1459152725435_0011_m_000003_2, Status : FAILED

Error: java.lang.ClassNotFoundException: org.fluttercode.datafactory.impl.DataFactory

        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

        at MyMapper.setup(MyMapper.java:54)

        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:346)

        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)

        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Outcomes