Working on UDF fuctions with PIG error java.lang.NoSuchMethodError:

Document created by wade on Feb 27, 2016Last modified by wade on Feb 27, 2016
Version 2Show Document
  • View in full screen mode

Author: Jitendra Yadav, last modified by Sanjamala Nayeem on May 7, 2015

 

Original Publication Date: May 1, 2015

 

Environment
MapR 3.x
Pig .0.12, 0.13

Symptom

Facing issue while working with UDF jars inside pig shell, even after register the UDF jar, we are till getting "
NoSuchMethodError" exception.

2015-04-11 13:01:06,911 [Thread-9] WARN org.apache.hadoop.mapred.LocalJobRunner – job_local_0001 
java.lang.NoSuchMethodError: org.tick.time.moveany(Lorg/joda/time/DateTimeZone;)Lorg/tick/time/movenay;
at com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:178)
at com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:12)

Root Cause

The custom UDF jar still not accessible to all the tasktracker nodes and tasks causing this issue, we need to make sure that UDF method should be accessible on all the tasktracker nodes.

Solution
There are multiple option available for this and at least one should work perfect in any case.

1) Registering this tick jar in the PIG script using REGISTER call.

 

Grunt> REGISTER /path/tick.jar

2) using -Dpig.additional.jars= /path/tick.jar

3) Set this jar in $HADOOP_CLASSPATH in pig-env.sh

 

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/tick.jar

4) Set this jar in the $classpath  in pig-env.sh

 

export CLASSPATH=$CLASSPATH:/path/tick.jar

5) Set the jar in the $PIG_CLASSPATH in pig-env.sh

export PIG_CLASSPATH=$PIG_CLASSPATH:/path/tick.jar

Attachments

    Outcomes