AnsweredAssumed Answered

NoSuchMethodException while using ToolRunner to execute jobs

Question asked by ravi on Feb 13, 2013
Latest reply on Feb 14, 2013 by gera
I was running my jobs directly on the cluster in the past and didn't use ToolRunner for MapR distribution. Since, my development team is started using the Hadoop, they want to have a flexibility using Hadoop within Eclipse for testing/debug cycle. I'm using MapR build version: 2.0.1.15869.GA

Followed the instructions for setting up the Windows [client][1].


  [1]: http://www.mapr.com/doc/display/MapR/Setting+Up+the+Client#SettingUptheClient-windowsclient

Now I'm unable to execute the same code with ToolRunner in place, I'm getting the following error:
<pre>
    13/02/13 12:07:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/02/13 12:07:16 INFO security.JniBasedUnixGroupsMappingWithFallback: Falling back to shell based
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.2--1, built on 08/30/2011 17:53 GMT
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:host.name=USENVEERAR1L1C.corp.emc.com
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_09
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:java.home=C:\Java\jdk1.7.0_09\jre
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:java.class.path=C:\opt\mapr\hadoop\hadoop-0.20.2\lib\amazon-s3.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\asm-3.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\aspectjrt-1.6.5.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\aspectjtools-1.6.5.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\aws-java-sdk-1.3.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-cli-1.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-codec-1.4.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-codec-1.5.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-configuration-1.8.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-daemon-1.0.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-el-1.0.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-httpclient-3.0.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-httpclient-3.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-lang-2.6.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-logging-1.0.4.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-logging-1.1.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-logging-api-1.0.4.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-math-2.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-net-1.4.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\commons-net-3.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\core-3.1.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\emr-metrics-1.0.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\eval-0.5.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\gson-1.4.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\guava-11.0.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\guava-13.0.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\guava-r09.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\hadoop-0.20.2-dev-capacity-scheduler.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\hadoop-0.20.2-dev-core.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\hadoop-0.20.2-dev-fairscheduler.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\hsqldb-1.8.0.10.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\httpclient-4.1.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\httpcore-4.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jackson-core-asl-1.5.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jackson-mapper-asl-1.5.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jasper-compiler-5.5.12.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jasper-runtime-5.5.12.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jersey-core-1.8.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jersey-json-1.8.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jersey-server-1.8.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jets3t-0.6.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jetty-6.1.14.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jetty-servlet-tester-6.1.14.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\jetty-util-6.1.14.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\junit-4.5.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\kfs-0.2.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\log4j-1.2.15.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\logging-0.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\maprfs-0.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\maprfs-test-0.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\mockito-all-1.8.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\mockito-all-1.8.5.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\mysql-connector-java-5.0.8-bin.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\oro-2.0.8.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\protobuf-java-2.4.1.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\servlet-api-2.5-6.1.14.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\slf4j-api-1.4.3.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\slf4j-log4j12-1.4.3.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\xmlenc-0.52.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\zookeeper-3.3.2.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\zookeeper-3.3.6.jar;C:\opt\mapr\hadoop\hadoop-0.20.2\conf
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:java.library.path=C:\opt\mapr\hadoop\hadoop-0.20.2\lib\native\Windows_7-amd64-64;C:\opt\mapr\hadoop\hadoop-0.20.2\lib\native\Windows_7-amd64-64
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=C:\Users\veerar1\AppData\Local\Temp\
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:os.name=Windows 7
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:os.version=6.1
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:user.name=veerar1
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:user.home=C:\Users\veerar1
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Client environment:user.dir=C:\Users\veerar1\workspace\myhadoop
13/02/13 12:07:17 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=10.5.152.222:5181,10.5.152.224:5181,10.5.152.232:5181 sessionTimeout=30000 watcher=com.mapr.fs.JobTrackerWatcher@40cd33b8
13/02/13 12:07:17 INFO zookeeper.ClientCnxn: Opening socket connection to server /10.5.152.222:5181
13/02/13 12:07:22 INFO zookeeper.ClientCnxn: Socket connection established to 10.5.152.222/10.5.152.222:5181, initiating session
13/02/13 12:07:22 INFO zookeeper.ClientCnxn: Session establishment complete on server 10.5.152.222/10.5.152.222:5181, sessionid = 0x3c63f8703630c8, negotiated timeout = 30000
13/02/13 12:07:22 INFO fs.JobTrackerWatcher: Current running JobTracker is: eng-poc02.archerlab.local/10.5.152.222:9001
13/02/13 12:07:23 WARN snappy.LoadSnappy: Snappy native library not loaded
13/02/13 12:07:23 INFO mapred.FileInputFormat: Total input paths to process : 1
13/02/13 12:07:23 INFO mapred.JobClient: Creating job's output directory at maprfs:/user/root/count
13/02/13 12:07:23 INFO mapred.JobClient: Creating job's user history location directory at maprfs:/user/root/count/_logs
13/02/13 12:07:24 INFO mapred.JobClient: Running job: job_201301221616_0024
13/02/13 12:07:25 INFO mapred.JobClient:  map 0% reduce 0%
13/02/13 12:07:25 ERROR ipc.RPC: FailoverProxy: Failing this Call: getTaskCompletionEventList for error(RemoteException): org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.NoSuchMethodException: org.apache.hadoop.mapred.JobSubmissionProtocol.getTaskCompletionEventList(org.apache.hadoop.mapred.JobID, int, int)
at java.lang.Class.getMethod(Class.java:1622)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:986)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1318)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1314)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1312)

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.NoSuchMethodException: org.apache.hadoop.mapred.JobSubmissionProtocol.getTaskCompletionEventList(org.apache.hadoop.mapred.JobID, int, int)
at java.lang.Class.getMethod(Class.java:1622)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:986)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1318)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1314)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1312)

at org.apache.hadoop.ipc.Client.call(Client.java:1095)
at org.apache.hadoop.ipc.Client.call(Client.java:1041)
at org.apache.hadoop.ipc.RPC$FailoverInvoker.invoke(RPC.java:540)
at org.apache.hadoop.mapred.$Proxy0.getTaskCompletionEventList(Unknown Source)
at org.apache.hadoop.mapred.JobClient$NetworkedJob.getTaskCompletionEventList(JobClient.java:418)
at org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1428)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1333)
at com.rsa.itsrm.poc.hadoop.WordCount.run(WordCount.java:78)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at com.rsa.itsrm.poc.hadoop.WordCount.main(WordCount.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Some error on socket 1144

</pre>

Outcomes