AnsweredAssumed Answered

Why do i get Communication exception in mapr cluster?

Question asked by reihane on Sep 15, 2013
Latest reply on Sep 27, 2013 by abhinav
I am working with a cluster of mapr.I am running a program.when i st the the number of reduce tasks to 10, 9 reducers become compelete but 1 of them fails.the output of 9 reducers is about 2 G but the output of 10th reduce is about 6 g. when its output becomes about 3.4 g, it fails with error:

<pre>
INFO org.apache.hadoop.mapred.Task: Communication exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: JvmValidate Failed. Ignoring request from task: attempt_201309161041_0007_r_000005_0, with JvmId: jvm_201309161041_0007_r_-1236217948
at org.apache.hadoop.mapred.TaskTracker.validateJVM(TaskTracker.java:4716)
at org.apache.hadoop.mapred.TaskTracker.ping(TaskTracker.java:4860)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:993)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1326)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1322)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1320)
at org.apache.hadoop.ipc.Client.call(Client.java:1095)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:275)
at sun.proxy.$Proxy0.ping(Unknown Source)
at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:680)
at java.lang.Thread.run(Thread.java:679)

2013-09-16 13:22:14,997 ERROR com.mapr.fs.Inode: Write failed for file: /output/random/50/compare.txt/_temporary/_attempt_201309161041_0007_r_000005_0/part-r-00005, error: Stale file handle


cldb.log:
Exception in thread "MetricsStorageTimer" java.lang.NoClassDefFoundError: com/mapr/fs/MapRFileStatus
at com.mapr.fs.cldb.dialhome.metrics.MetricsManager$2.run(MetricsManager.java:167)
at java.util.TimerThread.mainLoop(Timer.java:534)
at java.util.TimerThread.run(Timer.java:484)
Caused by: java.lang.ClassNotFoundException: com.mapr.fs.MapRFileStatus
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)

Outcomes