AnsweredAssumed Answered

Blocking MapRClient.getZkConnectString()

Question asked by oae on Sep 29, 2014
Latest reply on Oct 30, 2014 by nabeel
Hey,:

Submitting multiple map-reduce jobs concurrently from within one JVM to an mapr cluster.
Before each job is submitted we query the cluster-status with a separate job client (JobClient#.getClusterStatus()).

Now a customer occasionally runs into following problem. On getClusterStatus() call on job submission seem to hang in:
<code>
at com.mapr.fs.jni.MapRClient.getZkConnectString(Native Method)
at com.mapr.fs.MapRClientImpl.getZkConnectString(MapRClientImpl.java:433)
-  locked com.mapr.fs.MapRClientImpl@6c900ec2
at com.mapr.fs.MapRClientImpl.getJobTrackerAddrs(MapRClientImpl.java:448)
-  locked com.mapr.fs.MapRClientImpl@6c900ec2
at com.mapr.fs.MapRFileSystem.getJobTrackerAddrs(MapRFileSystem.java:1542)
at org.apache.hadoop.ipc.RPC$FailoverInvoker.invoke(RPC.java:526)
-  locked org.apache.hadoop.ipc.RPC$FailoverInvoker@30626825
at org.apache.hadoop.mapred.$Proxy102.getClusterStatus(Unknown Source)
at org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:1276)
at org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:1263)
</code>
Any reasons or explanations how this can happen ?

All other job-submission then are locked that way:

<code>
"JobScheduler worker1-thread-1891" Id=233477 BLOCKED on com.mapr.fs.MapRClientImpl@6c900ec2 owned by "JobScheduler worker1-thread-1890" Id=233375
at com.mapr.fs.MapRClientImpl.getJobTrackerAddrs(MapRClientImpl.java:442)
-  blocked on com.mapr.fs.MapRClientImpl@6c900ec2
at com.mapr.fs.MapRFileSystem.getJobTrackerAddrs(MapRFileSystem.java:1542)
at org.apache.hadoop.ipc.RPC$FailoverInvoker.invoke(RPC.java:526)
-  locked org.apache.hadoop.ipc.RPC$FailoverInvoker@2c66ce98
at org.apache.hadoop.mapred.$Proxy102.getClusterStatus(Unknown Source)
at org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:1276)
at org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:1263)
</code>

Any help appreciated!
Johannes

Outcomes