AnsweredAssumed Answered

APIServer - no graphs and Client Retrying connect to server: 0.0.0.0/0.0.0.0:8032

Question asked by pmichalski on Jul 13, 2018
Latest reply on Jul 17, 2018 by MichaelSegel

We have got problem with  MCS / apiserver. 

After login into service, for couple of seconds there is only message:

 

    Loading Graphs...

 

After that, all graphs in "Utilization Trend" panel do not have any "data points" (see screenshot).No data points

In the logs:

 

2018-07-13 16:19:21,376 INFO Client Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-07-13 16:19:22,377 INFO Client Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-07-13 16:19:23,377 INFO Client Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-07-13 16:19:24,378 INFO Client Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2018-07-13 16:19:25,375 ERROR MapRCliUtil TimeoutException caught during Getting Yarn Cluster Metrics from ResourceManager java.util.concurrent.TimeoutException
at java.util.concurrent.AbstractExecutorService.doInvokeAny(AbstractExecutorService.java:184)
at java.util.concurrent.AbstractExecutorService.invokeAny(AbstractExecutorService.java:225)
at com.mapr.cli.MapRCliUtil.asyncInvoke(MapRCliUtil.java:437)
at com.mapr.cli.MapRCliUtil.asyncInvoke(MapRCliUtil.java:429)
at com.mapr.cli.Dashboard.addYarnStats(Dashboard.java:404)
at com.mapr.cli.Dashboard.executeRealCommand(Dashboard.java:797)
at com.mapr.cliframework.base.CLIBaseClass.executeCommand(CLIBaseClass.java:238)
at com.mapr.admin.lib.MapRCliUtils.executeCLI(MapRCliUtils.java:54)
at com.mapr.admin.controller.RestController.executePost(RestController.java:56)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)

According to Ports Used by MapR 

ResourceManager

Admin RPC

Applications that

access the ResourceManager

Nodes running

ResourceManager

8033 

yarn.resourcemanager.admin.address in

/opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml

ResourceManager

Client RPC

Clients that

submit YARN applications

Nodes running

ResourceManager

8032 

yarn.resourcemanager.address in

/opt/mapr/hadoop/hadoop-<version>/etc/hadoop/yarn-site.xml


yarn-site.xml

<configuration>
<!-- Resource Manager MapR HA Configs -->
<property>
<name>yarn.resourcemanager.ha.custom-ha-enabled</name>
<value>true</value>
<description>MapR Zookeeper based RM Reconnect Enabled. If this is true, set the failover proxy to be the class MapRZKBasedRMFailoverProxyProvider</description>
</property>
<property>
<name>yarn.client.failover-proxy-provider</name>
<value>org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider</value>
<description>Zookeeper based reconnect proxy provider. Should be set if and only if mapr-ha-enabled property is true.</description>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
<description>RM Recovery Enabled</description>
</property>
<property>
<name>yarn.resourcemanager.ha.custom-ha-rmaddressfinder</name>
<value>org.apache.hadoop.yarn.client.MapRZKBasedRMAddressFinder</value>
</property>

We have got configured 5 Resources Managers (Active - Standby).
How can eliminate the errors from the log?

Outcomes