AnsweredAssumed Answered

Flink with Yarn on MapR

Question asked by ani.desh1512 on Jan 20, 2017
Latest reply on Feb 2, 2017 by maprcommunity

Hi,

I am trying to setup flink with Yarn on Mapr cluster. Mapr version is 5.2.0. I built flink (flink-1.3-SNAPSHOT) as follows:

 

mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.0-mapr-1607

 

The build is successful. Then I try to run ./bin/yarn-session.sh -n 4 (without changing any config or whatsoever) and get the following error:

  

 

Error while deploying YARN cluster: Couldn't deploy Yarn cluster

java.lang.RuntimeException: Couldn't deploy Yarn cluster

                at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:425)

                at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:620)

                at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:476)

                at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:473)

                at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)

                at java.security.AccessController.doPrivileged(Native Method)

                at javax.security.auth.Subject.doAs(Subject.java:422)

                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)

                at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)

                at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:473)

Caused by: java.lang.NumberFormatException: For input string: "${nodemanager.resource.cpu-vcores}"

                at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

                at java.lang.Integer.parseInt(Integer.java:569)

                at java.lang.Integer.parseInt(Integer.java:615)

                at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1271)

                at org.apache.flink.yarn.AbstractYarnClusterDescriptor.isReadyForDeployment(AbstractYarnClusterDescriptor.java:315)

                at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:434)

                at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:423)

                ... 9 more

 

Now, the property that is causing this error nodemanager.resource.cpu-vcores is appropriately set in yarn-site.xml. The cluster is 3 ResourceManager (2 on standby) and 5 NodeManager. To be extra safe, I changed the value for this property at ALL the Nodemanager’s yarn-site.xml.

I believe that this property is default set to 4 . So I am trying to understand as to why is this error cropping up.

The required environment variable is set as follows:

YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/

 

I also tried setting the fs.hdfs.hadoopconf property (to point to the Hadoop conf directory) in flink-config.yaml. But I still get the same error. Another thing I tried is setting fs.default-scheme: maprfs:/// in flink-config.yaml, but with no luck

 

 

Any help with this error would be greatly appreciated.

Outcomes