AnsweredAssumed Answered

Flink with Yarn on MapR

Question asked by ani.desh1512 on Jan 20, 2017
Latest reply on Feb 2, 2017 by maprcommunity


I am trying to setup flink with Yarn on Mapr cluster. Mapr version is 5.2.0. I built flink (flink-1.3-SNAPSHOT) as follows:


mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.0-mapr-1607


The build is successful. Then I try to run ./bin/ -n 4 (without changing any config or whatsoever) and get the following error:



Error while deploying YARN cluster: Couldn't deploy Yarn cluster

java.lang.RuntimeException: Couldn't deploy Yarn cluster

                at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(


                at org.apache.flink.yarn.cli.FlinkYarnSessionCli$

                at org.apache.flink.yarn.cli.FlinkYarnSessionCli$


                at Method)




                at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(

Caused by: java.lang.NumberFormatException: For input string: "${nodemanager.resource.cpu-vcores}"

                at java.lang.NumberFormatException.forInputString(

                at java.lang.Integer.parseInt(

                at java.lang.Integer.parseInt(

                at org.apache.hadoop.conf.Configuration.getInt(

                at org.apache.flink.yarn.AbstractYarnClusterDescriptor.isReadyForDeployment(

                at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(

                at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(

                ... 9 more


Now, the property that is causing this error nodemanager.resource.cpu-vcores is appropriately set in yarn-site.xml. The cluster is 3 ResourceManager (2 on standby) and 5 NodeManager. To be extra safe, I changed the value for this property at ALL the Nodemanager’s yarn-site.xml.

I believe that this property is default set to 4 . So I am trying to understand as to why is this error cropping up.

The required environment variable is set as follows:



I also tried setting the fs.hdfs.hadoopconf property (to point to the Hadoop conf directory) in flink-config.yaml. But I still get the same error. Another thing I tried is setting fs.default-scheme: maprfs:/// in flink-config.yaml, but with no luck



Any help with this error would be greatly appreciated.