AnsweredAssumed Answered

Flink's yarn-session fails to read configuration files.

Question asked by rapplean on Jul 10, 2017
Latest reply on Jul 11, 2017 by rapplean

I'm attempting to set up a Flink session to run on a Mapr Yarn cluster. I've followed all of the instructions I could find, but it can't find Maprfs.

 

Update: when I added fs.defaultFS =  maprfs:/// to the yarn-site.xml, it stopped complaining about the file system being "file", and started complaining that maprfs wasn't a known file system.

 

I then exported HADOOP_CLASSPATH=$(hadoop classpath), which allowed it to find maprfs, but then it failed to interpret ${nodemanager.resource.cpu-vcores} into a number that the cluster could use.

 

Here are the initial symptoms:

 

 

user@flinkhost:~/flink-1.3.1$ echo $YARN_CONF_DIR
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
user@flinkhost:~/flink-1.3.1$ echo $HADOOP_CONF_DIR
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
user@flinkhost:~/flink-1.3.1$ ls -latr $HADOOP_CONF_DIR/*-site.xml
-rw-r--r-- 1 mapr root 5511 Jul 25 2016 /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop//kms-site.xml
-rw-r--r-- 1 mapr root 775 Jul 25 2016 /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop//hdfs-site.xml
-rw-r--r-- 1 mapr root 1110 Jun 21 21:32 /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop//mapred-site.xml
-rw-r--r-- 1 mapr root 901 Jun 21 21:32 /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop//core-site.xml
-rw-r--r-- 1 mapr root 3534 Jun 22 20:53 /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop//yarn-site.xml

user@flinkhost:~/flink-1.3.1$ bin/yarn-session.sh -n 7 -tm 8192 -s 4
..... (removed for readability)
2017-07-10 19:34:23,012 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
..... (removed for readability)
2017-07-10 19:34:23,316 WARN org.apache.flink.yarn.YarnClusterDescriptor - The file system scheme is 'file'. This indicates that the specified Hadoop configuration path is wrong and the system is using the default Hadoop configuration values.The Flink YARN client needs to store its files in a distributed file system
2017-07-10 19:34:23,317 WARN org.apache.flink.yarn.YarnClusterDescriptor - The configuration directory ('/home/etl/flink-1.3.1/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2017-07-10 19:34:23,324 INFO org.apache.flink.yarn.Utils - Copying from file:/home/etl/flink-1.3.1/conf/log4j.properties to file:/home/etl/.flink/application_1499449861390_0020/log4j.properties
..... (removed for readability)
2017-07-10 19:34:23,670 INFO org.apache.flink.yarn.YarnClusterDescriptor - Submitting application master application_1499449861390_0020
2017-07-10 19:34:23,890 INFO org.apache.flink.yarn.YarnClusterDescriptor - Deploying cluster, current state ACCEPTED
Error while deploying YARN cluster: Couldn't deploy Yarn cluster
Diagnostics: File file:///home/etl/.flink/application_1499449861390_0020/flink-conf.yaml does not exist
java.io.FileNotFoundException: File file:///home/etl/.flink/application_1499449861390_0020/flink-conf.yaml does not exist

Any help would be welcome.

Outcomes