Hi, I am using Spark 1.6.1. In my application I am submitting spark in cluster mode, there is possibility my application will wait for input to arrive before starting spark context. I have specified spark.yarn.am.waitTime=7200s so that even if spark context is not invoked in 2 hours still application times out, however my application ends much before 2 hours even if input has not arrived.
On further analysis I found in yarn-site.xml, yarn.resource.connect.max-wait.ms=1500000 which seems to be reason my application is getting timed out. But I am not sure. Is spark.yarn.am.waitTime capped by yarn.resource.connect.max-wait.ms. It would be great if someone can explain what is relation between these 2 parameters.