AnsweredAssumed Answered

spark am timeout

Question asked by mvashi2 on Aug 13, 2017
Latest reply on Aug 29, 2017 by mvashi2

Hi, I am using Spark 1.6.1.  In my application I am submitting spark in cluster mode, there is possibility my application will wait for input to arrive before starting spark context. I have specified spark.yarn.am.waitTime=7200s so that even if spark context is not invoked in 2 hours still application times out, however my application ends much before 2 hours even if input has not arrived.

On further analysis I found in yarn-site.xml, yarn.resource.connect.max-wait.ms=1500000 which seems to be reason my application is getting timed out. But I am not sure. Is spark.yarn.am.waitTime capped by yarn.resource.connect.max-wait.ms. It would be great if someone can explain what is relation between these 2 parameters.

Outcomes