AnsweredAssumed Answered

Spark job fails with memory limits

Question asked by i'msuku on Jun 23, 2016
Latest reply on Jul 10, 2016 by Deependra singh

Hi Experts,

 

Could you please help me out here..

 

I got a hive query which completes in 1hr 45min with hive-1.2, im trying to execute the same in spark with

exec-memory 14g

exec-instances 100

exec-cores 4

driver memory 14g

even though i get the below error

Caused by: java.io.IOException: Failed to connect to host:port
13 ERROR YarnScheduler: Lost executor 71 on host Container killed by YARN for exceeding memory limits. 9.3 GB of 9 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/06/23 09:49:13 WARN TaskSetManager: Lost task 25.0 in stage 84.0 (TID 40955, host ExecutorLostFailure (executor 71 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 9.3 GB of 9 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.
16/06/23 09:49:13 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 9.3 GB of 9 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

 

I have tried to run the same with spark.yarn.executor.memoryOverhead 1600, even though it complains the same.

the yarn container size is 16g, not sure why its failing out at 9g.

 

I would appreciate if someone help me out here.

 

regards,

Sukumar

Outcomes