AnsweredAssumed Answered

spark SQL increase number of executors

Question asked by sgudavalliR on Oct 4, 2017
Latest reply on Oct 5, 2017 by cathy



I have a 5 node MapR FS cluster with total 10 cores and each node contains 2 Cores Available.

Total Memory available Is 35.84 GB. 


I am using MEP 1.1 on MapR 5.2 with Spark 1.6.1 version

My question Is how can i increase the number of executors, executor cores and spark.executor.memory



configurations passed thru spark-submit is not making any impact, and it is always two executors and with executor memory of 1G each.


I have configure following settings in yarn-site.xml, (and my assumption Is that each application submitted will get 4 cores, 10 GB each) .







can you please let me know if i am missing anything here.