AnsweredAssumed Answered

spark SQL increase number of executors

Question asked by sgudavalliR on Oct 4, 2017
Latest reply on Oct 5, 2017 by cathy

Hello,

 

I have a 5 node MapR FS cluster with total 10 cores and each node contains 2 Cores Available.

Total Memory available Is 35.84 GB. 

 

I am using MEP 1.1 on MapR 5.2 with Spark 1.6.1 version

My question Is how can i increase the number of executors, executor cores and spark.executor.memory

 

 

configurations passed thru spark-submit is not making any impact, and it is always two executors and with executor memory of 1G each.

 

I have configure following settings in yarn-site.xml, (and my assumption Is that each application submitted will get 4 cores, 10 GB each) .

 

<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>4</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>10240</value>
</property>

 

can you please let me know if i am missing anything here. 

 

Regards

Shiv

Outcomes