AnsweredAssumed Answered

Can we have the NodeManager auto create the shuffle volumes upon restart?

Question asked by apuranik on Mar 12, 2018
Latest reply on Mar 25, 2018 by apuranik

We recently switched to the option of writing shuffle data into local mfs volumes as opposed to local worker fs.

====================

/mapr/<cluster>/var/mapr/local/<host>/spark/usercache/<user>/appcache/

====================

We are looking for an option to have these local shuffle volumes auto created every time the nodemanager restarts or a new node is added to the cluster. We need something similar to the tasktracker local.mapred volumes from mrv1. 

Please let me know if I need to share any additional information here.

Outcomes