AnsweredAssumed Answered

Can we point spark scratch directory to /tmp on nfs instead of local?

Question asked by kiranjntukkd on Aug 21, 2017
Latest reply on Sep 20, 2017 by maprcommunity

Can we set yarn.nodemanager.local-dirs for only spark jobs to pick this nfs temp location , to avoid overburdening nfs service it with YARN or other tools using this nfs temp location?

 

We are using spark on yarn with spark at 1.6.1 version. Currently spark jobs are using yarn.nodemanager.local-dirs which is by default pointing to /tmp (local fs) which got limited storage capacity. When some big spark jobs are running /tmp is running out of disk. 

Outcomes