AnsweredAssumed Answered

YARN Cluster settings...

Question asked by MichaelSegel on Jul 31, 2017
Latest reply on Jul 31, 2017 by MichaelSegel

Having a bit of a brain dead moment.

 

I was under the impression that YARN only had a single maximum size setting for YARN controlled containers.

 

The issue is that if one wants to use YARN to control Docker Containers, they may want to use one maximum setting, yet want another for their spark jobs running on the cluster.

 

Also does anyone have a good idea if YARN has fixed the issues with long running jobs?  So while a spark job may run for a couple of hours,  Docker containers can run for weeks/months without issue and then there's the security issue.

 

The reason that I am asking this is that I'm in the camp that its not a good idea to have YARN control all containers and that you don't really want to mix/match your cluster design to allow for docker containers anywhere.  I believe in a bit more architecture than complete chaos.

 

Am I being old school?  ;-)

Outcomes