AnsweredAssumed Answered

Cascading TemplateTap keep alive being ignored?

Question asked by chriscurtin on Oct 26, 2012
Latest reply on Oct 29, 2012 by gera

Short version: we have a Cascading TemplateTap output that is creating 10's of thousands of 'bins' and during the 'committing' step of the reducer Hadoop often kills out jobs because they did not report status in 60 seconds.

Long version is here, with Chris W's thought that the keep alive isn't being processed:!topic/cascading-user/sOv24j8859s

Note that we increased the timeout beyond 600 seconds but that's a band-aid.

Not sure if it matters, but this volume has tens of millions of files in it (no real idea of how many any more) and mfs is taking 30%+ CPU during this step.