AnsweredAssumed Answered

Cannot create VM thread. Out of system resources.

Question asked by cooper6581 on Apr 12, 2013
When attempting to run a job, we are getting intermittent cores on a few of the nodes. The jobs are completing successfully.

We are running M3 (2.1.1.17042.GA-1), on a 20 node cluster.

This last instance, we had two nodes core 3 seconds apart.

Warden's logs show something like this for the first node (no errors in the second nodes warden log):

    2013-04-12 22:06:51,627 ERROR com.mapr.warden.service.baseservice.Service executeSimpleSHHCommand [pool-3-thread-1]: Error while running command: [hadoop, job, -list]
    2013-04-12 22:06:51,629 ERROR com.mapr.warden.service.baseservice.Service executeSimpleSHHCommand [pool-3-thread-1]: Error occurred during initialization of VM Cannot create VM thread. Out of system resources.

The cores show a generic JVM insufficient memory print out.  There is plenty of free physical memory when this occurs (or at least when the core is written out).

Could this possibly be due to the number of ParallelGCThreads?  We have tuned down the map / reduce slots to 6 / 2, but it didn't make any difference.  Does anyone happen to have any suggestions on where to look?

Outcomes