AnsweredAssumed Answered

Hadoop jobs fail with exitCode=7

Question asked by milindshah on Jul 29, 2017
Latest reply on Aug 17, 2017 by rgavryliuk

While running wordcount or teragen hadoop jobs, I receive following exception and job fail.

 

2017-07-29 03:19:28,424 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e04_1501314036327_0007_02_000001
2017-07-29 03:19:28,459 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /tmp/hadoop-mapr/nm-local-dir/nmPrivate/container_e04_1501314036327_0007_02_000001.tokens. Credentials list:
2017-07-29 03:19:28,478 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.security.ExternalTokenLocalizerFactory: Initialized external token localizer class - com.mapr.hadoop.yarn.nodemanager.MapRTicketLocalizer
2017-07-29 03:19:29,730 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource maprfs:///var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1501314036327_0007/job.splitmetainfo(->/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1501314036327_0007/filecache/10/job.splitmetainfo) transitioned from DOWNLOADING to LOCALIZED
2017-07-29 03:19:29,766 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource maprfs:///var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1501314036327_0007/job.jar(->/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1501314036327_0007/filecache/11/job.jar) transitioned from DOWNLOADING to LOCALIZED
2017-07-29 03:19:29,786 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource maprfs:///var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1501314036327_0007/job.split(->/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1501314036327_0007/filecache/12/job.split) transitioned from DOWNLOADING to LOCALIZED
2017-07-29 03:19:29,810 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource maprfs:///var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1501314036327_0007/job.xml(->/tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1501314036327_0007/filecache/13/job.xml) transitioned from DOWNLOADING to LOCALIZED
2017-07-29 03:19:29,812 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e04_1501314036327_0007_02_000001 transitioned from LOCALIZING to LOCALIZED
2017-07-29 03:19:29,852 ERROR com.mapr.hadoop.mapred.LocalVolumeAuxService: Can not find metadata for a job. Returning service metadata
2017-07-29 03:19:29,857 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_e04_1501314036327_0007_02_000001 transitioned from LOCALIZED to RUNNING
2017-07-29 03:19:29,863 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container container_e04_1501314036327_0007_02_000001 is : 7
2017-07-29 03:19:29,864 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception from container-launch with container ID: container_e04_1501314036327_0007_02_000001 and exit code: 7
ExitCodeException exitCode=7:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:563)
at org.apache.hadoop.util.Shell.run(Shell.java:460)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:748)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:354)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:87)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_e04_1501314036327_0007_02_000001
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 7
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=7:
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:563)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:460)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:748)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:305)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:354)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:87)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745)
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Shell output: main : command provided 1
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : user is mapr
2017-07-29 03:19:29,869 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : requested yarn user is mapr
2017-07-29 03:19:29,870 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 7

 

Job does not even launch.

Outcomes