AnsweredAssumed Answered

OOZIE -4.2.0 Yarn jobs on MAPR - throws Failed startup of context error

Question asked by kalpanamurthyl3 on Dec 16, 2015
OOZIE -4.2.0 Yarn jobs on MAPR - throws Failed startup of context o rg.mortbay.jetty.webapp.WebAppContext@414aa3df{/,jar:file:/opt/mapr/hadoop/hadoo p-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1506.jar!/webapps/mapred uce}
Hi
I am running the oozie 4.20 on MAPR 5.0 I used the following command to run the oozie job for map-reduce
/opt/mapr/oozie/oozie-4.2.0/bin/oozie job oozie="http://myhost.com:11000/oozie" -config /opt/mapr/oozie/oozie-4.2.0/examples/apps/map-reduce/job.properties -run
I am getting the following error.
I have run the standalone YARN jobs and it runs fine on my MAPR cluster. It is only when run through oozie 4.2.0 installed on MAPR there seems to be conflict in the classpath for the OOZIE web server and it fails to start the webserver. This happens when the following command is run too' ( a hive job)
/opt/mapr/oozie/oozie-4.2.0/bin/oozie job oozie="http://myhost.com:11000/oozie" -config /opt/mapr/oozie/oozie-4.2.0/examples/apps/hive/job.properties -run
Jobs fail beaccuse of oozie web server not finding the explode of the following jar file. ANy lead toward this will greatly help moving forward with the MAPR - oozie job . Agian My standalone YARN jobs on my mapr clusters work fine
Thanks in advance
2015-12-14 14:08:57,108 INFO [main] org.mortbay.log: Extract jar:file:/opt/mapr/ hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1506.jar!/we bapps/mapreduce to /opt/mapr/oozie/oozie-4.2.0/bin/tmp/Jetty_0_0_0_0_47454_mapre duce_.mk1gvr/webapp 2015-12-14 14:08:57,118 WARN [main] org.mortbay.log: Failed startup of context o rg.mortbay.jetty.webapp.WebAppContext@414aa3df{/,jar:file:/opt/mapr/hadoop/hadoo p-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1506.jar!/webapps/mapred uce} java.io.FileNotFoundException: /opt/mapr/oozie/oozie-4.2.0/bin/tmp/Jetty_0_0_0_047454_mapreduce.mk1gvr/webapp/webapps/mapreduce/.keep (No such file or dire ctory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:221) at java.io.FileOutputStream.(FileOutputStream.java:171) at org.mortbay.resource.JarResource.extract(JarResource.java:215) at org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.ja va:974) at org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:8 32) at org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(Web InfConfiguration.java:62) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489 ) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java: 50) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection .java:152) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHan dlerCollection.java:156) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java: 50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java: 130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java: 50) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:861) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:285) at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStar t(MRClientService.java:142) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast er.java:1088) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java :1500) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1595) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster( MRAppMaster.java:1496) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java: 1429) 2015-12-14 14:08:57,135 INFO [main] org.mortbay.log: Started HttpServer2$SelectC hannelConnectorWithSafeStartup@0.0.0.0:47454 2015-12-14 14:08:57,135 INFO [main] org.mortbay.log: Stopped HttpServer2$SelectC hannelConnectorWithSafeStartup@0.0.0.0:47454 2015-12-14 14:08:57,136 WARN [294349250@qtp-1937992831-1 - Acceptor0 HttpServer2 $SelectChannelConnectorWithSafeStartup@0.0.0.0:47454] org.apache.hadoop.http.Htt pServer2: HttpServer Acceptor: isRunning is false. Rechecking. 2015-12-14 14:08:57,136 WARN [294349250@qtp-1937992831-1 - Acceptor0 HttpServer2 $SelectChannelConnectorWithSafeStartup@0.0.0.0:47454] org.apache.hadoop.http.Htt pServer2: HttpServer Acceptor: isRunning is false 2015-12-14 14:08:57,237 ERROR [main] org.apache.hadoop.mapreduce.v2.app.client.M RClientService: Webapps failed to start. Ignoring for now: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:290) at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStar t(MRClientService.java:142) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast er.java:1088) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java :1500) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1595) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster( MRAppMaster.java:1496) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java: 1429) Caused by: java.io.IOException: Unable to initialize WebAppContext at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:883) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:285) ... 10 more Caused by: java.io.FileNotFoundException: /opt/mapr/oozie/oozie-4.2.0/bin/tmp/Je tty_0_0_0_0_47454_mapreduce____.mk1gvr/webapp/webapps/mapreduce/.keep (No such f ile or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:221) at java.io.FileOutputStream.(FileOutputStream.java:171) at org.mortbay.resource.JarResource.extract(JarResource.java:215) at org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.ja va:974) at org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:8 32) at org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(Web InfConfiguration.java:62) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489 ) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java: 50) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection .java:152) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHan dlerCollection.java:156) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java: 50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java: 130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java: 50) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:861) ... 11 more 2015-12-14 14:08:57,241 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Usin g callQueue class java.util.concurrent.LinkedBlockingQueue 2015-12-14 14:08:57,241 INFO [Socket Reader #1 for port 57184] org.apache.hadoop .ipc.Server: Starting Socket Reader #1 for port 57184 2015-12-14 14:08:57,248 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server : IPC Server Responder: starting 2015-12-14 14:08:57,248 INFO [IPC Server listener on 57184] org.apache.hadoop.ip c.Server: IPC Server listener on 57184: starting 2015-12-14 14:08:57,318 ERROR [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCom municator: Exception while registering java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort (MRClientService.java:174) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu nicator.java:156) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMC ommunicator.java:121) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRout er.serviceStart(MRAppMaster.java:823) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ ice.java:120) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast er.java:1091) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java :1500) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1595) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster( MRAppMaster.java:1496) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java: 1429) 2015-12-14 14:08:57,318 INFO [main] org.apache.hadoop.service.AbstractService: S ervice RMCommunicator failed in state STARTED; cause: org.apache.hadoop.yarn.exc eptions.YarnRuntimeException: java.lang.NullPointerException org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerExc eption at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu nicator.java:177) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMC ommunicator.java:121) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRout er.serviceStart(MRAppMaster.java:823) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ ice.java:120) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast er.java:1091) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java :1500) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1595) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster( MRAppMaster.java:1496) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java: 1429) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort (MRClientService.java:174) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu nicator.java:156) ... 13 more 2015-12-14 14:08:57,319 INFO [main] org.apache.hadoop.service.AbstractService: S ervice org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter f ailed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeExce ption: java.lang.NullPointerException org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerExc eption at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu nicator.java:177) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMC ommunicator.java:121) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRout er.serviceStart(MRAppMaster.java:823) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ ice.java:120) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast er.java:1091) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java :1500) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1595) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster( MRAppMaster.java:1496) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java: 1429) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort (MRClientService.java:174) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu nicator.java:156) ... 13 more 2015-12-14 14:08:57,319 INFO [main] org.apache.hadoop.service.AbstractService: S ervice org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED; c ause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPoin terException org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerExc eption at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu nicator.java:177) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMC ommunicator.java:121) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRout er.serviceStart(MRAppMaster.java:823) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ ice.java:120) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast er.java:1091) at org.apache.hadoop.service.AbstractService.start(AbstractService.java: 193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java :1500) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1595) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster( MRAppMaster.java:1496) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java: 1429) Caused by: java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort (MRClientService.java:174) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu nicator.java:156) ... 13 more 2015-12-14 14:08:57,319 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi storyEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queu e size is 2 2015-12-14 14:08:57,319 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi storyEventHandler: In stop, writing event AM_STARTED 2015-12-14 14:08:57,380 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi storyEventHandler: Event Writer setup for JobId: job_1450117634168_0010, File: m aprfs:/var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1450117634168_0010/job _1450117634168_0010_1.jhist 2015-12-14 14:08:57,550 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi storyEventHandler: In stop, writing event JOB_SUBMITTED 2015-12-14 14:08:57,551 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi storyEventHandler: Stopped JobHistoryEventHandler. super.stop() 2015-12-14 14:08:57,551 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMast er: Skipping cleaning up the staging dir. assuming AM will be retried. 2015-12-14 14:08:57,552 INFO [main] org.apache.hadoop.ipc.Server: Stopping serve r on 57184 2015-12-14 14:08:57,552 INFO [IPC Server listener on 57184] org.apache.hadoop.ip c.Server: Stopping IPC Server listener on 57184 2015-12-14 14:08:57,552 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoo p.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 2015-12-14 14:08:57,552 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server : Stopping IPC Server Responder
Like äó¢  0 äó¢  äó¢ Add comment äó¢  Hide 1
 kmurthy äó¢ 22 hours ago 0
The above error is shown in
Respective application logs at
/opt/mapr/hadoop/hadoop-2.7.0/logs/userlogs.

Outcomes