AnsweredAssumed Answered

OOZIE -4.2.0 Yarn jobs on MAPR - throws  Failed startup of context o rg.mortbay.jetty.webapp.WebAppContext@414aa3df{/,jar:file:/opt/mapr/hadoop/hadoo p-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1506.jar!/webapps/mapred uce}

Question asked by kmurthy on Dec 14, 2015
Latest reply on Aug 30, 2017 by Sorabh
Hi

I am running the oozie 4.20 on MAPR 5.0  I used the following command to run the oozie job for map-reduce

/opt/mapr/oozie/oozie-4.2.0/bin/oozie job oozie="http://myhost.com:11000/oozie" -config /opt/mapr/oozie/oozie-4.2.0/examples/apps/map-reduce/job.properties -run

I am getting the following error.

I have run the standalone YARN jobs and it runs fine on my MAPR cluster. It is only when run through oozie 4.2.0  installed on MAPR there seems to be conflict in the classpath for the OOZIE web server and it fails to start the webserver.  This happens when the following command is run too' ( a hive job)

/opt/mapr/oozie/oozie-4.2.0/bin/oozie job oozie="http://myhost.com:11000/oozie" -config /opt/mapr/oozie/oozie-4.2.0/examples/apps/hive/job.properties -run

Jobs fail beaccuse of oozie web server not finding the explode of the following jar file.  ANy lead toward this will greatly help moving forward with the MAPR - oozie job . Agian My standalone YARN jobs on my mapr clusters work fine

Thanks in advance

2015-12-14 14:08:57,108 INFO [main] org.mortbay.log: Extract jar:file:/opt/mapr/
hadoop/hadoop-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1506.jar!/we
bapps/mapreduce to /opt/mapr/oozie/oozie-4.2.0/bin/tmp/Jetty_0_0_0_0_47454_mapre
duce____.mk1gvr/webapp
2015-12-14 14:08:57,118 WARN [main] org.mortbay.log: Failed startup of context o
rg.mortbay.jetty.webapp.WebAppContext@414aa3df{/,jar:file:/opt/mapr/hadoop/hadoo
p-2.7.0/share/hadoop/yarn/hadoop-yarn-common-2.7.0-mapr-1506.jar!/webapps/mapred
uce}
java.io.FileNotFoundException: /opt/mapr/oozie/oozie-4.2.0/bin/tmp/Jetty_0_0_0_0
_47454_mapreduce____.mk1gvr/webapp/webapps/mapreduce/.keep (No such file or dire
ctory)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
        at org.mortbay.resource.JarResource.extract(JarResource.java:215)
        at org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.ja
va:974)
        at org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:8
32)
        at org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(Web
InfConfiguration.java:62)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489
)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
        at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection
.java:152)
        at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHan
dlerCollection.java:156)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:
130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:861)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:285)
        at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStar
t(MRClientService.java:142)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast
er.java:1088)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java
:1500)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1595)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
MRAppMaster.java:1496)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:
1429)
2015-12-14 14:08:57,135 INFO [main] org.mortbay.log: Started HttpServer2$SelectC
hannelConnectorWithSafeStartup@0.0.0.0:47454
2015-12-14 14:08:57,135 INFO [main] org.mortbay.log: Stopped HttpServer2$SelectC
hannelConnectorWithSafeStartup@0.0.0.0:47454
2015-12-14 14:08:57,136 WARN [294349250@qtp-1937992831-1 - Acceptor0 HttpServer2
$SelectChannelConnectorWithSafeStartup@0.0.0.0:47454] org.apache.hadoop.http.Htt
pServer2: HttpServer Acceptor: isRunning is false. Rechecking.
2015-12-14 14:08:57,136 WARN [294349250@qtp-1937992831-1 - Acceptor0 HttpServer2
$SelectChannelConnectorWithSafeStartup@0.0.0.0:47454] org.apache.hadoop.http.Htt
pServer2: HttpServer Acceptor: isRunning is false
2015-12-14 14:08:57,237 ERROR [main] org.apache.hadoop.mapreduce.v2.app.client.M
RClientService: Webapps failed to start. Ignoring for now:
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:290)
        at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStar
t(MRClientService.java:142)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast
er.java:1088)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java
:1500)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1595)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
MRAppMaster.java:1496)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:
1429)
Caused by: java.io.IOException: Unable to initialize WebAppContext
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:883)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:285)
        ... 10 more
Caused by: java.io.FileNotFoundException: /opt/mapr/oozie/oozie-4.2.0/bin/tmp/Je
tty_0_0_0_0_47454_mapreduce____.mk1gvr/webapp/webapps/mapreduce/.keep (No such f
ile or directory)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
        at org.mortbay.resource.JarResource.extract(JarResource.java:215)
        at org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.ja
va:974)
        at org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:8
32)
        at org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(Web
InfConfiguration.java:62)
        at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489
)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
        at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection
.java:152)
        at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHan
dlerCollection.java:156)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
        at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:
130)
        at org.mortbay.jetty.Server.doStart(Server.java:224)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:
50)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:861)
        ... 11 more
2015-12-14 14:08:57,241 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Usin
g callQueue class java.util.concurrent.LinkedBlockingQueue
2015-12-14 14:08:57,241 INFO [Socket Reader #1 for port 57184] org.apache.hadoop
.ipc.Server: Starting Socket Reader #1 for port 57184
2015-12-14 14:08:57,248 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server
: IPC Server Responder: starting
2015-12-14 14:08:57,248 INFO [IPC Server listener on 57184] org.apache.hadoop.ip
c.Server: IPC Server listener on 57184: starting
2015-12-14 14:08:57,318 ERROR [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCom
municator: Exception while registering
java.lang.NullPointerException
        at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort
(MRClientService.java:174)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu
nicator.java:156)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMC
ommunicator.java:121)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRout
er.serviceStart(MRAppMaster.java:823)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast
er.java:1091)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java
:1500)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1595)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
MRAppMaster.java:1496)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:
1429)
2015-12-14 14:08:57,318 INFO [main] org.apache.hadoop.service.AbstractService: S
ervice RMCommunicator failed in state STARTED; cause: org.apache.hadoop.yarn.exc
eptions.YarnRuntimeException: java.lang.NullPointerException
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerExc
eption
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu
nicator.java:177)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMC
ommunicator.java:121)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRout
er.serviceStart(MRAppMaster.java:823)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast
er.java:1091)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java
:1500)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1595)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
MRAppMaster.java:1496)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:
1429)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort
(MRClientService.java:174)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu
nicator.java:156)
        ... 13 more
2015-12-14 14:08:57,319 INFO [main] org.apache.hadoop.service.AbstractService: S
ervice org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter f
ailed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeExce
ption: java.lang.NullPointerException
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerExc
eption
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu
nicator.java:177)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMC
ommunicator.java:121)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRout
er.serviceStart(MRAppMaster.java:823)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast
er.java:1091)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java
:1500)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1595)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
MRAppMaster.java:1496)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:
1429)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort
(MRClientService.java:174)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu
nicator.java:156)
        ... 13 more
2015-12-14 14:08:57,319 INFO [main] org.apache.hadoop.service.AbstractService: S
ervice org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED; c
ause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPoin
terException
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerExc
eption
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu
nicator.java:177)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMC
ommunicator.java:121)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRout
er.serviceStart(MRAppMaster.java:823)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeServ
ice.java:120)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMast
er.java:1091)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:
193)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java
:1500)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1595)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(
MRAppMaster.java:1496)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:
1429)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort
(MRClientService.java:174)
        at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommu
nicator.java:156)
        ... 13 more
2015-12-14 14:08:57,319 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi
storyEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queu
e size is 2
2015-12-14 14:08:57,319 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi
storyEventHandler: In stop, writing event AM_STARTED
2015-12-14 14:08:57,380 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi
storyEventHandler: Event Writer setup for JobId: job_1450117634168_0010, File: m
aprfs:/var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1450117634168_0010/job
_1450117634168_0010_1.jhist
2015-12-14 14:08:57,550 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi
storyEventHandler: In stop, writing event JOB_SUBMITTED
2015-12-14 14:08:57,551 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHi
storyEventHandler: Stopped JobHistoryEventHandler. super.stop()
2015-12-14 14:08:57,551 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMast
er: Skipping cleaning up the staging dir. assuming AM will be retried.
2015-12-14 14:08:57,552 INFO [main] org.apache.hadoop.ipc.Server: Stopping serve
r on 57184
2015-12-14 14:08:57,552 INFO [IPC Server listener on 57184] org.apache.hadoop.ip
c.Server: Stopping IPC Server listener on 57184
2015-12-14 14:08:57,552 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoo
p.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2015-12-14 14:08:57,552 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server
: Stopping IPC Server Responder

Outcomes