AnsweredAssumed Answered

Getting JA009 while running Java action in Oozie 3.3.2

Question asked by psbora on May 17, 2015
Latest reply on May 17, 2015 by psbora
Hi, I am trying to run Java action in Oozie. I am running the example provided in the MapR Oozie example. I am getting Error Code: JA009. I am giving below complete log and job details below.

**Command:**

    /opt/mapr/oozie/oozie-3.3.2/bin/oozie job -oozie http://gold-oozie.db.aexp.com:11000/oozie/ -config job.properties -run


**job.properties**

    nameNode=maprfs:///
    jobTracker=maprfs:///
    queueName=default
    examplesRoot=examples
    oozie.wf.application.path=maprfs:/idn/home/uslcg/oozie/${examplesRoot}/apps/java-main
    

**workflow.xml**

    <workflow-app xmlns="uri:oozie:workflow:0.2" name="java-main-wf-mytest">
        <start to="java-node"/>
        <action name="java-node">
            <java>
                <job-tracker>${jobTracker}</job-tracker>
                <name-node>${nameNode}</name-node>
                <configuration>
                    <property>
                        <name>mapred.job.queue.name</name>
                        <value>${queueName}</value>
                    </property>
                </configuration>
                <main-class>org.apache.oozie.example.DemoJavaMain</main-class>
                <arg>Hello</arg>
                <arg>Oozie!</arg>
            </java>
            <ok to="end"/>
            <error to="fail"/>
        </action>
        <kill name="fail">
            <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
        </kill>
        <end name="end"/>
    </workflow-app>
    

**Log:**

    2015-05-17 01:10:44,761  INFO ActionStartXCommand:539 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@:start:] Start action [0019198-150430001250210-oozie-mapr-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2015-05-17 01:10:44,762  WARN ActionStartXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@:start:] [***0019198-150430001250210-oozie-mapr-W@:start:***]Action status=DONE
    2015-05-17 01:10:44,762  WARN ActionStartXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@:start:] [***0019198-150430001250210-oozie-mapr-W@:start:***]Action updated in DB!
    2015-05-17 01:10:44,884  INFO ActionStartXCommand:539 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Start action [0019198-150430001250210-oozie-mapr-W@java-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2015-05-17 01:10:47,480  WARN JavaActionExecutor:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] credentials is null for the action
    2015-05-17 01:10:51,213  INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] callback for action [0019198-150430001250210-oozie-mapr-W@java-node]
    2015-05-17 01:10:51,216 ERROR CompletedActionXCommand:536 - USER[-] GROUP[-] TOKEN[] APP[-] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] XException,
    org.apache.oozie.command.CommandException: E0800: Action it is not running its in [PREP] state, action [0019198-150430001250210-oozie-mapr-W@java-node]
     at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:85)
     at org.apache.oozie.command.XCommand.call(XCommand.java:248)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
    2015-05-17 01:10:51,653  WARN ActionStartXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Error starting action [java-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: null]
    org.apache.oozie.action.ActionExecutorException: JA009: null
     at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
     at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:762)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:913)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
     at org.apache.oozie.command.XCommand.call(XCommand.java:277)
     at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
     at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.EOFException
     at java.io.DataInputStream.readInt(DataInputStream.java:375)
     at org.apache.hadoop.mapreduce.ID.readFields(ID.java:83)
     at org.apache.hadoop.mapreduce.JobID.readFields(JobID.java:122)
     at org.apache.oozie.service.MaprJobClient.submitJob(MaprJobClient.java:300)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:747)
     ... 10 more
    2015-05-17 01:10:51,654  INFO ActionStartXCommand:539 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Next Retry, Attempt Number [1] in [60,000] milliseconds
    2015-05-17 01:11:51,695  INFO ActionStartXCommand:539 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Start action [0019198-150430001250210-oozie-mapr-W@java-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2015-05-17 01:11:51,807  WARN JavaActionExecutor:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] credentials is null for the action
    2015-05-17 01:12:00,503  INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] callback for action [0019198-150430001250210-oozie-mapr-W@java-node]
    2015-05-17 01:12:00,517 ERROR CompletedActionXCommand:536 - USER[-] GROUP[-] TOKEN[] APP[-] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] XException,
    org.apache.oozie.command.CommandException: E0800: Action it is not running its in [START_RETRY] state, action [0019198-150430001250210-oozie-mapr-W@java-node]
     at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:85)
     at org.apache.oozie.command.XCommand.call(XCommand.java:248)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
    2015-05-17 01:12:00,944  WARN ActionStartXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Error starting action [java-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: null]
    org.apache.oozie.action.ActionExecutorException: JA009: null
     at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
     at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:762)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:913)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
     at org.apache.oozie.command.XCommand.call(XCommand.java:277)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.EOFException
     at java.io.DataInputStream.readInt(DataInputStream.java:375)
     at org.apache.hadoop.mapreduce.ID.readFields(ID.java:83)
     at org.apache.hadoop.mapreduce.JobID.readFields(JobID.java:122)
     at org.apache.oozie.service.MaprJobClient.submitJob(MaprJobClient.java:300)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:747)
     ... 8 more
    2015-05-17 01:12:00,947  INFO ActionStartXCommand:539 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Next Retry, Attempt Number [2] in [60,000] milliseconds
    2015-05-17 01:13:00,979  INFO ActionStartXCommand:539 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Start action [0019198-150430001250210-oozie-mapr-W@java-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2015-05-17 01:13:01,082  WARN JavaActionExecutor:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] credentials is null for the action
    2015-05-17 01:13:03,964  INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] callback for action [0019198-150430001250210-oozie-mapr-W@java-node]
    2015-05-17 01:13:03,968 ERROR CompletedActionXCommand:536 - USER[-] GROUP[-] TOKEN[] APP[-] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] XException,
    org.apache.oozie.command.CommandException: E0800: Action it is not running its in [START_RETRY] state, action [0019198-150430001250210-oozie-mapr-W@java-node]
     at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:85)
     at org.apache.oozie.command.XCommand.call(XCommand.java:248)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
    2015-05-17 01:13:04,758  WARN ActionStartXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Error starting action [java-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: null]
    org.apache.oozie.action.ActionExecutorException: JA009: null
     at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
     at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:762)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:913)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
     at org.apache.oozie.command.XCommand.call(XCommand.java:277)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.EOFException
     at java.io.DataInputStream.readInt(DataInputStream.java:375)
     at org.apache.hadoop.mapreduce.ID.readFields(ID.java:83)
     at org.apache.hadoop.mapreduce.JobID.readFields(JobID.java:122)
     at org.apache.oozie.service.MaprJobClient.submitJob(MaprJobClient.java:300)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:747)
     ... 8 more
    2015-05-17 01:13:04,759  INFO ActionStartXCommand:539 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Next Retry, Attempt Number [3] in [60,000] milliseconds
    2015-05-17 01:14:04,788  INFO ActionStartXCommand:539 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Start action [0019198-150430001250210-oozie-mapr-W@java-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2015-05-17 01:14:04,895  WARN JavaActionExecutor:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] credentials is null for the action
    2015-05-17 01:14:07,158  INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] callback for action [0019198-150430001250210-oozie-mapr-W@java-node]
    2015-05-17 01:14:07,168 ERROR CompletedActionXCommand:536 - USER[-] GROUP[-] TOKEN[] APP[-] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] XException,
    org.apache.oozie.command.CommandException: E0800: Action it is not running its in [START_RETRY] state, action [0019198-150430001250210-oozie-mapr-W@java-node]
     at org.apache.oozie.command.wf.CompletedActionXCommand.eagerVerifyPrecondition(CompletedActionXCommand.java:85)
     at org.apache.oozie.command.XCommand.call(XCommand.java:248)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
    2015-05-17 01:14:07,602  WARN ActionStartXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Error starting action [java-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: null]
    org.apache.oozie.action.ActionExecutorException: JA009: null
     at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
     at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:762)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:913)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
     at org.apache.oozie.command.XCommand.call(XCommand.java:277)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:662)
    Caused by: java.io.EOFException
     at java.io.DataInputStream.readInt(DataInputStream.java:375)
     at org.apache.hadoop.mapreduce.ID.readFields(ID.java:83)
     at org.apache.hadoop.mapreduce.JobID.readFields(JobID.java:122)
     at org.apache.oozie.service.MaprJobClient.submitJob(MaprJobClient.java:300)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:747)
     ... 8 more
    2015-05-17 01:14:07,602  WARN ActionStartXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Exceeded max retry count [3]. Suspending Job
    2015-05-17 01:14:07,602  WARN ActionStartXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] Suspending Workflow Job id=0019198-150430001250210-oozie-mapr-W
    2015-05-17 01:14:08,616  WARN CoordActionUpdateXCommand:542 - USER[uslcg] GROUP[-] TOKEN[] APP[java-main-wf-mytest] JOB[0019198-150430001250210-oozie-mapr-W] ACTION[0019198-150430001250210-oozie-mapr-W@java-node] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
    

Outcomes