AnsweredAssumed Answered

Problem running the examples on Oozie in a four-node cluster

Question asked by volans on Aug 27, 2013
Latest reply on Aug 27, 2013 by aaron
**Problem:**

Job failed with ErrorCode JA009: null

**Detail Information:**

So I am new to MapR Hadoop and Oozie. We have a four-node cluster installed with M5 and Oozie 3.3.0 already installed. Today I downloaded and unzipped ext-2.2.zip into /opt/mapr/oozie/. Then I edited /opt/mapr/oozie/oozie-3.3.0/bin/oozie-setup.sh with the following:

    addExtjs="true"
    extjsHome="/opt/mapr/oozie/ext-2.2"

After that I ran:

    /opt/mapr/oozie/oozie-3.3.0/bin/oozie-setup.sh
    /opt/mapr/oozie/oozie-3.3.0/bin/oozie-start.sh

At this point Oozie Web UI was up and then I followed the "**Examples**" instructions on this page, [http://www.mapr.com/doc/display/MapR2/Oozie][1].

As soon as I submitted the example job, I got the job failure with error code JA009: null. The full job log below. I didn't edit the job.properties file and everything seemed to be in place as far as I can tell. I even tried the workaround on this page: [http://answers.mapr.com/questions/3461/oozie-job-on-mapr-single-node-cluster][2] to run the oozie setup script with guava jar but I still saw the same issue.

Any help to run the example on the four-cluster node is appreciated.

**Job Log**(with actual user name replaced with foo below):

    2013-08-27 19:54:25,588  INFO ActionStartXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@:start:] Start action [0000001-130827155705456-oozie-mapr-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2013-08-27 19:54:25,589  WARN ActionStartXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@:start:] [***0000001-130827155705456-oozie-mapr-W@:start:***]Action status=DONE
    2013-08-27 19:54:25,589  WARN ActionStartXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@:start:] [***0000001-130827155705456-oozie-mapr-W@:start:***]Action updated in DB!
    2013-08-27 19:54:25,880  INFO ActionStartXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Start action [0000001-130827155705456-oozie-mapr-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2013-08-27 19:54:26,111  WARN MapReduceActionExecutor:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] credentials is null for the action
    2013-08-27 19:54:26,313  WARN ActionStartXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: null]
    org.apache.oozie.action.ActionExecutorException: JA009: null
     at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
     at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:739)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
     at org.apache.oozie.command.XCommand.call(XCommand.java:277)
     at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
     at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
     at java.lang.Thread.run(Thread.java:722)
    Caused by: java.io.EOFException
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at org.apache.hadoop.mapreduce.ID.readFields(ID.java:83)
     at org.apache.hadoop.mapreduce.JobID.readFields(JobID.java:122)
     at org.apache.oozie.service.MaprJobClient.submitJob(MaprJobClient.java:292)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:724)
     ... 10 more
    2013-08-27 19:54:26,314  INFO ActionStartXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Next Retry, Attempt Number [1] in [60,000] milliseconds
    2013-08-27 19:55:26,355  INFO ActionStartXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Start action [0000001-130827155705456-oozie-mapr-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2013-08-27 19:55:26,569  WARN MapReduceActionExecutor:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] credentials is null for the action
    2013-08-27 19:55:27,131  WARN ActionStartXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: null]
    org.apache.oozie.action.ActionExecutorException: JA009: null
     at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
     at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:739)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
     at org.apache.oozie.command.XCommand.call(XCommand.java:277)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
     at java.lang.Thread.run(Thread.java:722)
    Caused by: java.io.EOFException
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at org.apache.hadoop.mapreduce.ID.readFields(ID.java:83)
     at org.apache.hadoop.mapreduce.JobID.readFields(JobID.java:122)
     at org.apache.oozie.service.MaprJobClient.submitJob(MaprJobClient.java:292)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:724)
     ... 8 more
    2013-08-27 19:55:27,132  INFO ActionStartXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Next Retry, Attempt Number [2] in [60,000] milliseconds
    2013-08-27 19:56:27,182  INFO ActionStartXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Start action [0000001-130827155705456-oozie-mapr-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2013-08-27 19:56:27,392  WARN MapReduceActionExecutor:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] credentials is null for the action
    2013-08-27 19:56:27,627  WARN ActionStartXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: null]
    org.apache.oozie.action.ActionExecutorException: JA009: null
     at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
     at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:739)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
     at org.apache.oozie.command.XCommand.call(XCommand.java:277)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
     at java.lang.Thread.run(Thread.java:722)
    Caused by: java.io.EOFException
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at org.apache.hadoop.mapreduce.ID.readFields(ID.java:83)
     at org.apache.hadoop.mapreduce.JobID.readFields(JobID.java:122)
     at org.apache.oozie.service.MaprJobClient.submitJob(MaprJobClient.java:292)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:724)
     ... 8 more
    2013-08-27 19:56:27,628  INFO ActionStartXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Next Retry, Attempt Number [3] in [60,000] milliseconds
    2013-08-27 19:57:27,668  INFO ActionStartXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Start action [0000001-130827155705456-oozie-mapr-W@mr-node] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
    2013-08-27 19:57:27,887  WARN MapReduceActionExecutor:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] credentials is null for the action
    2013-08-27 19:57:28,121  WARN ActionStartXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Error starting action [mr-node]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: null]
    org.apache.oozie.action.ActionExecutorException: JA009: null
     at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:418)
     at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:739)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:889)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:211)
     at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:59)
     at org.apache.oozie.command.XCommand.call(XCommand.java:277)
     at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
     at java.lang.Thread.run(Thread.java:722)
    Caused by: java.io.EOFException
     at java.io.DataInputStream.readInt(DataInputStream.java:392)
     at org.apache.hadoop.mapreduce.ID.readFields(ID.java:83)
     at org.apache.hadoop.mapreduce.JobID.readFields(JobID.java:122)
     at org.apache.oozie.service.MaprJobClient.submitJob(MaprJobClient.java:292)
     at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:724)
     ... 8 more
    2013-08-27 19:57:28,122  WARN ActionStartXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Exceeded max retry count [3]. Suspending Job
    2013-08-27 19:57:28,122  WARN ActionStartXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] Suspending Workflow Job id=0000001-130827155705456-oozie-mapr-W
    2013-08-27 19:57:29,179  WARN CoordActionUpdateXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[0000001-130827155705456-oozie-mapr-W@mr-node] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
    2013-08-27 19:58:45,111  INFO KillXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[-] STARTED WorkflowKillXCommand for jobId=0000001-130827155705456-oozie-mapr-W
    2013-08-27 19:58:45,230  WARN CoordActionUpdateXCommand:542 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
    2013-08-27 19:58:45,231  INFO KillXCommand:539 - USER[foo] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000001-130827155705456-oozie-mapr-W] ACTION[-] ENDED WorkflowKillXCommand for jobId=0000001-130827155705456-oozie-mapr-W
    
    
[1]: http://www.mapr.com/doc/display/MapR2/Oozie
[2]: http://answers.mapr.com/questions/3461/oozie-job-on-mapr-single-node-cluster

Outcomes