AnsweredAssumed Answered

mapreduce jobs stuck in pending state

Question asked by alexwoolford on Feb 17, 2015
Latest reply on Feb 17, 2015 by alexwoolford
I installed the MapR Community version running on three Ubuntu boxes using the Quick Installer. According the the MCS console, everything seems to be healthy (i.e. no alarms).
Here are the services that are running on each of the three nodes:

    $ ansible cluster -a "ls /opt/mapr/roles" -u root -k
    SSH password:
    
    hadoop01 | success | rc=0 >>
    cldb
    fileserver
    historyserver
    jobtracker
    metrics
    nfs
    resourcemanager
    webserver
    zookeeper
    
    hadoop02 | success | rc=0 >>
    fileserver
    metrics
    resourcemanager
    webserver
    zookeeper
    
    hadoop03 | success | rc=0 >>
    fileserver
    metrics
    resourcemanager
    webserver
    zookeeper

I'm running Hadoop2, though I've tried running Terasort using both classic hadoop and Yarn:

    # maprcli cluster mapreduce get
    default_mode  mapreduce_version 
    yarn          2.5.1


When I attempt to run Terasort, the job appears to kick off:

    maprcli volume create -name data1 –replication 1 –mount 1 –path /data1
    
    hadoop jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar teragen 5000000 /data1/out1
    
    15/02/17 16:26:55 INFO terasort.TeraSort: Generating 5000000 using 2
    15/02/17 16:26:55 INFO mapreduce.JobSubmitter: number of splits:2
    15/02/17 16:26:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1424204730264_0005
    15/02/17 16:26:55 INFO security.ExternalTokenManagerFactory: Initialized external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager
    15/02/17 16:26:55 INFO impl.YarnClientImpl: Submitted application application_1424204730264_0005
    15/02/17 16:26:55 INFO mapreduce.Job: The url to track the job: http://hadoop02.woolford.io:8088/proxy/application_1424204730264_0005/
    15/02/17 16:26:55 INFO mapreduce.Job: Running job: job_1424204730264_0005

Unfortunately, although the mapreduce application gets accepted, its final status is listed as 'UNDEFINED'. The tracking URL is listed as 'UNASSIGNED'.

I initially suspected that this might be caused by a DNS issue, but I checked and both forward and reverse DNS appear to be working:

    [mapr@hadoop02 ~]$ nslookup hadoop01.woolford.io
    Server:  10.0.1.20
    Address: 10.0.1.20#53

    Name: hadoop01.woolford.io
    Address: 10.0.1.11
    
    [mapr@hadoop02 ~]$ nslookup hadoop01
    Server:  10.0.1.20
    Address: 10.0.1.20#53
    
    Name: hadoop01.woolford.io
    Address: 10.0.1.11
    
    [mapr@hadoop02 ~]$ nslookup 10.0.1.11
    Server:  10.0.1.20
    Address: 10.0.1.20#53
    
    11.1.0.10.in-addr.arpa name = hadoop01.


I looked through the logs in /opt/mapr/ and didn't see anything that was obvious to me. Does anyone have any suggestions on how to troubleshoot this?

Outcomes