AnsweredAssumed Answered

Taskracker start fail

Question asked by humphrey on Aug 11, 2013
Latest reply on Aug 14, 2013 by humphrey
the taskracker,random node, always start fialed when i restart my cluster.<br>here is my createTTVolume.2020.cmd.out

        2013-08-12 10:47:11,0993 ERROR Global mrconfig.cc:1160 x.x.0.0:0 PrintServerState: rpc failed 104.
        2013-08-12 10:47:11,0994 ERROR Global mrconfig.cc:3251 x.x.0.0:0 GetServerState failed Connection reset by peer.104.
part of createTTVolume.2020.log:
<br>
   

     2013-08-12 10:46:59 DEBUG Will launch command "/opt/mapr//server/mrconfig -p 5660 info fsstate" with a command attempt timeout of 60 seconds a maximum of 3 attempts an
    d a maximum cumulative timeout of 60 seconds
    2013-08-12 10:46:59 DEBUG Launching "/opt/mapr//server/mrconfig -p 5660 info fsstate"
    2013-08-12 10:47:00 DEBUG Command attempt 1 failed with return code 1 after 1 seconds
    2013-08-12 10:47:00 DEBUG Launching "/opt/mapr//server/mrconfig -p 5660 info fsstate"
    2013-08-12 10:47:01 DEBUG Command attempt 2 failed with return code 1 after 1 seconds
    2013-08-12 10:47:01 DEBUG Launching "/opt/mapr//server/mrconfig -p 5660 info fsstate"
    2013-08-12 10:47:03 DEBUG Command attempt 3 failed with return code 1 after 2 seconds
    2013-08-12 10:47:03 FATAL Command did not complete successfully after 3 attempts and after 4 seconds.
    2013-08-12 10:47:03 INFO The command run was:
    /opt/mapr//server/mrconfig -p 5660 info fsstate

    2013-08-12 10:47:03 INFO The output of the last failed command attempt:
    *** warn: MAPR_SUBNETS set to 192.168.0.0/24
    2013-08-12 10:47:01,9945 ERROR Global mrconfig.cc:1160 x.x.0.0:0 PrintServerState: rpc failed 104.
    2013-08-12 10:47:01,9947 ERROR Global mrconfig.cc:3251 x.x.0.0:0 GetServerState failed Connection reset by peer.104.
    2013-08-12 10:47:09 INFO This script was called with the arguments: hph05.hadoop /var/mapr/local/hph05.hadoop/mapred/ /var/mapr/local/hph05.hadoop/mapred/taskTracker/
    2013-08-12 10:47:09 INFO Checking if MapRFS is online
    2013-08-12 10:47:09 DEBUG Will launch command "hadoop fs -stat /" with a command attempt timeout of 60 seconds a maximum of 1000 attempts and a maximum cumulative time
    out of 600 seconds
    2013-08-12 10:47:09 DEBUG Launching "hadoop fs -stat /"
    
hadoop-mapr-tasktracker-hph05.log:

   
     ------------------------------------------------------------*/
        2013-08-12 00:04:43,958 INFO org.apache.hadoop.mapred.TaskTracker: /tmp is tmpfs. Java Hotspot Instrumentation will be enabled by default
        2013-08-12 00:04:43,960 INFO org.apache.hadoop.mapred.TaskTracker: Cleaning up config files from the job history folder
        2013-08-12 00:04:43,962 INFO org.apache.hadoop.mapred.TaskTracker: TT local config  is /opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml
        2013-08-12 00:04:43,962 INFO org.apache.hadoop.mapred.TaskTracker: Loading resource properties file : /opt/mapr//logs/cpu_mem_disk
        2013-08-12 00:04:43,963 INFO org.apache.hadoop.mapred.TaskTracker: Physical memory reserved for mapreduce tasks = 49297752064 bytes
        2013-08-12 00:04:43,963 INFO org.apache.hadoop.mapred.TaskTracker: CPUS: 24
        2013-08-12 00:04:43,963 INFO org.apache.hadoop.mapred.TaskTracker: Total MEM: 94.43953GB
        2013-08-12 00:04:43,963 INFO org.apache.hadoop.mapred.TaskTracker: Reserved MEM: 46814MB
        2013-08-12 00:04:43,963 INFO org.apache.hadoop.mapred.TaskTracker: Reserved MEM for Ephemeral slots 200
        2013-08-12 00:04:43,963 INFO org.apache.hadoop.mapred.TaskTracker: DISKS: 3
        2013-08-12 00:04:43,986 INFO org.apache.hadoop.mapred.TaskTracker: map and reduce slots have been computed based on requested
        heap sizes for map and reduce slots
        2013-08-12 00:04:43,986 INFO org.apache.hadoop.mapred.TaskTracker: maptask heapsize: 800
        2013-08-12 00:04:43,986 INFO org.apache.hadoop.mapred.TaskTracker: reducetask heapsize: 1500
        2013-08-12 00:04:43,986 INFO org.apache.hadoop.mapred.TaskTracker: Map slots 24, Default heapsize for map task 800 mb
        2013-08-12 00:04:43,986 INFO org.apache.hadoop.mapred.TaskTracker: Reduce slots 18, Default heapsize for reduce task 1500 mb
        2013-08-12 00:04:43,986 INFO org.apache.hadoop.mapred.TaskTracker: Ephemeral slots 1, memory given for each ephemeral slot 200 mb
        2013-08-12 00:04:43,986 INFO org.apache.hadoop.mapred.TaskTracker: Prefetch map slots 0
        2013-08-12 00:04:44,093 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
        2013-08-12 00:04:44,186 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
        2013-08-12 00:04:44,210 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
        2013-08-12 00:04:44,219 INFO org.apache.hadoop.mapred.TaskTracker: Checking for local volume. If volume is not present command will create and mount it. Command invoke
        d is : /opt/mapr//server/createTTVolume.sh hph05.hadoop /var/mapr/local/hph05.hadoop/mapred/ /var/mapr/local/hph05.hadoop/mapred/taskTracker/
        2013-08-12 00:04:49,359 ERROR org.apache.hadoop.mapred.TaskTracker: Failed to create and mount local mapreduce volume at /var/mapr/local/hph05.hadoop/mapred/. Please s
        ee logs at /opt/mapr//logs/createTTVolume.log
        2013-08-12 00:04:49,360 ERROR org.apache.hadoop.mapred.TaskTracker: Command ran /opt/mapr//server/createTTVolume.sh hph05.hadoop /var/mapr/local/hph05.hadoop/mapred/ /
        var/mapr/local/hph05.hadoop/mapred/taskTracker/
        2013-08-12 00:04:49,360 ERROR org.apache.hadoop.mapred.TaskTracker: Command output
        2013-08-12 00:04:49,361 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start TaskTracker because org.apache.hadoop.util.Shell$ExitCodeException:
                at org.apache.hadoop.util.Shell.runCommand(Shell.java:322)
                at org.apache.hadoop.util.Shell.run(Shell.java:249)
                at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:442)
                at org.apache.hadoop.mapred.TaskTracker.createTTVolume(TaskTracker.java:1879)
                at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:961)
                at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:2176)
                at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:5309)
        
        2013-08-12 00:04:49,363 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
        /************************************************************
        SHUTDOWN_MSG: Shutting down TaskTracker at hph05.hadoop/192.168.0.3
        ************************************************************/
        2013-08-12 00:04:54,553 INFO org.apache.hadoop.mapred.TaskTracker: STARTUP_MSG:
        /************************************************************

Outcomes