AnsweredAssumed Answered

Jobtacker and Tasktracker wont start after upgrade from v1 to v3

Question asked by dimamah on Oct 15, 2013
Latest reply on Oct 15, 2013 by dimamah
After upgrading from v1.2.9 to v3.0.1 the jobtracker and tasktrack are unable to start. 
Tasktracker is stuck in the `createTTVolume` part where createTTVolume.0.log says : 

    ............
    2013-10-15 16:37:10 DEBUG Command attempt 8 failed to return within 60 seconds, it was killed and returned code 137, exiting
    2013-10-15 16:37:10 DEBUG Launching "hadoop fs -mkdir /var/mapr/local/hadoop-1/mapred/taskTracker/"
    2013-10-15 16:38:10 DEBUG Command did not complete within 60 seconds, issuing "kill -9 16061"
    2013-10-15 16:38:10 DEBUG Successfully killed process

Tried to run the `mkdir` comman manualy and its hanging. 
When running like this : `HADOOP_ROOT_LOGGER=DEBUG,console hadoop fs -mkdir /var/mapr/local/hadoop-1/mapred/taskTracker/`

The output is : 

    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.maprfs.impl.disable.cache
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.spoof.user
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.security.authentication
    13/10/15 16:59:14 DEBUG security.Groups:  Creating new Groups object
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.security.group.mapping
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.native.lib
    13/10/15 16:59:14 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
    13/10/15 16:59:14 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    13/10/15 16:59:14 INFO security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
    13/10/15 16:59:14 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.security.groups.cache.secs
    13/10/15 16:59:14 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.security.auth_to_local
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.maprfs.impl
    13/10/15 16:59:14 DEBUG fs.FileSystem: Creating filesystem for maprfs:///
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.block.size
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: dfs.block.size
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.disable.namecache
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.aggregate.writes
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.threads
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.shmpool.size
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.trace
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.slowops.threshold
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.io.remoteonly
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.rpc.timeout
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.spoofed.user.username
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.spoofed.user.uid
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.spoofed.user.groupname
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: hadoop.spoofed.user.gid
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: db.mapr.putbuffer.threshold.mb
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: db.mapr.putbuffer.threshold.sec
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: db.mapr.putbuffer.size
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.mapr.readbuffering
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.automatic.close
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.maprfs.impl.disable.cache
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.trash.interval
    13/10/15 16:59:14 DEBUG conf.MapRConf: Fetching MapR default for: fs.maprfs.impl.disable.cache
    13/10/15 17:01:04 DEBUG conf.MapRConf: Fetching MapR default for: dfs.umask
    13/10/15 17:01:04 DEBUG conf.MapRConf: Fetching MapR default for: dfs.umaskmode 
and hanging...

As for the Jobtracker the end of the log is :

    2013-10-15 16:46:02,282 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
    2013-10-15 16:46:02,283 INFO org.mortbay.log: jetty-6.1.26
    2013-10-15 16:46:02,710 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
    2013-10-15 16:46:02,713 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
    2013-10-15 16:46:02,715 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 9001
    2013-10-15 16:46:02,715 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
    2013-10-15 16:46:02,796 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory


Both seem to be some MFS related actions. 
tried reading data from MFS : 
`hadoop fs -text /user/hive/warehouse/somedb.db/sometable/somefile` 
Worked fine.

Writing also seems to work even to /var : 
`touch a.a` 
`hadoop fs -copyFromLocal a.a /var/mapr/local/hadoop-1/mapred/` 
`hadoop fs -ls /var/mapr/local/hadoop-1/mapred/` 

`-rwxr-xr-x   1 root root          0 2013-10-15 17:39 /var/mapr/local/hadoop-1/mapred/a.a`


`hadoop fs -ls /` output is : 

    drwxrwxrwx   - root root          5 2013-10-15 14:00 /hbase
    drwxr-xr-x   - mapr mapr          1 2013-10-15 15:51 /tmp
    drwxr-xr-x   - mapr mapr          1 2013-10-15 14:15 /user
    drwxr-xr-x   - root root          1 2013-10-15 12:28 /var


Outcomes