AnsweredAssumed Answered

Tasktracker and jobtracker all down after upgrade - NoSuchMethodException setMaprClntLibLoaded

Question asked by gaw on Feb 1, 2016
Latest reply on May 2, 2016 by gaw
- Mapr 5.0.0.32987.GA-1 (recently upgraded from Mapr 4.1) w/ M3 license, Hadoop 2.7.0
 - 5-nodes cluster, of which 3 are ZK, of which 2 are tasktracker

All tasktrackers AND jobtrackers are down, according to MCS, and can't seem to come up.
Not so many google hits for the java error, any hints please ? All tasktracker and jobtracker logfiles each only contain the following stacktrace. Further down is a sample of warden logfile.


`/opt/mapr/hadoop/hadoop-0.20.2/logs/hadoop-mapr-tasktracker-FWPC-80306-U.out`

`/opt/mapr/hadoop/hadoop-0.20.2/logs/hadoop-mapr-jobtracker-FWPC-80306-U.out`

    jobtracker started, pid 11472
    java.lang.NoSuchMethodException: java.lang.System.setMaprClntLibLoaded()
           at java.lang.Class.getDeclaredMethod(Class.java:2009)
           at com.mapr.fs.ShimLoader.loadNativeLibrary(ShimLoader.java:336)
           at com.mapr.fs.ShimLoader.load(ShimLoader.java:225)
           at org.apache.hadoop.conf.CoreDefaultProperties.<clinit>(CoreDefaultProperties.java:61)
           at java.lang.Class.forName0(Native Method)
           at java.lang.Class.forName(Class.java:274)
           at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2147)
           at org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2362)
           at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2579)
           at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2531)
           at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2444)
           at org.apache.hadoop.conf.Configuration.get(Configuration.java:994)
           at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1044)
           at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1264)
           at org.apache.hadoop.mapreduce.counters.Limits.<clinit>(Limits.java:29)
           at org.apache.hadoop.mapred.Counters.<clinit>(Counters.java:71)
           at org.apache.hadoop.mapred.JobTracker.<clinit>(JobTracker.java:5060)
    Exception in thread "main" java.lang.ExceptionInInitializerError
          at com.mapr.fs.ShimLoader.load(ShimLoader.java:233)
          at org.apache.hadoop.conf.CoreDefaultProperties.<clinit>(CoreDefaultProperties.java:61)
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:274)
          at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2147)
          at org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2362)
          at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2579)
          at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2531)
          at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2444)
          at org.apache.hadoop.conf.Configuration.get(Configuration.java:994)
          at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1044)
          at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1264)
          at org.apache.hadoop.mapreduce.counters.Limits.<clinit>(Limits.java:29)
          at org.apache.hadoop.mapred.Counters.<clinit>(Counters.java:71)
          at org.apache.hadoop.mapred.JobTracker.<clinit>(JobTracker.java:5060)
    Caused by: java.lang.NoSuchMethodException: java.lang.System.setMaprClntLibLoaded()
          at java.lang.Class.getDeclaredMethod(Class.java:2009)
          at com.mapr.fs.ShimLoader.loadNativeLibrary(ShimLoader.java:336)
          at com.mapr.fs.ShimLoader.load(ShimLoader.java:225)
          ... 14 more

`/opt/mapr/logs/warden.log (grep tracker -i)`

    2016-02-01 16:25:40,382 INFO  com.mapr.warden.WardenManager [main]: Configured services: [resourcemanager:1:cldb, httpfs:1, nodemanager:all:resourcemanager, historyserver:1:resourcemanager, webserver:all:cldb, jobtracker:1:cldb, tasktracker:all:jobtracker, nfs:all:cldb, kvstore:all, cldb:all:kvstore, hoststats:all:kvstore]
    { [...] service.command.jt.monitor=org.apache.hadoop.mapred.JobTracker, service.command.jt.stop=/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh stop jobtracker, service.command.jt.monitorcommand=/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh status jobtracker, service.command.tt.start=/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh start tasktracker, service.command.tt.monitor=org.apache.hadoop.mapred.TaskTracker, service.command.tt.stop=/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh stop tasktracker,  services=webserver:all:cldb;jobtracker:1:cldb;tasktracker:all:jobtracker;nfs:all:cldb;service.nice.value=-10, service.command.jt.start=/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh start jobtracker, service.command.tt.monitorcommand=/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh status tasktracker [...] }
    2016-02-01 16:26:26,235 INFO  com.mapr.warden.service.baseservice.Service [main-EventThread]: -------------Service is starting for: tasktracker
    2016-02-01 16:26:26,235 INFO  com.mapr.warden.service.baseservice.Service$ServiceMonitorRun [main-EventThread]: Command: [/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh, status, tasktracker], Directory: /opt/mapr/hadoop/hadoop-0.20.2/bin/
    2016-02-01 16:26:26,247 INFO  com.mapr.warden.service.baseservice.Service [main-EventThread]: Process path: /services/tasktracker/epimethee. Event state: SyncConnected. Event type: NodeCreated
    2016-02-01 16:26:26,247 INFO  com.mapr.warden.service.baseservice.Service [main-EventThread]: Thread: 42, NodeCreated: /services/tasktracker/epimethee
    2016-02-01 16:26:26,248 INFO  com.mapr.warden.service.baseservice.Service [main-EventThread]: Process path: /services/tasktracker. Event state: SyncConnected. Event type: NodeChildrenChanged
    2016-02-01 16:26:26,325 INFO  com.mapr.warden.service.baseservice.Service [tasktracker_monitor]: Alarm clearing command: [/opt/mapr/bin/maprcli, alarm, clear, -alarm, NODE_ALARM_SERVICE_TT_DOWN, -entity, epimethee]
    2016-02-01 16:26:29,495 INFO  com.mapr.warden.service.baseservice.Service$ServiceRun [tasktracker_monitor]: Command: [/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh, start, tasktracker], Directory: /opt/mapr/hadoop/hadoop-0.20.2/bin
    2016-02-01 16:26:29,496 INFO  com.mapr.warden.WardenMapRed [tasktracker_monitor]: Before adjustment, memoryForTasks = 475, memory for service = 237.5
    2016-02-01 16:26:34,966 INFO  com.mapr.warden.service.TTService [tasktracker_monitor]: Set HADOOP_TASKTRACKER_OPTS to: -Dhadoop.login=maprsasl -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf  -Dzookeeper.sasl.clientconfig=Client_simple -Dzookeeper.saslprovider=com.mapr.security.simplesasl.SimpleSaslProvider -Dtt.tasks.mem=237 -Dtt.tasks.cpu=0 -Dtt.tasks.io-spindles=0.5
    2016-02-01 16:26:36,946 INFO  com.mapr.warden.service.baseservice.Service$ServiceRun [tasktracker_monitor]: starting tasktracker, logging to /opt/mapr/hadoop/hadoop-0.20.2/bin/../logs/hadoop-mapr-tasktracker-FWPC-80306-U.out
    2016-02-01 16:26:46,949 INFO  com.mapr.job.mngmnt.hadoop.metrics.WardenRequestBuilder [tasktracker_monitor]: [e_SERV_RUN, hostName, ma_host, ma_process]
    2016-02-01 16:26:46,949 INFO  com.mapr.job.mngmnt.hadoop.metrics.WardenRequestBuilder [tasktracker_monitor]: []
    2016-02-01 16:26:47,032 INFO  com.mapr.warden.service.baseservice.Service [main-EventThread]: Process path: /services/tasktracker/epimethee. Event state: SyncConnected. Event type: NodeDataChanged
    2016-02-01 16:26:47,455 ERROR com.mapr.warden.service.baseservice.Service$ServiceMonitorRun run [tasktracker_monitor]: Monitor command: [/opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh, status, tasktracker]can not determine if service: tasktracker is running. Retrying. Retrial #1. Total retries count is: 3
    2016-02-01 16:26:47,455 ERROR com.mapr.warden.service.baseservice.Service$ServiceMonitorRun run [tasktracker_monitor]: /opt/mapr/hadoop/hadoop-0.20.2/bin/hadoop-daemon.sh: line 207: kill: (11129) - No such process




Outcomes