AnsweredAssumed Answered

Warden won't start with NoClassDefFoundError

Question asked by bytemedwb on Jul 16, 2013
Latest reply on Jul 17, 2013 by nabeel
All:

I have set up a 5 node cluster as a set of Virtual Machines.
I followed the Mapr 5node setup guide.   However on two of the
machines I can not get the warden to start via the service
command.   However, I can manually start the process by running
the java command directly from the command line.

Below is the wardeninit.log file with output from a modified
start script to echo the actual commands.  The system can not
find the com.sun.security.auth.module.UnixSystem.

<pre>
Wed Jul 17 09:35:52 EDT 2013
Start warden
no jobtracker to stop
Warden Cmd
java -XX:ErrorFile='/opt/cores/hs_err_pid%p.log' -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath='/opt/cores' -XX:+UseConcMarkSweepGC -Dlog.file=/opt/mapr/logs/warden.log -Djava.library.path=/opt/mapr/lib -classpath /opt/mapr:/opt/mapr/conf:/opt/mapr/lib/adminuiapp-0.1.jar:/opt/mapr/lib/ant-1.7.1.jar:/opt/mapr/lib/antlr-2.7.7.jar:/opt/mapr/lib/baseutils-0.1.jar:/opt/mapr/lib/c3p0-0.9.1.2.jar:/opt/mapr/lib/cldb-0.1.jar:/opt/mapr/lib/cliframework-0.1.jar:/opt/mapr/lib/commons-codec-1.5.jar:/opt/mapr/lib/commons-collections-3.2.1.jar:/opt/mapr/lib/commons-el-1.0.jar:/opt/mapr/lib/commons-email-1.2.jar:/opt/mapr/lib/commons-lang-2.5.jar:/opt/mapr/lib/commons-logging-1.0.4.jar:/opt/mapr/lib/commons-logging-api-1.0.4.jar:/opt/mapr/lib/dom4j-1.6.1.jar:/opt/mapr/lib/eval-0.5.jar:/opt/mapr/lib/flexjson-2.1.jar:/opt/mapr/lib/globalfsck-0.1.jar:/opt/mapr/lib/google-collect-1.0.jar:/opt/mapr/lib/gson-2.1.jar:/opt/mapr/lib/hadoop-metrics-0.20.2-dev.jar:/opt/mapr/lib/hadoop-metrics2-0.20.2-dev.jar:/opt/mapr/lib/hibernate-c3p0-3.3.1.GA.jar:/opt/mapr/lib/hibernate-commons-annotations-3.2.0.Final.jar:/opt/mapr/lib/hibernate-core-3.6.8.Final.jar:/opt/mapr/lib/httpclient-4.2.jar:/opt/mapr/lib/httpclient-cache-4.2.jar:/opt/mapr/lib/httpcore-4.2.jar:/opt/mapr/lib/jasper-compiler-5.5.12.jar:/opt/mapr/lib/jasper-runtime-5.5.12.jar:/opt/mapr/lib/javassist-3.12.1.GA.jar:/opt/mapr/lib/jetty-6.1.26.jar:/opt/mapr/lib/jetty-plus-6.1.26.jar:/opt/mapr/lib/jetty-util-6.1.26.jar:/opt/mapr/lib/jobmngmnt-0.1.jar:/opt/mapr/lib/joda-time-2.0.jar:/opt/mapr/lib/JPam-1.1.jar:/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/jsp-2.1.jar:/opt/mapr/lib/jsp-api-2.1.jar:/opt/mapr/lib/jta-1.1.jar:/opt/mapr/lib/junit-3.8.1.jar:/opt/mapr/lib/junit-4.5.jar:/opt/mapr/lib/kvstore-0.1.jar:/opt/mapr/lib/libprotodefs.jar:/opt/mapr/lib/log4j-1.2.14.jar:/opt/mapr/lib/log4j-1.2.15.jar:/opt/mapr/lib/logging-0.1.jar:/opt/mapr/lib/mail.jar:/opt/mapr/lib/maprbuildversion.jar:/opt/mapr/lib/maprcli-0.1.jar:/opt/mapr/lib/maprfs-diagnostic-tools-0.20.2-2.1.3.jar:/opt/mapr/lib/maprfs-jni-0.20.2-2.1.3.jar:/opt/mapr/lib/maprfs-jni-0.20.2-2.1.3-tests.jar:/opt/mapr/lib/maprsecurity-0.1.jar:/opt/mapr/lib/maprutil-0.1.jar:/opt/mapr/lib/persistence-api-1.0.jar:/opt/mapr/lib/protobuf-java-2.4.1-lite.jar:/opt/mapr/lib/servlet-api-2.5-6.1.26.jar:/opt/mapr/lib/volumemirror-0.1.jar:/opt/mapr/lib/warden-0.1.jar:/opt/mapr/lib/zookeeper-3.3.6.jar -Dmapr.library.flatclass -Dcom.sun.management.jmxremote -Dpid=2966 -Dpname=warden -Dmapr.home.dir=/opt/mapr com.mapr.warden.WardenMain /opt/mapr/conf/warden.conf
</pre>

----
su -s /bin/sh -p mapr -c
----
<pre>
Exception in thread "main" java.lang.NoClassDefFoundError: com.mapr.warden.WardenManager
   at java.lang.Class.initializeClass(libgcj.so.10)
   at com.mapr.warden.WardenMain.adjustOOMKillerSettings(WardenMain.java:323)
   at com.mapr.warden.WardenMain.main(WardenMain.java:293)
Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.module.UnixSystem not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/opt/mapr/,file:/opt/mapr/conf/,file:/opt/mapr/lib/adminuiapp-0.1.jar,file:/opt/mapr/lib/ant-1.7.1.jar,file:/opt/mapr/lib/antlr-2.7.7.jar,file:/opt/mapr/lib/baseutils-0.1.jar,file:/opt/mapr/lib/c3p0-0.9.1.2.jar,file:/opt/mapr/lib/cldb-0.1.jar,file:/opt/mapr/lib/cliframework-0.1.jar,file:/opt/mapr/lib/commons-codec-1.5.jar,file:/opt/mapr/lib/commons-collections-3.2.1.jar,file:/opt/mapr/lib/commons-el-1.0.jar,file:/opt/mapr/lib/commons-email-1.2.jar,file:/opt/mapr/lib/commons-lang-2.5.jar,file:/opt/mapr/lib/commons-logging-1.0.4.jar,file:/opt/mapr/lib/commons-logging-api-1.0.4.jar,file:/opt/mapr/lib/dom4j-1.6.1.jar,file:/opt/mapr/lib/eval-0.5.jar,file:/opt/mapr/lib/flexjson-2.1.jar,file:/opt/mapr/lib/globalfsck-0.1.jar,file:/opt/mapr/lib/google-collect-1.0.jar,file:/opt/mapr/lib/gson-2.1.jar,file:/opt/mapr/lib/hadoop-metrics-0.20.2-dev.jar,file:/opt/mapr/lib/hadoop-metrics2-0.20.2-dev.jar,file:/opt/mapr/lib/hibernate-c3p0-3.3.1.GA.jar,file:/opt/mapr/lib/hibernate-commons-annotations-3.2.0.Final.jar,file:/opt/mapr/lib/hibernate-core-3.6.8.Final.jar,file:/opt/mapr/lib/httpclient-4.2.jar,file:/opt/mapr/lib/httpclient-cache-4.2.jar,file:/opt/mapr/lib/httpcore-4.2.jar,file:/opt/mapr/lib/jasper-compiler-5.5.12.jar,file:/opt/mapr/lib/jasper-runtime-5.5.12.jar,file:/opt/mapr/lib/javassist-3.12.1.GA.jar,file:/opt/mapr/lib/jetty-6.1.26.jar,file:/opt/mapr/lib/jetty-plus-6.1.26.jar,file:/opt/mapr/lib/jetty-util-6.1.26.jar,file:/opt/mapr/lib/jobmngmnt-0.1.jar,file:/opt/mapr/lib/joda-time-2.0.jar,file:/opt/mapr/lib/JPam-1.1.jar,file:/opt/mapr/lib/json-20080701.jar,file:/opt/mapr/lib/jsp-2.1.jar,file:/opt/mapr/lib/jsp-api-2.1.jar,file:/opt/mapr/lib/jta-1.1.jar,file:/opt/mapr/lib/junit-3.8.1.jar,file:/opt/mapr/lib/junit-4.5.jar,file:/opt/mapr/lib/kvstore-0.1.jar,file:/opt/mapr/lib/libprotodefs.jar,file:/opt/mapr/lib/log4j-1.2.14.jar,file:/opt/mapr/lib/log4j-1.2.15.jar,file:/opt/mapr/lib/logging-0.1.jar,file:/opt/mapr/lib/mail.jar,file:/opt/mapr/lib/maprbuildversion.jar,file:/opt/mapr/lib/maprcli-0.1.jar,file:/opt/mapr/lib/maprfs-diagnostic-tools-0.20.2-2.1.3.jar,file:/opt/mapr/lib/maprfs-jni-0.20.2-2.1.3.jar,file:/opt/mapr/lib/maprfs-jni-0.20.2-2.1.3-tests.jar,file:/opt/mapr/lib/maprsecurity-0.1.jar,file:/opt/mapr/lib/maprutil-0.1.jar,file:/opt/mapr/lib/persistence-api-1.0.jar,file:/opt/mapr/lib/protobuf-java-2.4.1-lite.jar,file:/opt/mapr/lib/servlet-api-2.5-6.1.26.jar,file:/opt/mapr/lib/volumemirror-0.1.jar,file:/opt/mapr/lib/warden-0.1.jar,file:/opt/mapr/lib/zookeeper-3.3.6.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
   at java.net.URLClassLoader.findClass(libgcj.so.10)
   at java.lang.ClassLoader.loadClass(libgcj.so.10)
   at java.lang.ClassLoader.loadClass(libgcj.so.10)
   at java.lang.Class.forName(libgcj.so.10)
   at java.lang.Class.initializeClass(libgcj.so.10)
   ...2 more
/etc/init.d/mapr-warden: line 224: kill: (3086) - No such process
Warden start failed

Outcomes