AnsweredAssumed Answered

HMaster not able to start

Question asked by pvharish on Sep 20, 2014
Latest reply on Oct 9, 2014 by nabeel
I have installed MapR HBase on my machine using quick installation procedure . All the processes are running and even I am able to access MapR web GUI. But HMaster process is not able to start showing following error in the logs:
<pre>

2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/mapr/hadoop/hadoop-2.4.1/lib/native
2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.13.0-32-generic
2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=mapr
2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/mapr
2014-09-20 20:12:13,661 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/mapr/hbase/hbase-0.98.4/bin
2014-09-20 20:12:13,663 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=node1:5181 sessionTimeout=180000 watcher=master:60000
2014-09-20 20:12:13,692 INFO  [main] zookeeper.RecoverableZooKeeper: The identifier of this process is 15332@node1
2014-09-20 20:12:13,805 INFO  [main-SendThread(node1.pvh.com:5181)] zookeeper.Login: successfully logged in.
2014-09-20 20:12:13,809 INFO  [main-SendThread(node1.pvh.com:5181)] client.ZooKeeperSaslClient: Client will use SIMPLE-SECURITY as SASL mechanism.
2014-09-20 20:12:13,821 INFO  [main-SendThread(node1.pvh.com:5181)] zookeeper.ClientCnxn: Opening socket connection to server node1.pvh.com/192.168.0.111:5181. Will attempt to SASL-authenticate using Login Context section 'Client_simple'
2014-09-20 20:12:13,828 INFO  [main-SendThread(node1.pvh.com:5181)] zookeeper.ClientCnxn: Socket connection established to node1.pvh.com/192.168.0.111:5181, initiating session
2014-09-20 20:12:13,855 INFO  [main-SendThread(node1.pvh.com:5181)] zookeeper.ClientCnxn: Session establishment complete on server node1.pvh.com/192.168.0.111:5181, sessionid = 0x14892e556ab0078, negotiated timeout = 40000
2014-09-20 20:12:13,893 INFO  [IPC Server listener on 60000] ipc.HBaseServer: IPC Server listener on 60000: starting
2014-09-20 20:12:13,895 INFO  [IPC Server Responder] ipc.HBaseServer: IPC Server Responder: starting
2014-09-20 20:12:13,916 INFO  [IPC Server handler 0 on 60000] ipc.HBaseServer: IPC Server handler 0 on 60000: starting
2014-09-20 20:12:13,917 INFO  [IPC Server handler 2 on 60000] ipc.HBaseServer: IPC Server handler 2 on 60000: starting
2014-09-20 20:12:13,917 INFO  [IPC Server handler 1 on 60000] ipc.HBaseServer: IPC Server handler 1 on 60000: starting
2014-09-20 20:12:13,917 INFO  [IPC Server handler 4 on 60000] ipc.HBaseServer: IPC Server handler 4 on 60000: starting
2014-09-20 20:12:13,917 INFO  [IPC Server handler 3 on 60000] ipc.HBaseServer: IPC Server handler 3 on 60000: starting
2014-09-20 20:12:13,918 INFO  [IPC Server handler 6 on 60000] ipc.HBaseServer: IPC Server handler 6 on 60000: starting
2014-09-20 20:12:13,918 INFO  [IPC Server handler 5 on 60000] ipc.HBaseServer: IPC Server handler 5 on 60000: starting
2014-09-20 20:12:13,918 INFO  [IPC Server handler 8 on 60000] ipc.HBaseServer: IPC Server handler 8 on 60000: starting
2014-09-20 20:12:13,918 INFO  [IPC Server handler 7 on 60000] ipc.HBaseServer: IPC Server handler 7 on 60000: starting
2014-09-20 20:12:13,919 INFO  [IPC Server handler 10 on 60000] ipc.HBaseServer: IPC Server handler 10 on 60000: starting
2014-09-20 20:12:13,919 INFO  [IPC Server handler 9 on 60000] ipc.HBaseServer: IPC Server handler 9 on 60000: starting
2014-09-20 20:12:13,919 INFO  [IPC Server handler 11 on 60000] ipc.HBaseServer: IPC Server handler 11 on 60000: starting
2014-09-20 20:12:13,923 INFO  [IPC Server handler 12 on 60000] ipc.HBaseServer: IPC Server handler 12 on 60000: starting
2014-09-20 20:12:13,925 INFO  [IPC Server handler 14 on 60000] ipc.HBaseServer: IPC Server handler 14 on 60000: starting
2014-09-20 20:12:13,925 INFO  [IPC Server handler 13 on 60000] ipc.HBaseServer: IPC Server handler 13 on 60000: starting
2014-09-20 20:12:13,926 INFO  [IPC Server handler 15 on 60000] ipc.HBaseServer: IPC Server handler 15 on 60000: starting
2014-09-20 20:12:13,926 INFO  [IPC Server handler 16 on 60000] ipc.HBaseServer: IPC Server handler 16 on 60000: starting
2014-09-20 20:12:13,929 INFO  [IPC Server handler 17 on 60000] ipc.HBaseServer: IPC Server handler 17 on 60000: starting
2014-09-20 20:12:13,931 INFO  [IPC Server handler 18 on 60000] ipc.HBaseServer: IPC Server handler 18 on 60000: starting
2014-09-20 20:12:13,931 INFO  [IPC Server handler 19 on 60000] ipc.HBaseServer: IPC Server handler 19 on 60000: starting
2014-09-20 20:12:13,935 INFO  [IPC Server handler 20 on 60000] ipc.HBaseServer: IPC Server handler 20 on 60000: starting
2014-09-20 20:12:13,936 INFO  [IPC Server handler 21 on 60000] ipc.HBaseServer: IPC Server handler 21 on 60000: starting
2014-09-20 20:12:13,936 INFO  [IPC Server handler 22 on 60000] ipc.HBaseServer: IPC Server handler 22 on 60000: starting
2014-09-20 20:12:13,936 INFO  [IPC Server handler 23 on 60000] ipc.HBaseServer: IPC Server handler 23 on 60000: starting
2014-09-20 20:12:13,936 INFO  [IPC Server handler 24 on 60000] ipc.HBaseServer: IPC Server handler 24 on 60000: starting
2014-09-20 20:12:13,936 INFO  [IPC Server handler 25 on 60000] ipc.HBaseServer: IPC Server handler 25 on 60000: starting
2014-09-20 20:12:13,936 INFO  [IPC Server handler 26 on 60000] ipc.HBaseServer: IPC Server handler 26 on 60000: starting
2014-09-20 20:12:13,937 INFO  [IPC Server handler 27 on 60000] ipc.HBaseServer: IPC Server handler 27 on 60000: starting
2014-09-20 20:12:13,937 INFO  [IPC Server handler 28 on 60000] ipc.HBaseServer: IPC Server handler 28 on 60000: starting
2014-09-20 20:12:13,937 INFO  [IPC Server handler 29 on 60000] ipc.HBaseServer: IPC Server handler 29 on 60000: starting
2014-09-20 20:12:13,941 INFO  [main] jvm.JvmMetrics: Initializing JVM Metrics with processName=Master, sessionId=node1.pvh.com,60000,1411224133512
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: hdfsRevision
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: version
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: user
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: date
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: url
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: hdfsVersion
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: revision
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: hdfsUrl
2014-09-20 20:12:13,949 INFO  [main] hbase.metrics: MetricsString added: hdfsDate
2014-09-20 20:12:13,950 INFO  [main] hbase.metrics: MetricsString added: hdfsUser
2014-09-20 20:12:13,950 INFO  [main] hbase.metrics: new MBeanInfo
2014-09-20 20:12:13,951 INFO  [main] hbase.metrics: new MBeanInfo
2014-09-20 20:12:13,951 INFO  [main] metrics.MasterMetrics: Initialized
2014-09-20 20:12:13,983 INFO  [master-node1.pvh.com,60000,1411224133512] master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/node1.pvh.com,60000,1411224133512 from backup master directory
2014-09-20 20:12:14,004 WARN  [master-node1.pvh.com,60000,1411224133512] zookeeper.RecoverableZooKeeper: Node /hbase/backup-masters/node1.pvh.com,60000,1411224133512 already deleted, and this is not a retry
2014-09-20 20:12:14,004 INFO  [master-node1.pvh.com,60000,1411224133512] master.ActiveMasterManager: Master=node1.pvh.com,60000,1411224133512
2014-09-20 20:12:14,241 INFO  [master-node1.pvh.com,60000,1411224133512] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2014-09-20 20:12:14,251 INFO  [master-node1.pvh.com,60000,1411224133512] master.SplitLogManager: found 0 orphan tasks and 0 rescan nodes
2014-09-20 20:12:14,290 WARN  [master-node1.pvh.com,60000,1411224133512] util.FSUtils: Cluster ID file maprfs:/hbase/hbase.id was empty
2014-09-20 20:12:14,394 INFO  [master-node1.pvh.com,60000,1411224133512] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-09-20 20:12:14,735 FATAL [master-node1.pvh.com,60000,1411224133512] master.HMaster: Unhandled exception. Starting shutdown.
**java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.toBytes(Bytes.java:426)
at org.apache.hadoop.hbase.zookeeper.ClusterId.setClusterId(ClusterId.java:72)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:490)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344)
at java.lang.Thread.run(Thread.java:745)**
2014-09-20 20:12:14,739 INFO  [master-node1.pvh.com,60000,1411224133512] master.HMaster: Aborting
2014-09-20 20:12:14,739 INFO  [master-node1.pvh.com,60000,1411224133512] ipc.HBaseServer: Stopping server on 60000
2014-09-20 20:12:14,740 INFO  [IPC Server handler 1 on 60000] ipc.HBaseServer: IPC Server handler 1 on 60000: exiting
2014-09-20 20:12:14,740 INFO  [IPC Server listener on 60000] ipc.HBaseServer: Stopping IPC Server listener on 60000
2014-09-20 20:12:14,741 INFO  [IPC Server handler 11 on 60000] ipc.HBaseServer: IPC Server handler 11 on 60000: exiting
2014-09-20 20:12:14,740 INFO  [IPC Server handler 2 on 60000] ipc.HBaseServer: IPC Server handler 2 on 60000: exiting
2014-09-20 20:12:14,741 INFO  [IPC Server handler 3 on 60000] ipc.HBaseServer: IPC Server handler 3 on 60000: exiting
2014-09-20 20:12:14,741 INFO  [IPC Server handler 12 on 60000] ipc.HBaseServer: IPC Server handler 12 on 60000: exiting
2014-09-20 20:12:14,741 INFO  [IPC Server handler 7 on 60000] ipc.HBaseServer: IPC Server handler 7 on 60000: exiting
2014-09-20 20:12:14,742 INFO  [IPC Server handler 14 on 60000] ipc.HBaseServer: IPC Server handler 14 on 60000: exiting
2014-09-20 20:12:14,742 INFO  [IPC Server handler 9 on 60000] ipc.HBaseServer: IPC Server handler 9 on 60000: exiting
2014-09-20 20:12:14,742 INFO  [IPC Server handler 16 on 60000] ipc.HBaseServer: IPC Server handler 16 on 60000: exiting
2014-09-20 20:12:14,742 INFO  [IPC Server handler 13 on 60000] ipc.HBaseServer: IPC Server handler 13 on 60000: exiting
2014-09-20 20:12:14,742 INFO  [IPC Server handler 22 on 60000] ipc.HBaseServer: IPC Server handler 22 on 60000: exiting
2014-09-20 20:12:14,743 INFO  [IPC Server handler 25 on 60000] ipc.HBaseServer: IPC Server handler 25 on 60000: exiting
2014-09-20 20:12:14,743 INFO  [IPC Server handler 15 on 60000] ipc.HBaseServer: IPC Server handler 15 on 60000: exiting
2014-09-20 20:12:14,743 INFO  [IPC Server handler 24 on 60000] ipc.HBaseServer: IPC Server handler 24 on 60000: exiting
2014-09-20 20:12:14,743 INFO  [IPC Server handler 19 on 60000] ipc.HBaseServer: IPC Server handler 19 on 60000: exiting
2014-09-20 20:12:14,743 INFO  [IPC Server handler 21 on 60000] ipc.HBaseServer: IPC Server handler 21 on 60000: exiting
2014-09-20 20:12:14,740 INFO  [IPC Server handler 0 on 60000] ipc.HBaseServer: IPC Server handler 0 on 60000: exiting
2014-09-20 20:12:14,748 INFO  [IPC Server handler 29 on 60000] ipc.HBaseServer: IPC Server handler 29 on 60000: exiting
2014-09-20 20:12:14,741 INFO  [IPC Server handler 10 on 60000] ipc.HBaseServer: IPC Server handler 10 on 60000: exiting
2014-09-20 20:12:14,740 INFO  [IPC Server handler 8 on 60000] ipc.HBaseServer: IPC Server handler 8 on 60000: exiting
2014-09-20 20:12:14,748 INFO  [IPC Server handler 28 on 60000] ipc.HBaseServer: IPC Server handler 28 on 60000: exiting
2014-09-20 20:12:14,740 INFO  [IPC Server handler 18 on 60000] ipc.HBaseServer: IPC Server handler 18 on 60000: exiting
2014-09-20 20:12:14,740 INFO  [IPC Server handler 6 on 60000] ipc.HBaseServer: IPC Server handler 6 on 60000: exiting
2014-09-20 20:12:14,740 INFO  [IPC Server handler 4 on 60000] ipc.HBaseServer: IPC Server handler 4 on 60000: exiting
2014-09-20 20:12:14,749 INFO  [IPC Server Responder] ipc.HBaseServer: Stopping IPC Server Responder
2014-09-20 20:12:14,749 INFO  [IPC Server Responder] ipc.HBaseServer: Stopping IPC Server Responder
2014-09-20 20:12:14,748 INFO  [IPC Server handler 26 on 60000] ipc.HBaseServer: IPC Server handler 26 on 60000: exiting
2014-09-20 20:12:14,748 INFO  [IPC Server handler 23 on 60000] ipc.HBaseServer: IPC Server handler 23 on 60000: exiting
2014-09-20 20:12:14,743 INFO  [IPC Server handler 27 on 60000] ipc.HBaseServer: IPC Server handler 27 on 60000: exiting
2014-09-20 20:12:14,743 INFO  [IPC Server handler 17 on 60000] ipc.HBaseServer: IPC Server handler 17 on 60000: exiting
2014-09-20 20:12:14,741 INFO  [IPC Server handler 20 on 60000] ipc.HBaseServer: IPC Server handler 20 on 60000: exiting
2014-09-20 20:12:14,741 INFO  [IPC Server handler 5 on 60000] ipc.HBaseServer: IPC Server handler 5 on 60000: exiting
2014-09-20 20:12:14,766 INFO  [node1.pvh.com,60000,1411224133512.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: node1.pvh.com,60000,1411224133512.splitLogManagerTimeoutMonitor exiting
2014-09-20 20:12:14,776 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2014-09-20 20:12:14,776 INFO  [master-node1.pvh.com,60000,1411224133512] zookeeper.ZooKeeper: Session: 0x14892e556ab0078 closed
2014-09-20 20:12:14,776 INFO  [master-node1.pvh.com,60000,1411224133512] master.HMaster: HMaster main thread exiting
2014-09-20 20:12:14,776 ERROR [main] master.HMasterCommandLine: Failed to start master
**java.lang.RuntimeException: HMaster Aborted
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:154)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1802)
Sat Sep 20 20:12:21 IST 2014 Waiting for filesystem to come up***
</pre>
please guide me with the issue.

Outcomes