AnsweredAssumed Answered

Hive Metastore service dies after 30 seconds; no explanatory log messages in /opt/mapr/hive/hive-0.13/logs/mapr/hive.log

Question asked by alexwoolford on Feb 19, 2015
About 30 seconds after starting the Hive Metastore service in the MCS, the status shows that the service is running. After a further 30 seconds, the service stops and the MCS describes the state of the Hive Metastore service as "failed".

The MCS console points to the `/opt/mapr/hive/hive-0.13/logs/mapr` directory for logging information. This is what I see in `hive.log`:

    2015-02-19 20:43:00,161 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(5056)) - Starting hive metastore on port 9083
    2015-02-19 20:43:00,256 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(493)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    2015-02-19 20:43:00,280 INFO  [main]: metastore.ObjectStore (ObjectStore.java:initialize(248)) - ObjectStore, initialize called
    2015-02-19 20:43:00,970 INFO  [main]: metastore.ObjectStore (ObjectStore.java:getPMF(317)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
    2015-02-19 20:43:01,897 INFO  [main]: metastore.ObjectStore (ObjectStore.java:setConf(231)) - Initialized ObjectStore
    2015-02-19 20:43:02,083 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles(551)) - Added admin role in metastore
    2015-02-19 20:43:02,085 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles(560)) - Added public role in metastore
    2015-02-19 20:43:02,138 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:addAdminUsers(588)) - No user is added in admin role, since config is empty
    2015-02-19 20:43:02,255 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(5173)) - Starting DB backed MetaStore Server with SetUGI enabled
    2015-02-19 20:43:02,257 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(5190)) - Started the new metaserver on port [9083]...
    2015-02-19 20:43:02,257 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(5192)) - Options.minWorkerThreads = 200
    2015-02-19 20:43:02,257 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(5194)) - Options.maxWorkerThreads = 100000
    2015-02-19 20:43:02,257 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(5196)) - TCP keepalive = true

There's no further information to say what went wrong. In case it's relevant, I uploaded `hive-site.xml` to https://gist.github.com/alexwoolford/cd9c793a98f6ac05ec3d , uploaded the results of `hadoop conf-details print-all-effective-properties` to https://gist.github.com/alexwoolford/9d02957f2c0ce77a7dac , and `hive -e 'set;'` to https://gist.github.com/alexwoolford/92a35b98d6b96ed16647 .

Do you have any suggestions for how I might troubleshoot my dodgy metastore/hs2?

Update: Strangely, the Hive Metastore and HiveServer2 processes appear to be running:

    awoolford@hadoop01:~$ ps -aux | grep hive
    mapr      2659  1.9  1.1 1155440 275144 ?      Sl   15:03   0:06 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xmx256m -Dhiveserver2.auth=NONE -Dhadoop.login=simple -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.clientconfig=Client_simple -Dzookeeper.saslprovider=com.mapr.security.simplesasl.SimpleSaslProvider -Dmapr.library.flatclass -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/opt/mapr/hadoop/hadoop-2.5.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/mapr/hadoop/hadoop-2.5.1 -Dhadoop.id.str=mapr -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/mapr/hadoop/hadoop-2.5.1/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dhadoop.security.logger=INFO,NullAppender -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.clientconfig=Client_simple -Dzookeeper.saslprovider=com.mapr.security.simplesasl.SimpleSaslProvider org.apache.hadoop.util.RunJar /opt/mapr/hive/hive-0.13/lib/hive-service-0.13.0-mapr-1501.jar org.apache.hadoop.hive.metastore.HiveMetaStore
    mapr      2664  1.1  0.9 1190032 223980 ?      Sl   15:03   0:03 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Xmx256m -Dhiveserver2.auth=NONE -Dhadoop.login=simple -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.clientconfig=Client_simple -Dzookeeper.saslprovider=com.mapr.security.simplesasl.SimpleSaslProvider -Dmapr.library.flatclass -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/opt/mapr/hadoop/hadoop-2.5.1/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/mapr/hadoop/hadoop-2.5.1 -Dhadoop.id.str=mapr -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/mapr/hadoop/hadoop-2.5.1/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx512m -Dhadoop.security.logger=INFO,NullAppender -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf -Dzookeeper.sasl.clientconfig=Client_simple -Dzookeeper.saslprovider=com.mapr.security.simplesasl.SimpleSaslProvider org.apache.hadoop.util.RunJar /opt/mapr/hive/hive-0.13/lib/hive-service-0.13.0-mapr-1501.jar org.apache.hive.service.server.HiveServer2

I can see Hive metadata appear in the MySQL database when I create tables despite the apparent failed status in the MCS:

![MCS show hivemeta and hs2 as having failed][1]


  [1]: /storage/temp/132-hivemeta-hs2-failed.png

Outcomes