AnsweredAssumed Answered

Upgrade issue Hive 0.12 to 0.13

Question asked by yatrus_analytics on Mar 18, 2015
Hi all,

I am running a three nodes cluster with installed M3 edition with YARN cluster manager on Ubuntu 14.04. I have installed and configured Hive 0.12 and it is working properly. I have tried to upgrade to 0.13 but I have faced the following problem. The upgrade seems to be accomplished without errors including the upgrade metastore schema. During this procedure the instance on which Hive is installed was stopped. I decided to use the same configuration file (hive-site.xml) as in Hive 0.12 in order to be sure that everything is consistent. So after accomplishing the upgrade procedure I started the instance. In the logs of Metastore and Hiveserver2 it seems that they are running correctly but according to the information in MCS console they fail after approximately 30 seconds. Nothing unusual in the Metastore and HS2 logs after failure. After that tried a fresh installation of Hive with Metastore and Hiveserver2 and the whole procedure with creating new mysql metastore table and user, giving him all the rights over metastore database but the same problem occured.

I have found something about this problem in the logs of Warden. It seems that it is running monitoring command and after the command execution it can not decide whether Metastore is running. After that Warden starts shutting down the processes which causes the failure in my opinion. Warden runs the script /opt/mapr/hive/hive-0.13/bin/status in order to obtain status information but it seems that this script does not provide enough information about the status of Hivemeta and HS2. How could I solve this problem.

My hive-site.xml file is as follows:

    <configuration>
    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/user/hive/warehouse</value>
        <description>Metastore directory to save tables</description>
    </property>
    
     <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>
        <description>JDBC connect string for a JDBC metastore</description>
    </property>
    
     <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>Driver class name for a JDBC metastore</description>
     </property>
    
     <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>mapr</value>
        <description>username to use against metastore database</description>
     </property>
    
     <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>mapr</value>
        <description>password to use against metastore database</description>
     </property>
    
    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://localhost:9083</value>
     </property>
    
    <property>
    <name>datanucleus.autoCreateTables</name>
    <value>True</value>
    </property>
    
    <property>
    <name>hive.server2.authentication</name>
    <value>NOSASL</value>
    </property>
    
    <property>
      <name>hive.aux.jars.path</name>
      <value>file:///opt/mapr/hive/hive-0.12/lib/hive-hbase-handler-0.12-mapr-1501.jar,file:///opt/mapr/hbase/hbase-0.94.24/hbase-0.94.24-mapr-1501.jar,file:///opt/mapr/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5-mapr-1406.jar</value>
      <description>A comma separated list (with no spaces) of the jar files required for Hive-HBase integration</description>
    </property>
    
    <property>
      <name>hbase.zookeeper.quorum</name>
      <value>ec2-54-93-121-145.eu-central-1.compute.amazonaws.com</value>
      <description>A comma separated list (with no spaces) of the IP addresses of all ZooKeeper servers in the cluster.</description>
    </property>
    
    <property>
      <name>hbase.zookeeper.property.clientPort</name>
      <value>5181</value>
      <description>The Zookeeper client port. The MapR default clientPort is 5181.</description>
    </property>
    
    <property>
      <name>oozie.service.WorkflowAppService.system.libpath</name>
      <value>/opt/mapr/oozie/oozie-4.0.1/lib</value>
    </property>
    
    </configuration>

Just to note that I have corrected the hive.aux.jars.path after installing hive-0.13 as appropriate so this should not be a problem.

All suggestions are welcome!

Outcomes