Reading hive table blocks creating tables in the same database when hive.support.concurrency=true.

Document created by Hao Zhu Employee on Feb 18, 2016
Version 1Show Document
  • View in full screen mode

Author: Hao Zhu

Original Publication Date: February 25, 2015

Symptom:

Reading table blocks creating tables in the same database when hive.support.concurrency=true.Creating table will fail with below errors due to database level lock.

Env:

Hive 0.13

Root Cause:

By default in Hive 0.13 when hive.support.concurrency=true, it is using

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager 
hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager

Some DDLs needs Exclusive lock on database using this transaction manager.This issue is reported in HIVE-9199 which is fixed in Hive 1.1.

Minimum reproduce:

1. Set hive.support.concurrency=true in hive-site.xml.2. Open 2 hive sessions.--Session A runs a long running query:

 

create table passwords (col0 string, col1 string, col2 string, col3 string, col4 string,

col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ":";

load data local inpath "/etc/passwd" into table passwords;

 

select count(*) from passwords a,passwords b, passwords c, passwords d

where a.col0=b.col0

and a.col0=c.col0

and a.col0=d.col0;

 

--Session B tries to create a table in the same database in the meantime:

hive> create table test(id int); 
conflicting lock present for default mode EXCLUSIVE

Solution:

This is current lock behavior in Hive 0.13. You can disable this feature by setting hive.support.concurrency=false in hive-site.xml.

Or upgrade to Hive 0.14 or newer version in which new lock manager DbLockManager and new transaction manager DbTxnManager are introduced.

Attachments

    Outcomes