AnsweredAssumed Answered

One secure (also kerberized enabled) cluster + 2 hive setups

Question asked by midair77 on Apr 12, 2016
Latest reply on May 2, 2016 by mufeed


Hi all,

 

I have a secure MapR 5.1 that is secure and also kerberized enabled cluster.  In my first hive server,  hive metastore and hive server2 both authenticate with MapR-SASL.  This setup works.

 

I would like to set up a second hive server where hive metastore and hive server2 will both authenticate using Kerberos.  My hive servers will each have their own mysql servers, warehouse dirs on MapR-FS, and scratch dirs.  Basically, they are independent Hive servers that will not know anything about the other server.  The only thing that they both share is the MapR-FS setup (not the dirs they use) and they will use different authentication methods.

 

Is this setup even possible where MapR-SASL and Kerberos being used in parallel for two different Hive servers?  Does zookeeper in this setup accept MapR-SASL and Kerberos authentication methods?

 

I already have the Mapr-SASL hive metastore and server2 worked but I have run into issues that I can not find answer for.

 

I saw these in the log of zookeeper leader server:

2016-04-12 16:24:59,017 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00000 type:create cxid:0x4 zxid:0x400000009 txntype:-1 reqpath:n/a Error Path:/servers Error:KeeperErrorCode = NodeExists for /servers
2016-04-12 16:24:59,038 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00000 type:create cxid:0x7 zxid:0x40000000b txntype:-1 reqpath:n/a Error Path:/services_config Error:KeeperErrorCode = NodeExists for /services_config
2016-04-12 16:24:59,041 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00000 type:create cxid:0x8 zxid:0x40000000c txntype:-1 reqpath:n/a Error Path:/services_config/hivemeta Error:KeeperErrorCode = NodeExists for /services_config/hivemeta
2016-04-12 16:24:59,055 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00000 type:create cxid:0xc zxid:0x40000000e txntype:-1 reqpath:n/a Error Path:/services_config/hs2 Error:KeeperErrorCode = NodeExists for /services_config/hs2
2016-04-12 16:24:59,071 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00000 type:create cxid:0x10 zxid:0x400000010 txntype:-1 reqpath:n/a Error Path:/services_config/hoststats Error:KeeperErrorCode = NodeExists for /services_config/hoststats
2016-04-12 16:24:59,120 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.240.250.179:42898
2016-04-12 16:24:59,122 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:ZooKeeperServer@839] - Client attempting to establish new session at /10.240.250.179:42898
2016-04-12 16:24:59,127 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@595] - Established session 0x1540cca58cd0000 with negotiated timeout 30000 for client /10.240.250.179:42898
2016-04-12 16:24:59,144 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5181:ZooKeeperServer@935] - adding SASL authorization for authorizationID: mapr
2016-04-12 16:24:59,188 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00002 type:create cxid:0x4 zxid:0x400000018 txntype:-1 reqpath:n/a Error Path:/services Error:KeeperErrorCode = NodeExists for /services
2016-04-12 16:24:59,189 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00001 type:create cxid:0x4 zxid:0x400000019 txntype:-1 reqpath:n/a Error Path:/services Error:KeeperErrorCode = NodeExists for /services
2016-04-12 16:24:59,189 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x1540cca58cd0000 type:create cxid:0x4 zxid:0x40000001a txntype:-1 reqpath:n/a Error Path:/services Error:KeeperErrorCode = NodeExists for /services
2016-04-12 16:24:59,191 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00002 type:create cxid:0x5 zxid:0x40000001b txntype:-1 reqpath:n/a Error Path:/services/hoststats Error:KeeperErrorCode = NodeExists for /services/hoststats
2016-04-12 16:24:59,192 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00001 type:create cxid:0x5 zxid:0x40000001c txntype:-1 reqpath:n/a Error Path:/services/hs2 Error:KeeperErrorCode = NodeExists for /services/hs2
2016-04-12 16:24:59,192 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x1540cca58cd0000 type:create cxid:0x5 zxid:0x40000001d txntype:-1 reqpath:n/a Error Path:/services/hivemeta Error:KeeperErrorCode = NodeExists for /services/hivemeta
2016-04-12 16:24:59,212 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x1540cca58cd0000 type:create cxid:0xb zxid:0x40000001e txntype:-1 reqpath:n/a Error Path:/hivemeta_locks Error:KeeperErrorCode = NodeExists for /hivemeta_locks
2016-04-12 16:24:59,212 [myid:1] - INFO  [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x540cca58d00001 type:create cxid:0xb zxid:0x40000001f txntype:-1 reqpath:n/a Error Path:/hs2_locks Error:KeeperErrorCode = NodeExists for /hs2_locks

 

I saw this in warden.log when trying to start warden for the Kerberized Hive server.

2016-04-12 16:24:59,183 ERROR com.mapr.warden.service.baseservice.Service addPluggableAlarm [Thread-14]: Need to recreate alarm to add: NODE_ALARM_SERVICE_HS2_DOWN
2016-04-12 16:24:59,186 ERROR com.mapr.warden.service.baseservice.Service addPluggableAlarm [Thread-13]: Need to recreate alarm to add: NODE_ALARM_SERVICE_HOSTSTATS_DOWN
2016-04-12 16:24:59,186 ERROR com.mapr.warden.service.baseservice.Service addPluggableAlarm [Thread-12]: Need to recreate alarm to add: NODE_ALARM_SERVICE_HIVEMETA_DOWN
2016-04-12 16:24:59,188 INFO  com.mapr.warden.service.baseservice.Service [Thread-13]: Node: /nodes/mapr51-ker-hive2.mynet.com/services/hoststats does not exist yet
2016-04-12 16:24:59,188 INFO  com.mapr.warden.service.baseservice.Service [Thread-14]: Node: /nodes/mapr51-ker-hive2.mynet.com/services/hs2 does not exist yet
2016-04-12 16:24:59,188 INFO  com.mapr.warden.service.baseservice.Service [Thread-12]: Node: /nodes/mapr51-ker-hive2.mynet.com/services/hivemeta does not exist yet
Exception in thread "Thread-13" java.lang.IllegalArgumentException: Invalid path string "/services//mapr51-ker-hive2.mynet.com" caused by empty node name specified @10
        at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:99)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1020)
        at com.mapr.warden.service.baseservice.common.ZKUtilsLocking.checkZKNodeForExistence(ZKUtilsLocking.java:64)
        at com.mapr.warden.service.baseservice.DependentService.checkifDependentServiceChanged(DependentService.java:101)
        at com.mapr.warden.service.baseservice.DependentService.createIPChild(DependentService.java:68)
        at com.mapr.warden.service.baseservice.Service.run(Service.java:1562)
        at java.lang.Thread.run(Thread.java:745)
2016-04-12 16:24:59,212 INFO  com.mapr.job.mngmnt.hadoop.metrics.WardenRequestBuilder [main]: [e_SERV_CONF, hostName, ma_host, ma_process]
2016-04-12 16:24:59,213 INFO  com.mapr.job.mngmnt.hadoop.metrics.WardenRequestBuilder [main]: []
Warden started

Thank you very much.

Outcomes