AnsweredAssumed Answered

mapr-warden will not start after enabling security -- SASL authentication failed using login context 'Client'

Question asked by amsimms01 on Feb 8, 2016
Latest reply on Sep 12, 2017 by asathhusain
I just enabled security for the first time on my previously functioning cluster. Now warden will not start, it seems to have trouble when attempting to authenticate.   I did run configure.sh with -genkeys on my CLDB host, and dutifully copied cldb.key, maprserverticket, ssl_keystore, ssl_truststore to my other zookeepers, and the same set less cldb.key to the non-zookeepers:

    $  ls -l /opt/mapr/conf/{ssl_truststore,ssl_keystore,maprserverticket,cldb.key}
    -rw------- 1 mapr mapr   89 Feb  7 19:13 /opt/mapr/conf/cldb.key
    -rw------- 1 mapr mapr  301 Feb  7 19:13 /opt/mapr/conf/maprserverticket
    -rw------- 1 mapr mapr 2102 Feb  7 19:13 /opt/mapr/conf/ssl_keystore
    -r--r--r-- 1 mapr mapr  811 Feb  7 19:13 /opt/mapr/conf/ssl_truststore

The complete error log for warden is below, but the trouble seems to be some sort SASL authentication failure: SASL authentication failed using login context 'Client'. 

I've tried clearing all zookeeper data, and I've run and rerun configure.sh repeatedly.  I can connect to the zookeepers without issue from command line--they are running and elected the second host as leader. 

**warden.log from CLDB host**

no jobtracker to stop
Header: hostName: hadoop-data01.lpi.localdomain, Time Zone: Hawaii Standard Time, processName: warden, processId: 6178, MapR Build Version: 5.0.0.32987.GA
2016-02-08 06:18:57,327 INFO  com.mapr.warden.WardenMain [main]: Log dir: /opt/mapr/hadoop/hadoop-0.20.2/logs
2016-02-08 06:18:57,327 INFO  com.mapr.warden.WardenMain [main]: Log dir: /opt/mapr/hadoop/hadoop-2.7.0/logs
2016-02-08 06:18:57,369 INFO  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [main]: init MAPRContext
2016-02-08 06:18:57,369 INFO  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [main]: init MAPRContextHS
2016-02-08 06:18:57,377 WARN  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [main]: Error while trying to get correct Ticket with errorCode: 2
2016-02-08 06:18:57,379 WARN  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [Thread-5]: Error while trying to get correct Ticket with errorCode: 2
2016-02-08 06:18:57,383 INFO  com.mapr.warden.WardenManager [main]: mapruserticket does not exist or it is older then maprserverticket. Regenerating it
2016-02-08 06:18:57,384 INFO  com.mapr.warden.WardenManager [maprUserTicketGetCheckThread]: maprUserTicketExpiration: 0
2016-02-08 06:18:57,384 INFO  com.mapr.warden.WardenMain [main]: My pid: 6412
2016-02-08 06:18:57,578 ERROR com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils getDataForParticularCLDB [main-SendThread(hadoop-data02.lpi.localdomain:5181)]: No data returned in RPC: CLDB Ips: 10.255.241.70-, Port: 7222. Continue searching for correct CLDB
2016-02-08 06:18:57,580 INFO  com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils [main-SendThread(hadoop-data02.lpi.localdomain:5181)]: Bad CLDB credentials removed: CLDB Ips: 10.255.241.70-, Port: 7222
2016-02-08 06:18:57,580 WARN  com.mapr.security.maprsasl.MaprSecurityLoginModule [main-SendThread(hadoop-data02.lpi.localdomain:5181)]: Unable to generate the server key.
2016-02-08 06:18:57,599 INFO  com.mapr.warden.service.baseservice.zksessionmgmnt.ZookeeperClientSessionManagement [main]: Connected to ZK: hadoop-data01.lpi.localdomain:5181,hadoop-data02.lpi.localdomain:5181,hadoop-data03.lpi.localdomain:5181With State: State:CONNECTED Timeout:30000 sessionid:0x152bfae62160005 local:/10.255.241.70:35187 remoteserver:hadoop-data02.lpi.localdomain/10.255.241.71:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
2016-02-08 06:18:57,600 INFO  com.mapr.warden.service.baseservice.zksessionmgmnt.ZookeeperClientSessionManagement [main-EventThread]: ZK Connect state:State:CONNECTED Timeout:30000 sessionid:0x152bfae62160005 local:/10.255.241.70:35187 remoteserver:hadoop-data02.lpi.localdomain/10.255.241.71:5181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
2016-02-08 06:18:57,600 INFO  com.mapr.warden.service.baseservice.zksessionmgmnt.ZookeeperClientSessionManagement [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
2016-02-08 06:18:57,612 ERROR org.apache.zookeeper.client.ZooKeeperSaslClient respondToServer [main-SendThread(hadoop-data02.lpi.localdomain:5181)]: SASL authentication failed using login context 'Client'.
2016-02-08 06:18:57,612 INFO  com.mapr.warden.service.baseservice.zksessionmgmnt.ZookeeperClientSessionManagement [main-EventThread]: Process path: null. Event state: AuthFailed. Event type: None
2016-02-08 06:18:57,728 ERROR com.mapr.warden.WardenManager manage [main]: Keeper Exception while trying to deal with: /nodes/hadoop-data01.lpi.localdomain/stop
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /nodes/hadoop-data01.lpi.localdomain/stop
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at com.mapr.warden.service.baseservice.common.ZKUtilsLocking.checkZKNodeForExistence(ZKUtilsLocking.java:64)
    at com.mapr.warden.WardenManager.manage(WardenManager.java:310)
    at com.mapr.warden.WardenMain.main(WardenMain.java:301)
2016-02-08 06:18:57,734 ERROR com.mapr.warden.WardenServer addWardenServerToZK [main]: Keeper Exception during znode creation: /servers
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /servers
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
    at com.mapr.warden.service.baseservice.common.ZKUtilsLocking.createZKNode(ZKUtilsLocking.java:141)
    at com.mapr.warden.WardenServer.addWardenServerToZK(WardenServer.java:177)
    at com.mapr.warden.WardenManager.manage(WardenManager.java:343)
    at com.mapr.warden.WardenMain.main(WardenMain.java:301)
2016-02-08 06:18:57,735 FATAL com.mapr.warden.WardenManager manage [main]: Cannot create: /servers node or it is child node. Exiting
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /servers
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
    at com.mapr.warden.service.baseservice.common.ZKUtilsLocking.createZKNode(ZKUtilsLocking.java:141)
    at com.mapr.warden.WardenServer.addWardenServerToZK(WardenServer.java:177)
    at com.mapr.warden.WardenManager.manage(WardenManager.java:343)
    at com.mapr.warden.WardenMain.main(WardenMain.java:301)
2016-02-08 06:18:57,737 INFO  com.mapr.warden.WardenMain [Thread-2]: ShutdownHook started
2016-02-08 06:18:57,737 INFO  com.mapr.warden.centralconfig.PullCentralConfigTaskScheduler [Thread-2]: Cancelling outstanding PullCentralConfigTask
2016-02-08 06:18:57,737 WARN  com.mapr.warden.service.baseservice.zksessionmgmnt.ZookeeperClientSessionManagement [Thread-7]: Reinit Service thread's for Warden was interrupted. Most likely because of the shutdown.
2016-02-08 06:18:57,738 INFO  com.mapr.warden.WardenMain [Thread-2]: Shutdown took: 2 ms
2016-02-08 06:18:57,738 INFO  com.mapr.warden.WardenMain [Thread-2]: ShutdownHook completed
Warden started
Warden started
In sysVol

Outcomes