AnsweredAssumed Answered

CLDB not starting after security is enabled

Question asked by hanje04 on Apr 27, 2016
Latest reply on Apr 29, 2016 by hanje04

I've been having difficulty getting my cluster to come back up after enabling security/kerberos for the cluster.

I'm running 5.0 on CentOS 6.7. I've followed the instructions here:

MapR 5.1 Documentation

and here:

MapR 5.1 Documentation

but on startup the CLDB never comes up. warden.log just shows:

 

Header: hostName: hanje04-cloudera01.actian.com, Time Zone: Pacific Standard Time, processName: warden, processId: 12081, MapR Build Version: 5.0.0.32987.GA

2016-04-27 15:45:42,236 INFO  com.mapr.warden.WardenMain [main]: Log dir: /opt/mapr/hadoop/hadoop-0.20.2/logs

2016-04-27 15:45:42,236 INFO  com.mapr.warden.WardenMain [main]: Log dir: /opt/mapr/hadoop/hadoop-2.7.0/logs

2016-04-27 15:45:42,272 INFO  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [main]: init MAPRContext

2016-04-27 15:45:42,273 INFO  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [main]: init MAPRContextHS

2016-04-27 15:45:42,277 WARN  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [main]: Error while trying to get correct Ticket with errorCode: 2

2016-04-27 15:45:42,280 INFO  com.mapr.warden.WardenManager [main]: mapruserticket does not exist or it is older then maprserverticket. Regenerating it

2016-04-27 15:45:42,280 WARN  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [Thread-5]: Error while trying to get correct Ticket with errorCode: 2

2016-04-27 15:45:42,281 INFO  com.mapr.warden.WardenMain [main]: My pid: 12513

2016-04-27 15:45:42,286 INFO  com.mapr.warden.WardenManager [maprUserTicketGetCheckThread]: maprUserTicketExpiration: 0

Warden started

Warden started

In sysVol

2016-04-27 15:45:43,281 WARN  com.mapr.job.mngmnt.hadoop.metrics.MaprRPCContext [Thread-5]: Error while trying to get correct Ticket with errorCode: 2

 

and the last error is repeated indefinitely and seems to be the crux of the issue. Zookeeper also fails to come up, failing with:

 

2016-04-27 15:45:54,210 [myid:0] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally

java.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: MapR user ticket not available! error = com.mapr.security.MutableInt@5474c6c

  at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:205)

  at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)

  at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130)

  at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)

  at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

 

The cause of the problems seems to be the lack of the "mapruserticket" under /opt/mapr/conf but I can't find any info about how this is created. I've double check the ownership, permissions and md5sums on:

 

/opt/mapr/conf/cldb.key

/opt/mapr/conf/ssl_keystore

/opt/mapr/conf/maprserverticket

/opt/mapr/conf/ssl_truststore

 

across all the node an they're all fine.

The problem doesn't seem to be related to kerberos directly as my KDC log is showing the mapr/my.cluster@MYREALM ticket being issued. The same problem also occurs when enabling security WITHOUT kerberos. Can anyone give me some pointers on how to debug this further or some insight into what's going wrong.

Outcomes