AnsweredAssumed Answered

Drill Won't Start on Secure Cluster

Question asked by mandoskippy on Mar 9, 2017
Latest reply on Apr 11, 2017 by maprcommunity

I am running my own instance of Apache Drill (not managed by MapR) on a MapR Secure Cluster.  First let me say that on a non-secure cluster, this Drill package is working great. It's MapR's Build of Drill 1.8, but repackaged so I am managing all the aspects (not started by warden etc). 

 

That said, without SECURE, I am running with Authentication and with HTTPS no issues. 

 

When I enable MapR Secure Cluster, I get the error below on Bit startup, I tried setting in the drill-env.sh

 

export MAPR_TICKETFILE_LOCATION="/zeta/mydcos/zeta/prod/drill/drillprod/TICKETS/drillticket"

export MAPR_IMPERSONATION_ENABLED=true

 

I tried using a user ticket (mapr is my user, that's what I am using for all these) a Service Ticket, and then a Service ticket with impersonation. None of them work for me to be able to get Drill Started.   I get the error below, any help would be appreciated. 

 

Note in my drill-env.sh. I echoed both the location of the drill ticket, and the I ran "cat $MAPR_TICKETFILE_LOCATION"   Both commands outputed to STDOUT and showed that the Drill instance had access to the ticket, and could output it correctly.

 

 

 

 

Error:

 

Exception in thread "main" org.apache.drill.exec.exception.DrillbitStartupException: Failure while initializing values in Drillbit.      at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:287)      at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)      at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267) Caused by: org.apache.drill.common.exceptions.ExecutionSetupException: A System Table provider was either not specified or could not be found or instantiated      at org.apache.drill.exec.store.sys.PersistentStoreRegistry.newPStoreProvider(PersistentStoreRegistry.java:61)      at org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:104)      at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)      ... 2 more Caused by: java.lang.reflect.InvocationTargetException      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)      at org.apache.drill.exec.store.sys.PersistentStoreRegistry.newPStoreProvider(PersistentStoreRegistry.java:57)      ... 4 more Caused by: org.apache.drill.exec.exception.StoreException: unable to get filesystem      at org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider.<init>(ZookeeperPersistentStoreProvider.java:63)      at org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider.<init>(ZookeeperPersistentStoreProvider.java:47)      ... 9 more Caused by: java.io.IOException: failure to login: Unable to obtain MapR credentials      at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:751)      at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:688)      at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:572)      at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2848)      at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2838)      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2704)      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:407)      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)      at org.apache.drill.exec.store.dfs.DrillFileSystem.<init>(DrillFileSystem.java:92)      at org.apache.drill.exec.store.dfs.DrillFileSystem.<init>(DrillFileSystem.java:88)      at org.apache.drill.exec.store.sys.store.LocalPersistentStore.getFileSystem(LocalPersistentStore.java:98)      at org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider.<init>(ZookeeperPersistentStoreProvider.java:61)      ... 10 more Caused by: javax.security.auth.login.LoginException: Unable to obtain MapR credentials      at com.mapr.security.maprsasl.MaprSecurityLoginModule.login(MaprSecurityLoginModule.java:227)      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)      at java.lang.reflect.Method.invoke(Method.java:498)      at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)      at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)      at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)      at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)      at java.security.AccessController.doPrivileged(Native Method)      at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)      at javax.security.auth.login.LoginContext.login(LoginContext.java:587)      at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:724)      ... 21 more Caused by: com.mapr.login.MapRLoginException: Unable to authenticate as ticket is not available      at com.mapr.login.client.MapRLoginHttpsClient.authenticateIfNeeded(MapRLoginHttpsClient.java:173)      at com.mapr.login.client.MapRLoginHttpsClient.authenticateIfNeeded(MapRLoginHttpsClient.java:115)      at com.mapr.security.maprsasl.MaprSecurityLoginModule.login(MaprSecurityLoginModule.java:221)      ... 33 more

Outcomes