AnsweredAssumed Answered

flink kerberos failures

Question asked by hsawhney on Aug 15, 2016
Latest reply on Aug 22, 2016 by hsawhney

I have been able to compile flink-1.1.1 with Mapr(2.7.0-mapr-1607). But when I try and run a single flink yarn job [Apache Flink 1.2-SNAPSHOT Documentation: YARN Setup] it fails cause of Kerberos authentication even though we do not have kerberos enabled in our Mapr cluster (stack trace below).

 

Is there a way I can make flink work without kerberos. I would have assumed the Mapr libraries would be able to handle it?

 

2016-08-16 00:07:39,119 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar

2016-08-16 00:07:39,119 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar

2016-08-16 00:07:39,216 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Using values:

2016-08-16 00:07:39,218 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - TaskManager count = 4

2016-08-16 00:07:39,219 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - JobManager memory = 1024

2016-08-16 00:07:39,219 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - TaskManager memory = 1024

java.security.PrivilegedActionException: javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm)]

  at java.security.AccessController.doPrivileged(Native Method)

  at javax.security.auth.Subject.doAs(Subject.java:415)

  at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslClient(ZooKeeperSaslClient.java:283)

  at org.apache.zookeeper.client.ZooKeeperSaslClient.<init>(ZooKeeperSaslClient.java:131)

  at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:949)

  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

Caused by: javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm)]

  at com.sun.security.sasl.gsskerb.GssKrb5Client.<init>(GssKrb5Client.java:150)

  at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:63)

  at javax.security.sasl.Sasl.createSaslClient(Sasl.java:372)

  at org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClient.java:288)

  at org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClient.java:283)

  ... 6 more

Caused by: GSSException: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm)

  at sun.security.jgss.krb5.Krb5NameElement.getInstance(Krb5NameElement.java:129)

  at sun.security.jgss.krb5.Krb5MechFactory.getNameElement(Krb5MechFactory.java:95)

  at sun.security.jgss.GSSManagerImpl.getNameElement(GSSManagerImpl.java:202)

  at sun.security.jgss.GSSNameImpl.getElement(GSSNameImpl.java:476)

  at sun.security.jgss.GSSNameImpl.init(GSSNameImpl.java:201)

  at sun.security.jgss.GSSNameImpl.<init>(GSSNameImpl.java:170)

  at sun.security.jgss.GSSManagerImpl.createName(GSSManagerImpl.java:137)

  at com.sun.security.sasl.gsskerb.GssKrb5Client.<init>(GssKrb5Client.java:108)

  ... 10 more

2016-08-16 00:07:39,625 ERROR org.apache.hadoop.yarn.client.MapRZKRMFinderUtils             - Unable to determine ResourceManager service address from Zookeeper at

2016-08-16 00:07:39,625 ERROR org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider  - Unable to create proxy to the ResourceManager null

2016-08-16 00:07:39,625 INFO  org.apache.hadoop.service.AbstractService                     - Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl failed in state STARTED; cause: java.lang.RuntimeException: Unable to create proxy to the ResourceManager null

java.lang.RuntimeException: Unable to create proxy to the ResourceManager null

  at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.getProxy(MapRZKBasedRMFailoverProxyProvider.java:135)

  at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73)

  at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64)

  at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:58)

  at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:95)

  at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:73)

  at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:193)

  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

  at org.apache.flink.yarn.AbstractYarnClusterDescriptor.getYarnClient(AbstractYarnClusterDescriptor.java:306)

  at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:383)

  at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:365)

  at org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:513)

  at org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:63)

  at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:891)

  at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)

  at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)

  at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)

Caused by: java.lang.RuntimeException: Unable to determine ResourceManager service address from Zookeeper at <private hostnames removed>

  at org.apache.hadoop.yarn.client.MapRZKRMFinderUtils.mapRZkBasedRMFinder(MapRZKRMFinderUtils.java:121)

  at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.updateCurrentRMAddress(MapRZKBasedRMFailoverProxyProvider.java:64)

  at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.getProxy(MapRZKBasedRMFailoverProxyProvider.java:131)

  ... 16 more

 

 

------------------------------------------------------------

The program finished with the following exception:

 

 

java.lang.RuntimeException: Error deploying the YARN cluster

  at org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:515)

  at org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:63)

  at org.apache.flink.client.CliFrontend.createClient(CliFrontend.java:891)

  at org.apache.flink.client.CliFrontend.run(CliFrontend.java:237)

  at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1005)

  at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1048)

Caused by: java.lang.RuntimeException: Couldn't deploy Yarn cluster

  at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:368)

  at org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:513)

  ... 5 more

Caused by: java.lang.RuntimeException: Unable to create proxy to the ResourceManager null

  at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.getProxy(MapRZKBasedRMFailoverProxyProvider.java:135)

  at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73)

  at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64)

  at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:58)

  at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:95)

  at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:73)

  at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:193)

  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

  at org.apache.flink.yarn.AbstractYarnClusterDescriptor.getYarnClient(AbstractYarnClusterDescriptor.java:306)

  at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:383)

  at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:365)

  ... 6 more

Caused by: java.lang.RuntimeException: Unable to determine ResourceManager service address from Zookeeper at <private hostname removed>   at org.apache.hadoop.yarn.client.MapRZKRMFinderUtils.mapRZkBasedRMFinder(MapRZKRMFinderUtils.java:121)

  at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.updateCurrentRMAddress(MapRZKBasedRMFailoverProxyProvider.java:64)

  at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.getProxy(MapRZKBasedRMFailoverProxyProvider.java:131)

  ... 16 more

Outcomes