AnsweredAssumed Answered

Is Spark with Yarn Supported on Windows client?

Question asked by chrisyeung on Nov 1, 2017
Latest reply on Nov 9, 2017 by vmeghraj

We have an app that uses Spark with Yarn client mode. The setup works with Linux client with Yarn, but we have difficulty getting it to work on Windows client. This is mainly for development work where the app can be launched from an IDE on Windows (i.e. the driver will run on Windows). As there isn't a Windows installer that includes Spark, we copied the jars from Linux client and setup classpaths, environment variables, etc., manually. However, we keep getting the following ZK exception. Is there a way to make it work?

 

(Just to add that there is a Json lib in the class path)

 

17/11/01 14:05:00 ERROR client.MapRZKRMFinderUtils: Zookeeper address not found from MapRFilesystem. Will try the configuration from yarn-site.xmljava.io.IOException: failure to login: java.lang.NoClassDefFoundError: org/json/JSONException
at com.mapr.security.maprsasl.MaprSecurityLoginModule.login(MaprSecurityLoginModule.java:219)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:724)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:688)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:572)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2848)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2838)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2704)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:407)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.hadoop.yarn.client.MapRZKRMFinderUtils.mapRZkBasedRMFinder(MapRZKRMFinderUtils.java:71)
at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.updateCurrentRMAddress(MapRZKBasedRMFailoverProxyProvider.java:64)
at org.apache.hadoop.yarn.client.MapRZKBasedRMFailoverProxyProvider.getProxy(MapRZKBasedRMFailoverProxyProvider.java:131)
at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73)
at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64)
at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:58)
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:95)
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:73)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart(YarnClientImpl.java:193)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.client.cli.LogsCLI.createYarnClient(LogsCLI.java:185)
at org.apache.hadoop.yarn.client.cli.LogsCLI.verifyApplicationState(LogsCLI.java:157)
at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:122)
at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:193)
Caused by: java.lang.ClassNotFoundException: org.json.JSONException
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 34 more

17/11/01 14:05:00 ERROR client.MapRZKRMFinderUtils: Zookeeper address not found from MapRFilesystem. Will try the configuration from yarn-site.xml
17/11/01 14:05:00 ERROR client.MapRZKRMFinderUtils: Zookeeper address can not be retrieved. Trying backup zk address
17/11/01 14:05:00 ERROR client.MapRZKRMFinderUtils: Zookeeper address not configured in Yarn configuration. Please check yarn-site.xml.
17/11/01 14:05:00 ERROR client.MapRZKRMFinderUtils: Unable to determine ResourceManager service address from Zookeeper.

Outcomes