AnsweredAssumed Answered

Spark Client on Windows Connect to Yarn Cluster

Question asked by endamcm on Jun 14, 2017
Latest reply on Jul 24, 2017 by vmeghraj

Hi,

 

I have MAPR client installed on a windows server and I am able to programmatically interact with a MAPR Cluster via this configuration.

 

I have now installed Apache Spark Client 2.1.0 on the windows server and setup the applicable environment variables.

 

I am supplying the following in the code

setMaster("yarn")

.setDeployMode("cluster")

 

along with the various arguments and properties that point to the spark job that is residing on the MAPR Cluster

 

When I attempt to interact with the Spark Instance on the MAPR Cluster programmatically I receive errors, The following been the most informative.

 

INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

Jun 14, 2017 5:06:29 PM org.apache.spark.launcher.OutputRedirector redirect

INFO: 17/06/14 17:06:29 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Jun 14, 2017 5:06:31 PM org.apache.spark.launcher.OutputRedirector redirect

INFO: 17/06/14 17:06:31 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

Jun 14, 2017 5:06:33 PM org.apache.spark.launcher.OutputRedirector redirect

 

What have I missed/What am I doing wrong?

 

Thanks,

Outcomes