AnsweredAssumed Answered

Connecting remote spark cluster from Rstudio by installing sparklyr package

Question asked by srinivas.k on Apr 17, 2018
Latest reply on May 15, 2018 by maprcommunity

Connecting remote spark cluster from Rstudio by installing sparklyr package
--------------------------------------------------------------------------
We installed MapR clinet & Sparklyr package@ & R & Rstudio

we set the environment setting like below


1.MAPR_HOME =C:\opt\mapr
2.SPARK_HOME =C:\opt\mapr\spark\spark-2.1.0-bin-hadoop2.7
3.JAVA_HOME =C:\Program Files\Java\jdk1.8.0_162
4.HADOOP_CONF_DIR =C:\opt\mapr\hadoop\hadoop-0.20.2
5.HADOOP_HOME =C:\hadoop-common-2.2.0-bin-master for winutils.exe

 

 

We are trying to connect remote spark cluster(Hadoop cluster) from rstduio using sparklyr package....


We are able to connect local spark but unable to connect remote spark cluster..


We used below r code to connect remote spark cluster
----------------------------------------------------


>library(sparklyr)

>sc <- spark_connect(master = "http://10.2.28.4:18080")

Here:IP is the Remote saprk cluster ip with port number

 


We are getting below error while connect remote spark cluster
------------------------------------------------------------

Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
cannot open file 'C:\Users\admin\AppData\Local\Temp\RtmpWmwyXi\file3ad004c7714b6_spark.log': Permission denied@@

Attachments

Outcomes