AnsweredAssumed Answered

R from SparkR Interactive shell

Question asked by Vinayak Meghraj on Sep 21, 2016
Latest reply on Sep 26, 2016 by Vinayak Meghraj

Steps to execute R from /opt/mapr/spark/spark-1.6.1/bin/sparkR interactive shell

 

Example 1) 

1) people <- read.df(sqlContext, "file:///opt/mapr/spark/spark-1.6.1/examples/src/main/resources/people.json", "json")

2) head(people)

 

Example 2)

1) sc <- sparkR.init()

2) sqlContext <- sparkRSQL.init(sc)

3)  df <- createDataFrame(sqlContext, faithful)

4) head(df)

 

[root@n2b bin]# /opt/mapr/spark/spark-1.6.1/bin/sparkR

 

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"

Copyright (C) 2016 The R Foundation for Statistical Computing

Platform: x86_64-redhat-linux-gnu (64-bit)

 

R is free software and comes with ABSOLUTELY NO WARRANTY.

You are welcome to redistribute it under certain conditions.

Type 'license()' or 'licence()' for distribution details.

 

  Natural language support but running in an English locale

 

R is a collaborative project with many contributors.

Type 'contributors()' for more information and

'citation()' on how to cite R or R packages in publications.

 

Type 'demo()' for some demos, 'help()' for on-line help, or

'help.start()' for an HTML browser interface to help.

Type 'q()' to quit R.

 

Launching java with spark-submit command /opt/mapr/spark/spark-1.6.1/bin/spark-submit   "sparkr-shell" /tmp/Rtmp7MFQW4/backend_port304d5af51abb

16/09/15 13:53:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

 

 Welcome to

    ____              __

   / __/__  ___ _____/ /__

  _\ \/ _ \/ _ `/ __/  '_/

 /___/ .__/\_,_/_/ /_/\_\   version  1.6.1

    /_/

 

 

 Spark context is available as sc, SQL context is available as sqlContext

> people <- read.df(sqlContext, "file:///opt/mapr/spark/spark-1.6.1/examples/src/main/resources/people.json", "json")

> head(people)

  age    name

1  NA Michael

2  30    Andy

3  19  Justin

> sc <- sparkR.init()

Re-using existing Spark Context. Please stop SparkR with sparkR.stop() or restart R to create a new Spark Context

> sqlContext <- sparkRSQL.init(sc)

> df <- createDataFrame(sqlContext, faithful)

> head(df)

  eruptions waiting

1     3.600      79

2     1.800      54

3     3.333      74

4     2.283      62

5     4.533      85

6     2.883      55

> quit;

 

[root@n2b bin]# rpm -qa | grep spark

mapr-spark-historyserver-1.6.1.201608302253-1.noarch

mapr-spark-1.6.1.201608302253-1.noarch

Outcomes