AnsweredAssumed Answered

Spark on MapR - Apache Provided vs. MapR Packages

Question asked by mandoskippy on Jun 26, 2017
Latest reply on Jun 30, 2017 by asukhenko

Greetings all -

 

I've been using Spark Packages directly from the Spark download links. I can choose "bin-without-hadoop" for any version, set the MapR Class path and still have that binary distribution work with MapR FS with no issues.  

 

That said, I started digging into Kafka/Streams integration with Spark Streaming, and am curious... does the same thing apply with MapR Packages vs Spark Packages on kafka stuff (to be able to use MapR Streams) or do I need to actually use the MapR Package in order to use MapR Streams with Spark?  

 

My issue is we needed a bug fix in Spark 2.1.1, there is no MapR version of this available yet, however, I'd like to play with Streaming a bit, so, back to my question, do I NEED the MapR Packages to use MapR Streams, or is there a Class Path hack like there is for MapR FS on the standard Spark Distro I could use?  (I guess this question sorta also applies to MapR DB and the Hbase API)

 

Thanks!

 

John

Outcomes