AnsweredAssumed Answered

Issue with "How to Get Started with Spark Streaming and MapR Streams Using the Kafka API" tutorial

Question asked by sagar.sonawane on Feb 21, 2017
Latest reply on Mar 17, 2017 by maprcommunity

I was reading about Spark Streaming and stumbled upon How to Get Started with Spark Streaming and MapR Streams Using the Kafka API | MapR blog. I followed all the instructions from github page GitHub - caroljmcdonald/mapr-streams-spark and here is my experience.

 

I can sucessfuly execute Producer part but for consumer part facing error like "Multiple versions of scala libraries detected". Now, if I use scala-maven-plugin(3.1.6) along with assembly plugin(without this build was failing) then I can have final jar sucessfully. But, then I was facing runtime exceptions like below:

 

Note: I have opened two console where producer(executed as shown on github page) was producing messages and following are the logs from second console where consumer was running with this, /opt/mapr/spark/spark-2.0.1/bin/spark-submit --class com.sparkkafka.example.SparkKafkaConsumer target/mapr-streams-spark-1.0-jar-with-dependencies.jar --master yarn-client

 

PFA attached logs snippet and also pom files(as the one from github was not failing to compile project). Please let me know if any further logs needed.

 

Few things to note are, when I have posted above description on above mentioned blog, it was marked as spam! That is why I have to create question over here.

 

carol mcdonald Please guide why I'm facing such errors. I am having 3 node MapR v5.2 cluster and running sample code/apps from client outside the cluster. I can successfully execute other spark demos using spark-sbumit with yarn.

 

Yours sincerely,

Sagar Sonawane

Outcomes