AnsweredAssumed Answered

MapR Streams + Streamsets: Pipeline start failed, no spark binaries(?)

Question asked by reedv on Jan 19, 2018
Latest reply on Jan 22, 2018 by rupal

Problem (so far as I can see):

Trying to use streamsets to create a pipeline from a mapr Stream consumer origin to a mapr FS destination, but getting an error that seems to indicate YARN is having some problems working with Spark. The parts of the streamsets log for the pipeline that seem relevant read

+ exec /opt/mapr/spark/spark-2.1.0/bin/spark-submit --master yarn --deploy-mode cluster --executor-memory 2048m --executor-cores 1 --num-executors 5 --archives /var/lib/sdc/temp/cluster-pipeline-sa2tenantsademodatabatch0ad4bb39-5057-4480-8325-19803bd5d744-0/staging/libs.tar.gz,/var/lib/sdc/temp/cluster-pipeline-sa2tenantsademodatabatch0ad4bb39-5057-4480-8325-19803bd5d744-0/staging/etc.tar.gz,/var/lib/sdc/temp/cluster-pipeline-sa2tenantsademodatabatch0ad4bb39-5057-4480-8325-19803bd5d744-0/staging/resources.tar.gz --files /var/lib/sdc/temp/cluster-pipeline-sa2tenantsademodatabatch0ad4bb39-5057-4480-8325-19803bd5d744-0/staging/log4j.properties --jars /opt/streamsets-datacollector/libexec/bootstrap-libs/main/streamsets-datacollector-bootstrap-3.0.1.0.jar,/opt/streamsets-datacollector/streamsets-libs/streamsets-datacollector-mapr_6_0-mep4-lib/lib/maprfs-6.0.0-mapr.jar,/opt/streamsets-datacollector/streamsets-libs/streamsets-datacollector-mapr_6_0-mep4-lib/lib/maprfs-6.0.0-mapr-tests.jar,/opt/streamsets-datacollector/container-lib/streamsets-datacollector-container-3.0.1.0.jar,/opt/streamsets-datacollector/container-lib/streamsets-datacollector-common-3.0.1.0.jar,/opt/streamsets-datacollector/api-lib/streamsets-datacollector-api-3.0.0.0.jar,/opt/streamsets-datacollector/libexec/bootstrap-libs/cluster/streamsets-datacollector-cluster-bootstrap-api-3.0.1.0.jar --conf 'spark.executor.extraJavaOptions=-javaagent:./streamsets-datacollector-bootstrap-3.0.1.0.jar -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dlog4j.debug -Dmaprlogin.password.enabled=true' --name 'StreamSets Data Collector: sa2tenant_sa_demodata_stream' --class com.streamsets.pipeline.BootstrapClusterStreaming /opt/streamsets-datacollector/libexec/bootstrap-libs/cluster/streamsets-datacollector-mapr-cluster-bootstrap-3.0.1.0.jar
18/01/19 09:23:17 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Exception in thread "main" org.apache.spark.SparkException: Application application_1513310498578_0058 finished with failed status

then

Application Report : 
     Application-Id : application_1513310498578_0058
     Application-Name : StreamSets Data Collector: sa2tenant_sa_demodata_stream
     Application-Type : SPARK
     User : mapr
     Queue : root.mapr
     Start-Time : 1516389806380
     Finish-Time : 1516389902766
     Progress : 100%
     State : FINISHED
     Final-State : FAILED
     Tracking-URL : http://mapr004.ucera.local:18080/history/application_1513310498578_0058/2
     RPC Port : 0
     AM Host : 172.18.4.101
     Aggregate Resource Allocation : 658759 MB-seconds, 242 vcore-seconds
     Diagnostics : User class threw exception: java.lang.IllegalStateException: Error trying to invoke BootstrapClusterStreaming.main: java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/v09/KafkaUtils

 

Preliminary stuff:

The pipeline in in cluster batch mode and is correctly configured for hadoop impersonation (eg. the pipeline can run in cluster batch mode when using a maprFS destination). I have no quota set on the volume holding the stream (since I thought it may be a problem, due to this post), though I don't really think it should matter since I was able to successfully create the stream in the MCS. Furthermore, I can see that the stream exists by:

[mapr@mapr001 clarity_eap_hph_sa]$ maprcli stream info -path /path/to/my/stream/streamname
produceperm copyperm ischangelog logicalsize adminperm topicperm ttl path numtopics defaultpartitions autocreate consumeperm compression physicalsize
u:mapr u:mapr false 0 u:mapr u:mapr 604800 /path/to/my/stream/streamname 1 1 false u:mapr lz4 0

I have also successfully set up a pipeline that uses a maprFS origin and mapr stream producer destination in batch cluster mode which can startup, run, and finish the job.

 

Extra problem details (that may or may not actually be relevant):

Looking into the problem a bit more these are some of the environment variables that I found looking at the failed job on the spark history server web UI:

spark.yarn.dist.archivesfile:/var/lib/sdc/temp/cluster-pipeline-sa2tenantsademodatabatch0ad4bb39-5057-4480-8325-19803bd5d744-0/staging/libs.tar.gz,file:/var/lib/sdc/temp/cluster-pipeline-sa2tenantsademodatabatch0ad4bb39-5057-4480-8325-19803bd5d744-0/staging/etc.tar.gz,file:/var/lib/sdc/temp/cluster-pipeline-sa2tenantsademodatabatch0ad4bb39-5057-4480-8325-19803bd5d744-0/staging/resources.tar.gz
spark.submit.deployModecluster
spark.masteryarn
spark.authenticate.enableSaslEncryptiontrue
spark.authenticatetrue
spark.yarn.dist.jarsfile:/opt/streamsets-datacollector/libexec/bootstrap-libs/main/streamsets-datacollector-bootstrap-3.0.1.0.jar,file:/opt/streamsets-datacollector/streamsets-libs/streamsets-datacollector-mapr_6_0-mep4-lib/lib/maprfs-6.0.0-mapr.jar,file:/opt/streamsets-datacollector/streamsets-libs/streamsets-datacollector-mapr_6_0-mep4-lib/lib/maprfs-6.0.0-mapr-tests.jar,file:/opt/streamsets-datacollector/container-lib/streamsets-datacollector-container-3.0.1.0.jar,file:/opt/streamsets-datacollector/container-lib/streamsets-datacollector-common-3.0.1.0.jar,file:/opt/streamsets-datacollector/api-lib/streamsets-datacollector-api-3.0.0.0.jar,file:/opt/streamsets-datacollector/libexec/bootstrap-libs/cluster/streamsets-datacollector-cluster-bootstrap-api-3.0.1.0.jar

and in the Pipeline/Cluster and Pipeline/Parameters tab, I have the following env. variable set:

 SPARK_SUBMIT_YARN_COMMAND = /opt/mapr/spark/spark-2.1.0/bin/spark-submit

 

The pipeline validates and can be previewed successfully and I don't know very much about spark (a bit more about YARN) to even know what to look for. Has anyone had this kind of problem before or know how to fix it?

Thanks.

Outcomes