AnsweredAssumed Answered

How to setup spark environment on YARN for streaming applications

Question asked by karthikSpark on Dec 29, 2016
Latest reply on Jan 4, 2017 by cathy

Hi Mapr folks,

Here i was given a simple cluster which was already configured by my admin, but i'm failing to write the data in to maprfs with spark streaming application.I think i'm running in to every possible configuration failures, or may be the application failure.

Before blaming myself or any one else, can anyone please point me in a direction where i will be able to check the entire environment to make my application work.

 

I have successfully installed and configured kafka in my cluster, i'm able to read data from the kafka topics also.

But i'm failing to write the same data in to the distributed file system( i can store it in to lfs though) using my spark streaming application.

Can anyone provide me a valid link, or set of configuration file properties where i have to check.

 

My project flow.

External source >>Kafka Topic >> Spark Streaming >> Maprfs/ external table's Maprfs location(hive)

The main thing is my application is not picking up the classpath.

Outcomes