AnsweredAssumed Answered

MapR 5.2 + Yarn + Spark: Is it possible to see both MRv2 and Spark jobs on one page

Question asked by davidehle on Mar 31, 2017
Latest reply on Aug 4, 2017 by rsotnychenko

I am looking for some clarification on the  (Apache Spark) documentation about aggregated logging with Yarn + Spark and how it works in a MapR environment

 

Running Spark on YARN - Spark 2.1.0 Documentation  says:

"You need to have both the Spark history server and the MapReduce history server running and configure yarn.log.server.url in yarn-site.xml properly. The log URL on the Spark history server UI will redirect you to the MapReduce history server to show the aggregated logs."

I can't find any documentation about if/how this would work in a MapR Environment beyond a question in the forums (Monitoring MapR Spark Activity) where  bmohanam Employee refers the the poster to the same quote above

It looks like the requester ended up viewing completed MR jobs at the JobHistory Server url (http://jobhistoryserver:19888/jobhistory) and Completed Spark jobs at the Spark History Server url (http://sparkhistoryserver:18080/

The quote from Apache Spark documentation seems to indicate that it would be possible to have a "Single Pane" to see all completed jobs both MR and Spark in one place. Is this possible/supported in a MapR Hadoop Environment?

Am I interpreting the Apache documentation correctly or is it just wishful thinking?

Thank you for any insight you can provide.

Outcomes