AnsweredAssumed Answered

Monitoring MapR Spark Activity

Question asked by Terry on Nov 2, 2016
Latest reply on Nov 7, 2016 by bmohanam

We are running 5.2 with MapR Spark (1.6.1) installed using YARN. I am able to run jobs and view them within the Resource Manager page until they complete. The History Server "Job History" does not show Spark jobs, only Hadoop jobs. Even while a Spark job is running, the Spark Master web page shows nothing:

  • Alive Workers: 0
  • Cores in use: 0 Total, 0 Used
  • Memory in use: 0.0 B Total, 0.0 B Used
  • Applications: 0 Running, 0 Completed
  • Drivers: 0 Running, 0 Completed
  • Status: ALIVE

- How can I find which nodes have MapR-spark installed and running? (It is not under services)

- How can I see which nodes are being used for a job and what their resource levels are, etc.?

- How can I tell what Spark workers are known to the master?

 

Thanks for any guidance.

Outcomes