AnsweredAssumed Answered

Querying Spark Statistics via API

Question asked by john.humphreys on Sep 20, 2017
Latest reply on Sep 20, 2017 by cathy

I can get plenty of statistics from the Spark UI on streaming, jobs, etc from this URL:

 

https://hostname.company.com:8090/proxy/application_1503172397325_28477/streaming

 

Spark documentation says there's a REST API for these statistics:

 

REST API

In addition to viewing the metrics in the UI, they are also available as JSON. This gives developers an easy way to create new visualizations and monitoring tools for Spark. The JSON is available for both running applications, and in the history server. The endpoints are mounted at /api/v1. Eg., for the history server, they would typically be accessible at http://<server-url>:18080/api/v1, and for a running application, at http://localhost:4040/api/v1.

In the API, an application is referenced by its application ID, [app-id]. When running on YARN, each application may have multiple attempts, but there are attempt IDs only for applications in cluster mode, not applications in client mode. Applications in YARN cluster mode can be identified by their [attempt-id]. In the API listed below, when running in YARN cluster mode, [app-id] will actually be [base-app-id]/[attempt-id], where [base-app-id] is the YARN application ID.

I can't seem to find the URL that this works on with MapR though.


Where would the REST API for the given spark application in my original URL be?

Outcomes