Spark Troubleshooting Guide: Running Spark: How to collect GC statistics for Spark (Garbage Collection)

Document created by hdevanath Employee on Jun 19, 2017
Version 1Show Document
  • View in full screen mode

To collect statistics on how frequently garbage collection occurs and the amount of time spent GC, consider the following examples.

 

Case 1) Collect GC for Spark executor

--conf "spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps”


Case 2) Collect GC for Spark driver

--conf "spark.driver.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps”

 
Case 3) Collect GC for NodeManager
To collect GC for NodeManager, use the parameters shown below in /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/yarn-env.sh
 

export YARN_NODEMANAGER_OPTS="${YARN_NODEMANAGER_OPTS} ${MAPR_LOGIN_OPTS} -XX:+PrintGCDetails -XX:+PrintGC -XX:+PrintGCTimeStamps "
1 person found this helpful

Attachments

    Outcomes