Since Spark application runs on JVM, the --verbose and the --verbose:class options are both available.
Two switches are relevant in this context. The --verbose option provides configuration details and --verbose:class option reveals the classes loaded by the driver and executor. This debugging utility helps you trace class path conflicts for driver and executor.
1) To list the classes loaded by JVM while running a Java program, use --verbose option. The output is a list of all the classes loaded by the Class loader and the source that called the class. The following is a sample code using the --verbose:class option.
./spark-shell --conf "spark.executor.extraJavaOptions=-verbose:class" --conf "spark.driver.extraJavaOptions=-verbose:class"
What to expect
The screen shot below shows the console output showing class loaded and the source that called the class.
2) To launch the spark-shell or to run a Spark program using spark-submit, use --verbose option. The --verbose option outputs fine-grained debugging information like where the application loading source.
Tags: Spark 2.0, Apache Spark, Spark 2.1, Spark 2.x,