Spark JVM Profiler Plugin

Fork Origin

This project is forked from the Apache Spark connector/profiler module, copied from the master branch at commit c35e98fb487224d07a1051a35a78ef1f5efe2740.

The initial import preserves the upstream Scala source code, package names, and copyright headers unchanged. The Maven configuration has been adapted for standalone builds, and this document has been updated with build and runtime instructions. This project does not require the Spark source checkout or inherit its parent POM and is not an official Apache Spark release. The code is licensed under the Apache License 2.0; the upstream license and notices are retained in LICENSE and NOTICE.

Build

Requires JDK 17+ and Maven 3.6.3+. The default dependencies are Spark 4.2.0 and Scala 2.13.18, with async-profiler loader 4.5-13 matching upstream master.

mvn clean verify

Output: target/spark-profiler-plugin_2.13-0.1.0-SNAPSHOT.jar. The upstream module contains no tests; verify compiles the source code and packages the JAR.

Use -Dspark.version=... -Dscala.version=... to match your cluster versions. The source code uses Spark internal APIs, so rebuild and verify compatibility whenever you change versions. The Scala binary version is 2.13. The current upstream source depends on SparkContext.isDriver and cannot compile against Spark 4.1.x without changes.

The generated JAR does not bundle dependencies. Spark and Scala are provided by the cluster; supply me.bechberger:ap-loader-all:4.5-13 at runtime, for example through --packages as shown below. If the cluster cannot access Maven repositories, add a previously downloaded loader JAR to --jars alongside the plugin JAR. Since the upstream class names are preserved, avoid loading another Spark profiler module JAR at the same time.

Executor Code Profiling

The spark profiler module enables code profiling of executors in cluster mode based on the async profiler, a low overhead sampling profiler. This allows a Spark application to capture CPU and memory profiles for application running on a cluster which can later be analyzed for performance issues. The profiler captures Java Flight Recorder (jfr) files for each executor; these can be read by many tools including Java Mission Control and Intellij.

The profiler writes the jfr files to the executor's working directory in the executor's local file system and the files can grow to be large, so it is advisable that the executor machines have adequate storage. The profiler can be configured to copy the jfr files to a hdfs location before the executor shuts down.

Code profiling is currently only supported for

• Linux (x64)
• Linux (arm64)
• Linux (musl, x64)
• MacOS

To get maximum profiling information set the following jvm options for the executor :

spark.executor.extraJavaOptions=-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer

For more information on async_profiler see the Async Profiler Manual

To enable code profiling, first enable the code profiling plugin via

spark.plugins=org.apache.spark.profiler.ProfilerPlugin

Then enable the profiling in the configuration.

Task spans

Set spark.profiler.executor.taskSpan.enabled=true to record each task attempt as a profiler.Span event on profiled executors. Task spans are disabled by default and require executor profiling to be enabled; executor sampling still applies.

Spans are written to the same JFR file as the profiling samples. The span starts in the task start callback and ends in the success or failure callback, including result serialization on successful tasks. Task deserialization before the start callback is not included.

The span tag starts with SparkTask, followed by comma-separated stageId, stageAttemptNumber, partitionId, taskAttemptId, attemptNumber, and status (succeeded or failed) key-value pairs. These tags allow profiling samples to be associated with individual task attempts, including retries and speculative attempts. Spans are only emitted while a JFR recording is active.

Code profiling configuration

Property Name	Default	Meaning	Since Version
spark.profiler.driver.enabled	false	If true, turn on profiling in driver.	4.0.0
spark.profiler.executor.enabled	false	If true, turn on profiling in executors.	4.0.0
spark.profiler.executor.fraction	0.10	The fraction of executors on which to enable profiling. The executors to be profiled are picked at random.	4.0.0
spark.profiler.executor.taskSpan.enabled	false	If true, record task attempt spans in the JFR output of executors selected for profiling. Requires spark.profiler.executor.enabled=true and respects spark.profiler.executor.fraction.	5.0.0
spark.profiler.dfsDir	(none)	An HDFS compatible path to which the profiler's output files are copied. The output files will be written as dfsDir/{{APP_ID}}/profile-exec-{{EXECUTOR_ID}}.jfr If no dfsDir is specified then the files are not copied over. Users should ensure there is sufficient disk space available otherwise it may lead to corrupt jfr files.	4.0.0
spark.profiler.localDir	. i.e. the executor's working dir	The local directory in the executor container to write the jfr files to. If not specified the file will be written to the executor's working directory. Users should ensure there is sufficient disk space available on the system as running out of space may result in corrupt jfr file and even cause jobs to fail on systems like K8s.	4.0.0
spark.profiler.asyncProfiler.args	event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s	Arguments to pass to the Async Profiler. Detailed options are documented in the comments here: Profiler options. Note that the arguments to start, stop, specify output format, and output file do not have to be specified.	4.0.0
spark.profiler.dfsWriteInterval	30	Time interval, in seconds, after which the profiler output will be synced to DFS.	4.0.0

Kubernetes

On Kubernetes, spark will try to shut down the executor pods while the profiler files are still being saved. To prevent this set

  spark.kubernetes.executor.deleteOnTermination=false

Example

./bin/spark-submit \
  --jars target/spark-profiler-plugin_2.13-0.1.0-SNAPSHOT.jar \
  --packages me.bechberger:ap-loader-all:4.5-13 \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  -c spark.executor.extraJavaOptions="-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:+PreserveFramePointer" \
  -c spark.plugins=org.apache.spark.profiler.ProfilerPlugin \
  -c spark.profiler.executor.enabled=true \
  -c spark.profiler.executor.fraction=0.10 \
  -c spark.profiler.dfsDir=s3a://my-bucket/spark/profiles/ \
  -c spark.profiler.asyncProfiler.args=event=wall,interval=10ms,alloc=2m,lock=10ms,chunktime=300s \
  -c spark.kubernetes.executor.deleteOnTermination=false \
  <application-jar> \
  [application-arguments]