Skip to content

Extract SQL-plan instrumentation into standalone repo#78

Closed
minskya wants to merge 1 commit into
mainfrom
claude/instrumentation-extraction-2K3Ip
Closed

Extract SQL-plan instrumentation into standalone repo#78
minskya wants to merge 1 commit into
mainfrom
claude/instrumentation-extraction-2K3Ip

Conversation

@minskya
Copy link
Copy Markdown
Contributor

@minskya minskya commented May 29, 2026

Move the operator-timing instrumentation feature out of dataflint/spark into
a self-contained project seed (instrumentation-repo/) that depends on the
published io.dataflint:dataflint-common artifact and publishes to GitHub
Packages.

Removed from spark-plugin:

  • common: TimedExec, MetricsUtils, Accelerator, GraphDurationAttribution
  • pluginspark3/pluginspark4: DataFlintInstrumentationExtension
  • pluginspark3/src/test: the entire instrumentation regression suite
  • pluginspark4 build block pulling in the moved DataFlintCodegenFallbackSpec

Kept as the loose-coupling contract in dataflint-common:

  • INSTRUMENT_* config constants + registerInstrumentationExtension
    (loads the extension by class name via spark.sql.extensions)
  • api.ResolvedStageGroup

The new repo seed compiles (spark3/spark4 + test suite) and assembles fat
jars; the trimmed spark-plugin (common/spark3/spark4) still compiles.
See INSTRUMENTATION-EXTRACTION.md and instrumentation-repo/NEW-REPO-SETUP.md.

Move the operator-timing instrumentation feature out of dataflint/spark into
a self-contained project seed (instrumentation-repo/) that depends on the
published io.dataflint:dataflint-common artifact and publishes to GitHub
Packages.

Removed from spark-plugin:
- common: TimedExec, MetricsUtils, Accelerator, GraphDurationAttribution
- pluginspark3/pluginspark4: DataFlintInstrumentationExtension
- pluginspark3/src/test: the entire instrumentation regression suite
- pluginspark4 build block pulling in the moved DataFlintCodegenFallbackSpec

Kept as the loose-coupling contract in dataflint-common:
- INSTRUMENT_* config constants + registerInstrumentationExtension
  (loads the extension by class name via spark.sql.extensions)
- api.ResolvedStageGroup

The new repo seed compiles (spark3/spark4 + test suite) and assembles fat
jars; the trimmed spark-plugin (common/spark3/spark4) still compiles.
See INSTRUMENTATION-EXTRACTION.md and instrumentation-repo/NEW-REPO-SETUP.md.
@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@minskya minskya closed this May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants