Add Apache Gluten/Velox support to DataFlint UI

[menishmueli]

Summary

Adds first-class support for native query accelerators in DataFlint — both in the Spark UI side (node classification, parsers, stage attribution) and the JVM-side plugin (instrumentation that no longer fights the accelerator's plan substitution). Includes runnable Docker examples for Apache Gluten/Velox and Apache DataFusion Comet.

What's in

UI / SqlFlow

• Recognize Gluten/Velox, Comet/DataFusion, RAPIDS GPU, Photon node types: nodeTypeDict, nodeRenamerDict, exchange/blocking sets in GraphDurationAttribution.ts, and accelerator badges on StageNode.
• Parser updates so WindowParser, FilterParser, ProjectParser, ExchangeParser, hashAggregateParser accept transformer/columnar variants (e.g. WindowExecTransformer, ColumnarExchange, CometExchange).
• SqlReducer infers wholeStageCodegenId for Gluten's WholeStageCodegenTransformer (no parent edges in the graph) by ID-ordered scan up to stage boundaries; AQE codegen renumbering is reconciled at the stage-attribution step.
• Velox-native metrics are surfaced in the allowlist: time of aggregation/filter/sort/window/generate/hash build/probe, number of hash build/probe input rows, number of spilled bytes, peak memory bytes. Existing metrics like agg time, build time, duration continue to work.
• New regression test GlutenStageAssignment.spec.ts with a real fixture exercising the full stage-attribution pipeline against a Gluten plan.

JVM Plugin

• Accelerator.scala (new, in shared plugin/ module): registry of known accelerators (Gluten/Velox, Comet/DataFusion, RAPIDS GPU, Photon) — each with its detection classes and the set of Spark *Exec operators it substitutes. Accelerator.active probes the classpath once and caches the result.
• DataFlint runs last via postColumnarTransitions: ApplyColumnarRulesAndInsertTransitions applies pre rules in registration order then post rules in reverse order. The DataFlint plugin auto-registers its SessionExtension first, so its columnar rule sits at index 0 and naturally runs last in post. By that point every accelerator has already swapped FilterExec → FilterExecTransformer etc. in pre, and DataFlint's Spark-class match list misses the transformer variants naturally — no explicit skip list needed, robust to plugin registration order.
• Idempotence preserved (!isInstanceOf[TimedExec] guard) so the rule re-runs cleanly under AQE.

Examples / Docker

• docker/gluten/: runnable end-to-end script that downloads the Gluten nightly bundle (cached), builds the DataFlint UI/plugin/example, copies jars into the Docker context, brings up docker compose. GlutenVeloxExample.scala exercises filter/project/agg/sort/joins/window/explode/union with DataFlint instrumentation enabled. Spark UI on localhost:10000.
• docker/comet/: mirrors the Gluten setup for Apache DataFusion Comet 0.4.0. Comet's Maven jar bundles Linux x86_64/aarch64 natives in a single artifact; macOS users go through the Linux container.
• Both run scripts pick the example jar by mtime (ls -t ... | head -1) so stale sbt-dynver outputs in target/ don't get used.

How it solves the coexistence problem

TimedExec is transparent on Spark 3.2+ (children = child.children), which hides the wrapped operator from any other rule's plan traversal. Before this PR, wrapping FilterExec in preColumnarTransitions made the accelerator's rule walk past it — the query silently fell back to the Spark JVM path. Moving the wrap to postColumnarTransitions exploits Spark's reverse-iteration of post rules, so the accelerator substitutes operators first and DataFlint instruments whatever's left (fallback ops, writes, Python UDFs).

Test plan

• UI unit tests pass (incl. GlutenStageAssignment.spec.ts with a real Gluten fixture)
• Plugin unit tests pass — DataFlintPythonExecSpec, DataFlintWindowExecSpec, DataFlintSqlNodesSpec, TimedExecMetricsSpec
• ./docker/gluten/run-gluten-example.sh brings up SparkUI at localhost:10000, log shows Gluten/Velox detected, Velox ops execute, DataFlint wraps the fallback scans
• ./docker/comet/run-comet-example.sh brings up SparkUI, log shows Comet/DataFusion detected, Comet native runtime initializes, DataFlint wraps the fallback ops
• Visual smoke test in the SqlFlow view: accelerator badges show on Velox/Comet/Gpu/Photon nodes, exchange split write/read renders correctly, codegen IDs map to stages
• No regressions on standard (non-accelerated) Spark plans

🤖 Generated with Claude Code

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ minskya
❌ menishmueli
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[menishmueli]

Screenshots

Status Page — GlutenVeloxExample with 16 SQL Queries

The DataFlint status page correctly identifies the Gluten/Velox application.

GroupBy Aggregation (SQL 3) — Two stages with split exchange

Shows Stage 5 (write side) and Stage 7 (read side) with the ColumnarExchange visually split. All Velox-accelerated nodes have orange Velox badges. Native metrics like Aggregation Time and Peak Memory are displayed.

Window Functions (SQL 10) — Three stages with window partition fields

Shows Stage 47, Stage 49, and Stage 52 with two split ColumnarExchange nodes. Window nodes display Partition Fields (play_name), Sort Fields, and Select Fields parsed from the physical plan.

---

Note: To view live screenshots, run ./docker/gluten/run-gluten-example.sh and visit http://localhost:4000

[Copilot]

Pull request overview

Adds Apache Gluten/Velox (and related accelerators) support to the DataFlint UI’s SQL plan flow view, including new node classifications/labels, metrics, accelerator badges, and stage-assignment improvements, plus a runnable Docker + example app environment for validation.

Changes:

• Extend SQL node typing/renaming and metric handling to cover Gluten/Velox operators and Velox-native metrics.
• Improve stage identification for Gluten WholeStageCodegenTransformer and broaden exchange/stage-boundary handling.
• Add UI accelerator badges and a Docker + Scala example app for running/validating Gluten/Velox plans.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
spark-ui/src/reducers/SqlReducerUtils.ts	Add Velox/Gluten node mappings, metric allowlists/transformers/renames, exchange/aggregate lists, and accelerator lookup.
spark-ui/src/reducers/SqlReducer.ts	Add fallback plan-description parsing, Gluten codegen-id inference, and extend exchange/broadcast metric handling.
spark-ui/src/reducers/PlanParsers/ProjectParser.ts	Support parsing ProjectExecTransformer.
spark-ui/src/reducers/PlanParsers/hashAggregateParser.ts	Support alternate Keys: / Functions [...] formats.
spark-ui/src/reducers/PlanParsers/FilterParser.ts	Support parsing FilterExecTransformer.
spark-ui/src/reducers/PlanParsers/ExchangeParser.ts	Broaden exchange type parsing to include Columnar/Broadcast exchange variants.
spark-ui/src/reducers/PlanGraphUtils.ts	Extend “skip-through” node list for input-row backtracking with Velox/Gluten node names.
spark-ui/src/reducers/tests/GlutenStageAssignment.spec.ts	New test validating stage assignment with a real Gluten fixture.
spark-ui/src/reducers/tests/gluten-sql4-fixture.json	New fixture data for Gluten stage assignment test.
spark-ui/src/components/SqlFlow/StageNode.tsx	Show accelerator badge (Velox/Photon/RAPIDS/DataFusion) on nodes.
spark-ui/src/components/SqlFlow/SqlLayoutService.ts	Treat Columnar/Comet/GPU exchanges as split exchange nodes in layout.
spark-plugin/example_3_5_1/src/main/scala/io/dataflint/example/GlutenVeloxExample.scala	New Scala example app exercising common operators under Gluten/Velox.
spark-plugin/example_3_5_1/src/main/scala/io/dataflint/example/DataFusionCometExample.scala	Comment out a hardcoded local-path example line.
docker/gluten/run-gluten-example.sh	Automation script to build artifacts, download Gluten bundle, and run Docker.
docker/gluten/Dockerfile	New container image to run Spark + Gluten/Velox + DataFlint example.
docker/gluten/docker-compose.yml	Compose definition for running the Gluten example container.
docker/gluten/.gitignore	Ignore generated jars/test data/event logs for the Docker example.

Comments suppressed due to low confidence (1)

spark-ui/src/reducers/SqlReducer.ts:824

• In calcBroadcastExchangeDuration, the time to build / time to collect durations are currently not added to duration because they’re on a separate statement starting with unary +, so the function always returns only time to broadcast. Combine these into a single summed expression (or duration += ...) so the returned duration reflects all broadcast phases.

  if (nodeName === "BroadcastExchange" || nodeName === "ColumnarBroadcastExchange") {
    const duration = getMetricDuration("time to broadcast", metrics) ?? 0;
    +(getMetricDuration("time to build", metrics) ?? 0) +
      (getMetricDuration("time to collect", metrics) ?? 0);
    return duration;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[minskya]

Re Copilot's low-confidence note about calcBroadcastExchangeDuration at SqlReducer.ts:824 — that one's a real bug, not low-confidence. Lines 822-823 are a dead expression statement (+(getMetricDuration(...))...) with the unary + and + doing nothing because the previous statement already terminated at ;. Only time to broadcast was being returned. Fixed in upcoming commit by chaining all three durations into a single sum.