chore(deps): migrate DataFusion 53 → 54 #467

Problem

GraphForge is pinned to DataFusion 53 (Cargo.toml workspace datafusion = "53"). Dependabot opened #464 for 54.1.0, but a bump-only PR does not compile: catalog adapters still target the 53 trait surface, and companion crates stayed on 53, producing a dual datafusion_session graph.

Until this lands, Dependabot will keep failing majors (currently ignored via @dependabot ignore this major version on #464).

Objective

Ship a single, intentional upgrade to DataFusion 54.x (and matching datafusion-* companions) so graphforge-storage catalog providers, Rust quality, bindings, and Bazel drift checks are green on the new major.

Debt / regime

• Debt type: development / architecture (upstream API break + workspace pin hygiene)
• Quality regime: A (compute) — correctness of catalog/SQL execution over product UX

Evidence from #464 (fail-closed)

From CI on dependabot/cargo/datafusion-54.1.0 (Test Suite run 31201787417):

1. as_any removed from provider traits (E0407) in crates/graphforge-storage/src/catalog.rs for:
   • TableProvider: TopologyNodeTable, TypedEdgeTable, UnionEdgeTable, EdgePropertyTable, PropertyTable
   • SchemaProvider: GraphSchema
   • CatalogProvider: GraphCatalog
2. scan signature / Session trait mismatch (E0053 / E0308):
   • expected datafusion::catalog::Session, found datafusion_catalog::Session
   • note: multiple different versions of crate datafusion_session in the dependency graph
   • root cause on that PR: datafusion moved to 54 while datafusion-catalog = "53" (and related pins) stayed on 53

Upstream guide: DataFusion 54.0.0 upgrade — remove as_any impls; downcast via trait-object downcast_ref / is instead of .as_any().downcast_ref().

Requirements

1. Bump all DataFusion workspace/companion pins together to 54.x (no mixed 53/54 graph):
   • workspace: datafusion, datafusion-datasource
   • crate pins: datafusion-catalog, datafusion-functions-aggregate (and any other direct datafusion-* version literals found by rg)
2. Update GraphForge catalog adapters in crates/graphforge-storage/src/catalog.rs:
   • delete fn as_any from TableProvider / SchemaProvider / CatalogProvider impls
   • fix scan to use the 54 Session type from the unified graph
   • migrate provider downcasts in tests/helpers (provider.as_any().is::<…>() / schema.as_any().downcast_ref::<GraphSchema>()) to DF54 trait-object downcast APIs
3. Fix any follow-on compile breaks in DF consumers (graphforge-exec, graphforge-rel, graphforge-plan, graphforge-api) revealed by the unified bump
4. Regenerate legal notices + Bazel Cargo fingerprint/lock as required by repo policy:
   • python3 scripts/generate_third_party_notices.py
   • python3 scripts/ci/cargo-bazel-drift-check.py --write
   • CARGO_BAZEL_REPIN=1 bazelisk build --repo_env=CARGO_BAZEL_REPIN=1 //:first_party_libs
5. Keep Arrow array .as_any().downcast_ref::<…Array>() call sites unless Arrow itself requires a change (those are not DF provider as_any)

Acceptance criteria

• Workspace + direct crate pins are consistently on DataFusion 54.x (cargo tree -i datafusion-session shows a single major)
• cargo check / Clippy workspace green with catalog adapters compiling under DF54 traits
• Existing catalog/provider unit tests in graphforge-storage updated and passing
• Rust Quality + Python/Node binding jobs green on the PR
• Repository Policy notices check green; Bazel drift/bootstrap green (cargo-bazel-lock.json + fingerprint committed)
• ADR/docs that assert “DataFusion 53.x” as current ground truth are updated or explicitly annotated as historical (at least docs/adr/0008-heterogeneous-lists.md)

BDD completion scenarios

1. Given a workspace on DataFusion 54.x, when cargo tree -i datafusion-session is run, then only one major version appears (no 53+54 mix).
2. Given a GraphForge project catalog (GraphCatalog / table providers), when Rust quality and storage catalog tests run, then provider registration and scan paths succeed under DF54 Session/TableProvider APIs.
3. Given the upgrade PR, when CI Gate runs, then Rust Quality, bindings, Repository Policy, and Bazel Bootstrap are green at the exact head SHA.
4. Given Dependabot later opens a DF54 patch bump, when it only touches lockfiles within 54.x, then it is eligible to merge without repeating this adapter migration.

Testing

• Scenario 1: cargo tree -i datafusion-session (and companion crates) on the PR branch
• Scenario 2: cargo test -p graphforge-storage (catalog/provider coverage) + any graphforge-exec / graphforge-api tests that construct sessions/catalogs
• Scenario 3: required GitHub checks on the PR (CI Gate CLEAN)
• Scenario 4: observational — leave DF major ignore in place until this issue closes; afterward Dependabot majors can be re-enabled for 55+ only

Local gates appropriate to the Rust surface (from AGENTS.md):

cargo fmt --all -- --check
cargo clippy --workspace -- -D warnings
cargo test -p graphforge-storage
python3 scripts/ci/cargo-bazel-drift-check.py
# optional heavy: make bazel-test / //:ci_rust_tests

Documentation

• Update or annotate DF-version-grounded prose in docs/adr/0008-heterogeneous-lists.md
• Link the upstream 54.0.0 upgrade guide from the PR body
• No user-facing API doc change expected unless public Rust facade types leak DF types (call out if discovered)

Observability / security

• No new runtime telemetry required; failures should surface as compile/CI errors
• No auth/privacy surface; treat upgrade as correctness-preserving for query/catalog behavior

Non-goals

• Arrow major upgrades unrelated to what DF54 requires
• Adopting new DF54 features (higher-order UDFs, new optimizer rules) beyond what compiles/tests need
• Performance retuning / M4 baselines (#334 / #335) — re-run only if behavior or plans regress in existing tests
• Re-opening Dependabot chore(deps): bump datafusion from 53.1.0 to 54.1.0 #464 as-is (prefer a human PR that bumps all pins + adapters together)

Likely touchpoints

• Cargo.toml (workspace deps)
• crates/graphforge-storage/Cargo.toml (datafusion-catalog)
• crates/graphforge-rel/Cargo.toml (datafusion-functions-aggregate)
• crates/graphforge-storage/src/catalog.rs (primary adapter work)
• crates/graphforge-{exec,rel,plan,api}/ (follow-on compile fixes)
• Cargo.lock, cargo-bazel-lock.json, tools/bazel/drift/cargo_feature_fingerprint.json
• legal/THIRD_PARTY_NOTICES.md (+ packaging copies)
• docs/adr/0008-heterogeneous-lists.md

Related

• Closed Dependabot PR: #464
• Upstream: DF 54 upgrade guide, as_any removal on providers
• Sibling ignored major: TypeScript 7 tracking #468

Open questions

• None blocking implementation; start from a branch that bumps every datafusion* pin to 54.x in one commit, then fix catalog adapters.