Merge upstream DataFusion 55.2.0-rc1 into spiceai-55

[krinart]

Summary

Merges upstream apache/datafusion 55.2.0-rc1 (500744d5c25499055e3b2ca46b022bc425989203, the head of branch-55) into spiceai-55 (02550cf9462b9f56506b6b1cd07eabf08c46e200). This takes the same route as the earlier patch releases on this line (52.2–52.5, 54.1, 55.1): a merge of the upstream tag rather than a re-cut, so every Spice patch stays a separate commit.

55.2.0-rc1 has no final 55.2.0 tag yet. If the release vote succeeds, the 55.2.0 tag should point at this same commit. If an rc2 is cut instead, merge it into this branch before pinning.

Upstream changes (55.1.0..55.2.0-rc1, 8 commits)

Commit	Change
5e62bc399	fix: preserve fetch across distribution reoptimization (apache#24809, apache#25821)
c1514b53e	fix: remove serde_json/preserve_order from library crates (apache#25884, apache#25915)
5c4db5a90	Reduce datafusion-spark dependency footprint (apache#24351, apache#25277)
ec747460f	Preserve field metadata when coercing INT96 timestamps (apache#24790, apache#25895)
cf7cfd188	fix: update rustls for RUSTSEC-2026-0285 (apache#25309, apache#25514)
180db4c1d, 9f5714b0c	CI-only (MinIO image source, rustfs for datafusion-cli tests)
500744d5c	version bump to 55.2.0 and changelog

None of the 8 was on spiceai-55 before (git cherry -v origin/spiceai-55 55.2.0-rc1 55.1.0 marks all 8 +).

Conflict resolution

Cargo.lock was the only conflict. The merge keeps this branch's lockfile, because upstream's lacks the fork's own dependencies (for example vortex-datafusion under datafusion-proto). It then:

• re-resolves it for the 55.2.0 workspace versions (cargo update -w);
• carries upstream's rustls bump from cf7cfd188: rustls 0.23.45, rustls-webpki 0.103.15, aws-lc-rs 1.18.1, aws-lc-sys 0.45.0.

Compared with spiceai-55, the lockfile now differs only in the DataFusion crate versions and those four packages, which are the same four upstream moved. cargo metadata --locked resolves.

The datafusion-spark quote.rs import fix this branch carried (4524751d, 4085e114) is now upstream via 5c4db5a90. The merge left that file unchanged.

Why this matters downstream

5e62bc399 fixes a wrong-results bug in Ballista's static distributed planner. That planner re-runs EnsureRequirements below each shuffle boundary, and the rerun dropped a fetch held on a SortPreservingMergeExec. The repro below drives ballista_scheduler::planner::DefaultDistributedPlanner (spiceai/datafusion-ballista@a7c4c585) over the optimized plan of:

SELECT k, count(*) AS c FROM (SELECT k, v FROM t ORDER BY v LIMIT 5) s GROUP BY k ORDER BY k

It uses a 100-row table in 4 partitions and executes the first stage. The correct number of rows past LIMIT 5 is 5.

DataFusion	Stage 1 plan above TopK(fetch=5)	Rows counted past LIMIT 5
spiceai-55 @ 02550cf	SortPreservingMergeExec … fetch=5 removed	20
this branch's tree	SortPreservingMergeExec: [v@1 ASC NULLS LAST], fetch=5 kept	5

Single-node execution of the same query returns the correct (0,2), (1,2), (2,1) on both.

c1514b53e: a downstream crate that got serde_json/preserve_order only through DataFusion loses it with this merge. Its serde_json::Map then serializes keys sorted instead of in insertion order. Downstream crates that depend on key order need to opt in themselves.

Verification

All runs on 2026-10-05, macOS arm64. "Merge" is this branch at 47b9ec42a; "base" is 02550cf (spiceai-55). Both ran the same command with the test-data submodules (datafusion-testing, parquet-testing, testing) checked out at the same pinned commits.

Full workspace suite

This is the command the fork CI's cargo test (amd64) job runs, with --no-fail-fast so that every failure is reported:

$ cargo test --profile ci --no-fail-fast --exclude datafusion-examples --exclude ffi_example_table_provider \
    --exclude datafusion-cli --workspace --lib --tests --bins \
    --features serde,avro,json,backtrace,integration-tests,parquet_encryption,substrait

	base 02550cf	merge 47b9ec42a
failing targets	8	the same 8
libtest tests outside core_integration	9796 passed, 94 failed, 4 ignored	9797 passed, 94 failed, 4 ignored
core_integration (re-run with RUST_MIN_STACK=16777216, see below)	1050 passed, 44 failed	1058 passed, 44 failed
sqllogictests	258 errors in 30 .slt files	258 errors in the same 30 files

The failing targets, with identical counts on both sides:

datafusion --lib                                     460 passed; 4 failed
datafusion --test core_integration                  1050 / 1058 passed; 44 failed (16 MiB stack)
datafusion --test parquet_integration                221 passed; 1 failed
datafusion-optimizer --test optimizer_integration     25 passed; 1 failed
datafusion-physical-plan --lib                      1708 passed; 1 failed
datafusion-sql --test sql_integration                747 passed; 82 failed
datafusion-substrait --test substrait_integration    205 passed; 5 failed
datafusion-sqllogictest --test sqllogictests         258 errors in 30 files

Failures were compared one test at a time:

• each libtest test … FAILED, keyed by target;
• each sqllogictests error the runner printed, keyed by file:line;
• the stack-overflow abort.

There are 287 on base and 287 on merge, all 287 in common. None fails only on merge, and none only on base. The sqllogictests runner prints at most 10 errors per file, so 152 of the 258 errors are compared by location; the rest are compared by per-file error count, which is identical for all 30 files.

Tests that succeed on merge but not on base. The first 8 come from 5e62bc399 and the last from ec747460f (git log -S "fn <name>" 02550cf..47b9ec42a):

core_integration :: physical_optimizer::enforce_distribution::move_fetch_to_replacement_sort
core_integration :: physical_optimizer::enforce_distribution::preserve_fetch_below_filter_when_reoptimizing
core_integration :: physical_optimizer::enforce_distribution::preserve_fetch_below_filter_with_constant_ordering
core_integration :: physical_optimizer::enforce_distribution::preserve_fetch_in_nested_distribution_operators
core_integration :: physical_optimizer::enforce_distribution::preserve_fetch_when_parallelizing_sort_above_filter
core_integration :: physical_optimizer::enforce_distribution::preserve_fetch_when_reoptimizing_coalesce_partitions
core_integration :: physical_optimizer::enforce_distribution::preserve_fetch_when_reoptimizing_ordered_merge
core_integration :: physical_optimizer::enforce_distribution::preserve_topk_when_parent_changes_ordering
datafusion_datasource_parquet :: schema_coercion::tests::coerce_int96_to_resolution_preserves_field_metadata

No test succeeds on base but not on merge.

Notes on the pre-existing failures:

• sqllogictests: of the 152 printed errors, 151 are EXPLAIN plan-text mismatches, 114 of which involve the fork's HashJoinExec display accumulator=MinMaxLeftAccumulator. The other one is SHOW ALL VERBOSE in information_schema.slt. None of the printed errors is a mismatch in query results.
• The 138 libtest failures (94 outside core_integration, 44 in it) were not reviewed one by one. The two inspected, enforce_distribution::hash_join_key_ordering and union::tests::test_union_partition_statistics_with_mismatched_nullability, are respectively a plan snapshot without the accumulator= display and a byte-size statistic (69 vs 49).
• core_integration aborts on both base and merge at sql::unparser::bigquery_optimized_recursive_cte_roundtrip (has overflowed its stack) under the default test-thread stack. That leaves the target's results incomplete, so it was re-run on both sides with RUST_MIN_STACK=16777216. With the larger stack the test passes and the target completes; the numbers above come from that re-run.

fetch regression tests, with and without the fix

$ cargo test -p datafusion --test core_integration -- enforce_distribution::preserve_fetch enforce_distribution::move_fetch enforce_distribution::preserve_topk
merge 47b9ec42a:  test result: ok. 8 passed; 0 failed; 1092 filtered out
same tests, physical-optimizer src reverted to 02550cf:
  test result: FAILED. 0 passed; 8 failed
  e.g. preserve_topk_when_parent_changes_ordering  left: [9,8,7,6,5,4,3,2,1,0]  right: [4,3,2,1,0]
       preserve_fetch_when_parallelizing_sort_above_filter  left: 6  right: 3

Other checks (merge)

$ cargo check -p datafusion-spark --no-default-features   -> exit 0
$ cargo fmt --all -- --check                              -> clean
$ cargo metadata --locked --format-version 1              -> ok

Coverage. Not run:

• the datafusion-cli job;
• doc tests (cargo test --doc);
• sqllogictests with INCLUDE_TPCH=true;
• the Postgres-compatibility job;
• clippy;
• a downstream Spice build pinned to this revision.

The repository's Rust workflow does not run on this PR. gh run list --branch spiceai-55-patches-4 shows only Running Copilot Code Review, so nothing on this list ran in CI either.

Follow-ups (not in this PR)

• Patches on spiceai-54 that are not on spiceai-55:
  • the footer-cache hit counter fix (fix(execution): drop an evicted footer's hit counter together with its entry (refs spiceai/spiceai#12952) #245: e4918932, ae5d2299, 73e58151), which has no port yet;
  • metadata-column listing pruning (feat(listing): prune file listing by metadata-column predicates #229), whose port is open as feat(listing): prune file listing by metadata-column predicates (port of #229 to spiceai-55) #240.
• When open spiceai-54 PRs feat(physical-plan): estimate IN lists and OR-of-equalities from NDV #228, fix(unparser): keep a join that is another join's right input on that join's right (refs spiceai/spiceai#14373) #233, fix(unparser): derive a Limit that is a join input under the scan's own name, unconditionally (refs spiceai/spiceai#14375) #234, fix(unparser): keep a qualifier that an UNNEST or FLATTEN alias, a schema-qualified table or a struct field path binds (refs spiceai/spiceai#13442) #246 and fix(unparser): scope a sort key reading a projection output that cannot be repeated (refs spiceai/spiceai#14109) #247 merge, each needs a spiceai-55 port.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The broad merge changes optimizer semantics, dependencies, and containerized CI without full workspace or downstream validation.

Review effort: Balanced
Findings: None

What changed in this PR

Merges upstream DataFusion 55.2.0-rc1 into spiceai-55, retaining fork-specific dependencies and patches.

Changes:

• Bumps workspace crates and security-sensitive TLS dependencies to 55.2.0-compatible versions.
• Fixes optimizer fetch preservation, INT96 metadata coercion, and ordered pgjson output.
• Replaces MinIO with RustFS for CLI storage integration tests and reduces Spark dependencies.

File	Description
docs/​source/​user-guide/​example-usage.md	Updates dependency example to 55.2.0.
docs/​source/​user-guide/​crate-configuration.md	Updates feature configuration example.
docs/​source/​user-guide/​configs.md	Updates generated version default.
docs/​source/​download.md	Updates download dependency example.
dev/​changelog/​55.2.0.md	Adds the 55.2.0 changelog.
datafusion/​spark/​src/​lib.rs	Uses lower-level registry and session APIs.
datafusion/​spark/​src/​function/​table/​mod.rs	Moves TableFunction import to session.
datafusion/​spark/​Cargo.toml	Reduces Spark dependency footprint.
datafusion/​physical-plan/​src/​display.rs	Preserves pgjson field ordering without a global feature.
datafusion/​physical-plan/​Cargo.toml	Replaces preserve_order with direct Serde support.
datafusion/​physical-optimizer/​src/​ensure_requirements/​enforce_sorting/​replace_with_order_preserving_variants.rs	Protects fetched coalesces during sorting rewrites.
datafusion/​physical-optimizer/​src/​ensure_requirements/​enforce_sorting/​mod.rs	Retains fetched coalesce boundaries.
datafusion/​physical-optimizer/​src/​ensure_requirements/​enforce_distribution.rs	Preserves fetch semantics during distribution reoptimization.
datafusion/​expr/​src/​logical_plan/​display.rs	Adds deterministic logical pgjson serialization.
datafusion/​expr/​Cargo.toml	Adds Serde and removes feature-dependent ordering.
datafusion/​datasource-parquet/​src/​schema_coercion.rs	Preserves nested field metadata during INT96 coercion.
datafusion/​core/​tests/​physical_optimizer/​replace_with_order_preserving_variants.rs	Updates fetched-coalesce regression coverage.
datafusion/​core/​tests/​physical_optimizer/​enforce_sorting.rs	Updates the optimized-plan expectation.
datafusion/​core/​tests/​physical_optimizer/​enforce_distribution.rs	Adds fetch-preservation execution tests.
datafusion-cli/​tests/​cli_integration.rs	Migrates S3 integration tests to RustFS.
datafusion-cli/​CONTRIBUTING.md	Documents RustFS-based testing.
datafusion-cli/​Cargo.toml	Removes the MinIO testcontainers feature.
Cargo.toml	Bumps workspace crates and adds Serde.
Cargo.lock	Resolves 55.2.0 and patched TLS dependencies.
.github/​workflows/​rust.yml	Pre-pulls the pinned RustFS image.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.