Skip to content

fix: use nested loop join for nested piecewise merge keys - #26043

Open
mikamikasuki wants to merge 1 commit into
apache:mainfrom
mikamikasuki:fix/25957-nested-piecewise-merge-join
Open

mikamikasuki wants to merge 1 commit into
apache:mainfrom
mikamikasuki:fix/25957-nested-piecewise-merge-join

Conversation

@mikamikasuki

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

With piecewise merge joins enabled, nested range-key comparisons can return incorrect results when an array contains an inner NULL. Struct range keys can also fail with an unsupported-sort error. Both cases should preserve SQL comparison semantics and execute successfully.

What changes are included in this PR?

  • Fall back to NestedLoopJoinExec whenever either piecewise merge join range key has a nested datatype.
  • Update expected plans for existing nested-key right semi join cases.
  • Add SQL logic coverage for an array comparison containing an inner NULL and for a Struct comparison.

What is the testing strategy for this PR?

  • cargo test --test sqllogictests -- piecewise_merge_join_nested_nulls
  • cargo test --test sqllogictests -- pwmj
  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings
  • Full extended workspace tests from the repository instructions: RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
  • git diff --check

Are there any user-facing changes?

Nested range joins return SQL-correct results when nested values contain NULLs, and Struct range keys no longer fail with an unsupported-sort error. There are no public API changes.

@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Oct 5, 2026

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mikamikasuki

@jayzhan211
jayzhan211 enabled auto-merge October 5, 2026 13:53
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.66%. Comparing base (c3ef346) to head (b897fd9).
⚠️ Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/core/src/physical_planner.rs 78.57% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26043      +/-   ##
==========================================
- Coverage   82.66%   82.66%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      446357   446395      +38     
  Branches   446357   446395      +38     
==========================================
+ Hits       368971   369002      +31     
- Misses      54997    55001       +4     
- Partials    22389    22392       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jayzhan211
jayzhan211 added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026
@2010YOUY01

Copy link
Copy Markdown
Contributor

Thanks for the fix.

The root cause seem to be sorting and comparison don't agree in PWMJ, is it possible to fix that directly?

If that's tricky, this PR's approach totally makes sense, but we would keep the issue open, and add a note for the remaining fix to the root cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PiecewiseMergeJoin returns wrong results for List keys with NULL elements and fails on Struct keys

4 participants