Repository navigation
Prune hash join probe side via a build-side key range bitmap - #25602
Merged
jayzhan211 merged 6 commits intoOct 2, 2026
Merged
Conversation
github-actions
Bot
added
sqllogictest
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25602 +/- ##
=========================================
Coverage 82.60% 82.61%
=========================================
Files 1144 1146 +2
Lines 441971 443110 +1139
Branches 441971 443110 +1139
=========================================
+ Hits 365081 366065 +984
- Misses 54807 54874 +67
- Partials 22083 22171 +88 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
sunchao
reviewed
Sep 22, 2026
sunchao
left a comment
sunchao
left a comment
Member
There was a problem hiding this comment.
Thanks for working on this. I compared adbb3ce with base 714956b and found two regressions, detailed inline. Both have executable base/head reproductions. The 109 pruning unit tests and 64 ordinary SQL cases per revision passed; the separate nullable-CASE and memory-limit reproducers below fail only on head.
Comment on lines
+1875
to
+1880
| // A missing `ELSE` yields NULL, which never matches, so it adds nothing. | ||
| return case | ||
| .when_then_expr() | ||
| .iter() | ||
| .map(|(_, then)| then) | ||
| .chain(case.else_expr()) |
Member
There was a problem hiding this comment.
[P1] Preserve the implicit NULL branch during full-match inference
Omitting the missing ELSE is conservative for ordinary filtering, but this rewriter is also used to build the inverse predicate that proves an entire Parquet row group matches. That makes this change return incorrect rows when datafusion.execution.parquet.pushdown_filters=true.
I reproduced this with required Int64 columns (a,b), two one-row groups containing (2,1) and (1,2), and:
SELECT * FROM t
WHERE NOT (CASE WHEN a = 1 THEN false END) AND b > 0
ORDER BY a;Base 714956b3 returns only (1,2); head adbb3ce4 also returns (2,1), although its predicate result is NULL. With LIMIT 1 instead of ORDER BY, head returns the invalid (2,1). Disabling filter pushdown restores the correct result.
The forward rewrite falls back for NOT CASE, so it still permits full-match inversion. The inverse exposes CASE ... OR b <= 0; dropping the implicit NULL branch lets it prune both groups, which are then marked fully matched and skip row filtering. Head reports two fully matched groups and row_filter_skipped_fully_matched=1.
Could we keep CASE expressions without an explicit ELSE conservatively unhandled here, and add an end-to-end regression test? The join-generated CASE already has an explicit ELSE. Merely setting has_filter_semantics_only inside this branch would miss the forward NOT CASE path, which never descends into it.
Contributor
Author
There was a problem hiding this comment.
Good point, added a bail-out for that case as well as an SLT.
Comment on lines
+3113
to
+3114
| let pruning_bitmap = match (left_values.as_slice(), bounds.as_ref()) { | ||
| ([keys], Some(bounds)) if !keys.is_empty() => bounds |
Member
There was a problem hiding this comment.
[P2] Skip bitmap construction when dynamic filtering is inactive
bounds can exist solely for perfect-hash-join candidacy even when should_compute_dynamic_filters is false. This branch still constructs and reserves a pruning bitmap before those bounds are cleared below, so a join with dynamic filtering disabled can now fail for memory that provides no pruning benefit.
Using a native CollectLeft join with 151 Int64 build keys i * 10_000 (i = 0..151), probe keys [0, 500_000, 1_500_000], and enable_join_dynamic_filter_pushdown=false (otherwise default configuration), base 714956b3 succeeds in a 100,000-byte memory pool with 6,220 bytes reserved. Head adbb3ce4 fails with ResourcesExhausted requesting another 131,072 bytes. At a 1,000,000-byte limit both return the exact expected rows, but head reserves 137,292 bytes. I also reproduced the same failure with a Full join.
Could we gate this bitmap construction on should_compute_dynamic_filters and cover the disabled-filter case with a memory-limit regression test?
Contributor
Author
There was a problem hiding this comment.
Nice catch. I also noticed that the same problem happens for the InList path too.
So instead of guarding just the (bit)map path with should_compute_dynamic_filters, i made it guard both by extending the PushdownStrategy::Empty arm with || !should_compute_dynamic_filters; let me know if you see issues with this.
Also a unit test added.
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 23, 2026 07:20
6c48fab to
a6d5391
Compare
jayzhan211
reviewed
Sep 23, 2026
Contributor
|
run benchmarks |
Contributor
|
I think we need benchmark number |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing hash-join-dynamic-pruning-bitmap (a6d5391) to 7570366 (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 24, 2026 05:30
90415e6 to
ed6ba05
Compare
Contributor
Author
Looks like there's a genuine improvement for TPCH Q18, but it's a side-effect of the changes not really benefiting from the bitmap pruning in that case. That side-effect is wiring the |
Contributor
|
Per file, the adapter often rewrites the probe child to - let column = column_expr.downcast_ref::<phys_expr::Column>()?;
+ let column = match column_expr.downcast_ref::<phys_expr::CastExpr>() {
+ Some(cast) => {
+ let column = cast.expr().downcast_ref::<phys_expr::Column>()?;
+ let from = schema.fields().get(column.index())?.data_type();
+ if !is_integer_widening(from, cast.cast_type()) {
+ return None;
+ }
+ column
+ }
+ None => column_expr.downcast_ref::<phys_expr::Column>()?,
+ };/// Casts that keep the bitmap's `u64` key ordering (sign/zero extension).
fn is_integer_widening(from: &DataType, to: &DataType) -> bool {
use DataType::*;
from == to
|| matches!(
(from, to),
(Int8, Int16 | Int32 | Int64)
| (Int16, Int32 | Int64)
| (Int32, Int64)
| (UInt8, Int16 | Int32 | Int64 | UInt16 | UInt32 | UInt64)
| (UInt16, Int32 | Int64 | UInt32 | UInt64)
| (UInt32, Int64 | UInt64)
)
} |
jayzhan211
approved these changes
Sep 24, 2026
jayzhan211
left a comment
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
A non-blocking issue left
Contributor
Author
Makes sense; added conditional casting unwrapping in both |
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
3 times, most recently
from
September 25, 2026 07:28
0908a99 to
1012903
Compare
Contributor
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 27, 2026
Contributor
|
@gruuya . It seems the extended tests failed |
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 28, 2026 07:50
1012903 to
1dec8d6
Compare
Contributor
Author
Rebased and resolved, thanks. |
Contributor
Author
|
@sunchao @jayzhan211 we needed one more fix since #25491 clashed with this: for the new prepared builds we now always compute the bitmap in advance since we don't know whether or not we'll need it down the line. Let me know if this sounds ok. |
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 30, 2026 05:29
3d99dd9 to
8258fc5
Compare
Contributor
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 30, 2026
Map the build side's keys onto a fixed-size bitmap over their range and test container min/max against it, so a scan can skip containers whose values fall in the gaps between keys. The bitmap is bounded at 128 KiB regardless of build side size, and one whose buckets are all set is discarded, so a contiguous key set costs nothing. The key range and distinct count already exist for the bounds predicate, so a dense key set is ruled out before allocating. Integer keys only; other types go unpruned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
Co-authored-by: Jay Zhan <jayzhan211@gmail.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Skipping unused pruning state made membership depend on `should_compute_dynamic_filters`, but a prepared build doesn't know whether a future consuming join will attach a dynamic filter, so it must always compute it. Also updates a pre-existing test whose byte accounting assumed a bitmap is never added on top of the map. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
gruuya
force-pushed
the
hash-join-dynamic-pruning-bitmap
branch
from
September 30, 2026 14:18
8258fc5 to
324f67f
Compare
Under `force_hash_collisions` every key hashes to the same bucket, so Map::num_of_distinct_key() collapses to 1 and the IN-list threshold check no longer reflects the test's real 200 distinct keys, sending it down the IN-list branch instead of the Map+bitmap branch under test. Force the threshold off directly instead of relying on distinct-key counting to cross it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Author
|
@jayzhan211 fixed the |
rgbuilds
added a commit
to rgbuilds/datafusion
that referenced
this pull request
Oct 6, 2026
Remove the idle Bloom pruning metric from the hash-join regression introduced by apache#25602. Statistics retain two of four row groups, while the fixture has no Bloom filters and records no Bloom pruning outcome.
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
…25602) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#25291. - Alternative to apache#25292. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> Avoid scanning redundant files/row-groups/pages in the probe side of hash joins, based on the values dictated by the build side. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> - wiring up PushdownStrategy::Map/HashTableLookupExpr to carry a representation of the build side values from a hash join, through - `KeyRangeBitmap` implementation, which maps the build-side values array into a finite-sized bitmap bucket, and can answer probing questions for certain ranges - extend `build_predicate_expression` to build the associated pruning expression from `HashTableLookupExpr` using the new `KeyRangeBitmapPruningExpr`, which implements `PhysicalExpr` on top of `KeyRangeBitmap` - also extend `build_predicate_expression` so that it now pushes down pruning for `CaseExprs`, since that also unlocks the partitioned hash-join scenario this pr targets ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> A number of unit tests added, and one SLT added. Also verified on the query shape that motivated the original issue ```sql > copy (select i as k, random() as v from generate_series(0, 1999999) t(i)) to '/tmp/fact.parquet' stored as parquet options ('format.max_row_group_size' '1000'); +---------+ | count | +---------+ | 2000000 | +---------+ 1 row(s) fetched. Elapsed 0.093 seconds. > create external table fact stored as parquet location '/tmp/fact.parquet'; 0 row(s) fetched. Elapsed 0.008 seconds. > create table dim as select i as k from generate_series(0, 1999999) t(i) where i % 10000 < 200; 0 row(s) fetched. Elapsed 0.014 seconds. > select count(*), sum(v) from fact join dim on fact.k = dim.k; select count(*), sum(v) from fact join dim on fact.k = dim.k; select count(*), sum(v) from fact join dim on fact.k = dim.k; select count(*), sum(v) from fact join dim on fact.k = dim.k; +----------+--------------------+ | count(*) | sum(fact.v) | +----------+--------------------+ | 40000 | 20121.308621485692 | +----------+--------------------+ 1 row(s) fetched. Elapsed 0.031 seconds. +----------+--------------------+ | count(*) | sum(fact.v) | +----------+--------------------+ | 40000 | 20121.308621485692 | +----------+--------------------+ 1 row(s) fetched. Elapsed 0.017 seconds. +----------+-------------------+ | count(*) | sum(fact.v) | +----------+-------------------+ | 40000 | 20121.30862148569 | +----------+-------------------+ 1 row(s) fetched. Elapsed 0.011 seconds. +----------+-------------------+ | count(*) | sum(fact.v) | +----------+-------------------+ | 40000 | 20121.30862148569 | +----------+-------------------+ 1 row(s) fetched. Elapsed 0.011 seconds. > explain analyze select count(*), sum(v) from fact join dim on fact.k = dim.k; ... | | DataSourceExec: file_groups={12 groups: [[tmp/fact.parquet:0..1927333], [tmp/fact.parquet:1927333..3854666], [tmp/fact.parquet:3854666..5781999], [tmp/fact.parquet:5781999..7709332], [tmp/fact.parquet:7709332..9636665], ...]}, projection=[k, v], output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=DynamicFilter [ k@0 >= 0 AND k@0 <= 1990199 AND hash_lookup ], dynamic_rg_pruning=eligible, pruning_predicate=k_null_count@1 != row_count@2 AND k_max@0 >= 0 AND k_null_count@1 != row_count@2 AND k_min@3 <= 1990199 AND k_null_count@1 != row_count@2 AND KEY_RANGE_BITMAP(k_min@3, k_max@0, 20000/1048576 buckets), required_guarantees=[], metrics=[output_rows=200.0 K, elapsed_compute=9.65ms, output_bytes=25.0 MB, output_batches=200, files_ranges_pruned_statistics=12 total → 12 matched, row_groups_pruned_statistics=2.00 K total → 200 matched, row_groups_pruned_bloom_filter=200 total → 200 matched, page_index_pages_pruned=200 total → 200 matched, page_index_rows_pruned=200.0 K total → 200.0 K matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, bytes_processed=22.1 MB, bytes_scanned=2.1 MB, file_open_errors=0, file_scan_errors=0, files_opened=12, files_processed=12, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_pruned_dynamic_filter=0, predicate_cache_inner_records=0, predicate_cache_records=0, bloom_filter_eval_time=64.23µs, metadata_load_time=345.27µs, page_index_eval_time=1.33ms, row_pushdown_eval_time=36ns, statistics_eval_time=896.06µs, time_elapsed_opening=6.99ms, time_elapsed_processing=24.90ms, time_elapsed_scanning_total=224.99ms, time_elapsed_scanning_until_data=25.01ms, output_rows_skew=1.19%, scan_efficiency_ratio=9.7% (2.24 M/23.13 M)] | ... ``` ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Avoid scanning redundant files/row-groups/pages in the probe side of hash joins, based on the values dictated by the build side.
What changes are included in this PR?
KeyRangeBitmapimplementation, which maps the build-side values array into a finite-sized bitmap bucket, and can answer probing questions for certain rangesbuild_predicate_expressionto build the associated pruning expression fromHashTableLookupExprusing the newKeyRangeBitmapPruningExpr, which implementsPhysicalExpron top ofKeyRangeBitmapbuild_predicate_expressionso that it now pushes down pruning forCaseExprs, since that also unlocks the partitioned hash-join scenario this pr targetsWhat is the testing strategy for this PR?
A number of unit tests added, and one SLT added.
Also verified on the query shape that motivated the original issue
Are there any user-facing changes?