IN LIST: retain short floating-point and fixed-size binary lists - #25187
geoffreyclaude wants to merge 2 commits into
Conversation
0186b14 to
ef1ad44
Compare
ef1ad44 to
11b0a61
Compare
11b0a61 to
41aaaa1
Compare
41aaaa1 to
46e59a1
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25187 +/- ##
==========================================
+ Coverage 82.60% 82.66% +0.06%
==========================================
Files 1145 1147 +2
Lines 444057 446437 +2380
Branches 444057 446437 +2380
==========================================
+ Hits 366796 369050 +2254
- Misses 54996 54998 +2
- Partials 22265 22389 +124 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
46e59a1 to
d0ec00e
Compare
688a97d to
04a830c
Compare
04a830c to
499119b
Compare
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=2/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=3/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=2/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing 499119b (499119b) to bec8193 diff Run configurationrun benchmark in_list_rewrite
env:
BENCH_FILTER: "list=3/"
baseline:
ref: "bec81934341766b273dab7012ed127015d946cc6"
changed:
ref: "499119b2edbdecf40ce98fea090f50be6499750b"CPU Details (lscpu)Details
Resource Usagein_list_rewrite — base (merge-base)
in_list_rewrite — branch
File an issue against this benchmark runner |
499119b to
4a5d1a4
Compare
4a5d1a4 to
6c57810
Compare
Which issue does this PR close?
Rationale for this change
Expanding two- and three-item constant membership predicates into comparison chains prevents them from using DataFusion's optimized
INfilters. Paired expression benchmarks show broad gains from retaining these lists onFloat32,Float64, and fixed-size binary columns with widths 1, 2, 4, 8, and 16.The policy compares the cost of both alternatives. Types that share a membership kernel can have different comparison-chain costs: fixed-size binary comparisons operate on byte sequences, while native integers have inexpensive primitive comparisons and can benefit from short-circuiting. The benchmark measures neighboring representations directly to establish this boundary.
What changes are included in this PR?
INandNOT INlists onFloat32,Float64, and fixed-size binary columns of widths 1, 2, 4, 8, and 16 when the literals match the column type.in_list_rewrite, a paired comparison benchmark covering twelve representations. Shared typed SQL parameters keep setup compact and ensure that each expression evaluates the named type. Physical shapes, operand types, and equal outputs are checked before timing.What is the testing strategy for this PR?
The benchmark has 336 paired scenarios and 672 Criterion functions. All twelve representations cover misses, balanced matches, and uniform or skewed first-item hits. Floating-point and fixed-size binary candidates additionally cover nullable input, 64-row and single-row batches, and NULL list values. An unaligned
FixedSizeBinary(16)batch exercises the per-evaluation alignment copy. The benchmark lives indatafusion/core/benchesto plan both candidate expressions through SQL andSessionContext.Validation passed for
cargo fmt --all,cargo clippy --all-targets --all-features -- -D warnings, and./dev/rust_lint.sh. The extended workspace suite passed 12,317 Rust tests and all 526 SQL logic files. Focused optimizer and IN-list SQL logic checks also passed.All 672 benchmark functions passed their shape, type, and output checks on both the baseline and updated implementation. The full baseline matrix and 23 paired scenarios on the updated implementation were timed.
The TPC-H SQL logic check also passed with its existing plan snapshots.
Are there any user-facing changes?
Optimized plans retain eligible short membership predicates, enabling the specialized filters for the measured floating-point and fixed-size binary representations.
Performance evidence
Local measurements used an x86_64 AMD EPYC VM, Rust 1.98.1, and the
release-nonltoprofile. The complete paired matrix was measured onmainat4d167a167with the benchmark applied. Selected cases were repeated on the final implementation with a 500 ms warmup and a two-second measurement window. Planning is performed before timing; the measurements are physical-expression evaluation costs.For three-item
INon 8,192 rows, the longer repeats produced the following expression times. Each cell shows chain → IN, in microseconds:The
FixedSizeBinary(16)unaligned batch improved from 185.2 µs to 30.7 µs (6.0×). The UInt64 reference illustrates why rewrite decisions depend on comparison costs: its comparison chain has a much cheaper first-hit path.Uniform-hit Float64 cases have ranged from approximately tied in an earlier x86 repeat (overlapping 95% confidence intervals) to moderately faster in this repeat. Mixed-match gains are larger; the policy follows the broader workload matrix.
The posted AArch64 measurements also favor membership filters for
Float64andFixedSizeBinary(8)across the original nine workload profiles: two-item results and three-item results.Run the comparison with: