Skip to content

perf: rewrite decimal-to-integer cast predicates as source-domain ranges - #26216

Draft
discord9 wants to merge 18 commits into
apache:mainfrom
discord9:stack/pr22906-decimal-int-range
Draft

discord9 wants to merge 18 commits into
apache:mainfrom
discord9:stack/pr22906-decimal-int-range

Conversation

@discord9

@discord9 discord9 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Stacked on #22906 (1e127365510c83cfffec5fed6f72a56b5875d5b0); please review/merge that one first. The wrong-rows regression itself is fixed by #22906's allowlist; this PR adds the optimization on top.

Rationale for this change

CAST(decimal_col AS BIGINT) = 173665 is a many-to-one comparison (decimal→integer truncates toward zero), so #22906 correctly refuses to fold it and keeps the cast on the column. That is correct but leaves performance on the table: the predicate cannot be pushed into min/max statistics or bloom pruning, and every row pays the cast.

The same predicate has an exact source-domain form: price >= 173665.00 AND price < 173666.00. Rewriting to that bucket keeps identical results while removing the cast and enabling pruning.

What changes are included in this PR?

  • datafusion/expr-common/src/casts.rs: new decimal-to-integer CastPredicatePreimage::Range for Decimal128(p, s) sources with s > 0 (one level of Dictionary unwrapped). CAST(col AS INT) OP literal rewrites to source-domain decimal bounds using the same truncation-toward-zero buckets as timestamp precision narrowing (v > 0 → [v·10^s, (v+1)·10^s), v == 0 → [1-10^s, 10^s), v < 0 → [(v-1)·10^s+1, v·10^s+1)). trunc_toward_zero_bucket generalized i64 → i128 and reused; no second bucket implementation. Bounds use checked i128 arithmetic validated against the source precision: any unrepresentable bound retains the cast. Decimal32/64/256, scale 0, and non-integer literals are out of scope. No exact preimage can be produced for decimal→int (is_exact_cast_safe still rejects it), so IN lists are untouched.
  • datafusion/sqllogictest/test_files/simplify_expr.slt: end-to-end cases on the spiceai#14969 dataset (8 rows), including = 173665 → rows 2–5, strict/non-strict ranges, negative and zero buckets, NULL-row semantics for IS [NOT] DISTINCT FROM and !=, EXPLAIN plans showing the source-domain rewrite, and the retained-cast case for an unrepresentable bucket.
  • Logical (optimizer) and physical (physical-expr) consumers already dispatch Range preimages and guard volatile-input duplication, so they are unchanged.

What is the testing strategy for this PR?

  • cargo test -p datafusion-expr-common --lib: 239 passed, including 7 new unit tests for bucket bounds per sign, all operators, unrepresentable-bound rejection, scale-0 rejection, Dictionary sources, Decimal256 rejection, out-of-range literal rejection.
  • cargo test -p datafusion-optimizer --lib (928) and cargo test -p datafusion-physical-expr --lib (1695, 2 ignored): pass.
  • cargo test -p datafusion-sqllogictest --test sqllogictests -- simplify_expr: pass.
  • cargo fmt --all --check, cargo clippy -p datafusion-expr-common --all-targets -- -D warnings, git diff --check: pass.

Are there any user-facing changes?

Performance: predicates comparing CAST(decimal AS integer) to integer literals are now rewritten to source-domain decimal ranges, removing the per-row cast and enabling min/max and bloom pruning. Result sets are unchanged relative to this PR's base (#22906).

For values whose cast would error (out of target integer range), the rewritten predicate evaluates to a Boolean instead of erroring, matching the already-accepted overflow policy of the timestamp range preimages; this is documented on the new function. No API changes.

discord9 and others added 18 commits September 8, 2026 16:08
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Merge Apache main at 9547b09 without rewriting the reviewed PR history.
Reject naive-to-timezone-aware timestamp preimages in both comparison
and direct exact-conversion paths, retaining Arrow's timezone adjustment.
Remove redundant Date64 narrowing guards and return an internal error
for unsupported physical comparison operators.

Add result-level coverage for equal-unit, widening, narrowing and
multi-item IN predicates. Correct the existing cross-timezone equality
expectations using raw-instant and column-versus-column controls, while
preserving a true matching positive case.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Merge Apache main at 97c7593 after
publishing the reviewed cast-predicate follow-up. Preserve the existing
review history and timezone-aware cast safeguards without rebasing.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Deprecate the retained date-narrowing helper for 56.0.0 and keep its
existing behavior covered. Extract input-duplication metadata onto
CastPredicatePreimage and use it in the logical volatility guard.

Preserve existing cast comparison, timezone and NULL semantics without
changing physical rewrite logic or the supported cast allowlist.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Merge Apache main 55b2f09, the base
used by the failing PR CI checkout, without rewriting review history.

Preserve the original decimal(38,2) = 4.00 query and its result and
statistics assertions. Its retained decimal cast narrows integer
capacity, prevents a literal guarantee, and does not evaluate Bloom
filters. Account for that explicitly under the updated Bloom metrics.

Add a source-typed decimal equality case that actually evaluates both
surviving Bloom filters, and cover a representable comparison literal
whose column cast must still be retained because valid source values
can overflow the strict Arrow cast.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Casting DECIMAL(p, s) with s > 0 to an integer truncates toward zero, so
an integer literal describes a bucket of source values. Folding such a
comparison into an exact decimal equality keeps only the boundary value
and silently drops the other rows: for DECIMAL(15, 2) prices 173665.29,
173665.47, 173665.73 and 173665.91, CAST(price AS BIGINT) = 173665
returned no rows instead of four (spiceai/spiceai#14969).

Extend the cast-predicate preimage machinery with a decimal-to-integer
Range preimage. CAST(col AS INT) OP literal now rewrites to source-domain
decimal bounds using the same truncation-toward-zero buckets as
timestamp precision narrowing, for every supported comparison operator.
Buckets whose bounds the source decimal cannot represent retain the
cast, and Decimal32/64/256, scale 0, and non-integer literals are out of
scope.

The logical and physical consumers already dispatch Range preimages for
timestamp narrowing, so no consumer code changes; both guard input
duplication for volatile expressions on the equality family.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Oct 12, 2026
@github-actions

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion v55.1.0 (current)
       Built [  57.771s] (current)
     Parsing datafusion v55.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion v55.1.0 (baseline)
       Built [  57.121s] (baseline)
     Parsing datafusion v55.1.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.640s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 117.507s] datafusion
    Building datafusion-expr-common v55.1.0 (current)
       Built [  19.393s] (current)
     Parsing datafusion-expr-common v55.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-expr-common v55.1.0 (baseline)
       Built [  18.881s] (baseline)
     Parsing datafusion-expr-common v55.1.0 (baseline)
      Parsed [   0.018s] (baseline)
    Checking datafusion-expr-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.222s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure function_marked_deprecated: function #[deprecated] added ---

Description:
A function is now #[deprecated]. Downstream crates will get a compiler warning when using this function.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/function_marked_deprecated.ron

Failed in:
  function datafusion_expr_common::casts::is_date_narrowing_cast in /home/runner/work/datafusion/datafusion/datafusion/expr-common/src/casts.rs:827

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  39.603s] datafusion-expr-common
    Building datafusion-optimizer v55.1.0 (current)
       Built [  25.177s] (current)
     Parsing datafusion-optimizer v55.1.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-optimizer v55.1.0 (baseline)
       Built [  25.136s] (baseline)
     Parsing datafusion-optimizer v55.1.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-optimizer v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.165s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  51.370s] datafusion-optimizer
    Building datafusion-physical-expr v55.1.0 (current)
       Built [  27.452s] (current)
     Parsing datafusion-physical-expr v55.1.0 (current)
      Parsed [   0.046s] (current)
    Building datafusion-physical-expr v55.1.0 (baseline)
       Built [  27.181s] (baseline)
     Parsing datafusion-physical-expr v55.1.0 (baseline)
      Parsed [   0.049s] (baseline)
    Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.338s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  55.997s] datafusion-physical-expr
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [  93.054s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.015s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [  92.949s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.016s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.108s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 188.942s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Oct 12, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.08671% with 85 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.83%. Comparing base (55b2f09) to head (3ab2233).
⚠️ Report is 34 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/expr-common/src/casts.rs 94.63% 34 Missing and 20 partials ⚠️
...ptimizer/src/simplify_expressions/cast_preimage.rs 95.65% 12 Missing and 4 partials ⚠️
...fusion/physical-expr/src/simplifier/unwrap_cast.rs 96.74% 8 Missing and 3 partials ⚠️
...imizer/src/simplify_expressions/expr_simplifier.rs 81.81% 0 Missing and 2 partials ⚠️
datafusion/physical-expr/src/simplifier/mod.rs 66.66% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26216      +/-   ##
==========================================
+ Coverage   82.74%   82.83%   +0.08%     
==========================================
  Files        1147     1149       +2     
  Lines      449755   453468    +3713     
  Branches   449755   453468    +3713     
==========================================
+ Hits       372168   375629    +3461     
- Misses      54915    55015     +100     
- Partials    22672    22824     +152     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@discord9
discord9 marked this pull request as draft October 12, 2026 04:58
@discord9 discord9 changed the title fix: rewrite decimal-to-integer cast predicates as source-domain ranges perf: rewrite decimal-to-integer cast predicates as source-domain ranges Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants