Skip to content

perf: rewrite decimal-to-integer cast predicates as source-domain ranges - #1

Open
discord9 wants to merge 1 commit into
experiment/cast-predicate-preimagefrom
stack/pr22906-decimal-int-range
Open

discord9 wants to merge 1 commit into
experiment/cast-predicate-preimagefrom
stack/pr22906-decimal-int-range

Conversation

@discord9

Copy link
Copy Markdown
Owner

Stack-only review view. This PR is opened inside the fork so its base can be the apache#22906 branch and the diff shows only this PR's changes. Upstream PR: apache#26216 (base main, because GitHub does not allow a fork branch as an upstream PR base). Merge order upstream: apache#22906 first, then apache#26216.

Which issue does this PR close?

Stacked on apache#22906 (1e127365510c83cfffec5fed6f72a56b5875d5b0). The wrong-rows regression itself is fixed by apache#22906's allowlist; this PR adds the optimization on top.

Rationale for this change

CAST(decimal_col AS BIGINT) = 173665 is a many-to-one comparison (decimal→integer truncates toward zero), so apache#22906 correctly refuses to fold it and keeps the cast on the column. That is correct but leaves performance on the table: the predicate cannot be pushed into min/max statistics or bloom pruning, and every row pays the cast.

The same predicate has an exact source-domain form: price >= 173665.00 AND price < 173666.00. Rewriting to that bucket keeps identical results while removing the cast and enabling pruning.

What changes are included in this PR?

  • datafusion/expr-common/src/casts.rs: new decimal-to-integer CastPredicatePreimage::Range for Decimal128(p, s) sources with s > 0 (one level of Dictionary unwrapped). CAST(col AS INT) OP literal rewrites to source-domain decimal bounds using the same truncation-toward-zero buckets as timestamp precision narrowing (v > 0 → [v·10^s, (v+1)·10^s), v == 0 → [1-10^s, 10^s), v < 0 → [(v-1)·10^s+1, v·10^s+1)). trunc_toward_zero_bucket generalized i64 → i128 and reused; no second bucket implementation. Bounds use checked i128 arithmetic validated against the source precision: any unrepresentable bound retains the cast. Decimal32/64/256, scale 0, and non-integer literals are out of scope. No exact preimage can be produced for decimal→int (is_exact_cast_safe still rejects it), so IN lists are untouched.
  • datafusion/sqllogictest/test_files/simplify_expr.slt: end-to-end cases on the spiceai#14969 dataset (8 rows), including = 173665 → rows 2–5, strict/non-strict ranges, negative and zero buckets, NULL-row semantics for IS [NOT] DISTINCT FROM and !=, EXPLAIN plans showing the source-domain rewrite, and the retained-cast case for an unrepresentable bucket.
  • Logical (optimizer) and physical (physical-expr) consumers already dispatch Range preimages and guard volatile-input duplication, so they are unchanged.

What is the testing strategy for this PR?

  • cargo test -p datafusion-expr-common --lib: 239 passed, including 7 new unit tests for bucket bounds per sign, all operators, unrepresentable-bound rejection, scale-0 rejection, Dictionary sources, Decimal256 rejection, out-of-range literal rejection.
  • cargo test -p datafusion-optimizer --lib (928) and cargo test -p datafusion-physical-expr --lib (1695, 2 ignored): pass.
  • cargo test -p datafusion-sqllogictest --test sqllogictests -- simplify_expr: pass.
  • cargo fmt --all --check, cargo clippy -p datafusion-expr-common --all-targets -- -D warnings, git diff --check: pass.

Are there any user-facing changes?

Performance: predicates comparing CAST(decimal AS integer) to integer literals are now rewritten to source-domain decimal ranges, removing the per-row cast and enabling min/max and bloom pruning. Result sets are unchanged relative to this PR's base (apache#22906).

For values whose cast would error (out of target integer range), the rewritten predicate evaluates to a Boolean instead of erroring, matching the already-accepted overflow policy of the timestamp range preimages; this is documented on the new function. No API changes.

Casting DECIMAL(p, s) with s > 0 to an integer truncates toward zero, so
an integer literal describes a bucket of source values. Folding such a
comparison into an exact decimal equality keeps only the boundary value
and silently drops the other rows: for DECIMAL(15, 2) prices 173665.29,
173665.47, 173665.73 and 173665.91, CAST(price AS BIGINT) = 173665
returned no rows instead of four (spiceai/spiceai#14969).

Extend the cast-predicate preimage machinery with a decimal-to-integer
Range preimage. CAST(col AS INT) OP literal now rewrites to source-domain
decimal bounds using the same truncation-toward-zero buckets as
timestamp precision narrowing, for every supported comparison operator.
Buckets whose bounds the source decimal cannot represent retain the
cast, and Decimal32/64/256, scale 0, and non-integer literals are out of
scope.

The logical and physical consumers already dispatch Range preimages for
timestamp narrowing, so no consumer code changes; both guard input
duplication for volatile expressions on the equality family.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant