Repository navigation
fix: build the IN list static filter only from literal-only items - #26083
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #26083 +/- ##
==========================================
+ Coverage 82.72% 82.77% +0.05%
==========================================
Files 1147 1147
Lines 448013 450930 +2917
Branches 448013 450930 +2917
==========================================
+ Hits 370602 373252 +2650
- Misses 54920 54947 +27
- Partials 22491 22731 +240 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jayzhan211
approved these changes
Oct 8, 2026
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
Thanks @ranflarion , overall LGTM
Contributor
|
FYI @geoffreyclaude |
add is_volatile_node Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
alamb
approved these changes
Oct 9, 2026
Contributor
|
Thank you @ranflarion and @jayzhan211 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
INwhen a list item is aCASEthat reads a column #26082.Rationale for this change
n IN (2, 100, 101, CASE WHEN d > 5 THEN 1 ELSE d END * 5)returns wrong results on every row (repro in #26082).try_evaluate_constant_listdecides that a list is constant by evaluating it on an empty batch, andCaseExprreturns its THEN branch as a scalar for a zero-length WHEN mask, so the CASE item is frozen to5in the static filter and never evaluated per row.What changes are included in this PR?
try_evaluate_constant_listreturnsNoneunless every leaf of every list item is aLiteral, so any other list takes the per-batch evaluationInListExpralready uses for items that return arrays. Lists of literals, casts of literals and other expressions over literals still build the static filter from their empty-batch evaluation, as before.Checking leaves rather than
collect_columnskeeps the rule independent of which leaf types read the input (Column,LambdaVariable, or a leaf defined outside DataFusion). The cost is that an item with a non-literal leaf that is nonetheless constant, such as a stable function the optimizer did not fold, is now evaluated per batch instead of once.What is the testing strategy for this PR?
A new section in
in_list.sltruns the repro as a projection and as a filter, andtest_in_list_case_item_reading_a_columnbuilds the same expression throughin_list(). Both fail on main and pass with this change. The otherin_listunit tests, thephysical-exprlib tests, the full sqllogictest suite and thefilter_pushdownintegration tests pass unchanged.Are there any user-facing changes?
No API changes. IN lists whose items have a non-literal leaf return correct results, evaluated per batch rather than through the static filter.