fix(scanner): do not lower a negated InList into a take - #9097
Open
LuciferYang wants to merge 2 commits into
Open
LuciferYang wants to merge 2 commits into
LuciferYang wants to merge 2 commits into
Conversation
… ids TakeOperation::try_from_expr matched Expr::InList without consulting negated, so a filter like _rowid NOT IN (...) that survived the expression simplifier unexpanded (the simplifier only expands small lists) was planned as a take of exactly the listed ids — the complement of the predicate. Large NOT-IN lists on _rowid/_rowaddr/_rowoffset silently returned the wrong rows. Skip the lowering for negated lists; the filter then runs as a regular scan filter with correct semantics. The regression test uses a list large enough to survive unexpanded.
LuciferYang
force-pushed
the
fix/srid-rowid-not-in
branch
from
September 10, 2026 02:57
6dd903d to
4716b14
Compare
Contributor
There was a problem hiding this comment.
The guard removes the silent wrong-result path for all three system columns. _rowid and _rowaddr return the complement; _rowoffset NOT IN now fails explicitly because general _rowoffset scan filtering is not supported. #9105 tracks that bounded feature gap. An explicit error is safer than returning wrong rows, so it does not need to hold this fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9080.
TakeOperation::try_from_exprmatchedExpr::InListwithout consultingnegated, so a filter like_rowid NOT IN (...)was planned as a take of exactly the listed ids, the complement of the predicate. The arm returns no remainder, so the predicate was dropped from the plan rather than merely ignored by the take.Small lists never reached the lowering, because DataFusion's expression simplifier expands them into
!=conjunctions first. Only lists large enough to survive unexpanded hit it, which is why this is invisible in small examples.Skipping the lowering for negated lists lets
_rowidand_rowaddrrun as regular scan filters, which has the right semantics: theelsebranch leavesexpr_filter_planuntouched and hands the whole predicate to the filtered read._rowoffset_rowoffsetshares the arm, and the guard covers it in the sense that matters: a negated list is no longer answered as a take of the listed offsets. It cannot be answered as a scan filter either, though._rowoffsetis reachable only through this lowering, so it is absent from the filterable read schema and any predicate that survives to the planner is rejected. That is the column's existing limit rather than something the guard introduced: onmaintoday,_rowoffset > 2fails withColumn _rowoffset does not existfromProjection::union_column, which has arms for_rowidand_rowaddrbut none for_rowoffset. Adding that arm alone is not enough either; the filtered read then rejects the predicate atplanner.rs:1089withvalid_fields: [idx, _rowid, _rowaddr], because the derived column is never materialized into the schema the filter is planned against.Making every
_rowoffsetpredicate work means requesting_rowaddrwhenever a filter mentions_rowoffsetand computing the derived column before filter refinement. That is a feature rather than part of this correctness fix, so it is filed as #9105. I am happy to do it there, or here if you would rather have both together.Testing
test_filter_to_take_with_stable_row_idsgains a ten-element_rowid NOT INcase. Against the old code it returnsidx = [0..9]instead of[10, 11]. It also pins the_rowoffsetcontract end to end: a negated_rowoffsetlist must fail rather than return the complement.take_operation_rejects_negated_in_listexercises the lowering directly over all three columns that share the arm, so the coverage does not depend on the simplifier's expansion threshold staying where it is.