Skip to content

fix: remap nested TopK filter expressions before their children - #26031

Open
efegokdemir wants to merge 1 commit into
apache:mainfrom
efegokdemir:fix/topk-dynamic-filter-child-remap
Open

efegokdemir wants to merge 1 commit into
apache:mainfrom
efegokdemir:fix/topk-dynamic-filter-child-remap

Conversation

@efegokdemir

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

TopK dynamic filtering can return incorrect rows when one sort expression contains another sort key. Child-first remapping changes the parent before it can be matched.

What changes are included in this PR?

Remap matching parent expressions before their descendants and stop traversing each replacement. Add the reported query as a SQL logic test.

What is the testing strategy for this PR?

  • cargo test --profile=ci --test sqllogictests -- topk_dynamic_filter_remap.slt
  • cargo test --profile=ci -p datafusion-physical-expr --lib (1685 passed, 2 ignored)
  • cargo fmt --all --check
  • cargo clippy --profile ci -p datafusion-physical-expr --lib -- -D warnings
  • All-target Clippy passed with -A clippy::ignore-without-reason for an existing ignored test lacking a reason in planner.rs.

The broader topk.slt file could not be validated because this shallow checkout does not include the testing submodule data.

Are there any user-facing changes?

Yes. Queries matching this case now return the correct TopK rows; there is no API change.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Oct 4, 2026
# specific language governing permissions and limitations
# under the License.

statement ok

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The end-to-end SLT provides strong coverage for #26029. Since remap_children is also shared by HashJoin and projection/scan pushdown, would it be valuable to add a focused unit test that registers both a parent expression and one of its descendants, while also checking that an unmatched parent still allows descendant remapping? Non-blocking—the reported wrong-result path is already covered by the SLT.

@rgbuilds rgbuilds left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I traced the parent-first remapping behavior and the shared DynamicFilterPhysicalExpr consumers. transform_down with Jump correctly replaces the largest matching expression, preserves descendant remapping when the parent does not match, and continues processing sibling subtrees. The focused and broader TopK, dynamic-filter, Parquet-pushdown, physical-expression, formatting, and Clippy checks pass locally. No blocking issues found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TopK dynamic filter returns wrong rows when one sort key contains another (DynamicFilterPhysicalExpr remaps the inner column first)

2 participants