Skip to content

Reuse aggregate-derived join domains as detail-side payloads - #23113

Draft
pentschev wants to merge 2 commits into
NVIDIA:mainfrom
pentschev:cudf-polars/join-aggregate-domain-reuse
Draft

Reuse aggregate-derived join domains as detail-side payloads#23113
pentschev wants to merge 2 commits into
NVIDIA:mainfrom
pentschev:cudf-polars/join-aggregate-domain-reuse

Conversation

@pentschev

@pentschev pentschev commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

This extends join_filter_pushdown to recognize when an aggregate-derived join domain can be reused as the payload for a later detail-side join input.

The motivating case is PDSH Q18. The query computes the set of l_orderkey values whose sum(l_quantity) > 300, and current planning can already use that result as a semi-join domain to filter orders. However, the final orders x lineitem join still reads the raw lineitem detail side again.

With this change, when the aggregate-derived domain is known to contain the join key and the payload columns needed by the later join, the planner can reuse that derived domain to feed the detail side as well. The original full join is still preserved, so the rewrite is a row-reduction optimization rather than a semantic shortcut.

Q18 plan before, relevant subtree
JOIN Inner ('o_orderkey',) ('l_orderkey',)
  JOIN Semi ('o_orderkey',) ('l_orderkey',)
    STREAMINGSCAN orders
    FILTER (sum_quantity > 300.0)
      GROUPBY ('l_orderkey',)
        STREAMINGSCAN lineitem ('l_orderkey', 'l_quantity')
  STREAMINGSCAN lineitem ('l_orderkey', 'l_quantity')
Q18 plan after, relevant subtree
JOIN Inner ('o_orderkey',) ('l_orderkey',)
  JOIN Semi ('o_orderkey',) ('l_orderkey',)
    STREAMINGSCAN orders
    FILTER (sum_quantity > 300.0)
      GROUPBY ('l_orderkey',)
        STREAMINGSCAN lineitem ('l_orderkey', 'l_quantity')
  FILTER BooleanFunction ('l_orderkey', 'l_quantity')
    SELECT ('l_orderkey', 'l_quantity')
      FILTER (sum_quantity > 300.0)
        GROUPBY ('l_orderkey',)
          STREAMINGSCAN lineitem ('l_orderkey', 'l_quantity')

Materially, this reduces Q18 memory pressure and improves runtime by avoiding the final unfiltered detail-side lineitem input. In the latest standalone validation, the full SF30K Q1-Q22 workflow completed successfully on 6xNVL4 nodes with validation passing in 146s lukewarm/128s hot.

@pentschev pentschev self-assigned this Jul 5, 2026
@pentschev pentschev added 0 - Blocked Cannot progress due to external reasons improvement Improvement / enhancement to an existing function non-breaking Non-breaking change cudf-polars Issues specific to cudf-polars labels Jul 5, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 5, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Python Affects Python cuDF API. label Jul 5, 2026
@GPUtester GPUtester moved this to In Progress in cuDF Python Jul 5, 2026
Replace eligible detail-side join inputs with an existing aggregate-derived payload in join filter pushdown. This keeps the upstream semi-domain filtering from current main while removing the remaining raw detail scan from Q18-style plans, reducing memory pressure and improving runtime without depending on the separate reused semi-domain candidate branch.
@pentschev
pentschev force-pushed the cudf-polars/join-aggregate-domain-reuse branch from 01e286b to fbc8ea5 Compare August 10, 2026 20:39
@pentschev pentschev changed the title Reuse aggregate domains as join payloads Reuse aggregate-derived join domains as detail-side payloads Aug 10, 2026
@josephine-wolf-oberholtzer josephine-wolf-oberholtzer moved this to In Progress in cuDF Python Aug 12, 2026
@pentschev pentschev added 2 - In Progress Currently a work in progress and removed 0 - Blocked Cannot progress due to external reasons labels Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2 - In Progress Currently a work in progress cudf-polars Issues specific to cudf-polars improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API.

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

3 participants