Sync main with release#211
Merged
Merged
Conversation
DuckDB selects its IE_JOIN range-join operator only for INNER pure-inequality joins. The per-chromosome dynamic SQL for SEMI and ANTI INTERSECTS joins emitted a bare SEMI JOIN / ANTI JOIN, which DuckDB planned as a quadratic BLOCKWISE_NL_JOIN (a nested loop) rather than the range-join fast path -- roughly 80x slower at a million intervals per side. Emit the left-only shapes as a correlated WHERE EXISTS (SEMI) / WHERE NOT EXISTS (ANTI) subquery over the per-chromosome right partition instead. Both reach IE_JOIN while preserving exact semantics -- one row per qualifying left row, with no dedup-on-projection hazard. The INNER path, chromosome partitioning, empty-schema fallback, and WHERE-residual outer filter are unchanged. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Update the SQL-shape assertions to expect the correlated EXISTS / NOT EXISTS form instead of the SEMI JOIN / ANTI JOIN keyword, add an ANTI shape test, and add execution guards that EXPLAIN the generated per-chromosome SQL and assert it is not planned as a BLOCKWISE_NL_JOIN. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Correct the DuckDB IEJoin section: SEMI and ANTI no longer plan as BLOCKWISE_NL_JOIN; they are emitted as correlated EXISTS / NOT EXISTS subqueries that reach the IE_JOIN range-join fast path. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Three Hypothesis property tests cross-checked the duckdb SEMI/ANTI plan against the dialect=None naive plan. DuckDB 1.4.3 miscompiles the naive plan's bare SEMI JOIN / ANTI JOIN inequality overlap for inputs with duplicate right rows, so once the duckdb plan became correct these tests diverged from their now-unsound oracle and failed. Compare against the Python overlap reference (_python_semi_overlap / _python_anti_overlap, with the WHERE residual applied) instead -- ground truth on every DuckDB version -- and rename the tests to reflect the reference oracle. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-generated by the sync branches workflow.