DuckDB INTERSECTS SEMI/ANTI joins fall back to BLOCKWISE_NL_JOIN instead of IE_JOIN (~80x slowdown) — Closes #208

[conradbzura]

Summary

DuckDB selects its IE_JOIN range-join operator only for INNER pure-inequality joins. The DuckDB IEJoin override (src/giql/expanders/intersects_duckdb.py) emitted SEMI and ANTI column-to-column INTERSECTS joins as a per-chromosome SEMI JOIN / ANTI JOIN, which DuckDB plans as a quadratic BLOCKWISE_NL_JOIN (a nested loop) rather than the range-join fast path — roughly 80x slower than the INNER path (38 s vs 0.46 s single-threaded at one million intervals per side).

This change emits the left-only shapes as a correlated WHERE EXISTS (SEMI) / WHERE NOT EXISTS (ANTI) subquery over the per-chromosome right partition instead. Both reach IE_JOIN while preserving exact semantics — one row per qualifying left row, with no dedup-on-projection hazard. The INNER path, chromosome partitioning, empty-schema fallback, and WHERE-residual outer filter are unchanged. Verified end-to-end at one million intervals per side: SEMI 38 s → 0.46 s, ANTI → 0.37 s, and SEMI + ANTI row counts partition the left input exactly.

Scoped to SEMI/ANTI. The analogous count_overlaps LEFT-join slowdown shares the same root cause but is a larger change and is tracked separately in #209.

Closes #208

Proposed changes

• perf — In _build_sql, branch the per-chromosome dynamic-SQL template on the join shape. INNER keeps the a JOIN b ON <overlap> form; SEMI and ANTI now emit SELECT ... FROM a WHERE [NOT] EXISTS (SELECT 1 FROM b WHERE <overlap>). The overlap predicate and inline ON residuals move into the subquery WHERE.
• docs — Correct the class docstring and the DuckDB IEJoin section of docs/transpilation/performance.rst, which previously claimed SEMI/ANTI reached IE_JOIN.
• test — Update the SQL-shape assertions to expect the EXISTS / NOT EXISTS form, add an ANTI shape test, and add two execution guards that EXPLAIN the generated per-chromosome SQL and assert it is not planned as a BLOCKWISE_NL_JOIN.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	TestTranspileDuckDBIEJoinSQLStructure	A SEMI JOIN INTERSECTS query	Transpiled with dialect='duckdb'	Emits a correlated EXISTS subquery and no SEMI JOIN keyword	SEMI EXISTS shape
2	TestTranspileDuckDBIEJoinSQLStructure	An ANTI JOIN INTERSECTS query	Transpiled with dialect='duckdb'	Emits a correlated NOT EXISTS subquery and no ANTI JOIN keyword	ANTI NOT EXISTS shape
3	TestTranspileDuckDBIEJoinSQLStructure	An ANTI JOIN with a top-level WHERE residual	Transpiled with dialect='duckdb'	Applies the residual as an outer wrapper filter over the NOT EXISTS per-chromosome template	ANTI WHERE-residual placement
4	TestTranspileDuckDBIEJoinExecution	A dense single-chromosome SEMI join large enough to trigger a nested loop	The per-chromosome dynamic SQL is planned with EXPLAIN	The plan contains no BLOCKWISE_NL_JOIN	SEMI avoids nested loop
5	TestTranspileDuckDBIEJoinExecution	A dense single-chromosome ANTI join large enough to trigger a nested loop	The per-chromosome dynamic SQL is planned with EXPLAIN	The plan contains no BLOCKWISE_NL_JOIN	ANTI avoids nested loop

Existing property-based tests that execute the generated SEMI/ANTI SQL against DuckDB and compare against a Python overlap reference, and the bedtools integration oracle, continue to validate result correctness (unchanged, all green).

https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy