Route count_overlaps (LEFT JOIN + GROUP BY on INTERSECTS) through IE_JOIN with zero-fill — Closes #209

[conradbzura]

Summary

The idiomatic per-interval overlap count — SELECT a.cols, COUNT(b.col) FROM a LEFT JOIN b ON a.interval INTERSECTS b.interval GROUP BY a.cols — declined to the naive plan like every outer join (_has_outer_join_intersects), becoming a HASH_JOIN on the low-cardinality chromosome key with the position inequalities as a residual filter — quadratic, ~180× slower than achievable (81 s vs ~0.5 s at 1e6 intervals per side, single-thread).

This is a follow-up to #208 (same root cause: DuckDB only selects IE_JOIN for INNER pure-inequality joins). The key observation is that the existing INNER path already plans COUNT(b.col) + GROUP BY through IE_JOIN. So the fast path detects the LEFT-count shape, reuses the INNER machinery to compute the counts, and wraps them in a zero-fill LEFT JOIN back onto the distinct left keys, so left intervals with no overlap report 0. Verified end-to-end: count_overlaps at 1e6 goes from 81 s to 0.56 s single-threaded (0.21 s at 14 threads), faster than bioframe's 0.67 s. Results were validated against the naive plan and a Python reference across 500+ random and edge cases (empty inputs, one-sided chromosomes, duplicate left rows) on DuckDB 1.4.3 and 1.5.4 — exact match.

Scope (v1): a single COUNT(<right column>) / COUNT(DISTINCT <right column>) with a GROUP BY on the projected left columns. COUNT(*) (its LEFT-join semantics count the null-padded row for unmatched rows), non-COUNT aggregates (bedtools-map-style SUM/MIN/MAX — a clean follow-up), ORDER BY / LIMIT / HAVING, self-joins, and other outer-join shapes still take the naive-predicate plan.

Closes #209

Proposed changes

• perf — Add _match_count_overlaps (detect the LEFT-join COUNT shape, decline everything else) and _build_count_overlaps_sql (reuse transform_to_sql on an INNER copy, then wrap in WITH __giql_counts … + a zero-fill LEFT JOIN against the distinct left keys). Wire the detection into transform_to_sql ahead of the outer-join decline. Purely additive.
• test — A dedicated TestTranspileDuckDBIEJoinCountOverlaps class: shape, six decline cases, a Hypothesis property test against a Python count_overlaps reference, and an EXPLAIN plan guard.
• docs — A count_overlaps subsection in the DuckDB IEJoin performance guide.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	TestTranspileDuckDBIEJoinCountOverlaps	A LEFT-join COUNT(b.col) + GROUP BY query	Transpiled with dialect='duckdb'	Emits the SET VARIABLE + __giql_counts CTE + COALESCE(..., 0) zero-fill form and no a.chrom = b.chrom naive predicate	Fast-path shape
2	TestTranspileDuckDBIEJoinCountOverlaps	A COUNT(*) / SUM(b.score) / no-GROUP-BY / ORDER BY / self-join LEFT-join query	Transpiled with dialect='duckdb'	Declines the fast path (no __giql_counts wrapper)	Decline cases
3	TestTranspileDuckDBIEJoinCountOverlaps	Hypothesis-random peaks/genes	The count fast path is executed	Rows equal the Python count_overlaps reference (zero counts, one-sided chromosomes, empty right, duplicate left rows)	Result correctness
4	TestTranspileDuckDBIEJoinCountOverlaps	A dense single-chromosome pair	The generated inner dynamic SQL is planned with EXPLAIN	The plan contains no BLOCKWISE_NL_JOIN	Avoids nested loop

Full non-integration suite (1026 passed) and the bedtools integration oracle (79 passed) stay green.

https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy