Release v0.5#207
Merged
Merged
Conversation
DISJOIN is a FROM-clause table function that cuts each target interval at every reference breakpoint strictly interior to it, so no resulting sub-interval partially overlaps a reference interval. The target row passes through unchanged and the sub-interval is appended as disjoin_chrom, disjoin_start and disjoin_end. A coverage filter drops sub-intervals overlapping no reference interval. When no reference is given it defaults to the target set, so selecting the distinct sub-intervals reproduces a Bioconductor-style disjoin. The generator emits a portable WITH-CTE subquery using only UNION, LEAD and EXISTS, so it transpiles unchanged to DuckDB, PostgreSQL and SQLite. Sub-interval boundaries are canonicalized to 0-based half-open regardless of the target or reference coordinate system.
Add DISJOIN to the MCP server operator metadata and documentation path map so it is discoverable through list_operators, explain_operator and search_docs. The list_operators count assertion is updated to match the new operator total.
Add parsing tests for the GIQLDisjoin AST node, transpilation tests for the generated SQL shape, end-to-end execution tests on DuckDB, and a coordinate-space matrix asserting convention-invariant results across all four coordinate-system and interval-type encodings.
Add a DISJOIN section to the aggregation operators reference and a new disjoin recipe covering set partitioning and reference-grid splitting, wired into the recipes index and toctree.
Add tests/usage_patterns.py, a catalogue of the query contexts an operator appears in, and tests/test_usage_patterns.py, a functional suite that transpiles every DISJOIN usage pattern, executes it against a database-engine matrix (DuckDB now; SQLite and PostgreSQL are future entries), and snapshots the result rows with pytest-manifest. The catalogue is a per-operator-class descriptor framework. Only the table-function class is populated -- DISJOIN, exercised across 21 usage patterns in self- and reference-mode. AGGREGATE, PREDICATE, and SCALAR are defined extension points. Three patterns GIQL cannot transpile (a subquery, CTE, or nested operator in a table-function's target position) are catalogued as strict xfail. Add pytest-manifest as a dev dependency and register the usage marker.
DISJOIN bound only the first positional argument and silently discarded any extras, so DISJOIN(features, refs) dropped the intended reference set without warning. Unknown named arguments were passed through unchecked as well. from_arg_list now raises a ParseError naming the single-positional-arg limit and rejecting any named argument outside the DISJOIN schema, so a mistyped or misplaced reference fails loudly at parse time.
A bare DISJOIN reference name was matched against registered tables first and, failing that, assumed to be a canonical CTE. A CTE sharing a registered table's name therefore silently inherited that table's coordinate system, and a name matching neither produced SQL referencing a relation that does not exist. Reference resolution now checks enclosing WITH clauses first, letting an in-query CTE shadow a registered table the way SQL scoping does, and raises a ValueError when a bare name matches neither a CTE nor a registered table. Bare SELECT and UNION references are accepted alongside Subquery nodes, and a reference using the reserved __giql_dj_ prefix is rejected since that prefix names the operator's internal CTEs.
The end-to-end DISJOIN suite ran against DuckDB only. Parametrize it over a run fixture so every execution test also runs on in-memory SQLite, skipping SQLite below 3.25 where the LEAD() window function DISJOIN emits is unavailable.
DISJOIN was documented under the aggregation operators, but it multiplies rows rather than aggregating them. Move its reference into a new set-operators page, list the page in the dialect index and toctree, and point the MCP operator catalog at the relocated doc.
Ground each recipe in a concrete genomics scenario -- pooled ChIP-seq peak calls, ATAC-seq features against a callable mask, a fixed-width binned coverage matrix -- and note that the uniform-grid recipe relies on DuckDB-specific range() syntax and must span every chromosome present in the input.
DISJOIN, NEAREST, MERGE, and CLUSTER each bound the target only when a positional argument was present, so a call with no target -- such as DISJOIN(reference := refs) or MERGE(stranded := true) -- built a node with the target unset. The failure surfaced far downstream as the misleading "Target table 'None' not found". Each from_arg_list now raises a ParseError naming the missing target the moment the call is parsed, so the diagnostic points at the real problem.
DISJOIN names its internal CTEs with a __giql_dj_ prefix and already rejects a reference relation that uses it. A target table with the same prefix was unguarded and would emit a self-referential CTE that fails with no actionable error. giqldisjoin_sql now rejects a resolved target name carrying the reserved prefix, symmetric with the existing reference check.
The DISJOIN SQL was assembled as one ~30-fragment string concatenation, where a dropped inter-fragment space would silently produce invalid SQL. Build each __giql_dj_ CTE as a named local and join them once at the return, so every CTE reads as a discrete block. The generated SQL is unchanged.
The DISJOIN transpilation and execution test docstrings used a free-form GIVEN/WHEN/THEN prose block. Convert them to the structured Given/When/Then form the test guide specifies and add the Arrange-Act-Assert phase comments, matching test_disjoin_parsing.py. Also add a transpilation test for the reserved-prefix target guard.
Drop the structural banner-comment rule lines that divide usage_patterns.py into sections, keeping the explanatory prose. Make _annotations delegate to _features_like rather than duplicate it. Hoist the duckdb import in test_usage_patterns.py to module scope.
Note in the set-operators reference that disjoin_chrom, disjoin_start, and disjoin_end are reserved output column names that collide with a same-named target column. Trim the recipe's "Coming from Bedtools?" section to drop analogies it does not deliver, and correct two section underlines to their title lengths.
The usage-pattern snapshot suite depends on pytest-manifest, declared in the dev dependency group but never mirrored into the pixi environment that CI builds. CI therefore ran without the plugin and every snapshot test errored with "fixture 'manifest' not found". pytest-manifest has no conda-forge package, so add it under tool.pixi.pypi-dependencies -- the pixi environment now installs it from PyPI alongside the conda-managed test dependencies.
Add a DISJOIN section to the demo notebook covering self-mode and reference-mode. Self-mode is shown twice: once partitioning the features_a peak set at scale, and once on a small hand-built set of overlapping intervals where every cut is traceable by eye. The reference-mode example splits features_a at the breakpoints of features_b. The summary now lists DISJOIN as the eighth operator.
In self-reference mode -- DISJOIN(features) with the reference omitted, or DISJOIN(features, reference := features) where features is the same registered table -- the outer-WHERE EXISTS coverage subquery is provably always true. Every emitted segment lies inside its parent target row, and that row is itself a member of the reference set, so the semi-join filters nothing but still plans as a full equi+range hash join. Detect self-reference at the resolver and propagate a flag into the SQL builder. When the flag is set, omit the AND EXISTS clause; the other CTEs and the seg_end guards are untouched. Subquery references and CTE-shadowed names stay conservative -- proving relation equivalence for those is out of scope -- so EXISTS is retained. Benchmarks in the original issue measure 1.30x speedup at 10K intervals rising to 1.83x at 250K, with byte-identical row sets at every input size.
Cover behaviors orthogonal to the test-first cases: composition with non-default coordinate systems (1-based closed) on both self-mode and explicit-self-reference paths; composition with custom genomic column names; per-call independence inside a query that mixes self-mode and distinct-reference DISJOIN calls; and CTE shadowing detected via the ancestor walk past a nested subquery scope. These tests pin the optimization's contract against future regressions in adjacent code paths -- canonicalization, column resolution, the enclosing-CTE search, and per-expression flag computation -- where breakage would not surface in the basic truth-table tests.
End-to-end coverage through DuckDB for the optimization. The keystone test asserts row-set equivalence between DISJOIN(features) and DISJOIN(features, reference := features) across all four coordinate conventions -- proving the two self-reference paths take the same optimized branch at runtime. The remaining tests guard the load-bearing EXISTS that the optimization deliberately preserves: a partial-coverage distinct reference (with a coverage gap in the target), the same scenario under mixed target/reference encodings to exercise the canonical shift inside the retained EXISTS body, a CTE-shadowed self-reference where target and reference both resolve to the CTE, and a subquery reference whose filter induces partial coverage. Each of these would silently return extra rows if EXISTS were wrongly skipped on a non-self path.
Replace the inline exists_clause string-with-leading-space ternary with a where_clauses list joined by AND, mirroring the pattern already used in giqlnearest_sql. The previous form was one missing space away from emitting a syntactically broken WHERE; the list form removes that fragility and makes the conditional EXISTS appendage explicit. The emitted SQL is character-identical in both self-reference and distinct-reference modes. Also extend the leading comment to note that the __giql_dj_ref CTE itself stays live in self-reference mode because the breakpoint CTE still draws from it, preventing a future reader from trying to prune the seemingly unused reference CTE, and tighten the _resolve_disjoin_reference docstring to make the False semantics of is_self_reference explicit for subquery and CTE references.
The nested-CTE-shadow transpilation test asserted only that EXISTS appeared somewhere in the output. Tighten it to assert that exactly one EXISTS clause appears, matching the assertion style of the sibling mixed-mode test and ruling out a stray second EXISTS leaking in from the outer scope. Rename two integration tests on TestDisjoinCoordinateSpace from should_yield_X to the declarative-present form already used by the other five tests in the class and by every sibling coordinate-space test file.
…eGIQLGenerator The private static helper was placed before __init__, violating the class-member ordering convention where __init__/__new__ precede private methods. Move it next to the other private static helpers (_enclosing_cte_names) where readers expect private utilities to live, and tighten its docstring's exp.Boolean / exp.Literal references with :class: role markup.
…ansformer Two small drive-by fixes flagged during the PR #93 style review: - The __init__ docstring referenced DEFAULT_BIN_SIZE as bare text; switch to :data:`~giql.constants.DEFAULT_BIN_SIZE` so the reference resolves to an anchor in generated docs. - Five inline comments inside _build_pairs_replacement_joins ("Determine which INTERSECTS side maps to FROM vs JOIN table", "Ensure key-only bin CTEs exist", "Build and attach the pairs CTE", "join1: [SIDE] JOIN pairs ON from.key = pairs.from_key", and "join2: [SIDE] JOIN join_table ON join.key = pairs.join_key") restated what the next line of code did rather than explaining why. The four follow-on what-comments are dropped; the prefix-assignment block keeps a single why-comment that captures the symmetry invariant the code preserves. No behavior change.
The canonical-start / canonical-end conversions and their inverses lived inside BaseGIQLGenerator as private methods. With the DuckDB IEJoin dialect transformer about to land — which also needs to emit 0-based half-open SQL regardless of how each source table declares its coordinate system — the conversion logic needs to be shared across both surfaces. Lift canonical_start, canonical_end, decanonical_start, and decanonical_end out of BaseGIQLGenerator into a new module-level giql.canonical surface. Each function takes a raw column expression plus a Table (or None) and wraps the expression with whatever offset the table's (coordinate_system, interval_type) pair requires to reach the canonical 0-based half-open form. The decanonical counterparts apply the inverse offset so emitted endpoints land back in the table's own encoding. BaseGIQLGenerator now imports and delegates to the module-level helpers. Behavior is unchanged for every existing call site.
…RSECTS The binned equi-join plan that IntersectsBinnedJoinTransformer emits generalises across every SQL backend, but on DuckDB it loses to a per-chromosome dynamic-SQL plan that lets DuckDB pick from its range- join family (IE_JOIN / PIECEWISE_MERGE_JOIN). The new transformer emits a SET VARIABLE block whose value is a per-chromosome UNION ALL of pure inequality-predicate subqueries, then selects through query(getvariable(...)). This avoids the cross-chromosome candidate- pair materialization that hurts a global IEJoin and the UNNEST / DISTINCT overhead that hurts the binned plan. The transformer handles INNER, SEMI, and ANTI joins with exactly one column-to-column INTERSECTS predicate plus a broad set of common decorations: ORDER BY / LIMIT / OFFSET / DISTINCT on the outer SELECT, GROUP BY / HAVING with aggregate functions (COUNT, SUM, MIN, MAX, AVG, COUNT(DISTINCT), and any exp.AggFunc subclass), and extra non- INTERSECTS predicates ANDed onto the join ON or WHERE (both single- side and cross-side, including BETWEEN, IN (literal-list), LIKE, IS NULL). Unsupported shapes — outer-join INTERSECTS, self-joins, multi- INTERSECTS queries, three-or-more-table FROM clauses, top-level WITH / CTE clauses, DISTINCT ON, OR/NOT/paren-wrapped extras, unqualified-column extras, extras containing subqueries / aggregates / window functions, and subqueries inside GROUP BY / HAVING / ORDER BY — fall back to the binned plan so callers can opt in unconditionally without hitting silent correctness regressions. Hard-error projection shapes (bare SELECT *, unqualified columns, expression projections, unknown table qualifiers, a.* AS x, window aggregates, FILTER clauses, arithmetic-over-aggregate, scalar subqueries) raise ValueError with a dedicated message naming the offending form. Empty chromosome intersections route through a COALESCE empty-schema fallback so DuckDB receives a typed empty result instead of a runtime NULL crash. Chromosome literals are escaped manually because DuckDB has no quote_literal. Five INTERSECTS-walking helpers (_is_column_intersects, _find_column_intersects_in, _has_outer_join_intersects, _strip_intersects, _count_column_intersects) move out of IntersectsBinnedJoinTransformer to module scope so both transformers reuse one definition.
…cate Column-to-column INTERSECTS joins on the generic and DataFusion targets now emit the naive overlap predicate (a.chrom = b.chrom AND a.start < b.end AND b.start < a.end) as a plain ON/WHERE condition the engine plans as a range join, promoting the existing residual `_column_join` expander from fallback to the sole generic join plan. The generic binned equi-join transformer (`IntersectsBinnedJoinTransformer`), the `intersects_bin_size` parameter, and the `DEFAULT_BIN_SIZE` public constant are removed (breaking change). The naive predicate is correct for INNER and OUTER (LEFT/RIGHT/FULL) joins with no restructuring — verified on DuckDB and DataFusion, which both hash-partition on `chrom` with the position inequalities as a residual join filter (no nested-loop degradation). The DuckDB IEJoin transformer is unchanged; the shapes it declines now fall through to the naive predicate instead of the binned plan, so outer joins, self-joins, multiple INTERSECTS, extra predicates, non-base operands, and 3+ tables all get a correct plan. `range_join_strategy` gains the value "naive" (generic/datafusion) alongside "iejoin" (duckdb). Closes #167 Claude-Session: https://claude.ai/code/session_01KAYsMtN7vYuECZHeeFFzCM
Rename test_binned_join.py -> test_intersects_join.py and retarget it to the naive overlap predicate: structure tests assert the predicate is emitted with no CTE / UNNEST / bins, while the execution, outer-join, and bag-semantics tests (unchanged in behavior) now exercise the naive predicate. Remove the `intersects_bin_size` tests across test_transpile.py, test_duckdb_iejoin.py, and test_intersects.py; retarget the IEJoin fallback-test terminology from "binned plan" to "naive-predicate plan"; flip the `range_join_strategy` fixtures to "naive"; drop the `DEFAULT_BIN_SIZE` package-API reference; and de-stale the serialization / resolver pipeline comments. Add a cross-target LEFT JOIN oracle case proving outer-join result identity across generic / datafusion / duckdb, and an ANTI-join parity test now that the naive predicate returns correct results for the empty-right-table case the old binned plan dropped. Claude-Session: https://claude.ai/code/session_01KAYsMtN7vYuECZHeeFFzCM
…verride Relocate the DuckDB IEJoin column-to-column INTERSECTS join rewrite from the capability-gated pre-pass transformer (giql.transformer) into a registered (DuckDBTarget, Intersects) override expander (giql.expanders.intersects_duckdb), mirroring the CLUSTER / MERGE relocation (#144). INTERSECTS join strategy is now fully registry-driven: the (GenericTarget, Intersects) expander emits the naive overlap predicate on every target, and DuckDB's override supersedes it for the shapes the IEJoin path supports, deferring back to the same naive predicate for every shape it declines (LEFT/RIGHT/FULL, self-joins, multiple INTERSECTS, extra predicates, non-base operands, 3+ tables) and for literal-range / residual predicates. The IEJoin rewrite emits a whole-query multi-statement string, so the override produces it through the ExpansionContext.add_statement_finalizer seam, wrapping the built SQL in an exp.Command that serializes verbatim. A column-to-column INTERSECTS survives to pass 3 structurally intact (pass 1 attaches metadata only; pass 2 synthesizes no CTE for its column slots), so the override rebuilds the join from the same raw AST the former pre-pass read — output is byte-identical for every supported shape. - Remove the uses_iejoin / range_join_strategy capability branch from transpile; dispatch flows through the registry's resolve() fallback chain in pass 3. - Drop the now-vestigial range_join_strategy field from Capabilities (the (DuckDBTarget, Intersects) registration is the single source of truth). - git mv transformer.py -> expanders/intersects_duckdb.py; the module now also holds the registered expand_intersects_duckdb entry point. - has_override is retained as registry introspection but no longer gates a pre-pass (there is none); its docstrings and the CLUSTER/MERGE cross-references to the removed giql.transformer module are updated. Claude-Session: https://claude.ai/code/session_01KAYsMtN7vYuECZHeeFFzCM
…_strategy Update the tests for the #169 migration: - Add TestIEJoinRegistryDeferral::test_registry_should_resolve_duckdb_intersects_ to_iejoin_override, asserting the built-in (DuckDBTarget, Intersects) override is registered and resolves to expand_intersects_duckdb (the migrated pass-3 join strategy). - Reframe the two existing deferral tests for the new model: the built-in IEJoin is a registered override, and a user (DuckDBTarget, Intersects) registration replaces it via last-write-wins rather than skipping a pre-pass. Assertions are unchanged (byte-identical output), only the mechanism they document. - Drop range_join_strategy from every Capabilities construction / assertion in test_targets.py and test_expander.py (the field was removed); the equality-inequality test now differs on supports_qualify instead. - Refresh the stale test_serialization helper docstring (no pre-pass join rewrite remains). The DuckDB IEJoin behavioral oracle (tests/test_duckdb_iejoin.py, executed against real DuckDB) passes unchanged, confirming the refactor preserves output. Claude-Session: https://claude.ai/code/session_01KAYsMtN7vYuECZHeeFFzCM
The DuckDB IEJoin override collected JOIN-ON and top-level WHERE residuals into a single list and appended all of them to each per-chromosome join ON. That is sound for INNER (ON overlap WHERE r == ON overlap AND r) and for the equivalent SEMI case, but for ANTI it inverts the anti-join: a row failing the residual r satisfies NOT EXISTS b (overlap AND r) unconditionally, so rows that should be filtered out are instead resurrected (#200). Split the residuals by provenance in _extract_extra_predicates — ON residuals (genuine join conditions, always inlined for every kind) versus WHERE residuals (post-join filters). For the left-only shapes (SEMI / ANTI) the WHERE residuals are now layered as an outer WHERE on the wrapper relation instead of folded into the per-chromosome ON, reusing the existing wrapper-relation rewriter and pre-allocating their columns into the wrapper projection list so a filter can reference a column absent from the SELECT list. A right-side reference in such a residual raises the same left-only ValueError the SELECT list already enforces. INNER keeps inlining WHERE residuals (byte-identical output, provably equivalent). Closes #200 Claude-Session: https://claude.ai/code/session_01KAYsMtN7vYuECZHeeFFzCM
Adds regression coverage for #200: an ANTI join with a WHERE residual matches the naive plan (was resurrecting the anti-excluded rows below the threshold), a residual can filter on a column absent from the SELECT list, the SEMI case stays consistent, and a right-side residual raises. A structural test asserts the residual lands in the outer wrapper WHERE and never in the per-chromosome ON template, and a Hypothesis property test cross-checks ANTI+WHERE against the naive plan over random inputs. Claude-Session: https://claude.ai/code/session_01KAYsMtN7vYuECZHeeFFzCM
…plan Route four cross-backend inconsistencies in the DuckDB IEJoin INTERSECTS override to the naive-predicate plan so dialect="duckdb" stays consistent with dialect=None / "datafusion" for any shape the IEJoin cannot faithfully rebuild but the naive plan handles. Declining is cheap: the naive overlap predicate still plans as a chromosome-keyed hash/range join (#167). - #201: a SEMI/ANTI join whose column-to-column INTERSECTS sits in the top-level WHERE references the out-of-scope right table; the WHERE-branch relocated it into the join ON and invented anti/semi-overlap results. New guard _has_left_only_join_where_intersects declines so the naive plan surfaces the same binder error the reference plans raise. - #202: star projections (a.* / b.* / bare *) expanded to only the configured genomic columns, dropping user columns. New guard _has_star_projection declines every star so DuckDB expands the real star against the live schema. - #204, #205: projections the naive plan compiles but the IEJoin projection rebuild cannot express — expressions (a.start + 1), window aggregates, FILTER clauses, scalar subqueries, aggregates nested in expressions (COUNT(*) * 2), and stars nested in an aggregate argument (COUNT(a.*), MIN(COLUMNS(*))) — now decline via a projection pre-scan (_projection_declines_to_naive) instead of hard-erroring (#205) or miscompiling (#204). A projection referencing an out-of-scope column (unqualified / unknown-table / right-side) still raises a clean ValueError, since the naive plan rejects it too. The now-redundant specialized diagnostic branches (window / FILTER / scalar-subquery / arithmetic-over- aggregate messages) collapse into the generic qualified-projection error — the accurate guidance for the only cases that still reach it (qualify the column and the wrapper itself declines). Docs (performance.rst): the unsupported-but-naive-valid projection shapes move into the soft-fallback list; the corresponding hard-error entries are removed. Claude-Session: https://claude.ai/code/session_01KAYsMtN7vYuECZHeeFFzCM
Repoint the tests that pinned the old buggy / hard-error behavior to assert the corrected decline-to-naive behavior, and add regression classes for each declined shape. - #201 / #202 (from the first round): decline across ON TRUE/FALSE/score and SEMI/ANTI with binder-error parity; decline across every star form with full-column parity to the naive plan; idiomatic-form / explicit-column engage guards; execution parity for SEMI star, a.* AS x, b.* and c.*. - #204 / #205: convert six obsolete hard-error tests (window aggregate, paren-wrapped window, arithmetic-over-aggregate, scalar subquery, expression form, literal) to decline tests, and add TestTranspileDuckDBIEJoinUnsupportedProjectionFallback — a parametrized decline check over every unsupported-but-naive-valid projection, execution parity for COUNT(a.*) / MIN(COLUMNS(*)) / a.start + 1 / SUM(a.score) OVER (), a keep-raise test for an out-of-scope column inside an unsupported wrapper, and a plain-aggregate-still-engages guard. Claude-Session: https://claude.ai/code/session_01KAYsMtN7vYuECZHeeFFzCM
DuckDB selects its IE_JOIN range-join operator only for INNER pure-inequality joins. The per-chromosome dynamic SQL for SEMI and ANTI INTERSECTS joins emitted a bare SEMI JOIN / ANTI JOIN, which DuckDB planned as a quadratic BLOCKWISE_NL_JOIN (a nested loop) rather than the range-join fast path -- roughly 80x slower at a million intervals per side. Emit the left-only shapes as a correlated WHERE EXISTS (SEMI) / WHERE NOT EXISTS (ANTI) subquery over the per-chromosome right partition instead. Both reach IE_JOIN while preserving exact semantics -- one row per qualifying left row, with no dedup-on-projection hazard. The INNER path, chromosome partitioning, empty-schema fallback, and WHERE-residual outer filter are unchanged. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Update the SQL-shape assertions to expect the correlated EXISTS / NOT EXISTS form instead of the SEMI JOIN / ANTI JOIN keyword, add an ANTI shape test, and add execution guards that EXPLAIN the generated per-chromosome SQL and assert it is not planned as a BLOCKWISE_NL_JOIN. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Correct the DuckDB IEJoin section: SEMI and ANTI no longer plan as BLOCKWISE_NL_JOIN; they are emitted as correlated EXISTS / NOT EXISTS subqueries that reach the IE_JOIN range-join fast path. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Three Hypothesis property tests cross-checked the duckdb SEMI/ANTI plan against the dialect=None naive plan. DuckDB 1.4.3 miscompiles the naive plan's bare SEMI JOIN / ANTI JOIN inequality overlap for inputs with duplicate right rows, so once the duckdb plan became correct these tests diverged from their now-unsound oracle and failed. Compare against the Python overlap reference (_python_semi_overlap / _python_anti_overlap, with the WHERE residual applied) instead -- ground truth on every DuckDB version -- and rename the tests to reflect the reference oracle. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
The idiomatic per-interval overlap count -- SELECT a.cols, COUNT(b.col) FROM a LEFT JOIN b ON a.interval INTERSECTS b.interval GROUP BY a.cols -- declined to the naive plan like every outer join, becoming a HASH_JOIN on the low-cardinality chromosome key with the position inequalities as a residual filter (quadratic; ~180x slower than achievable at scale). Detect this shape and reuse the INNER IEJoin path (which already plans COUNT + GROUP BY through IE_JOIN), then wrap its counts in a zero-fill LEFT join back onto the distinct left keys so left intervals with no overlap report 0. Supports a single COUNT(<right column>) / COUNT(DISTINCT <right column>) with a GROUP BY on the projected left columns; COUNT(*), non-COUNT aggregates, ORDER BY / LIMIT / HAVING, and other outer-join shapes still take the naive-predicate plan. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Add shape, decline (COUNT(*), non-COUNT aggregate, no GROUP BY, ORDER BY, self-join), a Hypothesis property test against a Python count_overlaps reference (zero-overlap rows, one-sided chromosomes, empty right table, duplicate left rows), and an EXPLAIN plan guard asserting the generated inner SQL is not planned as a BLOCKWISE_NL_JOIN. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Describe the LEFT-join COUNT over INTERSECTS routing through the INNER IE_JOIN count plus a zero-fill LEFT join, and the shapes that remain on the naive-predicate plan. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Tighten the count_overlaps detection gate for two shapes a review found producing wrong results on matched queries: a GROUP BY carrying keys beyond the projected columns (the extra key inflates the count cardinality and drops zero-count rows the zero-fill base cannot reproduce), and a projected key whose output name collides with the COUNT alias (the zero-fill USING / COALESCE binds to the wrong duplicately-named column). Require the group keys to equal the projected keys exactly and all output names to be distinct, declining to the naive plan otherwise. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Add decline tests for a GROUP BY exceeding the projection, an output-name collision between a key and the COUNT alias, and a top-level WHERE, plus a COUNT(DISTINCT b.col) execution test asserting equivalence with the naive plan. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
CLUSTER (and MERGE, which composes on it) decided cluster boundaries with
LAG("end") -- the immediately preceding row's end -- so a later interval
contained within an earlier, wider one was spuriously split into a new
cluster once the preceding row ended early. Key the boundary off the
running maximum end of the preceding rows instead
(MAX("end") OVER (... ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING)),
which is the cluster's true right edge so far. The distance offset, the
first-row NULL behavior, and the separate PREV() predecessor-reference
predicate are unchanged; non-containment inputs are unaffected because the
running max equals LAG when no interval is contained.
Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Update the transpilation assertions from the LAG("end") adjacency to the
running-max MAX("end") OVER (... ROWS BETWEEN UNBOUNDED PRECEDING AND 1
PRECEDING) form, and add containment cases to the bedtools CLUSTER and
MERGE oracle suites (a wide interval containing a later narrower one that
ends before a third still-contained interval) -- the shape the previous
LAG boundary got wrong.
Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
…-max Reword the predicate comment: adjacency now keys off the cluster's running-max edge (not the immediate predecessor), and note the two notions can reference different rows under containment while the predicate keeps its documented immediate-predecessor semantics. Addresses review advisories. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
…red helper The DuckDB per-chromosome dynamic-SQL machinery -- the chromosome string-literal interpolation, the collision-resistant session-variable name, the SET VARIABLE ... = COALESCE((string_agg per chromosome), empty) builder, and the query(getvariable(...)) relation reference -- moves from IntersectsDuckDBIEJoinTransformer into a new giql.expanders._per_chrom module. The operator-specific parts (the per-chromosome SQL template, the partition source, the empty-schema fallback, and the outer projection) stay in the expander. Emitted SQL is byte-identical (no behavior change), readying the scaffolding for reuse by DISJOIN (#216) and NEAREST. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-generated by the cut release workflow.