Release v0.5

README.md

@@ -71,6 +71,8 @@ print(sql)
 
 The transpiled SQL can be executed with fast genome-unaware databases or in-memory analytic engines like DuckDB.
 
+By default a column-to-column `INTERSECTS` join emits the naive overlap predicate (`a.chrom = b.chrom AND a.start < b.end AND b.start < a.end`) as a plain `ON` condition, which each engine's optimizer plans as a range join. For DuckDB you can additionally pass `dialect="duckdb"` to opt into a per-chromosome IEJoin plan for INNER, SEMI, or ANTI joins; shapes it declines fall through to the naive predicate. See [DuckDB IEJoin Dialect](https://giql.readthedocs.io/en/latest/transpilation/performance.html#duckdb-iejoin-dialect) for the supported shapes and fallback rules.
+
 You can also use [oxbow](https://oxbow.readthedocs.io) to efficiently stream specialized genomics formats into DuckDB. 
 
 ```python

demo.ipynb

@@ -224,7 +224,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
    "metadata": {},
    "outputs": [
     {
@@ -3579,6 +3579,152 @@
     "result.head(15)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "\n",
+    "## 8. DISJOIN operator\n",
+    "\n",
+    "**Split intervals at breakpoints** so that no resulting sub-interval ever partially overlaps a reference interval.\n",
+    "\n",
+    "Unlike MERGE and CLUSTER, which *collapse* intervals, DISJOIN *multiplies* rows: one input interval becomes one or more sub-interval rows. It is a table-valued function, so it appears in the `FROM` clause like NEAREST. The full input row passes through unchanged and the sub-interval is appended as `disjoin_chrom`, `disjoin_start`, and `disjoin_end`.\n",
+    "\n",
+    "In **self-mode**, `DISJOIN(features)` splits a set against its own breakpoints. Selecting the distinct sub-intervals yields the globally non-overlapping partition — the equivalent of Bioconductor's `disjoin()`.\n",
+    "\n",
+    "**GIQL Query**: `SELECT DISTINCT disjoin_chrom, disjoin_start, disjoin_end FROM DISJOIN(features_a)`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "giql_query = \"\"\"\n",
+    "    SELECT DISTINCT disjoin_chrom, disjoin_start, disjoin_end\n",
+    "    FROM DISJOIN(features_a)\n",
+    "    ORDER BY disjoin_chrom, disjoin_start\n",
+    "\"\"\"\n",
+    "\n",
+    "sql = transpile(giql_query, tables=[\"features_a\"])\n",
+    "print(\"Transpiled SQL:\")\n",
+    "print(sqlglot.transpile(sql, pretty=True)[0])\n",
+    "print(\"\\n\" + \"=\" * 80 + \"\\n\")\n",
+    "\n",
+    "n_original = conn.execute(\"SELECT COUNT(*) FROM features_a\").fetchone()[0]\n",
+    "result = conn.execute(sql).fetchdf()\n",
+    "print(\n",
+    "    f\"Result: {len(result)} non-overlapping segments \"\n",
+    "    f\"(from {n_original} original peaks)\"\n",
+    ")\n",
+    "result.head(10)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Disjoining a set against itself\n",
+    "\n",
+    "The example above runs self-mode at scale; on a small, deliberately overlapping set every cut is visible. With no `reference`, `DISJOIN` defaults the reference to the target set, so each interval is split at every *other* interval's interior breakpoint: `A = [0, 100)` is cut where `B` starts (50); `B = [50, 150)` is cut at `A`'s end (100) and `C`'s start (120); `D = [0, 80)` is cut where `E` starts (40). Selecting `DISTINCT disjoin_*` collapses the pieces into the non-overlapping partition.\n",
+    "\n",
+    "**GIQL Query**: `SELECT * FROM DISJOIN(toy_peaks)`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# A small set of deliberately overlapping intervals, so the split is visible.\n",
+    "conn.execute(\"DROP TABLE IF EXISTS toy_peaks\")\n",
+    "conn.execute(\n",
+    "    'CREATE TABLE toy_peaks(chrom VARCHAR, \"start\" INTEGER, \"end\" INTEGER, name VARCHAR)'\n",
+    ")\n",
+    "conn.executemany(\n",
+    "    \"INSERT INTO toy_peaks VALUES (?, ?, ?, ?)\",\n",
+    "    [\n",
+    "        (\"chr1\", 0, 100, \"A\"),\n",
+    "        (\"chr1\", 50, 150, \"B\"),\n",
+    "        (\"chr1\", 120, 200, \"C\"),\n",
+    "        (\"chr2\", 0, 80, \"D\"),\n",
+    "        (\"chr2\", 40, 80, \"E\"),\n",
+    "    ],\n",
+    ")\n",
+    "print(\"Input intervals (overlapping):\")\n",
+    "print(\n",
+    "    conn.execute(\n",
+    "        'SELECT name, chrom, start, \"end\" FROM toy_peaks ORDER BY chrom, start'\n",
+    "    ).fetchdf()\n",
+    ")\n",
+    "print(\"\\n\" + \"=\" * 80 + \"\\n\")\n",
+    "\n",
+    "# With no reference, DISJOIN splits toy_peaks against its own breakpoints.\n",
+    "giql_query = \"\"\"\n",
+    "    SELECT name, disjoin_chrom, disjoin_start, disjoin_end\n",
+    "    FROM DISJOIN(toy_peaks)\n",
+    "    ORDER BY disjoin_chrom, disjoin_start, name\n",
+    "\"\"\"\n",
+    "\n",
+    "sql = transpile(giql_query, tables=[\"toy_peaks\"])\n",
+    "print(\"Transpiled SQL:\")\n",
+    "print(sqlglot.transpile(sql, pretty=True)[0])\n",
+    "print(\"\\n\" + \"=\" * 80 + \"\\n\")\n",
+    "\n",
+    "result = conn.execute(sql).fetchdf()\n",
+    "print(\n",
+    "    f\"Result: {len(result)} sub-intervals -- each interval split at its \"\n",
+    "    f\"neighbours' breakpoints\"\n",
+    ")\n",
+    "result"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Splitting against a reference set\n",
+    "\n",
+    "Pass a second interval set as `reference :=` to cut each target feature at that set's breakpoints, keeping only the pieces a reference interval covers. Here every peak in `features_a` is split at the breakpoints of `features_b`; the parent peak coordinates ride along with each sub-interval, and a peak that overlaps no feature in B drops out entirely.\n",
+    "\n",
+    "**GIQL Query**: `SELECT * FROM DISJOIN(features_a, reference := features_b)`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "giql_query = \"\"\"\n",
+    "    SELECT\n",
+    "        chrom,\n",
+    "        start AS peak_start,\n",
+    "        \"end\" AS peak_end,\n",
+    "        signal_value,\n",
+    "        disjoin_start,\n",
+    "        disjoin_end\n",
+    "    FROM DISJOIN(features_a, reference := features_b)\n",
+    "    ORDER BY chrom, disjoin_start\n",
+    "\"\"\"\n",
+    "\n",
+    "sql = transpile(giql_query, tables=[\"features_a\", \"features_b\"])\n",
+    "print(\"Transpiled SQL:\")\n",
+    "print(sqlglot.transpile(sql, pretty=True)[0])\n",
+    "print(\"\\n\" + \"=\" * 80 + \"\\n\")\n",
+    "\n",
+    "n_original = conn.execute(\"SELECT COUNT(*) FROM features_a\").fetchone()[0]\n",
+    "result = conn.execute(sql).fetchdf()\n",
+    "print(\n",
+    "    f\"Result: {len(result)} sub-intervals of A split at B's breakpoints \"\n",
+    "    f\"(from {n_original} original peaks)\"\n",
+    ")\n",
+    "result.head(10)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -3587,7 +3733,7 @@
     "\n",
     "## Summary\n",
     "\n",
-    "This demo showcased all 7 GIQL operators:\n",
+    "This demo showcased all 8 GIQL operators:\n",
     "\n",
     "1. **INTERSECTS**: Binary predicate for overlapping intervals\n",
     "2. **WITHIN**: Binary predicate for containment (A within B)\n",
@@ -3596,6 +3742,7 @@
     "5. **CLUSTER**: Aggregation operator to assign cluster IDs to overlapping intervals\n",
     "6. **DISTANCE**: UDF operator to calculate genomic distances between intervals\n",
     "7. **NEAREST**: Table-valued function for finding k-nearest genomic features\n",
+    "8. **DISJOIN**: Set operator to split intervals at reference breakpoints\n",
     "\n",
     "Each operator was:\n",
     "- Written in GIQL syntax\n",
@@ -3608,7 +3755,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "giql",
+   "display_name": "giql (3.13.3)",
    "language": "python",
    "name": "python3"
   },
@@ -3622,7 +3769,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.13.2"
+   "version": "3.13.3"
   }
  },
  "nbformat": 4,

docs/dialect/aggregation-operators.rst

@@ -40,6 +40,9 @@ Syntax
    -- Strand-specific clustering
    CLUSTER(interval, stranded := true) AS cluster_id
 
+   -- Predicate-gated clustering (run-length encoding on a column)
+   CLUSTER(interval, predicate := depth = PREV(depth)) AS cluster_id
+
    -- Combined parameters
    CLUSTER(interval, distance, stranded := true) AS cluster_id
 
@@ -56,6 +59,36 @@ Parameters
 **stranded** *(optional)*
    When ``true``, only cluster intervals on the same strand. Default: ``false``.
 
+**predicate** *(optional)*
+   A boolean expression evaluated between each interval and its sorted
+   predecessor. When supplied, the cluster-boundary condition becomes
+   **adjacent AND predicate**: an interval stays in the current cluster only
+   when it is within ``distance`` of its predecessor *and* the predicate holds
+   between the two. A change in the predicate forces a new cluster, so an
+   equality predicate yields a run-length encoding of the input sequence.
+   Omitting the predicate preserves the default adjacency-only behavior.
+
+   Bare column references resolve to the *current* interval; the predecessor's
+   value of a column is referenced with ``PREV(column)``
+   (e.g. ``depth = PREV(depth)``). The predicate composes with ``distance`` and
+   ``stranded`` and is evaluated under the operator's existing per-chromosome
+   (and per-strand) partition and start-position order.
+
+   Two constraints apply:
+
+   - **References existing columns only.** The predicate *gates* merging on
+     columns already present on the input rows; it does not synthesize a
+     statistic. Coverage depth, for example, must already be a column on the
+     rows (typically produced upstream by :ref:`DISJOIN <disjoin-operator>` and
+     aggregation).
+   - **Pairwise only, with single-linkage drift.** The predicate compares each
+     interval to its immediate sorted predecessor (everything ``LAG`` can
+     express). Whole-cluster conditions are out of scope. When the predicate is
+     not an equivalence relation (e.g. ``ABS(score - PREV(score)) < 5``),
+     consecutive pairs may each satisfy it while the cluster's extremes do not
+     — the same single-linkage behavior that ``distance``-based clustering
+     already exhibits.
+
 Return Value
 ~~~~~~~~~~~~
 
@@ -101,6 +134,19 @@ Cluster intervals separately by strand:
    FROM features
    ORDER BY chrom, strand, start
 
+**Predicate-Gated Clustering:**
+
+Cut adjacent intervals into clusters wherever a column's value changes
+(run-length encoding). ``PREV(column)`` references the predecessor row's value:
+
+.. code-block:: sql
+
+   SELECT
+       *,
+       CLUSTER(interval, predicate := depth = PREV(depth)) AS cluster_id
+   FROM features
+   ORDER BY chrom, start
+
 **Analyze Cluster Statistics:**
 
 Count features per cluster:
@@ -145,6 +191,10 @@ Find regions with multiple overlapping features:
    INNER JOIN cluster_sizes s ON c.cluster_id = s.cluster_id
    WHERE s.size >= 3
 
+.. note::
+
+   **Synthesized flag hidden under** ``SELECT *``. A ``SELECT *, CLUSTER(...)`` query materializes an internal ``__giql_is_new_cluster`` flag in a subquery, so the outer star is emitted as ``SELECT * EXCEPT (__giql_is_new_cluster)`` to keep that helper column out of the result (#184). ``* EXCEPT`` is a DataFusion-family extension: the generic and ``datafusion`` dialects emit it, while ``duckdb`` spells the exclusion ``EXCLUDE``. Transpile with ``dialect="duckdb"`` to execute on DuckDB — the portable generic ``* EXCEPT`` form is not DuckDB-runnable. A qualified ``SELECT t.*, CLUSTER(...)`` receives the same treatment: because ``CLUSTER`` runs over a single relation, the qualifier is dropped and the outer star is emitted as the same bare ``* EXCEPT (__giql_is_new_cluster)`` (#185). An explicitly-projected ``CLUSTER`` (no star) surfaces no helper column and needs no exclusion. A star may also be combined with additional explicit projection items — ``SELECT *, 1 AS extra, CLUSTER(...)`` — and each item surfaces exactly once. An *aliased* sibling (``expr AS name``) is materialized once in the inner subquery under a reserved ``__giql_sibling_N`` name, EXCEPTed from the outer star so it is not re-surfaced, and re-projected — aliased back to ``name`` — at its own position, so its output-column position is preserved even when the ``CLUSTER`` sits between the star and the item (``SELECT *, CLUSTER(...) AS cid, 1 AS extra`` keeps ``cid`` before ``extra``). Using a reserved inner name (rather than the user's alias) keeps the mechanism correct even when ``name`` collides with a base column the star surfaces (``SELECT *, score * 10 AS score`` yields both the base ``score`` and the computed one, matching the un-clustered projection) or with another sibling's alias (``SELECT *, 1 AS x, 2 AS x``). A *non-aliased* sibling (a bare column already covered by the star, or an unnamed expression) is materialized the same way, under its own reserved ``__giql_sibling_N`` name, and re-projected at its written slot — keeping its own column name (a bare column) or its rendered text (an unnamed expression) — so every sibling's output-column position matches the identical projection without ``CLUSTER`` (#190).
+
 Performance Notes
 ~~~~~~~~~~~~~~~~~
 
@@ -194,6 +244,9 @@ Syntax
    -- Strand-specific merge
    SELECT MERGE(interval, stranded := true) FROM features
 
+   -- Predicate-gated merge (merge only equal-valued adjacent runs)
+   SELECT MERGE(interval, predicate := depth = PREV(depth)) FROM features
+
    -- Merge with additional aggregations
    SELECT
        MERGE(interval),
@@ -214,6 +267,16 @@ Parameters
 **stranded** *(optional)*
    When ``true``, merge intervals separately by strand. Default: ``false``.
 
+**predicate** *(optional)*
+   A boolean expression that further restricts which adjacent intervals are
+   merged. ``MERGE`` decomposes into :ref:`CLUSTER <cluster-operator>` plus a
+   ``GROUP BY`` over the cluster id, so it inherits predicate-aware boundaries
+   directly — see the :ref:`CLUSTER predicate <cluster-operator>` description
+   for the full semantics, the ``PREV(column)`` convention, the
+   references-existing-columns-only constraint, and the pairwise-only /
+   single-linkage caveat. Omitting the predicate preserves the default
+   adjacency-only merge.
+
 Return Value
 ~~~~~~~~~~~~
 
@@ -256,6 +319,24 @@ Merge intervals separately by strand:
    SELECT MERGE(interval, stranded := true)
    FROM features
 
+**Predicate-Gated Merge (coverage depth):**
+
+Merge only adjacent intervals that share the same coverage depth, reconstructing
+a re-clustered, depth-segmented partition from per-breakpoint segments produced
+by :ref:`DISJOIN <disjoin-operator>` and aggregation:
+
+.. code-block:: sql
+
+   SELECT MERGE(interval, predicate := depth = PREV(depth))
+   FROM (
+       SELECT disjoin_chrom AS chrom,
+              disjoin_start AS start,
+              disjoin_end AS end,
+              COUNT(*) AS depth
+       FROM DISJOIN(features)
+       GROUP BY disjoin_chrom, disjoin_start, disjoin_end
+   ) AS segments
+
 **Merge with Feature Count:**
 
 Count how many features were merged into each region:
@@ -324,6 +405,59 @@ Notes
 - MERGE is an aggregate operation that processes all matching rows
 - The operation sorts data internally, so pre-sorting is not required
 
+.. note::
+
+   CLUSTER and MERGE cannot be combined in a single ``SELECT`` — MERGE
+   aggregates rows away while CLUSTER is a per-row window over those same rows,
+   so no single query expresses both. Transpiling ``SELECT MERGE(interval),
+   CLUSTER(interval) FROM features`` raises a ``ValueError``. Use them in
+   separate queries instead — for example, CLUSTER over a subquery, or MERGE
+   over one.
+
+.. note::
+
+   MERGE cannot be projected alongside a star. MERGE aggregates rows into one
+   row per merged region, so a ``SELECT *, MERGE(...)`` or ``SELECT t.*,
+   MERGE(...)`` has no coherent per-row meaning — the star names the
+   pre-aggregation input columns of a relation MERGE has already collapsed and
+   grouped away. Transpiling either shape raises a ``ValueError`` rather than
+   emitting non-executable SQL (a bare ``*`` re-surfaces non-grouped columns
+   under the synthesized ``GROUP BY``; a qualified ``rel.*`` dangles an alias the
+   aggregation no longer exposes). Drop the star, or project only grouping
+   columns and aggregates (e.g. ``COUNT(*)``) alongside ``MERGE`` — not raw input
+   columns, which are neither grouped nor aggregated. This is unlike
+   :ref:`CLUSTER <cluster-operator>`, a per-row window over which a star *is*
+   meaningful and supported.
+
+.. note::
+
+   **Projecting columns alongside** ``MERGE``. ``MERGE`` emits one row per merged
+   region and always projects the grouping keys (``chrom``, and ``strand`` when
+   ``stranded := true``) followed by ``MIN(start)`` / ``MAX(end)``. The other items in
+   a ``SELECT ..., MERGE(...)`` projection are reconciled against that grouping
+   (#192): an explicit grouping-key column (the ``chrom`` of the *Merge by Chromosome*
+   shape above) is emitted once, not duplicated; an aggregate (``COUNT(*)``,
+   ``AVG(score)``, ``STRING_AGG(name, ',')``) is computed per merged region; and an
+   expression over only grouping-key columns (``UPPER(chrom)``) is kept. A raw,
+   non-aggregated column that is neither a grouping key nor derived from one (e.g.
+   ``score``, or ``start`` within ``MAX(score) + start``) has no coherent value per
+   merged region and raises a ``ValueError`` — as does a *window* aggregate over such a
+   column (``SUM(score) OVER (...)``), since a window is evaluated after the grouping and
+   does not collapse it. An item whose output name collides with the ``chrom`` /
+   ``start`` / ``end`` columns ``MERGE`` synthesizes (e.g. ``chrom AS start``) also raises
+   rather than emitting two columns of that name — alias it to a distinct name instead.
+   The collision check folds case (``chrom AS Start`` raises too, since SQL binds the
+   unquoted alias onto ``start``), while an unaliased expression over a grouping key
+   (``CAST(chrom AS VARCHAR)``) is kept — its emitted column name is the expression, not a
+   bare grouping-key name. Likewise, a ``GROUP BY`` may name only the grouping-key columns
+   ``MERGE`` groups by, as a *bare column reference*: ``GROUP BY chrom`` is honored (it is
+   subsumed by the synthesized per-merge grouping), while ``GROUP BY`` over any other
+   column — or an expression (``GROUP BY UPPER(chrom)``) or ordinal (``GROUP BY 1``) that
+   ``MERGE`` cannot map to its grouping — raises a ``ValueError`` rather than being
+   silently discarded. A ``HAVING`` is honored against the per-merge grouping
+   (``HAVING COUNT(*) > 1`` keeps only regions built from more than one input interval)
+   rather than being dropped.
+
 Related Operators
 ~~~~~~~~~~~~~~~~~