CLUSTER and MERGE split contained intervals into separate clusters — Closes #214#215
Merged
Merged
Conversation
CLUSTER (and MERGE, which composes on it) decided cluster boundaries with
LAG("end") -- the immediately preceding row's end -- so a later interval
contained within an earlier, wider one was spuriously split into a new
cluster once the preceding row ended early. Key the boundary off the
running maximum end of the preceding rows instead
(MAX("end") OVER (... ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING)),
which is the cluster's true right edge so far. The distance offset, the
first-row NULL behavior, and the separate PREV() predecessor-reference
predicate are unchanged; non-containment inputs are unaffected because the
running max equals LAG when no interval is contained.
Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
Update the transpilation assertions from the LAG("end") adjacency to the
running-max MAX("end") OVER (... ROWS BETWEEN UNBOUNDED PRECEDING AND 1
PRECEDING) form, and add containment cases to the bedtools CLUSTER and
MERGE oracle suites (a wide interval containing a later narrower one that
ends before a third still-contained interval) -- the shape the previous
LAG boundary got wrong.
Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
…-max Reword the predicate comment: adjacency now keys off the cluster's running-max edge (not the immediate predecessor), and note the two notions can reference different rows under containment while the predicate keeps its documented immediate-predecessor semantics. Addresses review advisories. Claude-Session: https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CLUSTERandMERGE(which composes onCLUSTER) produced incorrect results when an interval is fully contained within an earlier, wider interval. They decided cluster boundaries withCASE WHEN LAG("end") OVER (...) >= "start"— comparing each interval to the immediately preceding row's end rather than the running maximum end of the cluster so far. When a wide interval contains a later narrower one, a subsequent interval that still overlaps the wide one (but not the narrow predecessor) was spuriously split into a new cluster.For
[0,100),[10,20),[30,40), GIQLMERGEreturned[0,100),[30,40)(wrong) vs the bedtools/bioframe oracle's[0,100). On a 1e6 interval setMERGEreturned 847,897 regions vs the oracle's 844,107 (~0.5% wrongly split). Contained intervals are common in real genomic data (genes over exons, broad peaks over narrow sub-peaks).The fix keys the boundary flag off the running maximum end of the preceding rows —
MAX("end") OVER (PARTITION BY chrom ORDER BY start ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING)— the cluster's true right edge so far. The distance offset, first-row NULL behavior, and the separatePREV()predecessor-reference predicate are unchanged; non-containment inputs are unaffected (the running max equalsLAGwhen nothing is contained). Verified: containment now yields the single correct region, and the 1e6 set returns exactly 844,107 (oracle match), at similar/better runtime (0.26s vs 0.43s).Discovered by the benchmark suite via a row-count discrepancy against the bedtools/bioframe oracle.
Closes #214
Proposed changes
src/giql/expanders/cluster.py: the adjacency-flag window becomes a running-maxMAX("end")framed window instead ofLAG("end"); docstring updated.test_cluster.py,test_genomic_columns.py,test_cluster_predicate_transpilation.py,test_expander.py) from theLAGshape to theMAX(...) OVER (... ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING)shape, and add containment cases to the bedtoolsCLUSTERandMERGEoracle suites.Test cases
test_mergeMERGEis executedtest_clusterCLUSTERis executedTestClusterExpanderMAX("end") OVER (... ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING)boundary (with+ distancewhen positive)Full non-integration suite (1112 passed) and the bedtools CLUSTER/MERGE oracle (13 passed, incl. the two new containment cases) stay green.
https://claude.ai/code/session_01TERWBHov76DQM3nQeT8yyy