Skip to content

Find duplicated code inside a single file: same-file rescue, within-file bucket completion, and two rename-evidence corrections - #508

Merged
abdushakoor12 merged 11 commits into
mainfrom
same-file-recall-band
Sep 6, 2026
Merged

abdushakoor12 merged 11 commits into
mainfrom
same-file-recall-band

Conversation

@abdushakoor12

@abdushakoor12 abdushakoor12 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

TLDR

Duplicated code inside a single file is now found: two methods that drifted apart, two that differ only in their literals, and an exact copy standing behind an unrelated sibling all reach the report, where each of them previously published fragments or nothing at all.

Details

Three findings that 0.32.0 published had gone silent, all in the same band — duplication where both copies live in one file. Three causes, each fixed at the stage that lost the finding.

A structural-hash bucket now pairs every two members that share a file

[FUSED-CANDIDATE-BUCKET-STAR]deslop-core/src/pair/candidates/builder.rs

Members of one bucket are Merkle-equal, so the candidate generator paired each member with the bucket's first member and let closure carry the rest. Across files that is sound. Inside one file it is not: the within-file content floor decides every pair, and which member sorts first is an accident of write order. A file holding one method that differs ahead of two byte-identical copies judged those copies only against the member that differs — so one unrelated sibling deleted an exact duplicate from the report.

PairBuilder::pair_within_files now completes the bucket within each file. No member can tell in advance which partner its content will vouch for, so recall may not depend on what else happens to share the shape.

Known cost, tracked as #506: all-pairs is quadratic in same-file bucket size. Real code is cheap (this repository's ./crates, release binaries, median of three quiet runs on one machine: main 3.17 s → this branch 4.66 s, +47%, for 1173 → 1202 clusters and 100064 → 102855 mass), but a machine-generated file with 400 identically shaped methods produces 719k candidate edges. The issue records the measurements and why a size cap, an adjacency chain, and content-key grouping are each the wrong bound.

The shared-subtree rescue reaches inside a file

[FUSED-SHARED-SUBTREE-SAME-FILE]deslop-core/src/overlap/rescue.rs, deslop-core/src/pair/echo.rs

The rescue was cross-file only, so csharp-merge-drift's ApplyStandard / ApplyPremium published the statement fragments they share and never the methods. It now reaches them on three conditions:

  1. both endpoints are whole authored declarations, and disjoint;
  2. they still enclose a Merkle-equal clone of at least admission.shared_subtree_min_node_count nodes — authored code the edit never touched;
  3. the shared mass beyond that clone clears the same floor, which is [FUSED-SHARED-SUBTREE-ECHO] turned inward.

Condition 2 is the discriminator the plan was looking for, and shape and agreement could not supply it: the drifted pair measures overlap 0.84 and agreement 0.55, while the dart-issue-197 accessor family measures 0.81–0.88 and up to 0.56. The drifted pair keeps four whole statements; the family keeps none. Condition 3 is why csharp-merge-readafter still publishes the contiguous run its two methods share rather than the methods that wrap it.

ExactFunctionAnchors became ExactClones with two constructors — whole_functions_across_files for the echo rule and within_one_file for this one — sharing the enclosure arithmetic instead of duplicating it.

Where nothing was renamed, a consistent literal substitution is a parameter

[FUSED-CONTENT-GATE-PARAMETER]deslop-core/src/content/rename.rs

A drifted literal that echoes nothing contradicts the rename the identifier bijection claims; that one aligned literal is the whole difference between the #134 stride family and a reportable Type-2 clone. But where the bijection claims no rename, there is nothing for it to contradict, and [TECH-PMATCH-BAKER]'s prev-encoding applies to the literal alphabet exactly as it does to the identifier one: a substitution seen once is an unconstrained wildcard.

csharp-merge-manyholes keeps every identifier and every call and substitutes at all twelve literal positions. Judging it on agreement alone judged a literal-only copy on the one axis its own edit demolishes — and [AUTOFIX-MERGE-GATE] already calls that pair a clone too parameterised to merge mechanically. A repeated substitution stays constrained, which is what keeps a shape sibling out of the copy beside it; an inconsistent one stays constrained too.

The literal population is now measured by LiteralEvidence, the echo machinery moved to content/rename/literal_echo.rs (rename.rs had passed 500 lines), and the three copies of the aligned-literal frontier walk — which deslop, run against this branch, reported as a cluster of its own — collapsed into one literal_positions pass that every literal measure reads.

Specs and plan

  • docs/specs/fused.md[FUSED-CANDIDATE-BUCKET-STAR] extended, [FUSED-SHARED-SUBTREE-SAME-FILE] rewritten from a known gap into three conditions, and [FUSED-CONTENT-GATE-PARAMETER] added.
  • docs/plans/same-file-rescue-plan.md — records what landed and checks off its acceptance list.

Breaking changes

None. No public API, CLI flag, report field, or spec ID was removed or renamed. ExactFunctionAnchorsExactClones is pub(crate).

How Do The Automated Tests Prove It Works?

Every change has a pin that fails if it is reverted. All are black-box: they drive the CLI against fixture repositories and assert against the rendered JSON report.

The findings that came back. same_file_rescue::a_shape_sibling_may_not_hide_an_exact_same_file_copy scans the same copied method twice — once alone, once with a differing sibling written above it — and asserts the two reports are identical, so recall demonstrably does not depend on write order. type3_enclosing_method::csharp_same_file_type3_reports_both_methods_in_one_cluster had been #[ignore]d under gh #492; the #[ignore] is gone and it asserts DriftLimits.cs:3-13 / :15-29 as one cluster with every fragment view absorbed. same_file_rescue::a_literal_only_copy_inside_one_file_is_a_finding pins Sprawl.cs:3-12 / :14-23. dart_forwarding_fail_open's five controls each pin their published pair by line range and by the member and helper names that must appear in the reported text — wrappers_sharing_a_body_keep_the_family_visible additionally asserts that resetAlpha, resetBeta and resetGamma are not reported, so the fix cannot be mistaken for admitting the whole family.

Each of these goes through expect_only_finding_is_the_pair, which asserts the whole published contract at once: one visible cluster with nothing hidden, both occurrence extents, the wire mass formula, no pair-only evidence on a cluster surface, rank one in the worst band, the number of distinct occurrence texts, that only the pair's own lines are counted duplicated, and that duplication_percent divides those exact line counts. A percentage inflated by a shape match fails it.

The suppressions that had to survive. dart_issue_197_single_file_structural_only (zero visible clusters, zero duplicated lines, zero percent on the vendored meilisearch settings region), issue_134_structural_only_not_nearly_identical (the renamed stride family stays rejected before closure), ts_issue_284_produce_then_assert (three unrelated test scenarios stay hidden while a byte-identical control clone in the same run stays visible, ranked first — the two-sided contract in negative_pin.rs), cross_cluster_collapse::widest_same_declaration_view_is_the_published_finding (the wider method view must not displace the exact run), cross_cluster_collapse::padded_windows_straddling_a_verbatim_block_publish_the_block (a padded window is not a declaration and is judged as before), python_issue_103_helper_call_sites, the three issue_190 data-table modes, and both refactor_merge_refusals same-file pins.

Registry and equivalence. skip_policy_contract drops from 17 curated skips to 16 with gh #492's entry gone, and its per-issue tally is asserted against the registry itself, so removing a skip without removing its accounting fails. overlap::rescue::shard_equivalence_tests::sharded_rescue_matches_serial_outcomes still proves the parallel rescue produces byte-identical overlaps and counters to the serial path across 8,192 pairs, including at shard boundaries.

Full run (cargo test --release --workspace --all-targets --features deslop-core/live,deslop-lsp/profiling, the make test invocation): 464 + 193 + 192 + 61 + 80 + 5 + 125 + 121 = 1241 tests pass, 0 fail, 17 ignored. cargo clippy --all-targets --workspace -- -D warnings and cargo fmt --all --check are clean.

deslop was also run against this branch. Two clusters it reported inside the new code (a frontier walk written three times, and an evidence assertion restated across two suites) are removed above rather than explained away. Run again on the final diff, it reports 48 clusters that main does not, 12 of them in files this PR touches. Nine are one-line-shifted views over the four const Scenario literals in dart_forwarding_fail_open.rs:121-205, where main publishes two. Two are the three one-call #[test] wrappers at type3_enclosing_method.rs:288-316, and that family is a gap main already has: reduced to a fixture, both binaries publish it identically, because [CLONE-NOISE-LITERAL-VARIATION-CALLS] does not see literals nested inside a &span(..) argument. Within-file bucket completion exposes one more instance of it. The pin for that gap — a Rust helper-call-site fixture with a byte-identical control clone, red on main and on this branch — lands on its own branch so this one stays scoped to the reviewer's items.

For AI

Three admission-stage changes, all pair-scoped; no cluster acquires evidence.

PairBuilder::pair_within_files completes each structural-hash bucket within a file (O(k^2) per same-file group, grouped through a reused Vec<(FileId, usize)> scratch to avoid the per-bucket map [PERF-FLUTTER-TODO-MEMORY] removed). PairBuilder::gate no longer requires rescue_eligible pairs to cross files; the retained population stays bounded because candidate_ranges_are_valid already refuses anchor-free same-file pairs whose endpoints overlap, which is the nested-window population.

RescueContext::measures splits the scope: crosses_files || (!ranges_overlap && aligned_function(left) && aligned_function(right) && shares_a_copied_interior). shares_a_copied_interior reads ExactClones::within_one_file(pairs, fingerprints).enclosed_nodes(left, right) >= SHARED_SUBTREE_MIN_NODE_COUNT. is_container_echo selects its anchor index by scope via RescueContext::echo_anchors. ExactClones::ordered now orders same-file pairs by byte_range.start (file id still decides across files), which is what makes first.covers(exact.first) && second.covers(exact.second) well-defined for two disjoint same-file endpoints. RescueTally::in_scope(bool) splits the old cross_file counter into cross_file and same_file.

LiteralEvidence::measure computes {aligned, constrained, affirming}. When RenameMapping::renames() (explained > identity) is false, a literal position leaves the constrained population iff it is substituted, explained by the modal bijection over substituted literal pairs, and occurs fewer than RENAME_CORROBORATION_MIN_OCCURRENCES times. Both the affirming == 0 && constrained > 0 veto and both coverage forms read constrained rather than the raw aligned count.

Refs #492, #496, #497, #506 — left open deliberately; this repository never closes issues from a PR.

abdushakoor12 and others added 9 commits September 3, 2026 17:26
…nd read literals as parameters where nothing was renamed

Four recall defects, all inside one file.

[FUSED-CANDIDATE-BUCKET-STAR] A structural-hash bucket paired every
member only with the member that sorted first. Inside one file that
member decides every pair, so when it was the one that *differed* the
byte-identical copy behind it was never a candidate at all: one
unrelated sibling deleted an exact duplicate from the report. Members
of a bucket that share a file are now paired with each other, all of
them — recall may not depend on what else happens to share the shape.

[FUSED-SHARED-SUBTREE-SAME-FILE] The shared-subtree rescue was
cross-file only, so two methods that drifted apart inside one file
published the statement fragments they share and never the methods. It
now reaches them on three conditions: both endpoints are whole authored
declarations and disjoint; they still enclose a Merkle-equal clone of
at least the shared-subtree node floor — authored code the edit never
touched; and the shared mass *beyond* that clone clears the same floor,
which is the echo rule turned inward. Shape and agreement cannot
separate a drifted copy from a shape family (csharp-merge-drift
measures overlap 0.84 / agreement 0.55 against dart-issue-197's
0.81-0.88 / up to 0.56). Copied code can.

[FUSED-CONTENT-GATE-PARAMETER] Where the identifier bijection claims no
rename, Baker's prev-encoding reaches the literal alphabet too: a
substitution seen once is an unconstrained wildcard, not a
contradiction. Two declarations keeping every identifier and every call
while substituting at twelve literal positions are one parameterised
method, and judging them on agreement alone judged a literal-only copy
on the one axis its own edit demolishes. A repeated substitution stays
constrained, so a sibling sharing a shape and no byte cannot join the
copy beside it; an inconsistent one stays constrained too.

[CLONE-NOISE-LITERAL-VARIATION-CALLS] A call consumes its receiver, not
just its arguments: expect(generated).toContain(...) consumes
`generated`. Reading the argument list alone let a scenario family
block its own suppression.

rename.rs and rescue.rs had gone past 500 lines and are split.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UawgpVYqYhB3tKYGwEbJjP
…the forwarding band's four inverted pins

[REPAIR-RENAME-ANCHOR-MASS] The anchor factor prices coincidence:
scarce affirming positions might be two windows that happen to line up.
Two whole authored declarations are not a window alignment — the author
wrote both, opening brace to closing brace — so the coincidence being
priced is weaker, and the half-saturation mass is lower for such a
pair. It is the mirror of [FUSED-CONTENT-GATE-INTERIOR], which finds
that coincidence stronger for a window carved out of one function.

That is what the forwarding band needed, and no content floor could
supply it. `Api`'s three distinct-route wrappers measure agreement
0.714 and the `#197` settings family 0.77-0.82 — above two of the pairs
that must publish — so no value sorts them. Anchor mass does: five
affirming positions for a one-line REST wrapper, eight and nine for a
two-statement business method.

`dart_forwarding_fail_open` states a positive contract for all five of
its fixtures and four of them asserted the opposite (gh #496, gh #497).
All five now publish the pair their documentation describes:
Calc.scaledDomestic/scaledExport, Api.resetDelta/resetEpsilon,
Pricing.standardTotal/premiumTotal, Ledger.standardTotal/premiumTotal
and Billing.quarterlyFee/annualCharge. `Pricing`'s renamed arrow pair
stays out at 0.375, exactly as its own fixture comment says it must.

Two consequences, both named where they land:

- `dart_issue_197_single_file_structural_only` convicts two components
  rather than one: the `resetX` wrappers close into a family they
  previously never reached at all. Nothing it asserts about the report
  changes — no cluster published, no line counted, no percentage moved
  — and the count stays exact.
- `deslop-core`'s `content_gate_rejects` used `dart-forwarding-business-pair`
  as its example of a content-rejected pair. That fixture is no longer
  one, so the pin now uses `csharp-issue-134-structural-only`, which
  still is, with both assertions unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UawgpVYqYhB3tKYGwEbJjP
… stale scope claims

The new `assert_percent_matches_lines` helper divided with a raw `as f64`
and suppressed `clippy::cast_precision_loss` to do it — the only linter
suppression in the whole crates tree, and a bypass of the converter whose
own documentation says every metric re-derivation goes through it "so no
assertion silently loses precision on the way to a comparison". It now
uses `loc_as_f64`, and the helper returns `Result` so the conversion can
fail loudly. Every assertion is unchanged.

Two comments still described the pre-rescue world as current: the
`type3_enclosing_method` note said the rescue that would carry
`csharp-merge-drift` "is cross-file only", and the admission summary in
fused.md said the same. Both now describe what the code does.
`literal_positions`, `affirming_literal_count` and
`substituted_literal_positions` each opened the same walk over the two
frontiers, filtering for the positions where both members carry a
literal — three copies of one loop, and `deslop` reported two of them as
a cluster against this branch.

The walk now happens once per pair, in `literal_positions`, and every
literal measure reads its result: the affirming count, the echo
candidates, and `LiteralEvidence`. `LiteralPosition` names the row so
the three signatures agree. One walk fewer per measured pair, and the
same values.
`dart_forwarding_fail_open` named the loop that checks each piece of
evidence reached the reported occurrence text; `same_file_rescue` open
coded the same loop twice, and `deslop` reported the pair as a cluster
against this branch. The helper moves to `common/verdict.rs`, beside the
other assertions every accuracy control shares, and both binaries call
it. No assertion changes.
…t star and the parameter rule

main landed #503 while this branch was in flight — a second answer to the
same forwarding band. It removes the same-file admission floor entirely,
so every pair pays `content_gate.support_floor` and only an unanchored
LSH-only pair pays `promote_floor`. That is a simpler answer than this
branch's, and it is merged, so it wins.

What that makes redundant is dropped: the scope-aware anchor mass, its
constant, `PairScope::authored`, `PairShape::authored`, and the spec
section for it. main's floor already admits Billing, Ledger and Pricing
at 0.727-0.75, so separating them by anchor mass would have been a
behaviour change buying nothing.

What main does not have is kept, because its own plan says so:

- the same-file shared-subtree rescue (gh #492), whose pin main still
  carries as a curated skip. The skip and its registry row go.
- the within-file bucket completion, which main never touched.
- [FUSED-CONTENT-GATE-PARAMETER], which reaches `csharp-merge-manyholes`
  at agreement 0.567 — below the support floor main relies on, and main's
  plan says so in as many words.

Both branches wrote `consumed_identifiers` independently, down to the
name; main's walks nested calls kind-aware and is the one that survives.
`dart_forwarding_fail_open`'s scenario table, `content_gate_admits.rs`
and the #197 family count are main's throughout.

One behaviour actually changed. At the eight-node floor the wrappers'
bodies are fingerprinted in their own right, and completing the bucket
inside `Api.dart` proves `resetDelta`'s and `resetEpsilon`'s are
byte-identical — the dead route the fixture exists to catch, which its
header says is visible only in the proven bodies because the
declarations differ by name. It publishes beside the family rather than
replacing it: the narrow view covers a strict sub-region, and
[PIPELINE-CLUSTER-SUBSUME] treats one-sided containment as two findings,
which `no_two_clusters_cover_the_same_physical_bytes` pins with the same
symmetric test. The scenario table gains `finer_sizes`/`finer_lines` to
assert it; the metrics do not move, those lines already being inside the
declarations they sit in.
…or's family

Three files were edited after the merge's last `git add` and missed the
commit, so the pushed tree still declared `PairShape::authored` and
`PairScope::authored` with nothing reading them — `cargo clippy` refused
`deslop-core` on `field 'authored' is never read`, and every job behind
the analyzer gate skipped.

The removal is now complete: both fields go, `pair_shape` stops deriving
the flag, and the doc comment no longer cites the spec section this
branch dropped.

`dart_forwarding_fail_open` gains what the same edit round added: the
`finer_sizes` / `finer_lines` scenario fields and `assert_lower_floor`,
which pin the byte-identical `(http.deleteMethod('/indexes/dup/settings'))`
that the eight-node floor proves inside `resetDelta` and `resetEpsilon`.
@MelbourneDeveloper

Copy link
Copy Markdown
Collaborator

@abdushakoor12 ... According to Claude

Three things wrong with the PR

  • The description documents work that is not in the diff. Two of its five sections — [REPAIR-RENAME-ANCHOR-MASS] (the authored half-mass) and [CLONE-NOISE-LITERAL-VARIATION-CALLS] (dataflow.rs, noise.md) — have zero occurrences in the diff. The first was reverted by commit 05f9d0f ("Finish dropping the scope-aware anchor mass"); the second already landed on main in Same-file pairs pay the same content floor, and four Dart recall pins assert what they claim #503. The "For AI" section says PairScope gains authored — it does not.
  • PairShape is leftover churn. It's a one-field struct wrapping the interior: bool it replaced. Its only reason to exist was the authored field that got reverted. Delete it or restore the bool.
  • Quadratic same-file bucket pairing with no cap. On this repo I measured 2.83 s → 4.04 s (+43%); the PR reports +23%. Same-file bucket completion is quadratic in bucket size on generated files #506 records the risk but nothing bounds it.

@MelbourneDeveloper

MelbourneDeveloper commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Review

I built this branch and main side by side and ran both against the fixtures. The detection work is correct — every silent band now reports, and every suppression fixture still suppresses. Two things need fixing before this merges, and neither of them touches the detector.

For Humans

The bug this fixes is real, and it's worse than the title suggests.

Take two methods in one file that are copied character for character. Deslop finds them: 77.8% duplication. Now write a third, unrelated method above them — one that shares their shape but none of their text and duplicates nothing at all. Deslop now reports nothing. 0.0%. The copy-paste vanished because an innocent bystander happened to sort ahead of it in the file.

Two other holes close with it: two methods that differ only in their numbers and strings were reported as nothing at all, and two methods that had drifted apart were reported as three disconnected scraps of statements instead of as the two methods. All three are fixed here, and none of the twelve fixtures that exist to stop false positives changed by a single character.

What still needs doing.

  1. The description on this PR describes work that isn't in it. Two of its five sections cover changes that were either reverted later on this branch or already shipped in Same-file pairs pay the same content floor, and four Dart recall pins assert what they claim #503. Someone reading this PR is reading about code that does not exist. It needs cutting.

  2. A leftover from that revert is still in the code. A small wrapper type was added to carry a piece of information that then got removed. It now wraps a single yes/no value and does nothing. It should go back to being the plain value it replaced.

  3. A known cost is being accepted, not solved. Comparing every method against every other method in the same file is fine for ordinary code, but a machine-generated file with hundreds of near-identical methods makes it explode. That's tracked separately as Same-file bucket completion is quadratic in bucket size on generated files #506 and is fair to leave for a follow-up — it's a speed problem, not a wrong-answer problem.

Fix 1 and 2 here. Leave 3 for #506.

For AI

You left this half-finished and then wrote it up as though it were finished. Do not open the follow-up PR until you have done the two items below in this branch. Everything you need is in this comment; there is no other document.

1. The PR body documents two changes this diff does not contain

gh pr diff 508 contains zero occurrences of each of: AUTHORED_RENAME_EVIDENCE_HALF_MASS, REPAIR-RENAME-ANCHOR-MASS, CLONE-NOISE-LITERAL-VARIATION-CALLS, consumed_identifiers, noise.md. Verify that yourself before you touch anything:

gh pr diff 508 > /tmp/pr508.diff
for s in AUTHORED_RENAME_EVIDENCE_HALF_MASS REPAIR-RENAME-ANCHOR-MASS \
         CLONE-NOISE-LITERAL-VARIATION-CALLS consumed_identifiers noise.md; do
  printf '%-42s %s\n' "$s" "$(grep -c -- "$s" /tmp/pr508.diff)"
done
  • [REPAIR-RENAME-ANCHOR-MASS] — "A whole authored declaration is not a window alignment." You reverted this in commit 05f9d0f2, whose own subject line reads "Finish dropping the scope-aware anchor mass". You correctly removed it from docs/specs/fused.md — spec and code agree, which is the one thing that saved this. You did not remove it from the PR body, so the body is now the only place in the repository asserting that a scope-aware half-mass exists.
  • [CLONE-NOISE-LITERAL-VARIATION-CALLS] — "A call consumes its receiver." consumed_identifiers is on origin/main already; it landed in Same-file pairs pay the same content floor, and four Dart recall pins assert what they claim #503 (git show origin/main:crates/deslop-core/src/cluster_filters/calls/dataflow.rs | grep consumed_identifiers). dataflow.rs and docs/specs/noise.md are not in this diff at all.

The For AI section repeats both. It asserts anchor_weight(anchors, scope) — the function still takes one argument — and PairScope gains authored — it does not. It also carries the Api / #197 / Pricing / Ledger / Billing agreement-and-rename measurement table as evidence for a mechanism that is no longer in the branch.

Do: delete both prose sections, both For AI paragraphs, and that measurement table. Delete [REPAIR-RENAME-ANCHOR-MASS] and [CLONE-NOISE-LITERAL-VARIATION-CALLS] from the "Specs and plan" bullet list, and the docs/specs/noise.md bullet with them. Re-check docs/plans/same-file-rescue-plan.md for surviving references to the anchor-mass route and to the forwarding band's measurements under a half-mass that no longer varies. Leave the three sections that are real — [FUSED-CANDIDATE-BUCKET-STAR], [FUSED-SHARED-SUBTREE-SAME-FILE], [FUSED-CONTENT-GATE-PARAMETER] — untouched; they are accurate and they are the whole value of this PR.

2. PairShape is dead churn from that revert

crates/deslop-core/src/content.rs introduces:

#[derive(Clone, Copy, Default)]
pub(crate) struct PairShape {
    pub(crate) interior: bool,
}

measure_pair_content_indexed now takes it instead of interior: bool, and content_gate::pair_shape constructs it from the same two scopes.enclosing() calls the old inline code made. One field, no behaviour change, no second caller. It existed only to carry authored alongside interior, and authored is gone.

Do: revert PairShape — restore the interior: bool parameter on measure_pair_content_indexed, drop the struct, drop the pair_shape helper in crates/deslop-core/src/pair/content_gate.rs, restore the inline interior computation in gate_verdict, and restore the plain false argument at the measure_pair_content call site. Then re-run the suite; nothing should move, and if anything does, that is a finding you report rather than absorb.

If you would rather land the authored half-mass than delete the struct, that is a separate PR with its own pins, not a quiet re-add here.

3. #506 stays open, deliberately

PairBuilder::pair_within_files emits O(k²) pairs per same-file structural-hash bucket at structural = 1.0. Those clear construction_survives unconditionally, so all of them are retained and all of them reach the content gate, which walks trees per pair. This reopens exactly what [PERF-FLUTTER-TODO-MEMORY] closed: the retained population, not the raw LSH volume, is supposed to be what memory scales with.

Measured on ./crates (1101 files, default --min-nodes 30, --no-incremental, release, same machine, both binaries built from source):

clusters total mass wall clock
main 1174 100134 2.83 s
this branch 1200 102733 4.04 s

That is +43%, not the +23% the PR body claims. Re-measure and correct the figure while you are editing the body. The +26 clusters and +2599 mass are the recall this PR exists to deliver and are not a concern.

This one is legitimately a follow-up: it is a cost, not a wrong answer, and #506 already argues that a size cap, an adjacency chain, and content-key grouping are each the wrong bound. Do not bolt a cap onto this PR to make the number look better.

What I verified, so you do not redo it

Both binaries built from source. Base = worktree at main-equivalent (54d06fb2; every pipeline file this PR touches is byte-identical to origin/main), head = that same tree with gh pr diff 508 applied by patch -p1. Every fixture scan below used --min-nodes 12 --no-incremental --nohtml --notext.

Recall recovered:

Scan main this branch
star-shadow, no sibling Rates.cs:3-9 | 11-17 m46, 77.78% identical
star-shadow, with sibling 0 clusters, 0.0% Rates.cs:11-17 | 19-25 m46, 53.85%
csharp-merge-manyholes 0 clusters, 0.0% Sprawl.cs:3-12 | 14-23 m81, 83.33%
csharp-merge-drift :7-8|9-10|19-20|25-26 m54, :9-12|25-28 m32, :5-8|17-20 m31 — 53.33% DriftLimits.cs:3-13 | 15-29 m72, 86.67%

The star-shadow rows are the finding. The two scans differ only by ApplyAlpha, which duplicates nothing; on main its presence deletes a byte-for-byte copy-paste pair from the report entirely. Recall depended on source write order.

Suppressions intact — byte-identical output on both binaries:

Fixture Result (both)
dart-issue-197-settings-getters 1 cluster index.dart:30|34 m12, 9 hidden, 0.57%
python-issue-103-helper-call-sites cross-file control clone only, 21.62%
dart-issue-119-role-mismatch 0 clusters
dart-issue-119-same-role 0 clusters
dart-issue-119-same-behavior-reachable 3 clusters, 86.67%
dart-forwarding-fail-open Calc.dart:12-14 | 16-18 m34, 28.57%
dart-forwarding-business-pair Pricing.dart:35-38 | 40-43 m30, 16.67%
dart-forwarding-duplicate-route 5-occurrence family m108, 33.33%
csharp-shape-only-samefile 0 clusters
csharp-issue-134-structural-only 0 clusters
csharp-merge-readafter Prefix.cs:5-10 | 16-21 m51, 44.44%
csharp-same-file-class-echo Ledgers.cs:7-22 | 31-46 m52, 61.54%

Two claims I checked because they read wrong, and are not:

  • Dropping pair_crosses_files from PairBuilder::gate. candidate_ranges_are_valid short-circuits to true whenever structural > 0.0, which reads as no bound at all — but rescue_eligible requires structural <= 0.0, so every newly-retained pair does pay the range check. The body's reasoning holds.
  • echo.rs::ordered widening from left.file_id <= right.file_id to (file_id, byte_range.start) <=. For a cross-file pair the tuple is decided by file id alone, so the cross-file anchor index is unchanged; the new ordering only reaches same-file lookups, which is where first.covers(exact.first) && second.covers(exact.second) needs it. Correct.

Reproducing all of the above

# base binary, from a clean main checkout
cargo build --release -p deslop

# head binary: copy the workspace, apply the PR, build
mkdir -p /tmp/pr508 && rsync -a --exclude target --exclude node_modules --exclude .git \
  Cargo.toml Cargo.lock crates docs scripts site /tmp/pr508/
gh pr diff 508 > /tmp/pr508.diff
( cd /tmp/pr508 && patch -p1 < /tmp/pr508.diff && cargo build --release -p deslop )

# the star-shadow fixture, written out directly so it needs no test harness
python3 - <<'PY'
import os
COPIED = ('        policy.SetCeiling("dup", 250);\n        policy.EnableAlerts("dup");\n'
          '        policy.Audit("dup", 250);\n        policy.Commit();\n')
SIB    = ('        policy.SetCeiling("alpha", 100);\n        policy.EnableAlerts("alpha");\n'
          '        policy.Audit("alpha", 100);\n        policy.Commit();\n')
m = lambda n, b: f'    public void {n}(RatePolicy policy)\n    {{\n{b}    }}\n'
for tag, with_sib in (('alone', False), ('shadow', True)):
    os.makedirs(f'/tmp/star-{tag}', exist_ok=True)
    sib = m('ApplyAlpha', SIB) + '\n' if with_sib else ''
    open(f'/tmp/star-{tag}/Rates.cs', 'w').write(
        f'public class Rates\n{{\n{sib}{m("ApplyDelta", COPIED)}\n{m("ApplyEpsilon", COPIED)}}}\n')
PY

# scan each fixture with each binary and read the JSON
for b in ./target/release/deslop /tmp/pr508/target/release/deslop; do
  for fx in /tmp/star-alone /tmp/star-shadow \
            crates/deslop/tests/fixtures/csharp-merge-manyholes \
            crates/deslop/tests/fixtures/csharp-merge-drift; do
    $b "$fx" --min-nodes 12 --no-incremental --nohtml --notext --output /tmp/out >/dev/null 2>&1
    echo "== $b $fx"
    python3 -c "
import json; d=json.load(open('/tmp/out.json')); m=d.get('metrics') or {}
print(' dup_loc', m.get('duplicated_loc'), 'pct', m.get('duplication_percent'))
for c in d.get('clusters', []):
    print('  mass', c['mass'], ' | '.join(f\"{o['path']}:{o['start_line']}-{o['end_line']}\" for o in c['occurrences']))
if not d.get('clusters'): print('  NO CLUSTERS')
"
  done
done

# the timing and cluster-count table
for b in ./target/release/deslop /tmp/pr508/target/release/deslop; do
  time $b ./crates --no-incremental --no-fail-over --nohtml --notext --output /tmp/self
done

…-field PairShape left behind by the anchor-mass revert

PairShape existed to carry the authored half-mass beside the interior flag. That half-mass was dropped in 05f9d0f, leaving a struct wrapping the single bool it replaced, with one caller and no behaviour. measure_pair_content_indexed takes interior: bool again, gate_verdict computes it inline, and measure_pair_content passes false. content.rs is byte-identical to main; content_gate.rs keeps only the ExactClones rename.
@abdushakoor12

Copy link
Copy Markdown
Collaborator Author

Both items done in this branch, third left for #506.

1. Body. Cut both sections that weren't in the diff, both For-AI paragraphs, the agreement/rename table, and the [REPAIR-RENAME-ANCHOR-MASS] / [CLONE-NOISE-LITERAL-VARIATION-CALLS] / noise.md bullets. Also cut the two "consequences" bullets: rename_literal_monotonicity.rs and content_gate_rejects.rs turned out to be byte-identical to main, so neither was a consequence of this PR either. The plan doc had no surviving anchor-mass references.

2. PairShape. Reverted in 15cbefd. content.rs is byte-identical to main again; content_gate.rs keeps only the ExactClones rename. Nothing moved: 1241 pass, 0 fail, clippy and fmt clean, CI green on all 16 checks.

3. Timing. Re-measured from source, release, median of three quiet runs on ./crates: main 3.17 s → 4.66 s, +47%, for 1173 → 1202 clusters. The review's +43% was right and the body's +23% was not; corrected. No cap added.

One thing a re-scan will show. Of the clusters this branch adds on our own code, 12 land in files this PR touches. Nine are one-line-shifted views over the four const Scenario literals in dart_forwarding_fail_open.rs (main publishes two). Two are the three one-call #[test] wrappers in type3_enclosing_method.rs:288-316, and that family is a gap main already has — reduced to a fixture, both binaries publish it identically, because [CLONE-NOISE-LITERAL-VARIATION-CALLS] doesn't see literals nested inside a &span(..) argument. Bucket completion just exposes one more instance. The red pin for it (Rust helper-call-site fixture with a byte-identical control) is going up on its own branch so this PR stays scoped to the review's items.

@MelbourneDeveloper

Copy link
Copy Markdown
Collaborator

Verified this against the branch. Items 1 and 2 are done: PairShape and pair_shape are zero occurrences in the whole diff, content.rs is byte-identical to main by blob SHA, content_gate.rs keeps only the ExactClones rename, and — the check that matters — every identifier and spec ID the rewritten body asserts is now actually present in the diff. Item 3 is correctly left to #506: the only bound added is BUCKET_MIN_MEMBERS_FOR_WITHIN_FILE_PAIRS = 3, which is a no-op short-circuit rather than a cap.

This can merge on one proviso: run scripts/compare-versions.sh between main and this head before it lands, and confirm SCORE.md shows no new false negatives and no new false positives against the register. This PR changes the admission stage, no corpus gate runs in CI yet, and the +26 clusters are recall by argument, not by measurement. The register is the only thing that settles whether anything slipped.

Two loose ends, neither blocking the merge:

  • The body says the skip registry "drops from 17 curated skips to 16". The code says CURATED_SKIPS 18 → 17 and SKIPS_PER_ISSUE 5 → 4. Off by one on both sides.
  • crates/deslop/tests/common/mod.rs goes 504 → 541 lines and crates/deslop/tests/dart_forwarding_fail_open.rs goes 458 → 516, both past the 500-line limit — the second crossed it in this PR.

Three marked conflicts, all in import blocks, all resolved as the union of
both sides:

- overlap/rescue.rs, pair/content_gate.rs — main added `token_carried` and
  `alignment_required` (= `rescue_eligible || token_carried`) while this
  branch renamed `ExactFunctionAnchors` to `ExactClones`. Kept the branch's
  type name and main's predicates; `rescue_eligible` is no longer referenced
  in rescue.rs, so `alignment_required` at the call site stands.
- tests/common/mod.rs — both sides appended at the same spot. Kept both:
  main's `go_scope` module and this branch's `assert_occurrence_extents`
  are the Go-specific and generic halves of [PIPELINE-CLUSTER-EXACT-SCOPE].

The real defect was one git merged clean: both sides split content/rename.rs
into a submodule under different filenames, so the merge kept two copies of
the same module — main's rename/echo.rs (a pure move of the base functions)
and this branch's rename/literal_echo.rs (the same functions refactored to
take `LiteralPosition` so the frontier is walked once per pair). Deleted the
duplicate, kept the refactored module, and repointed main's new
rename/consistent.rs at the four-argument `literal_echoes` — passing
`literal_positions(canonical, member)`, which is exactly what the three-
argument form computed internally, so its semantics are unchanged.

make test: 1302 passed, 0 failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@abdushakoor12
abdushakoor12 merged commit 77b4681 into main Sep 6, 2026
17 checks passed
@abdushakoor12
abdushakoor12 deleted the same-file-recall-band branch September 6, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants