Independent clone registers, a corpus accuracy gate, and the detector fixes that take it to 100%

[MelbourneDeveloper]

TLDR

Gives Deslop an independent ground truth — 12 blind-judged clone registers covering 186 judged pairs — wires it into CI as a pass/fail accuracy gate, and lands the detector fixes that take the register score from 98.4% to 100.0%.

Details

The clone register — independent ground truth ([CORPUS-REGISTER])

Until now nothing outside Deslop said what a duplicate is. corpus/register/*.json now does: for each pinned repository, a list of pairs judged CLEARLY IN (a real duplicate) or CLEARLY OUT (not one), decided by judges working from the code alone. A CLEARLY IN nobody reports is a false negative; a CLEARLY OUT that gets reported is a false positive. Both are bugs, and both now fail a build.

Twelve registers ship: axios, bloc, click, cobra, django, fsharp.data, guzzle, nest, polly, ripgrep, tokio, zod.

The judging pipeline is built so a verdict can't be contaminated by this codebase:

• [CORPUS-REGISTER-WORKSPACE] — make judging-folder (scripts/corpus/prepare-judging.sh, scripts/corpus/register-workspace.mjs) builds the judge's folder outside this repository. A judge who can read Deslop's source is contaminated and every verdict from that pass is void. the_judging_protocol_never_names_this_project enforces it.
• [CORPUS-REGISTER-QUEUE] — corpus/judging-queue.json is how a new repository gets in; it has no register until it's been judged.
• [CORPUS-REGISTER-MERGE] — make merge-verdicts folds independent passes back in and imports only pairs every source agrees on. Everything else is left out and listed, mechanically, in docs/reports/verdict-merge.md (1405 imported, 363 left out across 3 sources).
• [CORPUS-PIN] — every target is url#<40-char commit id>#language. Never a tag, branch or version, so a corpus run is reproducible.

The accuracy score and the gate ([CORPUS-SCORE])

New corpus-score binary in deslop-test-support (src/bin/corpus-score.rs, src/corpus_score/). It measures a scan (measure), scores every judged pair against the registers (score), and renders SCORE.md mechanically — no figure in that document is typed by a human or an agent.

• Score is 100 × correct / judged across the corpus, never the mean of per-repo scores.
• The gate is absolute FN/FP counts in corpus/register/score-thresholds.json (defaults 0/0), never a percentage. Thresholds live in that one file — not in the Makefile, not in CI YAML.
• Clusters, wall time, CPU and peak RSS are reported beside the score and never folded into it. Cost is description; accuracy is the score.
• The scorecard now opens by stating its own scope — how many repositories, how many distinct languages, and which — so a partial run can never be read as a full one. Rendered by scope_line in corpus_score/render.rs.

Wiring:

• CI — new corpus-score job in .github/workflows/ci.yml, a required dependency of the CI gate. It runs scripts/corpus/score-gate.sh over the two smallest judged repositories so it finishes in seconds, and uploads the scorecard if: always() — a failing run is exactly when someone needs to read which pair broke.
• Make — make score-gate runs the identical script CI runs, so green locally means green in CI. make compare runs the two-engine comparison over the whole register corpus.
• scripts/compare-versions.sh was rewritten (525 lines changed) to rebuild each engine from a clean git archive extract, abort if both cycles produce the same binary hash, and score both with one scorer built from the working tree.

Detector fixes the register exposed

• [FUSED-CONTENT-GATE-RENAME] — a consistent rename is now admitted whatever its literals do. The pooled rename axis prices a rename by its anchors, and a rename can be complete yet anchor-poor: sibling accessors whose literal keys rename with the accessor ('Name' beside getName) preserve no literal and read 0.0. Three such families were being dropped before closure despite being one code written twice. The measured route now admits on pooled support or on a contradiction-free bijection with at least one corroborated substitution and at least as many names kept as renamed. The refusals that matter stay: an inconsistent substitution still blocks (this is what keeps a minified polyfill out of axios's createInstance), as does a family that renames more names than it keeps. Implemented in content/rename.rs, split into rename/consistent.rs and rename/echo.rs; read by pair/content_gate.rs.
• [PIPELINE-FINGERPRINT-MERKLE-ROOT] — the synthetic __file__ root stays a candidate view by default (a module copied whole, import line and all, is one duplication at the extent of the file), but it is denied a view when the file carries a prologue its language mandates. Go's package clause is dictated by the directory the file lives in, not chosen and copied by an author; as the widest range in the file it was winning the same-file collapse over the declaration that was actually copied, publishing two Go files each holding one renamed function as alpha.go:1-13 vs beta.go:1-13, package clause included. The mandated clauses are named per language in boilerplate.rs (is_mandated_prologue, today Go's package_clause); fingerprint.rs (is_viewless_root) applies it beside the existing only-child rule. fpcache::blob::SEMANTIC_EPOCH moves 4 → 5 so a cache warmed before this change cannot keep serving Go whole-file fingerprints.
• [PIPELINE-CLUSTER-ELECT] — a token bridge may no longer weld two structural families into one cluster. New cluster_filters/structural_families.rs (+ containers.rs) splits such a component into one cluster per region, within a single language only — a cross-language component exists only because [CONFIG-CROSS-LANGUAGE] was opted into, and splitting it would delete the finding that opt-in asks for.
• [PIPELINE-CLUSTER-ELECT-CONTAINER] — a container that merely concatenates a family the component already holds is elected out, so the family publishes at its own extent.
• [FUSED-SHARED-SUBTREE] / content gate — a pair is "unanchored" only once its alignment has been measured and failed the overlap floor. An unmeasured overlap was reading as no alignment and sending token-carried pairs to the stricter promote floor.

Deduplication of Deslop's own code

The tool detects duplication, so its own codebase has to be exemplary:

• cluster_filters/node_search.rs — one range-bounded tree-sitter search shared by every filter that gathers nodes inside a member's reported bytes, replacing per-filter copies across calls, python* and rust filters.
• lang/merge_emit.rs — the merged-helper emission shape shared by C#, Dart and Rust; each language now supplies only placement, parameter spelling, call rendering, its brace style (BraceStyle: C# puts the body's brace on its own line, Dart and Rust on the declaration line) and its insertion seam (InsertionPoint: C# writes the helper directly after the class brace, Dart and Rust at the start of the first occurrence's line). The C# golden RateLimits.merged.cs pins the Allman shape; refactor_merge::csharp_leafgap_cluster_merges_to_golden fails if the shared emitter ever imposes one language's layout on another.
• deslop-mcp/tests/common/rpc.rs and deslop-lsp/tests/common/session.rs — shared test harnesses; deslop-mcp/tests/cli.rs drops 436 lines.
• Skills moved to .agents/skills/ with .claude/skills/* as symlinks, so one copy serves every agent host.

Test harness

deslop-mcp/tests/common/mod.rs no longer leaves a child's log stream on a pipe nothing reads. The LSP logs about 9 KiB per analysis pass to stderr; a pipe with no reader blocks its writer at the OS buffer (64 KiB), and on Linux, where the file watcher runs a pass for every edit a test makes, the server wedged mid-pass holding the session lock while the MCP refresh request behind it waited forever (t6_seed_cache_does_not_advance_on_incremental_edits hung until the job timed out; on macOS the same sequence stays 244 bytes under the buffer). The LSP's stderr now goes to .deslop/lsp-stderr.log in the test workspace, and the LSP's post-handshake stdout and the MCP's stderr are drained on background threads.

Portability

deslop-mcp/tests/common/mod.rs imports std::io::BufRead only inside the #[cfg(unix)] helper that reads a pid line, since the stdio request loop moved to rpc.rs; the Windows build of tcp_transport.rs, which compiles that module without the unix helpers, no longer trips -D unused-imports.

Files over the limits were split, not suppressed

corpus_score/render.rs → render/cells.rs; corpus_score/tests.rs → tests/render.rs; content/rename.rs → consistent.rs + echo.rs; structural_families.rs → containers.rs + tests/. Every file stays under 500 lines with no #[allow] anywhere.

Breaking changes

None to the CLI, the report format, or any public API. The CI aggregate job gains a new required dependency (corpus-score), and corpus/*.json single-repo manifests were replaced by corpus/register/*.json plus corpus/judging-queue.json.

How Do The Automated Tests Prove It Works?

The register itself is tested, not trusted — crates/deslop/tests/corpus_register_contract.rs:

• every_register_pins_a_commit_and_at_least_one_judged_pair — no empty register can silently score 100%.
• every_judged_entry_names_a_pair_it_actually_read and every_range_is_a_well_formed_non_empty_span — a verdict must cite two real spans, so a malformed entry can't be scored as a pass.
• an_empty_clearly_out_list_states_why_it_is_empty — a register with no false-positive ground truth has to say so rather than quietly grading only recall.
• the_judging_protocol_never_names_this_project — the blind-judging guarantee, asserted rather than assumed.

Pins are reproducible — crates/deslop/tests/corpus_commit_pins.rs: every_corpus_repository_is_pinned_to_a_full_commit_id, no_corpus_repository_carries_a_version_beside_its_commit, and nothing_fetches_a_corpus_repository_by_name fail if any target drifts to a tag or branch — which would make every score irreproducible.

The rename fix — crates/deslop/tests/type2_rename_literal_drift.rs drives the CLI against three new fixture repos and asserts the clusters, at the right spans:

• renamed_accessor_pairs_with_renamed_literal_keys_publish (PHP cookie.php — literal keys rename with the accessor, so zero preserved literals)
• renamed_constant_family_across_files_publishes (tls_version.php / curl_factory.php — a prefix rename seen three times)
• renamed_test_groups_in_one_file_publish (Dart edit_todo_event_test.dart)

Each asserted through assert_visible_clone_covering, which pins the file and the line spans — reverting the gate change drops all three clusters to nothing.

The Merkle-root rule, both sides — crates/deslop/tests/cluster_extent_alignment.rs: a_go_pair_is_not_padded_with_the_prologue_and_types_of_one_file against the new go-cluster-extent-alignment fixture, plus assert_only_authored_rows_are_duplicated and assert_duplicated_loc_within_honest_ceiling — these fail if a package clause or import block is ever counted as duplicated LOC again. The Go scope contract (crates/deslop/tests/common/go_scope.rs) is called by every Go suite, so the regression can't come back in a corner one test doesn't look at. The other side — a whole module copied with its import line is still reported from line 1 — is pinned by the unchanged python_inherited_contract_boundary (invoice_worker.py:1-14 / user_worker.py:1-14), js_ts_extensions (inventory.js:1-17 / ledger.mjs:1-17 beside the three-file function family), verbatim_subgroup_survives_noise (the four-line constant table) and js_ts_false_positive_filters (the byte-distinct real_a.ts:1-8 pair). crates/deslop-core/src/fingerprint/tests.rs isolates the rule per language: a Python module gets exactly one whole-file view, a Go file gets none while both of its functions stay views, and a root that re-describes its only child yields to that child. fpcache/tests.rs::the_blob_format_revisions_are_pinned pins the epoch at 5.

The cluster-election rules — crates/deslop/tests/csharp_merged_clone_families.rs: a summing loop copied across two files and a multiplying loop copied across two more are each reported alone, and without the rule are reported nothing at all when scanned together. Region cases the E2E can't reach (the four-file bridge, nesting that must survive it, an enclosing view whose nested run reaches a file it does not) are pinned in cluster_filters/structural_families/tests.rs and tests/container_tests.rs.

The scorer — crates/deslop-test-support/src/corpus_score/tests/ (15 tests): the gate tests prove a breach names the measure, the allowance and the actual (a_breached_gate_renders_as_a_failure_that_names_the_measure), that a failing card never also claims to pass, and that an unmeasured run renders as absent rather than as free (an_unmeasured_run_renders_as_absent_rather_than_as_zero — asserts 0 MB never appears). the_scorecard_states_how_many_repositories_and_languages_it_covers renders three repositories across two languages, with one language deliberately carrying two repos, so a language count can't be a repository count in disguise; it also pins the singular (1 repository, never 1 repositories).

The wiring can't drift — three new node --test gates run in make lint: score-gate-source.test.mjs (CI's job and make score-gate run the same script, thresholds read from score-thresholds.json and hardcoded nowhere), judging-workspace.test.mjs (the blinded folder contains no path back to this repo), verdict-merge.test.mjs (only unanimous pairs are imported; everything else is reported).

End to end, the two-engine comparison over 7 registered repositories across 7 languages (csharp, go, javascript, php, python, rust, typescript) moves 183/186 → 186/186 judged pairs correct, 98.4% → 100.0%, with false negatives 2 → 0 and false positives 1 → 0. GATE PASS.

For AI

Ground truth is corpus/register/<repo>.json: { commit, protocol, clearly_in: [...], clearly_out: [...] }, each entry a pair of {file, range} endpoints. Scoring is a set-membership check of judged pairs against reported cluster occurrences — an IN pair is correct when some reported cluster holds occurrences covering both endpoints; an OUT pair is correct when none does. Cluster ids are not comparable across engines (the id hash re-keys), so shared/gained/lost cluster columns are descriptive only and gate nothing.

corpus_score is a three-stage pipeline: measure wraps a scan and records RunCost { elapsed_ms, peak_rss_mb, cpu_seconds, binary_sha256 }; score reads the per-engine reports named by run.json and emits score.json + SCORE.md; --gate exits non-zero on any Breach. Rendering is pure — Scorecard -> String — so the render tests need no filesystem. scripts/compare-versions-summary.mjs renders INDEX.md and deliberately performs no arithmetic; every figure it prints is pre-computed in Rust, per the one-place-for-calculations rule. That is also why the scope line is not yet in INDEX.md: counting repositories in JS would breach it, and doing it properly requires the scorer to emit a pre-rendered scope string into score.json.

Gate-relevant invariant for [FUSED-CONTENT-GATE-RENAME]: admission is now support >= support_floor || pair_rename_is_consistent(pair), where the second disjunct requires aligned frontiers, no identifier position contradicting the modal bijection, corroborated_substitutions >= 1, and kept_names >= renamed_names. The pooled rename_consistency axis is unchanged and still routes; only the refusal it alone could issue against a rename is removed. Corroboration sources: same symbol renamed identically twice, a literal echoing the substitution ([REPAIR-RENAME-LITERAL-ECHO], content/rename/echo.rs), or sibling symbols sharing one transformation (prefix/suffix), which is what admits the STREAM_CRYPTO_PROTO_TLSv1_{0,1,2} -> CURL_SSLVERSION_TLSv1_{0,1,2} family off three occurrences of one prefix rename.

🤖 Generated with Claude Code