Localization vNext shadow comparison - #3
Open
harneet2512 wants to merge 14 commits into
Open
Conversation
…all-first legacy floor Generic typed structure was closing issue-specific behavioral coverage. _roles_for maps RAISES/CATCHES to expected_behavior/transition and guard/conditional_return to invariant/expected_behavior regardless of the issue; EvidenceUnit.create certifies every role at confidence >= 0.9, and typed graph rows arrive at 1.0. _coverage_admit then closed a required role on the first certified carrier and deferred the rest as redundant, so an unrelated property could evict the relevant region (ext2_js_express_init_middleware: gold lib/middleware/init.js displaced by lib/application.js, gold region first seen around discovery 89). Evidence now separates roles (what it describes), certified_roles (what the source certifies) and issue_roles (what it may satisfy for THIS issue). Admission, coverage, redundancy, coverability, source-region roles and role-class tracking read issue_roles. Edges and properties get issue eligibility only when attached to a node conditioned by exact identifiers, lexical retrieval, node FTS or body BM25; generic rows stay support-only, and a conditioned support row consolidated with a certified structural fact at the same region still yields certified marginal coverage. The comparison carries the exact model-visible legacy file order into shadow discovery as ranking_prior_only rows: they rank, carry no issue roles and are never admissible as behavioral evidence, so legacy top-1/top-8 cannot be displaced by support-only novel noise. Explicit paths and tracebacks remain hard-provenance overrides. Two further defects fixed here: - EvidenceUnit.create hashed the un-normalized issue_roles into evidence_id while storing the normalized (intersected) value, so two state-identical units could split into two ids and break dedup/consolidation. Byte-neutral for every in-tree caller. - The floor makes vnext_ranked_files a superset of the legacy order, which would have made a post-fix ordering change unattributable. Prior-only regions are now marked deterministically (never inherited from whichever support row sorts first) and the sealed artifact carries ranked_discovery_files_shadow_only plus a new_shadow_only scored column. Additive diagnostic: ranking, admission and the verdict are unchanged. vNext remains shadow-only behind GT_LOC_VNEXT_SHADOW; legacy rendering and delivery bytes are untouched. Verified: 110 passed across the four vnext suites (4 new tests, each RED before the change and RED again under targeted mutation); tests/pretask 912 passed with the same 12 failures that reproduce on pristine c5bd7d1; ruff check clean; pyright 0/0; git diff --check clean; ss_gate.py twice at PASS=11 / SKIP=1 (S8) / FAIL=0 / EXIT 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e name The first attempt (529db86) diagnosed correctly and implemented wrongly. Two independent adversarial verifications found eight defects; six were reproduced end to end. This replaces the eligibility apparatus with a fix at the origin. ROOT CAUSE, precisely: _roles_for granted BEHAVIORAL roles from pure structure. Any RAISES/CATCHES edge claimed `expected_behavior`; any guard or conditional_return property claimed `expected_behavior`; return_shape and exception_type claimed it too. Measured on an unrelated helper in an unrelated file, every one of those returned `expected_behavior` - a role required in 55 of 60 corpus cases. Typed graph rows arrive at confidence 1.0, so EvidenceUnit.create auto-certified them, and _coverage_admit closes a required role on the first certified carrier and defers everything behind it. Hence 53/60 cases reporting "all required roles resolved" while the admitted regions contained the gold file in 26/60 and nothing gold-matching in 34/60. Structure proves structure. A typed edge or property proves what the code DOES at that span; it never proves that this is the behavior the ISSUE is asking about. RAISES/CATCHES now prove `exception` and `transition` (raising IS control flow); guards prove `invariant`; READS/WRITES prove `state`; flow relations prove `transition`. None of them prove `expected_behavior`, which only issue-driven retrieval can support - at 0.6 confidence, uncertified by design. That makes the node-conditioning predicate unnecessary, so it is DELETED rather than patched, and three defects go with it: - conditioning was a NAME MATCH: it read non-empty issue_roles, whose broadening was gated on issue_mode == "behavior_described". 27/60 corpus cases are symbol_anchored/explicit_path/traceback, where it degenerated to "the node identifier contains the issue verb" - dropping gold regions, and reaching zero admitted regions on some shapes; - edge eligibility keyed on each side's OWN node, so an edge could never carry eligibility to a new region and graph traversal could not reach anything lexical retrieval had not already found; - the `if exact` narrowing inverted trust: the exact anchor node ended up LESS eligible than a fuzzy match, dropping the file the issue literally names. Remaining fixes: - _legacy_evidence: a ranking prior and an ordinary v7.4 row for one path were identity-identical (evidence_id excludes metadata/confidence/signal_rank), so dedup kept the v7.4 row and DELETED the floor pin. Corpus census: 159/218 pins evicted, 59/60 cases, gold's own pin in 45/60. Priors now carry provenance=("legacy_floor","model_visible_rank") and cannot collide. - _history_evidence rebuilt roles from the descriptive set and omitted issue_roles, re-granting every withheld role at whole-file granularity on every case (.git exists in all 60 harness repos). It now mirrors issue_roles. - consolidation unioned issue_roles and certified_roles independently, letting a certified-ineligible row lend certification to an eligible-uncertified row at the same key, while region_order used a per-unit predicate. One predicate now. - the attribution column is a TRUE unfloored order: the engine stamps each region with its rank under the ordering WITHOUT the legacy tier, and first_divergence is measured against it (floored value kept as first_divergence_floored). - new_shadow_only reports measured=false with None fields on artifacts sealed before the column existed, instead of a fabricated zero. Generality: the 60-case corpus has FULL graph capability on every case, so degraded regimes were never exercised. Adds a matrix dropping each of the 8 capabilities plus a nodes-only thin graph, asserting determinism, leakage 0, covered subset of required, unresolved as the honest remainder, and real spans. Verified: 91 engine + 24 comparison/oss/shadow tests; tests/pretask 926 passed with the same 12 failures that reproduce on pristine c5bd7d1; ruff clean; pyright 0/0; ss_gate.py twice at PASS=11 / SKIP=1 (S8) / FAIL=0 / EXIT 0. Six invariants are mutation-verified - each goes RED under a targeted mutation and GREEN on restore. One mutation exposed a vacuous test (no .git in the fixture, so the assertion loop never ran); it now builds a real repo and asserts the leg fired before checking roles. vNext remains shadow-only behind GT_LOC_VNEXT_SHADOW; delivery bytes unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sfied Run 30191986149 measured the previous commit paired within-run against the legacy arm on 52 clean cases: ranking safety WON (gold never ranked worse than legacy, 0 losses; strict Hit@8 dominance, 7 gains, McNemar p=0.0156; zero Hit@1 discordance) and admission REGRESSED (gold admitted 21/52 -> 16/52). The preregistered test was "gold admitted must rise, else the conditioning is over-suppressing and I was wrong". It fell. This is the diagnosis and the fix. SELECTION, NOT RETRIEVAL, IS THE DEFECT. Reading the four behavior_described losses: each reports unresolved=[] - coverage "satisfied" - while gold was not admitted. geopandas_file_io_driver RANKED gold #1 and admitted versioneer.py:1823-1824, a vendored build script, deferring 493 candidates. grpc_http2_client_transport ranked gold #2 and took a binarylog method_logger span. vue_renderer_patch ranked gold #1 and took two api* regions. _marginal was lexicographic (certified, independent, new_expected, new_fact, -dup, token_utility, fused_rank). The first four are role-label bookkeeping and fused_rank - the only relevance signal - sat LAST, breaking ties and nothing more, with token_utility above it creating pressure toward tiny spans. A region carrying a required role LABEL therefore beat the region retrieval ranked first, closed coverage, and everything behind it deferred as redundant. Removing expected_behavior from structural facts did not fix this: operation, transition and exception simply became the new vector. Any label works, because the objective was satisfied by labels rather than by relevance. The canonical selection objective is lambda*relevance - (1-lambda)*redundancy (MMR, Carbonell & Goldstein 1998; submodular summarization, Lin & Bilmes 2011). GT had the redundancy half and no relevance term. Repoformer (ICML 2024) shows retrieved repository context is often unnecessary or harmful and that selection must be evaluated on its own; RAGAs (EACL 2024) and ARES (NAACL 2024) require context relevance to be scored separately from coverage. Any interior lambda would be an invented constant, so this takes the lambda->1 limit with a hard novelty gate - novelty is a CONSTRAINT, relevance is the OBJECTIVE: before: (certified, independent, new_expected, new_fact, -dup, token_utility, fused_rank) after: (contributes, fused_rank, certified, independent, new_expected, new_fact, token_utility) Among candidates that add something new, the most relevant region wins the slot; coverage still terminates the loop but no longer decides what is delivered. The dead duplicate_penalty field (hardcoded 0) is dropped, keeping the ledger at 7. Because fused_rrf_score is file-granular this is implicitly two-stage - relevance picks the file, coverage picks the region within it - matching the file->element decomposition in Agentless and LocAgent. Measured: gold is the top file by fused relevance in 7/11 sampled cases, rank 2 in 1, rank 3 in 2, absent in 1. Two supporting fixes: - MODE-GATED STARVATION. With no structural fact granting expected_behavior, its only source was the query-driven broadening, gated on issue_mode == "behavior_described" (33/60 cases). In the other 27 nothing could cover a role required in 55/60 cases. Three of the seven admission losses are exactly that, with expected_behavior unresolved. The gate is removed: any node retrieved by FTS/BM25 or the embedder may cover required roles in every mode, at 0.6 confidence and uncertified, which is what prevents relaundering. Honest bound: 3/7, not 7/7 - the losses are NOT enriched for the starved modes (43% against a 44% base rate), so this was never the dominant cause. - EMBEDDER FAIL-OPEN. census_capabilities derived frozen_semantic from the presence of an .onnx file on disk - loadability, never execution. Across three sealed runs the embedder encoded ZERO passages on 0/60, 16/60 and 8/60 cases while every artifact reported the capability available. Because the legacy pipeline uses the same _EMBEDDER, a dark leg moved BOTH arms: legacy Hit@1 was 30 in run 30188004122 and 32 in run 30190408587 on identical inputs, with the legacy file list differing in 9/60 cases. _node_evidence now emits an execution witness; a capability claimed but never executed is downgraded before admission with reason declared_but_never_executed, so the roles it would have covered are reported UNAVAILABLE rather than as an ordinary retrieval miss. MEASUREMENT LAW: legacy and vNext share one graph.db and one _EMBEDDER in one process per case, so within-run paired comparison is valid and cross-run column comparison is not. Compare deltas, never raw columns. The earlier "1-case Hit@1 regression (29 vs 30)" tests at p=1.00 on 9-vs-8 discordant pairs and is retracted. An offline replay of the admission greedy over all 60 real sealed discovery sets was built to validate this before spending a run. Its control arm predicted 29/60 gold admitted under the OLD ordering against an actual 19/60, so it was discarded as unfaithful; the gap lives in _bounded_region, which needs the real repository files. On a 3-shard sample the control had tracked well (4 vs 5) - a small sample that agrees with you is not validation. Verified: 94 engine + 127 across four vnext suites; tests/pretask 929 passed with exactly the 12 failures that reproduce on pristine c5bd7d1; ruff clean; pyright 0/0; ss_gate.py twice at PASS=11 / SKIP=1 (S8) / FAIL=0 / EXIT 0; mechanical audit 17/17; nine invariants mutation-verified, each RED under a targeted mutation. Not benchmaxxed: no task id, repo name, gold label or corpus-tuned constant is read anywhere. vNext remains shadow-only behind GT_LOC_VNEXT_SHADOW. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ance fuse_by_evidence_class added a flat +1.0 to any file carrying explicit provenance. An RRF class term is capped at 1/(60+1) = 0.0164, so the bonus was worth 61 class-agreements against an achievable maximum of ~0.079. Measured over the 27,536 regions in run 30191986149: explicit-provenance regions have a median fused score of 1.032787 against 0.044404 for ordinary regions - a 23x gap. This was INERT while fused_rank was the last key in the lexicographic marginal, where it only broke ties. Making relevance the leading admission key (d1790c5) turned it load-bearing: any file the issue merely mentions textually now wins the admission slot regardless of retrieval agreement. Production case held_rust_serde_2950 admitted `my-binary/src/main.rs:0-0` - a whole-file span carrying only architectural_boundary, from a path mentioned in the issue - while gold sat at rank 9. It was also double-counting. Hard provenance already has its own top tier in region_order (slot 0) and its own qualifier in _marginal (new_fact). The additive bonus gave it a third, unbounded vote inside the relevance signal itself. Removing it leaves RRF in its textbook form and leaves provenance handled where it belongs. The `explicit` set that populated the bonus is removed with it rather than left populated-and-unread. Scope, measured before committing: exactly ONE case in the corpus admits an explicit-provenance region (held_rust_serde_2950), so this is correctness work on a latent defect, not a scoring lever. No run is dispatched for it; the measurement is banked for the next run that has other reasons to exist. RED first: a merely-mentioned path scored 1.016393 against four agreeing retrieval classes at 0.065574. The test pins that a corroborated region outscores a path the issue merely names. Mutation-verified: forcing fused to ignore retrieval agreement turns it RED. Verified: 128 across four vnext suites; tests/pretask 930 passed with exactly the 12 failures that reproduce on pristine c5bd7d1; ruff clean; pyright 0/0; ss_gate.py twice at PASS=11 / SKIP=1 (S8) / FAIL=0 / EXIT 0 (real exit codes, not through a pipe); mechanical audits 14/14 and 3/3. Not benchmaxxed: this REMOVES a magic constant rather than adding one. No task id, repo name, gold label or corpus-tuned value is read anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…9 defects behind it An adversarial 47-agent sweep confirmed 39 defects across the engine, the scorer and the new corpus builder. This lands them, and the most consequential is not on that list because nobody thought to check it. THE WINNER GATE HAS NEVER EVALUATED A SINGLE METRIC. evaluate_winner keyed a hard gate off a FIXED five-language tuple counted over region-scorable rows and returned INCONCLUSIVE before computing anything. Verified against the real completed run 30196352388: 0/60 rows are region_scorable, so it short-circuited - hit@1, hit@8, precision, latency and memory were never computed on ANY run in this corpus's history. It is also structurally unsatisfiable on a monolingual corpus, and the workflow fails only on OLD_WINS, so an empty verdict reads GREEN. The gate now counts the languages the corpus actually contains, treats absent region gold as UNMEASURED region metrics rather than an unjudgeable run, and its first real verdict on run 30196352388 is OLD_WINS: hit@8 0.750 -> 0.792 but file_precision@8 0.253 -> 0.115 over the same 60 paired rows. The recall gain reported all session had a paired precision cost that no instrument was measuring. SCORING (comparison.py): - file_recall counted matching CANDIDATES, not matched GOLD files - it could report 1.0 while most of the gold was missed, and could exceed 1.0. _matched_gold makes the numerator the gold set it covers. - _same_file is now unidirectional: a candidate may carry a checkout prefix the gold lacks, but a shallower `utils.py` is NOT `src/deep/utils.py`. The old bidirectional rule credited a wrong file as gold, which on multi-file gold inflates recall, precision and rank alike. - _paired_means aggregates each metric over the rows where BOTH arms measured it. Averaging each arm over its own population put two different populations on the two sides of one gate. - An UNMEASURED metric can neither show a regression nor clear one, and is reported as None with its paired_cases count - never as a measured 0.0. ENGINE (engine.py): - `independent` was dead by construction: role_classes[role] is always empty for a role in new_required, so it collapsed to a class-breadth count that fused_rank already scores, and NEW_MANDATORY_INDEPENDENT was a false reason code. - _bounded_region bounded in LINES at an assumed 20 chars/line while the rail judged in TOKENS at 4 chars/token; measured real source is 42.4 (Python) and 37.8 (Go), so the engine built regions its own rail then REJECTed. - stopping_reason was unconditionally overwritten by candidate_rail, so a run that stopped because its roles were covered reported the same string as one that hit a rail. - The FTS5 lexical legs failed open with no execution witness - the same class as the embedder fail-open, which is what made the legacy control arm drift. CORPUS (build_localization_gold_corpus.py): - Gold line numbers are now PRE-IMAGE, taken from the hunk's '-' side. The engine indexes at base_commit, the pre-fix tree, so post-image numbers pointed at lines it can never see. Cases carry gold_line_coordinates so no consumer can join across coordinate systems by accident. - The symbol extractor aborts if any hunk falls back to the header while its own changed lines define something - the exact regression that produced ['t','s','y']. Guards are per-language so a majority language cannot dilute a total corruption of a minority one. - git's C-style OCTAL path quoting is decoded properly; the report accounts for every file (kept 754, unsupported_language 164, doc_path 89, no_post_image 6, test_path 2) instead of hiding skips behind a coverage claim. - _INPUT_KEYS is now ENFORCED as a fail-closed allowlist, so a gold field added to the corpus later is excluded by default instead of leaking until someone remembers it. Gold isolation re-proven five ways over all 294 prepared rows. NEW CORPUS: benchmarks/data/swebench_live_gold_cases.json - 294 cases, 70 repos, 122 multi-file (41%), 3248 pre-image line ranges, 1355 symbols, splits random 237 / held 57. It is NOT dispatched here: evaluate_winner could not judge a monolingual corpus until this commit, and the 294-case geometry needs a sharding decision first. Verified: 979 passed in tests/pretask with exactly the 12 failures that reproduce on pristine c5bd7d1; 176 across the five vnext suites; ruff clean; pyright 0/0; ss_gate.py twice at PASS=11 / SKIP=1 (S8) / FAIL=0 / EXIT 0 (real exit codes, not through a pipe). Not benchmaxxed: no task id, repo name, gold label or corpus-tuned constant is read anywhere. vNext remains shadow-only behind GT_LOC_VNEXT_SHADOW. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the deterministic shadow localization engine and a 20-shard offline old/new comparison workflow. The sealing jobs receive gold-free manifests; gold is joined only after all outputs are sealed. No LLM API calls.