tool-review-injection: certify the reviewer by injected-defect catch rate - #285
Merged
Conversation
…rate Adds the review-mode half of the per-mode eval harness (project-translation#11), built to answer project-translation#28: what does the auto-merge gate actually catch? Field observation cannot answer it — ~1.2 eligible PRs/week means a <=1% regression bound needs ~4.6 years — and the estate holds four engine-era human-labelled commits, so fitting the gate to shadow data is the machine grading the machine. Injection manufactures the positive class instead. The object under test is whatever QuantEcon/action-translation@v0 resolves to, driven through the harness's own review-translations.yml. This tool builds nothing, patches nothing and pins nothing: fixtures are content in pull requests. Two structural facts the design turns on, both read off the deployed engine: scope is computed from the SOURCE diff, so every injection needs a paired source-side edit or it measures scope suppression rather than detection; and reviewPR filters to .md before any model call, so _toc.yml never reaches the reviewer and the ToC half of de-localisation scores zero by construction. Contents: build-sites.py (five injection sites derived from harness base state by exact anchored substitution, failing loudly on drift), run.mjs (plan / sources / targets / fire / capture / replicate / runs / status), score.mjs (catch rate by tier and class with Wilson intervals, severity-margin consistency, false positives, integrity), the two workflow scripts that author and blindly adjudicate the fixture set, and the pre-registered analysis rules — written before any fixture ran. First run, 2026-08-20: 140 reviews of 45 fixtures at @v0/v0.26.0, zh-cn. Report and raw data live in QuantEcon/project-translation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new tool-review-injection/ harness to measure the deployed review-mode gate (QuantEcon/action-translation@v0) by running known injected defects through the real review-translations.yml, capturing verdict comments, and scoring catch-rate / margin stability / false positives with pre-registered rules.
Changes:
- Introduces an end-to-end runner (
run.mjs+lib.mjs+drive.sh) to plan fixtures, open source/target PRs, trigger review runs, and capture verdict blocks idempotently. - Adds scoring and reporting (
score.mjs) plus adjudication preparation/workflow scripts to attribute findings to the injected defect and compute catch-rate metrics. - Adds fixture authoring + base-state tooling and data (
build-sites.py, workflow scripts, base fixture files, fixture addenda) and ignores derived artefacts.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tool-review-injection/score.mjs | Aggregates captured verdicts into per-tier/class metrics and emits report.md + scored.json. |
| tool-review-injection/run.mjs | Orchestrates planning, PR creation, firing review runs, capture, replication, and run-status inspection. |
| tool-review-injection/README.md | Documents method, surfaces, and pre-registered analysis rules for the harness. |
| tool-review-injection/merge-fixtures.mjs | Merges crafting-workflow results into fixtures/fixtures.json with normalization/repairs. |
| tool-review-injection/lib.mjs | Shared GitHub API helpers, PR-body shaping, verdict parsing via engine parser, and gate re-implementation for scoring. |
| tool-review-injection/fixtures/toc-addendum.json | Adds a ToC-caption “extra file” fixture variant to test .md-filter visibility limits. |
| tool-review-injection/fixtures/rejected.json | Records rejected crafted variants (audit trail for rule “nothing silently dropped”). |
| tool-review-injection/fixtures/base/zh.lecture.md | Base zh-cn lecture fixture used to derive sites/variants. |
| tool-review-injection/fixtures/base/zh.lecture-minimal.md | Base zh-cn minimal lecture fixture used to derive sites/variants. |
| tool-review-injection/fixtures/base/zh._toc.yml | Base zh-cn _toc.yml fixture used by ToC variants. |
| tool-review-injection/fixtures/base/src.lecture.md | Base English lecture fixture used to derive source-side site edits. |
| tool-review-injection/fixtures/base/src.lecture-minimal.md | Base English minimal lecture fixture used to derive source-side site edits. |
| tool-review-injection/fixtures/base/src._toc.yml | Base English _toc.yml fixture. |
| tool-review-injection/extract-workflow-result.mjs | Extracts a workflow return value from a journal transcript (with reconstruction fallback). |
| tool-review-injection/drive.sh | Bash driver to run the full harness loop in waves (fire/replicate/capture/runs). |
| tool-review-injection/craft-fixtures.workflow.mjs | Workflow script to craft and adversarially verify fixture variants per defect class. |
| tool-review-injection/build-sites.py | Builds the “site” substrate files by anchored substitution from base fixtures. |
| tool-review-injection/adjudicate.workflow.mjs | Workflow script for blind attribution adjudication (finding indices attributable to injection). |
| tool-review-injection/adjudicate-prep.mjs | Prepares adjudication inputs from captured verdicts while hiding outcomes. |
| tool-review-injection/.gitignore | Ignores derived artefacts (runs/, fixtures/sites/, fixtures/variants/). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ilot review) Three defects in the harness's own "nothing is silently dropped" guarantee — pre-registered rule 8, which is the rule the certification's credibility rests on. All three were latent in the 2026-08-20 run and none change its figures, verified by re-scoring the archived data. - Dedupe every row shape, not only those carrying a verdict. The inline comment claimed failed rows were keyed by fixture+replicate; they were not, so the concurrent-capture race that already produced one duplicate verdict row could equally duplicate a reviewFailed row with nothing to catch it. reviewIdentity() now keys on reviewedHeadSha, headSha or commentUpdatedAt as available. - Record the full fixture identity on the unparseable and failed-run rows. score.mjs partitions on kind, so a row without it was excluded from every aggregate and vanished from the denominator — the one outcome rule 8 forbids, in the code path written to honour it. fixtureFields() is now shared by all three append sites so they cannot drift apart again. - Partition on the fixture rather than on whatever the row recorded, so an older or partial row is recovered instead of dropped, and report how many rows needed that recovery rather than repairing them quietly. Re-scored against runs/m0: m0-core 50/69 (72%), deterministic 20/30, regression 8/15, 140 reviews after dedupe, 0 rows needing recovery — identical to the published report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the review-mode half of the per-mode eval harness (project-translation#11), built to answer project-translation#28: what does the auto-merge gate actually catch?
Why this exists
Field observation cannot answer it. At ~1.2 auto-merge-eligible PRs/week estate-wide, a ≤1% regression bound needs ~300 consecutive clean auto-merges — about 4.6 years. And the estate holds four engine-era human-labelled commits, so fitting the gate to shadow data is the machine grading the machine. Injection manufactures the positive class instead: ground truth is constructed, and a review costs ~$0.49 / ~40s.
What it does not do
It builds nothing, patches nothing and pins nothing. The object under test is whatever
QuantEcon/action-translation@v0resolves to, exercised through the harness's ownreview-translations.yml; the fixtures are content in pull requests. Every verdict in the first run carriesengineVersion: 0.26.0/engineRef: v0, which is the production discriminator.Two structural facts the design turns on
Both were read off the deployed engine rather than assumed, and both change what a fixture can mean.
Scope is computed from the SOURCE diff. The reviewer's prompt ends with "findings MUST relate ONLY to the sections that were changed in this PR", and
identifyChangedSectionsderives that list from the source PR's before/after — target-only changes never add a section. So every injection needs a paired source-side edit covering its site, or it measures scope suppression rather than detection. That is why the harness opens a source PR per site.reviewPRfilters to.mdbefore any model call._toc.ymltherefore never reaches the reviewer, and the ToC half of sync-path de-localisation scores zero by construction — not by model failure. A fixture in this set demonstrates it: byte-identical clean translation plus a_toc.ymlwith English captions reachedauto-mergeat 9/9/9/9 in 2 of 3 replicates, with zero findings mentioning the table of contents.Contents
build-sites.pyrun.mjsplan/sources/targets/fire/capture/replicate/runs/status, all idempotent and keyed onreviewedHeadShascore.mjs*.workflow.mjsREADME.mdDerived artefacts (
fixtures/sites/,fixtures/variants/,runs/) are gitignored: a stale copy that no longer matches base state is exactly the silent-drift class the tool exists to measure. Run data is archived deliberately in project-translation instead, because review mode overwrites its comment in place and the bodies survive nowhere else.Not covered by
lintorformat:check, which scope tosrc/**and top-level*.mjs— same astool-test-action-on-github/.First run
140 reviews of 45 fixtures, zh-cn, 2026-08-20. M0-core catch rate 72% (Wilson [61%, 82%]); 24 of the 36 non-catches detected the defect and filed it below the gate; both negative controls were false-flagged 6/6; and the routing decision flipped on 3 of 5 byte-identical clean-control re-reviews. Full analysis: reports/2026-08-20-m0-reviewer-certification.md; raw data: research/2026-08-20-m0-injection-run/.
That run also surfaced #283 and #284.
Reviewing this
The load-bearing parts are
lib.mjs(isGatingFinding/surfaces— the gate re-implemented for scoring, deliberately duplicated rather than imported so a silent change upstream shows up as a disagreement) and the pre-registration inREADME.md. One correction made during the build is recorded in the README rather than hidden: the first implementation required a gating category at every severity, butblocker/majorgate in any category, which understated the reviewer.🤖 Generated with Claude Code