docs: grading fixes — an eval whose answer key corral already wrote - #339
Merged
Merged
Conversation
Follows directly from the churn-vs-drain measurement: 5 of round three's 13 findings were created by round two's own fix, and those five were MORE SEVERE than the eight pre-existing ones. That raises a question with no execution-proven answer today — is a given model any good at FIXING, as opposed to writing? The design's whole argument is that corral already owns the answer key. A claim called REPRODUCED must ship a script that exits 0 only if the defect is demonstrated, and corral runs it. So a fix is graded by exit code, not by opinion: before the fix the script exits 0; after a correct one it must not. 42 claims declared REPRODUCED on the public branch, 36 held — 36 findings with working scripts, past models rank's five-observation floor. Three mechanical facts per fixer seat: CLOSED (the finding's own script no longer exits 0), INTACT (the repo's suite still passes), CLEAN (a cold reviewer, blinded to which model wrote the patch, finds nothing new in the touched files). Plus diff size recorded but not graded, since a fix that deletes the feature satisfies the first two. Prior art named up front: SWE-bench. Two differences carry the argument — its corpus is fixed and public, so contaminated, while these findings are generated fresh from the operator's own repo by an adversary that has never seen it; and it never asks whether the patch INTRODUCED something, which is the fact we just measured at 3 highs from 10 fixes. Also recorded: why the retrospective shortcut does not work. 1,259 of this repo's 1,477 commits carry a model trailer, so findings can be attributed without running anything — and the resulting table should be believed by nobody. Blame attributes to the most recent toucher, so the newest model always looks guiltiest; the models were not doing the same job; the findings are not a random sample because the round was aimed at one model's fixes; numerator and denominator do not span the same thing; and co-authorship is not authorship. Those five failures are the eval's design constraints. What is missing is one thing: seats are read-only by design, so a fixer seat is a new, jailed seat kind that must not be able to edit the script, the test command, or the finding. The first slice is one finding, N seats, three facts printed — because that is where the only real unknown lives. Stance: this is not corral acquiring a builder. The fixer models are the AUDITED party, as the operator's suite is in certify. Recorded as a founder call rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NgkB3eLPBhgfqU1XopR1V
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows straight from the churn-vs-drain measurement in #332: 5 of round three's 13 findings were created by round two's own fix, and those five were more severe than the eight pre-existing ones. That raises a question with no execution-proven answer today — is a given model any good at fixing, as opposed to writing?
The argument is that corral already owns the answer key. A claim called REPRODUCED must ship a script that exits 0 only if the defect is demonstrated, and corral runs it rather than reading it. So a fix is graded by exit code, not opinion: before, the script exits 0; after a correct fix, it must not. The public branch holds 42 such claims of which 36 held — 36 findings with working scripts, comfortably past
models rank's five-observation floor.Three mechanical facts per fixer seat: closed (the finding's own script no longer exits 0), intact (the repo's suite still passes), clean (a cold reviewer, blinded to which model wrote the patch, finds nothing new in the touched files). Diff size is recorded but not graded, because a fix that deletes the feature satisfies the first two.
Prior art named up front: SWE-bench. Two differences carry the whole argument — its corpus is fixed and public, therefore contaminated, while these findings are generated fresh from your own repo by an adversary that has never seen it; and it never asks whether the patch introduced something, which is the fact we just measured at 3 highs from 10 fixes.
The doc also records why the retrospective shortcut does not work, because it is instructive and it is wrong. 1,259 of this repo's 1,477 commits carry a model trailer, so findings can be attributed with no new machinery — and the resulting table should be believed by nobody: blame attributes to the most recent toucher so the newest model always looks guiltiest; the models were not doing the same job; the findings are not a random sample; numerator and denominator do not span the same thing; co-authorship is not authorship. Those five failures are the eval's design constraints.
One thing is genuinely missing. Seats are read-only by design (
--tools Read,Grep,Glob,--sandbox read-only), so a fixer is a new jailed seat kind that must not be able to edit the script, the test command, or the finding.Stance, recorded as a founder call rather than assumed: this is not corral acquiring a builder. The fixer models are the audited party, as the operator's suite is in
certify.Every figure re-verified against the repo before committing.
🤖 Generated with Claude Code
https://claude.ai/code/session_011NgkB3eLPBhgfqU1XopR1V