Close the changelog gap and retire the swe-bench evidence - #210
Merged
Merged
Conversation
Four pull requests merged to main without a CHANGELOG entry. The last merge to touch the file was #198, and the highest issue referenced in [Unreleased] was #195, so #205, #207, #208 and #209 shipped undocumented. release_drift.py did not catch this and could not: it checks that entries exist, not that recent merges are among them, so 102 commits against 28 entries reported OK while the four most recent merges were all missing. Add them: #209 phoenix_mission --backend mixed, the execution half of the hybrid mission (#86). Previously the binary took one --backend for the whole mission, so no proof could observe the hybrid path from outside the process. #208 eval-gate.ps1 enforces the instrument-validity rule MISSION.md has stated since 2026-08-07. #207 the acceptance contract restores every file it names, not only tests/*.py. #205 Check.timeout_secs is enforced, and truncate no longer panics on a non-char-boundary. Separately, retire swe-bench as evidence rather than delete it. The README presented the swe-bench-style result as a live row in the results table, and a limitations bullet described the gate as having "no headroom left" -- written in the present tense about a gate that is no longer used. The benchmark is out of scope for this project and the measurement has not been refreshed since 2026-07-03, so presenting it as current evidence overstates it. Mark the row and the hypothesis-table entry retired, kept for the record, and rewrite the limitations bullet to say the gate is withdrawn and that Tier 3 now runs under the instrument-validity rule from #208. The numbers stay: the eval did run and the result is real. Deleting it would be the dishonest option. Methodology citations are deliberately untouched. docs/autonomous-workflows.md cites the swe-bench discipline -- confirm the check fails, fix, confirm it passes -- which is the origin of Phoenix's failure-first gate, and src/accept.rs says the same in a comment. That lineage is not a stale claim about our results, and removing it would erase where the idea came from. BUILDLOG.md, the CHANGELOG history, and the research report are historical records and are left as written. handoff.md's mention is a live operational note about pytest collection, still accurate. No eval code or eval results were modified. Verified: 49 tests across the doc-pinning suites (README accept claim, release drift, version consistency, OKF claim disclosure, scoreboard, auto-merge gate, ralph example gate) green; release_drift reports 38 [Unreleased] entries, up from 28. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retiring a claim in prose is not durable on its own. The next person editing the results table has nothing telling them the row is historical, and a revert restores a current-tense claim about a withdrawn benchmark with nothing to catch it. Guard it the way test_okf_claim_discloses_its_limit.py guards the OKF cost/quality disclosure. Deliberately does NOT assert the numbers are gone: the eval ran, the result is real, and deleting it would be the dishonest option. It asserts only that the result is never presented as current evidence, and it fails if the row disappears entirely. Table-row scoped on purpose. Prose citing the swe-bench discipline (confirm the check fails, fix, confirm it passes) is where Phoenix's failure-first gate comes from, not a claim about our results, and must not trip this. The detector carries its own unit tests, without which the file would pass trivially if the mentions were simply deleted -- the outcome it exists to tell apart from retirement. Point the acceptance contract at it: verified RED against the pre-retirement docs (3 failed) and GREEN with them (6 passed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit staged README.md and docs/intent-to-outcome.md at their pre-retirement revisions: verifying the new guard goes RED without the change used git checkout origin/main -- <paths>, which writes the index as well as the working tree, and the working-tree restore afterwards did not re-stage them. The guard shipped sitting on top of the docs it exists to reject. Re-commit the retirement so the guard is green against the tree it guards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two documentation defects, both of the "advertised in one place, true nowhere" class this repo keeps finding.
1. Four merges shipped undocumented
The last merge to touch
CHANGELOG.mdwas #198, and the highest issue referenced in[Unreleased]was #195. Everything since shipped with no entry:timeout_secsenforced,truncatepanicscripts/release_drift.pydid not catch this and structurally cannot: it verifies entries exist, not that recent merges are among them. It reportedOK -- 102 commit(s) since 0.5.0, 28 [Unreleased] entr(ies)while the four most recent merges were all absent. Worth noting as a gap in the drift check itself; not fixed here.Entries added for all four.
[Unreleased]now reports 38.2. The README presented a retired benchmark as live evidence
The swe-bench-style result sat as a row in the results table, and a limitations bullet described the gate in the present tense: "has no headroom left" — language about a gate still in use. The benchmark is out of scope for this project and the measurement has not been refreshed since 2026-07-03.
Marked retired in the results table, the hypothesis table in
docs/intent-to-outcome.md, and the limitations bullet, which now says the gate is withdrawn and points at the instrument-validity rule from #208 that governs Tier 3 instead.The numbers stay. The eval ran and the result is real. Deleting it would be the dishonest option; the fix is to stop presenting it as current.
What was deliberately left alone
Methodology citations are not touched.
docs/autonomous-workflows.mdcites the swe-bench discipline — confirm the check fails, fix, confirm it passes — which is the origin of Phoenix's failure-first gate, andsrc/accept.rssays the same in a comment. That lineage is not a stale claim about our results, and erasing it would erase where the idea came from.BUILDLOG.md, the CHANGELOG's own history, andresearch/autonomous-workflows-research.mdare historical records, left as written.handoff.md's mention is a live operational note about pytest collection, still accurate.No eval code or eval results were modified.
The guard
Retiring a claim in prose is not durable on its own: the next person editing the results table has nothing telling them the row is historical, and a revert restores a current-tense claim with nothing to catch it.
tests/test_swebench_evidence_is_retired.pypins it, the same waytest_okf_claim_discloses_its_limit.pypins the OKF cost/quality disclosure. It asserts only that the result is never presented as current evidence — and it also fails if the row disappears entirely, because retirement keeps the record rather than deleting it. Table-row scoped so methodology prose cannot trip it. The detector carries its own unit tests, without which the file would pass trivially if the mentions were simply deleted.Verified RED against the pre-retirement docs (3 failed) and GREEN with them (6 passed), which is what the acceptance contract now points at.
Verification
49 tests across the doc-pinning suites — README accept claim, release drift, version consistency, OKF claim disclosure, scoreboard, scoreboard validity, auto-merge gate, ralph example gate — green.
One thing worth flagging in review: an earlier commit on this branch staged the two doc files at their pre-retirement revisions, because verifying the guard goes RED used
git checkout origin/main -- <paths>, which writes the index as well as the working tree.9b26553restores them. The final diff againstmainis what matters and is correct; the intermediate commit is not.