Skip to content

Close the changelog gap and retire the swe-bench evidence - #210

Merged
shyamsridhar123 merged 3 commits into
mainfrom
docs/changelog-gap-and-retire-swebench
Aug 16, 2026
Merged

shyamsridhar123 merged 3 commits into
mainfrom
docs/changelog-gap-and-retire-swebench

Conversation

@shyamsridhar123

Copy link
Copy Markdown
Contributor

Two documentation defects, both of the "advertised in one place, true nowhere" class this repo keeps finding.

1. Four merges shipped undocumented

The last merge to touch CHANGELOG.md was #198, and the highest issue referenced in [Unreleased] was #195. Everything since shipped with no entry:

Merge Entry?
#209 — hybrid mission across two backends (#86) missing
#208 — eval-gate instrument validity missing
#207 — contract restores every file it names missing
#205timeout_secs enforced, truncate panic missing

scripts/release_drift.py did not catch this and structurally cannot: it verifies entries exist, not that recent merges are among them. It reported OK -- 102 commit(s) since 0.5.0, 28 [Unreleased] entr(ies) while the four most recent merges were all absent. Worth noting as a gap in the drift check itself; not fixed here.

Entries added for all four. [Unreleased] now reports 38.

2. The README presented a retired benchmark as live evidence

The swe-bench-style result sat as a row in the results table, and a limitations bullet described the gate in the present tense: "has no headroom left" — language about a gate still in use. The benchmark is out of scope for this project and the measurement has not been refreshed since 2026-07-03.

Marked retired in the results table, the hypothesis table in docs/intent-to-outcome.md, and the limitations bullet, which now says the gate is withdrawn and points at the instrument-validity rule from #208 that governs Tier 3 instead.

The numbers stay. The eval ran and the result is real. Deleting it would be the dishonest option; the fix is to stop presenting it as current.

What was deliberately left alone

Methodology citations are not touched. docs/autonomous-workflows.md cites the swe-bench discipline — confirm the check fails, fix, confirm it passes — which is the origin of Phoenix's failure-first gate, and src/accept.rs says the same in a comment. That lineage is not a stale claim about our results, and erasing it would erase where the idea came from.

BUILDLOG.md, the CHANGELOG's own history, and research/autonomous-workflows-research.md are historical records, left as written. handoff.md's mention is a live operational note about pytest collection, still accurate.

No eval code or eval results were modified.

The guard

Retiring a claim in prose is not durable on its own: the next person editing the results table has nothing telling them the row is historical, and a revert restores a current-tense claim with nothing to catch it.

tests/test_swebench_evidence_is_retired.py pins it, the same way test_okf_claim_discloses_its_limit.py pins the OKF cost/quality disclosure. It asserts only that the result is never presented as current evidence — and it also fails if the row disappears entirely, because retirement keeps the record rather than deleting it. Table-row scoped so methodology prose cannot trip it. The detector carries its own unit tests, without which the file would pass trivially if the mentions were simply deleted.

Verified RED against the pre-retirement docs (3 failed) and GREEN with them (6 passed), which is what the acceptance contract now points at.

Verification

49 tests across the doc-pinning suites — README accept claim, release drift, version consistency, OKF claim disclosure, scoreboard, scoreboard validity, auto-merge gate, ralph example gate — green.

One thing worth flagging in review: an earlier commit on this branch staged the two doc files at their pre-retirement revisions, because verifying the guard goes RED used git checkout origin/main -- <paths>, which writes the index as well as the working tree. 9b26553 restores them. The final diff against main is what matters and is correct; the intermediate commit is not.

shyamsridhar and others added 3 commits August 16, 2026 06:23
Four pull requests merged to main without a CHANGELOG entry. The last merge
to touch the file was #198, and the highest issue referenced in [Unreleased]
was #195, so #205, #207, #208 and #209 shipped undocumented.

release_drift.py did not catch this and could not: it checks that entries
exist, not that recent merges are among them, so 102 commits against 28
entries reported OK while the four most recent merges were all missing.

Add them:

  #209  phoenix_mission --backend mixed, the execution half of the hybrid
        mission (#86). Previously the binary took one --backend for the whole
        mission, so no proof could observe the hybrid path from outside the
        process.
  #208  eval-gate.ps1 enforces the instrument-validity rule MISSION.md has
        stated since 2026-08-07.
  #207  the acceptance contract restores every file it names, not only
        tests/*.py.
  #205  Check.timeout_secs is enforced, and truncate no longer panics on a
        non-char-boundary.

Separately, retire swe-bench as evidence rather than delete it.

The README presented the swe-bench-style result as a live row in the results
table, and a limitations bullet described the gate as having "no headroom
left" -- written in the present tense about a gate that is no longer used.
The benchmark is out of scope for this project and the measurement has not
been refreshed since 2026-07-03, so presenting it as current evidence
overstates it.

Mark the row and the hypothesis-table entry retired, kept for the record,
and rewrite the limitations bullet to say the gate is withdrawn and that
Tier 3 now runs under the instrument-validity rule from #208. The numbers
stay: the eval did run and the result is real. Deleting it would be the
dishonest option.

Methodology citations are deliberately untouched. docs/autonomous-workflows.md
cites the swe-bench discipline -- confirm the check fails, fix, confirm it
passes -- which is the origin of Phoenix's failure-first gate, and
src/accept.rs says the same in a comment. That lineage is not a stale claim
about our results, and removing it would erase where the idea came from.
BUILDLOG.md, the CHANGELOG history, and the research report are historical
records and are left as written. handoff.md's mention is a live operational
note about pytest collection, still accurate.

No eval code or eval results were modified.

Verified: 49 tests across the doc-pinning suites (README accept claim,
release drift, version consistency, OKF claim disclosure, scoreboard,
auto-merge gate, ralph example gate) green; release_drift reports 38
[Unreleased] entries, up from 28.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retiring a claim in prose is not durable on its own. The next person editing
the results table has nothing telling them the row is historical, and a
revert restores a current-tense claim about a withdrawn benchmark with
nothing to catch it.

Guard it the way test_okf_claim_discloses_its_limit.py guards the OKF
cost/quality disclosure. Deliberately does NOT assert the numbers are gone:
the eval ran, the result is real, and deleting it would be the dishonest
option. It asserts only that the result is never presented as current
evidence, and it fails if the row disappears entirely.

Table-row scoped on purpose. Prose citing the swe-bench discipline (confirm
the check fails, fix, confirm it passes) is where Phoenix's failure-first
gate comes from, not a claim about our results, and must not trip this.

The detector carries its own unit tests, without which the file would pass
trivially if the mentions were simply deleted -- the outcome it exists to
tell apart from retirement.

Point the acceptance contract at it: verified RED against the pre-retirement
docs (3 failed) and GREEN with them (6 passed).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit staged README.md and docs/intent-to-outcome.md at their
pre-retirement revisions: verifying the new guard goes RED without the change
used git checkout origin/main -- <paths>, which writes the index as well as
the working tree, and the working-tree restore afterwards did not re-stage
them. The guard shipped sitting on top of the docs it exists to reject.

Re-commit the retirement so the guard is green against the tree it guards.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@shyamsridhar123
shyamsridhar123 merged commit 63fc1bc into main Aug 16, 2026
1 check passed
@shyamsridhar123
shyamsridhar123 deleted the docs/changelog-gap-and-retire-swebench branch August 16, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants