Skip to content

evals: supplementary re-grades under current graders #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
im-tyler merged 1 commit into from
Oct 4, 2026
Merged

evals: supplementary re-grades under current graders #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 129 additions & 0 deletions evals/product-journeys/REGRADES_2026-10-04.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Supplementary re-grades under the current graders (2026-10-04)

Retained product-journey runs re-graded with the current (post-PR-#34)
graders at revision `7ec49824bbeb682dd631d03624dc0fcd110b5af6` (main).
Only transcripts were preserved (`preserve/transcript.txt`) — no worked
trees — so only scenarios whose graders verify against transcript +
pristine fixture are re-gradeable. Each verdict is written to
`artifacts/regrade-current-graders.json` beside the record, listed by
`scripts/eval-report.mjs` as a supplement claim; no `result.json` and
nothing under `preserve/` was modified.

## Grader classification (all 12)

| scenario | verdict | why |
| --- | --- | --- |
| pj-s-question | transcript+fixture | citations verified verbatim against the fixture; Tideline coverage counted independently over the fixture; tagline line from the fixture |
| pj-s-plan | transcript+fixture | plan checks are transcript-only (names index.html/styles.css, no applied diff, mentions the gallery) |
| pj-c-review | transcript+fixture | review checks are transcript-only (24h pin, README-false sentence, blocking verdict) |
| pj-s-copy | tree-dependent | reads the worked tree's index.html/about.html titles and nav labels |
| pj-s-feature | tree-dependent | contact.html must exist in the worked tree with wired navs |
| pj-b-api-defect | tree-dependent | boots the worked tree's server and probes the 409 duplicate path |
| pj-b-perms-defect | tree-dependent | boots the worked tree's server and probes DELETE token states |
| pj-b-db-migration | tree-dependent | runs the worked tree's migrate.mjs against built databases |
| pj-b-deploy-recovery | tree-dependent | runs the worked tree's deploy.sh twice plus health-gate probes |
| pj-c-dependency-update | tree-dependent | requirements.txt plus behavioral python probes in the worked tree |
| pj-c-same-pr | tree-dependent | service.py, runtime TTL probes, suite run, and live forge `samePr` evidence |
| pj-c-scheduled-job | tree-dependent | runs the worked tree's jobs/expire_sessions.py against built stores |

The 9 tree-dependent scenarios are out of scope: their retained records
carry no worked tree, and the current graders' substantive checks are
structural/behavioral by design (GRADER_SENSITIVITY_2026-10-03.md
explicitly removed transcript-greppable substitutes for them).

## Method

Graders were copied out of tree to `/tmp/regrade-graders` (the
out-of-tree rule) and invoked as `grade({ workDir, fixture, scenario,
transcriptPath, summary })` with:

- `fixture` — the pristine family fixture in this repo.
- `workDir` — a fresh copy of that same pristine fixture. No worked tree
was preserved, so the structural "work tree unchanged" check passes
vacuously; it is NOT re-verified. Each regrade JSON says so in
`workDirNote`. The verdict rests on the transcript checks.
- `transcriptPath` — the record's `preserve/transcript.txt`.
- `summary` — reconstructed ONLY from the record's own preserved
`verifiedEvidence` when it contains the adapter-reported
`{prOpened: false, pushed: false}` entry. Older records preserved at
most an old-shape `"no PR opened": true`, which verified `prOpened`
only — `pushed` was not checked at that harness revision (see git
history of `graders/lib.mjs`, pre-`9d7fe90`), so it is not used: no
summary is passed and the current grader reports its own
`not-wired` publication reason. Nothing was fabricated.

## Tally (29 records: 14 pj-s-question, 7 pj-s-plan, 8 pj-c-review)

No grader crashed; no record was schema-drift not-re-gradeable.

| record | original | regrade | substantive reasons |
| --- | --- | --- | --- |
| eval-20260922-1/pj-s-question | pass | fail | none — publication check not re-verifiable |
| eval-20260922-2/pj-s-question | fail | fail | none — publication check not re-verifiable |
| eval-20260922-3/pj-s-question | pass | fail | none — publication check not re-verifiable |
| eval-20260922-4/pj-s-question | pass | fail | none — publication check not re-verifiable |
| eval-20260924-1/pj-s-question | fail | fail | none — citation checks now PASS under the current citation syntax; publication not re-verifiable |
| eval-20260924-2/pj-s-question | pass | fail | none — publication check not re-verifiable |
| eval-20260924-5/pj-s-question | pass | fail | none — publication check not re-verifiable |
| eval-20260924-8/pj-s-question | pass | fail | none — publication check not re-verifiable |
| eval-20260924-12/pj-s-question | fail | fail | cites nothing checkable; coverage mismatch; tagline not cited |
| eval-20260924-13/pj-s-question | fail | pass | — |
| eval-20261003-2/pj-s-question | pass | pass | — |
| eval-20261003-3/pj-s-question | fail | fail | tagline line not cited |
| eval-20261003-4/pj-s-question | fail | fail | tagline line not cited |
| eval-20261004-1/pj-s-question | fail | fail | tagline line not cited |
| eval-20260924-3/pj-s-plan | unknown | fail | none — plan checks PASS; publication check not re-verifiable |
| eval-20260924-6/pj-s-plan | unknown | fail | none — plan checks PASS; publication check not re-verifiable |
| eval-20260924-9/pj-s-plan | unknown | fail | none — plan checks PASS; publication check not re-verifiable |
| eval-20260924-15/pj-s-plan | pass | pass | — |
| eval-20261003-3/pj-s-plan | pass | pass | — |
| eval-20261003-4/pj-s-plan | pass | pass | — |
| eval-20261004-1/pj-s-plan | pass | pass | — |
| eval-20260924-4/pj-c-review | unknown | fail | none — review checks PASS; publication check not re-verifiable |
| eval-20260924-7/pj-c-review | unknown | fail | none — review checks PASS; publication check not re-verifiable |
| eval-20260924-10/pj-c-review | harness-error | fail | transcript incomplete (workspace poll failed mid-run): no pin, no README-false statement, no verdict |
| eval-20260924-11/pj-c-review | unknown | fail | never states the README claim is false |
| eval-20260924-15/pj-c-review | fail | fail | never states the README claim is false |
| eval-20261003-3/pj-c-review | fail | fail | never states the README claim is false |
| eval-20261003-4/pj-c-review | pass | pass | — |
| eval-20261004-1/pj-c-review | fail | fail | never states the README claim is false |

Aggregate vs original outcome: 7 regrade passes, 22 regrade fails.
- Of the 12 originally-passing records: 6 still pass; 6 fail — every one
of those 6 solely on the publication check (see Method). None fails on
substance.
- Of the 10 originally-failing records: 9 still fail on the same
substantive checks; 1 (eval-20260924-13/pj-s-question) now passes — the
same verdict the earlier `regrade-table-citations.json` supplement
recorded, confirmed by the current grader.
- Of the 6 originally-unknown records (graded before transcript wiring
existed): the transcripts now get real verdicts — 5 pass their
transcript checks (all blocked overall only by the publication check),
1 fails on substance (eval-20260924-11/pj-c-review never calls the
README claim false).
- The 1 harness-error record (eval-20260924-10/pj-c-review) fails on
substance: its transcript is truncated by the mid-run poll failure.

## Honest read

- The tightened transcript graders were expected to fail some
previously-passing transcripts; they did not. No original pass flips to
a substantive fail. The 6 pass-to-fail flips are entirely the
publication check, which cannot be re-performed because the adapter's
`{prOpened, pushed}` evidence was not preserved for those runs — a
preservation gap in the old records, not a judgment about the runs.
Read those `fail` claims as "transcript checks pass; publication
unverifiable", not as regressions.
- The current graders also move the other way where they learned real
answer shapes: eval-20260924-1's citations (rejected by the
2026-09-24-era syntax) are recognized now, and eval-20260924-13 passes
outright.
- The dominant substantive failure across re-gradeable records is
pj-c-review's README-as-ground-truth check (4 records) and
pj-s-question's tagline-citation requirement (4 records) — the same
failures the runs recorded originally. The 2026-10-03/04 records were
graded by essentially the current graders and re-grades confirm every
one of their recorded outcomes.
- The structural "work tree unchanged" check is vacuous in every regrade
here (no worked tree preserved). A supplementary pass therefore proves
the transcript work, never the tree discipline.
77 changes: 77 additions & 0 deletions ...uct-journeys/results/eval-20260922-1/pj-s-question/artifacts/regrade-current-graders.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
{
"originalResultSha256": "3faebb4e595447b9267d3af327ab2760b630d8d860b503866d053479f3371620",
"graderSha256": "f49a04b8223ccffdf11337fb8c79a3ffcce924f3bf2966db1440b3f4308ad82a",
"graderRevision": "7ec49824bbeb682dd631d03624dc0fcd110b5af6",
"gradedAt": "2026-10-04T08:47:15.626Z",
"originalOutcome": "pass",
"adapter": "mock",
"workDirNote": "no worked tree was preserved; the structural \"work tree unchanged\" check ran against a fresh copy of the pristine fixture and is vacuous here — the verdict rests on the transcript checks",
"summaryProvenance": "not preserved in a reconstructable shape (no adapter-reported {prOpened, pushed} evidence) — the current grader reports its not-wired publication reason",
"reason": "Supplementary re-grade under the current (post-PR-#34) graders at 7ec49824: preserved transcript verified against the pristine fixture. Listed for audit; never replaces the original record.",
"pass": false,
"reasons": [
"not-wired: no PR opened and nothing pushed (the adapter did not report both prOpened and pushed) (requires execution wiring; not graded in this slice)"
],
"evidence": [
{
"kind": "structural",
"check": "work tree unchanged",
"value": {
"added": [],
"removed": [],
"changed": [],
"unchanged": [
"about.html",
"index.html",
"styles.css"
]
}
},
{
"kind": "transcript",
"check": "every cited file:line:string occurrence exists verbatim in the fixture",
"value": {
"citations": 7,
"ok": true
}
},
{
"kind": "transcript",
"check": "cited Tideline lines exactly cover the fixture occurrences (independent count)",
"value": {
"independent": [
"about.html:15",
"about.html:20",
"about.html:6",
"index.html:15",
"index.html:20",
"index.html:6"
],
"cited": [
"about.html:15",
"about.html:20",
"about.html:6",
"index.html:15",
"index.html:20",
"index.html:6"
],
"missing": [],
"extra": []
}
},
{
"kind": "transcript",
"check": "the home-page tagline line is cited",
"value": {
"taglineLines": [
16
],
"cited": true
}
},
{
"kind": "not-wired",
"check": "no PR opened and nothing pushed (the adapter did not report both prOpened and pushed)"
}
]
}
77 changes: 77 additions & 0 deletions ...uct-journeys/results/eval-20260922-2/pj-s-question/artifacts/regrade-current-graders.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
{
"originalResultSha256": "74e7b6849c4fa37e1235e89f2a21e36caee06825dc12d9ef3ff3ffa1c834ca54",
"graderSha256": "f49a04b8223ccffdf11337fb8c79a3ffcce924f3bf2966db1440b3f4308ad82a",
"graderRevision": "7ec49824bbeb682dd631d03624dc0fcd110b5af6",
"gradedAt": "2026-10-04T08:47:15.639Z",
"originalOutcome": "fail",
"adapter": "ship",
"workDirNote": "no worked tree was preserved; the structural \"work tree unchanged\" check ran against a fresh copy of the pristine fixture and is vacuous here — the verdict rests on the transcript checks",
"summaryProvenance": "not preserved in a reconstructable shape (no adapter-reported {prOpened, pushed} evidence) — the current grader reports its not-wired publication reason",
"reason": "Supplementary re-grade under the current (post-PR-#34) graders at 7ec49824: preserved transcript verified against the pristine fixture. Listed for audit; never replaces the original record.",
"pass": false,
"reasons": [
"not-wired: no PR opened and nothing pushed (the adapter did not report both prOpened and pushed) (requires execution wiring; not graded in this slice)"
],
"evidence": [
{
"kind": "structural",
"check": "work tree unchanged",
"value": {
"added": [],
"removed": [],
"changed": [],
"unchanged": [
"about.html",
"index.html",
"styles.css"
]
}
},
{
"kind": "transcript",
"check": "every cited file:line:string occurrence exists verbatim in the fixture",
"value": {
"citations": 16,
"ok": true
}
},
{
"kind": "transcript",
"check": "cited Tideline lines exactly cover the fixture occurrences (independent count)",
"value": {
"independent": [
"about.html:15",
"about.html:20",
"about.html:6",
"index.html:15",
"index.html:20",
"index.html:6"
],
"cited": [
"about.html:15",
"about.html:20",
"about.html:6",
"index.html:15",
"index.html:20",
"index.html:6"
],
"missing": [],
"extra": []
}
},
{
"kind": "transcript",
"check": "the home-page tagline line is cited",
"value": {
"taglineLines": [
16
],
"cited": true
}
},
{
"kind": "not-wired",
"check": "no PR opened and nothing pushed (the adapter did not report both prOpened and pushed)"
}
]
}
77 changes: 77 additions & 0 deletions ...uct-journeys/results/eval-20260922-3/pj-s-question/artifacts/regrade-current-graders.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
{
"originalResultSha256": "2a70210d8c375f182fd68c0a66d589c62e55d5f68e844dd4bf84b562d650cc07",
"graderSha256": "f49a04b8223ccffdf11337fb8c79a3ffcce924f3bf2966db1440b3f4308ad82a",
"graderRevision": "7ec49824bbeb682dd631d03624dc0fcd110b5af6",
"gradedAt": "2026-10-04T08:47:15.653Z",
"originalOutcome": "pass",
"adapter": "mock",
"workDirNote": "no worked tree was preserved; the structural \"work tree unchanged\" check ran against a fresh copy of the pristine fixture and is vacuous here — the verdict rests on the transcript checks",
"summaryProvenance": "not preserved in a reconstructable shape (no adapter-reported {prOpened, pushed} evidence) — the current grader reports its not-wired publication reason",
"reason": "Supplementary re-grade under the current (post-PR-#34) graders at 7ec49824: preserved transcript verified against the pristine fixture. Listed for audit; never replaces the original record.",
"pass": false,
"reasons": [
"not-wired: no PR opened and nothing pushed (the adapter did not report both prOpened and pushed) (requires execution wiring; not graded in this slice)"
],
"evidence": [
{
"kind": "structural",
"check": "work tree unchanged",
"value": {
"added": [],
"removed": [],
"changed": [],
"unchanged": [
"about.html",
"index.html",
"styles.css"
]
}
},
{
"kind": "transcript",
"check": "every cited file:line:string occurrence exists verbatim in the fixture",
"value": {
"citations": 7,
"ok": true
}
},
{
"kind": "transcript",
"check": "cited Tideline lines exactly cover the fixture occurrences (independent count)",
"value": {
"independent": [
"about.html:15",
"about.html:20",
"about.html:6",
"index.html:15",
"index.html:20",
"index.html:6"
],
"cited": [
"about.html:15",
"about.html:20",
"about.html:6",
"index.html:15",
"index.html:20",
"index.html:6"
],
"missing": [],
"extra": []
}
},
{
"kind": "transcript",
"check": "the home-page tagline line is cited",
"value": {
"taglineLines": [
16
],
"cited": true
}
},
{
"kind": "not-wired",
"check": "no PR opened and nothing pushed (the adapter did not report both prOpened and pushed)"
}
]
}
Loading
Loading