feat(bench): Cw control + verification metrics, per-scenario, Holm, oracle (#116/#105)#123
Merged
Merged
Conversation
…racle (#116/#105) Phases 5 + 6 of the v2 multi-turn track — the offline analysis layer. Both are coupled (the Cw control only has meaning once the metrics compute the C3-vs-Cw contrast), so they ship together. Phase 5 — Cw warning-only control (#116): - prompts.py: a fifth condition Cw = stale doc + a generic "may be out of date" note with NO corrected code / file / line / value. Isolates whether C3's recovery is Surface's concrete fix (H6: C3 > Cw) or merely the suspicion any drift signal would raise. Deliberately content-free so it can't leak the truth and collapse the contrast. Phase 6 — verification metrics + reporting + oracle (#105): - metrics.py: verification_rate + verified_then_correct (multi-turn rows), the H4 suppression delta (C0−C1 verification) and the H5 C1 mediation (verifiers correct vs non-verifiers misled), a per-scenario breakdown, the C3-Cw / Cw-C1 success pairs, a bootstrap p-value on each success delta, and Holm-Bonferroni across the whole delta family. - oracle.py: post-run tripwires (C2-fresh ≈100% per scenario; cascade C1 must mislead at least sometimes), exit non-zero so a run can be gated. - report.py: a Verification section (the multi-turn headline), a per-scenario success table, Cw labels/gloss, a Holm flag on each delta, and a verification_rate.png hero figure. - tests: 6 offline tests (Cw prompt, verification/mediation, Holm + Cw pairs, per-scenario, oracle pass + trip). Suite: 30 passing. Verified end-to-end via a mock multi run -> report -> oracle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phases 5 + 6 of the v2 multi-turn track — the offline analysis layer. Grouped into one PR because they're coupled: the Cw control only means anything once the metrics compute the C3-vs-Cw contrast. Branches off
main; independent of the open adapter PRs (#121/#122).Phase 5 — Cw warning-only control (#116)
prompts.py: a fifth condition Cw = stale doc + a generic "may be out of date" note with no corrected code, file, line, or value. It isolates whether C3's recovery comes from Surface's concrete fix (H6: C3 > Cw) or merely the suspicion any drift signal would raise. Kept deliberately content-free so it can't leak the truth and collapse the contrast.Phase 6 — verification metrics + reporting + oracle (#105)
metrics.py: the multi-turn headline —verification_rate+verified_then_correct; the H4 suppression delta (C0−C1 verification) and the H5 C1 mediation (verifiers correct vs non-verifiers misled); a per-scenario breakdown; the C3-Cw / Cw-C1 success pairs; a bootstrap p-value per success delta; and Holm-Bonferroni across the whole delta family.oracle.py(new): post-run tripwires — C2-fresh ≈100% per scenario, and cascade C1 must mislead at least sometimes — exiting non-zero so a run can be gated in CI. (This is the guard against the Pilot smoke test (haiku×2): success-rate ceiling, but a real wasted-token signal surface-bench#9 authoring-bug failure mode.)report.py: a Verification section, a per-scenario success table, Cw labels/gloss, a Holm flag on each delta, and averification_rate.pnghero figure.Verification
uv run pytest(inbench/) — 30 passed (6 new), no network.cargo fmt --all --check— clean.mock·--mode multi· 2 scenarios): run emits all 5 conditions;reportrenders the Verification + Per-scenario sections andsummary.jsoncarriesverification/by_scenario/mediation;oraclecorrectly fires (mock can't solve cascades, so C2-fresh=0% and C1-never-misleads both trip — the tripwires working as designed).Scope
Closes the metrics/reporting half of #105 and implements #116. Remaining: scenario authoring (#104) and the pre-registration + staged run (#117/#106).
🤖 Generated with Claude Code