fix: Enhancement: Eval history observability with repo SHA provenance and judge drift markers - #5
Merged
coston merged 5 commits intoAug 31, 2026
Conversation
6 tasks
…ombie-reap race) Three tests failed on GitHub Actions while passing locally: - Ink checks `is-in-ci` at module load and, in CI, defers all frame writes until unmount (and the CI unmount path overwrites the final frame with a bare newline in debug mode), so tests asserting on frames written to a fake stdout captured nothing. A new shared force-non-ci.ts helper — imported first in every ink-rendering test file — pins CI="false", which is-in-ci short-circuits on even when GITHUB_ACTIONS/CI_* vars are set. - The process-group timeout test asserted ESRCH on the first signal-0 probe, but a just-SIGTERMed grandchild can still be an unreaped zombie (signal 0 succeeds on zombies), so it raced on Linux. The assertion now polls up to 2s for the process to disappear. Repro for the ink failures: `CI=true npm test` fails identically on macOS; green after this change both with and without CI=true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019thBZNhFBYfFP7Dk5cgzMW
An observability feature must never produce records that are confidently wrong. Four ways it could, found by running it, now fixed: - A resumed run (cached --output-csv results) appended a history record stamped with fresh provenance — new runAt/gitSha/fingerprint — for scores produced under the previous run's regime. recordHistory now refuses to append when any case was reused, and says why. - The judge model was asserted, not observed: provenance recorded EXECUTANT_MODEL ?? sonnet while the judge ran with no --model flag at all, i.e. the user's CLI default. judge.ts now pins the model it runs to the exact value provenance records (resolveJudgeModel, shared). Likewise the bare single-model path hard-coded the "claude/sonnet" series label; it now records the model that actually ran. - judgeVersion was captured but excluded from comparisonFingerprint, though the docs and the regime-marker text claim version changes are flagged — a Claude CLI upgrade shifted judging silently. The version is now part of the fingerprint. - One corrupt history line (a half-written append) aborted eval:trend entirely; it is now skipped with a warning naming the file and line. Plus smaller honesty fixes: judgePromptHash now hashes the header-stripped prompt (what the judge actually receives), evalHash is insensitive to YAML var-key order, trend flags missing a value now error instead of silently falling back (a silent --mode strict miss hands the user non-comparable data), an --eval filter that matches nothing says so instead of claiming no history exists, and trend group identity no longer round-trips through the "::" map key. Every fix carries a regression test (9 new). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019thBZNhFBYfFP7Dk5cgzMW
`npm run eval:trend -- --html <path>` now also renders the history log as one self-contained HTML file: a leaderboard card per eval (latest run per model, ranked by score then cost, LOCAL/API badges) with expandable per-model run histories carrying cost, duration, git sha, and the same regime-change markers as the terminal view. Themed with the @coston/design-tokens purple-dark theme the TUI already uses (src/ui/theme.ts) — the tokens are CSS-native oklch() strings, so they are injected verbatim as custom properties at generation time. No build step, no script, no external stylesheet, font, or asset: the file works offline and can be attached to a PR or CI artifact as-is. renderHtmlReport is pure (history entries in, HTML string out); the only I/O is the flag handling in trend-index.ts. HistoryEntry is now exported for consumers of loadHistory's return type. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019thBZNhFBYfFP7Dk5cgzMW
A 10-bin histogram over each model's latest run per eval, in the card header — it shows at a glance whether the eval separates models or everyone clusters at a ceiling. Pure CSS bars (no script, keeping the single-file guarantee); per-bin counts live in title tooltips and every score is also in the table, so the strip summarizes rather than carries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019thBZNhFBYfFP7Dk5cgzMW
|
🎉 This PR is included in version 2.14.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Multi-model eval comparisons currently only show a single run in isolation, so decision-makers can't tell whether a score change over time reflects the model under test getting better/worse, or the judge/eval regime itself shifting underneath them. This adds run provenance (repo, git SHA, judge/prompt/eval fingerprints), persists cost and duration alongside score for every run, and adds a historical trend view with regime-change markers so drift is explicit rather than hidden.
Changes
src/eval/provenance.ts(new) — builds aRunProvenancerecord per run:runAt,repo(from theoriginremote),gitSha,judgeProvider/judgeModel/judgeVersion,judgePromptHash,evalHash, and acomparisonFingerprinthashing all of the above for strict-comparability checks.src/eval/runner.ts—runPromptnow captures the Claudeoutput:costevent and returns cost alongside output, so per-case API cost flows through toTestResult.src/eval/index.ts— threads cost and duration throughrunEval, attachesprovenanceto everyEvalComparison(single- and multi-model paths), and adds an opt-in--history <path>flag that appends one JSONL record per model/eval to a history log.src/eval/export.ts— JSON/CSV output now includesprovenanceandcost_usd/duration_mscolumns;loadExistingResultsrestorescostUsdfrom prior CSVs.src/eval/history.ts(new) —appendHistory/loadHistoryfor the JSONL log, andbuildTrends, which groups records by eval+model, sorts by time, supportsstrict(matchingcomparisonFingerprint) vsallmodes, and flags regime-change points.src/eval/report.ts—printTrends()renders the terminal trend view, printing a regime-change marker line wherever the judge model/version, judge prompt, or eval spec changed since the previous run.src/eval/trend-index.ts(new) —npm run eval:trendCLI entry point (--eval,--mode,--historyflags).src/eval/types.ts— adds `RunProvenanc(output truncated)
Related Issue
Fixes #4
Generated by Operator