Enhancement: Eval history observability with repo SHA provenance and judge drift markers

[coston]

Summary

Implement eval-history observability for multi-model comparisons so historical trends remain useful while clearly signaling comparability caveats.

Problem

Current comparison outputs are useful per run, but decision-makers need reliable trend visibility over time. With LLM-as-judge, score changes can come from model-under-test changes or judge/eval regime changes, which can make historical interpretation ambiguous.

Proposed enhancement (v1)

Capture and report score, cost, and duration over time with run provenance and event markers.

1) Persist run provenance metadata

Add these fields to persisted run artifacts (JSON and CSV):

• run_at (ISO timestamp)
• repo (owner/repo)
• git_sha (commit evaluated)
• judge_provider
• judge_model
• judge_version (when available)
• judge_prompt_hash
• eval_hash (or per-eval spec hash)
• comparison_fingerprint (stable hash of judge+prompt+eval config)

2) Capture decision metrics

For each model/eval run, persist:

• quality score (existing pass-rate metrics)
• total/normalized cost (if available from provider/runtime)
• total duration / wall-clock time

3) Historical trend reporting

Add/extend report generation to show time-series trends per model/eval with two modes:

• Strict comparable: same comparison_fingerprint
• All runs: includes all historical runs with caveat labeling

4) Regime-change markers

In trend outputs (charts/tables), show vertical event markers when any of these change:

• judge model/version
• judge prompt hash
• eval hash/spec

These markers should behave like release/event bars so jumps are interpretable.

Why

• Preserves all historical data (no loss of context)
• Improves reproducibility with repo+SHA provenance
• Makes drift explicit rather than hidden
• Supports practical model decisions using quality + cost + speed together

Acceptance criteria

• JSON/CSV artifacts include provenance metadata fields listed above
• repo and git_sha are captured for every run
• score, cost, and duration are persisted for each model/eval run
• report supports strict-comparable and all-runs modes
• regime-change markers appear when judge/prompt/eval fingerprints change
• each trend point can be traced back to run timestamp, repo SHA, and judge config

Non-goals (v1)

• statistical normalization across different judge versions
• mandatory human re-judging workflow
• full BI/dashboard stack

References

• docs/eval-comparison.md
• Feedback: preserve historical numbers, record judge/date/SHA, and mark major judge/prompt changes as vertical bars.

[coston]

Operator has opened a pull request.

PR: #5

Status: Awaiting review

[coston]

Implementation Plan

Files to modify:

• src/eval/types.ts — add RunProvenance, costUsd/totalCostUsd fields on TestResult/EvalRun, provenance field on EvalComparison
• src/eval/runner.ts — capture output:cost events from runAgent so runPrompt returns { output, costUsd }
• src/eval/index.ts — thread cost through runEval, build provenance once per eval-file run, attach it to single- and multi-model EvalComparison objects, add --history <path> flag that appends each run to a JSONL history log
• src/eval/export.ts — add provenance + cost_usd columns to CSV, restore costUsd in loadExistingResults
• src/eval/report.ts — add printTrends() for the terminal trend view with regime-change marker rows
• docs/eval-comparison.md — document provenance fields, cost tracking, --history, and the new eval:trend command
• package.json — add eval:trend script; have eval:compare pass --history results/eval-history.jsonl
• src/tests/eval-comparison.test.ts, src/tests/eval.test.ts — update fixtures/assertions for the new CSV columns and runPrompt return shape

Files to create:

• src/eval/provenance.ts — getGitSha, getRepoSlug, getJudgeVersion, hashContent, getJudgePromptHash, getEvalHash, computeComparisonFingerprint, buildProvenance
• src/eval/history.ts — appendHistory/loadHistory (JSONL), buildTrends (strict-comparable vs all-runs modes, regime-change marker detection per group)
• src/eval/trend-index.ts — npm run eval:trend CLI entry point
• src/tests/eval-provenance.test.ts, src/tests/eval-history.test.ts — unit tests for the new modules

Steps:

1. Add provenance types + src/eval/provenance.ts (git SHA/repo via git CLI, judge model/version, sha256 hashes of the judge prompt and the resolved eval spec, and a stable comparisonFingerprint combining them).
2. Capture per-case cost in runPrompt/runEval (Claude only — OpenCode has no cost signal, left undefined) and roll it up into EvalRun.totalCostUsd.
3. Attach provenance to every EvalComparison built in runEvalFile (single- and multi-model paths) and extend JSON/CSV export with the provenance + cost fields.
4. Add src/eval/history.ts to append one JSONL record per model/eval per run (score, cost, duration, provenance) behind an opt-in --history <path> flag, plus buildTrends to group records by eval+model, sort by time, support strict-comparable (matching the latest comparisonFingerprint) vs all-runs modes, and flag regime-change points.
5. Add printTrends (report.ts) and the eval:trend CLI (trend-index.ts) to render the time series with a marker line wherever the judge/prompt/eval fingerprint changes.
6. Update docs/eval-comparison.md and package.json scripts.
7. Add/update unit tests, run lint/typecheck/tests, and confirm coverage.

Note on scope: the separate end-to-end workflow-eval harness (src/eval/workflow.ts, git-worktree based) is left untouched — this issue and docs/eval-comparison.md are about the criterion-judge multi-model comparison system, so provenance/history land there.

[github-actions]

🎉 This issue has been resolved in version 2.14.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀