fix: Enhancement: Eval history observability with repo SHA provenance and judge drift markers

ARCHITECTURE.md

@@ -159,19 +159,27 @@ Large text passed to Claude lives in `src/prompts/*.txt`. They use `{{PLACEHOLDE
 
 The eval system tests and iteratively refines the prompt templates in `src/prompts/`. It is not user-facing — run via `npm run eval` during development.
 
-**`src/eval/index.ts`** — CLI entry point. Parses `--refine`, `--max-iter`, `--models`, `--cases`, `--output-json`, and `--output-csv` flags. Accepts one or more eval file paths as positional arguments. `--cases` accepts comma-separated case IDs or 1-based index ranges (e.g. `simple,1-3`) to run a subset without editing YAML. Single-model mode: loads existing CSV results for resume (skips already-scored cases), runs remaining cases, optional refine loop. Multi-model mode (2+ models via `--models`): runs each model independently, builds an `EvalComparison`, prints a side-by-side table. When multiple files are passed, output paths are auto-suffixed per eval name.
+**`src/eval/index.ts`** — CLI entry point. Parses `--refine`, `--max-iter`, `--models`, `--cases`, `--output-json`, `--output-csv`, and `--history` flags. Accepts one or more eval file paths as positional arguments. `--cases` accepts comma-separated case IDs or 1-based index ranges (e.g. `simple,1-3`) to run a subset without editing YAML. Single-model mode: loads existing CSV results for resume (skips already-scored cases), runs remaining cases, optional refine loop. Multi-model mode (2+ models via `--models`): runs each model independently, builds an `EvalComparison`, prints a side-by-side table. When multiple files are passed, output paths are auto-suffixed per eval name. Every `EvalComparison` carries a `provenance` record (see `src/eval/provenance.ts`); `--history <path>` appends it (plus score/cost/duration) to a JSONL log for `npm run eval:trend`.
 
 **`src/eval/load.ts`** — Parses `evals/*.eval.yaml` via Zod. Resolves fixture paths (values in `vars` that end in `.md` / `.txt` are read and substituted with file contents).
 
-**`src/eval/runner.ts`** — `runPrompt(templatePath, vars, model?)`: substitutes `{{PLACEHOLDER}}` vars, runs the prompt through the specified model via `runAgent`, and returns the raw text output. Claude receives `METHODOLOGY` as `appendSystemPrompt`; OpenCode does not (flag not supported).
+**`src/eval/runner.ts`** — `runPrompt(templatePath, vars, model?)`: substitutes `{{PLACEHOLDER}}` vars, runs the prompt through the specified model via `runAgent`, and returns `{ output, costUsd? }` — the raw text output plus API cost when the provider reports one (Claude only). Claude receives `METHODOLOGY` as `appendSystemPrompt`; OpenCode does not (flag not supported).
 
 **`src/eval/judge.ts`** — `judgeOutput()`: takes a single output string and a criterion string, always uses Claude for judgment (the authoritative judge), and returns `{ pass: boolean, reason: string }`.
 
 **`src/eval/refine.ts`** — `refinePrompt()`: given the current template and a list of failures, calls Claude with the prompt-refiner prompt and returns a rewritten template.
 
-**`src/eval/report.ts`** — Terminal output: `printRun()` for single-model pass/fail table; `printComparison()` for multi-model side-by-side comparison table.
+**`src/eval/provenance.ts`** — `buildProvenance(evalFile)`: captures `RunProvenance` for a comparison run — `git rev-parse HEAD` / origin remote for repo+SHA, `claude --version` for judge version, and sha256 hashes (truncated to 12 hex chars) of the judge prompt template and the resolved eval spec. `comparisonFingerprint` combines judge provider+model+prompt hash+eval hash into the key that decides whether two runs are strictly comparable.
 
-**`src/eval/export.ts`** — `toJson(comparison)` and `toCsv(comparison)`: serialize `EvalComparison` for benchmark analysis. CSV is denormalized (one row per criterion judgment per model) with columns `eval_name, template_path, case_id, criterion, model_label, provider, model, pass, reason, duration_ms`.
+**`src/eval/history.ts`** — `appendHistory`/`loadHistory`: persist one JSONL record per model/eval per run (score, cost, duration, provenance) to a history log. `buildTrends(entries, mode)`: groups records by eval+model into time-ordered series; `"strict"` keeps only runs matching the group's latest `comparisonFingerprint`, `"all"` keeps every run and flags the points where the fingerprint changed (`regimeChange`).
+
+**`src/eval/trend-index.ts`** — `npm run eval:trend` CLI. Reads a history JSONL file, filters by `--eval`, builds trends in `--mode strict|all`, renders them via `printTrends`, and with `--html <path>` also writes the report as a single self-contained HTML file (`html-report.ts`).
+
+**`src/eval/html-report.ts`** — `renderHtmlReport(entries, mode)`: pure entries-in/HTML-out renderer for `--html`. One leaderboard card per eval (latest run per model, ranked) with expandable per-model run histories and regime-change markers, themed inline with the `@coston/design-tokens` purple-dark theme the TUI uses — no build step, script, or external asset.
+
+**`src/eval/report.ts`** — Terminal output: `printRun()` for single-model pass/fail table; `printComparison()` for multi-model side-by-side comparison table; `printTrends()` for the `eval:trend` time-series view, with a marker line at each regime-change point.
+
+**`src/eval/export.ts`** — `toJson(comparison)` and `toCsv(comparison)`: serialize `EvalComparison` for benchmark analysis. CSV is denormalized (one row per criterion judgment per model) with columns `eval_name, template_path, case_id, criterion, model_label, provider, model, pass, reason, duration_ms, cost_usd, run_at, repo, git_sha, judge_provider, judge_model, judge_version, judge_prompt_hash, eval_hash, comparison_fingerprint`.
 
 **`src/eval/prompts/`** — Eval-specific prompts (`criterion-judge.txt`, `prompt-refiner.txt`). Same `{{PLACEHOLDER}}` convention as `src/prompts/`.
 

BACKLOG.md

@@ -28,6 +28,18 @@ Known improvements deferred from code reviews and audits.
 
 - **Vars-aware caching in `resolveWorkflow`** — two `workflow:` steps referencing the same file are currently fetched and parsed independently. A cache would need to key on the resolved path/URL *and* the step's `vars:` override (serialized), since two steps referencing the same file with different overrides must not share a resolved `Workflow` — added complexity that isn't justified without evidence repeated-reference workflows are common.
 
+## Implemented (status bar, 2026-08)
+
+- ✅ **Status bar wraps to a second line under a long repo name or branch** — `StatusBar` (`src/ui/StatusBar.tsx`) rendered the repo name and branch at their full length with no regard for terminal width; a sufficiently long value (a long feature-branch name, or a checkout directory not named after the repo) wrapped the row onto a second line, breaking the fixed single-line layout the rest of the footer assumes. `fitRepoLabel` (`src/lib/statusline.ts`) now shrinks name and/or branch with an ellipsis to fit the terminal width minus the gauge segment's reserved width, handing a short name's unused budget to the branch. `App.tsx` passes `stdout?.columns ?? 80` down as the new required `columns` prop. Caught via `src/tests/statusline-ui.test.ts`, whose "names the repo and branch" test also hardcoded the literal string "executant" as the expected repo name (true only when the checkout directory happens to be named after the repo) — updated to assert the shape of the segment rather than its exact text, so the suite doesn't assume a specific checkout location.
+
+## Implemented (subprocess lifecycle, 2026-08)
+
+- ✅ **`timeout_seconds` on a script step actually kills the command on dash-family shells** — `runCommand` (`src/tasks/command.ts`) spawns `sh -c "<command>"` and, on timeout, called `proc.kill()` against that `sh` process alone. On shells that fork a real child for the command rather than exec-replacing themselves (verified with `dash`, Debian/Ubuntu's default `/bin/sh`, for anything beyond a single tail-callable command — e.g. `sleep 60`), killing only the `sh` PID left the actual command running as an orphan, still holding stdout/stderr open. The step's reader loop was waiting on EOF from that pipe, so it never saw one and hung indefinitely instead of throwing `TimeoutError` — silently defeating the feature on any Linux box where `/bin/sh` is dash (i.e. most). Fixed by spawning `detached: true` (making `proc.pid` the leader of its own process group) and killing the whole group (`process.kill(-proc.pid!, "SIGTERM")`) on timeout — `startTimeout` (`src/tasks/stream.ts`) now takes an optional `onTimeout` override for this, defaulting to the original `proc.kill()` for `claude.ts`/`opencode.ts`, which spawn the CLI binary directly and don't have this problem. Detaching moves the child out of executant's own process group, so `command.ts` also gained explicit `SIGINT`/`SIGTERM`/`SIGHUP` cleanup handlers (mirroring the pattern already used in `claude.ts`/`opencode.ts` for `SIGTERM`/`SIGHUP`) so a terminal Ctrl+C during a running script step still reaches the whole process tree instead of just executant itself. Caught via `src/tests/command.test.ts`'s existing timeout test, which used to hang for the full 60s test-runner timeout on this sandbox rather than failing fast — a new regression test spawns a real detached grandchild and asserts it's gone (not orphaned) after the timeout fires.
+
+## Implemented (eval comparison observability, 2026-08)
+
+- ✅ **Run provenance + judge drift markers for eval comparisons** — historical eval-comparison trends were previously opaque: a score change on the same eval/model could come from the model under test, or silently from a judge/prompt/eval regime change, with no way to tell which. Every `EvalComparison` (`src/eval/index.ts`) now carries a `provenance` record (`src/eval/provenance.ts`): `runAt`, `repo`/`gitSha` (from `git`), `judgeProvider`/`judgeModel`/`judgeVersion` (`claude --version`, when readable), and sha256 hashes of the judge prompt template and the resolved eval spec, combined into a `comparisonFingerprint` — the strict-comparability key. Per-case API cost (`TestResult.costUsd`, Claude only — OpenCode/local models don't report one) is captured alongside score and duration and rolled up into `EvalRun.totalCostUsd`. JSON/CSV output (`src/eval/export.ts`) include all of it; CSV repeats the values per row the same way `duration_ms` already did, keeping the pivot-table shape. `--history <path>` (new flag, wired into `eval:compare`) appends one JSONL record per model/eval per run (`src/eval/history.ts`); `npm run eval:trend` (`src/eval/trend-index.ts`) reads that log and renders per eval+model time series in two modes — `all` (every run, with a marker line wherever the judge/prompt/eval fingerprint changed since the previous run) and `strict` (only runs matching the latest fingerprint). The separate git-worktree-based workflow-eval harness (`src/eval/workflow.ts`) was left untouched — this only covers the criterion-judge comparison system `docs/eval-comparison.md` documents.
+
 ## Implemented (TUI, 2026-08)
 
 - ✅ **Context gauge reads per-call usage, not the step's cumulative total** — the gauge shipped reading `output:usage`, which is parsed from the Claude CLI's final `result` message. Those counts are summed across every API call the step made: each turn re-reads the whole cached prefix, so `cache_read_input_tokens` grows by roughly the context size *per turn*. Dividing that by a 200k window is a category error — it compares throughput to capacity. On a long agentic step it read **3781.1k/200k** (~25 turns' worth), with the bar pinned at 100% because `buildGauge` caps `pct`. Verified against a real three-turn `claude -p` transcript: the calls occupied 37,670 → 37,844 → 38,166 tokens while the result message reported 113,680 — exactly the sum of the three. The fix adds a separate `output:context` event emitted from the stream's `assistant` messages (deduplicated by message id, since the CLI emits one per content block sharing that turn's usage) carrying the session's `input + cacheCreation + cacheRead` as of its latest turn; the gauge replaces on each one, and `output:usage` goes back to being purely the run report's throughput total. The gauge is also cleared on a prompt step's `step:start`, so each step's session starts from an empty window rather than inheriting the previous step's fill — one session per step, one gauge per session, never carried across or summed. A side benefit is that the gauge now moves *during* a step rather than only at its end.

README.md

@@ -580,7 +580,7 @@ npm run eval -- \
 npm run eval -- evals/plan-decompose.eval.yaml evals/judge-evaluation.eval.yaml
 ```
 
-The `--output-csv` file is denormalized (one row per criterion judgment per model) — ready for pivot tables and charts. See [docs/eval-comparison.md](docs/eval-comparison.md) for column definitions and interpretation guidance.
+The `--output-csv` file is denormalized (one row per criterion judgment per model) — ready for pivot tables and charts. Every run also carries provenance (repo, git SHA, judge model/prompt, eval spec hash) and per-case cost. Pass `--history results/eval-history.jsonl` to accumulate a time series, then `npm run eval:trend` to view it, with markers wherever the judge/prompt/eval config changed. See [docs/eval-comparison.md](docs/eval-comparison.md) for column definitions and interpretation guidance.
 
 ### Workflow evals (end-to-end agentic testing)