feat(eval): add jobbench harness and correct token accounting

[David Koleczek (DavidKoleczek)]

Two related changes to the evaluation harnesses. Scope of impact: evaluation-only — nothing under src/, wrappers/, or protocol/ is touched, so no engine, wrapper, or protocol release is implied.

Token accounting (deep-swe)

total_tokens was computed as input + output, which dropped cache reads entirely. Because the four agent arms have very different cache behavior, this did not just understate a number — it inverted the ordering. On one real run an opencode trial reported 95,147 tokens against an amplifier trial's 1,218,757, an apparent 12x gap, when opencode had actually processed ~19.6M tokens against amplifier's ~10.2M. The old figure discarded opencode's entire 19.3M cache-read volume.

total_tokens is now the sum of four disjoint fields:

input + cache_read + cache_write + output

For that sum to be valid, input_tokens is normalized to fresh-only on both sources. opencode's tokens_input column is already fresh-only and is accumulated as-is; the amplifier stacks fold cache_read into their reported input_tokens, so parse_events subtracts it back out.

When input < cache_read the convention has broken. Rather than emit a silently wrong number, fresh input is clamped to 0, the occurrence is counted, and the run notes record that input_tokens is a FLOOR and total_tokens may undercount.

cost_usd is unaffected — it is priced from raw per-source counts against MODEL_RATES_PER_M and was never derived from total_tokens.

Any prior cross-arm token comparison from this harness should be treated as invalid.

The OpenCode install is also unpinned in the same change. The benchmark measures opencode as it ships today; pinning a version freezes the control arm against a release that is no longer what users get. The retry loop around the install endpoint is retained, since that flakiness is unrelated.

jobbench harness

A new harness running the same four agent arms against JobBench, a multi-source knowledge-work benchmark: reconciling contradictory records, cross-referencing, and citation tracing across 35 white-collar occupations. It is not a coding benchmark, which is what makes it worth running alongside deep-swe rather than instead of it.

Each task hands the agent a folder of source files and expects deliverables (xlsx, docx, pdf, ipynb, sqlite, pptx). An LLM judge scores them against a weighted, criterion-level rubric, all-or-nothing per rubric. Two splits: main (65 tasks, 569 rubrics) withholds reference material the agent must find via live web search; easy (63 tasks) is self-contained.

run.py            six subcommands: fetch, list-tasks, dtu-check, bake, run, grade
src/jobbench/     dataset fetch, DTU lifecycle, golden image bake, matrix,
                  scheduler, trial, grading, judge
profiles/         a base image plus one layer per agent arm
tests/            127 tests, all synthetic fixtures

Golden Incus images are baked per arm because cold-provisioning LibreOffice and the analysis stack for every trial is untenable at 260 trials; a warm launch is ~15s. No provider, model, or API key is baked into any image layer — credentials are injected at trial launch through DTU passthrough.

grading.py invokes the judge as a module subprocess rather than through upstream's run_judge.sh, because that script silently skips an empty output directory and writes no result file, making a crash indistinguishable from a task that never ran.

metrics.py is vendored byte-identical from deep-swe. The token normalization above was developed here and ported back.

Known issues that bound what the numbers mean

Documented in jobbench/README.md, surfaced here because they limit how the output should be read:

1. Tool results are lost on the opencode-amplifier arm (22–27 occurrences/run, ~2x wall clock, trial still exits 0). Root cause is outside this harness; flagged via warnings in trial.json.
2. That same arm reports all token and cost fields as not_available, by construction.
3. Judge token usage is captured but never priced, and is in no total.
4. Scores are not comparable to the published leaderboard — the authors validated the rubrics against a different judge model.
5. opencode-vanilla's cost_usd is recomputed from this harness's rate card, not opencode's self-reported figure.

Licensing

judge.py and prompt.py are derived from the upstream benchmark under Apache-2.0, with modifications enumerated in their module docstrings and full attribution in THIRD-PARTY-NOTICES.md. They are kept in-tree rather than under vendor/ so local adaptations stay diffable against upstream, and are excluded from lint and format for that reason. The dataset itself is fetched at runtime, not redistributed.

Verification

deep-swe:     27 passed
jobbench:    127 passed
make verify:  lint, format, pyright, verify-codegen, verify-wheel all pass

make verify-parity could not be run to completion locally — the TypeScript conformance runner crashes on a missing esbuild in node_modules, a local environment gap unrelated to this change. CI runs that gate, and nothing here touches src/, wrappers/, or protocol/.

No CHANGELOG entry: this is harness-only and not user-visible in any shipped artifact.