feat(eval): add jobbench harness and correct token accounting

.amplifier/evaluation/README.md

@@ -5,6 +5,12 @@ single canonical definition per agent and per task. Each trial provisions an
 isolated Digital Twin Universe (DTU) environment, drives the agent, extracts its
 work, and grades the result.
 
+Two third-party benchmarks live beside it in their own directories, `deep-swe/`
+and `jobbench/`, because each owns a task format, prompt contract, and grading
+path that must be reproduced exactly for its scores to mean anything. Both are
+described below and each has its own README. All three share the same four
+agent arms.
+
 ## Layout
 
 ```
@@ -34,6 +40,7 @@ run.py                  entry point
 runs/                   gitignored per-run outputs
 
 deep-swe/               separate harness for the deep-swe benchmark (see below)
+jobbench/               separate harness for the JobBench benchmark (see below)
 ```
 
 ## Task groups
@@ -168,6 +175,62 @@ agents installed into pier's venv), and there are several correctness constraint
 that make numbers comparable or worthless. See `deep-swe/README.md` before running
 it.
 
+## jobbench
+
+`jobbench/` is a self-contained harness for
+[JobBench](https://github.com/Job-Bench/job-bench-eval), which measures
+multi-source knowledge work rather than coding: reconciling contradictory
+records, cross-referencing data, tracing citations, across 35 white-collar
+occupations. The agent gets a folder of source files and must produce
+deliverables (xlsx, docx, pdf, ipynb, sqlite, pptx); an LLM judge scores those
+deliverables against a weighted, criterion-level rubric.
+
+It is separate from the matrix harness above because JobBench owns its own task
+format, prompt contract, and judge, all of which must be reproduced exactly for
+scores to mean anything. It does use DTUs, one per (agent, task), launched from
+a per-agent golden Incus image so a container is warm in about 15 seconds
+instead of provisioning from scratch.
+
+```
+jobbench/run.py                    entry point (fetch, list-tasks, dtu-check,
+                                   bake, run, grade)
+jobbench/src/jobbench/             the harness package
+jobbench/src/jobbench/agents/      agent adapters
+jobbench/src/jobbench/judge.py     the JobBench judge, with documented changes
+jobbench/profiles/                 golden-image bake profiles + trial template
+jobbench/tests/                    unit tests (synthetic fixtures only)
+jobbench/THIRD-PARTY-NOTICES.md    Apache-2.0 attribution for the above
+```
+
+`judge.py` is substantially derived from upstream, and `prompt.py` carries
+upstream's prompt template verbatim. Both are Apache-2.0 and stay in-tree
+rather than vendored under a separate directory, so the local adaptations
+documented in their headers remain diffable against upstream. The license and
+the file-by-file attribution are in `jobbench/THIRD-PARTY-NOTICES.md`; the rest
+of the harness is MIT under this repo's top-level `LICENSE`.
+
+The same four arms as the matrix harness: `amplifier-agent`,
+`amplifier-foundation`, `opencode-vanilla`, `opencode-amplifier`.
+
+```
+python run.py fetch --split main
+python run.py bake --agent amplifier-agent
+python run.py run --agent all --all-tasks --split main --max-parallel 4
+```
+
+Two splits: `main` (65 tasks, what the public leaderboard reports, withholds
+reference material the agent is expected to find via live web search) and `easy`
+(63 tasks, self-contained, a different corpus rather than simplified versions of
+the main tasks). Scoring is a weighted rubric score per task, all-or-nothing per
+rubric. Results land under gitignored `jobbench/runs/`, not `runs/`.
+
+The judge defaults to `gpt-5.6-terra` at medium reasoning effort. The JobBench
+authors validated their rubrics against `grok-4.3`, so scores produced here are
+internally consistent and valid for agent-vs-agent comparison but are not
+comparable to the published leaderboard. See `jobbench/README.md` for the
+prompt-contract constraint, the baked baseline toolchain, and known issues
+before running it.
+
 ## Runtime-fetched data
 
 Some task groups store only a selector and pull content on the fly, so no
@@ -178,6 +241,9 @@ benchmark content is committed here:
   (`ScaleAI/SWE-bench_Pro`) and the official scaleapi repo.
 - `tasks/automation-bench/` stores the `task` name plus `example_id` and fetches
   the prompt, tool set, seeded world state, and assertions from AutomationBench.
+- `jobbench/` stores no task content at all. `run.py fetch` downloads the split
+  from HuggingFace (`JobBench/job-bench`) into a gitignored local cache, and the
+  per-run output tree that quotes task and rubric text is gitignored too.
 
 For automation-bench the harness clones `zapier/AutomationBench` (pinned commit)
 exactly once into a machine-local cache under the system temp dir, then extracts

.amplifier/evaluation/deep-swe/.gitignore

@@ -6,3 +6,8 @@ __pycache__/
 .ruff_cache/
 *.egg-info/
 rendered-dockerfiles.txt
+
+# Spurious: this package declares no dependencies and is installed into pier's
+# own venv (`uv pip install -e .`), so a lock file here describes nothing. It
+# only appears as a side effect of ad-hoc `uv run` invocations.
+uv.lock

.amplifier/evaluation/deep-swe/run.py

@@ -426,6 +426,11 @@ def summarize_trial(trial_dir: Path) -> tuple[str, float | None]:
     name = result.get("task_name") or trial_dir.name
     agent_result = result.get("agent_result") or {}
     cost = _num(agent_result.get("cost_usd"))
+    # Every token processed: fresh input + cache + output. This is additive and
+    # double-counts nothing because metrics.py normalizes `input_tokens` to
+    # fresh-only in BOTH branches (the amplifier sources natively fold
+    # cache_read into it; see `parse_events`). Matches metrics.json's
+    # `total_tokens`.
     token_parts = [
         _num(agent_result.get(key))
         for key in ("n_input_tokens", "n_cache_tokens", "n_output_tokens")

.amplifier/evaluation/deep-swe/src/deepswe_agents/base.py

@@ -117,12 +117,16 @@
     InstallStep(user="root", run="curl -LsSf https://astral.sh/uv/install.sh | sh"),
 ]
 
+# opencode is installed UNPINNED, on purpose: the benchmark is meant to measure
+# opencode as it actually ships today, so a pin here would quietly freeze the
+# control arm (and the opencode-backed amplifier arm) at an old build. The
+# retry loop is for install-endpoint flakiness only, not for version drift.
 OPENCODE_PRELUDE = [
     InstallStep(
         user="root",
         run=(
             "for i in 1 2 3 4 5; do\n"
-            "  curl -fsSL https://opencode.ai/install | VERSION=1.17.20 bash && break\n"
+            "  curl -fsSL https://opencode.ai/install | bash && break\n"
             '  echo "opencode install attempt $i failed; retrying in $((i*10))s..." >&2\n'
             "  sleep $((i*10))\n"
             "done\n"

.amplifier/evaluation/deep-swe/src/deepswe_agents/metrics.py

@@ -17,6 +17,35 @@
 
 `not_available` discipline (never fabricate): every normalized field is either
 a real number or the exact string `"not_available"` -- never a silent 0.
+
+TOKEN ACCOUNTING. Every token field means exactly one thing, in both branches,
+and the four are disjoint so they add up:
+
+    input_tokens        fresh input only, never previously cached
+    cache_read_tokens   input served from cache
+    cache_write_tokens  input written into cache
+    output_tokens       generated output
+    total_tokens        the sum of all four: every token actually processed
+
+Reaching that required normalizing the two sources, which do NOT agree on what
+"input" means:
+
+- opencode's `session.tokens_input` column is already fresh-only. Verified on a
+  real run: `tokens_input=322` alongside `tokens_cache_read=19,299,708`.
+- The amplifier stacks fold cache_read INTO their reported `input_tokens` (but
+  not cache_write), so `parse_events` subtracts it back out. Verified across
+  114 events of a real run: 0/114 had input < cache_read, while 114/114 had
+  input < cache_read + cache_write -- e.g. `input=872` with `cache_read=0` and
+  `cache_write=12354`, which only a fresh-plus-cache_read reading explains.
+
+Why it matters: while `total_tokens` was `input + output`, an opencode trial
+reported 95,147 against an amplifier trial's 1,218,757 on the same run -- an
+apparent 12x gap that INVERTED the true ordering, since opencode had actually
+processed ~19.6M tokens to amplifier's ~10.2M. The old figure silently dropped
+opencode's entire 19.3M cache-read volume.
+
+`cost_usd` is unaffected by any of this: it is priced from the raw per-source
+counts against `MODEL_RATES_PER_M`, never derived from `total_tokens`.
 """
 
 from __future__ import annotations
@@ -137,6 +166,10 @@ class _Usage:
     files_read: int = 0
     min_ts: float | None = None
     max_ts: float | None = None
+    #: Records where reported input was somehow smaller than cache_read, i.e.
+    #: the source's convention is not what `parse_events` assumes. Surfaced in
+    #: `notes` rather than swallowed, because the resulting figure is wrong.
+    negative_fresh_input: int = 0
 
     def observe_time(self, epoch: float | None) -> None:
         """Widen the earliest-to-latest span with one timestamp; None is a no-op."""
@@ -158,7 +191,16 @@ def as_dict(self) -> dict[str, Any]:
             "output_tokens": self.output_tokens,
             "cache_read_tokens": self.cache_read_tokens,
             "cache_write_tokens": self.cache_write_tokens,
-            "total_tokens": self.input_tokens + self.output_tokens,
+            # Every token the model actually processed. `input_tokens` is
+            # fresh-only in both branches by construction, so cache_read and
+            # cache_write are additive here and nothing is double-counted.
+            "total_tokens": (
+                self.input_tokens
+                + self.cache_read_tokens
+                + self.cache_write_tokens
+                + self.output_tokens
+            ),
+            "negative_fresh_input": self.negative_fresh_input,
             "cost_usd": self.cost_usd,
             "cost_from_events": self.saw_cost,
             "llm_responses": self.llm_responses,
@@ -337,6 +379,11 @@ def parse_opencode_db(db_paths: list[str], workspace_dir: str) -> dict[str, Any]
         # run with clear usage is never reported as 0 responses.
         usage.llm_responses += assistant or len(sessions)
         for s in sessions:
+            # opencode's `tokens_input` column is already FRESH-ONLY (it
+            # excludes both cache figures), which is the convention this module
+            # normalizes to, so it is accumulated as-is. The amplifier branch
+            # has to strip cache_read out to reach the same meaning; see
+            # `parse_events`.
             s_in = _to_int(s.get("tokens_input"))
             s_out = _to_int(s.get("tokens_output"))
             s_cr = _to_int(s.get("tokens_cache_read"))
@@ -510,14 +557,28 @@ def parse_events(events_paths: list[str]) -> dict[str, Any]:
             # Field names differ by runtime/provider: amplifier-agent emits the
             # `_tokens`-suffixed names, the Python Anthropic provider emits the
             # bare names. Accept either.
-            usage.input_tokens += _to_int(_pick(event_usage, "input_tokens", "input"))
+            reported_in = _to_int(_pick(event_usage, "input_tokens", "input"))
+            ev_cache_read = _to_int(_pick(event_usage, "cache_read_tokens", "cache_read"))
+            ev_cache_write = _to_int(_pick(event_usage, "cache_write_tokens", "cache_write"))
+            # The amplifier stacks report an `input_tokens` that ALREADY
+            # contains cache_read (but not cache_write), while opencode reports
+            # a fresh-only figure. Strip cache_read here so `input_tokens` means
+            # exactly one thing -- genuinely-new, never-cached input -- no
+            # matter which source produced the record. Measured on a real run:
+            # 114/114 events had input >= cache_read and input < cache_read +
+            # cache_write, e.g. input=872 with cache_read=0, cache_write=12354.
+            fresh_in = reported_in - ev_cache_read
+            if fresh_in < 0:
+                # The invariant above broke, so the assumption no longer holds
+                # for this source. Clamp rather than emit a negative token
+                # count, and say so: a silently wrong number is the failure
+                # this whole normalization exists to prevent.
+                usage.negative_fresh_input += 1
+                fresh_in = 0
+            usage.input_tokens += fresh_in
             usage.output_tokens += _to_int(_pick(event_usage, "output_tokens", "output"))
-            usage.cache_read_tokens += _to_int(
-                _pick(event_usage, "cache_read_tokens", "cache_read")
-            )
-            usage.cache_write_tokens += _to_int(
-                _pick(event_usage, "cache_write_tokens", "cache_write")
-            )
+            usage.cache_read_tokens += ev_cache_read
+            usage.cache_write_tokens += ev_cache_write
             # cost_usd is only emitted by the amplifier-agent stack. Track
             # whether we ever saw it so a $0 from a stack that does not record
             # cost is not reported as a real, free run.
@@ -721,6 +782,21 @@ def _finalize(
     if isinstance(record["cost_usd"], (int, float)):
         record["cost_usd"] = round(float(record["cost_usd"]), 6)
 
+    # A source whose input figure was smaller than its own cache_read violates
+    # the convention `parse_events` normalizes against, so the fresh-input
+    # figure for those records is a clamped 0 rather than the truth. Say so.
+    if parsed.get("negative_fresh_input"):
+        notes = [
+            *notes,
+            (
+                f"{parsed['negative_fresh_input']} record(s) reported input_tokens "
+                "below their own cache_read, which contradicts the "
+                "fresh-plus-cache_read convention this parser normalizes; fresh "
+                "input was clamped to 0 for those, so input_tokens is a FLOOR and "
+                "total_tokens may undercount."
+            ),
+        ]
+
     record["source"] = source
     record["events_files"] = list(source_files)
     record["notes"] = " ".join([*notes, "total_wallclock_s is not measured by this harness."])