diff --git a/bench/ABC_CHECKLIST.md b/bench/ABC_CHECKLIST.md new file mode 100644 index 0000000..1eb7b61 --- /dev/null +++ b/bench/ABC_CHECKLIST.md @@ -0,0 +1,57 @@ +# ABC checklist — benchmark-rigor self-audit + +A self-audit against the **Agentic Benchmark Checklist** (ABC) from *Establishing Best Practices for +Building Rigorous Agentic Benchmarks* (Zhu et al., NeurIPS 2025 D&B; arXiv **2507.02825**). The ABC +groups failure modes into **task validity** (can the task be solved as specified?), **outcome +validity** (does the grader actually measure success — no shortcuts, no false pass/fail?), and +**reporting**. The paper found *all 10* benchmarks it audited had reporting gaps; this file is our +attempt not to be one of them. Item ids follow the paper's grouping. + +Legend: ✅ satisfied · ⚠️ partial / by-design limitation · n/a not applicable to this benchmark. + +## Task validity + +| Item | How this benchmark addresses it | +|---|---| +| **T.1** tool/version specificity | Model `model_id`s pinned in `config.toml`, copied into `run.json`; `surf --version` recorded. Python env pinned by `uv.lock`. ✅ | +| **T.2/T.3** API availability / interruption handling | Per-request 120 s timeout + bounded retries; a cell that still fails is logged with `error` and **excluded** from rates (not counted as a failure). ✅ | +| **T.4** clean state between tasks | Each grade runs in a **fresh temp workspace** (`grade_code`) / each multi-turn trial in a **fresh sandbox** (`tools_runtime.scenario_sandbox`). No cross-task leakage. ✅ | +| **T.5** isolate agent from ground truth | The drifted dependency is withheld from the prompt (`hidden_paths`); graders compute ground truth separately by probing the real dependency. The agent never sees the grader. ✅ | +| **T.6** freeze environment at release | Scenario set frozen + git-tagged at pre-registration (see `PREREGISTRATION.md` freeze gate). ✅ | +| **T.7/T.8** verify ground-truth annotations | Ground truth is **derived from the real code**, not hand-annotated: code graders import and probe the hidden dependency; QA rubrics encode the T1 behaviour the committed code implements. ✅ | +| **T.9** oracle/reference solution | Every scenario ships `.author/solution_correct.*` and `solution_stale.*`; `tools/validate_scenario.py` runs the live graders on both and proves they discriminate, offline, before any spend. ✅ | +| **T.10** inspect pilot outliers | Staged rollout (`mock` → per-provider smoke → small-N pilot → full) with the oracle as a tripwire; the v1 pilot's outlier (a leaked stale value) was caught this way (#113). ✅ | + +## Outcome validity + +| Item | How this benchmark addresses it | +|---|---| +| **O.a** semantic-equivalence / redundant-word robustness | QA grading parses a **fixed `VERDICT:` field format** (regex per field) rather than free-text matching; "last match wins" tolerates the model restating the format. Code grading runs real tests, so wording is irrelevant. ✅ | +| **O.b.2/O.b.3** no trivial success (list-all / empty answer) | Code: an empty or non-compiling answer fails the hidden test. QA: a missing/garbled `VERDICT` parses as neither correct nor misled (not a pass). ✅ | +| **O.d.1** manually verify test cases | Each grader's correct/misled tests are validated by `validate_scenario.py` against the reference solutions; the **misled** test must pass on the stale solution and fail on the correct one (and vice-versa). ✅ | +| **O.f.2** eliminate non-determinism in tests | Graders are deterministic (no clocks/network/randomness in the checked logic; fixed probe inputs). ✅ | +| **O.h.1** specify output format | `task.md` states the exact contract: a `FILE: ` block (code) or a `VERDICT: …` line (QA). ✅ | +| **O.h.2** tasks resistant to guessing | The reason multi-turn is the centerpiece: in single-shot a hidden value is only knowable from the doc, so "follow the stale doc" is partly rational. Multi-turn lets the agent *verify*, removing the tautology. QA verdicts use **two fields** to cut a 50/50 guess. ⚠️ (single-shot cascade is guess-resistant only because the value is genuinely unknowable without the doc — stated as a limitation) | +| **O.i.1** metric correlates with the construct | `verified_then_correct` validates the verification metric (reading the truth should rescue the answer); the misled metric directly encodes "asserted the stale claim". ✅ | + +## Reporting + +| Item | How this benchmark addresses it | +|---|---| +| **R.1/R.2** open data + eval code | The entire harness, scenarios, graders, and reference solutions are in this repo. `raw.jsonl` is preserved per run. ✅ | +| **R.3/R.4** contamination | Scenarios are bespoke fixtures, not scraped; the drift values live only in `hub_stale.md`. (Frontier models may have seen similar idioms — noted as residual risk.) ⚠️ | +| **R.5/R.6** state the capability assessed | `PREREGISTRATION.md` §9 states exactly what is and isn't assessed. ✅ | +| **R.7** mitigation for unavoidable flaws | The neutral system prompt + neutral `task.md` rules (#113 lesson) mitigate doc-trust bias; the oracle mitigates fixture defects. ✅ | +| **R.8/R.9** quantify limitation impact | Per-scenario and per-tier breakdowns expose any single-fixture or single-difficulty artifact. ✅ | +| **R.10** statistical significance | Wilson intervals, bootstrap CIs + p-values, **Holm–Bonferroni** across the confirmatory family. ✅ | +| **R.11** interpretation guidance | `README.md` §4 explains what each metric means; `PREREGISTRATION.md` separates confirmatory from exploratory. ✅ | +| **R.12/R.13** baselines/controls | **C0** (no-doc baseline) and **Cw** (warning-only control) isolate, respectively, the value of accurate prose and whether recovery is Surface's *fix* vs generic suspicion. ✅ | + +## Known limitations (disclosed, per R.7) + +- **Curated fixtures, not real repositories** (external validity); a real-OSS case study is deferred + (#109). +- **Read-only agent loop** — no edit/run/test "thrash"; deferred by design. +- **Languages:** Python + TypeScript only. +- **`author.py` seals only the first anchor**, so all current scenarios are single-anchor (no genuine + T3 multi-claim); a tooling follow-up would lift this. diff --git a/bench/PREREGISTRATION.md b/bench/PREREGISTRATION.md new file mode 100644 index 0000000..59bb269 --- /dev/null +++ b/bench/PREREGISTRATION.md @@ -0,0 +1,95 @@ +# Pre-registration — Surface agent-impact benchmark (v2 / multi-turn) + +**Status:** to be **frozen and git-tagged before the headline run.** This file states the +hypotheses, design, metrics, and analysis *in advance* so the result can't be accused of HARKing +(hypothesizing after results are known) or of cherry-picking an analysis. Nothing below is decided +by looking at outcomes. Operating instructions are in [`README.md`](README.md). + +> **Freeze gate.** Tag this only once: (a) the scenario set is final (all `cascade-*` PRs merged), +> (b) `python tools/author.py --all` and `python tools/validate_scenario.py --all` are green across +> the set, and (c) a full `--models mock` run + `python -m surface_bench.oracle` is clean. Record +> the tag and the `surf --version` here when frozen: `__TBD__`. + +## 1. Question + +Does keeping documentation accurate (what Surface enforces) change an agent's task performance, and +*through what mechanism*? Surface's value to an agent = the performance gap between working from +fresh vs rotted docs, plus whether surfacing the drift recovers it. + +## 2. Conditions (the only thing that varies) + +Same code + same task + same model in every cell; only the documentation block changes: +**C0** code only · **C1** code + stale doc · **C2** code + fresh doc · **C3** code + stale doc + real +`surf check --format json` report · **Cw** code + stale doc + a generic "may be outdated" warning +(no corrected code). Run in **multi mode** (the agent has read-only `read_file`/`grep`/`list_dir` +tools and may choose to verify) as the primary design; **single mode** is run as a secondary, +cheaper comparison. + +## 3. Hypotheses (confirmatory) + +- **H1 — accuracy beats rot:** success(C2) > success(C1). +- **H2 — rot is worse than nothing:** misled(C1) > misled(C0). +- **H3 — surfacing drift recovers it:** success(C3) ≈ success(C2) (and ≫ C1). +- **H4 — a confident stale doc suppresses verification:** verification_rate(C1) < verification_rate(C0). +- **H5 — the harm flows through skipped verification (mediation):** within C1, success among rows + that verified > success among rows that did not; verifiers are correct, non-verifiers misled. +- **H6 — recovery is the corrected code, not mere suspicion:** success(C3) > success(Cw). + +Direction is pre-specified for every hypothesis. Anything not listed here (e.g. per-provider +contrasts, tier gradients, token-cost deltas) is **exploratory** and will be reported as such. + +## 4. Metrics (pre-defined) + +- **success** — `ok`: the agent produced the current (T1) answer. Code scenarios: hidden tests that + probe the real dependency. QA scenarios: a `VERDICT:` line matched against a fixed rubric. **No LLM + judge.** +- **misled** — `misled`: the agent asserted the stale (T0) claim. +- **verification_rate** (multi only) — the agent called `read_file`/`grep` on a path matching the + scenario's `hidden_paths` glob *before* `final_answer`. Computed from the logged `verified_hidden` + per row. `verified_then_correct` (success among verifiers) is its validity check. +- **output tokens** and **cost** — secondary (token-tax + spend), exploratory. + +## 5. Models, scale, sampling (pre-specified) + +- **Models:** `haiku`, `sonnet`, `opus` (Anthropic) + `gpt` (OpenAI) + `gemini` (Google). Exact + `model_id`s and prices are pinned in `config.toml` at freeze time and copied into `run.json`. +- **Trials (tiered):** N = 10 per cell for the **cascade** family (the headline); N = 5 for the + **comprehension** family (success-ceilinged — kept only for the token story). Comprehension may be + omitted from multi mode entirely (it doesn't test verification); that choice is recorded in + `run.json`, not chosen by results. +- **temperature = 1.0** (stochasticity is part of what we measure); **max_turns = 8**; + `max_tokens = 1024`. + +## 6. Analysis plan (pre-specified) + +- Rates reported with 95% **Wilson** intervals. +- Each hypothesis tested as an unpaired difference of proportions with a 95% **bootstrap** CI and a + bootstrap two-sided p-value (10,000 resamples, fixed seed). +- **Holm–Bonferroni** correction applied across the family of confirmatory success-delta tests + (every model × pre-registered pair). A hypothesis is **confirmed** if its CI excludes 0 **and** it + survives Holm; reported as suggestive if CI-significant but not Holm-significant. +- H5 (mediation) reported as the within-C1 success split (verified vs not) per model. +- Per-scenario and per-tier breakdowns reported for transparency (exploratory). + +## 7. Exclusions / data handling (pre-specified) + +- A cell that errors (API failure) is logged with `error` and excluded from rates; the count of + excluded cells is reported. No silent retries beyond the client's built-in retry. +- The oracle (§ README) must be clean on the real run: any scenario failing C2-fresh ≥ 90% or (for + cascade) C1-misled > 0 is treated as a **fixture defect** and is flagged; whether to drop it is + decided by the pre-stated oracle rule, not by whether it helps the result. + +## 8. Stopping rule + +The matrix runs to completion (fixed N above). No data-dependent stopping, no N top-ups chosen by +looking at significance. If a stage smoke (mock → per-provider → small-N pilot) reveals a *harness* +bug, it is fixed and the affected stage re-run; the confirmatory full run begins only after this file +is tagged. + +## 9. What this does and does not assess + +Assesses: whether documentation accuracy changes single-shot and multi-turn task outcomes on curated +cascade fixtures across five models, and whether the effect is mediated by verification. Does **not** +assess: real-repository generalization (curated fixtures by design — see deferred #109), an +edit/run/test agent loop (tools are read-only by design), or non-English / languages beyond Python +and TypeScript. diff --git a/bench/README.md b/bench/README.md index 61846b5..f8321ab 100644 --- a/bench/README.md +++ b/bench/README.md @@ -1,110 +1,269 @@ -# surface-bench +# surface-bench — operator manual -Empirically measuring how much **documentation accuracy** changes an agent's task performance — -the gap [Surface](../README.md) exists to protect. +Empirically measuring how much **documentation accuracy** changes an agent's task performance — the +gap [Surface](../README.md) exists to protect. Surface doesn't make an agent smarter; it stops docs +silently rotting. So its value to an agent equals the performance delta between working from **fresh** +docs and **rotted** docs, plus whether *surfacing* the drift recovers the loss. This bench measures +those deltas directly, using drift of exactly the kind `surf check` catches (flipped operators, +dropped guards, changed constants, reordered keys). -Surface doesn't make an agent smarter; it stops docs silently rotting. So its value to an agent -equals the performance delta between working from **fresh** docs and **rotted** docs. This bench -measures that delta directly, using drift of exactly the kind `surf check` catches (flipped -operators, dropped guards, changed constants). +This is the single place to start when you (or a future agent) pick this up cold. It covers **what the +experiment is, how to run it, how to read the results, how to author scenarios, and the sharp edges.** +Lives in the Surface repo but has no inbound dependency on the Rust core — it only *consumes* the +`surf` binary's output — so it can be extracted later. -This lives in the Surface repo for now but has no inbound dependency on the Rust core — it only -*consumes* the `surf` binary's output — so it can be extracted to a standalone repo later. +> **Companion docs:** [`PREREGISTRATION.md`](PREREGISTRATION.md) (the frozen hypotheses + analysis +> plan for the headline run), [`ABC_CHECKLIST.md`](ABC_CHECKLIST.md) (benchmark-rigor self-audit), +> [`scenarios/CHECKLIST.md`](scenarios/CHECKLIST.md) (how to author one scenario), and the committed +> v1 write-up at [`results/2026-06-13-pilot-full-matrix/report.md`](results/2026-06-13-pilot-full-matrix/report.md). -## The experiment +--- -Same code + same task in every run; **only the documentation block changes**: +## 1. The experiment in one screen + +Same code + same task in every run; **only the documentation block changes**. There are five +conditions: | | Context shown to the agent | Represents | |---|---|---| -| **C0** | code only | baseline | +| **C0** | code only (no doc) | baseline | | **C1** | code + **stale** doc (true at T0, code moved to T1) | the ungoverned world | | **C2** | code + **fresh** doc (matches T1) | the Surface-governed world | | **C3** | code + stale doc + real `surf check --format json` report | "just surface the drift" | +| **Cw** | code + stale doc + a generic "may be outdated" warning (no corrected code) | control: is it the *fix* or just *suspicion*? | -Hypotheses: **H1** C2 > C1 (accuracy beats rot — the core value) · **H2** C1 < C0 (rotted docs are -*worse than nothing*) · **H3** C3 ≈ C2 (surfacing drift recovers the loss). +And two run **modes**: -Three metrics per condition: **success rate** (deterministic grader), **misled rate** (the agent -asserted the stale claim), and **output-token cost** (does a stale doc cost extra generation to -course-correct?). +- **single** (v1): one prompt → one completion. Cheap, reproducible, the original pilot. +- **multi** (v2, the centerpiece): a **multi-turn agent loop** with **read-only** tools + (`read_file`, `grep`, `list_dir`, `final_answer`). The agent can *choose* to read the hidden + dependency the doc describes. This is what makes the headline non-tautological — see §4. -Every run also reports **estimated spend** (`Total: $X`) — token usage × the per-model -`input_per_mtok` / `output_per_mtok` prices in `config.toml`. The report's per-call estimate is for -relative comparison; your Anthropic console invoice is the authoritative total for the "I spent $X -validating Surface" figure. +### Hypotheses -## Layout +| | Claim | Read on | +|---|---|---| +| **H1** | C2 > C1 — accuracy beats rot (the core value) | success rate | +| **H2** | C1 < C0 — rotted docs are *worse than nothing* | misled rate | +| **H3** | C3 ≈ C2 — surfacing drift recovers the loss | success rate | +| **H4** | verification_rate(C1) < verification_rate(C0) — a confident stale doc **suppresses verification** | verification rate (multi only) | +| **H5** | within C1, agents that read the hidden dep are correct; those that don't are misled | mediation (multi only) | +| **H6** | C3 > Cw — recovery is Surface's *corrected code*, not mere suspicion | success rate | -``` -bench/ - config.toml models, trials, temperature - surface_bench/ models · prompts · run · grade_qa · grade_code · metrics · report - scenarios// meta.toml · code/ · hub_stale.md · hub_fresh.md · surf_report.json · task.md · grader/ - tools/author.py regenerate a scenario's hashes + surf_report.json with the real surf binary - results// raw.jsonl · run.json · summary.json · report.md (gitignored) +### Two scenario families + +- **Cascade** (the headline): the agent edits/answers about a **visible** thing whose correctness + depends on a **hidden dependency** (listed in `meta.toml` `hidden_paths` — present in `code/` for + grading and for `surf` to seal a real divergence, but withheld from the prompt). The dependency + has drifted from what the stale doc says, so the doc is the agent's only (single-shot) or + *optional* (multi-turn) window onto the truth. `cascade-*` scenarios. +- **Comprehension**: the drifted code *and* its contradicting doc are both visible. The model can + just re-read the code, so success ceilings near 100% — useful for the **token-tax** story (a stale + doc costs extra generation to reconcile), not the success story. + +--- + +## 2. Setup + +The harness env is managed with [uv](https://docs.astral.sh/uv/) (the committed `uv.lock` pins it so +the spend figures are reproducible). Code-edit graders run under the *same* interpreter `uv run` +selects — no `python3`-on-PATH guessing. + +```sh +# from repo root: build the real surf binary (author.py + the C3 report need it) +cargo build --release # provides target/release/surf + +cd bench +uv sync # base deps (anthropic) +uv sync --extra dev # + pytest (to run the test suite) +uv sync --extra plots # + matplotlib (report figures) +uv sync --extra providers # + openai, google-genai (cross-provider runs) ``` -## Run it +**Toolchains the graders need at run time** (only when those scenarios are actually graded): -The harness env is managed with [uv](https://docs.astral.sh/uv/) (the committed `uv.lock` pins it -so the "I spent $X validating Surface" figure is reproducible). `uv run` puts the project venv's -interpreter first, and the code-edit graders execute under that *same* interpreter — so there is no -`python3`-on-PATH guessing. +- **Python** — always (via `sys.executable`). +- **Node ≥ 22.18** on PATH — for TypeScript code scenarios (`node --test`; that release strips TS + types at load, so no `npm install`/`tsc`). uv does not manage Node; install it separately. + +**API keys** (only for the providers you select): `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, +`GEMINI_API_KEY` (or `GOOGLE_API_KEY`). The convenience env file `~/.surface-bench.env` is sourced +per command in practice (`set -a; source ~/.surface-bench.env; set +a`). + +--- + +## 3. Running it + +The pipeline is four commands. **Everything except `run` against a real provider is free/offline.** ```sh -cargo build --release # from repo root: provides target/release/surf -cd bench && uv sync # add --extra plots for charts, --extra dev for pytest +# (a) RUN the matrix -> results//{raw.jsonl, run.json} +uv run python -m surface_bench.run --models mock # offline pipeline check, no key +uv run python -m surface_bench.run --models haiku --mode single --trials 10 +uv run python -m surface_bench.run --models haiku --mode multi --trials 10 --max-turns 8 -# offline pipeline check — no API key, no tokens -uv run python -m surface_bench.run --models mock +# (b) REPORT -> results//{summary.json, report.md, *.png} +uv run python -m surface_bench.report results/ -# real pilot -export ANTHROPIC_API_KEY=... -uv run python -m surface_bench.run --models haiku --trials 10 -uv run python -m surface_bench.report results/ +# (c) ORACLE — post-run sanity tripwires (exits non-zero if any fire) +uv run python -m surface_bench.oracle results/ ``` -> Prefer plain pip? `pip install -e .` then drop the `uv run` prefix — everything still works. +**CLI flags** (`run`): `--models ` (subset of `config.toml` models; default all), +`--scenarios `, `--conditions C0 C1 C2 C3 Cw`, `--trials N`, `--mode {single,multi}`, +`--max-turns N` (multi), `--out `, `--config `. Config defaults live in `config.toml` +(`trials`, `temperature`, `max_tokens`, `mode`, `max_turns`, and a `[models.]` block per +model with `provider`, `model_id`, and `input_per_mtok` / `output_per_mtok` pricing). -**Polyglot scenarios.** Code-edit scenarios graded with `node --test` (e.g. the TypeScript -`pagination-ts-code`) need **Node ≥ 22.18** on `PATH` — that is the first release where TypeScript -type-stripping is on by default, so the agent's `.ts` runs with no `npm install` or `tsc` step. -uv does not manage Node; install it separately. +**Providers:** `provider = "mock" | "anthropic" | "openai" | "gemini"`. Mock needs no key and is the +offline workhorse; in `--mode multi` it scripts a canned answer so the whole loop runs for free. -## Authoring a scenario +### The matrix size (and why you stage spend) -1. Write `code/` (the **drifted/T1** state the agent sees) and `.author/code_t0/` (the pre-drift - files that differ — usually just the anchored symbol). -2. Write `hub_stale.md` (describes T0) and `hub_fresh.md` (describes T1), both anchoring the same - symbol; leave `hash: 000000000000`. -3. Write `task.md`. **QA:** end with a `VERDICT:` line the rubric parses. **code-edit:** ask for - full files in `FILE: ` blocks. -4. Write the grader: `grader/rubric.toml` (QA) or `grader/grader.toml` + `grader/tests/` (code). -5. `python tools/author.py scenarios/` — seals both hubs against the real binary and emits the - genuine `surf_report.json`. It fails if the stale hub doesn't actually diverge. +`#scenarios × #conditions × #models × N`. Multi mode multiplies each cell by the number of agent +turns *and* re-sends the growing transcript each turn, so it is **much** more expensive than single. +**Always stage:** `mock` → one scenario × each provider (smoke the tool round-trip) → cascade-only +multi at small N (pilot the verification metric) → the full matrix. Gate each stage on the oracle +(§5) passing. Cost levers: `--max-turns` cap, tiered `--trials` (e.g. N=10 cascade / N=5 +comprehension), and dropping comprehension from multi mode (it doesn't test verification). -Keep the correct answer **non-obvious from a quick code read**, or the documentation carries no -weight and the bench measures nothing. +--- -### Two scenario families +## 4. Reading the results + +`run` writes **`raw.jsonl`** (one row per completion — the source of truth; grading can be re-run +offline without re-spending) and **`run.json`** (the run's parameters). `report` turns those into +`summary.json` (machine-readable) + `report.md` (the human read) + figures. + +**Per-row fields** (multi-only fields absent in single rows): `scenario, task_type, tier, condition, +model, trial, mode, output, input_tokens, output_tokens, cost_usd, ok, misled, detail, parsed` and, +for multi: `turns, stop_reason, tool_calls, verified_hidden, per_turn_tokens`. + +**The metrics, and what each tells you:** + +- **success rate** (`ok`) — got the current (T1) answer. The H1/H3/H6 axis. Deterministic: code + scenarios run hidden tests; QA scenarios parse a `VERDICT:` line against a rubric. **No LLM judge.** +- **misled rate** (`misled`) — asserted the *stale* (T0) claim. The H2 axis: a rotted doc doesn't + just fail to help, it *causes* the wrong answer. +- **verification_rate** (multi) — did the agent read/grep a `hidden_path` before answering? The + **headline of the agentic track** (H4). `verified_then_correct` is its validity check (reading the + truth should rescue you). +- **output tokens** — generation cost. Input tokens differ by construction (doc-block size) so are + ignored; output tokens carry the token-tax signal in the comprehension family. +- **cost_usd** — estimated spend (tokens × `config.toml` prices). The report's total is for relative + comparison; your provider console invoice is authoritative. + +**Statistics:** rates carry 95% **Wilson** intervals; condition deltas use a 95% **bootstrap** CI +*and* a bootstrap p-value, with **Holm–Bonferroni** applied across the whole success-delta family +(the report flags `(Holm ✓/✗)` per delta). CI-significance is the headline; Holm is the conservative +cross-check. The report also slices deltas **by tier** (the difficulty gradient) and **by scenario** +(catches one broken fixture hiding in a family average). + +**`report.md` sections:** Spend · the gradient (C2−C1 by tier) · **Verification** (multi: the H4 +deltas + H5 mediation) · per-model rate tables + deltas (with Holm flags) · output-token tables + +deltas · per-scenario success. **Figures:** `overview.png`, `cascade_success.png`, and (multi) +`verification_rate.png` — the "does a stale doc stop the agent checking?" hero chart. + +--- + +## 5. The oracle (sanity tripwires) + +`python -m surface_bench.oracle results/` is a cheap post-run gate that catches authoring/harness +bugs **before** they reach a write-up (the failure mode that bit us in issue #113). It exits non-zero +(so it can gate CI / a staged run) if, per scenario × model: + +- **C2-fresh < 90%** — with a *fresh* doc the task must be solvable; a low cell means the scenario is + mis-authored (leaked stale value, broken grader, impossible task), not a real effect. +- **a cascade C1 never misleads** — if a stale doc never produces the wrong answer, the drift isn't + load-bearing and the scenario measures nothing. + +The mock can't satisfy these (it doesn't actually solve scenarios), so run the oracle on **real** +outputs. + +--- + +## 6. Layout + +``` +bench/ + config.toml models, trials, temperature, mode, max_turns, per-model pricing + PREREGISTRATION.md frozen hypotheses + analysis plan for the headline run + ABC_CHECKLIST.md benchmark-rigor self-audit (arXiv 2507.02825) + surface_bench/ + run.py the matrix runner (single + multi); writes raw.jsonl + run.json + prompts.py assembles (system, user) per condition; hides hidden_paths + models.py provider adapters: complete() (single) + step() (multi, tool-use); + Anthropic / OpenAI / Gemini + Mock; neutral Step/ToolCall types + agent.py run_agent() — the multi-turn loop over the read-only tools + tools_runtime.py read-only tool surface (read_file/grep/list_dir/final_answer) + sandbox + grade_qa.py VERDICT-line rubric grader (QA) + grade_code.py FILE-block overlay + hidden-test grader (code) + metrics.py rates, Wilson/bootstrap CIs, Holm, verification, by_tier, by_scenario + report.py summary.json + report.md + figures + oracle.py post-run tripwires + tools/ + author.py seal a scenario's hub hashes + emit genuine surf_report.json + validate_scenario.py grader-polarization self-test (offline, no spend) + scenarios/ + CHECKLIST.md how to author one scenario + / meta.toml · task.md · hub_stale.md · hub_fresh.md · surf_report.json + code/ (T1) · .author/code_t0/ (T0 overlay) · .author/solution_{correct,stale}.* + grader/ (rubric.toml | grader.toml + tests/) + tests/ offline pytest (tools, agent loop, adapters, metrics, scenario polarity) + results// raw.jsonl · run.json · summary.json · report.md · *.png (gitignored, + except committed snapshots like 2026-06-13-pilot-full-matrix) +``` + +--- + +## 7. Authoring a scenario (summary — full steps in `scenarios/CHECKLIST.md`) + +A cascade scenario clones `scenarios/cascade-quota-batcher-code/`. The loop: + +1. Pick a **load-bearing** drift: name an input where T0 and T1 give *different* outputs (else the + doc carries no weight and the bench measures nothing). +2. Write `code/` (T1, the visible file stubbed) + `.author/code_t0/` (only the changed files, T0) + + `hub_stale.md`/`hub_fresh.md` (placeholder `hash: 000000000000`) + a **neutral** `task.md` + + the grader + `.author/solution_{correct,stale}.*` reference solutions. +3. `python tools/author.py scenarios/` — seals the hub hashes against the real binary and emits + `surf_report.json`; **fails loudly if the stale hub doesn't actually diverge.** +4. `python tools/validate_scenario.py scenarios/` — runs the live graders on the two reference + solutions and proves they **discriminate** (correct → ok & not misled; stale → not ok & misled). +5. `--models mock` run + `oracle` to confirm the pipeline + tripwires. + +**Two rules with teeth:** + +- **Graders probe the real hidden dependency** for ground truth — never hardcode the T1 value. + `check_misled` hardcodes the *stale* value. +- **Neutrality** (the #113 lesson): the stale value appears *only* in `hub_stale.md` — never in + `task.md` or the visible code, and never a "the doc may be wrong" hint. A leak re-introduces the + doc-trust bias the neutral system prompt is designed to remove. + +--- + +## 8. Sharp edges (read before authoring or debugging) + +- **`surf` alpha-renames identifiers when hashing.** A drift expressed only as a *named-constant + swap* (e.g. `ROUND_HALF_EVEN` → `ROUND_HALF_UP`) is **invisible** to `surf check` — the two hash + identically. A detectable drift must change a **literal, operator, or structure** (a number, a + string, `<`→`<=`, an added/removed statement, a reordered call). `author.py` fails with "expected + a 'changed' divergence" if you trip this — redesign the drift, don't fight the hasher. +- **`author.py` seals only the *first* anchor's hash.** Multi-anchor hubs (a genuine T3 "multi-claim" + scenario) need a small tooling change to seal every anchor; until then keep hubs single-anchor. +- **Read-only tools by design.** The multi-turn agent can read but not edit or run code. Giving it a + test runner would let it brute-force ground truth and wash out the doc-trust signal we measure. A + full edit/run "thrash" loop is deliberately deferred future work. +- **Don't stack PRs.** Scenario PRs are additive and independent — open each against `main` directly. + (A stacked PR once merged into its base branch instead of `main` and silently lost its content.) +- **Spend is in the *run*, not the docs/tooling.** `author.py`, `validate_scenario.py`, `report`, + `oracle`, the test suite, and any `mock` run cost nothing. Only `run` against a real provider does. + +--- + +## 9. Reproducibility -- **Comprehension** (e.g. `refresh-single-use-qa`): the drifted code *and* its contradicting doc - are both in the prompt. Measures whether an agent re-derives truth from code it can see. -- **Cascade** (e.g. `cascade-quota-batcher-code`): the realistic context-rot shape. The agent edits - a visible function whose correctness depends on a **hidden dependency** — the drifted code is - listed in `meta.toml`'s `hidden_paths` so it stays in `code/` (for `surf` to seal a real - divergence and for the grader to run against it) but is withheld from the prompt. The agent knows - the dependency only through its doc, so a stale doc propagates into a wrong cascaded edit; in C3 - the `surf` report's `new_code` is the only window onto the truth. The grader derives the expected - value from the real hidden dependency, so the test stays honest: - - ```toml - # meta.toml - hidden_paths = ["code/limiter/*.py"] # present for grading, hidden from the prompt - ``` - - Author cascades so the contract is **only** knowable from the doc (the dependency's code is - hidden), and avoid leaking the stale value through the visible file's wording or worked examples - — and through the task text: phrase it neutrally, never "work from the documentation", or you - re-introduce the doc-trust bias the neutral system prompt removes. +`uv.lock` pins the Python env. `run.json` records every parameter (models + ids, trials, +temperature, max_tokens, mode, max_turns, conditions, scenarios, `surf --version`). `raw.jsonl` +preserves raw outputs so grading/metrics re-run offline. To reproduce a committed snapshot, check out +the repo at its tag, `cargo build --release`, `cd bench && uv sync`, and re-run `report` on the +snapshot dir (it regenerates `summary.json` + figures from the frozen `raw.jsonl`).