feat(bench): phase-7 tooling + 4 Python cascade scenarios (#104)

bench/scenarios/CHECKLIST.md

@@ -0,0 +1,59 @@
+# Authoring a cascade scenario
+
+A *cascade* scenario: the agent edits a **visible** file whose correctness depends on a **hidden**
+dependency (in `code/`, listed in `meta.toml` `hidden_paths`, filtered out of the prompt). The
+dependency has drifted from what the **stale** doc says, so an agent that trusts the stale doc is
+wrong. The graders probe the *real* hidden dependency for ground truth. Clone
+`cascade-quota-batcher-code/` as the canonical template.
+
+## 0. Pick a load-bearing drift
+- [ ] Choose a drift archetype **not already in the family** (avoid: boundary `<`→`<=`, plain
+      constant, allow→block, page-size constant — see existing `cascade-*`).
+- [ ] Name a concrete input where **T0 and T1 produce different observable outputs**. If you can't,
+      the drift isn't load-bearing — stop and rethink. (The whole effect dies if T0 ≈ T1.)
+
+## 1. Directory skeleton  (`scenarios/<id>/`)
+- [ ] `meta.toml`: `id`, `title`, `lang`, `task_type`, `tier`, `anchor = "code/<file> > Symbol >
+      method"`, `invariant`, `drift`, `hidden_paths` (glob over the dependency), `edit_path` (the
+      visible file the agent returns — code scenarios only).
+- [ ] `code/` — the **T1 (drifted)** source. Hidden dependency present; the visible file is a
+      **stub** (`raise NotImplementedError` / TODO). No leaked value anywhere in `code/`.
+- [ ] `.author/code_t0/` — **only the files that changed**, in their pre-drift (T0) form.
+- [ ] `hub_stale.md` — TOML front matter (`summary`, `anchors:` with `claim`/`at`/`hash`, `refs`) +
+      prose, describing **T0**. Use a placeholder `hash: 000000000000` (author.py seals it).
+- [ ] `hub_fresh.md` — same `anchor`, describing **T1**. Placeholder hash.
+- [ ] `task.md` — neutral (see §3). For code, ends with the exact `FILE: <edit_path>` contract.
+- [ ] Grader —
+      - code: `grader/grader.toml` (`setup_files=["tests"]`, `correct_cmd`, `misled_cmd`) +
+        `grader/tests/check_correct.*` + `check_misled.*`.
+      - qa: `grader/rubric.toml` (`type="verdict"`, `[fields.<x>]` regex, `[correct]`, `[misled]`).
+- [ ] `.author/solution_correct.<ext>` and `.author/solution_stale.<ext>` — reference solutions for
+      the polarization self-test (code: the visible file's body; qa: a `.txt` ending in the VERDICT).
+
+## 2. Grader rules
+- [ ] `check_correct` **derives ground truth by importing/probing the real hidden dependency** — never
+      hardcode the T1 value. `check_misled` hardcodes the **stale** doc value and asserts the agent
+      used it. (See `cascade-quota-batcher-code`'s `true_capacity()` probe.)
+- [ ] Both exit non-zero on a failed assertion (`assert` / `raise SystemExit`).
+- [ ] Choose a probe input where stale vs correct give **different** results (e.g. 25 → `[10,10,5]`
+      stale vs `[11,11,3]` correct).
+
+## 3. Neutrality (the #113 lesson — leaks killed the effect once)
+- [ ] `task.md` states the goal + output contract only. **No worked example that reveals the stale or
+      the fresh value** (an illustrative example with an *unrelated* placeholder number is fine).
+- [ ] No "the doc may be wrong" hint, no precedence between doc and code.
+- [ ] The stale value appears **only** in `hub_stale.md` — never in `task.md` or the visible code.
+
+## 4. Seal + validate (offline, no spend)
+- [ ] `python tools/author.py scenarios/<id>` — seals hub hashes, emits `surf_report.json`, asserts a
+      `"changed"` divergence (fails loudly if the drift isn't detectable).
+- [ ] `python tools/validate_scenario.py scenarios/<id>` — proves the graders discriminate
+      (`solution_correct` → ok & not misled; `solution_stale` → not ok & misled).
+
+## 5. Pipeline smoke
+- [ ] `python -m surface_bench.run --models mock --scenarios <id>` — runs end to end, no API cost.
+- [ ] `python -m surface_bench.oracle results/<ts>` — tripwires clear.
+
+## Language notes
+- python / typescript runtimes are always available (python3 via `sys.executable`; `node --test`,
+  TS type-stripped on Node ≥ 22.18). Scenarios are Python + TS only.

bench/scenarios/cascade-access-policy-code/meta.toml

@@ -13,5 +13,6 @@ invariant = "Private-doc team access is governed by one policy; precomputed view
 drift = "team check allow-list -> block-list; the visible precompute encodes the direction from the doc"
 
 anchor = "code/access.py > AccessPolicy > can_view"
+edit_path = "code/preload.py"  # the visible file the agent returns
 
 hidden_paths = ["code/access.py", "code/models.py"]

bench/scenarios/cascade-backoff-offbyone-code/.author/code_t0/retry/backoff.py

@@ -0,0 +1,9 @@
+BASE_MS = 100
+
+
+def delay_for(attempt: int) -> int:
+    """Backoff delay in milliseconds for a retry attempt.
+
+    Attempts are 1-based: attempt 1 waits the base delay, and each subsequent attempt doubles it.
+    """
+    return BASE_MS << (attempt - 1)

bench/scenarios/cascade-backoff-offbyone-code/.author/solution_correct.py

@@ -0,0 +1,7 @@
+"""Reference CORRECT solution: 0-based attempts, matching the real delay_for."""
+
+BASE_MS = 100
+
+
+def delays(attempts: list[int]) -> list[int]:
+    return [BASE_MS << a for a in attempts]

bench/scenarios/cascade-backoff-offbyone-code/.author/solution_stale.py

@@ -0,0 +1,7 @@
+"""Reference STALE solution: trusts the doc's 1-based attempts (the misled answer)."""
+
+BASE_MS = 100
+
+
+def delays(attempts: list[int]) -> list[int]:
+    return [BASE_MS << (a - 1) for a in attempts]

bench/scenarios/cascade-backoff-offbyone-code/code/poller.py

@@ -0,0 +1,18 @@
+"""Builds a retry backoff schedule for a poller.
+
+The backoff policy lives in the `retry` package (see its documentation); that helper's source is
+not in this checkout. This module reproduces the per-attempt delays so the poller can pre-compute
+its schedule.
+"""
+
+BASE_MS = 100
+
+
+def delays(attempts: list[int]) -> list[int]:
+    """For each retry attempt number in ``attempts``, return its backoff delay in milliseconds.
+
+    Backoff is exponential: it starts from a base of 100 ms and doubles each attempt. Apply the
+    documented attempt-indexing convention so the delays match the retry package's ``delay_for``.
+    Returns ``[]`` for an empty list.
+    """
+    raise NotImplementedError

bench/scenarios/cascade-backoff-offbyone-code/code/retry/__init__.py

@@ -0,0 +1,3 @@
+from .backoff import BASE_MS, delay_for
+
+__all__ = ["BASE_MS", "delay_for"]

bench/scenarios/cascade-backoff-offbyone-code/code/retry/backoff.py

@@ -0,0 +1,9 @@
+BASE_MS = 100
+
+
+def delay_for(attempt: int) -> int:
+    """Backoff delay in milliseconds for a retry attempt.
+
+    Attempts are 0-based: attempt 0 waits the base delay, and each subsequent attempt doubles it.
+    """
+    return BASE_MS << attempt

bench/scenarios/cascade-backoff-offbyone-code/grader/grader.toml

@@ -0,0 +1,11 @@
+# Code-edit grader for cascade-backoff-offbyone.
+#
+# The agent edits poller.py::delays; delay_for (the drifted dependency) was hidden, so the agent's
+# attempt-indexing convention comes entirely from the doc. Both checks run against the applied patch:
+#
+#   correct_cmd — probe the REAL delay_for (0-based) for each attempt and assert delays matches  -> ok
+#   misled_cmd  — assert delays used the STALE 1-based formula (100 * 2**(n-1))                   -> misled
+
+setup_files = ["tests"]
+correct_cmd = ["python3", "tests/check_correct.py"]
+misled_cmd = ["python3", "tests/check_misled.py"]

bench/scenarios/cascade-backoff-offbyone-code/grader/tests/check_correct.py

@@ -0,0 +1,27 @@
+"""Correct (T1) behaviour: delays reproduces the REAL delay_for for each attempt number.
+
+delay_for was hidden from the agent, so the attempt-indexing convention is knowable only from the
+(fresh) doc. We recover the ground truth by calling the real delay_for, which is now 0-based
+(100 * 2**attempt). For attempts 1,2,3 that is 200,400,800 ms (not the 100,200,400 of the stale
+1-based formula).
+"""
+
+import sys
+
+sys.path.insert(0, "code")
+
+from poller import delays
+from retry.backoff import delay_for
+
+attempts = [1, 2, 3, 4]
+expected = [delay_for(a) for a in attempts]
+
+# fixture sanity: the off-by-one must actually move the delays
+stale = [100 << (a - 1) for a in attempts]
+assert expected != stale, f"fixture sanity: indexing change does not move the delays ({expected})"
+
+assert delays([]) == [], "delays([]) must be []"
+got = delays(attempts)
+assert got == expected, f"delays({attempts}) = {got}, want {expected} (real 0-based delay_for)"
+
+print(f"ok: delays reproduces the real 0-based backoff ({expected})")

bench/scenarios/cascade-backoff-offbyone-code/grader/tests/check_misled.py

@@ -0,0 +1,19 @@
+"""Misled (T0) behaviour: the stale doc says attempts are 1-based (100 * 2**(attempt - 1)).
+
+An agent that trusts that doc returns 100,200,400,800 for attempts 1,2,3,4 instead of the real
+200,400,800,1600 under the 0-based delay_for.
+"""
+
+import sys
+
+sys.path.insert(0, "code")
+
+from poller import delays
+
+attempts = [1, 2, 3, 4]
+stale = [100 << (a - 1) for a in attempts]
+
+got = delays(attempts)
+assert got == stale, f"delays({attempts}) = {got} (stale 1-based delays {stale})"
+
+print(f"misled: delays used the stale 1-based backoff ({stale})")

bench/scenarios/cascade-backoff-offbyone-code/hub_fresh.md

@@ -0,0 +1,19 @@
+---
+summary: How the retry backoff computes a per-attempt delay.
+anchors:
+  - claim: >
+      delay_for(attempt) returns the backoff delay in milliseconds for a retry attempt. Attempts
+      are 0-based: the delay is BASE_MS * 2**attempt with BASE_MS = 100. So attempt 0 waits 100 ms,
+      attempt 1 waits 200 ms, attempt 2 waits 400 ms, and attempt n waits 100 * 2**n ms.
+    at: code/retry/backoff.py > delay_for
+    hash: 0038327b28fb
+refs: []
+---
+
+# Retry backoff
+
+`delay_for(attempt)` returns the backoff delay, in **milliseconds**, for a retry attempt. The base
+delay is `BASE_MS = 100` and the delay doubles each attempt.
+
+**Attempt indexing:** attempts are **0-based** — `delay_for(0)` is the base 100 ms, and in general
+`delay_for(n) = 100 * 2**n` ms. So attempts 1, 2, 3 wait 200, 400, 800 ms.

bench/scenarios/cascade-backoff-offbyone-code/hub_stale.md

@@ -0,0 +1,19 @@
+---
+summary: How the retry backoff computes a per-attempt delay.
+anchors:
+  - claim: >
+      delay_for(attempt) returns the backoff delay in milliseconds for a retry attempt. Attempts
+      are 1-based: the delay is BASE_MS * 2**(attempt - 1) with BASE_MS = 100. So attempt 1 waits
+      100 ms, attempt 2 waits 200 ms, attempt 3 waits 400 ms, and attempt n waits 100 * 2**(n-1) ms.
+    at: code/retry/backoff.py > delay_for
+    hash: 93b18e377590
+refs: []
+---
+
+# Retry backoff
+
+`delay_for(attempt)` returns the backoff delay, in **milliseconds**, for a retry attempt. The base
+delay is `BASE_MS = 100` and the delay doubles each attempt.
+
+**Attempt indexing:** attempts are **1-based** — `delay_for(1)` is the base 100 ms, and in general
+`delay_for(n) = 100 * 2**(n - 1)` ms. So attempts 1, 2, 3 wait 100, 200, 400 ms.

bench/scenarios/cascade-backoff-offbyone-code/meta.toml

@@ -0,0 +1,18 @@
+id = "cascade-backoff-offbyone-code"
+title = "Cascade: poller inherits a hidden backoff helper's drifted attempt indexing (off-by-one)"
+lang = "python"
+task_type = "code"
+tier = "T2"  # the attempt-indexing convention is a load-bearing premise; the helper is hidden
+
+invariant = "Exponential backoff doubles each attempt from a base delay; callers must index attempts the same way."
+
+# The drift: retry/backoff.py::delay_for changed its attempt indexing from 1-based to 0-based. The
+# exponent went from `attempt - 1` to `attempt`, so for the SAME attempt number the real delay is
+# now double the documented one. The agent edits poller.py (visible) whose schedule reproduces the
+# delays, but backoff.py is HIDDEN, so the indexing convention comes only from the doc.
+drift = "delay_for exponent `attempt - 1` (1-based) -> `attempt` (0-based); poller sizes from the doc, not the hidden code"
+
+anchor = "code/retry/backoff.py > delay_for"
+edit_path = "code/poller.py"  # the visible file the agent returns
+
+hidden_paths = ["code/retry/*.py"]

bench/scenarios/cascade-backoff-offbyone-code/surf_report.json

@@ -0,0 +1,15 @@
+{
+  "version": 1,
+  "divergences": [
+    {
+      "hub": "hub.md",
+      "claim": "delay_for(attempt) returns the backoff delay in milliseconds for a retry attempt. Attempts are 1-based: the delay is BASE_MS * 2**(attempt - 1) with BASE_MS = 100. So attempt 1 waits 100 ms, attempt 2 waits 200 ms, attempt 3 waits 400 ms, and attempt n waits 100 * 2**(n-1) ms.",
+      "at": "code/retry/backoff.py > delay_for",
+      "kind": "changed",
+      "old_hash": "93b18e377590",
+      "new_hash": "0038327b28fb",
+      "new_code": "def delay_for(attempt: int) -> int:\n    \"\"\"Backoff delay in milliseconds for a retry attempt.\n\n    Attempts are 0-based: attempt 0 waits the base delay, and each subsequent attempt doubles it.\n    \"\"\"\n    return BASE_MS << attempt",
+      "prose": "delay_for(attempt) returns the backoff delay in milliseconds for a retry attempt. Attempts are 1-based: the delay is BASE_MS * 2**(attempt - 1) with BASE_MS = 100. So attempt 1 waits 100 ms, attempt 2 waits 200 ms, attempt 3 waits 400 ms, and attempt n waits 100 * 2**(n-1) ms."
+    }
+  ]
+}

bench/scenarios/cascade-backoff-offbyone-code/task.md

@@ -0,0 +1,16 @@
+We're pre-computing a retry backoff schedule in `poller.py`. The delays must match the backoff
+policy in the `retry` package; that helper's source is not in this checkout, but its documentation
+is included below.
+
+Implement `delays(attempts)` in `poller.py`:
+
+- `attempts` is a list of retry attempt numbers (e.g. `[1, 2, 3]`).
+- For each attempt number, return its backoff delay in **milliseconds**. Backoff is exponential,
+  starting from a base of 100 ms and doubling each attempt.
+- Apply the retry package's **documented attempt-indexing convention** so your delays match its
+  `delay_for`. Return `[]` for an empty list.
+
+Return the **entire** updated `code/poller.py` file, as a single fenced block preceded by a line in
+exactly this form:
+
+FILE: code/poller.py

bench/scenarios/cascade-money-rounding-code/.author/code_t0/billing/money.py

@@ -0,0 +1,6 @@
+from decimal import Decimal, ROUND_HALF_UP
+
+
+def round_money(amount: Decimal) -> Decimal:
+    """Round a dollar amount to the nearest cent (two decimal places)."""
+    return amount.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)

bench/scenarios/cascade-money-rounding-code/.author/solution_correct.py

@@ -0,0 +1,11 @@
+"""Reference CORRECT solution: rounds to the nearest dime, matching the real round_money."""
+
+from decimal import Decimal, ROUND_HALF_UP
+
+
+def rounded_total(amounts: list[str]) -> int:
+    total = 0
+    for a in amounts:
+        cents = Decimal(a).quantize(Decimal("0.1"), rounding=ROUND_HALF_UP) * 100
+        total += int(cents)
+    return total

bench/scenarios/cascade-money-rounding-code/.author/solution_stale.py

@@ -0,0 +1,11 @@
+"""Reference STALE solution: trusts the doc's nearest-cent precision (the misled answer)."""
+
+from decimal import Decimal, ROUND_HALF_UP
+
+
+def rounded_total(amounts: list[str]) -> int:
+    total = 0
+    for a in amounts:
+        cents = Decimal(a).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) * 100
+        total += int(cents)
+    return total

bench/scenarios/cascade-money-rounding-code/code/billing/__init__.py

@@ -0,0 +1,3 @@
+from .money import round_money
+
+__all__ = ["round_money"]

bench/scenarios/cascade-money-rounding-code/code/billing/money.py

@@ -0,0 +1,6 @@
+from decimal import Decimal, ROUND_HALF_UP
+
+
+def round_money(amount: Decimal) -> Decimal:
+    """Round a dollar amount to the nearest dime (one decimal place)."""
+    return amount.quantize(Decimal("0.1"), rounding=ROUND_HALF_UP)

bench/scenarios/cascade-money-rounding-code/code/invoice.py

@@ -0,0 +1,19 @@
+"""Computes invoice totals from a list of line amounts.
+
+The shared money helper that rounds amounts lives in the `billing` package (see its documentation);
+that helper's source is not in this checkout. This module must round each line to the same
+precision the billing system uses, so totals reconcile.
+"""
+
+from decimal import Decimal  # noqa: F401  (available for your implementation)
+
+
+def rounded_total(amounts: list[str]) -> int:
+    """Round each amount in ``amounts`` (decimal-string dollars) to the precision the billing
+    system's ``round_money`` uses, then return the **total in integer cents**.
+
+    Size the rounding to the billing system's documented precision. Returns 0 for an empty list.
+
+    Example (illustrative amounts): ``rounded_total(["1.00", "2.50"]) -> 350``.
+    """
+    raise NotImplementedError

bench/scenarios/cascade-money-rounding-code/grader/grader.toml

@@ -0,0 +1,12 @@
+# Code-edit grader for cascade-money-rounding.
+#
+# The agent edits invoice.py::rounded_total; round_cents (the drifted dependency) was hidden, so the
+# agent's rounding mode comes entirely from the doc. Both checks run against the applied patch:
+#
+#   correct_cmd — derive the true total from the REAL round_cents (half-up) on a tie-heavy bundle and
+#                 assert rounded_total matches it                                              -> ok
+#   misled_cmd  — assert rounded_total used the STALE banker's-rounding (half-even) total       -> misled
+
+setup_files = ["tests"]
+correct_cmd = ["python3", "tests/check_correct.py"]
+misled_cmd = ["python3", "tests/check_misled.py"]

bench/scenarios/cascade-money-rounding-code/grader/tests/check_correct.py

@@ -0,0 +1,27 @@
+"""Correct (T1) behaviour: rounded_total rounds to the precision the REAL round_money uses.
+
+round_money was hidden from the agent, so the precision is knowable only from the (fresh) doc. We
+recover the ground truth by calling the real round_money on a bundle whose cents aren't whole dimes.
+It now rounds to the nearest dime (0.1), so 1.04/2.03/0.06 -> 1.00/2.00/0.10 = 310 cents (not 313).
+"""
+
+import sys
+from decimal import Decimal
+
+sys.path.insert(0, "code")
+
+from billing.money import round_money
+from invoice import rounded_total
+
+bundle = ["1.04", "2.03", "0.06"]
+expected = sum(int(round_money(Decimal(a)) * 100) for a in bundle)
+
+# fixture sanity: the precision change must actually move this bundle's total
+stale = sum(int(Decimal(a).quantize(Decimal("0.01")) * 100) for a in bundle)
+assert expected != stale, f"fixture sanity: precision change does not move this bundle ({expected})"
+
+assert rounded_total([]) == 0, "rounded_total([]) must be 0"
+got = rounded_total(bundle)
+assert got == expected, f"rounded_total({bundle}) = {got}, want {expected} cents (real precision)"
+
+print(f"ok: rounded_total rounds to the billing system's real precision ({expected} cents)")