Codegen, cost capture, no more mocks — plus a hardened sandbox and an authentication hole closed

[barancan]

Completes every phase of the original market-ready plan: the builder now writes Python checks, runs report what they cost, and mock adapters no longer resolve outside tests. Generated code runs behind a genuinely hardened boundary, and an authentication hole that made the approval gate decorative is closed.

704 tests passing with zero API keys set, up from 499.

The sandbox first, deliberately

Codegen means model-written Python executes by default, so the containment boundary was built before any of it. Four layers added:

• Empty environment. The sandbox subprocess was inheriting the parent's entire environment, including ANTHROPIC_API_KEY. os was import-blocked so it wasn't directly reachable, but shipping model-written code with the keys sitting in its environment wasn't defensible.
• Throwaway working directory — the child no longer inherits the repo.
• A real network namespace. Where unprivileged unshare works, generated checks run with no network interfaces at all rather than monkeypatched socket factories. Verified against an actual connection attempt before being wired in.
• The child never needs open — the trusted parent reads the source and passes it in.

sandbox_tier() reports which layers are actually live, because the namespace needs Linux with user namespaces and isn't universal. The README says where it degrades rather than claiming a flat guarantee. The containment tests attempt the escape rather than asserting on internals.

Codegen (P4)

generator/codegen.py writes the check, then gates it three ways: a static AST check (contract, import allowlist, no __globals__/__subclasses__ crawling — caught as attribute access and as string constants), a sandboxed dry-run, and a discrimination test. That last one matters: a check returning passed: True on a sample only proves it runs. A check that passes everything is testing nothing.

Worth reading the discrimination design — the first implementation falsely rejected the repo's own worked example, because severity_monotonic is a conditional assertion legitimately satisfied by a response with no blocked findings. Rejecting the very check held up as a model is a bad gate, so it now tries degraded samples first and falls back to asking the check's author for a counter-example. A vacuous check still fails, because it passes its own counter-example too.

On failure after repairs, the intent degrades to a judge check with the reason recorded in build_meta.codegen_failures, and the review screen badges it — a reviewer can see a mechanical check was wanted and not achieved.

Cost capture (P5)

Per-case tokens and cost, with judge spend attributed to the case that caused it including every self-consistency sample. An unpriced model records NULL and renders as "unknown", never as $0.00 — free and unknown are different answers, and a silently wrong cost is worse than an absent one. Price table dated and overridable via ASSAY_PRICING_FILE; a malformed override raises rather than silently billing at list rates.

Model matching tolerates version decoration but refuses to cross families: claude-haiku-4-5-20251001 finds claude-haiku-4-5, but gpt-4o-mini never inherits gpt-4o's rate.

No more mocks (P6)

mock resolves only under ASSAY_ALLOW_MOCK=1 or an explicit --offline. Removed from the wizard. build.py no longer substitutes a mock judge when none is configured. Verified end to end: with no keys and no opt-in, generate refuses and run refuses, both with messages naming what to configure instead.

This surfaced two silent-mock paths nobody knew about: the wizard collected a judge model and never sent it, so every graded check the UI ever built was scored by the substituted mock judge. The CLI's default path had the same result.

An authentication hole

Writing the reviewer and CI journeys surfaced that X-Assay-User is a header anyone able to reach the port can send, and enforced mode accepted it alone. Naming a seeded reviewer was enough to approve a report — which made the approval gate decorative and contradicted the product's central claim that automation produces evidence while only a human produces a decision.

Enforced mode now requires a matching X-Assay-Token (ASSAY_API_TOKEN) alongside the header, and refuses the header outright when no token is configured. Session cookies are unchanged; open mode stays frictionless. Ten of the thirteen new tests fail against the previous code.

Two routes had no gate at all: POST /hooks/run (the only mutating route without identity, passing a caller-supplied path straight to load_spec — an unauthenticated arbitrary file read) and POST /settings/{judge,builder}.

Journeys and other fixes

docs/user-journeys.md gains reviewer, CI and admin actors (J13–J20), each row grounded in code, with a test asserting every route and symbol cited actually exists. Writing them found the bugs above plus:

• docker-compose lost every exported report. The volume mounts /root/.assay, ASSAY_HOME was never set, and WORKDIR is /app — so reports were written outside the volume and lost when the container was replaced. The database survived; the signed artifacts didn't.
• STATUS claimed a state-machine back-edge that doesn't exist. ready_for_review → pending is declared legal and never invoked, so a reviewer cannot return a report for rework. Now stated honestly rather than fixed, since adding it is a product decision.

Behaviour changes worth review

• Enforced-mode CI callers now need ASSAY_API_TOKEN. This breaks existing header-only automation, deliberately — the previous behaviour was the hole.
• assay generate requires --adapter unless --offline; it no longer defaults to mock.
• A run costs more and now says so. Case generation plus codegen means more model calls; the reports finally show it.

Verification

• pytest -q — 704 passing with provider keys explicitly unset. No test makes a network call.
• Codegen driven end to end with a scripted model: the AST gate catches wrong signatures, blocked imports and dunder crawling; the repair loop converges carrying the complaint; a vacuous check is rejected.
• The no-mocks claim walked by hand with no keys and no opt-in.
• Egress block verified by an actual connection attempt inside the namespace.

Not in this PR

Run history, pass-rate trends, regression detection, spec export from the UI, run-level gating enforcement, and seeding the first reviewer from the UI in enforced mode. All tracked in the ranked-gap table in docs/user-journeys.md.

---

Generated by Claude Code