Real LLMs on every build path, honest docs, and three bugs behind shipped UI

[barancan]

Moves Assay off mocks and heuristics on the paths that matter, makes the docs match the code, and fixes three bugs that were reachable from the shipped UI.

273 tests passing with zero API keys set. Baseline was 149.

Why

A status audit found the runner half production-shaped and the builder half a v0 heuristic: the web UI never called an LLM, codegen did not exist, rubrics were templated, and mock adapters were the silent default. Docs claimed capabilities that were not there. This PR is the first three phases of closing that.

Bugs fixed (all were live in the UI)

• Judge rubrics were never materialised at run time. _resolve_spec returned only generated sources, so judges/judge.py hit FileNotFoundError and aborted the entire run. Any DB pipeline with a judge check was dead. Reproduced first, then fixed.
• The sandbox was not a sandbox at module level. The import allowlist and the removal of open/exec/eval/compile were installed after exec_module(). Verified as a real escape, not a theoretical one: against the old code a generated check's top-level open(path,'w').write(...) succeeded and the file existed. The worker now compiles the module while those builtins are available, then locks down before running any user code. A second bug found in passing: the socket patch ran after the import guard, so import socket was blocked and the patch was silently skipped.
• "Regenerate check" emitted code the sandbox could never load — wrong signature, return True instead of a dict. Now emits a contract-correct scaffold that fails with an explicit message until codegen lands.

Real LLMs (P0 + P1)

• New assay/llm/provider.py: one entry point for which model, which key, is it configured. Keys stay in the environment, referenced by name, never persisted and never rendered. An unconfigured provider raises LLMConfigError carrying the adapter and the variable to set.
• key_env now reaches the adapters. It was collected by the wizard and silently discarded, because the spec models did not declare it and pydantic ignored extras. openai_compat no longer hardcodes OPENAI_API_KEY, and judges get endpoint/key_env/params instead of only model — so a judge can point at a local vLLM.
• Both build routes call a real model. /pipelines/preview and /pipelines/generate resolve the configured builder model. No key produces a 422 naming the exact variable, never a silent heuristic fallback. assay generate --offline is the explicit opt-in to the old keyword path.
• Real requirement traceability. generator/ingest.py splits requirements into R1…Rn; the prompt carries those ids, every returned ref is resolved against them, and each requirement becomes its own suite. requirement_ref: "auto" is gone.
• The connection badge stopped lying. It reported a green "Connected" when there was no API key at all, because a missing-credential exception matched a "reachable" heuristic. Ping now distinguishes unreachable from unauthenticated and names the missing variable.
• The judge system prompt now actually reaches the model. It was passed as params["system"] and read by no adapter, so judge verdicts were being requested without their instructions.
• Settings gains a Providers card: per-adapter env var name and a configured badge. Names only, never values.

Run progress

A run blocked the HTTP request until every case finished — invisible against mocks, minutes of a dead button against real models. execute_run is split into setup and execution; setup stays on the calling thread so an unreachable target still raises synchronously, and only the case loop is deferred. Browsers get a polling progress view; programmatic callers keep synchronous semantics because CI depends on the response carrying report_id. Case results now commit per case, which is what makes progress observable and means a run that dies half way keeps what it produced.

Docs

docs/STATUS.md is now the source of truth: a built/partial/planned matrix over every subsystem. README's false claims are corrected rather than softened — no MCP/SDK targets, no custom adapter, no OpenAPI import, and the sandbox section says plainly there is no filesystem jail and no egress block. docs/user-journeys.md maps twelve builder journeys to UI touchpoint, route, business logic and state effect. The stale UI sprint plan is archived with a note on what actually shipped.

tests/test_docs_truth.py enforces the mechanically checkable claims: README adapter tables must match the registry, documented CLI verbs must exist or be declared planned, and a verb listed as planned must not already be implemented.

Behaviour changes worth review

• TargetSpec/JudgeSpec are now extra="forbid". Silently dropping an undeclared field is exactly what caused the key_env bug. A hand-written assay.yaml with an unknown key under target: now fails loudly at load. Spec itself is unchanged, so spec_dict["requirements"] still round-trips.
• /pipelines/preview now requires identity. It calls a real model on every request, so unauthenticated it was an open spend vector. No change in open mode; 401 in enforced mode.
• key_env: "" means "this target takes no credential" — the escape hatch for a keyless local vLLM or LM Studio, which would otherwise have been forced to invent an API key.
• SQLite now opens with check_same_thread=False so background runs can write, and the hand-rolled migrations are dialect-aware (they emitted SQLite-only DDL while the README documents a Postgres switch).

Verification

• pytest -q — 273 passing, verified with ANTHROPIC_API_KEY and OPENAI_API_KEY explicitly unset. No test makes a network call.
• Run progress verified against a real uvicorn server, not just TestClient, since threading plus SQLite is where TestClient can mislead: run → progress page → poll redirects to the report → ready_for_review, zero server errors.
• No-key behaviour verified against a real server: both build routes return an actionable 422, Settings renders the Providers card with no value leakage, the wizard carries the error banner.
• CLI verified: bare generate without a key explains the three ways forward, a malformed --judge gives an example, --offline works and emits real R1/R2 refs.

Not in this PR

Codegen (P4) still does not exist, case inputs are still empty, the target interface is still not parsed at build time, and mock adapters are still selectable as ordinary targets (P6). docs/STATUS.md marks all of these planned, and the ranked-gap table in docs/user-journeys.md tracks them with the phase that closes each.

---

Generated by Claude Code