Real judging and real test cases: structured verdicts, verified evidence, interface grounding

[barancan]

Phases P2 and P3. Judges now produce schema-forced verdicts whose evidence is verified, and generated pipelines carry test cases with real inputs grounded on the target's interface.

471 tests passing with zero API keys set, up from 273. Verified across repeated full-suite runs and against a real server.

Why

After P0/P1 the builder called a real model, but two things still made its output weaker than it looked. Judges asked for JSON in prose and scraped the reply, storing evidence quotes without ever checking them — a judge could invent a supporting quote and nothing noticed. And every generated test case had an empty input, so a pipeline invoked the target with nothing in it. A pipeline whose cases have no inputs is not testing anything.

Real judging (P2)

• Structured output is forced natively per provider — Anthropic via a required emit_verdict tool, openai_compat via json_schema (retrying as json_object on a 400, since some gateways reject it), Ollama via format. Validated with jsonschema; a reply that does not conform is an error rather than unvalidated data passed downstream. A text-only reply under a schema is no longer something to scrape.
• Evidence is verified. With require_evidence, a quote that does not appear in the response fails the check. Matching folds case, whitespace and typographic look-alikes and honours ... elision, so a re-cased or re-wrapped span still passes and only fabrication fails.
• Self-consistency takes the median score per dimension across N temperature-0 samples, flooring an even split — a judge torn between 1 and 2 has not agreed the dimension is met — and records the spread so a reviewer can see disagreement.
• An unscored dimension now fails by name instead of silently reading 0.
• Rubrics are generated, not templated: at least two anchored dimensions with observable 0/1/2 levels. The validator rejects grading-word anchors like "the response is very good", and constrains dimension ids to slugs — they become filenames, and this codebase has been bitten by path traversal before.

Real test cases (P3)

• generator/interface.py parses Postman, OpenAPI 3 (JSON and YAML, local $refs resolved) and MCP tool schemas into request fields, a response schema and JSONPath response paths. adapters/rest.py now uses the same reader, so the adapter and the builder cannot drift into disagreeing about what a collection says. Format detection is by content, not extension.
• generator/casegen.py produces concrete inputs per intent, grounded on the interface's real request fields, with nominal, empty, boundary and hostile variants — an eval that only tests the happy path is not an eval. A single gate in build.py means no path can emit a case with an empty input.
• Golden datasets bind via assay generate --dataset, taking precedence over generation; a malformed row names file and line.
• Ids are validated rather than sanitised: a ../ id means the model is confused, not that the id needs cleaning.

Also in this PR

• Requirement coverage in both directions. Counting cases per requirement only describes what was tested. A requirement with nothing testing it was invisible — exactly what a reviewer needs before signing off. Now reports uncovered requirements by id and text, plus orphan tests citing requirements that no longer exist. When the requirement list is unavailable it says so rather than rendering "0 uncovered" and implying full coverage.
• TargetModel.interface_hash is populated. The column has existed since the first schema and was never written, so a report could not tell you whether the interface had changed underneath it.
• A flaky-test fix that was a real bug. Background runs resolve the session factory from module globals on every DB touch, so a run still in flight when the store was reconfigured wrote somewhere unexpected. wait_for_runs() gives callers something to wait on; the same hazard applies to graceful shutdown.

Decisions worth review

• A missing interface file is now an error; an unreadable one is not. The two halves of P3 disagreed here, and both had a point. Parsing stays forgiving so a bad document cannot take a build down, but a path the user explicitly supplied that does not exist is checked at the boundary — silently building an ungrounded pipeline would hand them precisely what they asked not to have.
• The web path no longer produces weaker rubrics than the CLI. /pipelines/generate called rubric_for() without a model, so the wizard quietly got deterministic fallbacks even with a builder model configured.
• A run now costs more. Case generation is one model call per intent, and n=3 means three target invocations per intent instead of one empty-input case. That is the point of the phase, but it is a real change in what a run costs — and cost capture (P5) is not built yet, so runs still report zero spend.

Verification

• pytest -q — 471 passing with ANTHROPIC_API_KEY and OPENAI_API_KEY explicitly unset. No test makes a network call. Run repeatedly to confirm the threading flakiness is gone.
• End to end offline: assay generate --offline --interface api.postman_collection.json produced 9 cases across 3 requirement suites with 0 empty inputs, using the collection's real text/locale fields, including an injection probe.
• Against a real server with no keys: generate returns a 422 naming the variable, a missing interface path returns 422, the wizard carries the interface field, no tracebacks.

Not in this PR

Codegen (P4) still does not exist — an intent routed to generated produces a spec entry with no source behind it. That is now the top open item in the ranked-gap table and the last piece of the product's core claim. Cost capture (P5) and retiring the mock default (P6) also remain.

---

Generated by Claude Code