Critical audit: contract integrity, statistics, dead machinery, API drift, GEPA bridge, test quality

[drewstone]

Charter

Perform the most rigorous critical review of agent-eval ever done on this repo: every claim below must end with a receipt (file:line, grep count, test run, or measured number) and a verdict — CONFIRMED-FINE / DEFECT (severity + fix) / DEAD (delete PR) / UNVERIFIABLE (say why).
Work the sections in order; each produces findings ranked by severity and, where mechanical, a fix PR.
Do not soften findings.
A finding without a receipt does not ship.

Context anchors (found 2026-07-23 during live improvement-loop work in agent-runtime/supervisor-lab; verify each independently, do not trust this list):

• Lineage DAG (src/campaign/lineage.ts, 844 LOC + 653 test LOC, PRs feat(campaign): Lineage DAG — git-graph of candidates, multi-parent merges, visioned tracks, agent-managed governor #319/feat(campaign): runLineageLoop — Lineage DAG live seams #322, released 0.106.3) has zero production usage in agent-eval, agent-runtime, or loops — built and never wired.
• The improvement loop that actually runs (agent-runtime agenticGenerator) uses none of this repo's proposer families; several families are handrolled sketches of published methods (GEPA/ACE/HALO/FAPO) now superseded by the real GEPA bridge (feat(campaign): add GEPA Omni recipe adapter #408/test(campaign): skip optional GEPA smoke when unavailable #409). A purge PR is in flight on chore/purge-handrolled-optimizers — audit what remains after it, don't duplicate it.
• Exactly one production Ax signature exists (analyst/finding-signature.ts: question:string -> findings:json[]); the rest of the optimization machinery is free-text prompts.
• agent-runtime's self-improving-supervisor.test.ts fails against agent-eval 0.123.0 with ctx.cost.observe is not a function — an API drift a consumer caught, not this repo's CI.
• Historical pattern of statistics defects caught late: median ship-stat suppressed a real +0.18 lift (fixed to mean, fix(gates): ship on the MEAN paired delta, not the median — ties were suppressing real lifts #316); single-rep pass-rates swing ±7.5pts at n=40; a "gate tie-domination" class of bugs.

Sections

1. Contract integrity (the honesty firewall)

The repo's core promise is that proposers cannot see or influence judge scores (judgeScores?: never in the SurfaceProposer contract) and holdout/final cases stay sealed.
Adversarially audit every path: campaign loop, presets, the new GEPA bridge, trace analysts, KB adapters.
Can any proposer-visible artifact (evidence bundle, trace text, prior-candidate metadata, error message) leak per-case judge outcomes or holdout case content?
Write a leak test per channel; the tests are the deliverable.

2. Statistics correctness

Every statistic the repo computes that can gate a promotion or mint a claim: pairing, bootstrap/CI code, sign tests, multiplicity handling (the rising-bar logic), tie handling, sequential/peeking behavior when campaigns resume, seed handling.
For each: is the math right (cite the formula), is it tested against a hand-computed fixture, and does any consumer misuse it (mean-vs-median class)?
Reproduce #316's bug class as a regression test if not already present.

3. Dead and unwired machinery

Full usage census: for every exported module/preset/family, grep agent-eval + agent-runtime + loops + any other consumer for real usage (not self-tests).
Lineage DAG, runLineageLoop, steering-optimizer, RL/active-curriculum, belief-state/shadow-probe, multi-layer-verifier, release-confidence — and everything else index.ts exports.
Verdict per module: WIRED (where), UNWIRED-BUT-STRATEGIC (who is the named future consumer; issue link), or DEAD (deletion PR).
"Built and never wired" is the repo's most documented failure mode — quantify it.

4. API surface and drift

Measure the public surface (exports count, index.ts size).
Find drift like ctx.cost.observe: run the newest agent-eval against agent-runtime's test suite (its biggest consumer) and inventory every breakage; propose the compat contract (what is public API vs internal, and how breaking changes get flagged before a consumer's CI finds them).

5. The GEPA bridge (#408/#409)

Fresh-eyes review of the adapter: does the final-cases-stay-inside boundary hold under adversarial inputs (what exactly crosses the Python boundary — audit the serialized payloads); is the source pin verifiable; what happens on GEPA version bump; are engine failures loud; is there any path where an inner-loop evaluation writes state the outer gate later trusts?

6. Test quality

Beyond count (~3,383): find tests that assert nothing (snapshot-only, no-throw-only), tests that mock the thing they claim to test, and the coverage of the paths that gate promotions (the campaign accept/reject decision itself — is there a test where a candidate SHOULD be rejected and is?).
The promotion-decision paths must have adversarial tests (fake improvement, missing rep, holdout regression, cost blowout) — write any that are missing.

7. Docs vs reality

Every README/docs claim about capabilities: does the code do it, is the example runnable verbatim on a fresh clone, do the doc'd APIs exist under the doc'd names?
(The runtime repo's audit found its 30-second quickstart broken; apply the same standard here.)

8. Release hygiene

Changelog accuracy vs actual diffs for the last 5 releases; version-bump discipline; whether published artifacts match the tagged source; CI gaps (the Python smoke is skip-by-default — what else is skip-by-default that shouldn't be).

Deliverables

1. Findings report as an issue comment: ranked by severity, each with receipt + verdict, in the report style (verdict first, tables, threats-to-validity).
2. Mechanical fixes as PRs (dead-code deletions, missing regression tests, leak tests), each small and independently mergeable.
3. A follow-up issue per non-mechanical defect with a concrete fix design.
4. The one-page summary: the 5 things most likely to produce a wrong promotion decision or a false public claim, in plain language.

Non-goals

• No rewrites or architecture changes — findings and surgical fixes only.
• Do not touch the in-flight purge branch (chore/purge-handrolled-optimizers) or re-litigate it.
• Do not audit consumers' code (agent-runtime/loops) beyond what's needed to prove usage/drift claims.

Evidence standard

file:line for code claims; command + output for behavior claims; test name for coverage claims; PR/commit for history claims.
Any section where the evidence cannot be obtained gets UNVERIFIABLE + the reason, never silence.