abproof

Oracle / evidence plane · Stable — feature-complete; maintenance only. The scope is finished, not abandoned. See the component map for how this fits the rest.

Offline A/B change-validation for an agentic coding harness — stat-gated, seed-blocked, reusing the executor as the arm.

abproof answers one question: did this change to your agent setup actually make it better? It runs the same executor twice — baseline vs. treatment — over a corpus of RED-test-gated tasks, with seed-blocked pairing, task-typed scoring, and a gate that only fails a run when the regression is both beyond tolerance and statistically significant (paired Wilcoxon, α = 0.05 by default) — not a bare point estimate. A worse-but-noisy result honestly exits PASS rather than failing on chance; a run that could not have reached α exits UNDERPOWERED rather than borrowing that PASS. The node is the unit of replication, so power comes from a larger battery — reps only sharpen each node's rate.

Unlike a prompt-eval framework, abproof A/Bs the whole assembly running a real loop, not a single model call — the executor is the arm.

Part of the Barnett Studios agentic-harness toolkit → cxpak · commitward · abproof · …

Install

brew tap Barnett-Studios/tap && brew install abproof   # macOS/Linux
cargo install abproof                                   # any platform
docker run --rm -v "$PWD:/repo" ghcr.io/barnett-studios/abproof run experiment.yaml --dry-run

Run

# project the cost/shape of an experiment (no execution)
abproof run experiment.yaml --dry-run

# execute the A/B (bounded), print the R-table + gate verdict
abproof run experiment.yaml --confirm --max-cost 5.00 --max-calls 200

For programmatic consumers, abproof run-json speaks the ADR-0052 response envelope: a JSON request on stdin ({manifest_yaml, baseline_json?, dry_run?, max_cost?, max_calls?}), a {schema_version, status, body} envelope on stdout, exit 0 (the verdict — and any abort — is carried in the envelope). dry_run: true projects without executing (no baseline/driver/network). An aborted experiment is reported as status: error so a consumer falls open rather than trusting an invalid PASS.

echo '{"manifest_yaml":"...","dry_run":true}' | abproof run-json

abproof needs two things at run time. A standalone install sets these via env; an in-tree/dev checkout resolves them by walking up from the CWD.

Env	What	Resolution
ABPROOF_CORPUS	the RED-baseline corpus dir	standalone: point at the corpus repo's red-baseline/; dev checkout: walked up as measurement/corpus/red-baseline
ABPROOF_EXECUTE_NODE	the execute-node loop (execute_node.py)	standalone: your executor's execute_node.py; dev checkout: walked up as skills/execute-node/execute_node.py
ABPROOF_RESULTS	where --out-less results are written	./measurement/experiments (any writable dir)

The corpus is a separate component (the Corpus slot) — abproof ships none. The reference RED-baseline corpus is Exercism-derived and must be license-scrubbed + attributed before redistribution (that is the Corpus component's job, not abproof's).

The manifest

An experiment is a YAML manifest: {name, battery:[task-ids], reps, seed_base, baseline:{loop, model, context, backend}, treatment:{…}, metrics:{…}, tolerance:{…}}. Baseline outcomes live beside it as <stem>.baseline.json. abproof run --dry-run prints the projected loop-runs, judge-calls, minutes, and claude-cli calls before you spend anything.

Exit codes (fail-loud on measurement integrity)

Code	Meaning
0	projection/dry-run printed, or the gate PASSED
1	setup error (bad manifest, missing baseline, unreadable corpus)
3	experiment aborted — an invalid measurement (local runtime unavailable, cost cap hit mid-battery); never presented as a result
4	UNDERPOWERED — the battery ran and is internally valid, but alpha was unreachable, so it could not have failed its own gate. Not a PASS
gate	on --confirm, the process exits with the statistical gate's own code (non-zero = FAIL)
64	usage error

Underpowered is not a pass

With n discordant (non-zero) paired deltas, the exact two-sided sign-flip test has a hard floor of 2/2ⁿ: only the all-positive and all-negative assignments reach the extreme, out of 2ⁿ. So α = 0.05 is unreachable at n ≤ 5 (floor 0.0625) and reachable from n = 6 (floor 0.03125). A battery below that threshold cannot fail its own gate whatever the data say — reporting PASS there means "we couldn't have found a regression", not "we looked and found none". abproof reports such a run as UNDERPOWERED with its own exit code.

The guard is deliberately direction-blind: a battery with no power to detect a regression did not establish its absence just because the point estimate happened to improve. Every result carries n_discordant and min_attainable_p so the power denominator is never hidden — that is how underpowered nulls get misread as evidence of no effect. Power comes from more discordant nodes, not more reps on nodes whose arms already agree.

abproof is deliberately fail-loud, not fail-open: an offline oracle that silently returned a green verdict on a broken run would be worse than useless. (It still stays out of the way — offline, never in the live agent loop; "absent abproof" simply means your setup goes unmeasured.)

See CONTRACT.md for the full interface.

License

Licensed under either of MIT or Apache-2.0 at your option. Unless you explicitly state otherwise, any contribution you intentionally submit for inclusion in the work shall be dual-licensed as above, without any additional terms.

---

Built by Barnett Studios — part of the agentic-harness toolkit: cxpak · commitward · cascadr · abproof · cordon · slicr.