AC-035/AC-035A: end-to-end benchmark baseline with verified provenance - #6
Merged
Merged
Conversation
Adds the first rigorous end-to-end operational and performance baseline for AuthContract, measured at e4e1a97 against only the capabilities that exist. New (no existing file modified): benchmarks/run_benchmarks.py - phase runner benchmarks/harness.py - timing, memory, environment, scale specimens benchmarks/specimens/ - declarative specimen table benchmarks/results/ - raw JSON output, committed for audit benchmarks/README.md - methodology and reproduction docs/BENCHMARKS.md - measured results and limitations docs/ROADMAP.md - NOW/NEXT/LATER/RESEARCH, each item evidence-cited docs/TRL-ASSESSMENT.md - TRL 4, with what would move it to 5 Correctness: 7/7 end-to-end specimens, 38/38 adversarial specimens, 342 existing regression tests. E2E-05 mutates and truncates every protected receipt field in turn (20 variants), all detected. Performance: end-to-end p50 701.6us, p95 1083.3us, p99 1149.9us; 1314 complete E2E transactions/sec single-process. Canonicalization (~96us) dominates; the authorization check itself is 8.4us. Verification costs about as much as deciding because it recomputes every binding rather than trusting the receipt. Scaling: linear in both measurable dimensions - declared actions 1..1000 and required facts 10..10000 (10k facts = ~0.57s, ~13MiB). Multi-contract corpora, concurrency, and persistence are recorded NOT EVALUATED rather than estimated, because no such path exists at this commit. Determinism: 100 replays produce byte-identical receipts across all 10 protected fields. decision_time is stable because it binds to the fact bundle's declared now, not wall clock, so there is no intentionally-varying receipt field. Findings recorded, not repaired during the measurement run: F1 admission approvals are bound as evidence but do not gate authorization F2 canonicalization is repeated several times per transaction F3 no replay protection semantics exist (determinism is not protection) F4 the *_mutated.json fixtures are mutated AND re-sealed, so they are validly-bound variants; the first draft of this suite mis-expected them to refuse. The genuine stale-binding attack is built programmatically and is correctly refused with AC_DIGEST. Claim ceiling preserved: one synthetic specimen family, one machine, one process. No production-readiness, regulatory, security-certification, distributed-scalability, or comparative claim.
Two benchmark-method defects, no product change. A. Provenance. The harness is introduced by a later commit than the implementation it measures, so "which commit was benchmarked" could not be answered by HEAD alone, and the prior reproduction instructions told users to check out a commit at which benchmarks/run_benchmarks.py does not exist. Results now record DUT_BASE_SHA (the AuthContract implementation measured) and BENCHMARK_HARNESS_SHA (the commit containing the harness) as distinct values. verify_dut_unchanged() diffs the device-under-test paths against DUT_BASE_SHA before any measurement runs and refuses to proceed on drift, so the two SHAs are provably comparable rather than merely asserted. B. Sustained throughput. The prior rates were reciprocals of mean latency. That is arithmetic, not measurement: it assumes zero loop overhead and no drift under continuous operation. sustained_throughput() now runs a continuous single-threaded loop over a fixed wall-clock window and counts completed operations - 3 trials x 5s, 1s warmup, for decision+receipt, receipt verification, and complete E2E. Both figures are reported and kept separate; the latency-derived rate is relabelled LATENCY-DERIVED RATE. The two disagree, which is the point of measuring: observed E2E sustained throughput is ~1604/s median against a latency-derived 1478/s, because the per-sample timer overhead in the latency path is amortized by the continuous loop. No runtime, test, fixture, workflow, SOTA, or claim-ceiling change. Findings F1-F4 remain findings and are not repaired here.
One coherent run of the AC-035A harness (a7f6ba3) against DUT e4e1a97, verified byte-identical before measuring. No cherry-picking across runs. Docs now record DUT_BASE_SHA and BENCHMARK_HARNESS_SHA separately and give reproduction instructions that actually work: check out the harness commit, not the DUT commit, because benchmarks/ does not exist at the DUT commit. The prior instruction was wrong and is called out as such. Throughput is reported twice and kept apart: observed sustained rate (the measurement) and latency-derived rate (the arithmetic). On this run they agree to within 0.3% on the end-to-end path; the direction of the small gap is not stable across runs, and that is stated rather than smoothed over. Also fixes a real defect in the drift guard added earlier in this work order: it diffed DUT_BASE_SHA against HEAD, which compares two commits and therefore reported "verified unchanged" while an uncommitted edit to authcontract/ silently changed what was actually measured. It now diffs against the working tree. Negative-tested: appending a line to authcontract/digest.py makes the harness refuse with exit 2 and name the drifted file. Results: 7/7 E2E, 38/38 adversarial, determinism stable across 100 replays with all 10 protected receipt fields byte-identical, 342 pytest passed. Findings F1-F4 remain findings; none repaired here. TRL remains 4 - methodology improved, environment did not.
veraxis-protocol
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AC-035 — End-to-End Benchmark Baseline, as amended by AC-035A
The first rigorous end-to-end operational and performance baseline for AuthContract. Measures only capabilities that exist; dimensions the architecture cannot express are recorded
NOT EVALUATEDrather than estimated.Provenance — two distinct commits
A benchmark harness cannot exist at the commit it measures, so these are deliberately separate:
e4e1a97509df1a66c44b090c0a0ca0a03907f4dc— the implementation measureda7f6ba374b1362e624a3f8b912b265dd03da4cdd— the commit containing the harnessBefore measuring, the harness diffs every device-under-test path against
DUT_BASE_SHAagainst the working tree and refuses to run on any drift (exit 2). This run:verified: true, zero modified DUT files — so "these numbers describee4e1a975" is verified, not asserted.Scope — additions only
No pre-existing file was modified.
git diff --stat e4e1a975 -- authcontract/ tests/ fixtures/ .github/ pyproject.toml README.md docs/SOTA.md docs/SOTA-EVIDENCE.md docs/DEVELOPER-LANGUAGE.md docs/CLEANROOM-VALIDATION-RUNBOOK.mdis empty.Correctness
7/7 end-to-end · 38/38 adversarial · 342 existing regression tests pass.
E2E-05 mutates and truncates every protected receipt field in turn — 20 variants — rather than one representative field. All detected.
Performance (n=1000–2000, warm)
Throughput — measured and derived, kept separate
Observed sustained rate (3 trials × 5 s continuous loop, 1 s warmup, single-threaded) — the real measurement:
Latency-derived rate (reciprocal of mean latency — arithmetic, an upper-bound estimate, not an observed capacity): E2E 1,579.6/s, decisions 3,532.7/s, verifications 3,382.0/s.
The two agree to within ~0.3% on the E2E path here. The direction of the small gap is not stable across runs; that is stated in the docs rather than smoothed over.
Scaling — linear in both measurable dimensions
Declared actions 1→1000: 280 µs → 78.0 ms. Required facts 10→10,000: 954 µs → 648.2 ms (~13 MiB). No cliff in range.
NOT EVALUATED: multi-contract corpora, concurrency/distribution, persistent storage — none exist at this commit.Determinism
100 replays produce byte-identical receipts across all 10 protected fields.
decision_timeis stable because it binds to the fact bundle's declarednow, not wall-clock — so there is no intentionally-varying receipt field. Scoped to one process, one platform, one Python version; cross-environment reproducibility untested and not claimed.Findings — recorded, not repaired
admission_digestandreceipt_digestboth change; cross-verification fails) but do not gate authorization. Detectable after the fact, not prevented at decision time.*_mutated.jsonfixtures are mutated and correctly re-sealed, so they are validly-bound variants, not mutation attacks. The first draft of this suite expected them to refuse — that expectation was wrong, not the implementation. The genuine stale-binding attack is built programmatically (E2E-06, ADV-35, ADV-36) and is correctly refused withAC_DIGEST.AC-035A additionally fixed a defect in the harness's own drift guard: it originally diffed
DUT_BASE_SHAagainstHEAD(commit-to-commit), so it reported "verified unchanged" while an uncommitted edit silently changed what was measured. Caught by negative-testing the guard rather than trusting it.TRL
TRL 4, with partial TRL 5 methodological characteristics. Implemented, demonstrated, and benchmarked — but not externally validated (all specimens authored by the same party as the implementation) and not production validated. Improving benchmark methodology makes the TRL 4 assessment better supported, not higher.
Reproduce
Check out the harness commit, not the DUT commit —
benchmarks/does not exist ate4e1a975. Verified working from a genuinely fresh clone. Exit codes:0all pass,1correctness failure,2DUT drift detected and nothing measured.Claim ceiling preserved
One synthetic banking specimen family, one machine, one process. Establishes no production readiness, regulatory or legal correctness, universal source-to-rule derivation, arbitrary-domain compatibility, security certification, distributed scalability, formal proof, or comparative superiority.
Generated by Claude Code