From 63960248b9834a52c8312de2ce07304c0548361e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 13:15:28 +0000 Subject: [PATCH 1/3] AC-035: end-to-end system run and benchmark baseline Adds the first rigorous end-to-end operational and performance baseline for AuthContract, measured at e4e1a97 against only the capabilities that exist. New (no existing file modified): benchmarks/run_benchmarks.py - phase runner benchmarks/harness.py - timing, memory, environment, scale specimens benchmarks/specimens/ - declarative specimen table benchmarks/results/ - raw JSON output, committed for audit benchmarks/README.md - methodology and reproduction docs/BENCHMARKS.md - measured results and limitations docs/ROADMAP.md - NOW/NEXT/LATER/RESEARCH, each item evidence-cited docs/TRL-ASSESSMENT.md - TRL 4, with what would move it to 5 Correctness: 7/7 end-to-end specimens, 38/38 adversarial specimens, 342 existing regression tests. E2E-05 mutates and truncates every protected receipt field in turn (20 variants), all detected. Performance: end-to-end p50 701.6us, p95 1083.3us, p99 1149.9us; 1314 complete E2E transactions/sec single-process. Canonicalization (~96us) dominates; the authorization check itself is 8.4us. Verification costs about as much as deciding because it recomputes every binding rather than trusting the receipt. Scaling: linear in both measurable dimensions - declared actions 1..1000 and required facts 10..10000 (10k facts = ~0.57s, ~13MiB). Multi-contract corpora, concurrency, and persistence are recorded NOT EVALUATED rather than estimated, because no such path exists at this commit. Determinism: 100 replays produce byte-identical receipts across all 10 protected fields. decision_time is stable because it binds to the fact bundle's declared now, not wall clock, so there is no intentionally-varying receipt field. Findings recorded, not repaired during the measurement run: F1 admission approvals are bound as evidence but do not gate authorization F2 canonicalization is repeated several times per transaction F3 no replay protection semantics exist (determinism is not protection) F4 the *_mutated.json fixtures are mutated AND re-sealed, so they are validly-bound variants; the first draft of this suite mis-expected them to refuse. The genuine stale-binding attack is built programmatically and is correctly refused with AC_DIGEST. Claim ceiling preserved: one synthetic specimen family, one machine, one process. No production-readiness, regulatory, security-certification, distributed-scalability, or comparative claim. --- benchmarks/README.md | 102 +++ benchmarks/harness.py | 255 ++++++ .../results/AC-035-CORRECTNESS-MATRIX.json | 709 +++++++++++++++ benchmarks/results/AC-035-DETERMINISM.json | 79 ++ benchmarks/results/AC-035-E2E-RESULTS.json | 232 +++++ benchmarks/results/AC-035-PERFORMANCE.json | 404 +++++++++ .../results/AC-035-RESOURCE-PROFILE.json | 72 ++ benchmarks/run_benchmarks.py | 840 ++++++++++++++++++ benchmarks/specimens/__init__.py | 390 ++++++++ docs/BENCHMARKS.md | 293 ++++++ docs/ROADMAP.md | 163 ++++ docs/TRL-ASSESSMENT.md | 104 +++ 12 files changed, 3643 insertions(+) create mode 100644 benchmarks/README.md create mode 100644 benchmarks/harness.py create mode 100644 benchmarks/results/AC-035-CORRECTNESS-MATRIX.json create mode 100644 benchmarks/results/AC-035-DETERMINISM.json create mode 100644 benchmarks/results/AC-035-E2E-RESULTS.json create mode 100644 benchmarks/results/AC-035-PERFORMANCE.json create mode 100644 benchmarks/results/AC-035-RESOURCE-PROFILE.json create mode 100644 benchmarks/run_benchmarks.py create mode 100644 benchmarks/specimens/__init__.py create mode 100644 docs/BENCHMARKS.md create mode 100644 docs/ROADMAP.md create mode 100644 docs/TRL-ASSESSMENT.md diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..c613443 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,102 @@ +# AuthContract benchmark suite (AC-035) + +The first end-to-end operational and performance baseline for AuthContract. + +This suite measures **only capabilities that exist** at the commit under test. +Where the architecture cannot express a dimension, the result is recorded as +`NOT EVALUATED` rather than estimated. + +## Reproduce + +```bash +git clone https://github.com/veraxis-protocol/AuthContract.git +cd AuthContract +git checkout e4e1a97509df1a66c44b090c0a0ca0a03907f4dc +python3 -m venv .venv && source .venv/bin/activate +pip install -e ".[test]" +python3 benchmarks/run_benchmarks.py +``` + +Runtime is roughly 45 seconds. Results are written to `benchmarks/results/`. +The process exits non-zero if any correctness specimen fails. + +## Layout + +| Path | Purpose | +|---|---| +| `run_benchmarks.py` | Entry point; runs every phase and writes results | +| `harness.py` | Timing, memory, environment capture, synthetic specimen generation | +| `specimens/__init__.py` | Declarative specimen table with expected behaviour classes | +| `results/` | Raw JSON output (committed, so results are auditable without re-running) | + +## What is measured + +**Path exercised.** Contract artifact → parse → digest-scope validation and +sibling binding → RFC 8785 canonicalization → SHA-256 contract digest → +deterministic projection → runtime fact admission → action check → +ALLOW/REFUSE decision → receipt emission → independent receipt verification → +mutation detection. + +**Stage latency.** Each stage timed separately with warmup discarded, reported +as n/min/p50/p95/p99/max/mean/stdev in microseconds. Very fast stages use an +inner batch so they are measured above timer resolution rather than quantized +to zero. + +**Throughput.** Single-process, single-threaded, derived from warm mean +latency. Not a distributed or multi-core claim. + +**Scale curves.** Two dimensions the architecture can genuinely express: +declared mediated actions (1/10/100/1000) and required facts (10/100/1000/10000). +Synthetic specimens are re-sealed with a recomputed contract digest so they are +genuinely valid — otherwise the curve would measure the refusal path. + +**Correctness and adversarial matrices.** Expectations are stated as behaviour +*classes* (ALLOW vs fail-closed REFUSE) rather than exact reason codes. The +class is the security-relevant property; the observed reason code is recorded +alongside as evidence, so a change in refusal taxonomy shows up in the results +without the assertion being weakened into a tautology. + +**Determinism.** 100 repeated executions per specimen, comparing decision, +reason code, contract digest, projection, and every protected receipt field. + +## Methodology notes worth knowing + +- **`receipt_verification` re-runs the entire decision path.** `verify_receipt` + deliberately recomputes every binding from raw inputs rather than trusting any + field in the receipt. That is why verification costs about as much as the + original decision — it is a correctness property, not an inefficiency. +- **`decision_and_receipt` supersets `projection` and `action_check`.** Stage + figures are not additive into the end-to-end figure. +- **`decisions_per_second` and `receipts_per_second` are the same measurement.** + Receipt emission is not separately callable at this commit; `run_specimen` + decides and emits in one pass. +- **Percentiles use nearest-rank.** For these sample sizes the difference from + an interpolating definition is far below the measurement noise floor. +- **Memory is `tracemalloc` peak**, i.e. Python-level allocation attributable to + one execution, excluding interpreter baseline and allocator caching. + +## A finding about the `*_mutated.json` fixtures + +The repository's `banking_payment_specimen_contract_mutated.json` and +`banking_payment_specimen_admission_mutated.json` fixtures are mutated **and +correctly re-sealed** — their recomputed digest matches their declared digest. +They are therefore validly-bound *variant* contracts, not post-binding mutation +attacks, and `ALLOW` is the correct outcome for them. + +The first draft of this suite expected them to refuse. That expectation was +wrong, not the implementation. The genuine attack — altering contract-bound +material while leaving the declared digest stale — is constructed +programmatically in the runner as `E2E-06`, `ADV-35`, and `ADV-36`, and is +correctly refused with `AC_DIGEST`. + +This is recorded rather than quietly corrected because the naming of those +fixtures invites exactly this misreading. + +## Claim ceiling + +These measurements establish only what they measure: a bounded MVP-alpha +implementation, one synthetic banking specimen family, one machine, one +process. They do **not** establish production readiness, regulatory or legal +correctness, universal source-to-rule derivation, arbitrary-domain +compatibility, security certification, distributed scalability, formal +correctness, or comparative superiority over any other system. diff --git a/benchmarks/harness.py b/benchmarks/harness.py new file mode 100644 index 0000000..6373d92 --- /dev/null +++ b/benchmarks/harness.py @@ -0,0 +1,255 @@ +"""Shared measurement primitives for the AC-035 benchmark suite. + +Deliberately dependency-free (stdlib only) so the benchmark reproduces in the +same clean-room environment the runtime itself targets. Nothing here imports +from the benchmark bodies, so timing code stays separable from the specimens +being timed. +""" + +from __future__ import annotations + +import copy +import json +import platform +import statistics +import subprocess +import sys +import time +import tracemalloc +from pathlib import Path +from typing import Any, Callable + +REPO_ROOT = Path(__file__).resolve().parent.parent +FIXTURES = REPO_ROOT / "fixtures" + + +# -------------------------------------------------------------------------- +# fixture loading +# -------------------------------------------------------------------------- + +def load_fixture(relative: str) -> Any: + """Load a fixture as parsed JSON. Path is relative to fixtures/.""" + with (FIXTURES / relative).open(encoding="utf-8") as handle: + return json.load(handle) + + +def load_fixture_text(relative: str) -> str: + return (FIXTURES / relative).read_text(encoding="utf-8") + + +# -------------------------------------------------------------------------- +# environment capture (Phase 1) +# -------------------------------------------------------------------------- + +def _git(*args: str) -> str: + try: + out = subprocess.run( + ["git", *args], cwd=REPO_ROOT, capture_output=True, text=True, check=True + ) + return out.stdout.strip() + except (subprocess.CalledProcessError, FileNotFoundError): + return "UNAVAILABLE" + + +def _installed_versions() -> dict[str, str]: + """Report versions of the packages actually imported by the runtime path.""" + versions: dict[str, str] = {} + for name in ("rfc8785", "pytest"): + try: + module = __import__(name) + versions[name] = getattr(module, "__version__", "UNKNOWN") + except ImportError: + versions[name] = "NOT INSTALLED" + return versions + + +def capture_environment() -> dict[str, Any]: + return { + "commit_sha": _git("rev-parse", "HEAD"), + "tree_sha": _git("rev-parse", "HEAD^{tree}"), + "git_status_clean": _git("status", "--porcelain") == "", + "python_version": sys.version.split()[0], + "python_implementation": platform.python_implementation(), + "platform": platform.platform(), + "machine": platform.machine(), + "processor": platform.processor() or "UNKNOWN", + "dependency_versions": _installed_versions(), + } + + +# -------------------------------------------------------------------------- +# latency measurement (Phase 3) +# -------------------------------------------------------------------------- + +def _percentile(sorted_values: list[float], fraction: float) -> float: + """Nearest-rank percentile. Deterministic and dependency-free; for the + sample sizes used here the difference from an interpolating definition is + far below the measurement noise floor.""" + if not sorted_values: + raise ValueError("percentile of empty sample") + rank = max(1, min(len(sorted_values), int(round(fraction * len(sorted_values) + 0.5)))) + return sorted_values[rank - 1] + + +def summarize(samples_seconds: list[float]) -> dict[str, Any]: + """Distribution summary in microseconds.""" + micros = sorted(value * 1e6 for value in samples_seconds) + return { + "n": len(micros), + "unit": "microseconds", + "min": round(micros[0], 3), + "p50": round(_percentile(micros, 0.50), 3), + "p95": round(_percentile(micros, 0.95), 3), + "p99": round(_percentile(micros, 0.99), 3), + "max": round(micros[-1], 3), + "mean": round(statistics.fmean(micros), 3), + "stdev": round(statistics.stdev(micros), 3) if len(micros) > 1 else 0.0, + } + + +def measure( + operation: Callable[[], Any], + *, + repetitions: int, + warmup: int = 50, + inner_batch: int = 1, +) -> dict[str, Any]: + """Time `operation` and return a distribution summary. + + `inner_batch` amortizes clock granularity for very fast operations: each + recorded sample is the mean of `inner_batch` back-to-back executions, so a + sub-microsecond operation is still measured above the timer's resolution + rather than being reported as a quantized 0. Warmup executions are + discarded so the reported figures are steady-state (warm) numbers. + """ + for _ in range(warmup): + operation() + + samples: list[float] = [] + for _ in range(repetitions): + start = time.perf_counter() + for _ in range(inner_batch): + operation() + elapsed = time.perf_counter() - start + samples.append(elapsed / inner_batch) + + result = summarize(samples) + result["warmup_discarded"] = warmup + result["inner_batch"] = inner_batch + return result + + +def measure_cold(operation: Callable[[], Any]) -> dict[str, Any]: + """Single un-warmed execution, for comparison against the warm figure.""" + start = time.perf_counter() + operation() + return {"unit": "microseconds", "single_cold_execution": round((time.perf_counter() - start) * 1e6, 3)} + + +def throughput_per_second(summary: dict[str, Any]) -> float: + """Derive sustained single-process rate from a mean latency in microseconds.""" + mean_seconds = summary["mean"] / 1e6 + if mean_seconds <= 0: + return float("inf") + return round(1.0 / mean_seconds, 1) + + +# -------------------------------------------------------------------------- +# memory measurement (Phase 9) +# -------------------------------------------------------------------------- + +def measure_peak_memory(operation: Callable[[], Any]) -> dict[str, Any]: + """Peak Python heap attributable to one execution, via tracemalloc. + + This measures interpreter-level allocation, not RSS: it excludes the + interpreter's own baseline and any allocator caching, which is what makes + it comparable across runs. + """ + tracemalloc.start() + tracemalloc.reset_peak() + operation() + _current, peak = tracemalloc.get_traced_memory() + tracemalloc.stop() + return {"peak_traced_bytes": peak, "peak_traced_kib": round(peak / 1024, 2)} + + +# -------------------------------------------------------------------------- +# synthetic scale specimens (Phase 5) +# -------------------------------------------------------------------------- + +def _reseal(artifact: dict[str, Any]) -> dict[str, Any]: + """Recompute the contract digest and re-bind every sibling that carries it. + + Generated specimens must be genuinely valid, not merely well-shaped: if the + digest were left stale the specimen would refuse at the digest gate and the + scale curve would measure the refusal path instead of the authorization + path. + """ + from authcontract.digest import contract_digest + + digest = contract_digest(artifact["contract"]) + for sibling in ("activation", "admission", "proof"): + if isinstance(artifact.get(sibling), dict) and "contract_digest" in artifact[sibling]: + artifact[sibling]["contract_digest"] = digest + return artifact + + +def make_action_scaled_specimen(action_count: int) -> dict[str, Any]: + """Banking specimen widened to `action_count` declared mediated actions. + + Scales the projection domain (the 'rules' dimension) while leaving the + exercised action, the required facts, and the decision semantics identical, + so any latency change is attributable to domain size alone. + """ + artifact = copy.deepcopy(load_fixture("banking_payment_specimen.json")) + actions = artifact["contract"]["projection_domain"]["actions"] + template = copy.deepcopy(actions["send_payment"]) + mediated = artifact["contract"]["subject"]["mediated_actions"] + for index in range(action_count - 1): + name = f"synthetic_action_{index:05d}" + actions[name] = copy.deepcopy(template) + mediated.append(name) + return _reseal(artifact) + + +def make_fact_scaled_specimen(fact_count: int) -> tuple[dict[str, Any], dict[str, Any]]: + """Banking specimen and matching bundle widened to `fact_count` facts. + + Both sides are widened together: a contract requiring N facts is only + satisfiable by a bundle supplying all N, so this scales the admissibility + workload rather than manufacturing a refusal. + """ + artifact = copy.deepcopy(load_fixture("banking_payment_specimen.json")) + bundle = copy.deepcopy(load_fixture("runtime/facts_valid.json")) + + required_template = copy.deepcopy(artifact["contract"]["required_facts"][0]) + fact_template = copy.deepcopy(bundle["facts"][0]) + + for index in range(fact_count - 1): + fact_id = f"synthetic.fact_{index:05d}" + + requirement = copy.deepcopy(required_template) + requirement["fact_id"] = fact_id + artifact["contract"]["required_facts"].append(requirement) + + fact = copy.deepcopy(fact_template) + fact["fact_id"] = fact_id + fact["evidence"] = copy.deepcopy(fact_template["evidence"]) + fact["evidence"]["fact_id"] = fact_id + bundle["facts"].append(fact) + + return _reseal(artifact), bundle + + +# -------------------------------------------------------------------------- +# result serialization +# -------------------------------------------------------------------------- + +def write_result(filename: str, payload: dict[str, Any]) -> Path: + results_dir = Path(__file__).resolve().parent / "results" + results_dir.mkdir(parents=True, exist_ok=True) + path = results_dir / filename + with path.open("w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2, sort_keys=True) + handle.write("\n") + return path diff --git a/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json b/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json new file mode 100644 index 0000000..ad21627 --- /dev/null +++ b/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json @@ -0,0 +1,709 @@ +{ + "adversarial_matrix": { + "by_category": { + "action_outside_scope": { + "FAIL": 0, + "PASS": 1 + }, + "admission_evidence_binding": { + "FAIL": 0, + "PASS": 2 + }, + "control": { + "FAIL": 0, + "PASS": 1 + }, + "corroboration_missing": { + "FAIL": 0, + "PASS": 1 + }, + "digest_mutation": { + "FAIL": 0, + "PASS": 2 + }, + "digest_scope_violation": { + "FAIL": 0, + "PASS": 1 + }, + "evidence_mismatch": { + "FAIL": 0, + "PASS": 4 + }, + "inactive_contract": { + "FAIL": 0, + "PASS": 1 + }, + "malformed_structured_input": { + "FAIL": 0, + "PASS": 4 + }, + "malformed_type": { + "FAIL": 0, + "PASS": 5 + }, + "missing_required_field": { + "FAIL": 0, + "PASS": 2 + }, + "out_of_domain_value": { + "FAIL": 0, + "PASS": 1 + }, + "post_binding_mutation": { + "FAIL": 0, + "PASS": 2 + }, + "receipt_context_substitution": { + "FAIL": 0, + "PASS": 1 + }, + "reordered_structured_input": { + "FAIL": 0, + "PASS": 1 + }, + "replay": { + "FAIL": 0, + "PASS": 1 + }, + "stale_fact": { + "FAIL": 0, + "PASS": 1 + }, + "trust_basis_violation": { + "FAIL": 0, + "PASS": 1 + }, + "unknown_field_forbidden": { + "FAIL": 0, + "PASS": 5 + }, + "validly_bound_variant": { + "FAIL": 0, + "PASS": 1 + } + }, + "results": [ + { + "category": "control", + "description": "Control: the valid specimen must still be ALLOWed", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "ADV-00-control" + }, + { + "category": "missing_required_field", + "description": "Missing required action parameter", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-01" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown action parameter where unknown fields are forbidden", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-02" + }, + { + "category": "out_of_domain_value", + "description": "Out-of-domain enum value", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-03" + }, + { + "category": "malformed_type", + "description": "Lossy decimal representation in action parameter", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-04" + }, + { + "category": "action_outside_scope", + "description": "Action type outside declared mediated scope", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_UNCLASSIFIED_ACTION", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-05" + }, + { + "category": "missing_required_field", + "description": "Required fact absent from bundle", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_FACT_BUNDLE_INCOMPLETE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-06" + }, + { + "category": "stale_fact", + "description": "Stale fact beyond freshness window", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_STALE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-07" + }, + { + "category": "malformed_type", + "description": "Future-dated fact timestamp", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_FUTURE_TIMESTAMP", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-08" + }, + { + "category": "malformed_type", + "description": "Timezone-naive fact timestamp (unverifiable ordering)", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_TIME_UNVERIFIABLE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-09" + }, + { + "category": "trust_basis_violation", + "description": "Self-asserted fact where policy prohibits self-assertion", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_SELF_ASSERTED", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-10" + }, + { + "category": "malformed_type", + "description": "Lossy wire representation of fact value", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_REPRESENTATION", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-11" + }, + { + "category": "malformed_structured_input", + "description": "Duplicate fact_id in bundle", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-12" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown field on fact object", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-13" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown field on fact bundle", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-14" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown field on verified-evidence object", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-15" + }, + { + "category": "evidence_mismatch", + "description": "Claimed value diverges from verifier-established evidence value", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_EVIDENCE_MISMATCH", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-16" + }, + { + "category": "evidence_mismatch", + "description": "Claimed asserter diverges from verifier-established asserter", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_EVIDENCE_MISMATCH", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-17" + }, + { + "category": "evidence_mismatch", + "description": "Claimed fact_id diverges from verifier-established fact_id", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_IDENTITY_MISMATCH", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-18" + }, + { + "category": "evidence_mismatch", + "description": "Stale verified evidence presented with a fresh caller claim", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_EVIDENCE_MISMATCH", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-19" + }, + { + "category": "validly_bound_variant", + "description": "Variant contract, mutated AND correctly re-sealed (binding intact)", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "ADV-20" + }, + { + "category": "digest_mutation", + "description": "Sibling digest disagreement across bound locations", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-21" + }, + { + "category": "digest_scope_violation", + "description": "Self-referential contract digest", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST_SCOPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-22" + }, + { + "category": "digest_mutation", + "description": "Cross-object digest substitution", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-23" + }, + { + "category": "malformed_type", + "description": "Malformed artifact", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-24" + }, + { + "category": "inactive_contract", + "description": "Suspended contract (activation state not ACTIVE)", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_INACTIVE_CONTRACT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-25" + }, + { + "category": "admission_evidence_binding", + "description": "Admission carrying approvals, contract binding intact", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "ADV-26" + }, + { + "category": "malformed_structured_input", + "description": "Admission present as JSON null", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-27" + }, + { + "category": "malformed_structured_input", + "description": "Admission present as JSON list", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-28" + }, + { + "category": "malformed_structured_input", + "description": "Duplicate required-fact declaration", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-29" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown field on required-fact declaration", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-30" + }, + { + "category": "corroboration_missing", + "description": "Corroboration required but not satisfiable as declared", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-31" + }, + { + "category": "reordered_structured_input", + "description": "Reordered JSON object keys must not alter canonical identity or decision", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "digest invariant under key reordering: True", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": null, + "specimen": "ADV-32" + }, + { + "category": "replay", + "description": "Replayed identical request", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "Replay yields an identical receipt. NOTE: this documents determinism, not replay *protection* \u2014 there is no nonce, sequence number, or single-use semantics at this commit, so an intercepted receipt is indistinguishable from a legitimately re-derived one. Recorded as an architectural gap.", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": null, + "specimen": "ADV-33" + }, + { + "category": "receipt_context_substitution", + "description": "Receipt presented against a different action than it was issued for", + "expected_result": "REFUSE", + "failure_stage": "receipt_verification", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": true, + "receipt_verified": false, + "specimen": "ADV-34" + }, + { + "category": "post_binding_mutation", + "description": "Contract version altered, digest binding left stale", + "expected_result": "REFUSE", + "failure_stage": "digest_binding", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-35" + }, + { + "category": "post_binding_mutation", + "description": "Projection domain widened (amount retyped), digest binding left stale", + "expected_result": "REFUSE", + "failure_stage": "digest_binding", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-36" + }, + { + "category": "admission_evidence_binding", + "description": "Forged admission approvals must alter the bound evidence", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "Decision is ALLOW because approvals are not an authorization gate at this commit (finding AC-035-F1). The evidence binding nonetheless holds: admission_digest and receipt_digest both change (True), and a receipt issued for the unforged admission does not verify against the forged one (VEIP_RECEIPT_MISMATCH).", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": false, + "specimen": "ADV-37" + } + ], + "totals": { + "failed": 0, + "not_evaluated": 0, + "passed": 38, + "total": 38 + } + }, + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "end_to_end_matrix": [ + { + "category": "happy_path", + "description": "Happy path: valid contract, valid facts, permitted action", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "E2E-01" + }, + { + "category": "stale_fact", + "description": "Stale runtime fact: freshness window exceeded", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_STALE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-02" + }, + { + "category": "malformed_contract", + "description": "Malformed contract artifact", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-03" + }, + { + "category": "unsupported_domain", + "description": "Action outside the declared projection domain", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_UNCLASSIFIED_ACTION", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-04" + }, + { + "category": "source_version_mutation", + "description": "Source/version material changed with stale (non-recomputed) digest binding", + "expected_result": "REFUSE", + "failure_stage": "digest_binding", + "notes": "contract.identity.version altered; activation/admission/proof digests left stale", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-06" + }, + { + "category": "receipt_mutation", + "description": "Receipt mutation and truncation across every protected field", + "expected_result": "REFUSE", + "failure_stage": "receipt_verification", + "notes": "20 mutation/truncation variants tested; 20 detected", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_RECEIPT_MISMATCH/MALFORMED", + "receipt_generated": true, + "receipt_verified": false, + "specimen": "E2E-05" + }, + { + "category": "deterministic_replay", + "description": "Deterministic replay, 100 executions of the identical specimen", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "all protected receipt fields byte-identical across replays", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "E2E-07" + } + ], + "environment": { + "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + }, + "generated_at_utc": "2026-08-24T13:12:02Z", + "work_order": "AC-035" +} diff --git a/benchmarks/results/AC-035-DETERMINISM.json b/benchmarks/results/AC-035-DETERMINISM.json new file mode 100644 index 0000000..37469a2 --- /dev/null +++ b/benchmarks/results/AC-035-DETERMINISM.json @@ -0,0 +1,79 @@ +{ + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "determinism": { + "determinism_statement": "Every observed output is stable across repeated execution of a fixed specimen: decision, reason code, contract digest, projection, and all protected receipt fields including decision_time. decision_time is stable because it is bound to the fact bundle's own declared `now`, not to wall-clock time at invocation \u2014 so at this commit there is no intentionally-varying receipt field. This is determinism over fixed inputs in a single process; it is not a claim about cross-version, cross-platform, or cross-implementation reproducibility, none of which was tested.", + "fully_deterministic_over_fixed_inputs": true, + "primitives": { + "contract_digest_distinct_values": 1, + "contract_digest_stable": true, + "projection_distinct_values": 1, + "projection_stable": true + }, + "refused_specimen": { + "decision_stable": true, + "decision_values_observed": [ + "REFUSED" + ], + "executions": 100, + "reason_code_stable": true, + "reason_codes_observed": [ + "RUN_FACT_STALE" + ], + "receipt_emitted": false, + "stable_receipt_fields": [], + "varying_receipt_fields": [] + }, + "valid_specimen": { + "decision_stable": true, + "decision_values_observed": [ + "ALLOW" + ], + "executions": 100, + "reason_code_stable": true, + "reason_codes_observed": [ + "OK" + ], + "receipt_emitted": true, + "stable_receipt_fields": [ + "activation_id", + "admission_digest", + "contract_digest", + "decision", + "decision_time", + "exact_action_digest", + "execution_result", + "projection_digest", + "receipt_digest", + "runtime_fact_set_digest" + ], + "varying_receipt_fields": [] + } + }, + "environment": { + "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + }, + "generated_at_utc": "2026-08-24T13:12:02Z", + "work_order": "AC-035" +} diff --git a/benchmarks/results/AC-035-E2E-RESULTS.json b/benchmarks/results/AC-035-E2E-RESULTS.json new file mode 100644 index 0000000..24fa450 --- /dev/null +++ b/benchmarks/results/AC-035-E2E-RESULTS.json @@ -0,0 +1,232 @@ +{ + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "end_to_end": { + "receipt_mutation_detail": [ + { + "detected": true, + "mutated_field": "activation_id", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "admission_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "contract_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "decision", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "decision_time", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "exact_action_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "execution_result", + "reason_code": "VEIP_INVALID_EXECUTION_RESULT" + }, + { + "detected": true, + "mutated_field": "projection_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "receipt_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "runtime_fact_set_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "activation_id (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "admission_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "contract_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "decision (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "decision_time (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "exact_action_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "execution_result (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "projection_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "receipt_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "runtime_fact_set_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + } + ], + "specimens": [ + { + "category": "happy_path", + "description": "Happy path: valid contract, valid facts, permitted action", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "E2E-01" + }, + { + "category": "stale_fact", + "description": "Stale runtime fact: freshness window exceeded", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_STALE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-02" + }, + { + "category": "malformed_contract", + "description": "Malformed contract artifact", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-03" + }, + { + "category": "unsupported_domain", + "description": "Action outside the declared projection domain", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_UNCLASSIFIED_ACTION", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-04" + }, + { + "category": "source_version_mutation", + "description": "Source/version material changed with stale (non-recomputed) digest binding", + "expected_result": "REFUSE", + "failure_stage": "digest_binding", + "notes": "contract.identity.version altered; activation/admission/proof digests left stale", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-06" + }, + { + "category": "receipt_mutation", + "description": "Receipt mutation and truncation across every protected field", + "expected_result": "REFUSE", + "failure_stage": "receipt_verification", + "notes": "20 mutation/truncation variants tested; 20 detected", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_RECEIPT_MISMATCH/MALFORMED", + "receipt_generated": true, + "receipt_verified": false, + "specimen": "E2E-05" + }, + { + "category": "deterministic_replay", + "description": "Deterministic replay, 100 executions of the identical specimen", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "all protected receipt fields byte-identical across replays", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "E2E-07" + } + ], + "totals": { + "failed": 0, + "passed": 7, + "total": 7 + } + }, + "environment": { + "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + }, + "generated_at_utc": "2026-08-24T13:12:02Z", + "work_order": "AC-035" +} diff --git a/benchmarks/results/AC-035-PERFORMANCE.json b/benchmarks/results/AC-035-PERFORMANCE.json new file mode 100644 index 0000000..5e1bbc7 --- /dev/null +++ b/benchmarks/results/AC-035-PERFORMANCE.json @@ -0,0 +1,404 @@ +{ + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "environment": { + "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + }, + "generated_at_utc": "2026-08-24T13:12:02Z", + "performance": { + "cold_vs_warm": { + "complete_end_to_end_cold": { + "single_cold_execution": 948.726, + "unit": "microseconds" + }, + "note": "Single un-warmed execution in an already-imported interpreter. The dominant one-time cost is interpreter startup and module import, reported separately in the resource profile." + }, + "stages": { + "action_check": { + "summary": { + "inner_batch": 20, + "max": 17.956, + "mean": 8.434, + "min": 6.581, + "n": 2000, + "p50": 7.825, + "p95": 11.235, + "p99": 13.395, + "stdev": 1.488, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "check_action: validate a proposed action against the projection" + }, + "canonical_digest": { + "summary": { + "inner_batch": 20, + "max": 180.803, + "mean": 98.778, + "min": 79.828, + "n": 2000, + "p50": 87.856, + "p95": 158.604, + "p99": 166.782, + "stdev": 23.999, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "contract_digest: JCS canonicalization plus SHA-256" + }, + "canonicalization": { + "summary": { + "inner_batch": 20, + "max": 186.59, + "mean": 96.423, + "min": 76.881, + "n": 2000, + "p50": 88.61, + "p95": 140.864, + "p99": 158.497, + "stdev": 19.025, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "canonical_bytes: RFC 8785 JCS serialization of the contract" + }, + "complete_end_to_end": { + "summary": { + "inner_batch": 1, + "max": 1519.071, + "mean": 760.837, + "min": 528.694, + "n": 1000, + "p50": 701.565, + "p95": 1083.29, + "p99": 1149.858, + "stdev": 172.467, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "parse \u2192 decide \u2192 emit receipt \u2192 independently verify receipt" + }, + "contract_parse": { + "summary": { + "inner_batch": 20, + "max": 40.473, + "mean": 10.631, + "min": 8.826, + "n": 2000, + "p50": 9.559, + "p95": 15.088, + "p99": 17.988, + "stdev": 2.286, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "json.loads of the raw contract artifact text" + }, + "decision_and_receipt": { + "summary": { + "inner_batch": 1, + "max": 3808.82, + "mean": 377.622, + "min": 250.815, + "n": 1000, + "p50": 376.382, + "p95": 484.69, + "p99": 562.39, + "stdev": 129.516, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "run_specimen: full orchestration \u2014 parse bundle, project, admit facts, decide, and emit the receipt. Supersets projection and action_check." + }, + "projection": { + "summary": { + "inner_batch": 20, + "max": 287.837, + "mean": 124.242, + "min": 92.451, + "n": 2000, + "p50": 122.922, + "p95": 170.734, + "p99": 181.503, + "stdev": 22.71, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "project: build the deterministic action-domain projection" + }, + "projection_digest": { + "summary": { + "inner_batch": 20, + "max": 101.318, + "mean": 59.444, + "min": 40.9, + "n": 2000, + "p50": 61.011, + "p95": 79.546, + "p99": 89.135, + "stdev": 10.971, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "projection_digest over the realized projection" + }, + "receipt_verification": { + "summary": { + "inner_batch": 1, + "max": 769.033, + "mean": 445.598, + "min": 256.907, + "n": 1000, + "p50": 453.604, + "p95": 568.965, + "p99": 634.309, + "stdev": 85.783, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "verify_receipt: independently recompute every binding from raw inputs and compare. Internally re-runs the full decision path." + }, + "validation_and_binding": { + "summary": { + "inner_batch": 20, + "max": 934.733, + "mean": 112.399, + "min": 81.993, + "n": 2000, + "p50": 103.982, + "p95": 159.629, + "p99": 173.914, + "stdev": 30.209, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "verify_artifact: digest-scope validation plus sibling binding agreement" + } + }, + "throughput": { + "complete_e2e_transactions_per_second": 1314.3, + "decisions_per_second": 2648.2, + "note": "Single-process, single-threaded, derived from warm mean latency. Not a distributed or multi-core claim.", + "receipt_verifications_per_second": 2244.2, + "receipts_per_second": 2648.2, + "receipts_per_second_caveat": "Receipt emission is not separately callable at this commit: run_specimen decides and emits in one pass, so decisions/sec and receipts/sec are the same measurement reported twice, not two independent figures." + } + }, + "scaling": { + "declared_action_scaling": { + "dimension": "count of declared mediated actions in the projection domain", + "levels": [ + { + "decision": "ALLOW", + "declared_actions": 1, + "latency": { + "inner_batch": 1, + "max": 711.098, + "mean": 407.433, + "min": 247.138, + "n": 400, + "p50": 424.174, + "p95": 532.946, + "p99": 616.399, + "stdev": 89.87, + "unit": "microseconds", + "warmup_discarded": 10 + }, + "peak_traced_memory": { + "peak_traced_bytes": 6214, + "peak_traced_kib": 6.07 + }, + "reason_code": "OK" + }, + { + "decision": "ALLOW", + "declared_actions": 10, + "latency": { + "inner_batch": 1, + "max": 2117.708, + "mean": 1201.454, + "min": 882.591, + "n": 400, + "p50": 1111.978, + "p95": 1622.925, + "p99": 1842.1, + "stdev": 250.461, + "unit": "microseconds", + "warmup_discarded": 10 + }, + "peak_traced_memory": { + "peak_traced_bytes": 9621, + "peak_traced_kib": 9.4 + }, + "reason_code": "OK" + }, + { + "decision": "ALLOW", + "declared_actions": 100, + "latency": { + "inner_batch": 1, + "max": 14053.368, + "mean": 8489.413, + "min": 7275.667, + "n": 400, + "p50": 8070.566, + "p95": 10956.776, + "p99": 13454.636, + "stdev": 1220.737, + "unit": "microseconds", + "warmup_discarded": 10 + }, + "peak_traced_memory": { + "peak_traced_bytes": 41774, + "peak_traced_kib": 40.79 + }, + "reason_code": "OK" + }, + { + "decision": "ALLOW", + "declared_actions": 1000, + "latency": { + "inner_batch": 1, + "max": 114677.333, + "mean": 81994.852, + "min": 74569.494, + "n": 60, + "p50": 80423.713, + "p95": 100705.244, + "p99": 114677.333, + "stdev": 6847.287, + "unit": "microseconds", + "warmup_discarded": 10 + }, + "peak_traced_memory": { + "peak_traced_bytes": 374877, + "peak_traced_kib": 366.09 + }, + "reason_code": "OK" + } + ], + "observed_shape": "sublinear (size x1000 -> time x201.2)" + }, + "not_evaluated": { + "concurrent_or_distributed_throughput": "NOT EVALUATED \u2014 no concurrency or distribution layer exists at this commit.", + "multi_contract_corpora": "NOT EVALUATED \u2014 the implementation evaluates one artifact per invocation; there is no multi-contract registry or cross-contract selection path at this commit whose scaling could be measured without inventing architecture.", + "persistent_storage_scaling": "NOT EVALUATED \u2014 the runtime is stateless over in-memory inputs; there is no storage backend to scale." + }, + "required_fact_scaling": { + "dimension": "count of required facts (contract) matched by supplied facts (bundle)", + "levels": [ + { + "decision": "ALLOW", + "latency": { + "inner_batch": 1, + "max": 1590.885, + "mean": 1016.326, + "min": 719.762, + "n": 200, + "p50": 942.157, + "p95": 1447.004, + "p99": 1522.569, + "stdev": 220.994, + "unit": "microseconds", + "warmup_discarded": 3 + }, + "peak_traced_memory": { + "peak_traced_bytes": 16262, + "peak_traced_kib": 15.88 + }, + "reason_code": "OK", + "required_facts": 10 + }, + { + "decision": "ALLOW", + "latency": { + "inner_batch": 1, + "max": 9921.777, + "mean": 6530.551, + "min": 5596.428, + "n": 200, + "p50": 6306.753, + "p95": 8274.561, + "p99": 9235.273, + "stdev": 802.916, + "unit": "microseconds", + "warmup_discarded": 3 + }, + "peak_traced_memory": { + "peak_traced_bytes": 143861, + "peak_traced_kib": 140.49 + }, + "reason_code": "OK", + "required_facts": 100 + }, + { + "decision": "ALLOW", + "latency": { + "inner_batch": 1, + "max": 71241.823, + "mean": 57994.791, + "min": 54316.178, + "n": 40, + "p50": 56670.824, + "p95": 64626.995, + "p99": 71241.823, + "stdev": 3765.218, + "unit": "microseconds", + "warmup_discarded": 3 + }, + "peak_traced_memory": { + "peak_traced_bytes": 1364965, + "peak_traced_kib": 1332.97 + }, + "reason_code": "OK", + "required_facts": 1000 + }, + { + "decision": "ALLOW", + "latency": { + "inner_batch": 1, + "max": 588081.258, + "mean": 566644.023, + "min": 550615.654, + "n": 5, + "p50": 563057.594, + "p95": 588081.258, + "p99": 588081.258, + "stdev": 15077.813, + "unit": "microseconds", + "warmup_discarded": 3 + }, + "peak_traced_memory": { + "peak_traced_bytes": 13788824, + "peak_traced_kib": 13465.65 + }, + "reason_code": "OK", + "required_facts": 10000 + } + ], + "observed_shape": "approximately linear (size x1000 -> time x557.5)" + } + }, + "work_order": "AC-035" +} diff --git a/benchmarks/results/AC-035-RESOURCE-PROFILE.json b/benchmarks/results/AC-035-RESOURCE-PROFILE.json new file mode 100644 index 0000000..3bd1c54 --- /dev/null +++ b/benchmarks/results/AC-035-RESOURCE-PROFILE.json @@ -0,0 +1,72 @@ +{ + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "environment": { + "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + }, + "generated_at_utc": "2026-08-24T13:12:02Z", + "resources": { + "artifact_sizes": { + "action": 148, + "canonical_contract_bytes": 773, + "contract_artifact": 1179, + "contract_body_only": 773, + "fact_bundle": 675, + "projection": 479, + "receipt": 715, + "unit": "bytes (compact JSON encoding)" + }, + "peak_memory": { + "note": "tracemalloc peak attributable to one execution; excludes interpreter baseline", + "single_decision": { + "peak_traced_bytes": 6214, + "peak_traced_kib": 6.07 + }, + "single_verification": { + "peak_traced_bytes": 6334, + "peak_traced_kib": 6.19 + } + }, + "process_max_rss": { + "note": "whole-process peak RSS at end of benchmark run, including interpreter and harness", + "unit": "kilobytes", + "value": 64804 + }, + "process_startup": { + "max": 122.74, + "mean": 79.64, + "min": 60.44, + "samples": [ + 91.1, + 122.74, + 62.11, + 60.44, + 61.84 + ], + "unit": "milliseconds", + "what": "python -c 'import authcontract.veip', out-of-process, 5 samples" + } + }, + "work_order": "AC-035" +} diff --git a/benchmarks/run_benchmarks.py b/benchmarks/run_benchmarks.py new file mode 100644 index 0000000..9ed1502 --- /dev/null +++ b/benchmarks/run_benchmarks.py @@ -0,0 +1,840 @@ +#!/usr/bin/env python3 +"""AC-035 — end-to-end operational and performance baseline for AuthContract. + +Exercises only capabilities that exist at the measured commit. No runtime code +is imported-and-patched, monkeypatched, or reimplemented here: every number +comes from calling the same public entry points a developer would call. + +Reproduce with: + + python3 -m pip install -e ".[test]" + python3 benchmarks/run_benchmarks.py + +Results are written to benchmarks/results/ as JSON. +""" + +from __future__ import annotations + +import copy +import json +import resource +import subprocess +import sys +import time +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from harness import ( # noqa: E402 + REPO_ROOT, + capture_environment, + load_fixture, + load_fixture_text, + make_action_scaled_specimen, + make_fact_scaled_specimen, + measure, + measure_cold, + measure_peak_memory, + throughput_per_second, + write_result, +) +from specimens import ADVERSARIAL_SPECIMENS, E2E_SPECIMENS # noqa: E402 + +from authcontract.digest import canonical_bytes, contract_digest, verify_artifact # noqa: E402 +from authcontract.projection import check_action, project, projection_digest, projection_to_dict # noqa: E402 +from authcontract.veip import run_specimen, verify_receipt # noqa: E402 + +EXECUTION_RESULT = "SIMULATED_SUCCESS" + +# Repetition counts. Chosen so each stage's sample is large enough for a stable +# p99 without making the suite take longer than a developer will tolerate. +STAGE_REPS = 2000 +STAGE_INNER_BATCH = 20 +E2E_REPS = 1000 +DETERMINISM_REPS = 100 + + +def _run(artifact: dict, action: dict, facts: dict): + return run_specimen(artifact, action, facts, execution_result=EXECUTION_RESULT) + + +def _load_triple(spec) -> tuple[dict, dict, dict]: + return ( + load_fixture(spec.artifact_fixture), + load_fixture(spec.action_fixture), + load_fixture(spec.facts_fixture), + ) + + +# -------------------------------------------------------------------------- +# Phase 2 + Phase 6 — end-to-end specimens and correctness matrix +# -------------------------------------------------------------------------- + +def _classify(result) -> str: + return "ALLOW" if result.decision == "ALLOW" else "REFUSE" + + +def _evaluate_specimen(spec) -> dict[str, Any]: + """Run one declarative specimen and record what actually happened.""" + artifact, action, facts = _load_triple(spec) + + failure_stage = None + receipt_generated = False + receipt_verified = None + notes = "" + + try: + result = _run(artifact, action, facts) + observed = _classify(result) + reason_code = result.reason_code + receipt_generated = result.receipt is not None + + if observed == "REFUSE": + failure_stage = "authorization_decision" + elif receipt_generated: + verification = verify_receipt(result.receipt, artifact, action, facts) + receipt_verified = verification.status == "PASS" + if not receipt_verified: + notes = f"receipt failed self-verification: {verification.reason_code}" + except Exception as exc: # noqa: BLE001 - an escaping exception is itself a finding + observed = "EXCEPTION" + reason_code = type(exc).__name__ + failure_stage = "uncaught_exception" + notes = ( + "Implementation raised instead of returning a refusal. An escaping " + f"exception is a fail-closed defect, not a refusal: {exc}" + ) + + passed = observed == spec.expected + if spec.expected == "ALLOW" and passed: + passed = receipt_generated and receipt_verified is True + + return { + "specimen": spec.specimen_id, + "description": spec.description, + "category": spec.category, + "expected_result": spec.expected, + "observed_result": observed, + "pass_fail": "PASS" if passed else "FAIL", + "failure_stage": failure_stage, + "receipt_generated": receipt_generated, + "receipt_verified": receipt_verified, + "reason_code": reason_code, + "notes": notes, + } + + +def _stale_bound_mutation(mutate) -> dict[str, Any]: + """Return a specimen whose contract body was altered but whose declared + digests were deliberately NOT recomputed — the real post-binding attack.""" + artifact = copy.deepcopy(load_fixture("banking_payment_specimen.json")) + mutate(artifact) + return artifact + + +def phase_e2e_and_correctness() -> dict[str, Any]: + rows = [_evaluate_specimen(spec) for spec in E2E_SPECIMENS] + + action_v = load_fixture("actions/send_payment_valid.json") + facts_v = load_fixture("runtime/facts_valid.json") + + # E2E-06: source/version-bound material changed with NO corresponding valid + # binding. Built programmatically because every on-disk "mutated" fixture is + # correctly re-sealed and therefore cannot express this attack. + def _bump_version(art: dict) -> None: + art["contract"]["identity"]["version"] = "9.9.9-attacker" + + mutated = _stale_bound_mutation(_bump_version) + mutated_result = _run(mutated, action_v, facts_v) + rows.append( + { + "specimen": "E2E-06", + "description": "Source/version material changed with stale (non-recomputed) digest binding", + "category": "source_version_mutation", + "expected_result": "REFUSE", + "observed_result": _classify(mutated_result), + "pass_fail": "PASS" if mutated_result.decision != "ALLOW" else "FAIL", + "failure_stage": "digest_binding", + "receipt_generated": mutated_result.receipt is not None, + "receipt_verified": None, + "reason_code": mutated_result.reason_code, + "notes": "contract.identity.version altered; activation/admission/proof digests left stale", + } + ) + + # E2E-05: receipt mutation. Every protected field is mutated in turn, so + # this measures whether the binding covers the whole payload rather than + # only the one field a single-shot test would happen to pick. + artifact = load_fixture("banking_payment_specimen.json") + action = load_fixture("actions/send_payment_valid.json") + facts = load_fixture("runtime/facts_valid.json") + baseline = _run(artifact, action, facts) + + mutation_rows = [] + if baseline.receipt is not None: + for field in sorted(baseline.receipt.keys()): + tampered = copy.deepcopy(baseline.receipt) + original = tampered[field] + tampered[field] = "TAMPERED" if not isinstance(original, str) else original + "-TAMPERED" + verification = verify_receipt(tampered, artifact, action, facts) + mutation_rows.append( + { + "mutated_field": field, + "detected": verification.status != "PASS", + "reason_code": verification.reason_code, + } + ) + + # Truncation: a receipt missing a protected field must not verify. + for field in sorted(baseline.receipt.keys()): + truncated = copy.deepcopy(baseline.receipt) + del truncated[field] + verification = verify_receipt(truncated, artifact, action, facts) + mutation_rows.append( + { + "mutated_field": f"{field} (removed)", + "detected": verification.status != "PASS", + "reason_code": verification.reason_code, + } + ) + + all_detected = all(row["detected"] for row in mutation_rows) and bool(mutation_rows) + rows.append( + { + "specimen": "E2E-05", + "description": "Receipt mutation and truncation across every protected field", + "category": "receipt_mutation", + "expected_result": "REFUSE", + "observed_result": "REFUSE" if all_detected else "ALLOW", + "pass_fail": "PASS" if all_detected else "FAIL", + "failure_stage": "receipt_verification", + "receipt_generated": True, + "receipt_verified": False, + "reason_code": "VEIP_RECEIPT_MISMATCH/MALFORMED", + "notes": ( + f"{len(mutation_rows)} mutation/truncation variants tested; " + f"{sum(1 for r in mutation_rows if r['detected'])} detected" + ), + } + ) + + # E2E-07: deterministic replay (detail recorded in the determinism phase). + replays = [_run(artifact, action, facts) for _ in range(DETERMINISM_REPS)] + receipts_identical = all(r.receipt == replays[0].receipt for r in replays) + decisions_identical = all(r.decision == replays[0].decision for r in replays) + rows.append( + { + "specimen": "E2E-07", + "description": f"Deterministic replay, {DETERMINISM_REPS} executions of the identical specimen", + "category": "deterministic_replay", + "expected_result": "ALLOW", + "observed_result": "ALLOW" if decisions_identical and replays[0].decision == "ALLOW" else "FAIL", + "pass_fail": "PASS" if (receipts_identical and decisions_identical) else "FAIL", + "failure_stage": None if receipts_identical else "determinism", + "receipt_generated": True, + "receipt_verified": True, + "reason_code": replays[0].reason_code, + "notes": ( + "all protected receipt fields byte-identical across replays" + if receipts_identical + else "receipt fields varied across replays" + ), + } + ) + + return { + "specimens": rows, + "receipt_mutation_detail": mutation_rows, + "totals": { + "total": len(rows), + "passed": sum(1 for row in rows if row["pass_fail"] == "PASS"), + "failed": sum(1 for row in rows if row["pass_fail"] == "FAIL"), + }, + } + + +# -------------------------------------------------------------------------- +# Phase 3 + Phase 4 — per-stage latency and throughput +# -------------------------------------------------------------------------- + +def phase_performance() -> dict[str, Any]: + artifact_text = load_fixture_text("banking_payment_specimen.json") + artifact = load_fixture("banking_payment_specimen.json") + action = load_fixture("actions/send_payment_valid.json") + facts = load_fixture("runtime/facts_valid.json") + contract = artifact["contract"] + + projection = project(artifact) + receipt = _run(artifact, action, facts).receipt + + stages = { + "contract_parse": { + "what": "json.loads of the raw contract artifact text", + "summary": measure( + lambda: json.loads(artifact_text), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "validation_and_binding": { + "what": "verify_artifact: digest-scope validation plus sibling binding agreement", + "summary": measure( + lambda: verify_artifact(artifact), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "canonicalization": { + "what": "canonical_bytes: RFC 8785 JCS serialization of the contract", + "summary": measure( + lambda: canonical_bytes(contract), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "canonical_digest": { + "what": "contract_digest: JCS canonicalization plus SHA-256", + "summary": measure( + lambda: contract_digest(contract), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "projection": { + "what": "project: build the deterministic action-domain projection", + "summary": measure( + lambda: project(artifact), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "projection_digest": { + "what": "projection_digest over the realized projection", + "summary": measure( + lambda: projection_digest(projection), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "action_check": { + "what": "check_action: validate a proposed action against the projection", + "summary": measure( + lambda: check_action(projection, action), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "decision_and_receipt": { + "what": ( + "run_specimen: full orchestration — parse bundle, project, admit facts, " + "decide, and emit the receipt. Supersets projection and action_check." + ), + "summary": measure( + lambda: _run(artifact, action, facts), + repetitions=E2E_REPS, + inner_batch=1, + ), + }, + "receipt_verification": { + "what": ( + "verify_receipt: independently recompute every binding from raw inputs " + "and compare. Internally re-runs the full decision path." + ), + "summary": measure( + lambda: verify_receipt(receipt, artifact, action, facts), + repetitions=E2E_REPS, + inner_batch=1, + ), + }, + } + + def full_e2e() -> None: + parsed = json.loads(artifact_text) + result = run_specimen(parsed, action, facts, execution_result=EXECUTION_RESULT) + verify_receipt(result.receipt, parsed, action, facts) + + stages["complete_end_to_end"] = { + "what": "parse → decide → emit receipt → independently verify receipt", + "summary": measure(full_e2e, repetitions=E2E_REPS, inner_batch=1), + } + + cold = { + "note": ( + "Single un-warmed execution in an already-imported interpreter. The dominant " + "one-time cost is interpreter startup and module import, reported separately " + "in the resource profile." + ), + "complete_end_to_end_cold": measure_cold(full_e2e), + } + + throughput = { + "note": "Single-process, single-threaded, derived from warm mean latency. Not a distributed or multi-core claim.", + "decisions_per_second": throughput_per_second(stages["decision_and_receipt"]["summary"]), + "receipts_per_second": throughput_per_second(stages["decision_and_receipt"]["summary"]), + "receipt_verifications_per_second": throughput_per_second(stages["receipt_verification"]["summary"]), + "complete_e2e_transactions_per_second": throughput_per_second(stages["complete_end_to_end"]["summary"]), + "receipts_per_second_caveat": ( + "Receipt emission is not separately callable at this commit: run_specimen " + "decides and emits in one pass, so decisions/sec and receipts/sec are the " + "same measurement reported twice, not two independent figures." + ), + } + + return {"stages": stages, "cold_vs_warm": cold, "throughput": throughput} + + +# -------------------------------------------------------------------------- +# Phase 5 — bounded scale curves +# -------------------------------------------------------------------------- + +def phase_scale() -> dict[str, Any]: + action = load_fixture("actions/send_payment_valid.json") + + action_curve = [] + for count in (1, 10, 100, 1000): + artifact = make_action_scaled_specimen(count) + facts = load_fixture("runtime/facts_valid.json") + control = _run(artifact, action, facts) + reps = 400 if count < 1000 else 60 + summary = measure(lambda a=artifact, f=facts: _run(a, action, f), repetitions=reps, warmup=10) + action_curve.append( + { + "declared_actions": count, + "decision": control.decision, + "reason_code": control.reason_code, + "latency": summary, + "peak_traced_memory": measure_peak_memory(lambda a=artifact, f=facts: _run(a, action, f)), + } + ) + + fact_curve = [] + for count in (10, 100, 1000, 10000): + artifact, facts = make_fact_scaled_specimen(count) + control = _run(artifact, action, facts) + reps = 200 if count <= 100 else (40 if count == 1000 else 5) + summary = measure(lambda a=artifact, f=facts: _run(a, action, f), repetitions=reps, warmup=3) + fact_curve.append( + { + "required_facts": count, + "decision": control.decision, + "reason_code": control.reason_code, + "latency": summary, + "peak_traced_memory": measure_peak_memory(lambda a=artifact, f=facts: _run(a, action, f)), + } + ) + + def _shape(curve: list[dict], key: str) -> str: + """Describe growth by comparing per-unit cost at the smallest and largest points.""" + first, last = curve[0], curve[-1] + size_ratio = last[key] / first[key] + time_ratio = last["latency"]["mean"] / first["latency"]["mean"] + if time_ratio < size_ratio * 0.25: + return f"sublinear (size x{size_ratio:.0f} -> time x{time_ratio:.1f})" + if time_ratio <= size_ratio * 1.5: + return f"approximately linear (size x{size_ratio:.0f} -> time x{time_ratio:.1f})" + return f"superlinear — bottleneck candidate (size x{size_ratio:.0f} -> time x{time_ratio:.1f})" + + return { + "declared_action_scaling": { + "dimension": "count of declared mediated actions in the projection domain", + "levels": action_curve, + "observed_shape": _shape(action_curve, "declared_actions"), + }, + "required_fact_scaling": { + "dimension": "count of required facts (contract) matched by supplied facts (bundle)", + "levels": fact_curve, + "observed_shape": _shape(fact_curve, "required_facts"), + }, + "not_evaluated": { + "multi_contract_corpora": ( + "NOT EVALUATED — the implementation evaluates one artifact per invocation; " + "there is no multi-contract registry or cross-contract selection path at this " + "commit whose scaling could be measured without inventing architecture." + ), + "concurrent_or_distributed_throughput": ( + "NOT EVALUATED — no concurrency or distribution layer exists at this commit." + ), + "persistent_storage_scaling": ( + "NOT EVALUATED — the runtime is stateless over in-memory inputs; there is no " + "storage backend to scale." + ), + }, + } + + +# -------------------------------------------------------------------------- +# Phase 7 — determinism +# -------------------------------------------------------------------------- + +def phase_determinism() -> dict[str, Any]: + action = load_fixture("actions/send_payment_valid.json") + artifact = load_fixture("banking_payment_specimen.json") + + findings = {} + for label, facts_fixture in ( + ("valid_specimen", "runtime/facts_valid.json"), + ("refused_specimen", "runtime/facts_stale.json"), + ): + facts = load_fixture(facts_fixture) + results = [_run(artifact, action, facts) for _ in range(DETERMINISM_REPS)] + + decisions = {r.decision for r in results} + reason_codes = {r.reason_code for r in results} + receipts = [r.receipt for r in results] + + stable_fields: list[str] = [] + varying_fields: list[str] = [] + if receipts[0] is not None: + for field in sorted(receipts[0].keys()): + values = {json.dumps(r[field], sort_keys=True) for r in receipts} + (stable_fields if len(values) == 1 else varying_fields).append(field) + + findings[label] = { + "executions": DETERMINISM_REPS, + "decision_values_observed": sorted(decisions), + "reason_codes_observed": sorted(reason_codes), + "decision_stable": len(decisions) == 1, + "reason_code_stable": len(reason_codes) == 1, + "receipt_emitted": receipts[0] is not None, + "stable_receipt_fields": stable_fields, + "varying_receipt_fields": varying_fields, + } + + # Independently confirm the digest/projection primitives are stable too. + contract = artifact["contract"] + digests = {contract_digest(contract) for _ in range(DETERMINISM_REPS)} + projections = { + json.dumps(projection_to_dict(project(artifact)), sort_keys=True) + for _ in range(DETERMINISM_REPS) + } + + findings["primitives"] = { + "contract_digest_distinct_values": len(digests), + "projection_distinct_values": len(projections), + "contract_digest_stable": len(digests) == 1, + "projection_stable": len(projections) == 1, + } + + valid = findings["valid_specimen"] + fully_stable = ( + valid["decision_stable"] + and valid["reason_code_stable"] + and not valid["varying_receipt_fields"] + and findings["primitives"]["contract_digest_stable"] + and findings["primitives"]["projection_stable"] + ) + + findings["determinism_statement"] = ( + ( + "Every observed output is stable across repeated execution of a fixed specimen: " + "decision, reason code, contract digest, projection, and all protected receipt " + "fields including decision_time. decision_time is stable because it is bound to " + "the fact bundle's own declared `now`, not to wall-clock time at invocation — so " + "at this commit there is no intentionally-varying receipt field. This is " + "determinism over fixed inputs in a single process; it is not a claim about " + "cross-version, cross-platform, or cross-implementation reproducibility, none of " + "which was tested." + ) + if fully_stable + else ( + "Determinism is NOT complete. Fields observed to vary across identical " + f"executions: {valid['varying_receipt_fields']}. Each must be classified as " + "intentionally variable or as a defect before any reproducibility claim is made." + ) + ) + findings["fully_deterministic_over_fixed_inputs"] = fully_stable + return findings + + +# -------------------------------------------------------------------------- +# Phase 8 — adversarial battery +# -------------------------------------------------------------------------- + +def phase_adversarial() -> dict[str, Any]: + rows = [_evaluate_specimen(spec) for spec in ADVERSARIAL_SPECIMENS] + + # Structural attacks that are not simple fixture swaps. + artifact = load_fixture("banking_payment_specimen.json") + action = load_fixture("actions/send_payment_valid.json") + facts = load_fixture("runtime/facts_valid.json") + + # Key reordering must not change canonical identity (RFC 8785 property). + reordered = {k: artifact[k] for k in reversed(list(artifact.keys()))} + reordered["contract"] = { + k: artifact["contract"][k] for k in reversed(list(artifact["contract"].keys())) + } + same_digest = contract_digest(reordered["contract"]) == contract_digest(artifact["contract"]) + reorder_result = _run(reordered, action, facts) + rows.append( + { + "specimen": "ADV-32", + "description": "Reordered JSON object keys must not alter canonical identity or decision", + "category": "reordered_structured_input", + "expected_result": "ALLOW", + "observed_result": _classify(reorder_result), + "pass_fail": "PASS" if (same_digest and reorder_result.decision == "ALLOW") else "FAIL", + "failure_stage": None if same_digest else "canonicalization", + "receipt_generated": reorder_result.receipt is not None, + "receipt_verified": None, + "reason_code": reorder_result.reason_code, + "notes": f"digest invariant under key reordering: {same_digest}", + } + ) + + # Replay: an identical request repeated must yield an identical decision. + first = _run(artifact, action, facts) + second = _run(artifact, action, facts) + replay_identical = first.receipt == second.receipt and first.decision == second.decision + rows.append( + { + "specimen": "ADV-33", + "description": "Replayed identical request", + "category": "replay", + "expected_result": "ALLOW", + "observed_result": _classify(second), + "pass_fail": "PASS" if replay_identical else "FAIL", + "failure_stage": None, + "receipt_generated": second.receipt is not None, + "receipt_verified": None, + "reason_code": second.reason_code, + "notes": ( + "Replay yields an identical receipt. NOTE: this documents determinism, not " + "replay *protection* — there is no nonce, sequence number, or single-use " + "semantics at this commit, so an intercepted receipt is indistinguishable " + "from a legitimately re-derived one. Recorded as an architectural gap." + ), + } + ) + + # Receipt verified against a different action than the one it was issued for. + other_action = load_fixture("actions/send_payment_out_of_enum_value.json") + cross = verify_receipt(first.receipt, artifact, other_action, facts) + rows.append( + { + "specimen": "ADV-34", + "description": "Receipt presented against a different action than it was issued for", + "category": "receipt_context_substitution", + "expected_result": "REFUSE", + "observed_result": "REFUSE" if cross.status != "PASS" else "ALLOW", + "pass_fail": "PASS" if cross.status != "PASS" else "FAIL", + "failure_stage": "receipt_verification", + "receipt_generated": True, + "receipt_verified": False, + "reason_code": cross.reason_code, + "notes": "", + } + ) + + # ADV-35/36: genuine post-binding mutation with stale digests. + for adv_id, label, mutate in ( + ( + "ADV-35", + "Contract version altered, digest binding left stale", + lambda art: art["contract"]["identity"].__setitem__("version", "9.9.9-attacker"), + ), + ( + "ADV-36", + "Projection domain widened (amount retyped), digest binding left stale", + lambda art: art["contract"]["projection_domain"]["actions"]["send_payment"][ + "parameters" + ]["amount"].__setitem__("value_type", "string"), + ), + ): + attacked = _stale_bound_mutation(mutate) + outcome = _run(attacked, action, facts) + rows.append( + { + "specimen": adv_id, + "description": label, + "category": "post_binding_mutation", + "expected_result": "REFUSE", + "observed_result": _classify(outcome), + "pass_fail": "PASS" if outcome.decision != "ALLOW" else "FAIL", + "failure_stage": "digest_binding", + "receipt_generated": outcome.receipt is not None, + "receipt_verified": None, + "reason_code": outcome.reason_code, + "notes": "", + } + ) + + # ADV-37: admission approvals are not an authorization gate, but they must + # still be bound into the evidence so a forged admission is distinguishable. + forged = copy.deepcopy(artifact) + forged["admission"]["approvals"] = [{"approval_id": "forged", "approver": "attacker"}] + forged_result = _run(forged, action, facts) + baseline_result = _run(artifact, action, facts) + binding_holds = ( + forged_result.receipt is not None + and baseline_result.receipt is not None + and forged_result.receipt["admission_digest"] != baseline_result.receipt["admission_digest"] + and forged_result.receipt["receipt_digest"] != baseline_result.receipt["receipt_digest"] + ) + cross_verify = verify_receipt(baseline_result.receipt, forged, action, facts) + rows.append( + { + "specimen": "ADV-37", + "description": "Forged admission approvals must alter the bound evidence", + "category": "admission_evidence_binding", + "expected_result": "ALLOW", + "observed_result": _classify(forged_result), + "pass_fail": "PASS" if (binding_holds and cross_verify.status != "PASS") else "FAIL", + "failure_stage": None, + "receipt_generated": True, + "receipt_verified": False, + "reason_code": forged_result.reason_code, + "notes": ( + "Decision is ALLOW because approvals are not an authorization gate at this " + "commit (finding AC-035-F1). The evidence binding nonetheless holds: " + f"admission_digest and receipt_digest both change ({binding_holds}), and a " + "receipt issued for the unforged admission does not verify against the forged " + f"one ({cross_verify.reason_code})." + ), + } + ) + + by_category: dict[str, dict[str, int]] = {} + for row in rows: + bucket = by_category.setdefault(row["category"], {"PASS": 0, "FAIL": 0}) + bucket[row["pass_fail"]] += 1 + + return { + "results": rows, + "totals": { + "total": len(rows), + "passed": sum(1 for row in rows if row["pass_fail"] == "PASS"), + "failed": sum(1 for row in rows if row["pass_fail"] == "FAIL"), + "not_evaluated": 0, + }, + "by_category": by_category, + } + + +# -------------------------------------------------------------------------- +# Phase 9 — resource profile +# -------------------------------------------------------------------------- + +def phase_resources() -> dict[str, Any]: + artifact = load_fixture("banking_payment_specimen.json") + action = load_fixture("actions/send_payment_valid.json") + facts = load_fixture("runtime/facts_valid.json") + result = _run(artifact, action, facts) + + # Interpreter + import cost, measured out-of-process so it is not masked by + # this process having already imported everything. + startup_samples = [] + for _ in range(5): + start = time.perf_counter() + subprocess.run( + [sys.executable, "-c", "import authcontract.veip"], + cwd=REPO_ROOT, + capture_output=True, + check=True, + ) + startup_samples.append((time.perf_counter() - start) * 1000) + + def _size(value: Any) -> int: + return len(json.dumps(value, separators=(",", ":")).encode("utf-8")) + + return { + "process_startup": { + "unit": "milliseconds", + "what": "python -c 'import authcontract.veip', out-of-process, 5 samples", + "samples": [round(s, 2) for s in startup_samples], + "min": round(min(startup_samples), 2), + "mean": round(sum(startup_samples) / len(startup_samples), 2), + "max": round(max(startup_samples), 2), + }, + "peak_memory": { + "note": "tracemalloc peak attributable to one execution; excludes interpreter baseline", + "single_decision": measure_peak_memory(lambda: _run(artifact, action, facts)), + "single_verification": measure_peak_memory( + lambda: verify_receipt(result.receipt, artifact, action, facts) + ), + }, + "process_max_rss": { + "unit": "kilobytes", + "note": "whole-process peak RSS at end of benchmark run, including interpreter and harness", + "value": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, + }, + "artifact_sizes": { + "unit": "bytes (compact JSON encoding)", + "contract_artifact": _size(artifact), + "contract_body_only": _size(artifact["contract"]), + "canonical_contract_bytes": len(canonical_bytes(artifact["contract"])), + "action": _size(action), + "fact_bundle": _size(facts), + "projection": _size(projection_to_dict(project(artifact))), + "receipt": _size(result.receipt), + }, + } + + +# -------------------------------------------------------------------------- +# runner +# -------------------------------------------------------------------------- + +CLAIM_CEILING = [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific.", +] + + +def main() -> int: + started = time.time() + environment = capture_environment() + + print("AC-035 benchmark — commit", environment["commit_sha"][:12]) + print(" phase: end-to-end + correctness matrix") + e2e = phase_e2e_and_correctness() + print(" phase: performance") + performance = phase_performance() + print(" phase: scale curves") + scale = phase_scale() + print(" phase: determinism") + determinism = phase_determinism() + print(" phase: adversarial battery") + adversarial = phase_adversarial() + print(" phase: resource profile") + resources = phase_resources() + + common = { + "work_order": "AC-035", + "environment": environment, + "claim_ceiling": CLAIM_CEILING, + "generated_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + } + + write_result("AC-035-E2E-RESULTS.json", {**common, "end_to_end": e2e}) + write_result("AC-035-PERFORMANCE.json", {**common, "performance": performance, "scaling": scale}) + write_result( + "AC-035-CORRECTNESS-MATRIX.json", + {**common, "end_to_end_matrix": e2e["specimens"], "adversarial_matrix": adversarial}, + ) + write_result("AC-035-DETERMINISM.json", {**common, "determinism": determinism}) + write_result("AC-035-RESOURCE-PROFILE.json", {**common, "resources": resources}) + + e2e_totals = e2e["totals"] + adv_totals = adversarial["totals"] + print() + print(f" E2E: {e2e_totals['passed']}/{e2e_totals['total']} passed") + print(f" Adversarial: {adv_totals['passed']}/{adv_totals['total']} passed") + print(f" E2E p50: {performance['stages']['complete_end_to_end']['summary']['p50']} us") + print(f" E2E tps: {performance['throughput']['complete_e2e_transactions_per_second']}") + print(f" Determinism: {'stable' if determinism['fully_deterministic_over_fixed_inputs'] else 'NOT STABLE'}") + print(f" elapsed: {time.time() - started:.1f}s") + + return 0 if (e2e_totals["failed"] == 0 and adv_totals["failed"] == 0) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/specimens/__init__.py b/benchmarks/specimens/__init__.py new file mode 100644 index 0000000..16909c3 --- /dev/null +++ b/benchmarks/specimens/__init__.py @@ -0,0 +1,390 @@ +"""Specimen definitions for the AC-035 benchmark. + +Specimens are declarative: each names the fixtures it composes and the +behaviour class the architecture is expected to produce. Expectations are +stated as *classes* (ALLOW vs fail-closed REFUSE, verification PASS vs FAIL) +rather than exact reason codes, because the class is the security-relevant +property. The exact observed reason code is recorded alongside as evidence, so +a change in refusal taxonomy is visible in the results without silently +weakening the assertion into a tautology. +""" + +from __future__ import annotations + +from typing import Any, NamedTuple + + +class Specimen(NamedTuple): + specimen_id: str + description: str + artifact_fixture: str + action_fixture: str + facts_fixture: str + expected: str # "ALLOW" or "REFUSE" + category: str + + +# -------------------------------------------------------------------------- +# Phase 2 — the seven mandated end-to-end specimens +# -------------------------------------------------------------------------- + +E2E_SPECIMENS: tuple[Specimen, ...] = ( + Specimen( + "E2E-01", + "Happy path: valid contract, valid facts, permitted action", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "ALLOW", + "happy_path", + ), + Specimen( + "E2E-02", + "Stale runtime fact: freshness window exceeded", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_stale.json", + "REFUSE", + "stale_fact", + ), + Specimen( + "E2E-03", + "Malformed contract artifact", + "malformed.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_contract", + ), + Specimen( + "E2E-04", + "Action outside the declared projection domain", + "banking_payment_specimen.json", + "actions/send_payment_unknown_action_type.json", + "runtime/facts_valid.json", + "REFUSE", + "unsupported_domain", + ), + # E2E-05 (receipt mutation), E2E-06 (source/version mutation) and E2E-07 + # (deterministic replay) are constructed programmatically by the runner + # rather than declared here. + # + # E2E-06 specifically must mutate contract-bound material while leaving the + # binding STALE. The repository's `*_mutated.json` fixtures are mutated + # *and correctly re-sealed* — their recomputed digest matches their declared + # digest — so they are validly-bound variant contracts, not mutation + # attacks. They are exercised below as ALLOW controls, and the genuine + # stale-binding attack is built in the runner. +) + + +# -------------------------------------------------------------------------- +# Phase 8 — adversarial battery +# -------------------------------------------------------------------------- +# Every entry composes real accepted fixtures. Each is expected to fail closed; +# the happy-path control is included so a battery that trivially refuses +# everything is distinguishable from one that discriminates correctly. + +ADVERSARIAL_SPECIMENS: tuple[Specimen, ...] = ( + Specimen( + "ADV-00-control", + "Control: the valid specimen must still be ALLOWed", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "ALLOW", + "control", + ), + # --- action-side attacks ------------------------------------------------- + Specimen( + "ADV-01", + "Missing required action parameter", + "banking_payment_specimen.json", + "actions/send_payment_missing_required_parameter.json", + "runtime/facts_valid.json", + "REFUSE", + "missing_required_field", + ), + Specimen( + "ADV-02", + "Unknown action parameter where unknown fields are forbidden", + "banking_payment_specimen.json", + "actions/send_payment_unknown_parameter.json", + "runtime/facts_valid.json", + "REFUSE", + "unknown_field_forbidden", + ), + Specimen( + "ADV-03", + "Out-of-domain enum value", + "banking_payment_specimen.json", + "actions/send_payment_out_of_enum_value.json", + "runtime/facts_valid.json", + "REFUSE", + "out_of_domain_value", + ), + Specimen( + "ADV-04", + "Lossy decimal representation in action parameter", + "banking_payment_specimen.json", + "actions/send_payment_lossy_value.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_type", + ), + Specimen( + "ADV-05", + "Action type outside declared mediated scope", + "banking_payment_specimen.json", + "actions/send_payment_unknown_action_type.json", + "runtime/facts_valid.json", + "REFUSE", + "action_outside_scope", + ), + # --- fact-side attacks --------------------------------------------------- + Specimen( + "ADV-06", + "Required fact absent from bundle", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_missing_required.json", + "REFUSE", + "missing_required_field", + ), + Specimen( + "ADV-07", + "Stale fact beyond freshness window", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_stale.json", + "REFUSE", + "stale_fact", + ), + Specimen( + "ADV-08", + "Future-dated fact timestamp", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_future_timestamp.json", + "REFUSE", + "malformed_type", + ), + Specimen( + "ADV-09", + "Timezone-naive fact timestamp (unverifiable ordering)", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_naive_timestamp.json", + "REFUSE", + "malformed_type", + ), + Specimen( + "ADV-10", + "Self-asserted fact where policy prohibits self-assertion", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_self_asserted_prohibited.json", + "REFUSE", + "trust_basis_violation", + ), + Specimen( + "ADV-11", + "Lossy wire representation of fact value", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_lossy_representation.json", + "REFUSE", + "malformed_type", + ), + Specimen( + "ADV-12", + "Duplicate fact_id in bundle", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_duplicate_fact_id.json", + "REFUSE", + "malformed_structured_input", + ), + Specimen( + "ADV-13", + "Unknown field on fact object", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_unknown_fact_field.json", + "REFUSE", + "unknown_field_forbidden", + ), + Specimen( + "ADV-14", + "Unknown field on fact bundle", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_unknown_bundle_field.json", + "REFUSE", + "unknown_field_forbidden", + ), + Specimen( + "ADV-15", + "Unknown field on verified-evidence object", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_unknown_evidence_field.json", + "REFUSE", + "unknown_field_forbidden", + ), + # --- caller-claim vs verifier-established evidence divergence ------------ + Specimen( + "ADV-16", + "Claimed value diverges from verifier-established evidence value", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_verified_value_mismatch.json", + "REFUSE", + "evidence_mismatch", + ), + Specimen( + "ADV-17", + "Claimed asserter diverges from verifier-established asserter", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_verified_asserter_mismatch.json", + "REFUSE", + "evidence_mismatch", + ), + Specimen( + "ADV-18", + "Claimed fact_id diverges from verifier-established fact_id", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_verified_fact_id_mismatch.json", + "REFUSE", + "evidence_mismatch", + ), + Specimen( + "ADV-19", + "Stale verified evidence presented with a fresh caller claim", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_verified_time_stale_claimed_fresh.json", + "REFUSE", + "evidence_mismatch", + ), + # --- contract/binding attacks ------------------------------------------- + Specimen( + "ADV-20", + "Variant contract, mutated AND correctly re-sealed (binding intact)", + "banking_payment_specimen_contract_mutated.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + # ALLOW is correct: this fixture's declared digest matches its + # recomputed digest, so it is a different-but-validly-bound contract. + # The stale-binding attack is ADV-35/ADV-36, built in the runner. + "ALLOW", + "validly_bound_variant", + ), + Specimen( + "ADV-21", + "Sibling digest disagreement across bound locations", + "sibling_digest_mismatch.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "digest_mutation", + ), + Specimen( + "ADV-22", + "Self-referential contract digest", + "self_referential.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "digest_scope_violation", + ), + Specimen( + "ADV-23", + "Cross-object digest substitution", + "cross_object_substitution.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "digest_mutation", + ), + Specimen( + "ADV-24", + "Malformed artifact", + "malformed.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_type", + ), + # --- activation / admission attacks ------------------------------------- + Specimen( + "ADV-25", + "Suspended contract (activation state not ACTIVE)", + "banking_payment_specimen_suspended.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "inactive_contract", + ), + Specimen( + "ADV-26", + "Admission carrying approvals, contract binding intact", + "banking_payment_specimen_admission_mutated.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + # ALLOW is correct at this commit: admission approvals are bound into + # admission_digest (and therefore into receipt_digest) as evidence, but + # they are not evaluated as an authorization gate. ADV-37 asserts the + # binding property; the missing gate is recorded as finding AC-035-F1. + "ALLOW", + "admission_evidence_binding", + ), + Specimen( + "ADV-27", + "Admission present as JSON null", + "banking_payment_specimen_admission_null.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_structured_input", + ), + Specimen( + "ADV-28", + "Admission present as JSON list", + "banking_payment_specimen_admission_list.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_structured_input", + ), + # --- contract-declaration attacks --------------------------------------- + Specimen( + "ADV-29", + "Duplicate required-fact declaration", + "banking_payment_specimen_duplicate_required_fact.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_structured_input", + ), + Specimen( + "ADV-30", + "Unknown field on required-fact declaration", + "banking_payment_specimen_unknown_required_fact_field.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "unknown_field_forbidden", + ), + Specimen( + "ADV-31", + "Corroboration required but not satisfiable as declared", + "banking_payment_specimen_bad_corroboration_required.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "corroboration_missing", + ), +) diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md new file mode 100644 index 0000000..862cb31 --- /dev/null +++ b/docs/BENCHMARKS.md @@ -0,0 +1,293 @@ +# AuthContract benchmark baseline + +**Commit measured:** `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` +**Tree:** `9967077e7f9f9c661199c728c5a2e7fe496be07a` +**Work order:** AC-035 +**Raw results:** [`benchmarks/results/`](../benchmarks/results/) + +This is the first end-to-end operational and performance baseline for +AuthContract. It measures the bounded MVP-alpha path that exists today. It is +not a marketing document and contains no projected or extrapolated figures. + +--- + +## 1. Environment + +| | | +|---|---| +| Python | 3.11.15 (CPython) | +| Platform | Linux x86_64 | +| Dependencies | `rfc8785` 0.1.4, `pytest` 9.1.1 | +| Process | single, single-threaded | + +Figures are environment-specific. Absolute latencies will differ on other +hardware; the *shape* of the curves and the relative cost of stages are the +transferable results. + +--- + +## 2. What was measured + +The complete implemented path: + +``` +contract artifact + → parse + → digest-scope validation + sibling binding agreement + → RFC 8785 (JCS) canonicalization + → SHA-256 contract digest + → deterministic projection into the declared action domain + → runtime fact admission (issuer / trust basis / freshness / evidence binding) + → action check against the projection + → ALLOW or REFUSE + → decision receipt + → independent receipt verification from raw inputs + → mutation / truncation detection +``` + +--- + +## 3. End-to-end correctness + +**7 / 7 end-to-end specimens pass. 38 / 38 adversarial specimens pass.** + +| ID | Specimen | Expected | Observed | +|---|---|---|---| +| E2E-01 | Happy path | ALLOW + receipt verifies | PASS | +| E2E-02 | Stale runtime fact | REFUSE | PASS (`RUN_FACT_STALE`) | +| E2E-03 | Malformed contract | fail closed | PASS | +| E2E-04 | Action outside projection domain | fail closed | PASS | +| E2E-05 | Receipt mutation + truncation | detected | PASS (20/20 variants detected) | +| E2E-06 | Source/version change, stale binding | REFUSE | PASS (`AC_DIGEST`) | +| E2E-07 | Deterministic replay ×100 | identical | PASS | + +E2E-05 mutates **every** protected receipt field in turn and additionally +removes each field in turn — 20 variants total — rather than testing one +representative field. All 20 are detected. + +The adversarial battery covers 38 specimens across: missing required fields, +unknown fields where forbidden, malformed types, out-of-domain values, stale +facts, future and timezone-naive timestamps, self-assertion where prohibited, +lossy representations, duplicate identifiers, caller-claim vs +verifier-established evidence divergence (value / asserter / fact_id / staleness), +digest mutation, sibling digest disagreement, self-referential digests, +cross-object substitution, suspended contracts, malformed admission shapes, +contract-declaration defects, key reordering, replay, and receipt context +substitution. + +--- + +## 4. Latency by stage + +Warm figures, warmup discarded. Microseconds. + +| Stage | n | p50 | p95 | p99 | mean | +|---|---:|---:|---:|---:|---:| +| contract parse | 2000 | 9.6 | 15.1 | 18.0 | 10.6 | +| validation + binding | 2000 | 104.0 | 159.6 | 173.9 | 112.4 | +| canonicalization (JCS) | 2000 | 88.6 | 140.9 | 158.5 | 96.4 | +| canonical digest | 2000 | 87.9 | 158.6 | 166.8 | 98.8 | +| projection | 2000 | 122.9 | 170.7 | 181.5 | 124.2 | +| projection digest | 2000 | 61.0 | 79.5 | 89.1 | 59.4 | +| action check | 2000 | 7.8 | 11.2 | 13.4 | 8.4 | +| decision + receipt | 1000 | 376.4 | 484.7 | 562.4 | 377.6 | +| receipt verification | 1000 | 453.6 | 569.0 | 634.3 | 445.6 | +| **complete end-to-end** | **1000** | **701.6** | **1083.3** | **1149.9** | **760.8** | + +Run-to-run variation on this shared machine is roughly ±15% on the end-to-end +figure (a prior run measured p50 616 µs). The relative cost of stages is stable +across runs; treat the absolute numbers as an order-of-magnitude baseline, not +a precise constant. + +Stage figures are **not additive** into the end-to-end figure: `decision + +receipt` already contains projection and action check. + +### What the distribution says + +- **Canonicalization is the dominant primitive cost.** At ~96 µs it is roughly + 11× the action check and accounts for most of validation, digest, and + projection cost. Every one of those stages canonicalizes. +- **The authorization logic itself is nearly free.** Action check is 8.4 µs — + about 1% of the end-to-end path. Cost is overwhelmingly in canonical identity, + not in deciding. +- **Verification costs as much as deciding** (446 µs vs 378 µs), because + `verify_receipt` recomputes every binding from raw inputs rather than trusting + any field in the receipt. This is the property that makes receipts + independently checkable; the cost is the point, not a defect. +- **p99 is ~1.6× p50**, with no long tail — consistent with a pure-CPU path with + no I/O, locking, or allocation cliffs. + +--- + +## 5. Throughput + +Single process, single thread, derived from warm mean latency. + +| Metric | Rate | +|---|---:| +| Decisions / sec | 2,648 | +| Receipts / sec | 2,648 | +| Receipt verifications / sec | 2,244 | +| Complete E2E transactions / sec | 1,314 | + +Decisions/sec and receipts/sec are the *same measurement*: receipt emission is +not separately callable at this commit. A complete E2E transaction is +decide-plus-independently-verify, which is why it is roughly half the decision +rate. + +**Not claimed:** distributed throughput, multi-core scaling, or sustained +throughput under concurrency. No concurrency layer exists at this commit. + +--- + +## 6. Scale curves + +### Declared mediated actions — sublinear to linear + +| Actions | mean latency | peak traced memory | decision | +|---:|---:|---:|---| +| 1 | 407 µs | 6.1 KiB | ALLOW | +| 10 | 1,202 µs | 9.4 KiB | ALLOW | +| 100 | 8,489 µs | 40.8 KiB | ALLOW | +| 1,000 | 81,995 µs | 366.1 KiB | ALLOW | + +1000× the domain size costs ~201× the time and ~60× the memory. Growth is +linear in the large-N regime (the 100→1000 step costs ~9.7× for 10× the size) +with a fixed overhead that dominates at small N, which is why the endpoint ratio +reads as sublinear. + +### Required facts — approximately linear + +| Facts | mean latency | peak traced memory | decision | +|---:|---:|---:|---| +| 10 | 1,016 µs | 15.9 KiB | ALLOW | +| 100 | 6,531 µs | 140.5 KiB | ALLOW | +| 1,000 | 57,995 µs | 1.3 MiB | ALLOW | +| 10,000 | 566,644 µs | 13.1 MiB | ALLOW | + +1000× the fact count costs ~558× the time and ~847× the memory — linear in +both, with no observed nonlinear degradation or cliff in the tested range. Each +10× step costs ~9–10× consistently. + +**Practical reading:** a 10,000-fact contract takes ~0.57 s and ~13 MiB per +decision. That is workable for batch evaluation and marginal for interactive +use. The linearity means cost is predictable; the constant is dominated by +repeated canonicalization (§8, finding F2). + +### NOT EVALUATED + +| Dimension | Why | +|---|---| +| Multi-contract corpora | The runtime evaluates one artifact per invocation. No registry or cross-contract selection path exists whose scaling could be measured without inventing architecture. | +| Concurrent / distributed throughput | No concurrency or distribution layer exists at this commit. | +| Persistent storage scaling | The runtime is stateless over in-memory inputs. No storage backend exists to scale. | + +--- + +## 7. Determinism + +**Fully deterministic over fixed inputs**, across 100 repeated executions: + +| Output | Stable? | +|---|---| +| Decision | yes | +| Reason code | yes | +| Contract digest | yes (1 distinct value / 100 runs) | +| Projection | yes (1 distinct value / 100 runs) | +| All 10 protected receipt fields | yes | + +Notably `decision_time` is **stable**, because it is bound to the fact bundle's +own declared `now` rather than to wall-clock time at invocation. At this commit +there is therefore **no intentionally-varying receipt field** — the entire +receipt is byte-identical across replays of a fixed specimen. + +**Scope of the claim:** this is determinism over fixed inputs in a single +process on one platform and one Python version. Cross-version, cross-platform, +and cross-implementation reproducibility were **not** tested and are not +claimed. + +--- + +## 8. Findings + +Recorded during the measurement run, not repaired during it. + +**AC-035-F1 — Admission approvals are bound as evidence but are not an +authorization gate.** +Forging `admission.approvals` yields `ALLOW`. The evidence binding does hold: +`admission_digest` and `receipt_digest` both change, and a receipt issued for +the unforged admission does not verify against the forged one. So the forgery is +*detectable after the fact* but is not *prevented at decision time*. Whether +approvals should gate the decision is an unimplemented policy capability, not a +break in the evidence chain. + +**AC-035-F2 — Canonicalization is repeated several times per transaction.** +Validation, digest, and projection each canonicalize, and `verify_receipt` +re-runs the whole decision path. At ~96 µs per canonicalization this is the +dominant cost and the clearest optimization target. It is *correct* — recomputing +rather than trusting is the security property — but it is not currently *cached* +within a single transaction. + +**AC-035-F3 — No replay protection semantics exist.** +A replayed identical request produces an identical receipt. This documents +determinism, not replay *protection*: there is no nonce, sequence number, or +single-use semantics, so an intercepted receipt is indistinguishable from a +legitimately re-derived one. Recorded as an architectural gap, not a defect +against current specified behaviour. + +**AC-035-F4 — `*_mutated.json` fixture names are misleading.** +`banking_payment_specimen_contract_mutated.json` and +`..._admission_mutated.json` are mutated *and correctly re-sealed*, so they are +validly-bound variant contracts rather than mutation attacks. The first draft of +this suite expected them to refuse; that expectation was wrong, not the +implementation. Genuine post-binding mutation (stale digest) is correctly +refused with `AC_DIGEST`. + +--- + +## 9. Resource profile + +| Metric | Value | +|---|---| +| Interpreter + import cost | 60–123 ms (mean 80 ms), out-of-process | +| Peak traced memory, one decision | 6.07 KiB | +| Peak traced memory, one verification | 6.19 KiB | +| Process max RSS (whole benchmark) | ~63 MiB | + +Artifact sizes (compact JSON, bytes): + +| Artifact | Size | +|---|---:| +| Contract artifact | 1,179 | +| Contract body | 773 | +| Canonical contract bytes | 773 | +| Action | 148 | +| Fact bundle | 675 | +| Projection | 479 | +| Receipt | 715 | + +Process startup (~80 ms) is **more than two orders of magnitude larger** than a +single decision (~0.38 ms). Any deployment shape that pays interpreter startup +per decision would be dominated entirely by startup. + +--- + +## 10. Limitations + +- One synthetic banking specimen family. No real institutional rule has been + integrated or evaluated. +- One machine, one Python version, one OS. No cross-platform comparison. +- Single process. No concurrency, no distribution, no persistence. +- Scale curves are synthetic: generated by widening a real specimen, not drawn + from production corpora. +- Percentiles are nearest-rank, computed from in-process `perf_counter` samples. +- Memory figures are Python-level allocation (`tracemalloc`), not RSS + attributable per decision. + +## 11. Claim ceiling + +This benchmark establishes only what it measures. It does **not** establish +production readiness, regulatory correctness, legal correctness, universal +source-to-rule derivation, arbitrary-domain compatibility, security +certification, distributed scalability, formal proof, or industry-wide +superiority. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md new file mode 100644 index 0000000..988a995 --- /dev/null +++ b/docs/ROADMAP.md @@ -0,0 +1,163 @@ +# AuthContract roadmap + +Derived from the AC-035 benchmark run at commit +`e4e1a97509df1a66c44b090c0a0ca0a03907f4dc`. Every item below cites the +observed evidence that motivates it. Items with no supporting measurement are +placed under **Research / not yet established** and are explicitly not +commitments. + +Raw evidence: [`benchmarks/results/`](../benchmarks/results/) · +Analysis: [`docs/BENCHMARKS.md`](BENCHMARKS.md) + +--- + +## NOW +*Required to move from current MVP-alpha to a stronger developer-testable alpha.* + +### N1 — Cache canonicalization within a transaction + +| | | +|---|---| +| **Evidence** | Canonicalization costs ~96 µs mean (p50 88.6, p95 140.9) and is performed by validation, digest, and projection independently; `verify_receipt` then re-runs the entire decision path. Action check by contrast is 8.4 µs. Canonicalization dominates the ~761 µs end-to-end path. | +| **Limitation** | The same contract body is canonicalized several times per transaction with no intra-transaction reuse. | +| **Capability** | Canonicalize once per artifact per transaction and reuse the bytes across validation, digest, and projection. | +| **Acceptance** | End-to-end p50 improves measurably against the 701.6 µs recorded baseline with **zero** change to any correctness or adversarial specimen result, and digests remain byte-identical to the recorded baseline. | +| **Dependency** | None. | +| **Maturity impact** | Removes the dominant cost without touching security semantics. | + +### N2 — Decide whether admission approvals gate authorization + +| | | +|---|---| +| **Evidence** | Finding AC-035-F1: forged `admission.approvals` yields `ALLOW`. The binding holds — `admission_digest` and `receipt_digest` both change and cross-verification fails — but nothing gates on approval content at decision time. | +| **Limitation** | Approvals are carried and bound as evidence but are never evaluated. A reader may reasonably assume otherwise. | +| **Capability** | Either (a) implement approval evaluation as a declared gate, or (b) document explicitly that approvals are evidence-only at this maturity. | +| **Acceptance** | If (a): a specimen with insufficient approvals REFUSEs with a distinct reason code, with positive and negative fixtures. If (b): README and contract-shape docs state the boundary and a regression test pins it. | +| **Dependency** | None. | +| **Maturity impact** | Closes an ambiguity that currently invites over-reading of what the system enforces. | + +### N3 — Rename the misleading `*_mutated.json` fixtures + +| | | +|---|---| +| **Evidence** | Finding AC-035-F4: the fixtures are mutated *and re-sealed*, so they are validly-bound variants. The first draft of the benchmark suite mis-expected them to refuse. | +| **Limitation** | Fixture names actively suggest a security property they do not exercise. | +| **Capability** | Rename to reflect what they are (e.g. `..._variant_resealed.json`) and add genuinely stale-bound counterparts as committed fixtures. | +| **Acceptance** | Names describe content; the stale-binding attack exists as a fixture, not only as programmatic construction in the benchmark. | +| **Dependency** | None. | +| **Maturity impact** | Removes a documented tripwire for external evaluators. | + +### N4 — Publish the benchmark as a regression gate + +| | | +|---|---| +| **Evidence** | The suite runs in ~45 s and exits non-zero on any correctness failure; 342 existing tests pass alongside it. | +| **Limitation** | Benchmark results are a point-in-time artifact; nothing prevents silent regression of latency or of the 38-specimen adversarial matrix. | +| **Capability** | Run the correctness and adversarial matrices in CI; track latency with a tolerance band rather than a hard threshold. | +| **Acceptance** | CI fails on any adversarial regression; latency drift is reported without failing on noise. | +| **Dependency** | N1 (so the tracked baseline is the post-optimization one). | +| **Maturity impact** | Converts a one-off measurement into a standing guarantee. | + +--- + +## NEXT +*Required for realistic external engineering evaluation.* + +### X1 — Multi-contract evaluation and selection + +| | | +|---|---| +| **Evidence** | Scale curves recorded `NOT EVALUATED` for multi-contract corpora: the runtime evaluates one artifact per invocation and has no registry or cross-contract selection path. | +| **Limitation** | Any realistic deployment holds many contracts; none of that behaviour exists or can be measured. | +| **Capability** | A contract registry with deterministic selection and explicit conflict/overlap semantics. | +| **Acceptance** | Scaling measured across 1/10/100/1000 contracts; overlapping-scope conflicts resolve deterministically or refuse explicitly; `select_matching_projection` semantics are covered by adversarial specimens. | +| **Dependency** | None. | +| **Maturity impact** | Removes the largest single `NOT EVALUATED` gap. | + +### X2 — Replay protection semantics + +| | | +|---|---| +| **Evidence** | Finding AC-035-F3: replayed identical requests produce identical receipts. Determinism is confirmed; replay *protection* does not exist. | +| **Limitation** | An intercepted receipt is indistinguishable from a legitimately re-derived one. | +| **Capability** | Nonce, sequence, or single-use decision semantics, with an explicit statement of the threat model addressed. | +| **Acceptance** | A replayed request is distinguishable from a fresh one; adversarial specimens cover replay within and across freshness windows. | +| **Dependency** | Requires deciding whether the runtime may hold state — it is currently stateless. | +| **Maturity impact** | Addresses a threat class the current design does not cover at all. | + +### X3 — Reduce verification cost, or justify it explicitly + +| | | +|---|---| +| **Evidence** | Verification (446 µs mean) costs ~as much as the original decision (378 µs mean) because it recomputes every binding. | +| **Limitation** | A verifier-heavy workload costs the same as a decider-heavy one; that is a deliberate trade but is undocumented as a capacity-planning input. | +| **Capability** | Either reduce redundant work inside `verify_receipt` (subject to N1) while preserving zero trust in receipt fields, or document the cost as an intentional property with guidance. | +| **Acceptance** | Verification remains fully independent — no receipt field trusted — and either measurably improves or is documented with a capacity-planning note. | +| **Dependency** | N1. | +| **Maturity impact** | Makes verification cost a designed, stated property rather than an emergent one. | + +### X4 — Cross-platform and cross-version determinism + +| | | +|---|---| +| **Evidence** | Determinism is confirmed for one process, one platform, one Python version. §7 explicitly does not claim more. | +| **Limitation** | Canonical identity is only useful across parties if it is stable across their environments — untested. | +| **Capability** | Reproduce digests across Python versions, OSes, architectures, and ideally an independent implementation. | +| **Acceptance** | Identical contract digests and receipts across at least two Python versions and two OSes, published as evidence. | +| **Dependency** | None. | +| **Maturity impact** | Upgrades determinism from a local observation to a portable property — a precondition for third-party verification. | + +### X5 — Real external rule evaluation + +| | | +|---|---| +| **Evidence** | Every measurement uses one synthetic banking specimen family. The runbook's own claim ceiling states no external rule has been integrated. | +| **Limitation** | Nothing demonstrates the contract shape can express a rule authored outside this project. | +| **Capability** | Encode at least one externally-authored rule end-to-end and publish the gaps found. | +| **Acceptance** | An external rule is evaluated, and every unsupported construct is reported as a finding rather than worked around by reshaping the rule. | +| **Dependency** | Possibly X1. | +| **Maturity impact** | First evidence of generality beyond the synthetic specimen. | + +--- + +## LATER +*Required for production-class deployment.* + +### L1 — Concurrency and sustained-load behaviour +**Evidence:** throughput is single-process only; concurrency recorded `NOT EVALUATED`. +**Acceptance:** throughput and tail latency under sustained concurrent load, with a stated saturation point. +**Dependency:** X1. + +### L2 — Interpreter startup amortization +**Evidence:** startup ~80 ms vs ~0.38 ms per decision — a ~210× ratio. Any per-decision-process deployment is dominated by startup. +**Acceptance:** a long-lived service or batch shape whose measured per-decision cost approaches the in-process figure. +**Dependency:** L1. + +### L3 — Large-corpus operating envelope +**Evidence:** 10,000 facts → ~0.57 s and ~13 MiB per decision, linear. +**Acceptance:** documented supported envelope with measured limits, plus explicit refusal or degradation beyond it. +**Dependency:** N1, X1. + +### L4 — Independent security review +**Evidence:** 38 adversarial specimens pass, but all were authored by the same party that wrote the implementation. +**Acceptance:** external review with findings published unmodified. +**Dependency:** X4, X5. + +--- + +## RESEARCH / NOT YET ESTABLISHED +*Requires architecture, formalization, or experiment that does not exist. Not commitments.* + +- **Automated natural-language source-to-rule derivation.** Not implemented; no + measurement exists. Would require a validation methodology establishing that a + derived rule faithfully represents its source — itself an open problem. +- **Formal proof of the authorization semantics.** No formal model exists. The + 38-specimen battery is testing, not proof. +- **Cross-jurisdiction / cross-domain generality.** One synthetic banking family + measured. Any claim of generality needs evidence from materially different + domains. +- **Distributed consensus on decisions.** No distribution layer exists; there is + no architecture to evaluate. +- **Regulatory or legal sufficiency of receipts as evidence.** Entirely outside + what any measurement here can establish; requires legal analysis, not + benchmarking. diff --git a/docs/TRL-ASSESSMENT.md b/docs/TRL-ASSESSMENT.md new file mode 100644 index 0000000..0b1e547 --- /dev/null +++ b/docs/TRL-ASSESSMENT.md @@ -0,0 +1,104 @@ +# AuthContract — TRL assessment (AC-035) + +**Commit assessed:** `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` +**Basis:** the AC-035 benchmark run only. Architecture documents, design intent, +and roadmap items are explicitly **not** counted as evidence. + +--- + +## Evidence classification + +The distinction that matters for TRL is *what kind* of evidence exists, not how +much of the system is written. + +| Class | Status | What supports it | +|---|---|---| +| **Implemented** | YES | Canonical digest, projection, fact admission, decision path, receipt emission, receipt verification, Git merge-result gate. 2,401 lines across 8 runtime modules. | +| **Demonstrated** | YES | Complete E2E path executes: 7/7 E2E specimens, 38/38 adversarial specimens, 342 regression tests pass. | +| **Benchmarked** | YES | Per-stage latency distributions (n=1000–2000), throughput, two scale curves, determinism over 100 replays, resource profile. This document's basis. | +| **Externally validated** | **NO** | Every specimen, fixture, and expectation was authored by the same party that authored the implementation. No external rule has been encoded; no third party has reproduced or reviewed the results. Clean-room external *testability* was established in AC-028, but by this same executor — not by an independent evaluator. | +| **Production validated** | **NO** | No deployment, no real workload, no concurrency, no persistence, no operational history. | + +--- + +## Current TRL: **4**, with partial TRL 5 characteristics + +**TRL 4 — component and/or breadboard validation in a laboratory environment.** + +### Why TRL 4 is met + +- The complete architectural path runs end to end, not merely in parts. +- Behaviour is validated against an adversarial battery of 38 specimens + spanning missing/unknown fields, malformed types, out-of-domain values, + staleness, evidence divergence, digest mutation, substitution, and + reordering — all failing closed. +- Tamper detection is exhaustive over the protected surface: all 20 + mutation-and-truncation variants of the receipt are detected. +- Behaviour is deterministic and measured, not asserted: 100 replays produce + byte-identical receipts, and the digest and projection primitives each yield + exactly one distinct value across 100 runs. +- Performance is characterized with distributions rather than single timings, + and scaling is linear with no observed cliff across 1000× in two dimensions. + +### Why TRL 5 is *not* met + +TRL 5 requires validation in a **relevant** environment. Every element of the +measured environment is laboratory-constructed: + +- One synthetic banking specimen family; no externally-authored rule has ever + been evaluated (roadmap X5). +- Single process, single machine, no concurrency, no persistence, no + operational load (roadmap L1, L2). +- Multi-contract evaluation — the shape any real deployment takes — does not + exist and was recorded `NOT EVALUATED` (roadmap X1). +- Determinism is confirmed on exactly one platform and one Python version; + cross-environment reproducibility, which is what makes canonical identity + useful between parties, is untested (roadmap X4). + +### Why the partial TRL 5 characteristics are real + +The rigor of the *evidence discipline* exceeds typical TRL 4: measured +distributions rather than anecdotes, an adversarial matrix rather than +happy-path demos, findings recorded rather than silently repaired (AC-035-F1 +through F4), and explicit `NOT EVALUATED` markers rather than estimates. That +methodological maturity is a genuine TRL 5 characteristic. It does not by +itself raise the TRL, because TRL is determined by the *environment* the +evidence was gathered in, not by the quality of the measurement. + +--- + +## What would move AuthContract to TRL 5 + +All four are necessary; none alone is sufficient. + +1. **An externally-authored rule evaluated end to end** (roadmap X5), with every + unsupported construct reported as a finding rather than worked around by + reshaping the rule to fit. +2. **Multi-contract evaluation implemented and measured** (roadmap X1), + including deterministic selection and explicit conflict semantics — closing + the largest `NOT EVALUATED` gap. +3. **Cross-platform and cross-version determinism demonstrated** (roadmap X4): + identical digests and receipts across at least two Python versions and two + operating systems. Canonical identity that only holds in one environment + cannot support third-party verification. +4. **Independent reproduction by a party that did not author the system** + (roadmap L4) — someone else running the benchmark, on their hardware, + reaching the same correctness results. + +## What would move it to TRL 6 + +Beyond the above: sustained operation under concurrent load with a stated +saturation point (L1), a documented operating envelope with measured limits +(L3), and an independent security review whose findings are published +unmodified (L4). + +--- + +## Claim ceiling + +This assessment is bounded by the measurements in +[`docs/BENCHMARKS.md`](BENCHMARKS.md). It does not establish production +readiness, regulatory or legal correctness, universal source-to-rule +derivation, arbitrary-domain compatibility, security certification, distributed +scalability, formal correctness, or comparative standing against any other +system. From a7f6ba374b1362e624a3f8b912b265dd03da4cdd Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 13:32:56 +0000 Subject: [PATCH 2/3] AC-035A: fix benchmark provenance and measure real sustained throughput Two benchmark-method defects, no product change. A. Provenance. The harness is introduced by a later commit than the implementation it measures, so "which commit was benchmarked" could not be answered by HEAD alone, and the prior reproduction instructions told users to check out a commit at which benchmarks/run_benchmarks.py does not exist. Results now record DUT_BASE_SHA (the AuthContract implementation measured) and BENCHMARK_HARNESS_SHA (the commit containing the harness) as distinct values. verify_dut_unchanged() diffs the device-under-test paths against DUT_BASE_SHA before any measurement runs and refuses to proceed on drift, so the two SHAs are provably comparable rather than merely asserted. B. Sustained throughput. The prior rates were reciprocals of mean latency. That is arithmetic, not measurement: it assumes zero loop overhead and no drift under continuous operation. sustained_throughput() now runs a continuous single-threaded loop over a fixed wall-clock window and counts completed operations - 3 trials x 5s, 1s warmup, for decision+receipt, receipt verification, and complete E2E. Both figures are reported and kept separate; the latency-derived rate is relabelled LATENCY-DERIVED RATE. The two disagree, which is the point of measuring: observed E2E sustained throughput is ~1604/s median against a latency-derived 1478/s, because the per-sample timer overhead in the latency path is amortized by the continuous loop. No runtime, test, fixture, workflow, SOTA, or claim-ceiling change. Findings F1-F4 remain findings and are not repaired here. --- benchmarks/harness.py | 125 ++++++++++++++++++++++++++++++++++- benchmarks/run_benchmarks.py | 94 +++++++++++++++++++++++--- 2 files changed, 206 insertions(+), 13 deletions(-) diff --git a/benchmarks/harness.py b/benchmarks/harness.py index 6373d92..eda4746 100644 --- a/benchmarks/harness.py +++ b/benchmarks/harness.py @@ -146,14 +146,135 @@ def measure_cold(operation: Callable[[], Any]) -> dict[str, Any]: return {"unit": "microseconds", "single_cold_execution": round((time.perf_counter() - start) * 1e6, 3)} -def throughput_per_second(summary: dict[str, Any]) -> float: - """Derive sustained single-process rate from a mean latency in microseconds.""" +def latency_derived_rate(summary: dict[str, Any]) -> float: + """Reciprocal of mean latency. This is an ARITHMETIC DERIVATION, not an + observed throughput measurement — it assumes zero per-iteration overhead + and no drift over time. Report it only alongside `sustained_throughput`, + never as a substitute for it.""" mean_seconds = summary["mean"] / 1e6 if mean_seconds <= 0: return float("inf") return round(1.0 / mean_seconds, 1) +def sustained_throughput( + operation: Callable[[], Any], + *, + duration_seconds: float = 5.0, + warmup_seconds: float = 1.0, + trials: int = 3, +) -> dict[str, Any]: + """Observed sustained rate: run `operation` continuously for a fixed wall-clock + window and count completed operations. + + This is a genuine throughput measurement rather than a reciprocal of mean + latency: it includes loop overhead, allocator behaviour, and any drift that + appears only under continuous operation, none of which a latency reciprocal + captures. Single process, single thread, no concurrency. + """ + trial_results: list[dict[str, Any]] = [] + + for trial_index in range(trials): + warmup_deadline = time.perf_counter() + warmup_seconds + while time.perf_counter() < warmup_deadline: + operation() + + operations = 0 + start = time.perf_counter() + deadline = start + duration_seconds + while time.perf_counter() < deadline: + operation() + operations += 1 + elapsed = time.perf_counter() - start + + trial_results.append( + { + "trial": trial_index + 1, + "operations": operations, + "elapsed_seconds": round(elapsed, 4), + "operations_per_second": round(operations / elapsed, 1), + } + ) + + rates = sorted(trial["operations_per_second"] for trial in trial_results) + return { + "method": "observed sustained rate — continuous single-threaded loop over a fixed window", + "warmup_seconds": warmup_seconds, + "measurement_seconds_per_trial": duration_seconds, + "trials": trials, + "trial_detail": trial_results, + "total_operations": sum(trial["operations"] for trial in trial_results), + "min_ops_per_second": rates[0], + "median_ops_per_second": statistics.median(rates), + "max_ops_per_second": rates[-1], + } + + +# -------------------------------------------------------------------------- +# device-under-test provenance (AC-035A) +# -------------------------------------------------------------------------- + +# Paths that constitute the device under test. The benchmark measures these and +# must not modify them; the harness itself lives outside this set. +DUT_PATHS = ( + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md", +) + + +def verify_dut_unchanged(dut_base_sha: str) -> dict[str, Any]: + """Prove the measured product files are byte-identical to `dut_base_sha`. + + The harness is introduced by a later commit than the implementation it + measures, so "which commit was measured" cannot be answered by HEAD alone. + This diffs only the DUT paths between HEAD and the declared base: an empty + diff establishes that the harness commit changed nothing under measurement. + """ + try: + # Deliberately diff against the WORKING TREE, not against HEAD: the + # benchmark imports and measures the files on disk, so a comparison + # between two commits would report "verified" while an uncommitted edit + # silently changed what was actually measured. Omitting the second + # revision makes git compare dut_base_sha to the working tree, catching + # committed and uncommitted drift alike. + completed = subprocess.run( + ["git", "diff", "--name-only", dut_base_sha, "--", *DUT_PATHS], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ) + except (subprocess.CalledProcessError, FileNotFoundError) as exc: + return { + "verified": False, + "error": f"could not diff against {dut_base_sha}: {exc}", + "dut_paths": list(DUT_PATHS), + } + + modified = [line for line in completed.stdout.splitlines() if line.strip()] + return { + "verified": not modified, + "dut_base_sha": dut_base_sha, + "dut_paths": list(DUT_PATHS), + "modified_dut_files": modified, + "statement": ( + f"All device-under-test paths are byte-identical to {dut_base_sha}; the " + "benchmark harness changed nothing under measurement." + if not modified + else f"DUT DRIFT: {len(modified)} file(s) differ from {dut_base_sha}. " + "Results do NOT describe that commit." + ), + } + + # -------------------------------------------------------------------------- # memory measurement (Phase 9) # -------------------------------------------------------------------------- diff --git a/benchmarks/run_benchmarks.py b/benchmarks/run_benchmarks.py index 9ed1502..bc9dfb6 100644 --- a/benchmarks/run_benchmarks.py +++ b/benchmarks/run_benchmarks.py @@ -29,6 +29,7 @@ from harness import ( # noqa: E402 REPO_ROOT, capture_environment, + latency_derived_rate, load_fixture, load_fixture_text, make_action_scaled_specimen, @@ -36,7 +37,8 @@ measure, measure_cold, measure_peak_memory, - throughput_per_second, + sustained_throughput, + verify_dut_unchanged, write_result, ) from specimens import ADVERSARIAL_SPECIMENS, E2E_SPECIMENS # noqa: E402 @@ -47,6 +49,18 @@ EXECUTION_RESULT = "SIMULATED_SUCCESS" +# The AuthContract implementation under measurement. The harness that measures +# it is introduced by a LATER commit, so these are deliberately distinct values: +# DUT_BASE_SHA identifies the product; the harness SHA is captured at runtime. +# `verify_dut_unchanged` proves the harness commit modified nothing under +# measurement, which is what makes the two safely comparable. +DUT_BASE_SHA = "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc" + +# Sustained-throughput measurement window. +THROUGHPUT_TRIALS = 3 +THROUGHPUT_SECONDS = 5.0 +THROUGHPUT_WARMUP_SECONDS = 1.0 + # Repetition counts. Chosen so each stage's sample is large enough for a stable # p99 without making the suite take longer than a developer will tolerate. STAGE_REPS = 2000 @@ -368,12 +382,17 @@ def full_e2e() -> None: "complete_end_to_end_cold": measure_cold(full_e2e), } - throughput = { - "note": "Single-process, single-threaded, derived from warm mean latency. Not a distributed or multi-core claim.", - "decisions_per_second": throughput_per_second(stages["decision_and_receipt"]["summary"]), - "receipts_per_second": throughput_per_second(stages["decision_and_receipt"]["summary"]), - "receipt_verifications_per_second": throughput_per_second(stages["receipt_verification"]["summary"]), - "complete_e2e_transactions_per_second": throughput_per_second(stages["complete_end_to_end"]["summary"]), + latency_derived = { + "method": "LATENCY-DERIVED RATE — reciprocal of warm mean latency, NOT an observed rate", + "caveat": ( + "This is arithmetic, not measurement: it assumes zero loop overhead and no " + "drift under continuous operation. Compare against observed_sustained_throughput " + "below; where they disagree, the observed figure is the real one." + ), + "decisions_per_second": latency_derived_rate(stages["decision_and_receipt"]["summary"]), + "receipts_per_second": latency_derived_rate(stages["decision_and_receipt"]["summary"]), + "receipt_verifications_per_second": latency_derived_rate(stages["receipt_verification"]["summary"]), + "complete_e2e_transactions_per_second": latency_derived_rate(stages["complete_end_to_end"]["summary"]), "receipts_per_second_caveat": ( "Receipt emission is not separately callable at this commit: run_specimen " "decides and emits in one pass, so decisions/sec and receipts/sec are the " @@ -381,7 +400,37 @@ def full_e2e() -> None: ), } - return {"stages": stages, "cold_vs_warm": cold, "throughput": throughput} + observed = { + "note": ( + "Observed sustained rates: continuous single-process, single-threaded loops over " + "fixed wall-clock windows. No concurrency. Not a distributed or multi-core claim." + ), + "decision_and_receipt": sustained_throughput( + lambda: _run(artifact, action, facts), + duration_seconds=THROUGHPUT_SECONDS, + warmup_seconds=THROUGHPUT_WARMUP_SECONDS, + trials=THROUGHPUT_TRIALS, + ), + "receipt_verification": sustained_throughput( + lambda: verify_receipt(receipt, artifact, action, facts), + duration_seconds=THROUGHPUT_SECONDS, + warmup_seconds=THROUGHPUT_WARMUP_SECONDS, + trials=THROUGHPUT_TRIALS, + ), + "complete_end_to_end": sustained_throughput( + full_e2e, + duration_seconds=THROUGHPUT_SECONDS, + warmup_seconds=THROUGHPUT_WARMUP_SECONDS, + trials=THROUGHPUT_TRIALS, + ), + } + + return { + "stages": stages, + "cold_vs_warm": cold, + "latency_derived_rate": latency_derived, + "observed_sustained_throughput": observed, + } # -------------------------------------------------------------------------- @@ -793,7 +842,25 @@ def main() -> int: started = time.time() environment = capture_environment() - print("AC-035 benchmark — commit", environment["commit_sha"][:12]) + provenance = { + "dut_base_sha": DUT_BASE_SHA, + "benchmark_harness_sha": environment["commit_sha"], + "note": ( + "DUT_BASE_SHA is the AuthContract implementation being measured. " + "BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it — " + "necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. " + "Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + ), + "dut_verification": verify_dut_unchanged(DUT_BASE_SHA), + } + + if not provenance["dut_verification"]["verified"]: + print("REFUSING TO RUN:", provenance["dut_verification"]["statement"], file=sys.stderr) + return 2 + + print("AC-035 benchmark") + print(" DUT :", DUT_BASE_SHA[:12], "(verified unchanged)") + print(" harness :", environment["commit_sha"][:12]) print(" phase: end-to-end + correctness matrix") e2e = phase_e2e_and_correctness() print(" phase: performance") @@ -808,7 +875,8 @@ def main() -> int: resources = phase_resources() common = { - "work_order": "AC-035", + "work_order": "AC-035 / AC-035A", + "provenance": provenance, "environment": environment, "claim_ceiling": CLAIM_CEILING, "generated_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), @@ -829,7 +897,11 @@ def main() -> int: print(f" E2E: {e2e_totals['passed']}/{e2e_totals['total']} passed") print(f" Adversarial: {adv_totals['passed']}/{adv_totals['total']} passed") print(f" E2E p50: {performance['stages']['complete_end_to_end']['summary']['p50']} us") - print(f" E2E tps: {performance['throughput']['complete_e2e_transactions_per_second']}") + print( + " E2E tps: " + f"{performance['observed_sustained_throughput']['complete_end_to_end']['median_ops_per_second']} observed median" + f" (latency-derived {performance['latency_derived_rate']['complete_e2e_transactions_per_second']})" + ) print(f" Determinism: {'stable' if determinism['fully_deterministic_over_fixed_inputs'] else 'NOT STABLE'}") print(f" elapsed: {time.time() - started:.1f}s") From 9c87f61215f34cedac45305535a29973ed18c861 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 13:45:37 +0000 Subject: [PATCH 3/3] AC-035A: refresh results and docs from the corrected harness run One coherent run of the AC-035A harness (a7f6ba3) against DUT e4e1a97, verified byte-identical before measuring. No cherry-picking across runs. Docs now record DUT_BASE_SHA and BENCHMARK_HARNESS_SHA separately and give reproduction instructions that actually work: check out the harness commit, not the DUT commit, because benchmarks/ does not exist at the DUT commit. The prior instruction was wrong and is called out as such. Throughput is reported twice and kept apart: observed sustained rate (the measurement) and latency-derived rate (the arithmetic). On this run they agree to within 0.3% on the end-to-end path; the direction of the small gap is not stable across runs, and that is stated rather than smoothed over. Also fixes a real defect in the drift guard added earlier in this work order: it diffed DUT_BASE_SHA against HEAD, which compares two commits and therefore reported "verified unchanged" while an uncommitted edit to authcontract/ silently changed what was actually measured. It now diffs against the working tree. Negative-tested: appending a line to authcontract/digest.py makes the harness refuse with exit 2 and name the drifted file. Results: 7/7 E2E, 38/38 adversarial, determinism stable across 100 replays with all 10 protected receipt fields byte-identical, 342 pytest passed. Findings F1-F4 remain findings; none repaired here. TRL remains 4 - methodology improved, environment did not. --- benchmarks/README.md | 47 +- .../results/AC-035-CORRECTNESS-MATRIX.json | 31 +- benchmarks/results/AC-035-DETERMINISM.json | 31 +- benchmarks/results/AC-035-E2E-RESULTS.json | 31 +- benchmarks/results/AC-035-PERFORMANCE.json | 411 +++++++++++------- .../results/AC-035-RESOURCE-PROFILE.json | 49 ++- docs/BENCHMARKS.md | 133 ++++-- docs/ROADMAP.md | 17 +- docs/TRL-ASSESSMENT.md | 21 +- 9 files changed, 533 insertions(+), 238 deletions(-) diff --git a/benchmarks/README.md b/benchmarks/README.md index c613443..4544a38 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -6,19 +6,48 @@ This suite measures **only capabilities that exist** at the commit under test. Where the architecture cannot express a dimension, the result is recorded as `NOT EVALUATED` rather than estimated. +## Two commits, and why that matters + +A benchmark harness cannot exist at the commit it measures — it is written +afterwards. So there are two distinct SHAs, and conflating them makes the +results unreproducible: + +| | | +|---|---| +| **`DUT_BASE_SHA`** | `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` — the AuthContract implementation **being measured** | +| **`BENCHMARK_HARNESS_SHA`** | `a7f6ba374b1362e624a3f8b912b265dd03da4cdd` — the commit containing the **harness that measured it** | + +**Check out the harness commit, not the DUT commit.** `benchmarks/` does not +exist at `e4e1a975`; an earlier draft of this file said otherwise and was +wrong. + +What makes the two safely comparable is not assertion but verification: before +any measurement runs, `verify_dut_unchanged()` diffs every device-under-test +path (`authcontract/`, `tests/`, `fixtures/`, `.github/`, `pyproject.toml`, +`README.md`, and the SOTA/language/runbook docs) against `DUT_BASE_SHA`. If any +byte differs, the harness **refuses to run** and exits 2, because results that +silently measured a drifted tree would not describe the commit they claim to. + ## Reproduce ```bash git clone https://github.com/veraxis-protocol/AuthContract.git cd AuthContract -git checkout e4e1a97509df1a66c44b090c0a0ca0a03907f4dc +git checkout a7f6ba374b1362e624a3f8b912b265dd03da4cdd # BENCHMARK_HARNESS_SHA python3 -m venv .venv && source .venv/bin/activate pip install -e ".[test]" python3 benchmarks/run_benchmarks.py ``` -Runtime is roughly 45 seconds. Results are written to `benchmarks/results/`. -The process exits non-zero if any correctness specimen fails. +Equivalently, check out the AC-035 branch head — the harness code is unchanged +between `a7f6ba3` and the branch tip, which adds only refreshed results and +documentation. + +The run prints the DUT and harness SHAs it verified before measuring. Runtime +is roughly 100 seconds (the sustained-throughput phase alone is 3 operations × +3 trials × 6 s). Results are written to `benchmarks/results/`. Exit codes: `0` all +specimens pass, `1` a correctness specimen failed, `2` DUT drift detected and +nothing was measured. ## Layout @@ -42,8 +71,16 @@ as n/min/p50/p95/p99/max/mean/stdev in microseconds. Very fast stages use an inner batch so they are measured above timer resolution rather than quantized to zero. -**Throughput.** Single-process, single-threaded, derived from warm mean -latency. Not a distributed or multi-core claim. +**Throughput — two figures, deliberately kept apart.** + +- *Latency-derived rate*: the reciprocal of warm mean latency. This is + arithmetic, not measurement — it assumes zero loop overhead and no drift. +- *Observed sustained rate*: a continuous single-threaded loop over a fixed + wall-clock window (3 trials × 5 s, 1 s warmup each), counting completed + operations. + +Both are reported. Where they disagree, the observed figure is the real one. +Neither is a concurrency or distribution claim — no such layer exists. **Scale curves.** Two dimensions the architecture can genuinely express: declared mediated actions (1/10/100/1000) and required facts (10/100/1000/10000). diff --git a/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json b/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json index ad21627..0d490ba 100644 --- a/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json +++ b/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json @@ -691,7 +691,7 @@ } ], "environment": { - "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", "dependency_versions": { "pytest": "9.1.1", "rfc8785": "0.1.4" @@ -702,8 +702,31 @@ "processor": "x86_64", "python_implementation": "CPython", "python_version": "3.11.15", - "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" }, - "generated_at_utc": "2026-08-24T13:12:02Z", - "work_order": "AC-035" + "generated_at_utc": "2026-08-24T13:43:12Z", + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, + "work_order": "AC-035 / AC-035A" } diff --git a/benchmarks/results/AC-035-DETERMINISM.json b/benchmarks/results/AC-035-DETERMINISM.json index 37469a2..938171a 100644 --- a/benchmarks/results/AC-035-DETERMINISM.json +++ b/benchmarks/results/AC-035-DETERMINISM.json @@ -61,7 +61,7 @@ } }, "environment": { - "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", "dependency_versions": { "pytest": "9.1.1", "rfc8785": "0.1.4" @@ -72,8 +72,31 @@ "processor": "x86_64", "python_implementation": "CPython", "python_version": "3.11.15", - "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" }, - "generated_at_utc": "2026-08-24T13:12:02Z", - "work_order": "AC-035" + "generated_at_utc": "2026-08-24T13:43:12Z", + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, + "work_order": "AC-035 / AC-035A" } diff --git a/benchmarks/results/AC-035-E2E-RESULTS.json b/benchmarks/results/AC-035-E2E-RESULTS.json index 24fa450..3a3b03e 100644 --- a/benchmarks/results/AC-035-E2E-RESULTS.json +++ b/benchmarks/results/AC-035-E2E-RESULTS.json @@ -214,7 +214,7 @@ } }, "environment": { - "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", "dependency_versions": { "pytest": "9.1.1", "rfc8785": "0.1.4" @@ -225,8 +225,31 @@ "processor": "x86_64", "python_implementation": "CPython", "python_version": "3.11.15", - "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" }, - "generated_at_utc": "2026-08-24T13:12:02Z", - "work_order": "AC-035" + "generated_at_utc": "2026-08-24T13:43:12Z", + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, + "work_order": "AC-035 / AC-035A" } diff --git a/benchmarks/results/AC-035-PERFORMANCE.json b/benchmarks/results/AC-035-PERFORMANCE.json index 5e1bbc7..4d76751 100644 --- a/benchmarks/results/AC-035-PERFORMANCE.json +++ b/benchmarks/results/AC-035-PERFORMANCE.json @@ -12,7 +12,7 @@ "Latency and throughput figures are single-process, single-machine, and environment-specific." ], "environment": { - "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", "dependency_versions": { "pytest": "9.1.1", "rfc8785": "0.1.4" @@ -23,29 +23,131 @@ "processor": "x86_64", "python_implementation": "CPython", "python_version": "3.11.15", - "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" }, - "generated_at_utc": "2026-08-24T13:12:02Z", + "generated_at_utc": "2026-08-24T13:43:12Z", "performance": { "cold_vs_warm": { "complete_end_to_end_cold": { - "single_cold_execution": 948.726, + "single_cold_execution": 598.022, "unit": "microseconds" }, "note": "Single un-warmed execution in an already-imported interpreter. The dominant one-time cost is interpreter startup and module import, reported separately in the resource profile." }, + "latency_derived_rate": { + "caveat": "This is arithmetic, not measurement: it assumes zero loop overhead and no drift under continuous operation. Compare against observed_sustained_throughput below; where they disagree, the observed figure is the real one.", + "complete_e2e_transactions_per_second": 1579.6, + "decisions_per_second": 3532.7, + "method": "LATENCY-DERIVED RATE \u2014 reciprocal of warm mean latency, NOT an observed rate", + "receipt_verifications_per_second": 3382.0, + "receipts_per_second": 3532.7, + "receipts_per_second_caveat": "Receipt emission is not separately callable at this commit: run_specimen decides and emits in one pass, so decisions/sec and receipts/sec are the same measurement reported twice, not two independent figures." + }, + "observed_sustained_throughput": { + "complete_end_to_end": { + "max_ops_per_second": 1587.5, + "measurement_seconds_per_trial": 5.0, + "median_ops_per_second": 1574.4, + "method": "observed sustained rate \u2014 continuous single-threaded loop over a fixed window", + "min_ops_per_second": 1512.9, + "total_operations": 23375, + "trial_detail": [ + { + "elapsed_seconds": 5.0003, + "operations": 7565, + "operations_per_second": 1512.9, + "trial": 1 + }, + { + "elapsed_seconds": 5.0002, + "operations": 7938, + "operations_per_second": 1587.5, + "trial": 2 + }, + { + "elapsed_seconds": 5.0001, + "operations": 7872, + "operations_per_second": 1574.4, + "trial": 3 + } + ], + "trials": 3, + "warmup_seconds": 1.0 + }, + "decision_and_receipt": { + "max_ops_per_second": 3423.9, + "measurement_seconds_per_trial": 5.0, + "median_ops_per_second": 3383.8, + "method": "observed sustained rate \u2014 continuous single-threaded loop over a fixed window", + "min_ops_per_second": 3258.9, + "total_operations": 50335, + "trial_detail": [ + { + "elapsed_seconds": 5.0004, + "operations": 17121, + "operations_per_second": 3423.9, + "trial": 1 + }, + { + "elapsed_seconds": 5.0, + "operations": 16919, + "operations_per_second": 3383.8, + "trial": 2 + }, + { + "elapsed_seconds": 5.0002, + "operations": 16295, + "operations_per_second": 3258.9, + "trial": 3 + } + ], + "trials": 3, + "warmup_seconds": 1.0 + }, + "note": "Observed sustained rates: continuous single-process, single-threaded loops over fixed wall-clock windows. No concurrency. Not a distributed or multi-core claim.", + "receipt_verification": { + "max_ops_per_second": 3237.4, + "measurement_seconds_per_trial": 5.0, + "median_ops_per_second": 3212.9, + "method": "observed sustained rate \u2014 continuous single-threaded loop over a fixed window", + "min_ops_per_second": 3178.2, + "total_operations": 48145, + "trial_detail": [ + { + "elapsed_seconds": 5.0004, + "operations": 16066, + "operations_per_second": 3212.9, + "trial": 1 + }, + { + "elapsed_seconds": 5.0003, + "operations": 15892, + "operations_per_second": 3178.2, + "trial": 2 + }, + { + "elapsed_seconds": 5.0001, + "operations": 16187, + "operations_per_second": 3237.4, + "trial": 3 + } + ], + "trials": 3, + "warmup_seconds": 1.0 + } + }, "stages": { "action_check": { "summary": { "inner_batch": 20, - "max": 17.956, - "mean": 8.434, - "min": 6.581, + "max": 18.275, + "mean": 5.633, + "min": 4.914, "n": 2000, - "p50": 7.825, - "p95": 11.235, - "p99": 13.395, - "stdev": 1.488, + "p50": 5.049, + "p95": 8.507, + "p99": 10.81, + "stdev": 1.362, "unit": "microseconds", "warmup_discarded": 50 }, @@ -54,14 +156,14 @@ "canonical_digest": { "summary": { "inner_batch": 20, - "max": 180.803, - "mean": 98.778, - "min": 79.828, + "max": 226.511, + "mean": 88.619, + "min": 78.961, "n": 2000, - "p50": 87.856, - "p95": 158.604, - "p99": 166.782, - "stdev": 23.999, + "p50": 85.554, + "p95": 107.888, + "p99": 146.427, + "stdev": 11.555, "unit": "microseconds", "warmup_discarded": 50 }, @@ -70,14 +172,14 @@ "canonicalization": { "summary": { "inner_batch": 20, - "max": 186.59, - "mean": 96.423, - "min": 76.881, + "max": 428.039, + "mean": 90.825, + "min": 75.79, "n": 2000, - "p50": 88.61, - "p95": 140.864, - "p99": 158.497, - "stdev": 19.025, + "p50": 83.872, + "p95": 143.6, + "p99": 155.003, + "stdev": 19.863, "unit": "microseconds", "warmup_discarded": 50 }, @@ -86,14 +188,14 @@ "complete_end_to_end": { "summary": { "inner_batch": 1, - "max": 1519.071, - "mean": 760.837, - "min": 528.694, + "max": 1438.889, + "mean": 633.078, + "min": 520.309, "n": 1000, - "p50": 701.565, - "p95": 1083.29, - "p99": 1149.858, - "stdev": 172.467, + "p50": 574.002, + "p95": 1052.539, + "p99": 1111.175, + "stdev": 154.163, "unit": "microseconds", "warmup_discarded": 50 }, @@ -102,14 +204,14 @@ "contract_parse": { "summary": { "inner_batch": 20, - "max": 40.473, - "mean": 10.631, - "min": 8.826, + "max": 2091.211, + "mean": 10.92, + "min": 8.822, "n": 2000, - "p50": 9.559, - "p95": 15.088, - "p99": 17.988, - "stdev": 2.286, + "p50": 9.148, + "p95": 12.668, + "p99": 17.553, + "stdev": 46.568, "unit": "microseconds", "warmup_discarded": 50 }, @@ -118,14 +220,14 @@ "decision_and_receipt": { "summary": { "inner_batch": 1, - "max": 3808.82, - "mean": 377.622, - "min": 250.815, + "max": 576.538, + "mean": 283.067, + "min": 247.537, "n": 1000, - "p50": 376.382, - "p95": 484.69, - "p99": 562.39, - "stdev": 129.516, + "p50": 264.921, + "p95": 365.349, + "p99": 492.532, + "stdev": 46.925, "unit": "microseconds", "warmup_discarded": 50 }, @@ -134,14 +236,14 @@ "projection": { "summary": { "inner_batch": 20, - "max": 287.837, - "mean": 124.242, - "min": 92.451, + "max": 780.841, + "mean": 105.502, + "min": 89.451, "n": 2000, - "p50": 122.922, - "p95": 170.734, - "p99": 181.503, - "stdev": 22.71, + "p50": 99.535, + "p95": 141.815, + "p99": 184.272, + "stdev": 26.567, "unit": "microseconds", "warmup_discarded": 50 }, @@ -150,14 +252,14 @@ "projection_digest": { "summary": { "inner_batch": 20, - "max": 101.318, - "mean": 59.444, - "min": 40.9, + "max": 388.926, + "mean": 45.351, + "min": 40.102, "n": 2000, - "p50": 61.011, - "p95": 79.546, - "p99": 89.135, - "stdev": 10.971, + "p50": 42.922, + "p95": 60.748, + "p99": 77.546, + "stdev": 10.701, "unit": "microseconds", "warmup_discarded": 50 }, @@ -166,14 +268,14 @@ "receipt_verification": { "summary": { "inner_batch": 1, - "max": 769.033, - "mean": 445.598, - "min": 256.907, + "max": 679.022, + "mean": 295.685, + "min": 252.573, "n": 1000, - "p50": 453.604, - "p95": 568.965, - "p99": 634.309, - "stdev": 85.783, + "p50": 279.364, + "p95": 382.414, + "p99": 491.106, + "stdev": 49.273, "unit": "microseconds", "warmup_discarded": 50 }, @@ -182,29 +284,44 @@ "validation_and_binding": { "summary": { "inner_batch": 20, - "max": 934.733, - "mean": 112.399, - "min": 81.993, + "max": 2054.891, + "mean": 94.558, + "min": 79.758, "n": 2000, - "p50": 103.982, - "p95": 159.629, - "p99": 173.914, - "stdev": 30.209, + "p50": 86.937, + "p95": 123.164, + "p99": 157.781, + "stdev": 63.977, "unit": "microseconds", "warmup_discarded": 50 }, "what": "verify_artifact: digest-scope validation plus sibling binding agreement" } - }, - "throughput": { - "complete_e2e_transactions_per_second": 1314.3, - "decisions_per_second": 2648.2, - "note": "Single-process, single-threaded, derived from warm mean latency. Not a distributed or multi-core claim.", - "receipt_verifications_per_second": 2244.2, - "receipts_per_second": 2648.2, - "receipts_per_second_caveat": "Receipt emission is not separately callable at this commit: run_specimen decides and emits in one pass, so decisions/sec and receipts/sec are the same measurement reported twice, not two independent figures." } }, + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, "scaling": { "declared_action_scaling": { "dimension": "count of declared mediated actions in the projection domain", @@ -214,14 +331,14 @@ "declared_actions": 1, "latency": { "inner_batch": 1, - "max": 711.098, - "mean": 407.433, - "min": 247.138, + "max": 556.088, + "mean": 279.782, + "min": 244.412, "n": 400, - "p50": 424.174, - "p95": 532.946, - "p99": 616.399, - "stdev": 89.87, + "p50": 265.108, + "p95": 353.737, + "p99": 455.387, + "stdev": 41.235, "unit": "microseconds", "warmup_discarded": 10 }, @@ -236,14 +353,14 @@ "declared_actions": 10, "latency": { "inner_batch": 1, - "max": 2117.708, - "mean": 1201.454, - "min": 882.591, + "max": 1662.417, + "mean": 1008.521, + "min": 855.607, "n": 400, - "p50": 1111.978, - "p95": 1622.925, - "p99": 1842.1, - "stdev": 250.461, + "p50": 967.498, + "p95": 1303.346, + "p99": 1589.686, + "stdev": 140.887, "unit": "microseconds", "warmup_discarded": 10 }, @@ -258,14 +375,14 @@ "declared_actions": 100, "latency": { "inner_batch": 1, - "max": 14053.368, - "mean": 8489.413, - "min": 7275.667, + "max": 13444.785, + "mean": 8126.808, + "min": 7167.499, "n": 400, - "p50": 8070.566, - "p95": 10956.776, - "p99": 13454.636, - "stdev": 1220.737, + "p50": 7811.029, + "p95": 10430.793, + "p99": 12830.666, + "stdev": 1045.757, "unit": "microseconds", "warmup_discarded": 10 }, @@ -280,14 +397,14 @@ "declared_actions": 1000, "latency": { "inner_batch": 1, - "max": 114677.333, - "mean": 81994.852, - "min": 74569.494, + "max": 116275.614, + "mean": 78031.881, + "min": 71024.273, "n": 60, - "p50": 80423.713, - "p95": 100705.244, - "p99": 114677.333, - "stdev": 6847.287, + "p50": 76607.789, + "p95": 87792.025, + "p99": 116275.614, + "stdev": 6752.494, "unit": "microseconds", "warmup_discarded": 10 }, @@ -298,7 +415,7 @@ "reason_code": "OK" } ], - "observed_shape": "sublinear (size x1000 -> time x201.2)" + "observed_shape": "approximately linear (size x1000 -> time x278.9)" }, "not_evaluated": { "concurrent_or_distributed_throughput": "NOT EVALUATED \u2014 no concurrency or distribution layer exists at this commit.", @@ -312,20 +429,20 @@ "decision": "ALLOW", "latency": { "inner_batch": 1, - "max": 1590.885, - "mean": 1016.326, - "min": 719.762, + "max": 4685.441, + "mean": 953.728, + "min": 717.66, "n": 200, - "p50": 942.157, - "p95": 1447.004, - "p99": 1522.569, - "stdev": 220.994, + "p50": 913.82, + "p95": 1215.565, + "p99": 1396.101, + "stdev": 293.21, "unit": "microseconds", "warmup_discarded": 3 }, "peak_traced_memory": { - "peak_traced_bytes": 16262, - "peak_traced_kib": 15.88 + "peak_traced_bytes": 16204, + "peak_traced_kib": 15.82 }, "reason_code": "OK", "required_facts": 10 @@ -334,20 +451,20 @@ "decision": "ALLOW", "latency": { "inner_batch": 1, - "max": 9921.777, - "mean": 6530.551, - "min": 5596.428, + "max": 9229.832, + "mean": 6978.442, + "min": 5406.586, "n": 200, - "p50": 6306.753, - "p95": 8274.561, - "p99": 9235.273, - "stdev": 802.916, + "p50": 6916.054, + "p95": 8445.045, + "p99": 9076.396, + "stdev": 725.641, "unit": "microseconds", "warmup_discarded": 3 }, "peak_traced_memory": { - "peak_traced_bytes": 143861, - "peak_traced_kib": 140.49 + "peak_traced_bytes": 144093, + "peak_traced_kib": 140.72 }, "reason_code": "OK", "required_facts": 100 @@ -356,20 +473,20 @@ "decision": "ALLOW", "latency": { "inner_batch": 1, - "max": 71241.823, - "mean": 57994.791, - "min": 54316.178, + "max": 83138.987, + "mean": 68148.353, + "min": 63971.47, "n": 40, - "p50": 56670.824, - "p95": 64626.995, - "p99": 71241.823, - "stdev": 3765.218, + "p50": 67261.842, + "p95": 74742.939, + "p99": 83138.987, + "stdev": 3699.0, "unit": "microseconds", "warmup_discarded": 3 }, "peak_traced_memory": { - "peak_traced_bytes": 1364965, - "peak_traced_kib": 1332.97 + "peak_traced_bytes": 1363979, + "peak_traced_kib": 1332.01 }, "reason_code": "OK", "required_facts": 1000 @@ -378,14 +495,14 @@ "decision": "ALLOW", "latency": { "inner_batch": 1, - "max": 588081.258, - "mean": 566644.023, - "min": 550615.654, + "max": 671062.884, + "mean": 648173.111, + "min": 597079.984, "n": 5, - "p50": 563057.594, - "p95": 588081.258, - "p99": 588081.258, - "stdev": 15077.813, + "p50": 657265.367, + "p95": 671062.884, + "p99": 671062.884, + "stdev": 29974.09, "unit": "microseconds", "warmup_discarded": 3 }, @@ -397,8 +514,8 @@ "required_facts": 10000 } ], - "observed_shape": "approximately linear (size x1000 -> time x557.5)" + "observed_shape": "approximately linear (size x1000 -> time x679.6)" } }, - "work_order": "AC-035" + "work_order": "AC-035 / AC-035A" } diff --git a/benchmarks/results/AC-035-RESOURCE-PROFILE.json b/benchmarks/results/AC-035-RESOURCE-PROFILE.json index 3bd1c54..77b37e6 100644 --- a/benchmarks/results/AC-035-RESOURCE-PROFILE.json +++ b/benchmarks/results/AC-035-RESOURCE-PROFILE.json @@ -12,7 +12,7 @@ "Latency and throughput figures are single-process, single-machine, and environment-specific." ], "environment": { - "commit_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", "dependency_versions": { "pytest": "9.1.1", "rfc8785": "0.1.4" @@ -23,9 +23,32 @@ "processor": "x86_64", "python_implementation": "CPython", "python_version": "3.11.15", - "tree_sha": "9967077e7f9f9c661199c728c5a2e7fe496be07a" + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" + }, + "generated_at_utc": "2026-08-24T13:43:12Z", + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." }, - "generated_at_utc": "2026-08-24T13:12:02Z", "resources": { "artifact_sizes": { "action": 148, @@ -51,22 +74,22 @@ "process_max_rss": { "note": "whole-process peak RSS at end of benchmark run, including interpreter and harness", "unit": "kilobytes", - "value": 64804 + "value": 64708 }, "process_startup": { - "max": 122.74, - "mean": 79.64, - "min": 60.44, + "max": 87.95, + "mean": 67.66, + "min": 60.54, "samples": [ - 91.1, - 122.74, - 62.11, - 60.44, - 61.84 + 65.18, + 87.95, + 63.09, + 60.54, + 61.51 ], "unit": "milliseconds", "what": "python -c 'import authcontract.veip', out-of-process, 5 samples" } }, - "work_order": "AC-035" + "work_order": "AC-035 / AC-035A" } diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 862cb31..f74c213 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -1,10 +1,19 @@ # AuthContract benchmark baseline -**Commit measured:** `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` +**DUT_BASE_SHA (implementation measured):** `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` **Tree:** `9967077e7f9f9c661199c728c5a2e7fe496be07a` -**Work order:** AC-035 +**BENCHMARK_HARNESS_SHA (harness that measured it):** `a7f6ba374b1362e624a3f8b912b265dd03da4cdd` +**Work order:** AC-035, amended by AC-035A **Raw results:** [`benchmarks/results/`](../benchmarks/results/) +> **Provenance.** These are two different commits by necessity — the harness +> cannot exist at the commit it measures. Reproduce from +> `BENCHMARK_HARNESS_SHA`; `benchmarks/` does not exist at `DUT_BASE_SHA`. +> Before measuring, the harness diffs every device-under-test path against +> `DUT_BASE_SHA` and **refuses to run on any drift** (exit 2), so the claim +> "these numbers describe `e4e1a975`" is verified rather than asserted. This +> run: `verified: true`, zero modified DUT files. + This is the first end-to-end operational and performance baseline for AuthContract. It measures the bounded MVP-alpha path that exists today. It is not a marketing document and contains no projected or extrapolated figures. @@ -83,60 +92,92 @@ Warm figures, warmup discarded. Microseconds. | Stage | n | p50 | p95 | p99 | mean | |---|---:|---:|---:|---:|---:| -| contract parse | 2000 | 9.6 | 15.1 | 18.0 | 10.6 | -| validation + binding | 2000 | 104.0 | 159.6 | 173.9 | 112.4 | -| canonicalization (JCS) | 2000 | 88.6 | 140.9 | 158.5 | 96.4 | -| canonical digest | 2000 | 87.9 | 158.6 | 166.8 | 98.8 | -| projection | 2000 | 122.9 | 170.7 | 181.5 | 124.2 | -| projection digest | 2000 | 61.0 | 79.5 | 89.1 | 59.4 | -| action check | 2000 | 7.8 | 11.2 | 13.4 | 8.4 | -| decision + receipt | 1000 | 376.4 | 484.7 | 562.4 | 377.6 | -| receipt verification | 1000 | 453.6 | 569.0 | 634.3 | 445.6 | -| **complete end-to-end** | **1000** | **701.6** | **1083.3** | **1149.9** | **760.8** | +| contract parse | 2000 | 9.1 | 12.7 | 17.6 | 10.9 | +| validation + binding | 2000 | 86.9 | 123.2 | 157.8 | 94.6 | +| canonicalization (JCS) | 2000 | 83.9 | 143.6 | 155.0 | 90.8 | +| canonical digest | 2000 | 85.6 | 107.9 | 146.4 | 88.6 | +| projection | 2000 | 99.5 | 141.8 | 184.3 | 105.5 | +| projection digest | 2000 | 42.9 | 60.7 | 77.5 | 45.4 | +| action check | 2000 | 5.0 | 8.5 | 10.8 | 5.6 | +| decision + receipt | 1000 | 264.9 | 365.3 | 492.5 | 283.1 | +| receipt verification | 1000 | 279.4 | 382.4 | 491.1 | 295.7 | +| **complete end-to-end** | **1000** | **574.0** | **1052.5** | **1111.2** | **633.1** | Run-to-run variation on this shared machine is roughly ±15% on the end-to-end -figure (a prior run measured p50 616 µs). The relative cost of stages is stable -across runs; treat the absolute numbers as an order-of-magnitude baseline, not -a precise constant. +figure. The relative cost of stages is stable across runs; treat the absolute +numbers as an order-of-magnitude baseline, not a precise constant. Stage figures are **not additive** into the end-to-end figure: `decision + receipt` already contains projection and action check. ### What the distribution says -- **Canonicalization is the dominant primitive cost.** At ~96 µs it is roughly - 11× the action check and accounts for most of validation, digest, and +- **Canonicalization is the dominant primitive cost.** At ~91 µs it is roughly + 16× the action check and accounts for most of validation, digest, and projection cost. Every one of those stages canonicalizes. -- **The authorization logic itself is nearly free.** Action check is 8.4 µs — - about 1% of the end-to-end path. Cost is overwhelmingly in canonical identity, +- **The authorization logic itself is nearly free.** Action check is 5.6 µs — + under 1% of the end-to-end path. Cost is overwhelmingly in canonical identity, not in deciding. -- **Verification costs as much as deciding** (446 µs vs 378 µs), because +- **Verification costs as much as deciding** (296 µs vs 283 µs), because `verify_receipt` recomputes every binding from raw inputs rather than trusting any field in the receipt. This is the property that makes receipts independently checkable; the cost is the point, not a defect. -- **p99 is ~1.6× p50**, with no long tail — consistent with a pure-CPU path with +- **p99 is ~1.9× p50**, with no long tail — consistent with a pure-CPU path with no I/O, locking, or allocation cliffs. --- ## 5. Throughput -Single process, single thread, derived from warm mean latency. +Two figures, deliberately reported separately. Conflating them is how benchmarks +overstate capacity. + +### 5a. Observed sustained rate — the real measurement + +Continuous single-process, single-threaded loop over a fixed wall-clock window. +**3 trials × 5 s each, 1 s warmup per trial.** + +| Operation | min | **median** | max | total ops | +|---|---:|---:|---:|---:| +| Decision + receipt | 3,258.9 | **3,383.8** | 3,423.9 | 50,335 | +| Receipt verification | 3,178.2 | **3,212.9** | 3,237.4 | 48,145 | +| **Complete end-to-end** | **1,512.9** | **1,574.4** | **1,587.5** | **23,375** | + +Trial spread is under 7% for every operation, so the median is a stable +estimate rather than a lucky draw. + +### 5b. Latency-derived rate — arithmetic, not measurement -| Metric | Rate | +| Operation | Derived rate | |---|---:| -| Decisions / sec | 2,648 | -| Receipts / sec | 2,648 | -| Receipt verifications / sec | 2,244 | -| Complete E2E transactions / sec | 1,314 | +| Decisions / sec | 3,532.7 | +| Receipts / sec | 3,532.7 | +| Receipt verifications / sec | 3,382.0 | +| Complete E2E transactions / sec | 1,579.6 | + +This is the reciprocal of mean latency. It assumes zero loop overhead and no +drift under continuous operation, so it is an *upper-bound estimate*, not an +observed capacity. + +### What the gap says + +The two agree to within ~0.3% on the end-to-end path (1,579.6 derived vs 1,574.4 +observed). The derived figure runs slightly high, which is the expected +direction: it excludes loop overhead that real sustained operation pays. The +direction is not perfectly consistent across runs — an earlier run had the +observed figure slightly *above* the derived one, because per-sample +`perf_counter` overhead in the latency measurement can itself exceed loop +overhead. Both readings sit inside run-to-run noise; the honest summary is that +the two methods agree closely here, and the observed figure is the one to quote. Decisions/sec and receipts/sec are the *same measurement*: receipt emission is not separately callable at this commit. A complete E2E transaction is decide-plus-independently-verify, which is why it is roughly half the decision rate. -**Not claimed:** distributed throughput, multi-core scaling, or sustained -throughput under concurrency. No concurrency layer exists at this commit. +**Not claimed:** distributed throughput, multi-core scaling, or throughput under +concurrency. No concurrency layer exists at this commit and none was introduced +to measure one. --- @@ -146,13 +187,13 @@ throughput under concurrency. No concurrency layer exists at this commit. | Actions | mean latency | peak traced memory | decision | |---:|---:|---:|---| -| 1 | 407 µs | 6.1 KiB | ALLOW | -| 10 | 1,202 µs | 9.4 KiB | ALLOW | -| 100 | 8,489 µs | 40.8 KiB | ALLOW | -| 1,000 | 81,995 µs | 366.1 KiB | ALLOW | +| 1 | 280 µs | 6.1 KiB | ALLOW | +| 10 | 1,009 µs | 9.4 KiB | ALLOW | +| 100 | 8,127 µs | 40.8 KiB | ALLOW | +| 1,000 | 78,032 µs | 366.1 KiB | ALLOW | -1000× the domain size costs ~201× the time and ~60× the memory. Growth is -linear in the large-N regime (the 100→1000 step costs ~9.7× for 10× the size) +1000× the domain size costs ~279× the time and ~60× the memory. Growth is +linear in the large-N regime (the 100→1000 step costs ~9.6× for 10× the size) with a fixed overhead that dominates at small N, which is why the endpoint ratio reads as sublinear. @@ -160,16 +201,16 @@ reads as sublinear. | Facts | mean latency | peak traced memory | decision | |---:|---:|---:|---| -| 10 | 1,016 µs | 15.9 KiB | ALLOW | -| 100 | 6,531 µs | 140.5 KiB | ALLOW | -| 1,000 | 57,995 µs | 1.3 MiB | ALLOW | -| 10,000 | 566,644 µs | 13.1 MiB | ALLOW | +| 10 | 954 µs | 15.8 KiB | ALLOW | +| 100 | 6,978 µs | 140.7 KiB | ALLOW | +| 1,000 | 68,148 µs | 1.3 MiB | ALLOW | +| 10,000 | 648,173 µs | 13.1 MiB | ALLOW | -1000× the fact count costs ~558× the time and ~847× the memory — linear in +1000× the fact count costs ~680× the time and ~852× the memory — linear in both, with no observed nonlinear degradation or cliff in the tested range. Each -10× step costs ~9–10× consistently. +10× step costs ~8–10× consistently. -**Practical reading:** a 10,000-fact contract takes ~0.57 s and ~13 MiB per +**Practical reading:** a 10,000-fact contract takes ~0.65 s and ~13 MiB per decision. That is workable for batch evaluation and marginal for interactive use. The linearity means cost is predictable; the constant is dominated by repeated canonicalization (§8, finding F2). @@ -223,7 +264,7 @@ break in the evidence chain. **AC-035-F2 — Canonicalization is repeated several times per transaction.** Validation, digest, and projection each canonicalize, and `verify_receipt` -re-runs the whole decision path. At ~96 µs per canonicalization this is the +re-runs the whole decision path. At ~91 µs per canonicalization this is the dominant cost and the clearest optimization target. It is *correct* — recomputing rather than trusting is the security property — but it is not currently *cached* within a single transaction. @@ -249,7 +290,7 @@ refused with `AC_DIGEST`. | Metric | Value | |---|---| -| Interpreter + import cost | 60–123 ms (mean 80 ms), out-of-process | +| Interpreter + import cost | 61–88 ms (mean 68 ms), out-of-process | | Peak traced memory, one decision | 6.07 KiB | | Peak traced memory, one verification | 6.19 KiB | | Process max RSS (whole benchmark) | ~63 MiB | @@ -266,8 +307,8 @@ Artifact sizes (compact JSON, bytes): | Projection | 479 | | Receipt | 715 | -Process startup (~80 ms) is **more than two orders of magnitude larger** than a -single decision (~0.38 ms). Any deployment shape that pays interpreter startup +Process startup (~68 ms) is **more than two orders of magnitude larger** than a +single decision (~0.28 ms). Any deployment shape that pays interpreter startup per decision would be dominated entirely by startup. --- diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 988a995..2ed52cb 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,7 +1,8 @@ # AuthContract roadmap -Derived from the AC-035 benchmark run at commit -`e4e1a97509df1a66c44b090c0a0ca0a03907f4dc`. Every item below cites the +Derived from the AC-035 benchmark run measuring implementation +`e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` (DUT), with harness +`a7f6ba374b1362e624a3f8b912b265dd03da4cdd`. Every item below cites the observed evidence that motivates it. Items with no supporting measurement are placed under **Research / not yet established** and are explicitly not commitments. @@ -18,10 +19,10 @@ Analysis: [`docs/BENCHMARKS.md`](BENCHMARKS.md) | | | |---|---| -| **Evidence** | Canonicalization costs ~96 µs mean (p50 88.6, p95 140.9) and is performed by validation, digest, and projection independently; `verify_receipt` then re-runs the entire decision path. Action check by contrast is 8.4 µs. Canonicalization dominates the ~761 µs end-to-end path. | +| **Evidence** | Canonicalization costs ~91 µs mean (p50 83.9, p95 143.6) and is performed by validation, digest, and projection independently; `verify_receipt` then re-runs the entire decision path. Action check by contrast is 5.6 µs. Canonicalization dominates the ~633 µs end-to-end path. | | **Limitation** | The same contract body is canonicalized several times per transaction with no intra-transaction reuse. | | **Capability** | Canonicalize once per artifact per transaction and reuse the bytes across validation, digest, and projection. | -| **Acceptance** | End-to-end p50 improves measurably against the 701.6 µs recorded baseline with **zero** change to any correctness or adversarial specimen result, and digests remain byte-identical to the recorded baseline. | +| **Acceptance** | End-to-end p50 improves measurably against the 574.0 µs recorded baseline with **zero** change to any correctness or adversarial specimen result, and digests remain byte-identical to the recorded baseline. | | **Dependency** | None. | | **Maturity impact** | Removes the dominant cost without touching security semantics. | @@ -51,7 +52,7 @@ Analysis: [`docs/BENCHMARKS.md`](BENCHMARKS.md) | | | |---|---| -| **Evidence** | The suite runs in ~45 s and exits non-zero on any correctness failure; 342 existing tests pass alongside it. | +| **Evidence** | The suite runs in ~100 s and exits non-zero on any correctness failure; 342 existing tests pass alongside it. | | **Limitation** | Benchmark results are a point-in-time artifact; nothing prevents silent regression of latency or of the 38-specimen adversarial matrix. | | **Capability** | Run the correctness and adversarial matrices in CI; track latency with a tolerance band rather than a hard threshold. | | **Acceptance** | CI fails on any adversarial regression; latency drift is reported without failing on noise. | @@ -89,7 +90,7 @@ Analysis: [`docs/BENCHMARKS.md`](BENCHMARKS.md) | | | |---|---| -| **Evidence** | Verification (446 µs mean) costs ~as much as the original decision (378 µs mean) because it recomputes every binding. | +| **Evidence** | Verification (296 µs mean) costs ~as much as the original decision (283 µs mean) because it recomputes every binding. | | **Limitation** | A verifier-heavy workload costs the same as a decider-heavy one; that is a deliberate trade but is undocumented as a capacity-planning input. | | **Capability** | Either reduce redundant work inside `verify_receipt` (subject to N1) while preserving zero trust in receipt fields, or document the cost as an intentional property with guidance. | | **Acceptance** | Verification remains fully independent — no receipt field trusted — and either measurably improves or is documented with a capacity-planning note. | @@ -129,12 +130,12 @@ Analysis: [`docs/BENCHMARKS.md`](BENCHMARKS.md) **Dependency:** X1. ### L2 — Interpreter startup amortization -**Evidence:** startup ~80 ms vs ~0.38 ms per decision — a ~210× ratio. Any per-decision-process deployment is dominated by startup. +**Evidence:** startup ~68 ms vs ~0.28 ms per decision — a ~240× ratio. Any per-decision-process deployment is dominated by startup. **Acceptance:** a long-lived service or batch shape whose measured per-decision cost approaches the in-process figure. **Dependency:** L1. ### L3 — Large-corpus operating envelope -**Evidence:** 10,000 facts → ~0.57 s and ~13 MiB per decision, linear. +**Evidence:** 10,000 facts → ~0.65 s and ~13 MiB per decision, linear. **Acceptance:** documented supported envelope with measured limits, plus explicit refusal or degradation beyond it. **Dependency:** N1, X1. diff --git a/docs/TRL-ASSESSMENT.md b/docs/TRL-ASSESSMENT.md index 0b1e547..15b4a55 100644 --- a/docs/TRL-ASSESSMENT.md +++ b/docs/TRL-ASSESSMENT.md @@ -1,7 +1,8 @@ # AuthContract — TRL assessment (AC-035) -**Commit assessed:** `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` -**Basis:** the AC-035 benchmark run only. Architecture documents, design intent, +**Implementation assessed (DUT):** `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` +**Measured by harness:** `a7f6ba374b1362e624a3f8b912b265dd03da4cdd` +**Basis:** the AC-035 benchmark run only, as amended by AC-035A. Architecture documents, design intent, and roadmap items are explicitly **not** counted as evidence. --- @@ -15,7 +16,7 @@ much of the system is written. |---|---|---| | **Implemented** | YES | Canonical digest, projection, fact admission, decision path, receipt emission, receipt verification, Git merge-result gate. 2,401 lines across 8 runtime modules. | | **Demonstrated** | YES | Complete E2E path executes: 7/7 E2E specimens, 38/38 adversarial specimens, 342 regression tests pass. | -| **Benchmarked** | YES | Per-stage latency distributions (n=1000–2000), throughput, two scale curves, determinism over 100 replays, resource profile. This document's basis. | +| **Benchmarked** | YES | Per-stage latency distributions (n=1000–2000), *observed* sustained throughput (3 trials × 5 s per operation, not merely derived from latency), two scale curves, determinism over 100 replays, resource profile — all against a DUT verified byte-identical to the declared base commit before measuring. This document's basis. | | **Externally validated** | **NO** | Every specimen, fixture, and expectation was authored by the same party that authored the implementation. No external rule has been encoded; no third party has reproduced or reviewed the results. Clean-room external *testability* was established in AC-028, but by this same executor — not by an independent evaluator. | | **Production validated** | **NO** | No deployment, no real workload, no concurrency, no persistence, no operational history. | @@ -60,10 +61,16 @@ measured environment is laboratory-constructed: The rigor of the *evidence discipline* exceeds typical TRL 4: measured distributions rather than anecdotes, an adversarial matrix rather than happy-path demos, findings recorded rather than silently repaired (AC-035-F1 -through F4), and explicit `NOT EVALUATED` markers rather than estimates. That -methodological maturity is a genuine TRL 5 characteristic. It does not by -itself raise the TRL, because TRL is determined by the *environment* the -evidence was gathered in, not by the quality of the measurement. +through F4), explicit `NOT EVALUATED` markers rather than estimates, observed +sustained throughput rather than a latency reciprocal, and a device-under-test +whose identity is verified before measurement rather than asserted afterwards. +That methodological maturity is a genuine TRL 5 characteristic. + +It does not by itself raise the TRL. **TRL is determined by the environment the +evidence was gathered in, not by the quality of the measurement** — improving +benchmark methodology (as AC-035A did) makes the TRL 4 assessment *better +supported*, not higher. External independent reproduction remains absent, which +is the binding constraint. ---