diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..4544a38 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,139 @@ +# AuthContract benchmark suite (AC-035) + +The first end-to-end operational and performance baseline for AuthContract. + +This suite measures **only capabilities that exist** at the commit under test. +Where the architecture cannot express a dimension, the result is recorded as +`NOT EVALUATED` rather than estimated. + +## Two commits, and why that matters + +A benchmark harness cannot exist at the commit it measures — it is written +afterwards. So there are two distinct SHAs, and conflating them makes the +results unreproducible: + +| | | +|---|---| +| **`DUT_BASE_SHA`** | `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` — the AuthContract implementation **being measured** | +| **`BENCHMARK_HARNESS_SHA`** | `a7f6ba374b1362e624a3f8b912b265dd03da4cdd` — the commit containing the **harness that measured it** | + +**Check out the harness commit, not the DUT commit.** `benchmarks/` does not +exist at `e4e1a975`; an earlier draft of this file said otherwise and was +wrong. + +What makes the two safely comparable is not assertion but verification: before +any measurement runs, `verify_dut_unchanged()` diffs every device-under-test +path (`authcontract/`, `tests/`, `fixtures/`, `.github/`, `pyproject.toml`, +`README.md`, and the SOTA/language/runbook docs) against `DUT_BASE_SHA`. If any +byte differs, the harness **refuses to run** and exits 2, because results that +silently measured a drifted tree would not describe the commit they claim to. + +## Reproduce + +```bash +git clone https://github.com/veraxis-protocol/AuthContract.git +cd AuthContract +git checkout a7f6ba374b1362e624a3f8b912b265dd03da4cdd # BENCHMARK_HARNESS_SHA +python3 -m venv .venv && source .venv/bin/activate +pip install -e ".[test]" +python3 benchmarks/run_benchmarks.py +``` + +Equivalently, check out the AC-035 branch head — the harness code is unchanged +between `a7f6ba3` and the branch tip, which adds only refreshed results and +documentation. + +The run prints the DUT and harness SHAs it verified before measuring. Runtime +is roughly 100 seconds (the sustained-throughput phase alone is 3 operations × +3 trials × 6 s). Results are written to `benchmarks/results/`. Exit codes: `0` all +specimens pass, `1` a correctness specimen failed, `2` DUT drift detected and +nothing was measured. + +## Layout + +| Path | Purpose | +|---|---| +| `run_benchmarks.py` | Entry point; runs every phase and writes results | +| `harness.py` | Timing, memory, environment capture, synthetic specimen generation | +| `specimens/__init__.py` | Declarative specimen table with expected behaviour classes | +| `results/` | Raw JSON output (committed, so results are auditable without re-running) | + +## What is measured + +**Path exercised.** Contract artifact → parse → digest-scope validation and +sibling binding → RFC 8785 canonicalization → SHA-256 contract digest → +deterministic projection → runtime fact admission → action check → +ALLOW/REFUSE decision → receipt emission → independent receipt verification → +mutation detection. + +**Stage latency.** Each stage timed separately with warmup discarded, reported +as n/min/p50/p95/p99/max/mean/stdev in microseconds. Very fast stages use an +inner batch so they are measured above timer resolution rather than quantized +to zero. + +**Throughput — two figures, deliberately kept apart.** + +- *Latency-derived rate*: the reciprocal of warm mean latency. This is + arithmetic, not measurement — it assumes zero loop overhead and no drift. +- *Observed sustained rate*: a continuous single-threaded loop over a fixed + wall-clock window (3 trials × 5 s, 1 s warmup each), counting completed + operations. + +Both are reported. Where they disagree, the observed figure is the real one. +Neither is a concurrency or distribution claim — no such layer exists. + +**Scale curves.** Two dimensions the architecture can genuinely express: +declared mediated actions (1/10/100/1000) and required facts (10/100/1000/10000). +Synthetic specimens are re-sealed with a recomputed contract digest so they are +genuinely valid — otherwise the curve would measure the refusal path. + +**Correctness and adversarial matrices.** Expectations are stated as behaviour +*classes* (ALLOW vs fail-closed REFUSE) rather than exact reason codes. The +class is the security-relevant property; the observed reason code is recorded +alongside as evidence, so a change in refusal taxonomy shows up in the results +without the assertion being weakened into a tautology. + +**Determinism.** 100 repeated executions per specimen, comparing decision, +reason code, contract digest, projection, and every protected receipt field. + +## Methodology notes worth knowing + +- **`receipt_verification` re-runs the entire decision path.** `verify_receipt` + deliberately recomputes every binding from raw inputs rather than trusting any + field in the receipt. That is why verification costs about as much as the + original decision — it is a correctness property, not an inefficiency. +- **`decision_and_receipt` supersets `projection` and `action_check`.** Stage + figures are not additive into the end-to-end figure. +- **`decisions_per_second` and `receipts_per_second` are the same measurement.** + Receipt emission is not separately callable at this commit; `run_specimen` + decides and emits in one pass. +- **Percentiles use nearest-rank.** For these sample sizes the difference from + an interpolating definition is far below the measurement noise floor. +- **Memory is `tracemalloc` peak**, i.e. Python-level allocation attributable to + one execution, excluding interpreter baseline and allocator caching. + +## A finding about the `*_mutated.json` fixtures + +The repository's `banking_payment_specimen_contract_mutated.json` and +`banking_payment_specimen_admission_mutated.json` fixtures are mutated **and +correctly re-sealed** — their recomputed digest matches their declared digest. +They are therefore validly-bound *variant* contracts, not post-binding mutation +attacks, and `ALLOW` is the correct outcome for them. + +The first draft of this suite expected them to refuse. That expectation was +wrong, not the implementation. The genuine attack — altering contract-bound +material while leaving the declared digest stale — is constructed +programmatically in the runner as `E2E-06`, `ADV-35`, and `ADV-36`, and is +correctly refused with `AC_DIGEST`. + +This is recorded rather than quietly corrected because the naming of those +fixtures invites exactly this misreading. + +## Claim ceiling + +These measurements establish only what they measure: a bounded MVP-alpha +implementation, one synthetic banking specimen family, one machine, one +process. They do **not** establish production readiness, regulatory or legal +correctness, universal source-to-rule derivation, arbitrary-domain +compatibility, security certification, distributed scalability, formal +correctness, or comparative superiority over any other system. diff --git a/benchmarks/harness.py b/benchmarks/harness.py new file mode 100644 index 0000000..eda4746 --- /dev/null +++ b/benchmarks/harness.py @@ -0,0 +1,376 @@ +"""Shared measurement primitives for the AC-035 benchmark suite. + +Deliberately dependency-free (stdlib only) so the benchmark reproduces in the +same clean-room environment the runtime itself targets. Nothing here imports +from the benchmark bodies, so timing code stays separable from the specimens +being timed. +""" + +from __future__ import annotations + +import copy +import json +import platform +import statistics +import subprocess +import sys +import time +import tracemalloc +from pathlib import Path +from typing import Any, Callable + +REPO_ROOT = Path(__file__).resolve().parent.parent +FIXTURES = REPO_ROOT / "fixtures" + + +# -------------------------------------------------------------------------- +# fixture loading +# -------------------------------------------------------------------------- + +def load_fixture(relative: str) -> Any: + """Load a fixture as parsed JSON. Path is relative to fixtures/.""" + with (FIXTURES / relative).open(encoding="utf-8") as handle: + return json.load(handle) + + +def load_fixture_text(relative: str) -> str: + return (FIXTURES / relative).read_text(encoding="utf-8") + + +# -------------------------------------------------------------------------- +# environment capture (Phase 1) +# -------------------------------------------------------------------------- + +def _git(*args: str) -> str: + try: + out = subprocess.run( + ["git", *args], cwd=REPO_ROOT, capture_output=True, text=True, check=True + ) + return out.stdout.strip() + except (subprocess.CalledProcessError, FileNotFoundError): + return "UNAVAILABLE" + + +def _installed_versions() -> dict[str, str]: + """Report versions of the packages actually imported by the runtime path.""" + versions: dict[str, str] = {} + for name in ("rfc8785", "pytest"): + try: + module = __import__(name) + versions[name] = getattr(module, "__version__", "UNKNOWN") + except ImportError: + versions[name] = "NOT INSTALLED" + return versions + + +def capture_environment() -> dict[str, Any]: + return { + "commit_sha": _git("rev-parse", "HEAD"), + "tree_sha": _git("rev-parse", "HEAD^{tree}"), + "git_status_clean": _git("status", "--porcelain") == "", + "python_version": sys.version.split()[0], + "python_implementation": platform.python_implementation(), + "platform": platform.platform(), + "machine": platform.machine(), + "processor": platform.processor() or "UNKNOWN", + "dependency_versions": _installed_versions(), + } + + +# -------------------------------------------------------------------------- +# latency measurement (Phase 3) +# -------------------------------------------------------------------------- + +def _percentile(sorted_values: list[float], fraction: float) -> float: + """Nearest-rank percentile. Deterministic and dependency-free; for the + sample sizes used here the difference from an interpolating definition is + far below the measurement noise floor.""" + if not sorted_values: + raise ValueError("percentile of empty sample") + rank = max(1, min(len(sorted_values), int(round(fraction * len(sorted_values) + 0.5)))) + return sorted_values[rank - 1] + + +def summarize(samples_seconds: list[float]) -> dict[str, Any]: + """Distribution summary in microseconds.""" + micros = sorted(value * 1e6 for value in samples_seconds) + return { + "n": len(micros), + "unit": "microseconds", + "min": round(micros[0], 3), + "p50": round(_percentile(micros, 0.50), 3), + "p95": round(_percentile(micros, 0.95), 3), + "p99": round(_percentile(micros, 0.99), 3), + "max": round(micros[-1], 3), + "mean": round(statistics.fmean(micros), 3), + "stdev": round(statistics.stdev(micros), 3) if len(micros) > 1 else 0.0, + } + + +def measure( + operation: Callable[[], Any], + *, + repetitions: int, + warmup: int = 50, + inner_batch: int = 1, +) -> dict[str, Any]: + """Time `operation` and return a distribution summary. + + `inner_batch` amortizes clock granularity for very fast operations: each + recorded sample is the mean of `inner_batch` back-to-back executions, so a + sub-microsecond operation is still measured above the timer's resolution + rather than being reported as a quantized 0. Warmup executions are + discarded so the reported figures are steady-state (warm) numbers. + """ + for _ in range(warmup): + operation() + + samples: list[float] = [] + for _ in range(repetitions): + start = time.perf_counter() + for _ in range(inner_batch): + operation() + elapsed = time.perf_counter() - start + samples.append(elapsed / inner_batch) + + result = summarize(samples) + result["warmup_discarded"] = warmup + result["inner_batch"] = inner_batch + return result + + +def measure_cold(operation: Callable[[], Any]) -> dict[str, Any]: + """Single un-warmed execution, for comparison against the warm figure.""" + start = time.perf_counter() + operation() + return {"unit": "microseconds", "single_cold_execution": round((time.perf_counter() - start) * 1e6, 3)} + + +def latency_derived_rate(summary: dict[str, Any]) -> float: + """Reciprocal of mean latency. This is an ARITHMETIC DERIVATION, not an + observed throughput measurement — it assumes zero per-iteration overhead + and no drift over time. Report it only alongside `sustained_throughput`, + never as a substitute for it.""" + mean_seconds = summary["mean"] / 1e6 + if mean_seconds <= 0: + return float("inf") + return round(1.0 / mean_seconds, 1) + + +def sustained_throughput( + operation: Callable[[], Any], + *, + duration_seconds: float = 5.0, + warmup_seconds: float = 1.0, + trials: int = 3, +) -> dict[str, Any]: + """Observed sustained rate: run `operation` continuously for a fixed wall-clock + window and count completed operations. + + This is a genuine throughput measurement rather than a reciprocal of mean + latency: it includes loop overhead, allocator behaviour, and any drift that + appears only under continuous operation, none of which a latency reciprocal + captures. Single process, single thread, no concurrency. + """ + trial_results: list[dict[str, Any]] = [] + + for trial_index in range(trials): + warmup_deadline = time.perf_counter() + warmup_seconds + while time.perf_counter() < warmup_deadline: + operation() + + operations = 0 + start = time.perf_counter() + deadline = start + duration_seconds + while time.perf_counter() < deadline: + operation() + operations += 1 + elapsed = time.perf_counter() - start + + trial_results.append( + { + "trial": trial_index + 1, + "operations": operations, + "elapsed_seconds": round(elapsed, 4), + "operations_per_second": round(operations / elapsed, 1), + } + ) + + rates = sorted(trial["operations_per_second"] for trial in trial_results) + return { + "method": "observed sustained rate — continuous single-threaded loop over a fixed window", + "warmup_seconds": warmup_seconds, + "measurement_seconds_per_trial": duration_seconds, + "trials": trials, + "trial_detail": trial_results, + "total_operations": sum(trial["operations"] for trial in trial_results), + "min_ops_per_second": rates[0], + "median_ops_per_second": statistics.median(rates), + "max_ops_per_second": rates[-1], + } + + +# -------------------------------------------------------------------------- +# device-under-test provenance (AC-035A) +# -------------------------------------------------------------------------- + +# Paths that constitute the device under test. The benchmark measures these and +# must not modify them; the harness itself lives outside this set. +DUT_PATHS = ( + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md", +) + + +def verify_dut_unchanged(dut_base_sha: str) -> dict[str, Any]: + """Prove the measured product files are byte-identical to `dut_base_sha`. + + The harness is introduced by a later commit than the implementation it + measures, so "which commit was measured" cannot be answered by HEAD alone. + This diffs only the DUT paths between HEAD and the declared base: an empty + diff establishes that the harness commit changed nothing under measurement. + """ + try: + # Deliberately diff against the WORKING TREE, not against HEAD: the + # benchmark imports and measures the files on disk, so a comparison + # between two commits would report "verified" while an uncommitted edit + # silently changed what was actually measured. Omitting the second + # revision makes git compare dut_base_sha to the working tree, catching + # committed and uncommitted drift alike. + completed = subprocess.run( + ["git", "diff", "--name-only", dut_base_sha, "--", *DUT_PATHS], + cwd=REPO_ROOT, + capture_output=True, + text=True, + check=True, + ) + except (subprocess.CalledProcessError, FileNotFoundError) as exc: + return { + "verified": False, + "error": f"could not diff against {dut_base_sha}: {exc}", + "dut_paths": list(DUT_PATHS), + } + + modified = [line for line in completed.stdout.splitlines() if line.strip()] + return { + "verified": not modified, + "dut_base_sha": dut_base_sha, + "dut_paths": list(DUT_PATHS), + "modified_dut_files": modified, + "statement": ( + f"All device-under-test paths are byte-identical to {dut_base_sha}; the " + "benchmark harness changed nothing under measurement." + if not modified + else f"DUT DRIFT: {len(modified)} file(s) differ from {dut_base_sha}. " + "Results do NOT describe that commit." + ), + } + + +# -------------------------------------------------------------------------- +# memory measurement (Phase 9) +# -------------------------------------------------------------------------- + +def measure_peak_memory(operation: Callable[[], Any]) -> dict[str, Any]: + """Peak Python heap attributable to one execution, via tracemalloc. + + This measures interpreter-level allocation, not RSS: it excludes the + interpreter's own baseline and any allocator caching, which is what makes + it comparable across runs. + """ + tracemalloc.start() + tracemalloc.reset_peak() + operation() + _current, peak = tracemalloc.get_traced_memory() + tracemalloc.stop() + return {"peak_traced_bytes": peak, "peak_traced_kib": round(peak / 1024, 2)} + + +# -------------------------------------------------------------------------- +# synthetic scale specimens (Phase 5) +# -------------------------------------------------------------------------- + +def _reseal(artifact: dict[str, Any]) -> dict[str, Any]: + """Recompute the contract digest and re-bind every sibling that carries it. + + Generated specimens must be genuinely valid, not merely well-shaped: if the + digest were left stale the specimen would refuse at the digest gate and the + scale curve would measure the refusal path instead of the authorization + path. + """ + from authcontract.digest import contract_digest + + digest = contract_digest(artifact["contract"]) + for sibling in ("activation", "admission", "proof"): + if isinstance(artifact.get(sibling), dict) and "contract_digest" in artifact[sibling]: + artifact[sibling]["contract_digest"] = digest + return artifact + + +def make_action_scaled_specimen(action_count: int) -> dict[str, Any]: + """Banking specimen widened to `action_count` declared mediated actions. + + Scales the projection domain (the 'rules' dimension) while leaving the + exercised action, the required facts, and the decision semantics identical, + so any latency change is attributable to domain size alone. + """ + artifact = copy.deepcopy(load_fixture("banking_payment_specimen.json")) + actions = artifact["contract"]["projection_domain"]["actions"] + template = copy.deepcopy(actions["send_payment"]) + mediated = artifact["contract"]["subject"]["mediated_actions"] + for index in range(action_count - 1): + name = f"synthetic_action_{index:05d}" + actions[name] = copy.deepcopy(template) + mediated.append(name) + return _reseal(artifact) + + +def make_fact_scaled_specimen(fact_count: int) -> tuple[dict[str, Any], dict[str, Any]]: + """Banking specimen and matching bundle widened to `fact_count` facts. + + Both sides are widened together: a contract requiring N facts is only + satisfiable by a bundle supplying all N, so this scales the admissibility + workload rather than manufacturing a refusal. + """ + artifact = copy.deepcopy(load_fixture("banking_payment_specimen.json")) + bundle = copy.deepcopy(load_fixture("runtime/facts_valid.json")) + + required_template = copy.deepcopy(artifact["contract"]["required_facts"][0]) + fact_template = copy.deepcopy(bundle["facts"][0]) + + for index in range(fact_count - 1): + fact_id = f"synthetic.fact_{index:05d}" + + requirement = copy.deepcopy(required_template) + requirement["fact_id"] = fact_id + artifact["contract"]["required_facts"].append(requirement) + + fact = copy.deepcopy(fact_template) + fact["fact_id"] = fact_id + fact["evidence"] = copy.deepcopy(fact_template["evidence"]) + fact["evidence"]["fact_id"] = fact_id + bundle["facts"].append(fact) + + return _reseal(artifact), bundle + + +# -------------------------------------------------------------------------- +# result serialization +# -------------------------------------------------------------------------- + +def write_result(filename: str, payload: dict[str, Any]) -> Path: + results_dir = Path(__file__).resolve().parent / "results" + results_dir.mkdir(parents=True, exist_ok=True) + path = results_dir / filename + with path.open("w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2, sort_keys=True) + handle.write("\n") + return path diff --git a/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json b/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json new file mode 100644 index 0000000..0d490ba --- /dev/null +++ b/benchmarks/results/AC-035-CORRECTNESS-MATRIX.json @@ -0,0 +1,732 @@ +{ + "adversarial_matrix": { + "by_category": { + "action_outside_scope": { + "FAIL": 0, + "PASS": 1 + }, + "admission_evidence_binding": { + "FAIL": 0, + "PASS": 2 + }, + "control": { + "FAIL": 0, + "PASS": 1 + }, + "corroboration_missing": { + "FAIL": 0, + "PASS": 1 + }, + "digest_mutation": { + "FAIL": 0, + "PASS": 2 + }, + "digest_scope_violation": { + "FAIL": 0, + "PASS": 1 + }, + "evidence_mismatch": { + "FAIL": 0, + "PASS": 4 + }, + "inactive_contract": { + "FAIL": 0, + "PASS": 1 + }, + "malformed_structured_input": { + "FAIL": 0, + "PASS": 4 + }, + "malformed_type": { + "FAIL": 0, + "PASS": 5 + }, + "missing_required_field": { + "FAIL": 0, + "PASS": 2 + }, + "out_of_domain_value": { + "FAIL": 0, + "PASS": 1 + }, + "post_binding_mutation": { + "FAIL": 0, + "PASS": 2 + }, + "receipt_context_substitution": { + "FAIL": 0, + "PASS": 1 + }, + "reordered_structured_input": { + "FAIL": 0, + "PASS": 1 + }, + "replay": { + "FAIL": 0, + "PASS": 1 + }, + "stale_fact": { + "FAIL": 0, + "PASS": 1 + }, + "trust_basis_violation": { + "FAIL": 0, + "PASS": 1 + }, + "unknown_field_forbidden": { + "FAIL": 0, + "PASS": 5 + }, + "validly_bound_variant": { + "FAIL": 0, + "PASS": 1 + } + }, + "results": [ + { + "category": "control", + "description": "Control: the valid specimen must still be ALLOWed", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "ADV-00-control" + }, + { + "category": "missing_required_field", + "description": "Missing required action parameter", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-01" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown action parameter where unknown fields are forbidden", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-02" + }, + { + "category": "out_of_domain_value", + "description": "Out-of-domain enum value", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-03" + }, + { + "category": "malformed_type", + "description": "Lossy decimal representation in action parameter", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-04" + }, + { + "category": "action_outside_scope", + "description": "Action type outside declared mediated scope", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_UNCLASSIFIED_ACTION", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-05" + }, + { + "category": "missing_required_field", + "description": "Required fact absent from bundle", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_FACT_BUNDLE_INCOMPLETE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-06" + }, + { + "category": "stale_fact", + "description": "Stale fact beyond freshness window", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_STALE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-07" + }, + { + "category": "malformed_type", + "description": "Future-dated fact timestamp", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_FUTURE_TIMESTAMP", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-08" + }, + { + "category": "malformed_type", + "description": "Timezone-naive fact timestamp (unverifiable ordering)", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_TIME_UNVERIFIABLE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-09" + }, + { + "category": "trust_basis_violation", + "description": "Self-asserted fact where policy prohibits self-assertion", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_SELF_ASSERTED", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-10" + }, + { + "category": "malformed_type", + "description": "Lossy wire representation of fact value", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_REPRESENTATION", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-11" + }, + { + "category": "malformed_structured_input", + "description": "Duplicate fact_id in bundle", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-12" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown field on fact object", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-13" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown field on fact bundle", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-14" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown field on verified-evidence object", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-15" + }, + { + "category": "evidence_mismatch", + "description": "Claimed value diverges from verifier-established evidence value", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_EVIDENCE_MISMATCH", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-16" + }, + { + "category": "evidence_mismatch", + "description": "Claimed asserter diverges from verifier-established asserter", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_EVIDENCE_MISMATCH", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-17" + }, + { + "category": "evidence_mismatch", + "description": "Claimed fact_id diverges from verifier-established fact_id", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_IDENTITY_MISMATCH", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-18" + }, + { + "category": "evidence_mismatch", + "description": "Stale verified evidence presented with a fresh caller claim", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_EVIDENCE_MISMATCH", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-19" + }, + { + "category": "validly_bound_variant", + "description": "Variant contract, mutated AND correctly re-sealed (binding intact)", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "ADV-20" + }, + { + "category": "digest_mutation", + "description": "Sibling digest disagreement across bound locations", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-21" + }, + { + "category": "digest_scope_violation", + "description": "Self-referential contract digest", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST_SCOPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-22" + }, + { + "category": "digest_mutation", + "description": "Cross-object digest substitution", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-23" + }, + { + "category": "malformed_type", + "description": "Malformed artifact", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-24" + }, + { + "category": "inactive_contract", + "description": "Suspended contract (activation state not ACTIVE)", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_INACTIVE_CONTRACT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-25" + }, + { + "category": "admission_evidence_binding", + "description": "Admission carrying approvals, contract binding intact", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "ADV-26" + }, + { + "category": "malformed_structured_input", + "description": "Admission present as JSON null", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-27" + }, + { + "category": "malformed_structured_input", + "description": "Admission present as JSON list", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-28" + }, + { + "category": "malformed_structured_input", + "description": "Duplicate required-fact declaration", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-29" + }, + { + "category": "unknown_field_forbidden", + "description": "Unknown field on required-fact declaration", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-30" + }, + { + "category": "corroboration_missing", + "description": "Corroboration required but not satisfiable as declared", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_MALFORMED_INPUT", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-31" + }, + { + "category": "reordered_structured_input", + "description": "Reordered JSON object keys must not alter canonical identity or decision", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "digest invariant under key reordering: True", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": null, + "specimen": "ADV-32" + }, + { + "category": "replay", + "description": "Replayed identical request", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "Replay yields an identical receipt. NOTE: this documents determinism, not replay *protection* \u2014 there is no nonce, sequence number, or single-use semantics at this commit, so an intercepted receipt is indistinguishable from a legitimately re-derived one. Recorded as an architectural gap.", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": null, + "specimen": "ADV-33" + }, + { + "category": "receipt_context_substitution", + "description": "Receipt presented against a different action than it was issued for", + "expected_result": "REFUSE", + "failure_stage": "receipt_verification", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": true, + "receipt_verified": false, + "specimen": "ADV-34" + }, + { + "category": "post_binding_mutation", + "description": "Contract version altered, digest binding left stale", + "expected_result": "REFUSE", + "failure_stage": "digest_binding", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-35" + }, + { + "category": "post_binding_mutation", + "description": "Projection domain widened (amount retyped), digest binding left stale", + "expected_result": "REFUSE", + "failure_stage": "digest_binding", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "ADV-36" + }, + { + "category": "admission_evidence_binding", + "description": "Forged admission approvals must alter the bound evidence", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "Decision is ALLOW because approvals are not an authorization gate at this commit (finding AC-035-F1). The evidence binding nonetheless holds: admission_digest and receipt_digest both change (True), and a receipt issued for the unforged admission does not verify against the forged one (VEIP_RECEIPT_MISMATCH).", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": false, + "specimen": "ADV-37" + } + ], + "totals": { + "failed": 0, + "not_evaluated": 0, + "passed": 38, + "total": 38 + } + }, + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "end_to_end_matrix": [ + { + "category": "happy_path", + "description": "Happy path: valid contract, valid facts, permitted action", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "E2E-01" + }, + { + "category": "stale_fact", + "description": "Stale runtime fact: freshness window exceeded", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_STALE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-02" + }, + { + "category": "malformed_contract", + "description": "Malformed contract artifact", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-03" + }, + { + "category": "unsupported_domain", + "description": "Action outside the declared projection domain", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_UNCLASSIFIED_ACTION", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-04" + }, + { + "category": "source_version_mutation", + "description": "Source/version material changed with stale (non-recomputed) digest binding", + "expected_result": "REFUSE", + "failure_stage": "digest_binding", + "notes": "contract.identity.version altered; activation/admission/proof digests left stale", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-06" + }, + { + "category": "receipt_mutation", + "description": "Receipt mutation and truncation across every protected field", + "expected_result": "REFUSE", + "failure_stage": "receipt_verification", + "notes": "20 mutation/truncation variants tested; 20 detected", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_RECEIPT_MISMATCH/MALFORMED", + "receipt_generated": true, + "receipt_verified": false, + "specimen": "E2E-05" + }, + { + "category": "deterministic_replay", + "description": "Deterministic replay, 100 executions of the identical specimen", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "all protected receipt fields byte-identical across replays", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "E2E-07" + } + ], + "environment": { + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" + }, + "generated_at_utc": "2026-08-24T13:43:12Z", + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, + "work_order": "AC-035 / AC-035A" +} diff --git a/benchmarks/results/AC-035-DETERMINISM.json b/benchmarks/results/AC-035-DETERMINISM.json new file mode 100644 index 0000000..938171a --- /dev/null +++ b/benchmarks/results/AC-035-DETERMINISM.json @@ -0,0 +1,102 @@ +{ + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "determinism": { + "determinism_statement": "Every observed output is stable across repeated execution of a fixed specimen: decision, reason code, contract digest, projection, and all protected receipt fields including decision_time. decision_time is stable because it is bound to the fact bundle's own declared `now`, not to wall-clock time at invocation \u2014 so at this commit there is no intentionally-varying receipt field. This is determinism over fixed inputs in a single process; it is not a claim about cross-version, cross-platform, or cross-implementation reproducibility, none of which was tested.", + "fully_deterministic_over_fixed_inputs": true, + "primitives": { + "contract_digest_distinct_values": 1, + "contract_digest_stable": true, + "projection_distinct_values": 1, + "projection_stable": true + }, + "refused_specimen": { + "decision_stable": true, + "decision_values_observed": [ + "REFUSED" + ], + "executions": 100, + "reason_code_stable": true, + "reason_codes_observed": [ + "RUN_FACT_STALE" + ], + "receipt_emitted": false, + "stable_receipt_fields": [], + "varying_receipt_fields": [] + }, + "valid_specimen": { + "decision_stable": true, + "decision_values_observed": [ + "ALLOW" + ], + "executions": 100, + "reason_code_stable": true, + "reason_codes_observed": [ + "OK" + ], + "receipt_emitted": true, + "stable_receipt_fields": [ + "activation_id", + "admission_digest", + "contract_digest", + "decision", + "decision_time", + "exact_action_digest", + "execution_result", + "projection_digest", + "receipt_digest", + "runtime_fact_set_digest" + ], + "varying_receipt_fields": [] + } + }, + "environment": { + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" + }, + "generated_at_utc": "2026-08-24T13:43:12Z", + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, + "work_order": "AC-035 / AC-035A" +} diff --git a/benchmarks/results/AC-035-E2E-RESULTS.json b/benchmarks/results/AC-035-E2E-RESULTS.json new file mode 100644 index 0000000..3a3b03e --- /dev/null +++ b/benchmarks/results/AC-035-E2E-RESULTS.json @@ -0,0 +1,255 @@ +{ + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "end_to_end": { + "receipt_mutation_detail": [ + { + "detected": true, + "mutated_field": "activation_id", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "admission_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "contract_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "decision", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "decision_time", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "exact_action_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "execution_result", + "reason_code": "VEIP_INVALID_EXECUTION_RESULT" + }, + { + "detected": true, + "mutated_field": "projection_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "receipt_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "runtime_fact_set_digest", + "reason_code": "VEIP_RECEIPT_MISMATCH" + }, + { + "detected": true, + "mutated_field": "activation_id (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "admission_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "contract_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "decision (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "decision_time (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "exact_action_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "execution_result (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "projection_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "receipt_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + }, + { + "detected": true, + "mutated_field": "runtime_fact_set_digest (removed)", + "reason_code": "VEIP_RECEIPT_MALFORMED" + } + ], + "specimens": [ + { + "category": "happy_path", + "description": "Happy path: valid contract, valid facts, permitted action", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "E2E-01" + }, + { + "category": "stale_fact", + "description": "Stale runtime fact: freshness window exceeded", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_FACT_STALE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-02" + }, + { + "category": "malformed_contract", + "description": "Malformed contract artifact", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_DOMAIN_ESCAPE", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-03" + }, + { + "category": "unsupported_domain", + "description": "Action outside the declared projection domain", + "expected_result": "REFUSE", + "failure_stage": "authorization_decision", + "notes": "", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "RUN_UNCLASSIFIED_ACTION", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-04" + }, + { + "category": "source_version_mutation", + "description": "Source/version material changed with stale (non-recomputed) digest binding", + "expected_result": "REFUSE", + "failure_stage": "digest_binding", + "notes": "contract.identity.version altered; activation/admission/proof digests left stale", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "AC_DIGEST", + "receipt_generated": false, + "receipt_verified": null, + "specimen": "E2E-06" + }, + { + "category": "receipt_mutation", + "description": "Receipt mutation and truncation across every protected field", + "expected_result": "REFUSE", + "failure_stage": "receipt_verification", + "notes": "20 mutation/truncation variants tested; 20 detected", + "observed_result": "REFUSE", + "pass_fail": "PASS", + "reason_code": "VEIP_RECEIPT_MISMATCH/MALFORMED", + "receipt_generated": true, + "receipt_verified": false, + "specimen": "E2E-05" + }, + { + "category": "deterministic_replay", + "description": "Deterministic replay, 100 executions of the identical specimen", + "expected_result": "ALLOW", + "failure_stage": null, + "notes": "all protected receipt fields byte-identical across replays", + "observed_result": "ALLOW", + "pass_fail": "PASS", + "reason_code": "OK", + "receipt_generated": true, + "receipt_verified": true, + "specimen": "E2E-07" + } + ], + "totals": { + "failed": 0, + "passed": 7, + "total": 7 + } + }, + "environment": { + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" + }, + "generated_at_utc": "2026-08-24T13:43:12Z", + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, + "work_order": "AC-035 / AC-035A" +} diff --git a/benchmarks/results/AC-035-PERFORMANCE.json b/benchmarks/results/AC-035-PERFORMANCE.json new file mode 100644 index 0000000..4d76751 --- /dev/null +++ b/benchmarks/results/AC-035-PERFORMANCE.json @@ -0,0 +1,521 @@ +{ + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "environment": { + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" + }, + "generated_at_utc": "2026-08-24T13:43:12Z", + "performance": { + "cold_vs_warm": { + "complete_end_to_end_cold": { + "single_cold_execution": 598.022, + "unit": "microseconds" + }, + "note": "Single un-warmed execution in an already-imported interpreter. The dominant one-time cost is interpreter startup and module import, reported separately in the resource profile." + }, + "latency_derived_rate": { + "caveat": "This is arithmetic, not measurement: it assumes zero loop overhead and no drift under continuous operation. Compare against observed_sustained_throughput below; where they disagree, the observed figure is the real one.", + "complete_e2e_transactions_per_second": 1579.6, + "decisions_per_second": 3532.7, + "method": "LATENCY-DERIVED RATE \u2014 reciprocal of warm mean latency, NOT an observed rate", + "receipt_verifications_per_second": 3382.0, + "receipts_per_second": 3532.7, + "receipts_per_second_caveat": "Receipt emission is not separately callable at this commit: run_specimen decides and emits in one pass, so decisions/sec and receipts/sec are the same measurement reported twice, not two independent figures." + }, + "observed_sustained_throughput": { + "complete_end_to_end": { + "max_ops_per_second": 1587.5, + "measurement_seconds_per_trial": 5.0, + "median_ops_per_second": 1574.4, + "method": "observed sustained rate \u2014 continuous single-threaded loop over a fixed window", + "min_ops_per_second": 1512.9, + "total_operations": 23375, + "trial_detail": [ + { + "elapsed_seconds": 5.0003, + "operations": 7565, + "operations_per_second": 1512.9, + "trial": 1 + }, + { + "elapsed_seconds": 5.0002, + "operations": 7938, + "operations_per_second": 1587.5, + "trial": 2 + }, + { + "elapsed_seconds": 5.0001, + "operations": 7872, + "operations_per_second": 1574.4, + "trial": 3 + } + ], + "trials": 3, + "warmup_seconds": 1.0 + }, + "decision_and_receipt": { + "max_ops_per_second": 3423.9, + "measurement_seconds_per_trial": 5.0, + "median_ops_per_second": 3383.8, + "method": "observed sustained rate \u2014 continuous single-threaded loop over a fixed window", + "min_ops_per_second": 3258.9, + "total_operations": 50335, + "trial_detail": [ + { + "elapsed_seconds": 5.0004, + "operations": 17121, + "operations_per_second": 3423.9, + "trial": 1 + }, + { + "elapsed_seconds": 5.0, + "operations": 16919, + "operations_per_second": 3383.8, + "trial": 2 + }, + { + "elapsed_seconds": 5.0002, + "operations": 16295, + "operations_per_second": 3258.9, + "trial": 3 + } + ], + "trials": 3, + "warmup_seconds": 1.0 + }, + "note": "Observed sustained rates: continuous single-process, single-threaded loops over fixed wall-clock windows. No concurrency. Not a distributed or multi-core claim.", + "receipt_verification": { + "max_ops_per_second": 3237.4, + "measurement_seconds_per_trial": 5.0, + "median_ops_per_second": 3212.9, + "method": "observed sustained rate \u2014 continuous single-threaded loop over a fixed window", + "min_ops_per_second": 3178.2, + "total_operations": 48145, + "trial_detail": [ + { + "elapsed_seconds": 5.0004, + "operations": 16066, + "operations_per_second": 3212.9, + "trial": 1 + }, + { + "elapsed_seconds": 5.0003, + "operations": 15892, + "operations_per_second": 3178.2, + "trial": 2 + }, + { + "elapsed_seconds": 5.0001, + "operations": 16187, + "operations_per_second": 3237.4, + "trial": 3 + } + ], + "trials": 3, + "warmup_seconds": 1.0 + } + }, + "stages": { + "action_check": { + "summary": { + "inner_batch": 20, + "max": 18.275, + "mean": 5.633, + "min": 4.914, + "n": 2000, + "p50": 5.049, + "p95": 8.507, + "p99": 10.81, + "stdev": 1.362, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "check_action: validate a proposed action against the projection" + }, + "canonical_digest": { + "summary": { + "inner_batch": 20, + "max": 226.511, + "mean": 88.619, + "min": 78.961, + "n": 2000, + "p50": 85.554, + "p95": 107.888, + "p99": 146.427, + "stdev": 11.555, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "contract_digest: JCS canonicalization plus SHA-256" + }, + "canonicalization": { + "summary": { + "inner_batch": 20, + "max": 428.039, + "mean": 90.825, + "min": 75.79, + "n": 2000, + "p50": 83.872, + "p95": 143.6, + "p99": 155.003, + "stdev": 19.863, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "canonical_bytes: RFC 8785 JCS serialization of the contract" + }, + "complete_end_to_end": { + "summary": { + "inner_batch": 1, + "max": 1438.889, + "mean": 633.078, + "min": 520.309, + "n": 1000, + "p50": 574.002, + "p95": 1052.539, + "p99": 1111.175, + "stdev": 154.163, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "parse \u2192 decide \u2192 emit receipt \u2192 independently verify receipt" + }, + "contract_parse": { + "summary": { + "inner_batch": 20, + "max": 2091.211, + "mean": 10.92, + "min": 8.822, + "n": 2000, + "p50": 9.148, + "p95": 12.668, + "p99": 17.553, + "stdev": 46.568, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "json.loads of the raw contract artifact text" + }, + "decision_and_receipt": { + "summary": { + "inner_batch": 1, + "max": 576.538, + "mean": 283.067, + "min": 247.537, + "n": 1000, + "p50": 264.921, + "p95": 365.349, + "p99": 492.532, + "stdev": 46.925, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "run_specimen: full orchestration \u2014 parse bundle, project, admit facts, decide, and emit the receipt. Supersets projection and action_check." + }, + "projection": { + "summary": { + "inner_batch": 20, + "max": 780.841, + "mean": 105.502, + "min": 89.451, + "n": 2000, + "p50": 99.535, + "p95": 141.815, + "p99": 184.272, + "stdev": 26.567, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "project: build the deterministic action-domain projection" + }, + "projection_digest": { + "summary": { + "inner_batch": 20, + "max": 388.926, + "mean": 45.351, + "min": 40.102, + "n": 2000, + "p50": 42.922, + "p95": 60.748, + "p99": 77.546, + "stdev": 10.701, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "projection_digest over the realized projection" + }, + "receipt_verification": { + "summary": { + "inner_batch": 1, + "max": 679.022, + "mean": 295.685, + "min": 252.573, + "n": 1000, + "p50": 279.364, + "p95": 382.414, + "p99": 491.106, + "stdev": 49.273, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "verify_receipt: independently recompute every binding from raw inputs and compare. Internally re-runs the full decision path." + }, + "validation_and_binding": { + "summary": { + "inner_batch": 20, + "max": 2054.891, + "mean": 94.558, + "min": 79.758, + "n": 2000, + "p50": 86.937, + "p95": 123.164, + "p99": 157.781, + "stdev": 63.977, + "unit": "microseconds", + "warmup_discarded": 50 + }, + "what": "verify_artifact: digest-scope validation plus sibling binding agreement" + } + } + }, + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, + "scaling": { + "declared_action_scaling": { + "dimension": "count of declared mediated actions in the projection domain", + "levels": [ + { + "decision": "ALLOW", + "declared_actions": 1, + "latency": { + "inner_batch": 1, + "max": 556.088, + "mean": 279.782, + "min": 244.412, + "n": 400, + "p50": 265.108, + "p95": 353.737, + "p99": 455.387, + "stdev": 41.235, + "unit": "microseconds", + "warmup_discarded": 10 + }, + "peak_traced_memory": { + "peak_traced_bytes": 6214, + "peak_traced_kib": 6.07 + }, + "reason_code": "OK" + }, + { + "decision": "ALLOW", + "declared_actions": 10, + "latency": { + "inner_batch": 1, + "max": 1662.417, + "mean": 1008.521, + "min": 855.607, + "n": 400, + "p50": 967.498, + "p95": 1303.346, + "p99": 1589.686, + "stdev": 140.887, + "unit": "microseconds", + "warmup_discarded": 10 + }, + "peak_traced_memory": { + "peak_traced_bytes": 9621, + "peak_traced_kib": 9.4 + }, + "reason_code": "OK" + }, + { + "decision": "ALLOW", + "declared_actions": 100, + "latency": { + "inner_batch": 1, + "max": 13444.785, + "mean": 8126.808, + "min": 7167.499, + "n": 400, + "p50": 7811.029, + "p95": 10430.793, + "p99": 12830.666, + "stdev": 1045.757, + "unit": "microseconds", + "warmup_discarded": 10 + }, + "peak_traced_memory": { + "peak_traced_bytes": 41774, + "peak_traced_kib": 40.79 + }, + "reason_code": "OK" + }, + { + "decision": "ALLOW", + "declared_actions": 1000, + "latency": { + "inner_batch": 1, + "max": 116275.614, + "mean": 78031.881, + "min": 71024.273, + "n": 60, + "p50": 76607.789, + "p95": 87792.025, + "p99": 116275.614, + "stdev": 6752.494, + "unit": "microseconds", + "warmup_discarded": 10 + }, + "peak_traced_memory": { + "peak_traced_bytes": 374877, + "peak_traced_kib": 366.09 + }, + "reason_code": "OK" + } + ], + "observed_shape": "approximately linear (size x1000 -> time x278.9)" + }, + "not_evaluated": { + "concurrent_or_distributed_throughput": "NOT EVALUATED \u2014 no concurrency or distribution layer exists at this commit.", + "multi_contract_corpora": "NOT EVALUATED \u2014 the implementation evaluates one artifact per invocation; there is no multi-contract registry or cross-contract selection path at this commit whose scaling could be measured without inventing architecture.", + "persistent_storage_scaling": "NOT EVALUATED \u2014 the runtime is stateless over in-memory inputs; there is no storage backend to scale." + }, + "required_fact_scaling": { + "dimension": "count of required facts (contract) matched by supplied facts (bundle)", + "levels": [ + { + "decision": "ALLOW", + "latency": { + "inner_batch": 1, + "max": 4685.441, + "mean": 953.728, + "min": 717.66, + "n": 200, + "p50": 913.82, + "p95": 1215.565, + "p99": 1396.101, + "stdev": 293.21, + "unit": "microseconds", + "warmup_discarded": 3 + }, + "peak_traced_memory": { + "peak_traced_bytes": 16204, + "peak_traced_kib": 15.82 + }, + "reason_code": "OK", + "required_facts": 10 + }, + { + "decision": "ALLOW", + "latency": { + "inner_batch": 1, + "max": 9229.832, + "mean": 6978.442, + "min": 5406.586, + "n": 200, + "p50": 6916.054, + "p95": 8445.045, + "p99": 9076.396, + "stdev": 725.641, + "unit": "microseconds", + "warmup_discarded": 3 + }, + "peak_traced_memory": { + "peak_traced_bytes": 144093, + "peak_traced_kib": 140.72 + }, + "reason_code": "OK", + "required_facts": 100 + }, + { + "decision": "ALLOW", + "latency": { + "inner_batch": 1, + "max": 83138.987, + "mean": 68148.353, + "min": 63971.47, + "n": 40, + "p50": 67261.842, + "p95": 74742.939, + "p99": 83138.987, + "stdev": 3699.0, + "unit": "microseconds", + "warmup_discarded": 3 + }, + "peak_traced_memory": { + "peak_traced_bytes": 1363979, + "peak_traced_kib": 1332.01 + }, + "reason_code": "OK", + "required_facts": 1000 + }, + { + "decision": "ALLOW", + "latency": { + "inner_batch": 1, + "max": 671062.884, + "mean": 648173.111, + "min": 597079.984, + "n": 5, + "p50": 657265.367, + "p95": 671062.884, + "p99": 671062.884, + "stdev": 29974.09, + "unit": "microseconds", + "warmup_discarded": 3 + }, + "peak_traced_memory": { + "peak_traced_bytes": 13788824, + "peak_traced_kib": 13465.65 + }, + "reason_code": "OK", + "required_facts": 10000 + } + ], + "observed_shape": "approximately linear (size x1000 -> time x679.6)" + } + }, + "work_order": "AC-035 / AC-035A" +} diff --git a/benchmarks/results/AC-035-RESOURCE-PROFILE.json b/benchmarks/results/AC-035-RESOURCE-PROFILE.json new file mode 100644 index 0000000..77b37e6 --- /dev/null +++ b/benchmarks/results/AC-035-RESOURCE-PROFILE.json @@ -0,0 +1,95 @@ +{ + "claim_ceiling": [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific." + ], + "environment": { + "commit_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dependency_versions": { + "pytest": "9.1.1", + "rfc8785": "0.1.4" + }, + "git_status_clean": false, + "machine": "x86_64", + "platform": "Linux-6.18.44-fc-v21-x86_64-with-glibc2.39", + "processor": "x86_64", + "python_implementation": "CPython", + "python_version": "3.11.15", + "tree_sha": "61375c762b1e9b9e9e602adf7921df7da17d9456" + }, + "generated_at_utc": "2026-08-24T13:43:12Z", + "provenance": { + "benchmark_harness_sha": "a7f6ba374b1362e624a3f8b912b265dd03da4cdd", + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_verification": { + "dut_base_sha": "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc", + "dut_paths": [ + "authcontract", + "tests", + "fixtures", + ".github", + "pyproject.toml", + "README.md", + "docs/SOTA.md", + "docs/SOTA-EVIDENCE.md", + "docs/DEVELOPER-LANGUAGE.md", + "docs/CLEANROOM-VALIDATION-RUNBOOK.md" + ], + "modified_dut_files": [], + "statement": "All device-under-test paths are byte-identical to e4e1a97509df1a66c44b090c0a0ca0a03907f4dc; the benchmark harness changed nothing under measurement.", + "verified": true + }, + "note": "DUT_BASE_SHA is the AuthContract implementation being measured. BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it \u2014 necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + }, + "resources": { + "artifact_sizes": { + "action": 148, + "canonical_contract_bytes": 773, + "contract_artifact": 1179, + "contract_body_only": 773, + "fact_bundle": 675, + "projection": 479, + "receipt": 715, + "unit": "bytes (compact JSON encoding)" + }, + "peak_memory": { + "note": "tracemalloc peak attributable to one execution; excludes interpreter baseline", + "single_decision": { + "peak_traced_bytes": 6214, + "peak_traced_kib": 6.07 + }, + "single_verification": { + "peak_traced_bytes": 6334, + "peak_traced_kib": 6.19 + } + }, + "process_max_rss": { + "note": "whole-process peak RSS at end of benchmark run, including interpreter and harness", + "unit": "kilobytes", + "value": 64708 + }, + "process_startup": { + "max": 87.95, + "mean": 67.66, + "min": 60.54, + "samples": [ + 65.18, + 87.95, + 63.09, + 60.54, + 61.51 + ], + "unit": "milliseconds", + "what": "python -c 'import authcontract.veip', out-of-process, 5 samples" + } + }, + "work_order": "AC-035 / AC-035A" +} diff --git a/benchmarks/run_benchmarks.py b/benchmarks/run_benchmarks.py new file mode 100644 index 0000000..bc9dfb6 --- /dev/null +++ b/benchmarks/run_benchmarks.py @@ -0,0 +1,912 @@ +#!/usr/bin/env python3 +"""AC-035 — end-to-end operational and performance baseline for AuthContract. + +Exercises only capabilities that exist at the measured commit. No runtime code +is imported-and-patched, monkeypatched, or reimplemented here: every number +comes from calling the same public entry points a developer would call. + +Reproduce with: + + python3 -m pip install -e ".[test]" + python3 benchmarks/run_benchmarks.py + +Results are written to benchmarks/results/ as JSON. +""" + +from __future__ import annotations + +import copy +import json +import resource +import subprocess +import sys +import time +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from harness import ( # noqa: E402 + REPO_ROOT, + capture_environment, + latency_derived_rate, + load_fixture, + load_fixture_text, + make_action_scaled_specimen, + make_fact_scaled_specimen, + measure, + measure_cold, + measure_peak_memory, + sustained_throughput, + verify_dut_unchanged, + write_result, +) +from specimens import ADVERSARIAL_SPECIMENS, E2E_SPECIMENS # noqa: E402 + +from authcontract.digest import canonical_bytes, contract_digest, verify_artifact # noqa: E402 +from authcontract.projection import check_action, project, projection_digest, projection_to_dict # noqa: E402 +from authcontract.veip import run_specimen, verify_receipt # noqa: E402 + +EXECUTION_RESULT = "SIMULATED_SUCCESS" + +# The AuthContract implementation under measurement. The harness that measures +# it is introduced by a LATER commit, so these are deliberately distinct values: +# DUT_BASE_SHA identifies the product; the harness SHA is captured at runtime. +# `verify_dut_unchanged` proves the harness commit modified nothing under +# measurement, which is what makes the two safely comparable. +DUT_BASE_SHA = "e4e1a97509df1a66c44b090c0a0ca0a03907f4dc" + +# Sustained-throughput measurement window. +THROUGHPUT_TRIALS = 3 +THROUGHPUT_SECONDS = 5.0 +THROUGHPUT_WARMUP_SECONDS = 1.0 + +# Repetition counts. Chosen so each stage's sample is large enough for a stable +# p99 without making the suite take longer than a developer will tolerate. +STAGE_REPS = 2000 +STAGE_INNER_BATCH = 20 +E2E_REPS = 1000 +DETERMINISM_REPS = 100 + + +def _run(artifact: dict, action: dict, facts: dict): + return run_specimen(artifact, action, facts, execution_result=EXECUTION_RESULT) + + +def _load_triple(spec) -> tuple[dict, dict, dict]: + return ( + load_fixture(spec.artifact_fixture), + load_fixture(spec.action_fixture), + load_fixture(spec.facts_fixture), + ) + + +# -------------------------------------------------------------------------- +# Phase 2 + Phase 6 — end-to-end specimens and correctness matrix +# -------------------------------------------------------------------------- + +def _classify(result) -> str: + return "ALLOW" if result.decision == "ALLOW" else "REFUSE" + + +def _evaluate_specimen(spec) -> dict[str, Any]: + """Run one declarative specimen and record what actually happened.""" + artifact, action, facts = _load_triple(spec) + + failure_stage = None + receipt_generated = False + receipt_verified = None + notes = "" + + try: + result = _run(artifact, action, facts) + observed = _classify(result) + reason_code = result.reason_code + receipt_generated = result.receipt is not None + + if observed == "REFUSE": + failure_stage = "authorization_decision" + elif receipt_generated: + verification = verify_receipt(result.receipt, artifact, action, facts) + receipt_verified = verification.status == "PASS" + if not receipt_verified: + notes = f"receipt failed self-verification: {verification.reason_code}" + except Exception as exc: # noqa: BLE001 - an escaping exception is itself a finding + observed = "EXCEPTION" + reason_code = type(exc).__name__ + failure_stage = "uncaught_exception" + notes = ( + "Implementation raised instead of returning a refusal. An escaping " + f"exception is a fail-closed defect, not a refusal: {exc}" + ) + + passed = observed == spec.expected + if spec.expected == "ALLOW" and passed: + passed = receipt_generated and receipt_verified is True + + return { + "specimen": spec.specimen_id, + "description": spec.description, + "category": spec.category, + "expected_result": spec.expected, + "observed_result": observed, + "pass_fail": "PASS" if passed else "FAIL", + "failure_stage": failure_stage, + "receipt_generated": receipt_generated, + "receipt_verified": receipt_verified, + "reason_code": reason_code, + "notes": notes, + } + + +def _stale_bound_mutation(mutate) -> dict[str, Any]: + """Return a specimen whose contract body was altered but whose declared + digests were deliberately NOT recomputed — the real post-binding attack.""" + artifact = copy.deepcopy(load_fixture("banking_payment_specimen.json")) + mutate(artifact) + return artifact + + +def phase_e2e_and_correctness() -> dict[str, Any]: + rows = [_evaluate_specimen(spec) for spec in E2E_SPECIMENS] + + action_v = load_fixture("actions/send_payment_valid.json") + facts_v = load_fixture("runtime/facts_valid.json") + + # E2E-06: source/version-bound material changed with NO corresponding valid + # binding. Built programmatically because every on-disk "mutated" fixture is + # correctly re-sealed and therefore cannot express this attack. + def _bump_version(art: dict) -> None: + art["contract"]["identity"]["version"] = "9.9.9-attacker" + + mutated = _stale_bound_mutation(_bump_version) + mutated_result = _run(mutated, action_v, facts_v) + rows.append( + { + "specimen": "E2E-06", + "description": "Source/version material changed with stale (non-recomputed) digest binding", + "category": "source_version_mutation", + "expected_result": "REFUSE", + "observed_result": _classify(mutated_result), + "pass_fail": "PASS" if mutated_result.decision != "ALLOW" else "FAIL", + "failure_stage": "digest_binding", + "receipt_generated": mutated_result.receipt is not None, + "receipt_verified": None, + "reason_code": mutated_result.reason_code, + "notes": "contract.identity.version altered; activation/admission/proof digests left stale", + } + ) + + # E2E-05: receipt mutation. Every protected field is mutated in turn, so + # this measures whether the binding covers the whole payload rather than + # only the one field a single-shot test would happen to pick. + artifact = load_fixture("banking_payment_specimen.json") + action = load_fixture("actions/send_payment_valid.json") + facts = load_fixture("runtime/facts_valid.json") + baseline = _run(artifact, action, facts) + + mutation_rows = [] + if baseline.receipt is not None: + for field in sorted(baseline.receipt.keys()): + tampered = copy.deepcopy(baseline.receipt) + original = tampered[field] + tampered[field] = "TAMPERED" if not isinstance(original, str) else original + "-TAMPERED" + verification = verify_receipt(tampered, artifact, action, facts) + mutation_rows.append( + { + "mutated_field": field, + "detected": verification.status != "PASS", + "reason_code": verification.reason_code, + } + ) + + # Truncation: a receipt missing a protected field must not verify. + for field in sorted(baseline.receipt.keys()): + truncated = copy.deepcopy(baseline.receipt) + del truncated[field] + verification = verify_receipt(truncated, artifact, action, facts) + mutation_rows.append( + { + "mutated_field": f"{field} (removed)", + "detected": verification.status != "PASS", + "reason_code": verification.reason_code, + } + ) + + all_detected = all(row["detected"] for row in mutation_rows) and bool(mutation_rows) + rows.append( + { + "specimen": "E2E-05", + "description": "Receipt mutation and truncation across every protected field", + "category": "receipt_mutation", + "expected_result": "REFUSE", + "observed_result": "REFUSE" if all_detected else "ALLOW", + "pass_fail": "PASS" if all_detected else "FAIL", + "failure_stage": "receipt_verification", + "receipt_generated": True, + "receipt_verified": False, + "reason_code": "VEIP_RECEIPT_MISMATCH/MALFORMED", + "notes": ( + f"{len(mutation_rows)} mutation/truncation variants tested; " + f"{sum(1 for r in mutation_rows if r['detected'])} detected" + ), + } + ) + + # E2E-07: deterministic replay (detail recorded in the determinism phase). + replays = [_run(artifact, action, facts) for _ in range(DETERMINISM_REPS)] + receipts_identical = all(r.receipt == replays[0].receipt for r in replays) + decisions_identical = all(r.decision == replays[0].decision for r in replays) + rows.append( + { + "specimen": "E2E-07", + "description": f"Deterministic replay, {DETERMINISM_REPS} executions of the identical specimen", + "category": "deterministic_replay", + "expected_result": "ALLOW", + "observed_result": "ALLOW" if decisions_identical and replays[0].decision == "ALLOW" else "FAIL", + "pass_fail": "PASS" if (receipts_identical and decisions_identical) else "FAIL", + "failure_stage": None if receipts_identical else "determinism", + "receipt_generated": True, + "receipt_verified": True, + "reason_code": replays[0].reason_code, + "notes": ( + "all protected receipt fields byte-identical across replays" + if receipts_identical + else "receipt fields varied across replays" + ), + } + ) + + return { + "specimens": rows, + "receipt_mutation_detail": mutation_rows, + "totals": { + "total": len(rows), + "passed": sum(1 for row in rows if row["pass_fail"] == "PASS"), + "failed": sum(1 for row in rows if row["pass_fail"] == "FAIL"), + }, + } + + +# -------------------------------------------------------------------------- +# Phase 3 + Phase 4 — per-stage latency and throughput +# -------------------------------------------------------------------------- + +def phase_performance() -> dict[str, Any]: + artifact_text = load_fixture_text("banking_payment_specimen.json") + artifact = load_fixture("banking_payment_specimen.json") + action = load_fixture("actions/send_payment_valid.json") + facts = load_fixture("runtime/facts_valid.json") + contract = artifact["contract"] + + projection = project(artifact) + receipt = _run(artifact, action, facts).receipt + + stages = { + "contract_parse": { + "what": "json.loads of the raw contract artifact text", + "summary": measure( + lambda: json.loads(artifact_text), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "validation_and_binding": { + "what": "verify_artifact: digest-scope validation plus sibling binding agreement", + "summary": measure( + lambda: verify_artifact(artifact), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "canonicalization": { + "what": "canonical_bytes: RFC 8785 JCS serialization of the contract", + "summary": measure( + lambda: canonical_bytes(contract), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "canonical_digest": { + "what": "contract_digest: JCS canonicalization plus SHA-256", + "summary": measure( + lambda: contract_digest(contract), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "projection": { + "what": "project: build the deterministic action-domain projection", + "summary": measure( + lambda: project(artifact), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "projection_digest": { + "what": "projection_digest over the realized projection", + "summary": measure( + lambda: projection_digest(projection), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "action_check": { + "what": "check_action: validate a proposed action against the projection", + "summary": measure( + lambda: check_action(projection, action), + repetitions=STAGE_REPS, + inner_batch=STAGE_INNER_BATCH, + ), + }, + "decision_and_receipt": { + "what": ( + "run_specimen: full orchestration — parse bundle, project, admit facts, " + "decide, and emit the receipt. Supersets projection and action_check." + ), + "summary": measure( + lambda: _run(artifact, action, facts), + repetitions=E2E_REPS, + inner_batch=1, + ), + }, + "receipt_verification": { + "what": ( + "verify_receipt: independently recompute every binding from raw inputs " + "and compare. Internally re-runs the full decision path." + ), + "summary": measure( + lambda: verify_receipt(receipt, artifact, action, facts), + repetitions=E2E_REPS, + inner_batch=1, + ), + }, + } + + def full_e2e() -> None: + parsed = json.loads(artifact_text) + result = run_specimen(parsed, action, facts, execution_result=EXECUTION_RESULT) + verify_receipt(result.receipt, parsed, action, facts) + + stages["complete_end_to_end"] = { + "what": "parse → decide → emit receipt → independently verify receipt", + "summary": measure(full_e2e, repetitions=E2E_REPS, inner_batch=1), + } + + cold = { + "note": ( + "Single un-warmed execution in an already-imported interpreter. The dominant " + "one-time cost is interpreter startup and module import, reported separately " + "in the resource profile." + ), + "complete_end_to_end_cold": measure_cold(full_e2e), + } + + latency_derived = { + "method": "LATENCY-DERIVED RATE — reciprocal of warm mean latency, NOT an observed rate", + "caveat": ( + "This is arithmetic, not measurement: it assumes zero loop overhead and no " + "drift under continuous operation. Compare against observed_sustained_throughput " + "below; where they disagree, the observed figure is the real one." + ), + "decisions_per_second": latency_derived_rate(stages["decision_and_receipt"]["summary"]), + "receipts_per_second": latency_derived_rate(stages["decision_and_receipt"]["summary"]), + "receipt_verifications_per_second": latency_derived_rate(stages["receipt_verification"]["summary"]), + "complete_e2e_transactions_per_second": latency_derived_rate(stages["complete_end_to_end"]["summary"]), + "receipts_per_second_caveat": ( + "Receipt emission is not separately callable at this commit: run_specimen " + "decides and emits in one pass, so decisions/sec and receipts/sec are the " + "same measurement reported twice, not two independent figures." + ), + } + + observed = { + "note": ( + "Observed sustained rates: continuous single-process, single-threaded loops over " + "fixed wall-clock windows. No concurrency. Not a distributed or multi-core claim." + ), + "decision_and_receipt": sustained_throughput( + lambda: _run(artifact, action, facts), + duration_seconds=THROUGHPUT_SECONDS, + warmup_seconds=THROUGHPUT_WARMUP_SECONDS, + trials=THROUGHPUT_TRIALS, + ), + "receipt_verification": sustained_throughput( + lambda: verify_receipt(receipt, artifact, action, facts), + duration_seconds=THROUGHPUT_SECONDS, + warmup_seconds=THROUGHPUT_WARMUP_SECONDS, + trials=THROUGHPUT_TRIALS, + ), + "complete_end_to_end": sustained_throughput( + full_e2e, + duration_seconds=THROUGHPUT_SECONDS, + warmup_seconds=THROUGHPUT_WARMUP_SECONDS, + trials=THROUGHPUT_TRIALS, + ), + } + + return { + "stages": stages, + "cold_vs_warm": cold, + "latency_derived_rate": latency_derived, + "observed_sustained_throughput": observed, + } + + +# -------------------------------------------------------------------------- +# Phase 5 — bounded scale curves +# -------------------------------------------------------------------------- + +def phase_scale() -> dict[str, Any]: + action = load_fixture("actions/send_payment_valid.json") + + action_curve = [] + for count in (1, 10, 100, 1000): + artifact = make_action_scaled_specimen(count) + facts = load_fixture("runtime/facts_valid.json") + control = _run(artifact, action, facts) + reps = 400 if count < 1000 else 60 + summary = measure(lambda a=artifact, f=facts: _run(a, action, f), repetitions=reps, warmup=10) + action_curve.append( + { + "declared_actions": count, + "decision": control.decision, + "reason_code": control.reason_code, + "latency": summary, + "peak_traced_memory": measure_peak_memory(lambda a=artifact, f=facts: _run(a, action, f)), + } + ) + + fact_curve = [] + for count in (10, 100, 1000, 10000): + artifact, facts = make_fact_scaled_specimen(count) + control = _run(artifact, action, facts) + reps = 200 if count <= 100 else (40 if count == 1000 else 5) + summary = measure(lambda a=artifact, f=facts: _run(a, action, f), repetitions=reps, warmup=3) + fact_curve.append( + { + "required_facts": count, + "decision": control.decision, + "reason_code": control.reason_code, + "latency": summary, + "peak_traced_memory": measure_peak_memory(lambda a=artifact, f=facts: _run(a, action, f)), + } + ) + + def _shape(curve: list[dict], key: str) -> str: + """Describe growth by comparing per-unit cost at the smallest and largest points.""" + first, last = curve[0], curve[-1] + size_ratio = last[key] / first[key] + time_ratio = last["latency"]["mean"] / first["latency"]["mean"] + if time_ratio < size_ratio * 0.25: + return f"sublinear (size x{size_ratio:.0f} -> time x{time_ratio:.1f})" + if time_ratio <= size_ratio * 1.5: + return f"approximately linear (size x{size_ratio:.0f} -> time x{time_ratio:.1f})" + return f"superlinear — bottleneck candidate (size x{size_ratio:.0f} -> time x{time_ratio:.1f})" + + return { + "declared_action_scaling": { + "dimension": "count of declared mediated actions in the projection domain", + "levels": action_curve, + "observed_shape": _shape(action_curve, "declared_actions"), + }, + "required_fact_scaling": { + "dimension": "count of required facts (contract) matched by supplied facts (bundle)", + "levels": fact_curve, + "observed_shape": _shape(fact_curve, "required_facts"), + }, + "not_evaluated": { + "multi_contract_corpora": ( + "NOT EVALUATED — the implementation evaluates one artifact per invocation; " + "there is no multi-contract registry or cross-contract selection path at this " + "commit whose scaling could be measured without inventing architecture." + ), + "concurrent_or_distributed_throughput": ( + "NOT EVALUATED — no concurrency or distribution layer exists at this commit." + ), + "persistent_storage_scaling": ( + "NOT EVALUATED — the runtime is stateless over in-memory inputs; there is no " + "storage backend to scale." + ), + }, + } + + +# -------------------------------------------------------------------------- +# Phase 7 — determinism +# -------------------------------------------------------------------------- + +def phase_determinism() -> dict[str, Any]: + action = load_fixture("actions/send_payment_valid.json") + artifact = load_fixture("banking_payment_specimen.json") + + findings = {} + for label, facts_fixture in ( + ("valid_specimen", "runtime/facts_valid.json"), + ("refused_specimen", "runtime/facts_stale.json"), + ): + facts = load_fixture(facts_fixture) + results = [_run(artifact, action, facts) for _ in range(DETERMINISM_REPS)] + + decisions = {r.decision for r in results} + reason_codes = {r.reason_code for r in results} + receipts = [r.receipt for r in results] + + stable_fields: list[str] = [] + varying_fields: list[str] = [] + if receipts[0] is not None: + for field in sorted(receipts[0].keys()): + values = {json.dumps(r[field], sort_keys=True) for r in receipts} + (stable_fields if len(values) == 1 else varying_fields).append(field) + + findings[label] = { + "executions": DETERMINISM_REPS, + "decision_values_observed": sorted(decisions), + "reason_codes_observed": sorted(reason_codes), + "decision_stable": len(decisions) == 1, + "reason_code_stable": len(reason_codes) == 1, + "receipt_emitted": receipts[0] is not None, + "stable_receipt_fields": stable_fields, + "varying_receipt_fields": varying_fields, + } + + # Independently confirm the digest/projection primitives are stable too. + contract = artifact["contract"] + digests = {contract_digest(contract) for _ in range(DETERMINISM_REPS)} + projections = { + json.dumps(projection_to_dict(project(artifact)), sort_keys=True) + for _ in range(DETERMINISM_REPS) + } + + findings["primitives"] = { + "contract_digest_distinct_values": len(digests), + "projection_distinct_values": len(projections), + "contract_digest_stable": len(digests) == 1, + "projection_stable": len(projections) == 1, + } + + valid = findings["valid_specimen"] + fully_stable = ( + valid["decision_stable"] + and valid["reason_code_stable"] + and not valid["varying_receipt_fields"] + and findings["primitives"]["contract_digest_stable"] + and findings["primitives"]["projection_stable"] + ) + + findings["determinism_statement"] = ( + ( + "Every observed output is stable across repeated execution of a fixed specimen: " + "decision, reason code, contract digest, projection, and all protected receipt " + "fields including decision_time. decision_time is stable because it is bound to " + "the fact bundle's own declared `now`, not to wall-clock time at invocation — so " + "at this commit there is no intentionally-varying receipt field. This is " + "determinism over fixed inputs in a single process; it is not a claim about " + "cross-version, cross-platform, or cross-implementation reproducibility, none of " + "which was tested." + ) + if fully_stable + else ( + "Determinism is NOT complete. Fields observed to vary across identical " + f"executions: {valid['varying_receipt_fields']}. Each must be classified as " + "intentionally variable or as a defect before any reproducibility claim is made." + ) + ) + findings["fully_deterministic_over_fixed_inputs"] = fully_stable + return findings + + +# -------------------------------------------------------------------------- +# Phase 8 — adversarial battery +# -------------------------------------------------------------------------- + +def phase_adversarial() -> dict[str, Any]: + rows = [_evaluate_specimen(spec) for spec in ADVERSARIAL_SPECIMENS] + + # Structural attacks that are not simple fixture swaps. + artifact = load_fixture("banking_payment_specimen.json") + action = load_fixture("actions/send_payment_valid.json") + facts = load_fixture("runtime/facts_valid.json") + + # Key reordering must not change canonical identity (RFC 8785 property). + reordered = {k: artifact[k] for k in reversed(list(artifact.keys()))} + reordered["contract"] = { + k: artifact["contract"][k] for k in reversed(list(artifact["contract"].keys())) + } + same_digest = contract_digest(reordered["contract"]) == contract_digest(artifact["contract"]) + reorder_result = _run(reordered, action, facts) + rows.append( + { + "specimen": "ADV-32", + "description": "Reordered JSON object keys must not alter canonical identity or decision", + "category": "reordered_structured_input", + "expected_result": "ALLOW", + "observed_result": _classify(reorder_result), + "pass_fail": "PASS" if (same_digest and reorder_result.decision == "ALLOW") else "FAIL", + "failure_stage": None if same_digest else "canonicalization", + "receipt_generated": reorder_result.receipt is not None, + "receipt_verified": None, + "reason_code": reorder_result.reason_code, + "notes": f"digest invariant under key reordering: {same_digest}", + } + ) + + # Replay: an identical request repeated must yield an identical decision. + first = _run(artifact, action, facts) + second = _run(artifact, action, facts) + replay_identical = first.receipt == second.receipt and first.decision == second.decision + rows.append( + { + "specimen": "ADV-33", + "description": "Replayed identical request", + "category": "replay", + "expected_result": "ALLOW", + "observed_result": _classify(second), + "pass_fail": "PASS" if replay_identical else "FAIL", + "failure_stage": None, + "receipt_generated": second.receipt is not None, + "receipt_verified": None, + "reason_code": second.reason_code, + "notes": ( + "Replay yields an identical receipt. NOTE: this documents determinism, not " + "replay *protection* — there is no nonce, sequence number, or single-use " + "semantics at this commit, so an intercepted receipt is indistinguishable " + "from a legitimately re-derived one. Recorded as an architectural gap." + ), + } + ) + + # Receipt verified against a different action than the one it was issued for. + other_action = load_fixture("actions/send_payment_out_of_enum_value.json") + cross = verify_receipt(first.receipt, artifact, other_action, facts) + rows.append( + { + "specimen": "ADV-34", + "description": "Receipt presented against a different action than it was issued for", + "category": "receipt_context_substitution", + "expected_result": "REFUSE", + "observed_result": "REFUSE" if cross.status != "PASS" else "ALLOW", + "pass_fail": "PASS" if cross.status != "PASS" else "FAIL", + "failure_stage": "receipt_verification", + "receipt_generated": True, + "receipt_verified": False, + "reason_code": cross.reason_code, + "notes": "", + } + ) + + # ADV-35/36: genuine post-binding mutation with stale digests. + for adv_id, label, mutate in ( + ( + "ADV-35", + "Contract version altered, digest binding left stale", + lambda art: art["contract"]["identity"].__setitem__("version", "9.9.9-attacker"), + ), + ( + "ADV-36", + "Projection domain widened (amount retyped), digest binding left stale", + lambda art: art["contract"]["projection_domain"]["actions"]["send_payment"][ + "parameters" + ]["amount"].__setitem__("value_type", "string"), + ), + ): + attacked = _stale_bound_mutation(mutate) + outcome = _run(attacked, action, facts) + rows.append( + { + "specimen": adv_id, + "description": label, + "category": "post_binding_mutation", + "expected_result": "REFUSE", + "observed_result": _classify(outcome), + "pass_fail": "PASS" if outcome.decision != "ALLOW" else "FAIL", + "failure_stage": "digest_binding", + "receipt_generated": outcome.receipt is not None, + "receipt_verified": None, + "reason_code": outcome.reason_code, + "notes": "", + } + ) + + # ADV-37: admission approvals are not an authorization gate, but they must + # still be bound into the evidence so a forged admission is distinguishable. + forged = copy.deepcopy(artifact) + forged["admission"]["approvals"] = [{"approval_id": "forged", "approver": "attacker"}] + forged_result = _run(forged, action, facts) + baseline_result = _run(artifact, action, facts) + binding_holds = ( + forged_result.receipt is not None + and baseline_result.receipt is not None + and forged_result.receipt["admission_digest"] != baseline_result.receipt["admission_digest"] + and forged_result.receipt["receipt_digest"] != baseline_result.receipt["receipt_digest"] + ) + cross_verify = verify_receipt(baseline_result.receipt, forged, action, facts) + rows.append( + { + "specimen": "ADV-37", + "description": "Forged admission approvals must alter the bound evidence", + "category": "admission_evidence_binding", + "expected_result": "ALLOW", + "observed_result": _classify(forged_result), + "pass_fail": "PASS" if (binding_holds and cross_verify.status != "PASS") else "FAIL", + "failure_stage": None, + "receipt_generated": True, + "receipt_verified": False, + "reason_code": forged_result.reason_code, + "notes": ( + "Decision is ALLOW because approvals are not an authorization gate at this " + "commit (finding AC-035-F1). The evidence binding nonetheless holds: " + f"admission_digest and receipt_digest both change ({binding_holds}), and a " + "receipt issued for the unforged admission does not verify against the forged " + f"one ({cross_verify.reason_code})." + ), + } + ) + + by_category: dict[str, dict[str, int]] = {} + for row in rows: + bucket = by_category.setdefault(row["category"], {"PASS": 0, "FAIL": 0}) + bucket[row["pass_fail"]] += 1 + + return { + "results": rows, + "totals": { + "total": len(rows), + "passed": sum(1 for row in rows if row["pass_fail"] == "PASS"), + "failed": sum(1 for row in rows if row["pass_fail"] == "FAIL"), + "not_evaluated": 0, + }, + "by_category": by_category, + } + + +# -------------------------------------------------------------------------- +# Phase 9 — resource profile +# -------------------------------------------------------------------------- + +def phase_resources() -> dict[str, Any]: + artifact = load_fixture("banking_payment_specimen.json") + action = load_fixture("actions/send_payment_valid.json") + facts = load_fixture("runtime/facts_valid.json") + result = _run(artifact, action, facts) + + # Interpreter + import cost, measured out-of-process so it is not masked by + # this process having already imported everything. + startup_samples = [] + for _ in range(5): + start = time.perf_counter() + subprocess.run( + [sys.executable, "-c", "import authcontract.veip"], + cwd=REPO_ROOT, + capture_output=True, + check=True, + ) + startup_samples.append((time.perf_counter() - start) * 1000) + + def _size(value: Any) -> int: + return len(json.dumps(value, separators=(",", ":")).encode("utf-8")) + + return { + "process_startup": { + "unit": "milliseconds", + "what": "python -c 'import authcontract.veip', out-of-process, 5 samples", + "samples": [round(s, 2) for s in startup_samples], + "min": round(min(startup_samples), 2), + "mean": round(sum(startup_samples) / len(startup_samples), 2), + "max": round(max(startup_samples), 2), + }, + "peak_memory": { + "note": "tracemalloc peak attributable to one execution; excludes interpreter baseline", + "single_decision": measure_peak_memory(lambda: _run(artifact, action, facts)), + "single_verification": measure_peak_memory( + lambda: verify_receipt(result.receipt, artifact, action, facts) + ), + }, + "process_max_rss": { + "unit": "kilobytes", + "note": "whole-process peak RSS at end of benchmark run, including interpreter and harness", + "value": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, + }, + "artifact_sizes": { + "unit": "bytes (compact JSON encoding)", + "contract_artifact": _size(artifact), + "contract_body_only": _size(artifact["contract"]), + "canonical_contract_bytes": len(canonical_bytes(artifact["contract"])), + "action": _size(action), + "fact_bundle": _size(facts), + "projection": _size(projection_to_dict(project(artifact))), + "receipt": _size(result.receipt), + }, + } + + +# -------------------------------------------------------------------------- +# runner +# -------------------------------------------------------------------------- + +CLAIM_CEILING = [ + "This benchmark measures a bounded MVP-alpha implementation on one synthetic banking specimen family.", + "It does NOT establish production readiness.", + "It does NOT establish regulatory or legal correctness.", + "It does NOT establish universal source-to-rule derivation.", + "It does NOT establish arbitrary-domain compatibility.", + "It does NOT establish security certification.", + "It does NOT establish distributed or concurrent scalability.", + "It does NOT constitute a formal proof.", + "It does NOT establish comparative superiority over any other system.", + "Latency and throughput figures are single-process, single-machine, and environment-specific.", +] + + +def main() -> int: + started = time.time() + environment = capture_environment() + + provenance = { + "dut_base_sha": DUT_BASE_SHA, + "benchmark_harness_sha": environment["commit_sha"], + "note": ( + "DUT_BASE_SHA is the AuthContract implementation being measured. " + "BENCHMARK_HARNESS_SHA is the commit containing the harness that measured it — " + "necessarily a later commit, since the harness did not exist at DUT_BASE_SHA. " + "Reproduce from BENCHMARK_HARNESS_SHA, not from DUT_BASE_SHA." + ), + "dut_verification": verify_dut_unchanged(DUT_BASE_SHA), + } + + if not provenance["dut_verification"]["verified"]: + print("REFUSING TO RUN:", provenance["dut_verification"]["statement"], file=sys.stderr) + return 2 + + print("AC-035 benchmark") + print(" DUT :", DUT_BASE_SHA[:12], "(verified unchanged)") + print(" harness :", environment["commit_sha"][:12]) + print(" phase: end-to-end + correctness matrix") + e2e = phase_e2e_and_correctness() + print(" phase: performance") + performance = phase_performance() + print(" phase: scale curves") + scale = phase_scale() + print(" phase: determinism") + determinism = phase_determinism() + print(" phase: adversarial battery") + adversarial = phase_adversarial() + print(" phase: resource profile") + resources = phase_resources() + + common = { + "work_order": "AC-035 / AC-035A", + "provenance": provenance, + "environment": environment, + "claim_ceiling": CLAIM_CEILING, + "generated_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + } + + write_result("AC-035-E2E-RESULTS.json", {**common, "end_to_end": e2e}) + write_result("AC-035-PERFORMANCE.json", {**common, "performance": performance, "scaling": scale}) + write_result( + "AC-035-CORRECTNESS-MATRIX.json", + {**common, "end_to_end_matrix": e2e["specimens"], "adversarial_matrix": adversarial}, + ) + write_result("AC-035-DETERMINISM.json", {**common, "determinism": determinism}) + write_result("AC-035-RESOURCE-PROFILE.json", {**common, "resources": resources}) + + e2e_totals = e2e["totals"] + adv_totals = adversarial["totals"] + print() + print(f" E2E: {e2e_totals['passed']}/{e2e_totals['total']} passed") + print(f" Adversarial: {adv_totals['passed']}/{adv_totals['total']} passed") + print(f" E2E p50: {performance['stages']['complete_end_to_end']['summary']['p50']} us") + print( + " E2E tps: " + f"{performance['observed_sustained_throughput']['complete_end_to_end']['median_ops_per_second']} observed median" + f" (latency-derived {performance['latency_derived_rate']['complete_e2e_transactions_per_second']})" + ) + print(f" Determinism: {'stable' if determinism['fully_deterministic_over_fixed_inputs'] else 'NOT STABLE'}") + print(f" elapsed: {time.time() - started:.1f}s") + + return 0 if (e2e_totals["failed"] == 0 and adv_totals["failed"] == 0) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/specimens/__init__.py b/benchmarks/specimens/__init__.py new file mode 100644 index 0000000..16909c3 --- /dev/null +++ b/benchmarks/specimens/__init__.py @@ -0,0 +1,390 @@ +"""Specimen definitions for the AC-035 benchmark. + +Specimens are declarative: each names the fixtures it composes and the +behaviour class the architecture is expected to produce. Expectations are +stated as *classes* (ALLOW vs fail-closed REFUSE, verification PASS vs FAIL) +rather than exact reason codes, because the class is the security-relevant +property. The exact observed reason code is recorded alongside as evidence, so +a change in refusal taxonomy is visible in the results without silently +weakening the assertion into a tautology. +""" + +from __future__ import annotations + +from typing import Any, NamedTuple + + +class Specimen(NamedTuple): + specimen_id: str + description: str + artifact_fixture: str + action_fixture: str + facts_fixture: str + expected: str # "ALLOW" or "REFUSE" + category: str + + +# -------------------------------------------------------------------------- +# Phase 2 — the seven mandated end-to-end specimens +# -------------------------------------------------------------------------- + +E2E_SPECIMENS: tuple[Specimen, ...] = ( + Specimen( + "E2E-01", + "Happy path: valid contract, valid facts, permitted action", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "ALLOW", + "happy_path", + ), + Specimen( + "E2E-02", + "Stale runtime fact: freshness window exceeded", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_stale.json", + "REFUSE", + "stale_fact", + ), + Specimen( + "E2E-03", + "Malformed contract artifact", + "malformed.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_contract", + ), + Specimen( + "E2E-04", + "Action outside the declared projection domain", + "banking_payment_specimen.json", + "actions/send_payment_unknown_action_type.json", + "runtime/facts_valid.json", + "REFUSE", + "unsupported_domain", + ), + # E2E-05 (receipt mutation), E2E-06 (source/version mutation) and E2E-07 + # (deterministic replay) are constructed programmatically by the runner + # rather than declared here. + # + # E2E-06 specifically must mutate contract-bound material while leaving the + # binding STALE. The repository's `*_mutated.json` fixtures are mutated + # *and correctly re-sealed* — their recomputed digest matches their declared + # digest — so they are validly-bound variant contracts, not mutation + # attacks. They are exercised below as ALLOW controls, and the genuine + # stale-binding attack is built in the runner. +) + + +# -------------------------------------------------------------------------- +# Phase 8 — adversarial battery +# -------------------------------------------------------------------------- +# Every entry composes real accepted fixtures. Each is expected to fail closed; +# the happy-path control is included so a battery that trivially refuses +# everything is distinguishable from one that discriminates correctly. + +ADVERSARIAL_SPECIMENS: tuple[Specimen, ...] = ( + Specimen( + "ADV-00-control", + "Control: the valid specimen must still be ALLOWed", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "ALLOW", + "control", + ), + # --- action-side attacks ------------------------------------------------- + Specimen( + "ADV-01", + "Missing required action parameter", + "banking_payment_specimen.json", + "actions/send_payment_missing_required_parameter.json", + "runtime/facts_valid.json", + "REFUSE", + "missing_required_field", + ), + Specimen( + "ADV-02", + "Unknown action parameter where unknown fields are forbidden", + "banking_payment_specimen.json", + "actions/send_payment_unknown_parameter.json", + "runtime/facts_valid.json", + "REFUSE", + "unknown_field_forbidden", + ), + Specimen( + "ADV-03", + "Out-of-domain enum value", + "banking_payment_specimen.json", + "actions/send_payment_out_of_enum_value.json", + "runtime/facts_valid.json", + "REFUSE", + "out_of_domain_value", + ), + Specimen( + "ADV-04", + "Lossy decimal representation in action parameter", + "banking_payment_specimen.json", + "actions/send_payment_lossy_value.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_type", + ), + Specimen( + "ADV-05", + "Action type outside declared mediated scope", + "banking_payment_specimen.json", + "actions/send_payment_unknown_action_type.json", + "runtime/facts_valid.json", + "REFUSE", + "action_outside_scope", + ), + # --- fact-side attacks --------------------------------------------------- + Specimen( + "ADV-06", + "Required fact absent from bundle", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_missing_required.json", + "REFUSE", + "missing_required_field", + ), + Specimen( + "ADV-07", + "Stale fact beyond freshness window", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_stale.json", + "REFUSE", + "stale_fact", + ), + Specimen( + "ADV-08", + "Future-dated fact timestamp", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_future_timestamp.json", + "REFUSE", + "malformed_type", + ), + Specimen( + "ADV-09", + "Timezone-naive fact timestamp (unverifiable ordering)", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_naive_timestamp.json", + "REFUSE", + "malformed_type", + ), + Specimen( + "ADV-10", + "Self-asserted fact where policy prohibits self-assertion", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_self_asserted_prohibited.json", + "REFUSE", + "trust_basis_violation", + ), + Specimen( + "ADV-11", + "Lossy wire representation of fact value", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_lossy_representation.json", + "REFUSE", + "malformed_type", + ), + Specimen( + "ADV-12", + "Duplicate fact_id in bundle", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_duplicate_fact_id.json", + "REFUSE", + "malformed_structured_input", + ), + Specimen( + "ADV-13", + "Unknown field on fact object", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_unknown_fact_field.json", + "REFUSE", + "unknown_field_forbidden", + ), + Specimen( + "ADV-14", + "Unknown field on fact bundle", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_unknown_bundle_field.json", + "REFUSE", + "unknown_field_forbidden", + ), + Specimen( + "ADV-15", + "Unknown field on verified-evidence object", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_unknown_evidence_field.json", + "REFUSE", + "unknown_field_forbidden", + ), + # --- caller-claim vs verifier-established evidence divergence ------------ + Specimen( + "ADV-16", + "Claimed value diverges from verifier-established evidence value", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_verified_value_mismatch.json", + "REFUSE", + "evidence_mismatch", + ), + Specimen( + "ADV-17", + "Claimed asserter diverges from verifier-established asserter", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_verified_asserter_mismatch.json", + "REFUSE", + "evidence_mismatch", + ), + Specimen( + "ADV-18", + "Claimed fact_id diverges from verifier-established fact_id", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_verified_fact_id_mismatch.json", + "REFUSE", + "evidence_mismatch", + ), + Specimen( + "ADV-19", + "Stale verified evidence presented with a fresh caller claim", + "banking_payment_specimen.json", + "actions/send_payment_valid.json", + "runtime/facts_verified_time_stale_claimed_fresh.json", + "REFUSE", + "evidence_mismatch", + ), + # --- contract/binding attacks ------------------------------------------- + Specimen( + "ADV-20", + "Variant contract, mutated AND correctly re-sealed (binding intact)", + "banking_payment_specimen_contract_mutated.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + # ALLOW is correct: this fixture's declared digest matches its + # recomputed digest, so it is a different-but-validly-bound contract. + # The stale-binding attack is ADV-35/ADV-36, built in the runner. + "ALLOW", + "validly_bound_variant", + ), + Specimen( + "ADV-21", + "Sibling digest disagreement across bound locations", + "sibling_digest_mismatch.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "digest_mutation", + ), + Specimen( + "ADV-22", + "Self-referential contract digest", + "self_referential.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "digest_scope_violation", + ), + Specimen( + "ADV-23", + "Cross-object digest substitution", + "cross_object_substitution.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "digest_mutation", + ), + Specimen( + "ADV-24", + "Malformed artifact", + "malformed.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_type", + ), + # --- activation / admission attacks ------------------------------------- + Specimen( + "ADV-25", + "Suspended contract (activation state not ACTIVE)", + "banking_payment_specimen_suspended.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "inactive_contract", + ), + Specimen( + "ADV-26", + "Admission carrying approvals, contract binding intact", + "banking_payment_specimen_admission_mutated.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + # ALLOW is correct at this commit: admission approvals are bound into + # admission_digest (and therefore into receipt_digest) as evidence, but + # they are not evaluated as an authorization gate. ADV-37 asserts the + # binding property; the missing gate is recorded as finding AC-035-F1. + "ALLOW", + "admission_evidence_binding", + ), + Specimen( + "ADV-27", + "Admission present as JSON null", + "banking_payment_specimen_admission_null.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_structured_input", + ), + Specimen( + "ADV-28", + "Admission present as JSON list", + "banking_payment_specimen_admission_list.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_structured_input", + ), + # --- contract-declaration attacks --------------------------------------- + Specimen( + "ADV-29", + "Duplicate required-fact declaration", + "banking_payment_specimen_duplicate_required_fact.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "malformed_structured_input", + ), + Specimen( + "ADV-30", + "Unknown field on required-fact declaration", + "banking_payment_specimen_unknown_required_fact_field.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "unknown_field_forbidden", + ), + Specimen( + "ADV-31", + "Corroboration required but not satisfiable as declared", + "banking_payment_specimen_bad_corroboration_required.json", + "actions/send_payment_valid.json", + "runtime/facts_valid.json", + "REFUSE", + "corroboration_missing", + ), +) diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md new file mode 100644 index 0000000..f74c213 --- /dev/null +++ b/docs/BENCHMARKS.md @@ -0,0 +1,334 @@ +# AuthContract benchmark baseline + +**DUT_BASE_SHA (implementation measured):** `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` +**Tree:** `9967077e7f9f9c661199c728c5a2e7fe496be07a` +**BENCHMARK_HARNESS_SHA (harness that measured it):** `a7f6ba374b1362e624a3f8b912b265dd03da4cdd` +**Work order:** AC-035, amended by AC-035A +**Raw results:** [`benchmarks/results/`](../benchmarks/results/) + +> **Provenance.** These are two different commits by necessity — the harness +> cannot exist at the commit it measures. Reproduce from +> `BENCHMARK_HARNESS_SHA`; `benchmarks/` does not exist at `DUT_BASE_SHA`. +> Before measuring, the harness diffs every device-under-test path against +> `DUT_BASE_SHA` and **refuses to run on any drift** (exit 2), so the claim +> "these numbers describe `e4e1a975`" is verified rather than asserted. This +> run: `verified: true`, zero modified DUT files. + +This is the first end-to-end operational and performance baseline for +AuthContract. It measures the bounded MVP-alpha path that exists today. It is +not a marketing document and contains no projected or extrapolated figures. + +--- + +## 1. Environment + +| | | +|---|---| +| Python | 3.11.15 (CPython) | +| Platform | Linux x86_64 | +| Dependencies | `rfc8785` 0.1.4, `pytest` 9.1.1 | +| Process | single, single-threaded | + +Figures are environment-specific. Absolute latencies will differ on other +hardware; the *shape* of the curves and the relative cost of stages are the +transferable results. + +--- + +## 2. What was measured + +The complete implemented path: + +``` +contract artifact + → parse + → digest-scope validation + sibling binding agreement + → RFC 8785 (JCS) canonicalization + → SHA-256 contract digest + → deterministic projection into the declared action domain + → runtime fact admission (issuer / trust basis / freshness / evidence binding) + → action check against the projection + → ALLOW or REFUSE + → decision receipt + → independent receipt verification from raw inputs + → mutation / truncation detection +``` + +--- + +## 3. End-to-end correctness + +**7 / 7 end-to-end specimens pass. 38 / 38 adversarial specimens pass.** + +| ID | Specimen | Expected | Observed | +|---|---|---|---| +| E2E-01 | Happy path | ALLOW + receipt verifies | PASS | +| E2E-02 | Stale runtime fact | REFUSE | PASS (`RUN_FACT_STALE`) | +| E2E-03 | Malformed contract | fail closed | PASS | +| E2E-04 | Action outside projection domain | fail closed | PASS | +| E2E-05 | Receipt mutation + truncation | detected | PASS (20/20 variants detected) | +| E2E-06 | Source/version change, stale binding | REFUSE | PASS (`AC_DIGEST`) | +| E2E-07 | Deterministic replay ×100 | identical | PASS | + +E2E-05 mutates **every** protected receipt field in turn and additionally +removes each field in turn — 20 variants total — rather than testing one +representative field. All 20 are detected. + +The adversarial battery covers 38 specimens across: missing required fields, +unknown fields where forbidden, malformed types, out-of-domain values, stale +facts, future and timezone-naive timestamps, self-assertion where prohibited, +lossy representations, duplicate identifiers, caller-claim vs +verifier-established evidence divergence (value / asserter / fact_id / staleness), +digest mutation, sibling digest disagreement, self-referential digests, +cross-object substitution, suspended contracts, malformed admission shapes, +contract-declaration defects, key reordering, replay, and receipt context +substitution. + +--- + +## 4. Latency by stage + +Warm figures, warmup discarded. Microseconds. + +| Stage | n | p50 | p95 | p99 | mean | +|---|---:|---:|---:|---:|---:| +| contract parse | 2000 | 9.1 | 12.7 | 17.6 | 10.9 | +| validation + binding | 2000 | 86.9 | 123.2 | 157.8 | 94.6 | +| canonicalization (JCS) | 2000 | 83.9 | 143.6 | 155.0 | 90.8 | +| canonical digest | 2000 | 85.6 | 107.9 | 146.4 | 88.6 | +| projection | 2000 | 99.5 | 141.8 | 184.3 | 105.5 | +| projection digest | 2000 | 42.9 | 60.7 | 77.5 | 45.4 | +| action check | 2000 | 5.0 | 8.5 | 10.8 | 5.6 | +| decision + receipt | 1000 | 264.9 | 365.3 | 492.5 | 283.1 | +| receipt verification | 1000 | 279.4 | 382.4 | 491.1 | 295.7 | +| **complete end-to-end** | **1000** | **574.0** | **1052.5** | **1111.2** | **633.1** | + +Run-to-run variation on this shared machine is roughly ±15% on the end-to-end +figure. The relative cost of stages is stable across runs; treat the absolute +numbers as an order-of-magnitude baseline, not a precise constant. + +Stage figures are **not additive** into the end-to-end figure: `decision + +receipt` already contains projection and action check. + +### What the distribution says + +- **Canonicalization is the dominant primitive cost.** At ~91 µs it is roughly + 16× the action check and accounts for most of validation, digest, and + projection cost. Every one of those stages canonicalizes. +- **The authorization logic itself is nearly free.** Action check is 5.6 µs — + under 1% of the end-to-end path. Cost is overwhelmingly in canonical identity, + not in deciding. +- **Verification costs as much as deciding** (296 µs vs 283 µs), because + `verify_receipt` recomputes every binding from raw inputs rather than trusting + any field in the receipt. This is the property that makes receipts + independently checkable; the cost is the point, not a defect. +- **p99 is ~1.9× p50**, with no long tail — consistent with a pure-CPU path with + no I/O, locking, or allocation cliffs. + +--- + +## 5. Throughput + +Two figures, deliberately reported separately. Conflating them is how benchmarks +overstate capacity. + +### 5a. Observed sustained rate — the real measurement + +Continuous single-process, single-threaded loop over a fixed wall-clock window. +**3 trials × 5 s each, 1 s warmup per trial.** + +| Operation | min | **median** | max | total ops | +|---|---:|---:|---:|---:| +| Decision + receipt | 3,258.9 | **3,383.8** | 3,423.9 | 50,335 | +| Receipt verification | 3,178.2 | **3,212.9** | 3,237.4 | 48,145 | +| **Complete end-to-end** | **1,512.9** | **1,574.4** | **1,587.5** | **23,375** | + +Trial spread is under 7% for every operation, so the median is a stable +estimate rather than a lucky draw. + +### 5b. Latency-derived rate — arithmetic, not measurement + +| Operation | Derived rate | +|---|---:| +| Decisions / sec | 3,532.7 | +| Receipts / sec | 3,532.7 | +| Receipt verifications / sec | 3,382.0 | +| Complete E2E transactions / sec | 1,579.6 | + +This is the reciprocal of mean latency. It assumes zero loop overhead and no +drift under continuous operation, so it is an *upper-bound estimate*, not an +observed capacity. + +### What the gap says + +The two agree to within ~0.3% on the end-to-end path (1,579.6 derived vs 1,574.4 +observed). The derived figure runs slightly high, which is the expected +direction: it excludes loop overhead that real sustained operation pays. The +direction is not perfectly consistent across runs — an earlier run had the +observed figure slightly *above* the derived one, because per-sample +`perf_counter` overhead in the latency measurement can itself exceed loop +overhead. Both readings sit inside run-to-run noise; the honest summary is that +the two methods agree closely here, and the observed figure is the one to quote. + +Decisions/sec and receipts/sec are the *same measurement*: receipt emission is +not separately callable at this commit. A complete E2E transaction is +decide-plus-independently-verify, which is why it is roughly half the decision +rate. + +**Not claimed:** distributed throughput, multi-core scaling, or throughput under +concurrency. No concurrency layer exists at this commit and none was introduced +to measure one. + +--- + +## 6. Scale curves + +### Declared mediated actions — sublinear to linear + +| Actions | mean latency | peak traced memory | decision | +|---:|---:|---:|---| +| 1 | 280 µs | 6.1 KiB | ALLOW | +| 10 | 1,009 µs | 9.4 KiB | ALLOW | +| 100 | 8,127 µs | 40.8 KiB | ALLOW | +| 1,000 | 78,032 µs | 366.1 KiB | ALLOW | + +1000× the domain size costs ~279× the time and ~60× the memory. Growth is +linear in the large-N regime (the 100→1000 step costs ~9.6× for 10× the size) +with a fixed overhead that dominates at small N, which is why the endpoint ratio +reads as sublinear. + +### Required facts — approximately linear + +| Facts | mean latency | peak traced memory | decision | +|---:|---:|---:|---| +| 10 | 954 µs | 15.8 KiB | ALLOW | +| 100 | 6,978 µs | 140.7 KiB | ALLOW | +| 1,000 | 68,148 µs | 1.3 MiB | ALLOW | +| 10,000 | 648,173 µs | 13.1 MiB | ALLOW | + +1000× the fact count costs ~680× the time and ~852× the memory — linear in +both, with no observed nonlinear degradation or cliff in the tested range. Each +10× step costs ~8–10× consistently. + +**Practical reading:** a 10,000-fact contract takes ~0.65 s and ~13 MiB per +decision. That is workable for batch evaluation and marginal for interactive +use. The linearity means cost is predictable; the constant is dominated by +repeated canonicalization (§8, finding F2). + +### NOT EVALUATED + +| Dimension | Why | +|---|---| +| Multi-contract corpora | The runtime evaluates one artifact per invocation. No registry or cross-contract selection path exists whose scaling could be measured without inventing architecture. | +| Concurrent / distributed throughput | No concurrency or distribution layer exists at this commit. | +| Persistent storage scaling | The runtime is stateless over in-memory inputs. No storage backend exists to scale. | + +--- + +## 7. Determinism + +**Fully deterministic over fixed inputs**, across 100 repeated executions: + +| Output | Stable? | +|---|---| +| Decision | yes | +| Reason code | yes | +| Contract digest | yes (1 distinct value / 100 runs) | +| Projection | yes (1 distinct value / 100 runs) | +| All 10 protected receipt fields | yes | + +Notably `decision_time` is **stable**, because it is bound to the fact bundle's +own declared `now` rather than to wall-clock time at invocation. At this commit +there is therefore **no intentionally-varying receipt field** — the entire +receipt is byte-identical across replays of a fixed specimen. + +**Scope of the claim:** this is determinism over fixed inputs in a single +process on one platform and one Python version. Cross-version, cross-platform, +and cross-implementation reproducibility were **not** tested and are not +claimed. + +--- + +## 8. Findings + +Recorded during the measurement run, not repaired during it. + +**AC-035-F1 — Admission approvals are bound as evidence but are not an +authorization gate.** +Forging `admission.approvals` yields `ALLOW`. The evidence binding does hold: +`admission_digest` and `receipt_digest` both change, and a receipt issued for +the unforged admission does not verify against the forged one. So the forgery is +*detectable after the fact* but is not *prevented at decision time*. Whether +approvals should gate the decision is an unimplemented policy capability, not a +break in the evidence chain. + +**AC-035-F2 — Canonicalization is repeated several times per transaction.** +Validation, digest, and projection each canonicalize, and `verify_receipt` +re-runs the whole decision path. At ~91 µs per canonicalization this is the +dominant cost and the clearest optimization target. It is *correct* — recomputing +rather than trusting is the security property — but it is not currently *cached* +within a single transaction. + +**AC-035-F3 — No replay protection semantics exist.** +A replayed identical request produces an identical receipt. This documents +determinism, not replay *protection*: there is no nonce, sequence number, or +single-use semantics, so an intercepted receipt is indistinguishable from a +legitimately re-derived one. Recorded as an architectural gap, not a defect +against current specified behaviour. + +**AC-035-F4 — `*_mutated.json` fixture names are misleading.** +`banking_payment_specimen_contract_mutated.json` and +`..._admission_mutated.json` are mutated *and correctly re-sealed*, so they are +validly-bound variant contracts rather than mutation attacks. The first draft of +this suite expected them to refuse; that expectation was wrong, not the +implementation. Genuine post-binding mutation (stale digest) is correctly +refused with `AC_DIGEST`. + +--- + +## 9. Resource profile + +| Metric | Value | +|---|---| +| Interpreter + import cost | 61–88 ms (mean 68 ms), out-of-process | +| Peak traced memory, one decision | 6.07 KiB | +| Peak traced memory, one verification | 6.19 KiB | +| Process max RSS (whole benchmark) | ~63 MiB | + +Artifact sizes (compact JSON, bytes): + +| Artifact | Size | +|---|---:| +| Contract artifact | 1,179 | +| Contract body | 773 | +| Canonical contract bytes | 773 | +| Action | 148 | +| Fact bundle | 675 | +| Projection | 479 | +| Receipt | 715 | + +Process startup (~68 ms) is **more than two orders of magnitude larger** than a +single decision (~0.28 ms). Any deployment shape that pays interpreter startup +per decision would be dominated entirely by startup. + +--- + +## 10. Limitations + +- One synthetic banking specimen family. No real institutional rule has been + integrated or evaluated. +- One machine, one Python version, one OS. No cross-platform comparison. +- Single process. No concurrency, no distribution, no persistence. +- Scale curves are synthetic: generated by widening a real specimen, not drawn + from production corpora. +- Percentiles are nearest-rank, computed from in-process `perf_counter` samples. +- Memory figures are Python-level allocation (`tracemalloc`), not RSS + attributable per decision. + +## 11. Claim ceiling + +This benchmark establishes only what it measures. It does **not** establish +production readiness, regulatory correctness, legal correctness, universal +source-to-rule derivation, arbitrary-domain compatibility, security +certification, distributed scalability, formal proof, or industry-wide +superiority. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md new file mode 100644 index 0000000..2ed52cb --- /dev/null +++ b/docs/ROADMAP.md @@ -0,0 +1,164 @@ +# AuthContract roadmap + +Derived from the AC-035 benchmark run measuring implementation +`e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` (DUT), with harness +`a7f6ba374b1362e624a3f8b912b265dd03da4cdd`. Every item below cites the +observed evidence that motivates it. Items with no supporting measurement are +placed under **Research / not yet established** and are explicitly not +commitments. + +Raw evidence: [`benchmarks/results/`](../benchmarks/results/) · +Analysis: [`docs/BENCHMARKS.md`](BENCHMARKS.md) + +--- + +## NOW +*Required to move from current MVP-alpha to a stronger developer-testable alpha.* + +### N1 — Cache canonicalization within a transaction + +| | | +|---|---| +| **Evidence** | Canonicalization costs ~91 µs mean (p50 83.9, p95 143.6) and is performed by validation, digest, and projection independently; `verify_receipt` then re-runs the entire decision path. Action check by contrast is 5.6 µs. Canonicalization dominates the ~633 µs end-to-end path. | +| **Limitation** | The same contract body is canonicalized several times per transaction with no intra-transaction reuse. | +| **Capability** | Canonicalize once per artifact per transaction and reuse the bytes across validation, digest, and projection. | +| **Acceptance** | End-to-end p50 improves measurably against the 574.0 µs recorded baseline with **zero** change to any correctness or adversarial specimen result, and digests remain byte-identical to the recorded baseline. | +| **Dependency** | None. | +| **Maturity impact** | Removes the dominant cost without touching security semantics. | + +### N2 — Decide whether admission approvals gate authorization + +| | | +|---|---| +| **Evidence** | Finding AC-035-F1: forged `admission.approvals` yields `ALLOW`. The binding holds — `admission_digest` and `receipt_digest` both change and cross-verification fails — but nothing gates on approval content at decision time. | +| **Limitation** | Approvals are carried and bound as evidence but are never evaluated. A reader may reasonably assume otherwise. | +| **Capability** | Either (a) implement approval evaluation as a declared gate, or (b) document explicitly that approvals are evidence-only at this maturity. | +| **Acceptance** | If (a): a specimen with insufficient approvals REFUSEs with a distinct reason code, with positive and negative fixtures. If (b): README and contract-shape docs state the boundary and a regression test pins it. | +| **Dependency** | None. | +| **Maturity impact** | Closes an ambiguity that currently invites over-reading of what the system enforces. | + +### N3 — Rename the misleading `*_mutated.json` fixtures + +| | | +|---|---| +| **Evidence** | Finding AC-035-F4: the fixtures are mutated *and re-sealed*, so they are validly-bound variants. The first draft of the benchmark suite mis-expected them to refuse. | +| **Limitation** | Fixture names actively suggest a security property they do not exercise. | +| **Capability** | Rename to reflect what they are (e.g. `..._variant_resealed.json`) and add genuinely stale-bound counterparts as committed fixtures. | +| **Acceptance** | Names describe content; the stale-binding attack exists as a fixture, not only as programmatic construction in the benchmark. | +| **Dependency** | None. | +| **Maturity impact** | Removes a documented tripwire for external evaluators. | + +### N4 — Publish the benchmark as a regression gate + +| | | +|---|---| +| **Evidence** | The suite runs in ~100 s and exits non-zero on any correctness failure; 342 existing tests pass alongside it. | +| **Limitation** | Benchmark results are a point-in-time artifact; nothing prevents silent regression of latency or of the 38-specimen adversarial matrix. | +| **Capability** | Run the correctness and adversarial matrices in CI; track latency with a tolerance band rather than a hard threshold. | +| **Acceptance** | CI fails on any adversarial regression; latency drift is reported without failing on noise. | +| **Dependency** | N1 (so the tracked baseline is the post-optimization one). | +| **Maturity impact** | Converts a one-off measurement into a standing guarantee. | + +--- + +## NEXT +*Required for realistic external engineering evaluation.* + +### X1 — Multi-contract evaluation and selection + +| | | +|---|---| +| **Evidence** | Scale curves recorded `NOT EVALUATED` for multi-contract corpora: the runtime evaluates one artifact per invocation and has no registry or cross-contract selection path. | +| **Limitation** | Any realistic deployment holds many contracts; none of that behaviour exists or can be measured. | +| **Capability** | A contract registry with deterministic selection and explicit conflict/overlap semantics. | +| **Acceptance** | Scaling measured across 1/10/100/1000 contracts; overlapping-scope conflicts resolve deterministically or refuse explicitly; `select_matching_projection` semantics are covered by adversarial specimens. | +| **Dependency** | None. | +| **Maturity impact** | Removes the largest single `NOT EVALUATED` gap. | + +### X2 — Replay protection semantics + +| | | +|---|---| +| **Evidence** | Finding AC-035-F3: replayed identical requests produce identical receipts. Determinism is confirmed; replay *protection* does not exist. | +| **Limitation** | An intercepted receipt is indistinguishable from a legitimately re-derived one. | +| **Capability** | Nonce, sequence, or single-use decision semantics, with an explicit statement of the threat model addressed. | +| **Acceptance** | A replayed request is distinguishable from a fresh one; adversarial specimens cover replay within and across freshness windows. | +| **Dependency** | Requires deciding whether the runtime may hold state — it is currently stateless. | +| **Maturity impact** | Addresses a threat class the current design does not cover at all. | + +### X3 — Reduce verification cost, or justify it explicitly + +| | | +|---|---| +| **Evidence** | Verification (296 µs mean) costs ~as much as the original decision (283 µs mean) because it recomputes every binding. | +| **Limitation** | A verifier-heavy workload costs the same as a decider-heavy one; that is a deliberate trade but is undocumented as a capacity-planning input. | +| **Capability** | Either reduce redundant work inside `verify_receipt` (subject to N1) while preserving zero trust in receipt fields, or document the cost as an intentional property with guidance. | +| **Acceptance** | Verification remains fully independent — no receipt field trusted — and either measurably improves or is documented with a capacity-planning note. | +| **Dependency** | N1. | +| **Maturity impact** | Makes verification cost a designed, stated property rather than an emergent one. | + +### X4 — Cross-platform and cross-version determinism + +| | | +|---|---| +| **Evidence** | Determinism is confirmed for one process, one platform, one Python version. §7 explicitly does not claim more. | +| **Limitation** | Canonical identity is only useful across parties if it is stable across their environments — untested. | +| **Capability** | Reproduce digests across Python versions, OSes, architectures, and ideally an independent implementation. | +| **Acceptance** | Identical contract digests and receipts across at least two Python versions and two OSes, published as evidence. | +| **Dependency** | None. | +| **Maturity impact** | Upgrades determinism from a local observation to a portable property — a precondition for third-party verification. | + +### X5 — Real external rule evaluation + +| | | +|---|---| +| **Evidence** | Every measurement uses one synthetic banking specimen family. The runbook's own claim ceiling states no external rule has been integrated. | +| **Limitation** | Nothing demonstrates the contract shape can express a rule authored outside this project. | +| **Capability** | Encode at least one externally-authored rule end-to-end and publish the gaps found. | +| **Acceptance** | An external rule is evaluated, and every unsupported construct is reported as a finding rather than worked around by reshaping the rule. | +| **Dependency** | Possibly X1. | +| **Maturity impact** | First evidence of generality beyond the synthetic specimen. | + +--- + +## LATER +*Required for production-class deployment.* + +### L1 — Concurrency and sustained-load behaviour +**Evidence:** throughput is single-process only; concurrency recorded `NOT EVALUATED`. +**Acceptance:** throughput and tail latency under sustained concurrent load, with a stated saturation point. +**Dependency:** X1. + +### L2 — Interpreter startup amortization +**Evidence:** startup ~68 ms vs ~0.28 ms per decision — a ~240× ratio. Any per-decision-process deployment is dominated by startup. +**Acceptance:** a long-lived service or batch shape whose measured per-decision cost approaches the in-process figure. +**Dependency:** L1. + +### L3 — Large-corpus operating envelope +**Evidence:** 10,000 facts → ~0.65 s and ~13 MiB per decision, linear. +**Acceptance:** documented supported envelope with measured limits, plus explicit refusal or degradation beyond it. +**Dependency:** N1, X1. + +### L4 — Independent security review +**Evidence:** 38 adversarial specimens pass, but all were authored by the same party that wrote the implementation. +**Acceptance:** external review with findings published unmodified. +**Dependency:** X4, X5. + +--- + +## RESEARCH / NOT YET ESTABLISHED +*Requires architecture, formalization, or experiment that does not exist. Not commitments.* + +- **Automated natural-language source-to-rule derivation.** Not implemented; no + measurement exists. Would require a validation methodology establishing that a + derived rule faithfully represents its source — itself an open problem. +- **Formal proof of the authorization semantics.** No formal model exists. The + 38-specimen battery is testing, not proof. +- **Cross-jurisdiction / cross-domain generality.** One synthetic banking family + measured. Any claim of generality needs evidence from materially different + domains. +- **Distributed consensus on decisions.** No distribution layer exists; there is + no architecture to evaluate. +- **Regulatory or legal sufficiency of receipts as evidence.** Entirely outside + what any measurement here can establish; requires legal analysis, not + benchmarking. diff --git a/docs/TRL-ASSESSMENT.md b/docs/TRL-ASSESSMENT.md new file mode 100644 index 0000000..15b4a55 --- /dev/null +++ b/docs/TRL-ASSESSMENT.md @@ -0,0 +1,111 @@ +# AuthContract — TRL assessment (AC-035) + +**Implementation assessed (DUT):** `e4e1a97509df1a66c44b090c0a0ca0a03907f4dc` +**Measured by harness:** `a7f6ba374b1362e624a3f8b912b265dd03da4cdd` +**Basis:** the AC-035 benchmark run only, as amended by AC-035A. Architecture documents, design intent, +and roadmap items are explicitly **not** counted as evidence. + +--- + +## Evidence classification + +The distinction that matters for TRL is *what kind* of evidence exists, not how +much of the system is written. + +| Class | Status | What supports it | +|---|---|---| +| **Implemented** | YES | Canonical digest, projection, fact admission, decision path, receipt emission, receipt verification, Git merge-result gate. 2,401 lines across 8 runtime modules. | +| **Demonstrated** | YES | Complete E2E path executes: 7/7 E2E specimens, 38/38 adversarial specimens, 342 regression tests pass. | +| **Benchmarked** | YES | Per-stage latency distributions (n=1000–2000), *observed* sustained throughput (3 trials × 5 s per operation, not merely derived from latency), two scale curves, determinism over 100 replays, resource profile — all against a DUT verified byte-identical to the declared base commit before measuring. This document's basis. | +| **Externally validated** | **NO** | Every specimen, fixture, and expectation was authored by the same party that authored the implementation. No external rule has been encoded; no third party has reproduced or reviewed the results. Clean-room external *testability* was established in AC-028, but by this same executor — not by an independent evaluator. | +| **Production validated** | **NO** | No deployment, no real workload, no concurrency, no persistence, no operational history. | + +--- + +## Current TRL: **4**, with partial TRL 5 characteristics + +**TRL 4 — component and/or breadboard validation in a laboratory environment.** + +### Why TRL 4 is met + +- The complete architectural path runs end to end, not merely in parts. +- Behaviour is validated against an adversarial battery of 38 specimens + spanning missing/unknown fields, malformed types, out-of-domain values, + staleness, evidence divergence, digest mutation, substitution, and + reordering — all failing closed. +- Tamper detection is exhaustive over the protected surface: all 20 + mutation-and-truncation variants of the receipt are detected. +- Behaviour is deterministic and measured, not asserted: 100 replays produce + byte-identical receipts, and the digest and projection primitives each yield + exactly one distinct value across 100 runs. +- Performance is characterized with distributions rather than single timings, + and scaling is linear with no observed cliff across 1000× in two dimensions. + +### Why TRL 5 is *not* met + +TRL 5 requires validation in a **relevant** environment. Every element of the +measured environment is laboratory-constructed: + +- One synthetic banking specimen family; no externally-authored rule has ever + been evaluated (roadmap X5). +- Single process, single machine, no concurrency, no persistence, no + operational load (roadmap L1, L2). +- Multi-contract evaluation — the shape any real deployment takes — does not + exist and was recorded `NOT EVALUATED` (roadmap X1). +- Determinism is confirmed on exactly one platform and one Python version; + cross-environment reproducibility, which is what makes canonical identity + useful between parties, is untested (roadmap X4). + +### Why the partial TRL 5 characteristics are real + +The rigor of the *evidence discipline* exceeds typical TRL 4: measured +distributions rather than anecdotes, an adversarial matrix rather than +happy-path demos, findings recorded rather than silently repaired (AC-035-F1 +through F4), explicit `NOT EVALUATED` markers rather than estimates, observed +sustained throughput rather than a latency reciprocal, and a device-under-test +whose identity is verified before measurement rather than asserted afterwards. +That methodological maturity is a genuine TRL 5 characteristic. + +It does not by itself raise the TRL. **TRL is determined by the environment the +evidence was gathered in, not by the quality of the measurement** — improving +benchmark methodology (as AC-035A did) makes the TRL 4 assessment *better +supported*, not higher. External independent reproduction remains absent, which +is the binding constraint. + +--- + +## What would move AuthContract to TRL 5 + +All four are necessary; none alone is sufficient. + +1. **An externally-authored rule evaluated end to end** (roadmap X5), with every + unsupported construct reported as a finding rather than worked around by + reshaping the rule to fit. +2. **Multi-contract evaluation implemented and measured** (roadmap X1), + including deterministic selection and explicit conflict semantics — closing + the largest `NOT EVALUATED` gap. +3. **Cross-platform and cross-version determinism demonstrated** (roadmap X4): + identical digests and receipts across at least two Python versions and two + operating systems. Canonical identity that only holds in one environment + cannot support third-party verification. +4. **Independent reproduction by a party that did not author the system** + (roadmap L4) — someone else running the benchmark, on their hardware, + reaching the same correctness results. + +## What would move it to TRL 6 + +Beyond the above: sustained operation under concurrent load with a stated +saturation point (L1), a documented operating envelope with measured limits +(L3), and an independent security review whose findings are published +unmodified (L4). + +--- + +## Claim ceiling + +This assessment is bounded by the measurements in +[`docs/BENCHMARKS.md`](BENCHMARKS.md). It does not establish production +readiness, regulatory or legal correctness, universal source-to-rule +derivation, arbitrary-domain compatibility, security certification, distributed +scalability, formal correctness, or comparative standing against any other +system.