From be71f26cc90979bf603d630e203a8c5230017c98 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Sat, 25 Jul 2026 23:17:37 -0400 Subject: [PATCH 01/14] docs(plan): pre-flight corrections to the Slice 4a plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three defects found by scanning the plan against the real repo before dispatching any implementer: - The Task 2 test helper called loop.contract.scaffold_contract, which does not exist. The scaffold entry point is loop.scaffold.scaffold (loop/scaffold.py:104). - Task 7 attested examples/flaky-test-triage, but no tracked examples/* contract ships an events.db — event stores are runtime artifacts and .loop/ is gitignored. An empty chain head makes Task 6 Step 4 skip the attest step, so the job would pass having attested nothing. It now seeds a chained workspace with the existing scripts/ci_anchor_probe.py (the pattern ci.yml chain-anchor already uses) and fails when no attestation URL is produced or the observed head differs from the seeded one. - Recorded the verified doctor_report shape: it also returns paths (absolute, and deliberately excluded from the predicate) and requested_mode; validation_mode is the single token jsonschema, so the no-whitespace assertion in Task 5 is safe. --- .../2026-07-25-slice4a-verdict-emission.md | 79 ++++++++++++++----- 1 file changed, 61 insertions(+), 18 deletions(-) diff --git a/docs/superpowers/plans/2026-07-25-slice4a-verdict-emission.md b/docs/superpowers/plans/2026-07-25-slice4a-verdict-emission.md index 270d334..44d1b59 100644 --- a/docs/superpowers/plans/2026-07-25-slice4a-verdict-emission.md +++ b/docs/superpowers/plans/2026-07-25-slice4a-verdict-emission.md @@ -311,10 +311,10 @@ import pytest def _scaffold_terminal(tmp_path, name, state="Succeeded", policy="all_required"): """A doctor-clean scaffold advanced to a terminal record.""" - from loop.contract import scaffold_contract # existing scaffold entry point + from loop.scaffold import scaffold target = tmp_path / name - scaffold_contract(target) + scaffold(target) terminal = { "schema": "loop-engineer/terminal@1", "state": state, @@ -357,11 +357,11 @@ def test_issue_codes_are_sorted_deduplicated_and_carry_no_detail(tmp_path): def test_build_verdict_refuses_a_workspace_with_no_terminal_record(tmp_path): - from loop.contract import scaffold_contract + from loop.scaffold import scaffold from loop.verdict import VerdictError, build_verdict target = tmp_path / "no-terminal" - scaffold_contract(target) + scaffold(target) with pytest.raises(VerdictError, match="no terminal record"): build_verdict(target) @@ -379,7 +379,7 @@ def test_build_verdict_refuses_a_nonexistent_target(tmp_path): Run: `uv run --with pyyaml --with jsonschema --with pytest python3 -B -m pytest -q -p no:cacheprovider scripts/test_verdict.py -v` Expected: FAIL with `ImportError: cannot import name 'build_verdict'` -> If `scaffold_contract` is not the exported scaffold name at HEAD, run `uv run --with pyyaml python3 -B -c "import loop.contract as c; print([n for n in dir(c) if 'scaffold' in n])"` and use the real one in the helper. Do not invent an API. +> Verified at HEAD: the scaffold entry point is `loop.scaffold.scaffold(target)` (`loop/scaffold.py:104`). There is no `scaffold_contract`. `LoopPaths` also exposes `.terminal` directly (`loop/paths.py:24`), so prefer `paths.terminal` over `paths.loop_dir / "terminal_state.json"`. - [ ] **Step 3: Implement the projection** @@ -1033,10 +1033,19 @@ git commit -m "feat(action): opt-in keyless attestation of the verdict predicate - Create: `.github/workflows/attest.yml` **Interfaces:** -- Consumes: the composite action from Task 6. -- Produces: a real attestation on every push to the default branch, over a **tracked** `examples/*` contract. +- Consumes: the composite action from Task 6; the existing `scripts/ci_anchor_probe.py`. +- Produces: a real attestation on every push to the default branch, over a **seeded chained workspace**. -Never point this at the live gitignored `.loop/` — CI runs on a fresh checkout where it does not exist. +**Pre-flight correction.** An earlier draft pointed this job at `examples/flaky-test-triage`. That is +wrong: **no tracked `examples/*` contract ships an `events.db`** (verified — event stores are runtime +artifacts and `.loop/` is gitignored). With no store the chain head is empty, Task 6 Step 4's guard +skips the attest step, and the job goes green having attested nothing — an unfalsifiable CI job, which +is the exact false-completion shape this project exists to refuse. + +Reuse the pattern the repo already proved. `ci.yml`'s `chain anchor (live end-to-end)` job seeds a +chained workspace via `python -B scripts/ci_anchor_probe.py "$workspace"`, which prints the resulting +head; its own comment states the rationale verbatim — *"action-dogfood gates a store-free example, so +the anchor surface has no live cover there."* Never point this at the live gitignored `.loop/`. - [ ] **Step 1: Write the workflow** @@ -1060,25 +1069,59 @@ jobs: id-token: write attestations: write steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v7 + + - uses: actions/setup-python@v7 + with: + python-version: "3.12" + + - name: Install probe dependencies + run: python -m pip install --upgrade pip pyyaml jsonschema + + - name: Seed a chained workspace + id: seed + # Same probe the chain-anchor job uses. A store-free example would make + # the attest step skip and this job unfalsifiable. + run: | + workspace="${RUNNER_TEMP}/verdict-ws" + head="$(python -B scripts/ci_anchor_probe.py "$workspace")" + echo "head=$head" >> "$GITHUB_OUTPUT" + echo "workspace=$workspace" >> "$GITHUB_OUTPUT" + - id: gate uses: ./ with: - path: examples/flaky-test-triage + path: ${{ steps.seed.outputs.workspace }} attest: "true" - - name: record + + - name: assert an attestation was actually created + # Without this the job passes when the attest step skips, which is the + # failure mode this whole task exists to avoid. + env: + SEEDED_HEAD: ${{ steps.seed.outputs.head }} + OBSERVED_HEAD: ${{ steps.gate.outputs.chain-head }} + ATTESTATION: ${{ steps.gate.outputs.attestation-url }} run: | - echo "chain head: ${{ steps.gate.outputs.chain-head }}" >> "$GITHUB_STEP_SUMMARY" - echo "attestation: ${{ steps.gate.outputs.attestation-url }}" >> "$GITHUB_STEP_SUMMARY" + if [ -z "$ATTESTATION" ]; then + echo "::error::attest was requested but no attestation URL was produced" + exit 1 + fi + if [ "$OBSERVED_HEAD" != "$SEEDED_HEAD" ]; then + echo "::error::gate observed head '$OBSERVED_HEAD', seed produced '$SEEDED_HEAD'" + exit 1 + fi + echo "chain head: $OBSERVED_HEAD" >> "$GITHUB_STEP_SUMMARY" + echo "attestation: $ATTESTATION" >> "$GITHUB_STEP_SUMMARY" ``` -- [ ] **Step 2: Confirm the checkout action major matches the repo's other workflows** +- [ ] **Step 2: Confirm the action majors match the repo's other workflows** ```bash -grep -rn "actions/checkout@" .github/workflows/ +grep -rn "actions/checkout@\|actions/setup-python@" .github/workflows/ ``` -Use whatever major `ci.yml` uses. A mismatched pin is a dependabot PR waiting to happen. +Pre-flight observed `actions/checkout@v7` and `actions/setup-python@v7` in `ci.yml`; confirm before +committing. A mismatched pin is a dependabot PR waiting to happen. - [ ] **Step 3: Commit** @@ -1227,6 +1270,6 @@ Carried from ADR 0002. None blocks starting; each blocks the task that touches i 1. **Subject digest algorithm** (Task 6). The chain head is not a hash of retrievable bytes, so the `sha256` DigestSet key licenses a false inference. A namespaced key is correct in in-toto terms but `actions/attest`'s `subject-digest` may accept only `sha256:`. Resolve by experiment. If the input constrains us, §23 must carry the disambiguation instead. 2. **`create-storage-record` / `push-to-registry` defaults** (Task 6). Pass both explicitly so the two-permission claim is true by construction. -3. **`scaffold_contract` name** (Task 2 Step 2). Confirm the real scaffold entry point before writing the test helper. -4. **`event_store` key names** (Task 2 Step 5). Confirm `run_id`, `chain.head.event_hash`, `chain.head.sequence`, `chain.unchained_prefix` against real doctor output. +3. ~~`scaffold_contract` name~~ — **RESOLVED in pre-flight.** The entry point is `loop.scaffold.scaffold`; the plan now uses it. +4. **`event_store` key names** (Task 2 Step 5). Pre-flight confirmed the store-absent shape is exactly `{"present": false}` and that `doctor_report` also returns `paths` (absolute filesystem paths — deliberately excluded from the predicate) and `requested_mode`. `validation_mode` is the single token `"jsonschema"`, so Task 5's no-whitespace assertion is safe. The populated `chain.head.*` key names still need confirming against a seeded store. 5. **The chain-bound evidence map producer** (Task 3 Step 4). Reuse `loop/contract.py`'s, never re-derive. From 7a00cdc682b0df8999ba69003bca4a624c33755d Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Sun, 26 Jul 2026 09:31:00 -0400 Subject: [PATCH 02/14] feat(verdict): add the verdict@1 predicate schema and module skeleton Task 1 of Slice 4a (ADR 0002). Adds schemas/verdict.schema.json, the loop/verdict.py skeleton (VERDICT_SCHEMA_ID, PREDICATE_TYPE, VerdictError, _load_verdict_schema), and three tests. The kernel builds a predicate body; it never signs one and never constructs an in-toto Statement. Every digest field pairs a 64-hex pattern with maxLength 64, because jsonschema pattern matching uses re.search semantics and would otherwise accept a trailing newline. Authored via the Claudex lane (gpt-5.6-terra/medium, session 019f9c74), verified outside the worker: scope 3/3, full suite 1308 passed / 18 skipped against a 1305/18 baseline, independent claude-sonnet-5 review PASS. Receipt cx_s4a_t1_verdict_schema_a1. --- loop/verdict.py | 25 +++++++++++++ schemas/verdict.schema.json | 75 +++++++++++++++++++++++++++++++++++++ scripts/test_verdict.py | 26 +++++++++++++ 3 files changed, 126 insertions(+) create mode 100644 loop/verdict.py create mode 100644 schemas/verdict.schema.json create mode 100644 scripts/test_verdict.py diff --git a/loop/verdict.py b/loop/verdict.py new file mode 100644 index 0000000..b75bfa9 --- /dev/null +++ b/loop/verdict.py @@ -0,0 +1,25 @@ +"""Project a loop run into a loop-engineer/verdict@1 predicate body. + +This module builds a document. It NEVER signs one, never verifies a signature, +never constructs an in-toto Statement, and never reads an environment variable. +The signer lane (action.yml -> actions/attest) owns the envelope, the subject, +and every cryptographic operation. See docs/adr/0002-ci-attested-verdict.md. +""" + +from __future__ import annotations + +import json +from typing import Any + +from ._resources import schemas_dir + +VERDICT_SCHEMA_ID = "loop-engineer/verdict@1" +PREDICATE_TYPE = "urn:loop-engineer:verdict:1" + + +class VerdictError(ValueError): + """A verdict cannot be projected from this workspace.""" + + +def _load_verdict_schema() -> dict[str, Any]: + return json.loads((schemas_dir() / "verdict.schema.json").read_text(encoding="utf-8")) diff --git a/schemas/verdict.schema.json b/schemas/verdict.schema.json new file mode 100644 index 0000000..70ad538 --- /dev/null +++ b/schemas/verdict.schema.json @@ -0,0 +1,75 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "loop-engineer/verdict@1", + "title": "Loop Engineer Verdict @1", + "description": "A CI-attestable projection of a loop run's doctor verdict, chain head, terminal record, and verified-evidence digests. Emitted as an in-toto predicate body; the signer constructs the Statement.", + "type": "object", + "additionalProperties": false, + "required": ["schema", "run_id", "tool", "doctor", "chain", "terminal", "evidence"], + "properties": { + "schema": { "const": "loop-engineer/verdict@1" }, + "run_id": { "type": "string", "minLength": 1, "maxLength": 256 }, + "tool": { + "type": "object", + "additionalProperties": false, + "required": ["name", "version"], + "properties": { + "name": { "const": "loop-engineer" }, + "version": { "type": ["string", "null"], "maxLength": 64 } + } + }, + "doctor": { + "type": "object", + "additionalProperties": false, + "required": ["ok", "validation_mode", "issue_codes", "schemas_checked"], + "properties": { + "ok": { "type": "boolean" }, + "validation_mode": { "type": "string", "maxLength": 64 }, + "issue_codes": { + "type": "array", + "items": { "type": "string", "pattern": "^[a-z0-9_]{1,64}$", "maxLength": 64 } + }, + "schemas_checked": { + "type": "array", + "items": { "type": "string", "maxLength": 128 } + } + } + }, + "chain": { + "type": "object", + "additionalProperties": false, + "required": ["head", "sequence", "unchained_prefix"], + "properties": { + "head": { "type": ["string", "null"], "pattern": "^[0-9a-f]{64}$", "maxLength": 64 }, + "sequence": { "type": ["integer", "null"], "minimum": 0 }, + "unchained_prefix": { "type": "integer", "minimum": 0 } + } + }, + "terminal": { + "type": "object", + "additionalProperties": false, + "required": ["state", "completion_policy", "false_completion"], + "properties": { + "state": { + "enum": ["Succeeded", "FailedUnverifiable", "FailedBlocked", "FailedBudget", + "FailedSafety", "FailedSpecGap", "AbortedByHuman"] + }, + "completion_policy": { "type": ["string", "null"], "maxLength": 64 }, + "false_completion": { "type": "boolean" } + } + }, + "evidence": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["digest", "code_digest", "policy_digest"], + "properties": { + "digest": { "type": "string", "pattern": "^[0-9a-f]{64}$", "maxLength": 64 }, + "code_digest": { "type": ["string", "null"], "pattern": "^[0-9a-f]{64}$", "maxLength": 64 }, + "policy_digest": { "type": ["string", "null"], "pattern": "^[0-9a-f]{64}$", "maxLength": 64 } + } + } + } + } +} diff --git a/scripts/test_verdict.py b/scripts/test_verdict.py new file mode 100644 index 0000000..61cb376 --- /dev/null +++ b/scripts/test_verdict.py @@ -0,0 +1,26 @@ +import json + + +def test_schema_id_and_predicate_type_are_pinned(): + from loop.verdict import PREDICATE_TYPE, VERDICT_SCHEMA_ID + + assert VERDICT_SCHEMA_ID == "loop-engineer/verdict@1" + assert PREDICATE_TYPE == "urn:loop-engineer:verdict:1" + + +def test_schema_file_declares_the_matching_id(): + from loop._resources import schemas_dir + from loop.verdict import VERDICT_SCHEMA_ID + + schema = json.loads((schemas_dir() / "verdict.schema.json").read_text(encoding="utf-8")) + assert schema["$id"] == VERDICT_SCHEMA_ID + assert schema["$schema"] == "https://json-schema.org/draft/2020-12/schema" + + +def test_verdict_schema_is_not_a_contract_artifact(): + # SCHEMA_IDS is the contract-object tuple (manifest/state/tasks/terminal). + # verdict@1 is a projection, not a contract object, and must stay out of it. + from loop.contract import SCHEMA_IDS + from loop.verdict import VERDICT_SCHEMA_ID + + assert VERDICT_SCHEMA_ID not in SCHEMA_IDS From 2ba257f62c19d1f5dbc7d48fbb3f4ef8a080f974 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Sun, 26 Jul 2026 14:02:46 -0400 Subject: [PATCH 03/14] feat(verdict): project doctor, chain, and terminal into verdict@1 Task 2 of Slice 4a (ADR 0002). Adds build_verdict(target, *, mode=None), projecting a workspace's doctor report, event-chain head, and terminal record into a verdict@1 predicate body. Evidence stays empty until Task 3. Fail-closed on the terminal record: a false_completion that is absent or not a bool raises VerdictError rather than projecting False. The predicate is signed and written to a public transparency log, so a missing safety flag is unprojectable, not reassuring. isinstance(x, bool) is deliberate -- isinstance(True, int) is True, so a truthiness check would admit 0 and 1. RuntimeError joins the resolution guard, matching the convention at evidence.py:192,295 and verifier.py:76-77. It is tested by monkeypatching the binding loop.verdict actually calls, not through a filesystem symlink loop: resolve_loop_paths resolves non-strictly and never raises for a loop, and strict resolution yields OSError on this interpreter. Authored via the Claudex lane (gpt-5.6-terra, sessions 019f9ea1 / 019f9eaa / 019f9f7c / 019f9f87). Verified outside the worker: scope clean, suite 1320 passed / 18 skipped against a 1314 baseline, and five mutation probes killed on a green baseline -- stubbed schema loader, nulled chain head, emptied issue codes, reverted fail-open false_completion, and RuntimeError dropped from the except tuple. Independent claude-sonnet-5 review PASS on all eight criteria. Receipt cx_s4a_t2b_verdict_correction_a2. --- loop/verdict.py | 80 ++++++++++++++++++ scripts/test_verdict.py | 181 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 261 insertions(+) diff --git a/loop/verdict.py b/loop/verdict.py index b75bfa9..f5489e2 100644 --- a/loop/verdict.py +++ b/loop/verdict.py @@ -9,9 +9,13 @@ from __future__ import annotations import json +from importlib import metadata +from pathlib import Path from typing import Any from ._resources import schemas_dir +from .contract import doctor_report +from .paths import LoopPaths, resolve_loop_paths VERDICT_SCHEMA_ID = "loop-engineer/verdict@1" PREDICATE_TYPE = "urn:loop-engineer:verdict:1" @@ -23,3 +27,79 @@ class VerdictError(ValueError): def _load_verdict_schema() -> dict[str, Any]: return json.loads((schemas_dir() / "verdict.schema.json").read_text(encoding="utf-8")) + + +def _tool_version() -> str | None: + try: + return metadata.version("loop-engineer") + except metadata.PackageNotFoundError: + return None + + +def _terminal_record(paths: LoopPaths) -> dict[str, Any]: + path = paths.loop_dir / "terminal_state.json" + if not path.is_file(): + raise VerdictError( + "no terminal record: a verdict projects a finished run " + f"({path.name} is absent)" + ) + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise VerdictError(f"terminal record is unreadable: {exc}") from exc + if not isinstance(data, dict): + raise VerdictError("terminal record is not an object") + if "false_completion" not in data: + raise VerdictError("terminal record is missing required false_completion") + if not isinstance(data["false_completion"], bool): + raise VerdictError("terminal record false_completion must be a boolean") + return data + + +def build_verdict(target: str | Path, *, mode: str | None = None) -> dict[str, Any]: + """Project local run state into a ``verdict@1`` predicate body. + + Pure over the workspace: no environment, network, signing, or verification. + """ + try: + paths = resolve_loop_paths(target) + report = doctor_report(paths.workspace, mode=mode) + except (OSError, ValueError, RuntimeError) as exc: + # RuntimeError is pathlib's symlink-loop signal on Python <= 3.12. + raise VerdictError(f"cannot resolve a loop workspace at {target}: {exc}") from exc + + terminal = _terminal_record(paths) + store = report.get("event_store") or {} + chain = store.get("chain") or {} + head = chain.get("head") or {} + policy = terminal.get("completion_policy") + policy_mode = policy.get("mode") if isinstance(policy, dict) else None + + return { + "schema": VERDICT_SCHEMA_ID, + "run_id": str(store.get("run_id") or paths.workspace.name), + "tool": {"name": "loop-engineer", "version": _tool_version()}, + "doctor": { + "ok": bool(report.get("ok")), + "validation_mode": str(report.get("validation_mode") or "unknown"), + "issue_codes": sorted({ + str(issue.get("code")) + for issue in report.get("issues", []) + if isinstance(issue, dict) and issue.get("code") + }), + "schemas_checked": sorted( + str(schema) for schema in report.get("schemas_checked", []) + ), + }, + "chain": { + "head": head.get("event_hash"), + "sequence": head.get("sequence"), + "unchained_prefix": int(chain.get("unchained_prefix") or 0), + }, + "terminal": { + "state": terminal.get("state"), + "completion_policy": policy_mode, + "false_completion": terminal["false_completion"], + }, + "evidence": [], + } diff --git a/scripts/test_verdict.py b/scripts/test_verdict.py index 61cb376..0dc1243 100644 --- a/scripts/test_verdict.py +++ b/scripts/test_verdict.py @@ -1,4 +1,27 @@ import json +import re + +import pytest + +from loop.events import SQLiteEventStore +from loop.scaffold import scaffold + + +def _workspace_with_terminal(tmp_path, name="workspace", *, completion_policy=None): + target = tmp_path / name + scaffold(target) + (target / ".loop" / "terminal_state.json").write_text( + json.dumps({ + "schema": "loop-engineer/terminal@1", + "state": "Succeeded", + "criteria_met": {"gate": True}, + "evidence": [], + "false_completion": False, + "completion_policy": {"mode": "all_required"} if completion_policy is None else completion_policy, + }), + encoding="utf-8", + ) + return target def test_schema_id_and_predicate_type_are_pinned(): @@ -24,3 +47,161 @@ def test_verdict_schema_is_not_a_contract_artifact(): from loop.verdict import VERDICT_SCHEMA_ID assert VERDICT_SCHEMA_ID not in SCHEMA_IDS + + +def test_build_verdict_has_the_required_top_level_shape(tmp_path): + from loop.verdict import build_verdict + + verdict = build_verdict(_workspace_with_terminal(tmp_path)) + + assert set(verdict) == {"schema", "run_id", "tool", "doctor", "chain", "terminal", "evidence"} + assert verdict["schema"] == "loop-engineer/verdict@1" + assert verdict["tool"]["name"] == "loop-engineer" + assert verdict["evidence"] == [] + + +def test_build_verdict_projects_nonempty_normalized_doctor_issue_codes(tmp_path): + from loop.verdict import build_verdict + + target = _workspace_with_terminal(tmp_path) + (target / "scripts" / "verify-fast").unlink() + (target / "scripts" / "verify-full").unlink() + + doctor = build_verdict(target)["doctor"] + + assert set(doctor) == {"ok", "validation_mode", "issue_codes", "schemas_checked"} + assert doctor["issue_codes"] + assert doctor["issue_codes"] == sorted(set(doctor["issue_codes"])) + assert "unresolved_task_verify" in doctor["issue_codes"] + assert all(" " not in code and "/" not in code for code in doctor["issue_codes"]) + + +def test_build_verdict_projects_terminal_and_requires_terminal_record(tmp_path): + from loop.verdict import VerdictError, build_verdict + + target = _workspace_with_terminal(tmp_path) + terminal = build_verdict(target)["terminal"] + + assert set(terminal) == {"state", "completion_policy", "false_completion"} + assert terminal["completion_policy"] == "all_required" + + no_policy = _workspace_with_terminal(tmp_path, "no-policy", completion_policy=None) + (no_policy / ".loop" / "terminal_state.json").write_text( + json.dumps({"state": "Succeeded", "false_completion": False}), encoding="utf-8" + ) + assert build_verdict(no_policy)["terminal"]["completion_policy"] is None + + non_object_policy = _workspace_with_terminal(tmp_path, "non-object-policy", completion_policy="all_required") + assert build_verdict(non_object_policy)["terminal"]["completion_policy"] is None + + plain = tmp_path / "plain" + scaffold(plain) + with pytest.raises(VerdictError, match="no terminal record"): + build_verdict(plain) + with pytest.raises(VerdictError): + build_verdict(tmp_path / "missing") + + +def test_build_verdict_projects_chain_head_from_real_store(tmp_path): + from loop.verdict import build_verdict + + target = _workspace_with_terminal(tmp_path) + head = SQLiteEventStore(target / ".loop" / "events.db").append( + "run-1", "contract_opened", {"workspace": target.name}, actor="test" + ) + + chain = build_verdict(target)["chain"] + + assert chain["head"] is not None + assert re.fullmatch(r"[0-9a-f]{64}", chain["head"]) + assert chain["sequence"] == head["sequence"] + + +def test_build_verdict_handles_an_absent_event_store(tmp_path): + from loop.verdict import build_verdict + + verdict = build_verdict(_workspace_with_terminal(tmp_path)) + + assert verdict["chain"] == {"head": None, "sequence": None, "unchained_prefix": 0} + + +def test_build_verdict_degrades_for_an_unreadable_event_store(tmp_path): + from loop.verdict import build_verdict + + target = _workspace_with_terminal(tmp_path) + store_path = target / ".loop" / "events.db" + SQLiteEventStore(store_path).append( + "run-1", "contract_opened", {"workspace": target.name}, actor="test" + ) + store_path.write_bytes(b"not a SQLite database") + + verdict = build_verdict(target) + + assert verdict["chain"] == {"head": None, "sequence": None, "unchained_prefix": 0} + assert verdict["doctor"]["issue_codes"] + + +def test_build_verdict_rejects_terminal_without_false_completion(tmp_path): + from loop.verdict import VerdictError, build_verdict + + target = _workspace_with_terminal(tmp_path) + terminal_path = target / ".loop" / "terminal_state.json" + terminal = json.loads(terminal_path.read_text(encoding="utf-8")) + del terminal["false_completion"] + terminal_path.write_text(json.dumps(terminal), encoding="utf-8") + + with pytest.raises(VerdictError, match="false_completion"): + build_verdict(target) + + +@pytest.mark.parametrize("value", ["false", 1]) +def test_build_verdict_rejects_non_boolean_false_completion(tmp_path, value): + from loop.verdict import VerdictError, build_verdict + + target = _workspace_with_terminal(tmp_path) + terminal_path = target / ".loop" / "terminal_state.json" + terminal = json.loads(terminal_path.read_text(encoding="utf-8")) + terminal["false_completion"] = value + terminal_path.write_text(json.dumps(terminal), encoding="utf-8") + + with pytest.raises(VerdictError, match="false_completion"): + build_verdict(target) + + +def test_build_verdict_rejects_non_object_terminal_record(tmp_path): + from loop.verdict import VerdictError, build_verdict + + target = _workspace_with_terminal(tmp_path) + (target / ".loop" / "terminal_state.json").write_text("[]", encoding="utf-8") + + with pytest.raises(VerdictError, match="not an object"): + build_verdict(target) + + +def test_build_verdict_wraps_path_resolution_runtime_errors_as_verdict_error( + tmp_path, monkeypatch +): + import loop.verdict as verdict + from loop.verdict import VerdictError, build_verdict + + def raise_symlink_loop(_target): + raise RuntimeError("Symlink loop from 'x'") + + monkeypatch.setattr(verdict, "resolve_loop_paths", raise_symlink_loop) + + with pytest.raises(VerdictError): + build_verdict(tmp_path / "anything") + + +def test_build_verdict_validates_against_its_loaded_schema(tmp_path): + jsonschema = pytest.importorskip("jsonschema") + from loop.verdict import _load_verdict_schema, build_verdict + + schema = _load_verdict_schema() + verdict = build_verdict(_workspace_with_terminal(tmp_path)) + invalid_verdict = {**verdict, "chain": {**verdict["chain"], "head": "not-a-hash"}} + + assert schema["$id"] == "loop-engineer/verdict@1" + jsonschema.validate(verdict, schema) + with pytest.raises(jsonschema.ValidationError): + jsonschema.validate(invalid_verdict, schema) From 924ce03f21c436075d8730326b4cbcd8bb779629 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Sun, 26 Jul 2026 14:21:13 -0400 Subject: [PATCH 04/14] fix(verdict): precise read-failure frame, and assert the two identity invariants Closes the three low follow-ups left open by Task 2. Split the resolution guard from the doctor_report guard. ValidationModeError subclasses RuntimeError, so a single guard reported an invalid mode= as 'cannot resolve a loop workspace' -- a misleading frame for an argument error. Both paths still raise VerdictError, so the fail-closed posture is unchanged; only the diagnostic improves. Add None to the non-boolean false_completion cases. JSON null round-trips to None, so the key is present and only the isinstance check stands between it and a signed false claim -- this is the exact value the original bool(None) is False defect turned on, and it was proven by inspection but untested. Assert PREDICATE_TYPE is derived from VERDICT_SCHEMA_ID. ADR 0002 chose a URN matching the schema $id; a URN cannot idiomatically carry '/' or '@', so the mapping is a transliteration and nothing stopped the two from drifting apart. Governor-authored and gated mechanically, not lane-delegated: full suite 1323 passed / 18 skipped against a 1320 baseline, and three mutation probes killed -- re-merging the guards fails the invalid-mode test, drifting PREDICATE_TYPE fails the derivation test, and reverting to fail-open fails all three parametrized cases including None. --- loop/verdict.py | 8 +++++++- scripts/test_verdict.py | 33 ++++++++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/loop/verdict.py b/loop/verdict.py index f5489e2..7c6673d 100644 --- a/loop/verdict.py +++ b/loop/verdict.py @@ -63,11 +63,17 @@ def build_verdict(target: str | Path, *, mode: str | None = None) -> dict[str, A """ try: paths = resolve_loop_paths(target) - report = doctor_report(paths.workspace, mode=mode) except (OSError, ValueError, RuntimeError) as exc: # RuntimeError is pathlib's symlink-loop signal on Python <= 3.12. raise VerdictError(f"cannot resolve a loop workspace at {target}: {exc}") from exc + # Separate from resolution so an invalid mode= is not reported as a path failure; + # ValidationModeError is a RuntimeError subclass and would otherwise land above. + try: + report = doctor_report(paths.workspace, mode=mode) + except (OSError, ValueError, RuntimeError) as exc: + raise VerdictError(f"cannot read the contract at {paths.workspace}: {exc}") from exc + terminal = _terminal_record(paths) store = report.get("event_store") or {} chain = store.get("chain") or {} diff --git a/scripts/test_verdict.py b/scripts/test_verdict.py index 0dc1243..9f8fa23 100644 --- a/scripts/test_verdict.py +++ b/scripts/test_verdict.py @@ -31,6 +31,18 @@ def test_schema_id_and_predicate_type_are_pinned(): assert PREDICATE_TYPE == "urn:loop-engineer:verdict:1" +def test_predicate_type_is_derived_from_the_schema_id(): + """The two constants are one identity in two encodings, not two names. + + ADR 0002 chose a URN matching the schema $id. A URN cannot idiomatically + carry '/' or '@', so the mapping is a transliteration -- which means + nothing stops the two from silently drifting apart unless it is asserted. + """ + from loop.verdict import PREDICATE_TYPE, VERDICT_SCHEMA_ID + + assert PREDICATE_TYPE == "urn:" + VERDICT_SCHEMA_ID.replace("/", ":").replace("@", ":") + + def test_schema_file_declares_the_matching_id(): from loop._resources import schemas_dir from loop.verdict import VERDICT_SCHEMA_ID @@ -154,7 +166,11 @@ def test_build_verdict_rejects_terminal_without_false_completion(tmp_path): build_verdict(target) -@pytest.mark.parametrize("value", ["false", 1]) +# None is the original defect value: bool(None) is False, so the fail-open +# projection this guard replaced would have claimed "not a false completion" +# for a null flag. JSON null round-trips to None, so the key IS present and +# only the isinstance check stands between it and a signed false claim. +@pytest.mark.parametrize("value", ["false", 1, None]) def test_build_verdict_rejects_non_boolean_false_completion(tmp_path, value): from loop.verdict import VerdictError, build_verdict @@ -168,6 +184,21 @@ def test_build_verdict_rejects_non_boolean_false_completion(tmp_path, value): build_verdict(target) +def test_build_verdict_reports_an_invalid_mode_as_a_contract_read_failure(tmp_path): + """An invalid mode= is a contract-read failure, not a path-resolution one. + + ValidationModeError subclasses RuntimeError, so a single guard around both + resolution and doctor_report would label it "cannot resolve a loop + workspace" -- a misleading frame for an argument error. + """ + from loop.verdict import VerdictError, build_verdict + + target = _workspace_with_terminal(tmp_path) + + with pytest.raises(VerdictError, match="cannot read the contract"): + build_verdict(target, mode="not-a-validation-mode") + + def test_build_verdict_rejects_non_object_terminal_record(tmp_path): from loop.verdict import VerdictError, build_verdict From b93d5e82258f02c76a843b36d787dd7a316833f5 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 07:57:02 -0400 Subject: [PATCH 05/14] feat(verdict): carry only chain-bound evidence that passes the strict bar The digest projected for each evidence entry is the sha256 of the evidence record FILE BYTES - the value the event chain committed - never the record's sha256 field, which hashes the cited artifact instead (the plan sketch lifted a 'digest' field evidence@1 does not have). Only entries that clear loop.contract._strict_evidence_failure project; an unreadable event store projects an empty list fail-closed rather than laundering into the store-less degradation; output is de-duplicated and canonically sorted. Claudex lane s4a-t3-evidence-digests, gpt-5.6-terra/medium attempt 1 accepted: scope 2/2, targeted 26 exact, extras 1331/18 exact (+8), pyyaml-only 1236/113 (+8), purity clean, mutation probes M1-M5 all KILLED, fresh claude-sonnet-5 review PASS (0 blockers). Receipt cx_s4a_t3_evidence_digests_a1. Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- loop/verdict.py | 61 +++++++++++++- scripts/test_verdict.py | 183 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 240 insertions(+), 4 deletions(-) diff --git a/loop/verdict.py b/loop/verdict.py index 7c6673d..6dc71e5 100644 --- a/loop/verdict.py +++ b/loop/verdict.py @@ -2,20 +2,22 @@ This module builds a document. It NEVER signs one, never verifies a signature, never constructs an in-toto Statement, and never reads an environment variable. -The signer lane (action.yml -> actions/attest) owns the envelope, the subject, -and every cryptographic operation. See docs/adr/0002-ci-attested-verdict.md. +The signer lane (action.yml -> actions/attest) owns the envelope claims and every +cryptographic operation. See docs/adr/0002-ci-attested-verdict.md. """ from __future__ import annotations +import hashlib import json from importlib import metadata from pathlib import Path from typing import Any from ._resources import schemas_dir -from .contract import doctor_report +from .contract import _strict_evidence_failure, doctor_report from .paths import LoopPaths, resolve_loop_paths +from .runtime import RuntimeStoreError, bound_artifact_digests VERDICT_SCHEMA_ID = "loop-engineer/verdict@1" PREDICATE_TYPE = "urn:loop-engineer:verdict:1" @@ -56,6 +58,51 @@ def _terminal_record(paths: LoopPaths) -> dict[str, Any]: return data +def _evidence_digests(entry: object, paths: LoopPaths) -> dict[str, str | None] | None: + """Return the chain-committed record digest and verifier digests for an entry.""" + if not isinstance(entry, str): + return None + try: + record_bytes = (paths.workspace / entry).read_bytes() + record = json.loads(record_bytes.decode("utf-8")) + except (OSError, ValueError, UnicodeDecodeError, json.JSONDecodeError): + return None + if not isinstance(record, dict): + return None + verified_by = record.get("verified_by") + return { + "digest": hashlib.sha256(record_bytes).hexdigest(), + "code_digest": verified_by.get("code_digest") if isinstance(verified_by, dict) else None, + "policy_digest": verified_by.get("policy_digest") if isinstance(verified_by, dict) else None, + } + + +def _bound_evidence(paths: LoopPaths) -> dict[str, tuple[str, ...]] | None: + """Read evidence record digests committed by the event chain.""" + return bound_artifact_digests(paths.workspace) + + +def _verified_evidence( + terminal: dict[str, Any], paths: LoopPaths, bound: dict[str, tuple[str, ...]] | None +) -> list[dict[str, str | None]]: + """Project only terminal evidence that clears the shared strict bar.""" + entries = terminal.get("evidence") + if not isinstance(entries, list): + return [] + projected = { + (digest["digest"], digest["code_digest"], digest["policy_digest"]) + for entry in entries + if _strict_evidence_failure(entry, paths, bound) is None + if (digest := _evidence_digests(entry, paths)) is not None + } + return [ + {"digest": digest, "code_digest": code_digest, "policy_digest": policy_digest} + for digest, code_digest, policy_digest in sorted( + projected, key=lambda item: (item[0], item[1] or "", item[2] or "") + ) + ] + + def build_verdict(target: str | Path, *, mode: str | None = None) -> dict[str, Any]: """Project local run state into a ``verdict@1`` predicate body. @@ -75,6 +122,12 @@ def build_verdict(target: str | Path, *, mode: str | None = None) -> dict[str, A raise VerdictError(f"cannot read the contract at {paths.workspace}: {exc}") from exc terminal = _terminal_record(paths) + try: + bound = _bound_evidence(paths) + except RuntimeStoreError: + evidence = [] + else: + evidence = _verified_evidence(terminal, paths, bound) store = report.get("event_store") or {} chain = store.get("chain") or {} head = chain.get("head") or {} @@ -107,5 +160,5 @@ def build_verdict(target: str | Path, *, mode: str | None = None) -> dict[str, A "completion_policy": policy_mode, "false_completion": terminal["false_completion"], }, - "evidence": [], + "evidence": evidence, } diff --git a/scripts/test_verdict.py b/scripts/test_verdict.py index 9f8fa23..6a76947 100644 --- a/scripts/test_verdict.py +++ b/scripts/test_verdict.py @@ -1,9 +1,13 @@ +import hashlib import json import re import pytest +from loop import emit +from loop.completion import VERIFIED_EVIDENCE_MODE from loop.events import SQLiteEventStore +from loop.runner import dispatch_once from loop.scaffold import scaffold @@ -24,6 +28,65 @@ def _workspace_with_terminal(tmp_path, name="workspace", *, completion_policy=No return target +def _ready(tmp_path): + """A real task workspace positioned for its first dispatched iteration.""" + workspace = tmp_path / "dispatched" + emit.open_contract(workspace) + task = { + "id": "T-1", "title": "T-1", "status": "pending", "criterion_ref": "T-1", + "verify": "./scripts/verify-fast.sh", "depends_on": [], "attempts": 0, + "evidence": None, + } + (workspace / "TASKS.json").write_text( + json.dumps({"schema": "loop-engineer/tasks@1", "tasks": [task]}), encoding="utf-8" + ) + verifier = workspace / "scripts" / "verify-fast.sh" + verifier.parent.mkdir(parents=True, exist_ok=True) + verifier.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8") + verifier.chmod(0o755) + store = SQLiteEventStore(workspace / ".loop" / "events.db") + store.append("run-1", "contract_opened", {"workspace": workspace.name}, actor="test") + for state in ("plan", "critique-plan", "queue-tasks", "execute-task"): + store.append("run-1", "iteration_appended", { + "iteration_id": 0, "outcome": "replanned", "state": state, + }, actor="test") + return workspace + + +def _dispatched_workspace(tmp_path): + workspace = _ready(tmp_path) + dispatch_once(workspace) + return workspace + + +def _handwritten_record(workspace, *, name="evidence-handwritten.json"): + """Create self-consistent evidence that is only chain-bound in storeless workspaces.""" + bundle = workspace / ".loop" / "artifacts" / "verify-handwritten.json" + bundle.parent.mkdir(parents=True, exist_ok=True) + bundle_bytes = b'{"outcome": "PASS", "passed": true}' + bundle.write_bytes(bundle_bytes) + record = { + "schema": "loop-engineer/evidence@1", "id": "hand:1:verify", + "kind": "verify-bundle", "uri": ".loop/artifacts/verify-handwritten.json", + "sha256": hashlib.sha256(bundle_bytes).hexdigest(), "media_type": "application/json", + "produced_by": {"run_id": "run-1", "task_id": None, "attempt": 1, + "executor": "worker-a"}, + "created_at": "2026-07-25T00:00:00+00:00", + "verified_by": {"by": "ci", "at": "2026-07-25T00:00:00+00:00"}, + } + path = workspace / ".loop" / "evidence" / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(record, sort_keys=True) + "\n", encoding="utf-8") + return f".loop/evidence/{name}" + + +def _rewrite_terminal_evidence(workspace, evidence): + path = workspace / ".loop" / "terminal_state.json" + terminal = json.loads(path.read_text(encoding="utf-8")) + terminal["evidence"] = evidence + path.write_text(json.dumps(terminal, sort_keys=True) + "\n", encoding="utf-8") + + def test_schema_id_and_predicate_type_are_pinned(): from loop.verdict import PREDICATE_TYPE, VERDICT_SCHEMA_ID @@ -236,3 +299,123 @@ def test_build_verdict_validates_against_its_loaded_schema(tmp_path): jsonschema.validate(verdict, schema) with pytest.raises(jsonschema.ValidationError): jsonschema.validate(invalid_verdict, schema) + + +def test_evidence_carries_only_entries_that_pass_the_strict_bar(tmp_path, monkeypatch): + import loop.verdict as verdict + + monkeypatch.setattr(verdict, "_strict_evidence_failure", + lambda entry, _paths, _bound: None if entry == "good" else "bad") + monkeypatch.setattr(verdict, "_evidence_digests", lambda entry, _paths: { + "digest": "a" * 64, "code_digest": None, "policy_digest": None, + } if entry == "good" else {"digest": "b" * 64, "code_digest": None, + "policy_digest": None}) + target = _workspace_with_terminal(tmp_path) + _rewrite_terminal_evidence(target, ["good", "bad"]) + + assert verdict.build_verdict(target)["evidence"] == [ + {"digest": "a" * 64, "code_digest": None, "policy_digest": None} + ] + + +def test_evidence_entries_carry_digests_only(tmp_path): + from loop.verdict import build_verdict + + workspace = _dispatched_workspace(tmp_path) + emit.terminate(workspace, state="Succeeded", criteria_met={"C-1": True}, + evidence=[".loop/evidence/evidence-iter1.json"], + completion_policy=VERIFIED_EVIDENCE_MODE) + + evidence = build_verdict(workspace)["evidence"] + assert evidence + assert all(set(entry) == {"digest", "code_digest", "policy_digest"} for entry in evidence) + + +def test_evidence_is_sorted_by_digest(tmp_path, monkeypatch): + import loop.verdict as verdict + + digests = { + "later": {"digest": "f" * 64, "code_digest": None, "policy_digest": None}, + "first": {"digest": "0" * 64, "code_digest": None, "policy_digest": None}, + } + monkeypatch.setattr(verdict, "_strict_evidence_failure", lambda *_args: None) + monkeypatch.setattr(verdict, "_evidence_digests", lambda entry, _paths: digests[entry]) + target = _workspace_with_terminal(tmp_path) + _rewrite_terminal_evidence(target, ["later", "first"]) + + assert [entry["digest"] for entry in verdict.build_verdict(target)["evidence"]] == [ + "0" * 64, "f" * 64, + ] + + +def test_evidence_digest_is_the_chain_committed_record_digest(tmp_path): + from loop.runtime import bound_artifact_digests + from loop.verdict import build_verdict + + workspace = _dispatched_workspace(tmp_path) + entry = ".loop/evidence/evidence-iter1.json" + emit.terminate(workspace, state="Succeeded", criteria_met={"C-1": True}, evidence=[entry], + completion_policy=VERIFIED_EVIDENCE_MODE) + record_path = workspace / entry + record = json.loads(record_path.read_text(encoding="utf-8")) + + projected = build_verdict(workspace)["evidence"] + expected = hashlib.sha256(record_path.read_bytes()).hexdigest() + assert projected == [{"digest": expected, + "code_digest": record["verified_by"]["code_digest"], + "policy_digest": record["verified_by"]["policy_digest"]}] + assert expected == bound_artifact_digests(workspace)[entry][0] + assert expected != record["sha256"] + + +def test_unbound_record_is_excluded_when_a_store_exists(tmp_path): + from loop.verdict import build_verdict + + workspace = _dispatched_workspace(tmp_path) + bound_entry = ".loop/evidence/evidence-iter1.json" + emit.terminate(workspace, state="Succeeded", criteria_met={"C-1": True}, + evidence=[bound_entry], completion_policy=VERIFIED_EVIDENCE_MODE) + unbound_entry = _handwritten_record(workspace) + _rewrite_terminal_evidence(workspace, [bound_entry, unbound_entry]) + + evidence = build_verdict(workspace)["evidence"] + assert [entry["digest"] for entry in evidence] == [ + hashlib.sha256((workspace / bound_entry).read_bytes()).hexdigest() + ] + + +def test_absent_store_projects_evidence_under_the_documented_degradation(tmp_path): + from loop.verdict import build_verdict + + workspace = _workspace_with_terminal(tmp_path) + entry = _handwritten_record(workspace) + _rewrite_terminal_evidence(workspace, [entry]) + + assert build_verdict(workspace)["evidence"] == [{ + "digest": hashlib.sha256((workspace / entry).read_bytes()).hexdigest(), + "code_digest": None, "policy_digest": None, + }] + + +def test_unreadable_store_projects_no_evidence_and_does_not_raise(tmp_path): + from loop.verdict import build_verdict + + workspace = _workspace_with_terminal(tmp_path) + entry = _handwritten_record(workspace) + _rewrite_terminal_evidence(workspace, [entry]) + (workspace / ".loop" / "events.db").write_bytes(b"not a SQLite database") + + assert build_verdict(workspace)["evidence"] == [] + + +def test_identical_evidence_entries_project_once(tmp_path): + from loop.verdict import build_verdict + + workspace = _workspace_with_terminal(tmp_path) + entry = _handwritten_record(workspace) + _rewrite_terminal_evidence(workspace, [entry, entry]) + + assert build_verdict(workspace)["evidence"] == [{ + "digest": hashlib.sha256((workspace / entry).read_bytes()).hexdigest(), + "code_digest": None, "policy_digest": None, + }] From 17fcb54fce7b1c391a040f8077c6af8ba7c1db5b Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 08:00:34 -0400 Subject: [PATCH 06/14] fix(verdict): re-verify the projection read against the chain-committed digest Closes the two substantive review minors from the T3 acceptance. The strict bar validates its own read of an evidence record; _evidence_digests then read the file a second time, leaving a window where the projected digest could diverge from the digest the chain committed. The projection now drops any entry whose second-read digest is not exactly the chain's one-element binding (store-less workspaces keep the documented degradation). Also adds the populated-evidence schema validation test the suite lacked (importorskip'd so the structural-fallback leg stays honest). Governor-authored, not lane-dispatched: the review itself prescribed the hardening and the change is eleven lines. Held to the follow-up evidence standard: targeted 28, extras 1333/18 (+2), pyyaml-only 1237/114 (+1 pass +1 skip), probes M6 (check removed -> TOCTOU test fails) and M7 (extra key leaked -> shape + schema tests fail) both KILLED, tree restored intact. Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- loop/verdict.py | 19 +++++++++++++------ scripts/test_verdict.py | 31 +++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 6 deletions(-) diff --git a/loop/verdict.py b/loop/verdict.py index 6dc71e5..c5cb62c 100644 --- a/loop/verdict.py +++ b/loop/verdict.py @@ -89,12 +89,19 @@ def _verified_evidence( entries = terminal.get("evidence") if not isinstance(entries, list): return [] - projected = { - (digest["digest"], digest["code_digest"], digest["policy_digest"]) - for entry in entries - if _strict_evidence_failure(entry, paths, bound) is None - if (digest := _evidence_digests(entry, paths)) is not None - } + projected = set() + for entry in entries: + if _strict_evidence_failure(entry, paths, bound) is not None: + continue + digests = _evidence_digests(entry, paths) + if digests is None: + continue + if bound is not None and (digests["digest"],) != bound.get(entry): + # The bar validated its own read of the record; this projection read + # hashed differently, so the bytes moved between the two reads. A + # digest the chain never committed must not enter a signed document. + continue + projected.add((digests["digest"], digests["code_digest"], digests["policy_digest"])) return [ {"digest": digest, "code_digest": code_digest, "policy_digest": policy_digest} for digest, code_digest, policy_digest in sorted( diff --git a/scripts/test_verdict.py b/scripts/test_verdict.py index 6a76947..e3aff5e 100644 --- a/scripts/test_verdict.py +++ b/scripts/test_verdict.py @@ -419,3 +419,34 @@ def test_identical_evidence_entries_project_once(tmp_path): "digest": hashlib.sha256((workspace / entry).read_bytes()).hexdigest(), "code_digest": None, "policy_digest": None, }] + + +def test_projection_digest_must_match_the_chain_committed_digest(tmp_path, monkeypatch): + """The bar validates its own read of the record; the projection read must + re-match the chain-committed digest, or the entry is dropped (TOCTOU).""" + import loop.verdict as verdict + + workspace = _dispatched_workspace(tmp_path) + entry = ".loop/evidence/evidence-iter1.json" + emit.terminate(workspace, state="Succeeded", criteria_met={"C-1": True}, evidence=[entry], + completion_policy=VERIFIED_EVIDENCE_MODE) + monkeypatch.setattr(verdict, "_evidence_digests", lambda _entry, _paths: { + "digest": "f" * 64, "code_digest": None, "policy_digest": None, + }) + + assert verdict.build_verdict(workspace)["evidence"] == [] + + +def test_populated_evidence_validates_against_the_schema(tmp_path): + """A verdict carrying a non-empty evidence list satisfies verdict.schema.json.""" + jsonschema = pytest.importorskip("jsonschema") + from loop.verdict import _load_verdict_schema, build_verdict + + workspace = _dispatched_workspace(tmp_path) + emit.terminate(workspace, state="Succeeded", criteria_met={"C-1": True}, + evidence=[".loop/evidence/evidence-iter1.json"], + completion_policy=VERIFIED_EVIDENCE_MODE) + + verdict = build_verdict(workspace) + assert verdict["evidence"] + jsonschema.validate(verdict, _load_verdict_schema()) From 9e1d5be92ae03c952fe465c3cf609b108b1decd9 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 08:47:26 -0400 Subject: [PATCH 07/14] feat(cli): add the loop verdict verb python3 -m loop verdict [--mode basic|strict|release] prints canonical_json(build_verdict(...)) to stdout and exits 0; every projection failure - VerdictError AND canonical_json's ChainHashError - degrades to one typed 'verdict: ...' stderr line and exit 2. Registered at all five CLI surfaces (_COMMANDS, _READ_COMMANDS, _USAGE, the --mode extraction set, _HELP with the signer-boundary description and content summary). Claudex lane s4a-t4-verdict-cli, gpt-5.6-terra, attempt 2 accepted after one verifier-directed repair: attempt 1 passed every deterministic gate but the fresh review caught the dispatch catching only VerdictError while a NaN-bearing terminal record (json.loads accepts NaN) crashes canonical_json with a raw ChainHashError traceback - governor-reproduced, then repaired with a regression test plus a valid --mode success-path test proving flag threading. Gates: targeted 11 exact, extras 1344/18 exact, pyyaml-only 1248/114, probes M1-M6 all KILLED. Receipts cx_s4a_t4_verdict_cli_a1 (repair_requested) / _a2 (accepted). Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- loop/__main__.py | 25 +++++-- scripts/test_verdict_cli.py | 129 ++++++++++++++++++++++++++++++++++++ 2 files changed, 149 insertions(+), 5 deletions(-) create mode 100644 scripts/test_verdict_cli.py diff --git a/loop/__main__.py b/loop/__main__.py index b0b3dce..2f5e1a8 100644 --- a/loop/__main__.py +++ b/loop/__main__.py @@ -12,13 +12,13 @@ _PROG = "python3 -m loop" -_COMMANDS = ("scaffold", "doctor", "validate", "verify", "inspect", "metrics", "plan-lint", "status", "replay", "simulate", "run", "approve", "pause", "resume", "cancel", "migrate", "architect") +_COMMANDS = ("scaffold", "doctor", "validate", "verify", "verdict", "inspect", "metrics", "plan-lint", "status", "replay", "simulate", "run", "approve", "pause", "resume", "cancel", "migrate", "architect") # Read commands operate on an EXISTING contract dir; scaffold CREATES one, so it # is exempt from the "target must exist" guard. -_READ_COMMANDS = ("doctor", "validate", "verify", "inspect", "metrics", "plan-lint", "status", "replay", "simulate", "run", "approve", "pause", "resume", "cancel", "migrate") +_READ_COMMANDS = ("doctor", "validate", "verify", "verdict", "inspect", "metrics", "plan-lint", "status", "replay", "simulate", "run", "approve", "pause", "resume", "cancel", "migrate") -_USAGE = f"usage: {_PROG} " +_USAGE = f"usage: {_PROG} " _HELP = f"""{_PROG} — validate, inspect, and measure a portable repo-OS loop contract. @@ -26,6 +26,7 @@ {_PROG} metrics [--baseline] {_PROG} doctor|validate|verify [--mode basic|strict|release] [--expect-chain-head SHA256] + {_PROG} verdict [--mode basic|strict|release] {_PROG} status [--mode basic|strict|release] {_PROG} replay [--mode basic|strict|release] {_PROG} simulate [--mode basic|strict|release] @@ -42,6 +43,9 @@ doctor Validate the contract objects; --mode selects validation strength. validate Alias for doctor. verify Alias for doctor — check the contract's state. + verdict Emit the predicate body only; the signer (actions/attest) constructs + the in-toto Statement. Never signs and never verifies a signature. + (schema loop-engineer/verdict@1: doctor status, chain head, terminal outcome, verified-evidence digests) inspect Score an existing loop against the prime-directive checklist (emits a weak/strong verdict and a gap report). metrics Derive false-completion-rate + repair-productivity from the loop's @@ -70,7 +74,7 @@ options: --mode {{basic,strict,release}} - (doctor/validate/verify/plan-lint/status/replay/simulate/run) basic forces structural + (doctor/validate/verify/verdict/plan-lint/status/replay/simulate/run) basic forces structural checks; strict/release require jsonschema. Default: auto-detect. --expect-chain-head SHA256 (doctor/validate/verify) fail unless the event store's chain head @@ -241,7 +245,7 @@ def main(argv: list[str] | None = None) -> int: return 2 mode = None - if command in {"doctor", "validate", "verify", "plan-lint", "status", "replay", "simulate", "run", "approve", "pause", "resume", "cancel"}: + if command in {"doctor", "validate", "verify", "verdict", "plan-lint", "status", "replay", "simulate", "run", "approve", "pause", "resume", "cancel"}: try: mode, argv = _extract_mode_flag(argv) except ValueError as exc: @@ -367,6 +371,17 @@ def main(argv: list[str] | None = None) -> int: print(f"{command}: {exc}", file=sys.stderr) return 2 + if command == "verdict": + from .verdict import VerdictError, build_verdict + from .chain import ChainHashError, canonical_json + + try: + print(canonical_json(build_verdict(target, mode=mode))) + return 0 + except (VerdictError, ChainHashError) as exc: + print(f"verdict: {exc}", file=sys.stderr) + return 2 + if command == "plan-lint": try: return _print_json(validate_plan(target, mode=mode)) diff --git a/scripts/test_verdict_cli.py b/scripts/test_verdict_cli.py new file mode 100644 index 0000000..14940ac --- /dev/null +++ b/scripts/test_verdict_cli.py @@ -0,0 +1,129 @@ +"""CLI contract tests for the read-only ``loop verdict`` projection.""" + +import importlib.util +import json +import pathlib +import subprocess +import sys + +import pytest + + +@pytest.fixture +def repo_root() -> pathlib.Path: + return pathlib.Path(__file__).resolve().parent.parent + + +def _run(repo_root: pathlib.Path, *args: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, "-B", "-m", "loop", *args], + capture_output=True, + text=True, + cwd=repo_root, + ) + + +def test_verdict_emits_canonical_json_and_exits_zero(repo_root: pathlib.Path): + result = _run(repo_root, "verdict", "examples/flaky-test-triage") + assert result.returncode == 0, result.stderr + doc = json.loads(result.stdout) + assert doc["schema"] == "loop-engineer/verdict@1" + + +def test_verdict_output_is_byte_stable(repo_root: pathlib.Path): + first = _run(repo_root, "verdict", "examples/flaky-test-triage") + second = _run(repo_root, "verdict", "examples/flaky-test-triage") + assert first.returncode == second.returncode == 0 + assert first.stdout == second.stdout + + +def test_verdict_emits_no_statement_envelope(repo_root: pathlib.Path): + result = _run(repo_root, "verdict", "examples/flaky-test-triage") + assert result.returncode == 0, result.stderr + doc = json.loads(result.stdout) + assert not {"_type", "subject", "predicateType", "predicate"} & doc.keys() + + +def test_verdict_fails_loud_without_a_terminal_record(repo_root: pathlib.Path, tmp_path: pathlib.Path): + workspace = tmp_path / "workspace" + scaffolded = _run(repo_root, "scaffold", str(workspace)) + assert scaffolded.returncode == 0, scaffolded.stderr + result = _run(repo_root, "verdict", str(workspace)) + assert result.returncode == 2 + assert result.stdout == "" + assert "Traceback" not in result.stderr + assert "no terminal record" in result.stderr + assert result.stderr.startswith("verdict:") + + +def test_verdict_reports_a_typed_error_when_the_document_cannot_be_canonicalized( + repo_root: pathlib.Path, tmp_path: pathlib.Path +): + workspace = tmp_path / "workspace" + scaffolded = _run(repo_root, "scaffold", str(workspace)) + assert scaffolded.returncode == 0, scaffolded.stderr + terminal = json.dumps({"state": "Succeeded", "false_completion": False}) + (workspace / ".loop" / "terminal_state.json").write_text( + terminal.replace('"Succeeded"', "NaN"), encoding="utf-8" + ) + result = _run(repo_root, "verdict", str(workspace)) + assert result.returncode == 2 + assert result.stdout == "" + assert "Traceback" not in result.stderr + assert result.stderr.startswith("verdict:") + + +def test_verdict_appears_in_usage_and_help(repo_root: pathlib.Path): + help_result = _run(repo_root, "--help") + assert "verdict" in help_result.stdout + assert "Never signs and never verifies" in help_result.stdout + missing_target = _run(repo_root, "verdict") + assert missing_target.returncode == 2 + assert "verdict" in missing_target.stderr + + +def test_verdict_stdout_is_the_canonical_json_of_build_verdict(repo_root: pathlib.Path): + from loop.chain import canonical_json + from loop.verdict import build_verdict + + result = _run(repo_root, "verdict", "examples/flaky-test-triage") + assert result.returncode == 0, result.stderr + assert result.stdout == canonical_json(build_verdict("examples/flaky-test-triage")) + "\n" + + +def test_verdict_missing_target_exits_2_with_the_read_command_hint(repo_root: pathlib.Path, tmp_path: pathlib.Path): + result = _run(repo_root, "verdict", str(tmp_path / "missing")) + assert result.returncode == 2 + assert "target path does not exist" in result.stderr + + +def test_verdict_rejects_an_invalid_mode_value(repo_root: pathlib.Path): + result = _run(repo_root, "verdict", "--mode", "bogus", "examples/flaky-test-triage") + assert result.returncode == 2 + assert "invalid --mode value" in result.stderr + + +def test_verdict_mode_flag_reaches_the_projection(repo_root: pathlib.Path): + basic = _run(repo_root, "verdict", "--mode", "basic", "examples/flaky-test-triage") + assert basic.returncode == 0, basic.stderr + basic_doc = json.loads(basic.stdout) + assert basic_doc["doctor"]["validation_mode"] == "structural-fallback" + + default = _run(repo_root, "verdict", "examples/flaky-test-triage") + assert default.returncode == 0, default.stderr + default_doc = json.loads(default.stdout) + has_jsonschema = importlib.util.find_spec("jsonschema") is not None + if has_jsonschema: + assert basic_doc["doctor"]["validation_mode"] != default_doc["doctor"]["validation_mode"] + + +def test_verdict_rejects_expect_chain_head(repo_root: pathlib.Path): + result = _run( + repo_root, + "verdict", + "--expect-chain-head", + "a" * 64, + "examples/flaky-test-triage", + ) + assert result.returncode == 2 + assert "only valid for doctor/validate/verify" in result.stderr From 3e26e9154f327290d3963bec470660367289337f Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 08:51:59 -0400 Subject: [PATCH 08/14] test(verdict): pin the typed-error contract for an undecodable terminal file The T4 re-review advisory (R-101) claimed a raw UnicodeDecodeError escapes _terminal_record; independent reproduction REFUTED the reachability - the doctor_report wrapper converts it first (its except tuple carries ValueError, and UnicodeDecodeError is one), so a guard widen there would be dead code and was not kept. What is worth pinning is the invariant itself: build_verdict on an invalid-UTF-8 terminal raises VerdictError at whichever site converts, and the new test holds under projection reorder. Teeth proven: narrowing the doctor wrapper's ValueError makes it fail (M8 KILLED, tree restored). The diagnosis surfaced the REAL pre-existing gap, out of 4a scope: doctor_report itself raises raw UnicodeDecodeError on an undecodable terminal file, so 'loop doctor' tracebacks (exit 1) instead of reporting a typed issue - recorded for an issue at PR time. Test-only change; loop/ byte-identical. Targeted 29, extras 1345/18 (+1), pyyaml-only 1249/114 (+1). Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- scripts/test_verdict.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/scripts/test_verdict.py b/scripts/test_verdict.py index e3aff5e..c869955 100644 --- a/scripts/test_verdict.py +++ b/scripts/test_verdict.py @@ -450,3 +450,18 @@ def test_populated_evidence_validates_against_the_schema(tmp_path): verdict = build_verdict(workspace) assert verdict["evidence"] jsonschema.validate(verdict, _load_verdict_schema()) + + +def test_terminal_record_with_invalid_utf8_raises_verdict_error(tmp_path): + """An undecodable terminal file must surface as VerdictError, never a raw + UnicodeDecodeError. Today the doctor_report wrapper converts it (doctor + raises UnicodeDecodeError, a ValueError); if projection order ever changes, + _terminal_record's own guard becomes the conversion site - either way the + typed contract holds, so no message is pinned.""" + from loop.verdict import VerdictError, build_verdict + + target = _workspace_with_terminal(tmp_path) + (target / ".loop" / "terminal_state.json").write_bytes(b"\xff\xfe{}") + + with pytest.raises(VerdictError): + build_verdict(target) From 17978e46a3e253648d6f50b2949ced2647660451 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 08:56:02 -0400 Subject: [PATCH 09/14] test(verdict): make the ADR 0002 boundary mechanical Six purity tests so the kernel/signer boundary cannot decay into an intention: no signing-stack token anywhere under loop/, no environment read, verdict.py imports only stdlib + loop.*, the emitted predicate holds the exact field allowlist at every level, carries no free text but run_id (whitespace as the prose proxy), and validates against its own schema. Governor-authored plan-verbatim (Task 5 ships its full test text in the plan; zero judgment surface), with one deviation: the schema-validation test uses pytest.importorskip, not bare __import__ - the plan's version ERRORS instead of skipping in the structural-fallback environment (plan defect five). Teeth proven per the plan's own probe step, extended: P1 os.environ read in verdict.py, P2 extra returned field, P3 prose string leak - all KILLED, tree restored byte-identical. Targeted 6; extras 1351/18 (+6); pyyaml-only 1254/115 (+5 passed, +1 honest skip). Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- scripts/test_verdict_purity.py | 111 +++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 scripts/test_verdict_purity.py diff --git a/scripts/test_verdict_purity.py b/scripts/test_verdict_purity.py new file mode 100644 index 0000000..7012c6f --- /dev/null +++ b/scripts/test_verdict_purity.py @@ -0,0 +1,111 @@ +"""The ADR 0002 boundary, made mechanical. + +These tests exist so the kernel/signer boundary cannot decay into an intention: +the kernel may hash, it must never sign, never read the environment, and never +emit anything but the allowlisted, prose-free predicate body. +""" +import ast +import json +import pathlib +import subprocess +import sys + +import pytest + +REPO = pathlib.Path(__file__).resolve().parent.parent +LOOP = REPO / "loop" + +_SIGNING_TOKENS = ("sigstore", "cosign", "fulcio", "rekor", "dsse", + "private_key", "PRIVATE KEY", "ACTIONS_ID_TOKEN", + "id_token", "oidc") + + +def test_kernel_never_references_a_signing_stack(): + """ADR 0002: the kernel may hash; it must never sign.""" + offenders = [] + for path in LOOP.rglob("*.py"): + text = path.read_text(encoding="utf-8").lower() + for token in _SIGNING_TOKENS: + if token.lower() in text: + offenders.append(f"{path.relative_to(REPO)}:{token}") + assert offenders == [], offenders + + +def test_kernel_reads_no_environment_variable(): + """Zero matches at HEAD 0025acc; this keeps it that way. A verdict that + read GITHUB_SHA would put a vendor identifier inside the portable layer.""" + offenders = [] + for path in LOOP.rglob("*.py"): + text = path.read_text(encoding="utf-8") + if "os.environ" in text or "getenv" in text: + offenders.append(str(path.relative_to(REPO))) + assert offenders == [], offenders + + +def test_verdict_module_imports_only_stdlib_and_loop(): + tree = ast.parse((LOOP / "verdict.py").read_text(encoding="utf-8")) + third_party = [] + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom): + if node.level == 0 and node.module and node.module.split(".")[0] not in sys.stdlib_module_names: + third_party.append(node.module) + elif isinstance(node, ast.Import): + for alias in node.names: + if alias.name.split(".")[0] not in sys.stdlib_module_names: + third_party.append(alias.name) + assert third_party == [], third_party + + +def _predicate(): + proc = subprocess.run( + [sys.executable, "-B", "-m", "loop", "verdict", "examples/flaky-test-triage"], + capture_output=True, text=True, cwd=REPO) + assert proc.returncode == 0, proc.stderr + return json.loads(proc.stdout) + + +def test_predicate_field_allowlist_holds(): + """Everything here is public, append-only, and permanent. Adding a field is + a one-way door — this test is the door.""" + doc = _predicate() + assert set(doc) == {"schema", "run_id", "tool", "doctor", "chain", "terminal", "evidence"} + assert set(doc["doctor"]) == {"ok", "validation_mode", "issue_codes", "schemas_checked"} + assert set(doc["chain"]) == {"head", "sequence", "unchained_prefix"} + assert set(doc["terminal"]) == {"state", "completion_policy", "false_completion"} + for entry in doc["evidence"]: + assert set(entry) == {"digest", "code_digest", "policy_digest"} + + +def test_predicate_carries_no_free_text_but_run_id(): + """run_id is the ONE operator-controlled string, allowlisted deliberately. + Any other prose in the document is a leak into a permanent public log.""" + doc = _predicate() + strings = [] + + def walk(node, path): + if isinstance(node, dict): + for k, v in node.items(): + walk(v, f"{path}.{k}") + elif isinstance(node, list): + for i, v in enumerate(node): + walk(v, f"{path}[{i}]") + elif isinstance(node, str): + strings.append((path, node)) + + walk(doc, "") + for path, value in strings: + if path == ".run_id": + continue + # Whitespace is the proxy for prose. Every other string in the document + # is a digest, an enum, a snake_case issue code, or a slash-separated + # schema id — none of which contain a space. + assert " " not in value, f"free text at {path}: {value!r}" + + +def test_predicate_validates_against_its_own_schema(): + # importorskip, not bare __import__: in the structural-fallback environment + # this must skip honestly rather than error. + jsonschema = pytest.importorskip("jsonschema") + from loop.verdict import _load_verdict_schema + + jsonschema.validate(_predicate(), _load_verdict_schema()) From c3fd7ec498dab12a6ba7d2118fdd0d4a2fc6233b Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 08:58:14 -0400 Subject: [PATCH 10/14] feat(action): opt-in keyless attestation of the verdict predicate New attest input (default false): the gate emits loop verdict's predicate body to RUNNER_TEMP and hands it to actions/attest@v4 with subject-name loop-chain-head + subject-digest sha256:, exposing attestation-url and attestation-id outputs. A legible permission precheck fails before the OIDC 403 would (ACTIONS_ID_TOKEN_REQUEST_URL is present only when the calling job declared id-token: write - a composite action cannot declare its own permissions), an empty chain head skips the attest step with a warning instead of shipping a malformed sha256: subject, and a doctor failure never reaches the attest steps at all. ADR 0002 open items 1 and 5 resolved against the LIVE actions/attest surface (gh api fetch of its action.yml, not memory or docs): subject-digest requires exactly the algorithm:hex form; subject-name is required alongside it; predicate-type + predicate-path are current; v4 is the current major; create-storage-record EXISTS (default true, effective only with push-to-registry true) and is pinned false explicitly. Governor-authored: plan-specified YAML whose one judgment surface WAS the live input verification, which a sandboxed worker cannot perform; the runtime experiment is Task 7's fail-loud CI job. YAML parse + structural lint green; step order chain-head -> verdict -> attest -> skipped-warning preserved. Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- action.yml | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/action.yml b/action.yml index 78a6779..ceda2b0 100644 --- a/action.yml +++ b/action.yml @@ -32,11 +32,26 @@ inputs: detection — the gate then only records the head for a later comparison. required: false default: "" + attest: + description: >- + Emit a loop-engineer/verdict@1 predicate and attest it keylessly via + GitHub OIDC. Requires the CALLING JOB to declare `id-token: write` and + `attestations: write` — a composite action cannot declare its own + permissions. Default false. The predicate is PUBLIC and permanent for + public repositories. + required: false + default: "false" outputs: chain-head: description: "Chain head event_hash observed by this gate run ('' when the store has no chained events)." value: ${{ steps.chain-head.outputs.chain-head }} + attestation-url: + description: "URL of the attestation created by this run ('' when attest is false)." + value: ${{ steps.attest.outputs.attestation-url }} + attestation-id: + description: "ID of the attestation created by this run ('' when attest is false)." + value: ${{ steps.attest.outputs.attestation-id }} runs: using: "composite" @@ -106,6 +121,45 @@ runs: open(sys.argv[3], "a").write(f"chain-head={value}\n") PY + - name: verdict predicate + id: verdict + if: ${{ inputs.attest == 'true' }} + shell: bash + env: + LOOP_PATH: "${{ inputs.path }}" + run: | + # Fail with a legible message rather than a raw OIDC 403 from the attest + # step: a composite action cannot declare permissions, so the CALLING + # job must. ACTIONS_ID_TOKEN_REQUEST_URL is present only when the job + # declared id-token: write. + if [ -z "${ACTIONS_ID_TOKEN_REQUEST_URL:-}" ]; then + echo "::error::attest: true requires the calling job to declare" \ + "'permissions: { id-token: write, attestations: write }'." \ + "A composite action cannot declare its own permissions." + exit 1 + fi + loop verdict "$LOOP_PATH" > "${RUNNER_TEMP}/verdict.json" + echo "predicate-path=${RUNNER_TEMP}/verdict.json" >> "$GITHUB_OUTPUT" + + - name: attest verdict + id: attest + if: ${{ inputs.attest == 'true' && steps.chain-head.outputs.chain-head != '' }} + uses: actions/attest@v4 + with: + subject-name: loop-chain-head + subject-digest: sha256:${{ steps.chain-head.outputs.chain-head }} + predicate-type: urn:loop-engineer:verdict:1 + predicate-path: ${{ steps.verdict.outputs.predicate-path }} + push-to-registry: false + # Only effective when push-to-registry is true, but pinned false + # explicitly per ADR 0002 open item 5: no storage record, ever. + create-storage-record: false + + - name: attest skipped (no chained events) + if: ${{ inputs.attest == 'true' && steps.chain-head.outputs.chain-head == '' }} + shell: bash + run: echo "::warning::attest requested but the store has no chained events; nothing to attest." + - name: loop inspect (scorecard) shell: bash env: From b8d854334c037dc6eeb4bc0af8e2a947333b4f6e Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 09:01:37 -0400 Subject: [PATCH 11/14] ci: attest the verdict over a runner-seeded chained workspace on push to main The job seeds its subject through the REAL writer path - contract + one task + event ramp, then two dispatch_once calls (execute-and-bind, then the runner's auto-terminal) - because the two shapes the plan considered do not survive contact: no tracked examples/* ships an events.db (empty head -> the attest step skips -> an unfalsifiable green job), and scripts/ci_anchor_probe.py seeds a chained-but-unterminated workspace where loop verdict refuses (no terminal record) - and hand-terminating it leaves doctor dirty (state_field_mismatch + desynced_terminal_window), which the composite doctor hard-gate would fail. The runner path was proven locally: doctor ok with zero issues, terminal Succeeded under all_required_verified_evidence, and the attested predicate carries a real chain-bound evidence digest. The job fails loud when no attestation URL is produced or the gate-observed head differs from the seeded head. Push-to-main only (ADR 0002 decision 5); job-scoped id-token/attestations permissions; checkout/setup-python pinned v7 matching ci.yml. Cannot execute before merge - first post-merge run is the live experiment; a failure there is a follow-up PR, not a revert (attest defaults false everywhere else). Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- .github/workflows/attest.yml | 109 +++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 .github/workflows/attest.yml diff --git a/.github/workflows/attest.yml b/.github/workflows/attest.yml new file mode 100644 index 0000000..7b3de20 --- /dev/null +++ b/.github/workflows/attest.yml @@ -0,0 +1,109 @@ +name: attest + +on: + push: + branches: [main] + +permissions: + contents: read + +jobs: + verdict: + # Push-to-default-branch only, by ADR 0002 decision 5: attesting on a PR + # would mint a signed verdict under the repository's identity before review. + runs-on: ubuntu-latest + permissions: + contents: read + id-token: write + attestations: write + steps: + - uses: actions/checkout@v7 + + - uses: actions/setup-python@v7 + with: + python-version: "3.12" + + - name: Install seed dependencies + run: python -m pip install --upgrade pip pyyaml jsonschema + + - name: Seed a terminated, chained workspace + id: seed + # Through the REAL writer path — the runner's dispatch + auto-terminal — + # never a hand-written terminal (doctor flags those as desynced) and + # never a tracked example (no examples/* contract ships an events.db, so + # the attest step would skip and this job would pass having attested + # nothing: unfalsifiable, the exact false-completion shape this project + # refuses). Two dispatches: the first executes the one task and binds + # its evidence into the chain, the second fires the auto-terminal. + run: | + workspace="${RUNNER_TEMP}/verdict-ws" + head="$(python -B - "$workspace" <<'PY' + import json + import pathlib + import sys + + sys.path.insert(0, ".") + from loop import emit + from loop.contract import doctor_report + from loop.events import SQLiteEventStore + from loop.runner import dispatch_once + + ws = pathlib.Path(sys.argv[1]) + emit.open_contract(ws) + task = {"id": "T-1", "title": "T-1", "status": "pending", + "criterion_ref": "T-1", "verify": "./scripts/verify-fast.sh", + "depends_on": [], "attempts": 0, "evidence": None} + (ws / "TASKS.json").write_text( + json.dumps({"schema": "loop-engineer/tasks@1", "tasks": [task]}), + encoding="utf-8") + script = ws / "scripts" / "verify-fast.sh" + script.parent.mkdir(parents=True, exist_ok=True) + script.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8") + script.chmod(0o755) + store = SQLiteEventStore(ws / ".loop" / "events.db") + store.append("run-1", "contract_opened", {"workspace": ws.name}, actor="ci") + for state in ("plan", "critique-plan", "queue-tasks", "execute-task"): + store.append("run-1", "iteration_appended", + {"iteration_id": 0, "outcome": "replanned", "state": state}, + actor="ci") + dispatch_once(ws) + dispatch_once(ws) + report = doctor_report(ws) + if not report["ok"]: + raise SystemExit( + f"seeded workspace is not doctor-clean: " + f"{[issue['code'] for issue in report['issues']]}") + head = (((report["event_store"].get("chain") or {}).get("head") or {}) + .get("event_hash")) or "" + if not head: + raise SystemExit("seeded workspace has no chain head") + print(head) + PY + )" + echo "head=$head" >> "$GITHUB_OUTPUT" + echo "workspace=$workspace" >> "$GITHUB_OUTPUT" + + - id: gate + uses: ./ + with: + path: ${{ steps.seed.outputs.workspace }} + attest: "true" + + - name: assert an attestation was actually created + # Without this the job passes when the attest step skips, which is the + # failure mode this whole task exists to avoid. + env: + SEEDED_HEAD: ${{ steps.seed.outputs.head }} + OBSERVED_HEAD: ${{ steps.gate.outputs.chain-head }} + ATTESTATION: ${{ steps.gate.outputs.attestation-url }} + run: | + if [ -z "$ATTESTATION" ]; then + echo "::error::attest was requested but no attestation URL was produced" + exit 1 + fi + if [ "$OBSERVED_HEAD" != "$SEEDED_HEAD" ]; then + echo "::error::gate observed head '$OBSERVED_HEAD', seed produced '$SEEDED_HEAD'" + exit 1 + fi + echo "chain head: $OBSERVED_HEAD" >> "$GITHUB_STEP_SUMMARY" + echo "attestation: $ATTESTATION" >> "$GITHUB_STEP_SUMMARY" From 241a930bcdc0df7944060bf11f3773149fc13d98 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 09:06:30 -0400 Subject: [PATCH 12/14] docs: normative verdict@1 section and honest limits reference/repo-os-contract.md gains an appended section 23 (no new reference/ file - the structural pin holds the list at eight): the predicate shape with a machine-pinned conformance vector, field-by-field semantics, the refusal/ degradation table (terminal required; store-less projects honestly; unreadable store projects empty evidence - an errored check fails, never skips), the urn:loop-engineer:verdict:1 identity and why it names no vendor host, and the subject-seam disambiguation ADR 0002 open item 1 required once actions/attest constrained the digest key: the chain head is a SHA-256 over a synthesized event preimage, not retrievable bytes - a consumer must never fetch-rehash- compare. The honest-limits block carries the ADR's standing limits in substance: context not correctness, the worker-can-edit-the-verifier path with CODEOWNERS as the control, fabricated-history indistinguishability, the one-run detection latency, and attestation-as-decoration until 4b consumes it. CHANGELOG Unreleased entry in the same voice; the phrase tamper-proof appears nowhere. Gates: self_eval 13/13, frontmatter 9/9, extras 1351/18, pyyaml-only 1254/115 - docs-only, counts unchanged. Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- CHANGELOG.md | 32 +++++++++ reference/repo-os-contract.md | 130 ++++++++++++++++++++++++++++++++++ 2 files changed, 162 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index de191c3..30ae48b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,38 @@ All notable changes to `loop-engineer` are documented here. `WORKFLOW.md` and `README.md` are reworded to describe the mechanism; the 0.3.4 history is left intact. +## Unreleased + +**A verdict you can hand to a signer (slice 4a of tamper-evident provenance).** +The kernel gains `loop verdict `: a pure projection of a finished +run — doctor verdict, chain head, terminal outcome, and the chain-bound +evidence digests that pass the strict verified-evidence bar — into one +canonical `loop-engineer/verdict@1` predicate body (`schemas/verdict.schema.json`, +normative in `reference/repo-os-contract.md` §23). Digests, enums, and issue +codes only; `run_id` is the single operator-controlled string; the field set is +an allowlist held by test. The kernel never signs, never builds an in-toto +Statement, and never reads an environment variable — `scripts/test_verdict_purity.py` +makes each boundary mechanical. + +The composite action gains an opt-in `attest` input (default false): it writes +the predicate to the runner temp dir and hands it to `actions/attest` with +`subject-name: loop-chain-head` / `subject-digest: sha256:`, +exposing `attestation-url`/`attestation-id` outputs; a legible permission +precheck replaces the raw OIDC 403, and an empty chain head skips with a +warning rather than shipping a malformed subject. `.github/workflows/attest.yml` +mints a real attestation on every push to main over a workspace seeded through +the runner's own dispatch + auto-terminal path, and fails loud if no +attestation URL is produced or the observed head differs from the seeded one. + +What this does not buy: the signature attests context — repo, workflow, +trigger, time — never correctness, so a signed verdict over a weakened gate is +just a signed weakened gate. An agent with ordinary merge rights can loosen +`loop/**`/`action.yml`/the workflow and then mint a perfectly genuine +attestation for the result (the control is CODEOWNERS review on those paths, +not signing), and an unattested chain rewrite is detected at best one run +late. Verification (`--compare`, anchor auto-resolution, signer-trust policy) +is slice 4b and does not ship here. + ## 0.11.0 — 2026-07-26 **Verifier identity, and evidence that is load-bearing.** Two slices of the diff --git a/reference/repo-os-contract.md b/reference/repo-os-contract.md index b818e83..a401fb9 100644 --- a/reference/repo-os-contract.md +++ b/reference/repo-os-contract.md @@ -1507,3 +1507,133 @@ match, the mismatch itself surfaces in doctor's issue list as Always pass an anchor in CI. A bare `loop doctor` treats a fully deleted store — no database, no sidecars — as a valid never-ran contract, so "delete the evidence" is a passing run without one. + + +## 23. `loop-engineer/verdict@1` — the CI-attested verdict predicate + +`loop verdict ` projects a **finished** run into one canonical JSON +document: the doctor verdict, the chain head, the terminal outcome, and the +chain-bound evidence digests that pass §17's strict verified-evidence bar. The +kernel emits the **predicate body only**. It never signs, never verifies a +signature, never constructs an in-toto Statement (`_type`, `subject`, +`predicateType`, and `predicate` keys are forbidden in its output), and never +reads an environment variable — every one of those boundaries is a mechanical +test, not an intention. The signer lane (`action.yml`'s opt-in `attest` input +→ `actions/attest`) owns the envelope, the subject, and all cryptography: +**the kernel disposes on contents; the CI lane notarizes context; neither is +the other.** + +Serialization is §16's canonical JSON — sorted keys, compact separators, +`ensure_ascii: false`, `allow_nan: false` — so the same run always projects +the same bytes, and a value canonical JSON cannot carry (a NaN smuggled into a +terminal record) is a typed refusal at the CLI, never a crash. + +**Conformance vector (machine-pinned).** `null` is legal for `chain.head`, +`chain.sequence`, `tool.version`, and `terminal.completion_policy`; every +other key is always present. The field set is an allowlist enforced by test — +everything in this document is public, append-only, and permanent, so adding a +field is a one-way door. + +```json +{ + "schema": "loop-engineer/verdict@1", + "run_id": "coverage-repair", + "tool": { "name": "loop-engineer", "version": "0.11.0" }, + "doctor": { + "ok": true, + "validation_mode": "jsonschema", + "issue_codes": [], + "schemas_checked": ["loop-engineer/manifest@1", "loop-engineer/state@1"] + }, + "chain": { + "head": "9f2c…64hex", + "sequence": 41, + "unchained_prefix": 0 + }, + "terminal": { + "state": "Succeeded", + "completion_policy": "all_required_verified_evidence", + "false_completion": false + }, + "evidence": [ + { + "digest": "a1b2…64hex", + "code_digest": "c3d4…64hex", + "policy_digest": "e5f6…64hex" + } + ] +} +``` + +Field semantics: + +- `run_id` — the **one operator-controlled string** in the document, + allowlisted deliberately so a verdict can be correlated to a run. It lands + in a permanent public log: run ids must not embed sensitive text. Every + other string is a digest, an enum, a snake_case issue code, or a schema id — + a whitespace-bearing value anywhere else is a conformance failure. +- `doctor.issue_codes` — **codes only, sorted, de-duplicated.** Never + `message`, never `path`. Free-text detail strings and workspace paths do + not leave the machine. +- `doctor.validation_mode` — which strength the contract was validated at + (`jsonschema` or `structural-fallback`); without it, `ok: true` from a + fallback environment would read as the stronger claim. +- `chain.head` / `chain.sequence` — the store's chained head (§22's `chain` + block). `null` for a store-less workspace **and** for a store whose chain + never established a head; `unchained_prefix` carries §16's honest count of + events the chain does not cover. +- `terminal.completion_policy` — the policy `mode` string, or `null` for a + legacy record that never declared one. Without it a reader cannot tell what + `Succeeded` meant (`all_required` vs `all_required_verified_evidence`). +- `evidence[]` — one entry per **chain-bound** terminal-evidence record that + passes §17's strict bar, digests only. `digest` is the SHA-256 of the + evidence@1 **record file bytes** — the exact digest the event chain + committed — never the record's own `sha256` field, which hashes the cited + artifact instead. `code_digest`/`policy_digest` lift from the record's + `verified_by`. Entries are de-duplicated and sorted by + `(digest, code_digest, policy_digest)` with `null` ordered as the empty + string. No URIs: a URI is a workspace path, and this document is public. + +**Refusals and degradations.** A workspace with no terminal record refuses, +typed — a verdict projects a finished run. A terminal record whose +`false_completion` is absent or non-boolean refuses: projecting `false` for an +unknown safety flag would trade an alarming truth for a reassuring lie. A +store-less workspace **projects** (`chain.head: null`, evidence under §17's +documented store-dependent degradation) — the projection is honest about what +it cannot prove rather than refusing to say anything. An **unreadable** store +is not that degradation: chain-boundness is then unestablished, so `evidence` +projects empty — an errored check fails, it never skips — while the store +failure itself surfaces in `doctor.issue_codes`. + +**Predicate identity.** `predicateType` is +`urn:loop-engineer:verdict:1` — the schema `$id` transliterated +(`/` and `@` → `:`), an equality asserted by test so the two names cannot +drift. It names no vendor host, no organization, and no repository: a +predicateType is written immutably into a public log, and a rename must not be +able to orphan it. + +**The subject seam — read this before verifying anything.** The signer binds +`subject-name: loop-chain-head` with +`subject-digest: sha256:`. That `sha256:` key satisfies the +signer's required `algorithm:hex_digest` form, but the chain head is a SHA-256 +over a **synthesized event preimage** (§16), not over any retrievable +artifact's bytes. A consumer must never conclude "fetch the bytes, re-hash, +compare" — there are no bytes to fetch. The only meaningful comparison is +equality against a chain head recomputed locally from the store (`loop +doctor`'s `chain` block, or the anchor check via `--expect-chain-head`). +Attested-vs-local agreement is a 4b (`--compare`) concern; authenticity +(`gh attestation verify`) and agreement are separate checks, in that order, +and neither implies the other. + +**What an attestation buys — and does not.** The signature attests *context*: +which repository, which workflow, which trigger, at what time. It never +attests correctness — a signed verdict over a weak gate is a signed weak +gate. A worker with ordinary merge rights can loosen the gate +(`loop/**`, `action.yml`, the workflow) and then mint a perfectly genuine +attestation for the loosened gate; the control is human review on those paths +(ADR 0002 decision 6), not signing. The chain proves order and non-tampering +relative to an anchor, not that the events happened when claimed — a history +fabricated wholesale at authoring time is byte-valid. Detection of an +unattested rewrite is at best one run late. And an attestation nothing +verifies is decoration: until a consumer checks it, this section describes a +publication surface, not a gate. From 1c624ac162de1e97d252325cbd95aafa86f3e7e0 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 09:07:01 -0400 Subject: [PATCH 13/14] chore: require human review on the gate-defining paths CODEOWNERS covers loop/, action.yml, and .github/workflows/ - the three paths that define what the gate checks - per ADR 0002 decision 6: this is the only control for the largest standing limit (a worker with merge rights can loosen the gate, then mint a genuine attestation for it). The file alone enforces nothing until the operator flips the main-protection ruleset to require code-owner review; until then decision 6 is documented, not in force, and the PR body records that state. Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- .github/CODEOWNERS | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 .github/CODEOWNERS diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..9d5d622 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,6 @@ +# Changes to what the gate checks require human review (ADR 0002, decision 6). +# Everything else in this repo stays autonomous. +/loop/ @SollanSystems +/action.yml @SollanSystems +/.github/workflows/ @SollanSystems +/.github/CODEOWNERS @SollanSystems From 67ee119980dd69b127e11f3fc031a669ca6b93c8 Mon Sep 17 00:00:00 2001 From: Sollan Systems Date: Wed, 29 Jul 2026 09:30:48 -0400 Subject: [PATCH 14/14] fix(docs+governance): fix-first findings from the whole-branch review R-001 (p0): the CHANGELOG and section-23 limits language claimed code-owner review as the operative control while the live main-protection ruleset does not yet require it (verified: require_code_owner_review false, approvals 0). Both now state the control is in force only once the ruleset requires code-owner review - a CODEOWNERS file the ruleset does not enforce is documentation, not a control. The PR body carries the operator action. R-002 (high): /schemas/ added to CODEOWNERS and to the loosen-the-gate path list in both docs - the contract schemas define what doctor accepts as ok and what the strict bar accepts as verified evidence, so they are the same enforcement surface as loop/ (an extension beyond ADR decision 6's literal three paths, consistent with its threat model). R-003 (low): the action's marketplace description now names the opt-in attestation capability. R-004 (seed-logic triplication) recorded as a non-blocking follow-up candidate in the PR body. Claude-Session: https://claude.ai/code/session_01JK76jSm45nHcoRoP1SdxXF --- .github/CODEOWNERS | 3 +++ CHANGELOG.md | 8 ++++---- action.yml | 2 +- reference/repo-os-contract.md | 9 ++++++--- 4 files changed, 14 insertions(+), 8 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 9d5d622..08e2a20 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,6 +1,9 @@ # Changes to what the gate checks require human review (ADR 0002, decision 6). +# schemas/ is included beyond the ADR's literal list: the contract schemas +# define what doctor accepts, so they are the same enforcement surface. # Everything else in this repo stays autonomous. /loop/ @SollanSystems +/schemas/ @SollanSystems /action.yml @SollanSystems /.github/workflows/ @SollanSystems /.github/CODEOWNERS @SollanSystems diff --git a/CHANGELOG.md b/CHANGELOG.md index 30ae48b..ff3e67d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,10 +40,10 @@ attestation URL is produced or the observed head differs from the seeded one. What this does not buy: the signature attests context — repo, workflow, trigger, time — never correctness, so a signed verdict over a weakened gate is just a signed weakened gate. An agent with ordinary merge rights can loosen -`loop/**`/`action.yml`/the workflow and then mint a perfectly genuine -attestation for the result (the control is CODEOWNERS review on those paths, -not signing), and an unattested chain rewrite is detected at best one run -late. Verification (`--compare`, anchor auto-resolution, signer-trust policy) +`loop/**`/`schemas/**`/`action.yml`/the workflow and then mint a perfectly +genuine attestation for the result — the control is code-owner review on those +paths, which is in force only once the repository ruleset requires it — and an +unattested chain rewrite is detected at best one run late. Verification (`--compare`, anchor auto-resolution, signer-trust policy) is slice 4b and does not ship here. ## 0.11.0 — 2026-07-26 diff --git a/action.yml b/action.yml index ceda2b0..de4a30d 100644 --- a/action.yml +++ b/action.yml @@ -1,5 +1,5 @@ name: "loop-engineer gate" -description: "Proof-of-done gate for agent-loop contracts: hard-fails on doctor, scores with inspect (warn-only by default)." +description: "Proof-of-done gate for agent-loop contracts: hard-fails on doctor, scores with inspect (warn-only by default), and can keylessly attest the run's verdict@1 predicate (opt-in)." branding: icon: "check-circle" color: "green" diff --git a/reference/repo-os-contract.md b/reference/repo-os-contract.md index a401fb9..3a55b17 100644 --- a/reference/repo-os-contract.md +++ b/reference/repo-os-contract.md @@ -1629,9 +1629,12 @@ and neither implies the other. which repository, which workflow, which trigger, at what time. It never attests correctness — a signed verdict over a weak gate is a signed weak gate. A worker with ordinary merge rights can loosen the gate -(`loop/**`, `action.yml`, the workflow) and then mint a perfectly genuine -attestation for the loosened gate; the control is human review on those paths -(ADR 0002 decision 6), not signing. The chain proves order and non-tampering +(`loop/**`, `schemas/**` — the contract schemas define what doctor accepts — +`action.yml`, or the workflow) and then mint a perfectly genuine attestation +for the loosened gate; the control is human review on those paths (ADR 0002 +decision 6), not signing — and CODEOWNERS is that control **only while the +repository's ruleset requires code-owner review**. A CODEOWNERS file the +ruleset does not enforce is documentation, not a control. The chain proves order and non-tampering relative to an anchor, not that the events happened when claimed — a history fabricated wholesale at authoring time is byte-valid. Detection of an unattested rewrite is at best one run late. And an attestation nothing