diff --git a/README.md b/README.md index ee26367..ae8bf75 100644 --- a/README.md +++ b/README.md @@ -216,7 +216,7 @@ terminal outcome: .loop/ manifest.yaml # contract metadata state.json # live FSM cursor - terminal_state.json # final exit record; written once — emit refuses overwrite without force=True + terminal_state.json # final exit record; immutable once written artifacts/ # evidence bundles and intermediate outputs approvals/ # approval requests and resolutions checkpoints/ # recoverable snapshots @@ -301,7 +301,8 @@ otherwise, the stop is blocked with the exact doctor issues. No-op without **Any Python runtime** — `loop.emit` is a pure-stdlib writer for foreign orchestrators (LangGraph, or anything that can call four functions): `open_contract`, `append_iteration`, `append_receipt`, `terminate`. The writer -refuses an evidence-free `Succeeded` at write time. Recipe: +refuses `Succeeded` unless every declared criterion is true and evidence is +present. Recipe: [docs/integrations/langgraph.md](docs/integrations/langgraph.md). **CI** — one workflow step validates the contract and publishes a scorecard: diff --git a/docs/adr/0001-proof-kernel-and-runtime.md b/docs/adr/0001-proof-kernel-and-runtime.md new file mode 100644 index 0000000..81a2e19 --- /dev/null +++ b/docs/adr/0001-proof-kernel-and-runtime.md @@ -0,0 +1,104 @@ +# ADR 0001: Separate the proof kernel from the execution runtime + +- **Status:** Accepted +- **Date:** 2026-07-12 +- **Decision owners:** Loop Engineer maintainers + +## Context + +Loop Engineer began as a portable contract and proof layer. Its current core is +valuable because it defines typed terminal states, evidence-gated completion, +anti-cheat checks, bounded repair, and repo-native state without binding those +concepts to a particular orchestration framework. + +The product goal is broader: a user should be able to provide a complex goal, +receive a reviewable loop design, execute it through interchangeable agents and +tools, pause for approvals, recover from failure, and obtain an independently +verified terminal result. + +Keeping execution out of the project would preserve a small scope, but it would +also leave the most important invariants as instructions that a host agent may +ignore. Folding every concern into one package would create the opposite +problem: provider-specific execution details would contaminate the portable +proof protocol. + +## Decision + +Loop Engineer will have two first-party layers with a strict dependency +direction. + +### 1. Proof kernel + +The proof kernel is the stable, runtime-neutral protocol. It owns: + +- contract and schema versions; +- deterministic completion-policy evaluation; +- legal state and terminal-state projection; +- evidence and provenance rules; +- verifier and anti-cheat interfaces; +- policy validation and conformance tests; +- event reduction and replay semantics. + +The kernel must not import model providers, agent frameworks, or workflow +engines. It may be embedded by foreign runtimes. + +### 2. Execution runtime + +The execution runtime interprets a validated Loop Plan and owns: + +- planning and task scheduling; +- worker leases and attempt numbers; +- agent and tool dispatch; +- budgets, retries, timeouts, pause, resume, and cancellation; +- approvals and side-effect policy; +- checkpointing and crash recovery; +- persistence of immutable events and artifacts. + +The runtime depends on the kernel. The kernel never depends on the runtime. + +## Governing rule + +**Agents propose; the kernel disposes.** + +An agent may propose a command, patch, transition, or completion claim. Only the +kernel may validate and commit a state transition or terminal result. + +## Immediate consequences + +1. `Succeeded` uses an explicit completion policy. The first supported policy is + `all_required`; every declared criterion must be proven true. +2. Terminal records are immutable. Corrections will be represented by separate, + auditable administrative events rather than file replacement. +3. New state writers use canonical integer iteration identifiers. Legacy + numeric strings remain a read-compatibility concern until a versioned state + migration removes them. +4. The next persistence milestone will introduce an `EventStore` protocol and a + SQLite/WAL implementation. JSON and Markdown files become projections rather + than the sole authoritative state. +5. Provider and model selection will be capability-based, not encoded in the + portable contract as vendor model names. + +## Non-goals of this decision + +This ADR does not select a distributed scheduler, hosted control plane, web UI, +or model provider. It also does not make structural evidence equivalent to +cryptographic attestation. Those require separate decisions. + +## Rejected alternatives + +### Remain contract-only + +Rejected because the desired product must execute and govern loops end to end. +A prose-only state machine cannot reliably enforce concurrency, approvals, +budgets, or immutable terminal decisions. + +### Build a monolithic provider-specific agent framework + +Rejected because it would erase the strongest differentiation: a portable proof +contract that can sit above multiple runtimes. + +### Permit terminal overwrite for operator convenience + +Rejected because an overwritten terminal record destroys audit history and +creates a race in which a later writer can launder an earlier result. A future +supersession event can preserve both the original decision and the correction. diff --git a/loop/completion.py b/loop/completion.py new file mode 100644 index 0000000..6d9a1ba --- /dev/null +++ b/loop/completion.py @@ -0,0 +1,73 @@ +"""Deterministic completion-policy evaluation. + +The first portable policy is intentionally narrow: every declared acceptance +criterion is required. Keeping the evaluator in a small, side-effect-free +module lets emitters, runtime adapters, and contract validation share exactly +the same success semantics. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Final, Literal, TypeAlias, cast + +CompletionMode: TypeAlias = Literal["all_required"] +DEFAULT_COMPLETION_MODE: Final[CompletionMode] = "all_required" +SUPPORTED_COMPLETION_MODES: Final[tuple[CompletionMode, ...]] = (DEFAULT_COMPLETION_MODE,) + + +class CompletionPolicyError(ValueError): + """The requested completion policy is malformed or unsupported.""" + + +def normalize_completion_policy(policy: object | None = None) -> dict[str, CompletionMode]: + """Return the canonical JSON form of a supported completion policy. + + ``None`` is the compatibility default for terminal@1 records created before + the policy field existed. New writers should always persist the returned + object explicitly. + """ + if policy is None: + mode: object = DEFAULT_COMPLETION_MODE + elif isinstance(policy, str): + mode = policy + elif isinstance(policy, Mapping): + unexpected = sorted(str(key) for key in policy if key != "mode") + if unexpected: + raise CompletionPolicyError( + "completion_policy contains unsupported fields: " + ", ".join(unexpected) + ) + if "mode" not in policy: + raise CompletionPolicyError("completion_policy.mode is required") + mode = policy.get("mode") + else: + raise CompletionPolicyError( + "completion_policy must be null, a mode string, or an object with a mode field" + ) + + if mode not in SUPPORTED_COMPLETION_MODES: + supported = ", ".join(SUPPORTED_COMPLETION_MODES) + raise CompletionPolicyError( + f"unsupported completion policy mode {mode!r}; expected one of: {supported}" + ) + return {"mode": cast(CompletionMode, mode)} + + +def criteria_satisfy_completion( + criteria_met: Mapping[str, object], + policy: object | None = None, +) -> bool: + """Return whether a criteria map satisfies the declared policy. + + An empty map never proves completion. Values must be the boolean singleton + ``True``; truthy substitutes such as ``1`` are deliberately rejected. + """ + normalized = normalize_completion_policy(policy) + if normalized["mode"] == "all_required": + return bool(criteria_met) and all(value is True for value in criteria_met.values()) + raise AssertionError(f"unhandled completion policy: {normalized!r}") + + +def unmet_required_criteria(criteria_met: Mapping[str, object]) -> tuple[str, ...]: + """Return stable string identifiers for criteria not proven true.""" + return tuple(sorted(key for key, value in criteria_met.items() if value is not True)) diff --git a/loop/contract.py b/loop/contract.py index 9009389..7336723 100644 --- a/loop/contract.py +++ b/loop/contract.py @@ -4,6 +4,12 @@ from pathlib import Path from typing import Any +from .completion import ( + CompletionPolicyError, + criteria_satisfy_completion, + normalize_completion_policy, + unmet_required_criteria, +) from .paths import LoopPaths, resolve_loop_paths TERMINAL_STATES = ( @@ -157,11 +163,39 @@ def _validate_state(data: dict[str, Any] | None, path: Path, issues: list[dict]) for key in ("iteration_id", "state", "plan_version", "budget_remaining"): if key not in data: issues.append(ContractIssue("missing_state_field", f"state missing {key}", path)) + + if "iteration_id" in data: + iteration_id = data.get("iteration_id") + canonical_integer = ( + isinstance(iteration_id, int) + and not isinstance(iteration_id, bool) + and iteration_id >= 0 + ) + legacy_decimal = ( + isinstance(iteration_id, str) + and ( + iteration_id == "0" + or ( + iteration_id.isascii() + and iteration_id.isdigit() + and not iteration_id.startswith("0") + ) + ) + ) + if not (canonical_integer or legacy_decimal): + issues.append( + ContractIssue( + "invalid_state", + "iteration_id must be a non-negative integer " + "(legacy canonical decimal strings remain read-compatible)", + path, + ) + ) + terminal = data.get("terminal_state") if terminal is not None and terminal not in TERMINAL_STATES: issues.append(ContractIssue("invalid_terminal_state", f"invalid terminal_state {terminal!r}", path)) - def _check_tasks_semantics(data: dict[str, Any] | None, path: Path, issues: list[dict]) -> None: """Cross-task rules JSON Schema cannot express: id uniqueness and the evidence-before-done invariant. Runs in both validation modes.""" @@ -202,49 +236,142 @@ def _validate_tasks(data: dict[str, Any] | None, path: Path, issues: list[dict]) _check_tasks_semantics(data, path, issues) -def _check_terminal_contradiction(data: dict[str, Any] | None, path: Path, issues: list[dict]) -> None: - """G1: a Succeeded terminal must not contradict its own evidence. +def _check_terminal_contradiction( + data: dict[str, Any] | None, + path: Path, + issues: list[dict], +) -> None: + """G1: a Succeeded terminal must prove every required criterion. - Runs in both validation modes because JSON Schema cannot express the - cross-field rule that a success claim requires false_completion=false, at - least one met criterion, AND a non-empty evidence list — mirroring the - write-time refusal in loop/emit.py (an evidence-free Succeeded is a claim - with nothing behind it). + This semantic rule runs in both validation modes because JSON Schema cannot + express the relationship between the completion policy, criteria map, + false-completion flag, and evidence list. """ - if not isinstance(data, dict) or data.get("state") != "Succeeded": return + + try: + policy = normalize_completion_policy(data.get("completion_policy")) + except CompletionPolicyError as exc: + issues.append( + ContractIssue( + "invalid_completion_policy", + f"Succeeded terminal has invalid completion_policy: {exc}", + path, + ) + ) + return + if data.get("false_completion") is True: issues.append( - ContractIssue("contradictory_terminal", "Succeeded terminal declares false_completion=true", path) + ContractIssue( + "contradictory_terminal", + "Succeeded terminal declares false_completion=true", + path, + ) ) + criteria = data.get("criteria_met") - if not isinstance(criteria, dict) or not any(v is True for v in criteria.values()): + if not isinstance(criteria, dict) or not criteria_satisfy_completion(criteria, policy): + unmet = unmet_required_criteria(criteria) if isinstance(criteria, dict) else () + detail = ", ".join(unmet) if unmet else "no criteria were declared" issues.append( - ContractIssue("contradictory_terminal", "Succeeded terminal has no met (true) entry in criteria_met", path) + ContractIssue( + "contradictory_terminal", + "Succeeded terminal criteria_met does not prove every required criterion: " + detail, + path, + ) ) + evidence = data.get("evidence") if not isinstance(evidence, list) or not evidence: issues.append( - ContractIssue("contradictory_terminal", "Succeeded terminal has empty evidence[] (G1)", path) + ContractIssue( + "contradictory_terminal", + "Succeeded terminal has empty evidence[] (G1)", + path, + ) ) - def _validate_terminal(data: dict[str, Any] | None, path: Path, issues: list[dict]) -> None: _require_schema(data, "loop-engineer/terminal@1", path, issues) if data is None: return if data.get("state") not in TERMINAL_STATES: issues.append(ContractIssue("invalid_terminal_state", f"invalid state {data.get('state')!r}", path)) - if not isinstance(data.get("criteria_met"), dict): + + criteria = data.get("criteria_met") + if not isinstance(criteria, dict): issues.append(ContractIssue("invalid_terminal", "criteria_met must be an object", path)) - if not isinstance(data.get("evidence"), list): + else: + if not all(isinstance(key, str) and key.strip() for key in criteria): + issues.append( + ContractIssue( + "invalid_terminal", + "criteria_met keys must be non-empty strings", + path, + ) + ) + if not all(isinstance(value, bool) for value in criteria.values()): + issues.append( + ContractIssue( + "invalid_terminal", + "criteria_met values must be booleans", + path, + ) + ) + + try: + normalize_completion_policy(data.get("completion_policy")) + except CompletionPolicyError as exc: + issues.append( + ContractIssue( + "invalid_terminal", + f"completion_policy is invalid: {exc}", + path, + ) + ) + + evidence = data.get("evidence") + if not isinstance(evidence, list): issues.append(ContractIssue("invalid_terminal", "evidence must be a list", path)) + else: + all_strings = all(isinstance(item, str) for item in evidence) + if not all_strings or any(not item.strip() for item in evidence): + issues.append( + ContractIssue( + "invalid_terminal", + "evidence entries must be non-empty strings", + path, + ) + ) + if all_strings and len(set(evidence)) != len(evidence): + issues.append( + ContractIssue( + "invalid_terminal", + "evidence entries must be unique", + path, + ) + ) + + iteration_id = data.get("iteration_id") + if iteration_id is not None and ( + not isinstance(iteration_id, int) + or isinstance(iteration_id, bool) + or iteration_id < 0 + ): + issues.append( + ContractIssue( + "invalid_terminal", + "iteration_id must be a non-negative integer", + path, + ) + ) + if not isinstance(data.get("false_completion"), bool): issues.append(ContractIssue("invalid_terminal", "false_completion must be bool", path)) _check_terminal_contradiction(data, path, issues) - def _validate_manifest(data: dict[str, Any] | None, path: Path, issues: list[dict]) -> None: if data is None: issues.append(ContractIssue("missing_file", "missing manifest.yaml", path)) diff --git a/loop/emit.py b/loop/emit.py index 3d4315a..097c312 100644 --- a/loop/emit.py +++ b/loop/emit.py @@ -1,8 +1,8 @@ """Writer API for foreign runtimes (B1). A writer, never a runtime: it renders contract artifacts and refuses dishonest ones — no orchestration, no execution. -The G1 cross-check (a Succeeded terminal needs evidence and a met criterion) -is enforced HERE, at write time, before doctor ever sees the file. +The G1 cross-check (a Succeeded terminal needs evidence and every required +criterion) is enforced HERE, at write time, before doctor ever sees the file. """ from __future__ import annotations @@ -14,6 +14,12 @@ from pathlib import Path from typing import Any, Sequence +from .completion import ( + CompletionPolicyError, + criteria_satisfy_completion, + normalize_completion_policy, + unmet_required_criteria, +) from .contract import ( TERMINAL_STATES, _validate_record, @@ -72,6 +78,8 @@ def _atomic_write_text(path: Path, text: str) -> None: try: with os.fdopen(fd, "w", encoding="utf-8") as fh: fh.write(text) + fh.flush() + os.fsync(fh.fileno()) os.replace(tmp_name, path) except BaseException: try: @@ -81,6 +89,35 @@ def _atomic_write_text(path: Path, text: str) -> None: raise +def _atomic_create_text(path: Path, text: str) -> None: + """Create ``path`` exactly once from a fully-written same-directory temp file. + + The hard-link step is atomic and refuses an existing destination, closing the + check-then-replace race that would otherwise let concurrent terminators + overwrite one another. The destination never names a partially-written file. + """ + fd, tmp_name = tempfile.mkstemp(dir=str(path.parent), prefix=path.name + ".", suffix=".tmp") + try: + with os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.write(text) + fh.flush() + os.fsync(fh.fileno()) + os.link(tmp_name, path) + finally: + try: + os.unlink(tmp_name) + except FileNotFoundError: + pass + + +def _require_iteration_id(value: object, *, optional: bool = False) -> int | None: + if optional and value is None: + return None + if not isinstance(value, int) or isinstance(value, bool) or value < 0: + raise EmitError("iteration_id must be a non-negative integer") + return value + + def _write_state(paths, state: dict[str, Any]) -> None: _atomic_write_text(paths.state, json.dumps(state, indent=2) + "\n") @@ -101,6 +138,7 @@ def append_iteration( .loop/state.json's iteration_id/active_task.""" if outcome not in _ITERATION_OUTCOMES: raise EmitError(f"unknown iteration outcome {outcome!r}; expected one of {_ITERATION_OUTCOMES}") + _require_iteration_id(iteration_id) paths = _require_contract(target) lines = [ @@ -139,7 +177,7 @@ def append_iteration( fh.write("\n".join(lines)) state = _read_state(paths) - state["iteration_id"] = str(iteration_id) + state["iteration_id"] = iteration_id if task_id: state["active_task"] = task_id _write_state(paths, state) @@ -199,36 +237,61 @@ def terminate( iteration_id: int | None = None, false_completion: bool = False, lessons_ref: str | None = None, + completion_policy: object | None = None, force: bool = False, ) -> Path: - """Write .loop/terminal_state.json (and stamp state.json.terminal_state). + """Write an immutable ``.loop/terminal_state.json`` and stamp state.json. - Refuses an evidence-free Succeeded — the G1 cross-check at write time: - Succeeded requires non-empty evidence, at least one met criterion, and - false_completion=False. + ``Succeeded`` requires non-empty evidence, ``false_completion=False``, and + every declared criterion to satisfy the explicit completion policy. The + compatibility default is ``{"mode": "all_required"}``, and new records + always persist it. - The terminal record is written once: a second terminate on an existing - terminal file is refused unless force=True (the deliberate-overwrite escape - hatch). + ``force`` remains temporarily in the signature so older callers receive an + actionable error instead of silently overwriting an audit record. It never + permits replacement. """ if state not in TERMINAL_STATES: raise EmitError(f"unknown terminal state {state!r}; expected one of {TERMINAL_STATES}") + if force: + raise EmitError( + "force=True is no longer supported: terminal records are immutable; " + "record any correction as a separate administrative event" + ) + if not isinstance(criteria_met, dict): + raise EmitError("criteria_met must be an object") + if not all(isinstance(key, str) and key.strip() for key in criteria_met): + raise EmitError("criteria_met keys must be non-empty strings") + if not all(isinstance(value, bool) for value in criteria_met.values()): + raise EmitError("criteria_met values must be booleans") + if not isinstance(evidence, list): + raise EmitError("evidence must be a list") + if any(not isinstance(item, str) or not item.strip() for item in evidence): + raise EmitError("evidence entries must be non-empty strings") + if len(set(evidence)) != len(evidence): + raise EmitError("evidence entries must be unique") + try: + normalized_policy = normalize_completion_policy(completion_policy) + except CompletionPolicyError as exc: + raise EmitError(str(exc)) from exc + _require_iteration_id(iteration_id, optional=True) + if state == "Succeeded": if false_completion: raise EmitError("refusing Succeeded with false_completion=True (G1 contradiction)") if not evidence: raise EmitError("refusing evidence-free Succeeded: evidence[] is empty (G1)") - if not any(v is True for v in criteria_met.values()): - raise EmitError("refusing Succeeded with no met (true) entry in criteria_met (G1)") - if not all(isinstance(v, bool) for v in criteria_met.values()): - raise EmitError("criteria_met values must be booleans") + if not criteria_satisfy_completion(criteria_met, normalized_policy): + unmet = unmet_required_criteria(criteria_met) + detail = ", ".join(unmet) if unmet else "no criteria were declared" + raise EmitError( + "refusing Succeeded because not all required criteria are proven true: " + detail + ) + paths = _require_contract(target) terminal_path = paths.loop_dir / "terminal_state.json" - if terminal_path.is_file() and not force: - raise EmitError( - f"terminal already written at {terminal_path} — the terminal record is " - f"written once; pass force=True to deliberately overwrite it" - ) + if terminal_path.is_file(): + raise EmitError(f"terminal already written at {terminal_path} — terminal records are immutable") current = _read_state(paths) terminal: dict[str, Any] = { @@ -236,6 +299,7 @@ def terminate( "project": paths.workspace.name, "state": state, "criteria_met": dict(criteria_met), + "completion_policy": normalized_policy, "evidence": list(evidence), "false_completion": false_completion, "terminated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), @@ -252,7 +316,45 @@ def terminate( if issues: raise EmitError(f"terminal failed validation before write: {issues}") - _atomic_write_text(terminal_path, json.dumps(terminal, indent=2) + "\n") + try: + _atomic_create_text(terminal_path, json.dumps(terminal, indent=2) + "\n") + except FileExistsError as exc: + raise EmitError( + f"terminal already written at {terminal_path} — terminal records are immutable" + ) from exc + except OSError as exc: + raise EmitError(f"terminal write failed at {terminal_path}: {exc}") from exc current["terminal_state"] = state - _write_state(paths, current) + try: + _write_state(paths, current) + except OSError as exc: + raise EmitError( + f"terminal written at {terminal_path} but state.json was not stamped: {exc} — " + "call emit.sync_state_to_terminal() to reconcile" + ) from exc return terminal_path + + +def sync_state_to_terminal(target: str | Path) -> Path: + """Stamp state.json's ``terminal_state`` from an existing terminal record. + + The narrow repair for a crash or failed write between the immutable + ``terminal_state.json`` creation and the state.json stamp — the two files + are not one transaction. Reads the terminal record and reconciles + state.json to it; never creates, alters, or removes the terminal file. + """ + paths = _require_contract(target) + terminal_path = paths.loop_dir / "terminal_state.json" + if not terminal_path.is_file(): + raise EmitError(f"no terminal record at {terminal_path} — nothing to sync") + try: + terminal = json.loads(terminal_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise EmitError(f"unreadable terminal_state.json: {exc}") from exc + if not isinstance(terminal, dict) or terminal.get("state") not in TERMINAL_STATES: + raise EmitError(f"terminal_state.json at {terminal_path} does not hold a valid terminal record") + current = _read_state(paths) + if current.get("terminal_state") != terminal["state"]: + current["terminal_state"] = terminal["state"] + _write_state(paths, current) + return paths.state diff --git a/loop/integrations.py b/loop/integrations.py index 7b3907f..8b5a736 100644 --- a/loop/integrations.py +++ b/loop/integrations.py @@ -9,8 +9,8 @@ The fixed precedence — safety -> human -> blocked -> budget -> spec-gap -> gate verdict — means a gamed (FailedSafety) or human-killed (AbortedByHuman) run can never launder itself into Succeeded. ``Succeeded`` is reachable ONLY -via a green gate verdict + anticheat clean of HIGH/CRITICAL findings + at -least one met criterion + non-empty evidence. ``false_completion`` is +via a green gate verdict + anticheat clean of HIGH/CRITICAL findings + every +required criterion met + non-empty evidence. ``false_completion`` is copied out of the gate result, never synthesized. A missing or structurally-empty gate/anticheat input fails closed to ``FailedUnverifiable`` — the same posture as ``holdout_gate`` on an empty @@ -25,6 +25,13 @@ from dataclasses import dataclass from typing import Sequence +from .completion import ( + CompletionPolicyError, + criteria_satisfy_completion, + normalize_completion_policy, + unmet_required_criteria, +) + TERMINAL_SCHEMA = "loop-engineer/terminal@1" @@ -71,6 +78,8 @@ def to_terminal_state( gate_verdict: dict | None, anticheat: dict | None, criteria_met: dict[str, bool | None], + *, + completion_policy: object | None = None, ) -> dict: """Project an engine terminal + gate/anticheat evidence into a terminal@1 body. @@ -83,12 +92,43 @@ def to_terminal_state( ac = anticheat if isinstance(anticheat, dict) else {} false_completion = gate.get("false_completion") is True + criteria_error: str | None = None + if not isinstance(criteria_met, dict): + criteria_error = "criteria_met must be an object" + canonical_criteria: dict[str, bool | None] = {} + else: + canonical_criteria = {} + for key, value in criteria_met.items(): + if not isinstance(key, str) or not key.strip(): + criteria_error = "criteria identifiers must be non-empty strings" + continue + if value is not True and value is not False and value is not None: + criteria_error = f"criterion {key!r} must be true, false, or null" + continue + canonical_criteria[key] = value + + artifacts = tuple(str(item) for item in outcome.artifacts) + evidence_error: str | None = None + if any(not item.strip() for item in artifacts): + evidence_error = "evidence artifact paths must be non-empty strings" + elif len(set(artifacts)) != len(artifacts): + evidence_error = "evidence artifact paths must be unique" + + try: + normalized_policy = normalize_completion_policy(completion_policy) + policy_error: str | None = None + except CompletionPolicyError as exc: + # Projection APIs fail closed rather than throwing a runtime result away. + normalized_policy = normalize_completion_policy() + policy_error = str(exc) + def body(state: str, reason: str) -> dict: return { "schema": TERMINAL_SCHEMA, "state": state, - "criteria_met": {str(k): v is True for k, v in criteria_met.items()}, - "evidence": list(outcome.artifacts), + "criteria_met": {key: value is True for key, value in canonical_criteria.items()}, + "completion_policy": normalized_policy, + "evidence": list(artifacts), "false_completion": false_completion, "reason": reason, } @@ -101,7 +141,11 @@ def body(state: str, reason: str) -> dict: return body("FailedBlocked", f"unrecoverable external block: {outcome.external_error}") if outcome.budget_exhausted: return body("FailedBudget", "engine budget cap hit (steps/tokens/wall-clock/cost)") - unmapped = sorted(str(k) for k, v in criteria_met.items() if v is None) + if policy_error is not None: + return body("FailedSpecGap", "invalid completion policy: " + policy_error) + if criteria_error is not None: + return body("FailedSpecGap", "invalid criteria map: " + criteria_error) + unmapped = sorted(key for key, value in canonical_criteria.items() if value is None) if unmapped: return body("FailedSpecGap", "criteria with no mapped check: " + ", ".join(unmapped)) if not _valid_anticheat(ac): @@ -116,10 +160,20 @@ def body(state: str, reason: str) -> dict: return body("FailedUnverifiable", f"gate verdict {gate['verdict']!r} — cannot certify Succeeded") if false_completion: return body("FailedUnverifiable", "gate flags false_completion — refusing Succeeded") - if not any(v is True for v in criteria_met.values()): - return body("FailedUnverifiable", "green gate but no met criterion — cannot certify") - if not outcome.artifacts: + if not criteria_satisfy_completion(canonical_criteria, normalized_policy): + unmet = unmet_required_criteria(canonical_criteria) + detail = ", ".join(unmet) if unmet else "no criteria were declared" + return body( + "FailedUnverifiable", + "green gate but not all required criteria are proven true: " + detail, + ) + if evidence_error is not None: + return body("FailedUnverifiable", "invalid evidence artifacts: " + evidence_error) + if not artifacts: return body("FailedUnverifiable", "green gate but no evidence artifacts — cannot certify") if not outcome.reached_end: return body("FailedUnverifiable", "engine did not reach its own terminal signal") - return body("Succeeded", "holdout gate green, anticheat clean, criteria met with evidence") + return body( + "Succeeded", + "holdout gate green, anticheat clean, all required criteria met with evidence", + ) diff --git a/reference/repo-os-contract.md b/reference/repo-os-contract.md index 3efa46b..cca4ede 100644 --- a/reference/repo-os-contract.md +++ b/reference/repo-os-contract.md @@ -296,6 +296,7 @@ sets it; resolution clears it; the loop never spawns a fresh untracked attempt ( | `state` | enum | One of the 7 above. | | `iteration_id` | int | Final iteration count. | | `criteria_met` | object | `{ "": true\|false }` for every `SPEC.md` criterion. | +| `completion_policy` | object | Completion rule for the criteria map. v1 supports `{ "mode": "all_required" }`; legacy records without the field are interpreted the same way. Optional (additive). Note: a pre-migration `Succeeded` record whose criteria map contains any `false` value fails this rule and needs re-verification. | | `evidence` | string[] | Paths to the verification bundles backing the verdict. | | `false_completion` | bool | True if the loop had earlier *claimed* success that verification later refuted (feeds the false-completion-rate metric). | | `reason` | string | One line: why this terminal state, especially for any `Failed*`/`Aborted*`. | @@ -307,6 +308,7 @@ sets it; resolution clears it; the loop never spawns a fresh untracked attempt ( "state": "Succeeded", "iteration_id": 2, "criteria_met": { "1": true, "2": true }, + "completion_policy": { "mode": "all_required" }, "evidence": [".loop/artifacts/verify-T2.json", ".loop/artifacts/verify-T1.json"], "false_completion": false, "reason": "All SPEC criteria verified: coverage 0.83 >= 0.80; validation tests pass.", @@ -521,8 +523,9 @@ land silently. (`terminal_state` is one of the canonical 7 **and** `terminal_state.json` is present and valid). - **B2** — `terminal_state.json`, when present, validates against `loop-engineer/terminal@1` with a `criteria_met` object, an `evidence` list, and an explicit `false_completion` boolean; a - `Succeeded` terminal additionally has `false_completion=false`, at least one true criterion, and - non-empty `evidence`. + `Succeeded` terminal additionally has `false_completion=false`, every declared criterion true + under `completion_policy.mode=all_required` (legacy records without the field are interpreted + the same way), and non-empty `evidence`. **C. Evidentiary trail (checked when present)** - **C1** — every `.loop/receipts/*.jsonl` line validates against `loop-engineer/receipt@1`. diff --git a/schemas/state.schema.json b/schemas/state.schema.json index 0c2cf5b..d899302 100644 --- a/schemas/state.schema.json +++ b/schemas/state.schema.json @@ -2,23 +2,88 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "loop-engineer/state@1", "title": "Loop Engineer State @1", - "description": "Reconciled with loop/contract.py and the shipped contracts (examples/coverage-repair, roadmap/v1.0). Loosened from the original: required is narrowed to the structural core the hand checks enforce (active_task/best_score/failure_mode/pending_approval/checkpoint_path moved to optional — an in-flight state legitimately omits them); iteration_id also accepts a string and best_score/pending_approval accept the richer forms real contracts emit.", + "description": "Runtime state for a Loop Engineer contract. New writers emit a non-negative integer iteration_id; legacy decimal strings remain read-compatible during the terminal@1/state@1 migration window.", "type": "object", - "required": ["schema", "iteration_id", "state", "plan_version", "budget_remaining"], + "required": [ + "schema", + "iteration_id", + "state", + "plan_version", + "budget_remaining" + ], "properties": { - "schema": { "const": "loop-engineer/state@1" }, - "iteration_id": { "type": ["integer", "string"] }, - "state": { "type": "string" }, - "plan_version": { "type": "integer", "minimum": 0 }, - "active_task": { "type": ["string", "null"] }, - "best_score": { "type": ["number", "string", "null"] }, - "failure_mode": { "type": ["string", "null"] }, - "pending_approval": { "type": ["string", "object", "null"] }, - "budget_remaining": { "type": "object" }, - "checkpoint_path": { "type": ["string", "null"] }, + "schema": { + "const": "loop-engineer/state@1" + }, + "iteration_id": { + "oneOf": [ + { + "type": "integer", + "minimum": 0 + }, + { + "type": "string", + "pattern": "^(0|[1-9][0-9]*)$" + } + ] + }, + "state": { + "type": "string" + }, + "plan_version": { + "type": "integer", + "minimum": 0 + }, + "active_task": { + "type": [ + "string", + "null" + ] + }, + "best_score": { + "type": [ + "number", + "string", + "null" + ] + }, + "failure_mode": { + "type": [ + "string", + "null" + ] + }, + "pending_approval": { + "type": [ + "string", + "object", + "null" + ] + }, + "budget_remaining": { + "type": "object" + }, + "checkpoint_path": { + "type": [ + "string", + "null" + ] + }, "terminal_state": { - "type": ["string", "null"], - "enum": [null, "Succeeded", "FailedUnverifiable", "FailedBlocked", "FailedBudget", "FailedSafety", "FailedSpecGap", "AbortedByHuman"] + "type": [ + "string", + "null" + ], + "enum": [ + null, + "Succeeded", + "FailedUnverifiable", + "FailedBlocked", + "FailedBudget", + "FailedSafety", + "FailedSpecGap", + "AbortedByHuman" + ] } }, "additionalProperties": true diff --git a/schemas/terminal.schema.json b/schemas/terminal.schema.json index b83dfa7..525113b 100644 --- a/schemas/terminal.schema.json +++ b/schemas/terminal.schema.json @@ -2,18 +2,85 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "loop-engineer/terminal@1", "title": "Loop Engineer Terminal State @1", - "description": "Reconciled with loop/contract.py and the shipped contracts. Loosened from the original: iteration_id, reason and lessons_ref moved from required to optional because a real terminal (roadmap/v1.0) records the equivalent under other keys — the load-bearing cross-field rule (a Succeeded terminal needs false_completion=false and a met criterion) is enforced in code, not the required list. reason keeps minLength:1 so, when present, it must be non-empty.", + "description": "A typed terminal result. Legacy terminal@1 records without completion_policy are interpreted as all_required. Cross-field success rules are enforced by loop.contract.", "type": "object", - "required": ["schema", "state", "criteria_met", "evidence", "false_completion"], + "required": [ + "schema", + "state", + "criteria_met", + "evidence", + "false_completion" + ], "properties": { - "schema": { "const": "loop-engineer/terminal@1" }, - "state": { "enum": ["Succeeded", "FailedUnverifiable", "FailedBlocked", "FailedBudget", "FailedSafety", "FailedSpecGap", "AbortedByHuman"] }, - "iteration_id": { "type": "integer", "minimum": 0 }, - "criteria_met": { "type": "object", "additionalProperties": { "type": "boolean" } }, - "evidence": { "type": "array", "items": { "type": "string" } }, - "false_completion": { "type": "boolean" }, - "reason": { "type": "string", "minLength": 1 }, - "lessons_ref": { "type": ["string", "null"] } + "schema": { + "const": "loop-engineer/terminal@1" + }, + "state": { + "enum": [ + "Succeeded", + "FailedUnverifiable", + "FailedBlocked", + "FailedBudget", + "FailedSafety", + "FailedSpecGap", + "AbortedByHuman" + ] + }, + "iteration_id": { + "type": "integer", + "minimum": 0 + }, + "criteria_met": { + "type": "object", + "additionalProperties": { + "type": "boolean" + }, + "propertyNames": { + "type": "string", + "minLength": 1, + "pattern": "\\S" + } + }, + "evidence": { + "type": "array", + "items": { + "type": "string", + "minLength": 1, + "pattern": "\\S" + }, + "uniqueItems": true + }, + "false_completion": { + "type": "boolean" + }, + "reason": { + "type": "string", + "minLength": 1 + }, + "lessons_ref": { + "type": [ + "string", + "null" + ] + }, + "completion_policy": { + "type": [ + "object", + "null" + ], + "required": [ + "mode" + ], + "properties": { + "mode": { + "const": "all_required" + } + }, + "additionalProperties": false, + "default": { + "mode": "all_required" + } + } }, "additionalProperties": true } diff --git a/scripts/runtime_monitor.py b/scripts/runtime_monitor.py index 80e5a44..ee97705 100644 --- a/scripts/runtime_monitor.py +++ b/scripts/runtime_monitor.py @@ -164,12 +164,23 @@ def _missing_report(paths, missing: list[str]) -> dict: } -def _terminal_disposition(state: dict) -> str | None: +def _terminal_disposition(paths, state: dict) -> str | None: """The loop's terminal state, if it has reached one. A finished loop must - not be told to `continue` — the in-flight detectors don't apply to it.""" + not be told to `continue` — the in-flight detectors don't apply to it. + An existing terminal_state.json is authoritative even when state.json was + never stamped (the writer's two files are not one transaction).""" terminal = state.get("terminal_state") if terminal: return terminal + terminal_path = paths.loop_dir / "terminal_state.json" + if terminal_path.is_file(): + try: + record = json.loads(terminal_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return "terminal" + if isinstance(record, dict) and record.get("state"): + return str(record["state"]) + return "terminal" if state.get("state") == "terminal": return "terminal" return None @@ -260,7 +271,7 @@ def health_report(loop_dir) -> dict: "paths": {"state": str(paths.state), "runlog": str(paths.runlog)}, } - terminal = _terminal_disposition(state) + terminal = _terminal_disposition(paths, state) if terminal is not None: return _terminal_report(paths, state, terminal) diff --git a/scripts/test_completion_policy.py b/scripts/test_completion_policy.py new file mode 100644 index 0000000..66b6513 --- /dev/null +++ b/scripts/test_completion_policy.py @@ -0,0 +1,118 @@ +"""Shared completion-policy semantics are deterministic and fail closed.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from loop.completion import ( + CompletionPolicyError, + criteria_satisfy_completion, + normalize_completion_policy, + unmet_required_criteria, +) +from loop.contract import _check_terminal_contradiction, _validate_state, _validate_terminal + + +def test_all_required_policy_requires_a_nonempty_all_true_map(): + assert criteria_satisfy_completion({"a": True, "b": True}) is True + assert criteria_satisfy_completion({"a": True, "b": False}) is False + assert criteria_satisfy_completion({}) is False + assert criteria_satisfy_completion({"a": 1}) is False + + +def test_policy_normalization_is_explicit_and_strict(): + assert normalize_completion_policy() == {"mode": "all_required"} + assert normalize_completion_policy("all_required") == {"mode": "all_required"} + assert normalize_completion_policy({"mode": "all_required"}) == {"mode": "all_required"} + with pytest.raises(CompletionPolicyError): + normalize_completion_policy({"mode": "any_required"}) + with pytest.raises(CompletionPolicyError): + normalize_completion_policy({"mode": "all_required", "threshold": 1}) + + +def test_unmet_criteria_are_stable_and_human_readable(): + assert unmet_required_criteria({"b": False, "a": True, "c": None}) == ("b", "c") + + +def test_contract_rejects_partial_success_claim(tmp_path: Path): + issues: list[dict] = [] + _check_terminal_contradiction( + { + "schema": "loop-engineer/terminal@1", + "state": "Succeeded", + "criteria_met": {"one": True, "two": False}, + "completion_policy": {"mode": "all_required"}, + "evidence": ["artifact.json"], + "false_completion": False, + }, + tmp_path / "terminal_state.json", + issues, + ) + assert any(issue["code"] == "contradictory_terminal" for issue in issues) + assert any("two" in issue["message"] for issue in issues) + + +def test_terminal_fallback_validation_rejects_bad_policy_and_evidence(tmp_path: Path): + issues: list[dict] = [] + _validate_terminal( + { + "schema": "loop-engineer/terminal@1", + "state": "FailedUnverifiable", + "criteria_met": {"one": False}, + "completion_policy": {"mode": "any_required"}, + "evidence": ["", "duplicate", "duplicate"], + "false_completion": False, + }, + tmp_path / "terminal_state.json", + issues, + ) + messages = "\n".join(issue["message"] for issue in issues) + assert "completion_policy" in messages + assert "non-empty strings" in messages + assert "unique" in messages + + +def test_state_fallback_validation_accepts_legacy_decimal_strings_only(tmp_path: Path): + base = { + "schema": "loop-engineer/state@1", + "state": "planned", + "plan_version": 1, + "budget_remaining": {}, + } + + for iteration_id in (0, 7, "0", "7"): + issues: list[dict] = [] + _validate_state({**base, "iteration_id": iteration_id}, tmp_path / "state.json", issues) + assert not any(issue["code"] == "invalid_state" for issue in issues) + + for iteration_id in (-1, True, "-1", "1.5", "next"): + issues = [] + _validate_state({**base, "iteration_id": iteration_id}, tmp_path / "state.json", issues) + assert any(issue["code"] == "invalid_state" for issue in issues) + +def test_explicit_null_policy_is_accepted_in_both_validation_modes(tmp_path: Path): + # An explicit "completion_policy": null must mean the same thing as an + # absent field in BOTH validation modes — a record must not be doctor-clean + # on a machine without the jsonschema extra and doctor-dirty on one with it. + record = { + "schema": "loop-engineer/terminal@1", + "state": "Succeeded", + "criteria_met": {"one": True}, + "completion_policy": None, + "evidence": ["artifact.json"], + "false_completion": False, + } + issues: list[dict] = [] + _validate_terminal(record, tmp_path / "terminal_state.json", issues) + assert issues == [] + + jsonschema = pytest.importorskip("jsonschema") + schema = json.loads( + (Path(__file__).resolve().parents[1] / "schemas" / "terminal.schema.json").read_text( + encoding="utf-8" + ) + ) + jsonschema.validate(record, schema) diff --git a/scripts/test_emit.py b/scripts/test_emit.py index 4af3882..df1e0c7 100644 --- a/scripts/test_emit.py +++ b/scripts/test_emit.py @@ -1,9 +1,12 @@ -"""B1 acceptance: emit writes schema-valid artifacts by construction and refuses -an evidence-free Succeeded at write time (G1 enforced before validate time).""" +"""B1 acceptance: emit writes schema-valid artifacts by construction and +refuses a Succeeded claim unless every required criterion has evidence-backed +proof (G1 enforced before validate time).""" from __future__ import annotations import json +import threading +from concurrent.futures import ThreadPoolExecutor from pathlib import Path import pytest @@ -43,7 +46,7 @@ def test_append_iteration_writes_parseable_runlog_and_updates_state(workspace): assert "`task_passed`" in text state = json.loads((workspace / ".loop" / "state.json").read_text(encoding="utf-8")) - assert state["iteration_id"] == "1" + assert state["iteration_id"] == 1 assert state["active_task"] == "T1" assert validate_contract(workspace)["ok"] is True @@ -71,12 +74,13 @@ def test_append_receipt_rejects_bad_role(workspace): def test_terminate_succeeded_with_evidence_passes_doctor(workspace): terminal = emit.terminate( - workspace, state="Succeeded", criteria_met={"1": True}, + workspace, state="Succeeded", criteria_met={"1": True, "2": True}, evidence=["artifact.txt"], reason="verified", iteration_id=1, ) data = json.loads(terminal.read_text(encoding="utf-8")) assert data["schema"] == "loop-engineer/terminal@1" assert data["false_completion"] is False + assert data["completion_policy"] == {"mode": "all_required"} state = json.loads((workspace / ".loop" / "state.json").read_text(encoding="utf-8")) assert state["terminal_state"] == "Succeeded" assert validate_contract(workspace)["ok"] is True @@ -87,6 +91,7 @@ def test_terminate_succeeded_with_evidence_passes_doctor(workspace): [ dict(criteria_met={"1": True}, evidence=[]), # evidence-free dict(criteria_met={"1": False}, evidence=["a.txt"]), # no met criterion + dict(criteria_met={"1": True, "2": False}, evidence=["a.txt"]), # partial proof dict(criteria_met={}, evidence=["a.txt"]), # empty criteria dict(criteria_met={"1": True}, evidence=["a.txt"], false_completion=True), # G1 contradiction ], @@ -124,6 +129,34 @@ def test_writes_refused_without_a_contract(tmp_path): emit.append_iteration(tmp_path / "nowhere", iteration_id=1, outcome="task_passed") +@pytest.mark.parametrize("iteration_id", [-1, True, "1"]) +def test_append_iteration_rejects_noncanonical_iteration_ids(workspace, iteration_id): + with pytest.raises(emit.EmitError, match="non-negative integer"): + emit.append_iteration(workspace, iteration_id=iteration_id, outcome="task_passed") + + +def test_terminate_rejects_unsupported_completion_policy(workspace): + with pytest.raises(emit.EmitError, match="unsupported completion policy"): + emit.terminate( + workspace, + state="Succeeded", + criteria_met={"1": True}, + evidence=["artifact.txt"], + completion_policy={"mode": "any_required"}, + ) + + +def test_terminate_rejects_duplicate_or_blank_evidence(workspace): + for evidence in (["a.txt", "a.txt"], [""]): + with pytest.raises(emit.EmitError): + emit.terminate( + workspace, + state="FailedUnverifiable", + criteria_met={"1": False}, + evidence=evidence, + ) + + def _loop_leftovers(workspace): return sorted(p.name for p in (workspace / ".loop").rglob("*.tmp")) @@ -141,30 +174,70 @@ def test_terminate_refuses_overwrite_of_existing_terminal(workspace): workspace, state="FailedBlocked", criteria_met={"1": False}, evidence=[], reason="second", ) - # names the written-once contract and the force escape hatch - assert "written once" in str(exc.value) - assert "force=True" in str(exc.value) + assert "immutable" in str(exc.value) # the refused call left the original terminal record byte-for-byte intact assert terminal_path.read_text(encoding="utf-8") == before assert not _loop_leftovers(workspace) -def test_terminate_force_overwrites(workspace): +def test_terminate_force_is_refused_and_preserves_original(workspace): emit.terminate( workspace, state="Succeeded", criteria_met={"1": True}, evidence=["artifact.txt"], reason="first", iteration_id=1, ) - emit.terminate( - workspace, state="FailedBlocked", criteria_met={"1": False}, - evidence=[], reason="deliberate override", force=True, - ) - data = json.loads((workspace / ".loop" / "terminal_state.json").read_text(encoding="utf-8")) - assert data["state"] == "FailedBlocked" + terminal_path = workspace / ".loop" / "terminal_state.json" + before = terminal_path.read_text(encoding="utf-8") + + with pytest.raises(emit.EmitError, match="immutable"): + emit.terminate( + workspace, state="FailedBlocked", criteria_met={"1": False}, + evidence=[], reason="deliberate override", force=True, + ) + + assert terminal_path.read_text(encoding="utf-8") == before state = json.loads((workspace / ".loop" / "state.json").read_text(encoding="utf-8")) - assert state["terminal_state"] == "FailedBlocked" + assert state["terminal_state"] == "Succeeded" assert not _loop_leftovers(workspace) + +def test_concurrent_terminators_create_exactly_one_terminal(workspace): + barrier = threading.Barrier(2) + + def attempt(state, criteria_met, evidence): + barrier.wait() + try: + emit.terminate( + workspace, + state=state, + criteria_met=criteria_met, + evidence=evidence, + reason=f"candidate {state}", + iteration_id=1, + ) + except emit.EmitError as exc: + return ("refused", str(exc)) + return ("created", state) + + with ThreadPoolExecutor(max_workers=2) as pool: + results = [ + pool.submit(attempt, "Succeeded", {"1": True}, ["artifact.txt"]), + pool.submit(attempt, "FailedBlocked", {"1": False}, []), + ] + outcomes = [future.result() for future in results] + + assert [kind for kind, _ in outcomes].count("created") == 1 + assert [kind for kind, _ in outcomes].count("refused") == 1 + assert "immutable" in next(message for kind, message in outcomes if kind == "refused") + + terminal_path = workspace / ".loop" / "terminal_state.json" + terminal = json.loads(terminal_path.read_text(encoding="utf-8")) + winner = next(value for kind, value in outcomes if kind == "created") + assert terminal["state"] == winner + state = json.loads((workspace / ".loop" / "state.json").read_text(encoding="utf-8")) + assert state["terminal_state"] == winner + assert not _loop_leftovers(workspace) + def test_terminate_leaves_no_tmp_litter_on_success(workspace): emit.terminate( workspace, state="Succeeded", criteria_met={"1": True}, @@ -185,3 +258,42 @@ def test_terminate_leaves_no_tmp_litter_on_invalid_terminate(workspace): def test_append_iteration_leaves_no_tmp_litter(workspace): emit.append_iteration(workspace, iteration_id=1, outcome="task_passed", task_id="T1") assert not _loop_leftovers(workspace) + + +def test_sync_state_to_terminal_reconciles_unstamped_state(workspace): + emit.terminate( + workspace, state="Succeeded", criteria_met={"1": True}, + evidence=["artifact.txt"], reason="ok", iteration_id=1, + ) + terminal_path = workspace / ".loop" / "terminal_state.json" + before = terminal_path.read_text(encoding="utf-8") + state_path = workspace / ".loop" / "state.json" + state = json.loads(state_path.read_text(encoding="utf-8")) + state["terminal_state"] = None + state_path.write_text(json.dumps(state, indent=2) + "\n", encoding="utf-8") + + synced = emit.sync_state_to_terminal(workspace) + + assert synced == state_path + assert json.loads(state_path.read_text(encoding="utf-8"))["terminal_state"] == "Succeeded" + assert terminal_path.read_text(encoding="utf-8") == before + assert not _loop_leftovers(workspace) + + +def test_sync_state_to_terminal_requires_a_terminal_record(workspace): + with pytest.raises(emit.EmitError, match="nothing to sync"): + emit.sync_state_to_terminal(workspace) + + +def test_terminate_wraps_link_failure_as_emit_error(workspace, monkeypatch): + def _refuse_link(src, dst): + raise PermissionError("hard links not supported") + + monkeypatch.setattr(emit.os, "link", _refuse_link) + with pytest.raises(emit.EmitError, match="terminal write failed"): + emit.terminate( + workspace, state="Succeeded", criteria_met={"1": True}, + evidence=["artifact.txt"], reason="ok", iteration_id=1, + ) + assert not (workspace / ".loop" / "terminal_state.json").exists() + assert not _loop_leftovers(workspace) diff --git a/scripts/test_integrations.py b/scripts/test_integrations.py index 0b25a19..d47fc3e 100644 --- a/scripts/test_integrations.py +++ b/scripts/test_integrations.py @@ -2,7 +2,7 @@ evidence to one of the 7 typed terminal states. Pins the fixed precedence (safety -> human -> blocked -> budget -> spec-gap -> gate verdict), the false-completion invariant, and the structural unreachability of Succeeded -without a green gate + clean anticheat + a met criterion + evidence.""" +without a green gate + clean anticheat + every required criterion + evidence.""" from __future__ import annotations @@ -54,9 +54,59 @@ def test_succeeded_via_green_gate_clean_anticheat_met_criterion(): assert body["state"] == "Succeeded" assert body["false_completion"] is False assert body["schema"] == "loop-engineer/terminal@1" + assert body["completion_policy"] == {"mode": "all_required"} assert body["evidence"] == ["a.txt"] +def test_partial_criteria_cannot_succeed_even_with_green_gate(): + body = to_terminal_state( + EngineOutcome(**_ENDED), + _gate(True, True), + _CLEAN_AC, + {"1": True, "2": False}, + ) + assert body["state"] == "FailedUnverifiable" + assert "2" in body["reason"] + + +def test_unsupported_completion_policy_fails_as_spec_gap(): + body = to_terminal_state( + EngineOutcome(**_ENDED), + _gate(True, True), + _CLEAN_AC, + {"1": True}, + completion_policy={"mode": "any_required"}, + ) + assert body["state"] == "FailedSpecGap" + assert "unsupported completion policy" in body["reason"] + + +def test_malformed_criteria_map_fails_as_spec_gap(): + blank_key = to_terminal_state( + EngineOutcome(**_ENDED), _gate(True, True), _CLEAN_AC, {"": True}, + ) + assert blank_key["state"] == "FailedSpecGap" + assert "criteria identifiers" in blank_key["reason"] + + bad_value = to_terminal_state( + EngineOutcome(**_ENDED), _gate(True, True), _CLEAN_AC, {"1": 1.0}, + ) + assert bad_value["state"] == "FailedSpecGap" + assert "true, false, or null" in bad_value["reason"] + + +def test_blank_or_duplicate_artifacts_cannot_certify_success(): + for artifacts in ([""], ["a.txt", "a.txt"]): + body = to_terminal_state( + EngineOutcome(reached_end=True, artifacts=artifacts), + _gate(True, True), + _CLEAN_AC, + {"1": True}, + ) + assert body["state"] == "FailedUnverifiable" + assert "invalid evidence artifacts" in body["reason"] + + def test_false_completion_invariant_visible_green_holdout_red(): gate = _gate(True, False) assert gate["false_completion"] is True # the real decide() flag @@ -155,7 +205,8 @@ def test_body_feeds_emit_terminate_round_trip(tmp_path): body = to_terminal_state(EngineOutcome(**_ENDED), _gate(True, True), _CLEAN_AC, {"1": True}) path = emit.terminate( ws, state=body["state"], criteria_met=body["criteria_met"], evidence=body["evidence"], - false_completion=body["false_completion"], reason=body["reason"], iteration_id=1, + false_completion=body["false_completion"], completion_policy=body["completion_policy"], + reason=body["reason"], iteration_id=1, ) assert path.is_file() @@ -165,6 +216,12 @@ def test_module_imports_no_engine_and_no_scripts(): imports = [l for l in source.splitlines() if re.match(r"\s*(import|from)\s", l)] for line in imports: assert "langgraph" not in line and "temporalio" not in line and "scripts" not in line, line - # pure stdlib: the only allowed import roots + # pure stdlib plus the shared, side-effect-free policy evaluator + allowed_prefixes = ( + "from __future__ import", + "from dataclasses import", + "from typing import", + "from .completion import", + ) for line in imports: - assert re.match(r"\s*(from\s+(__future__|dataclasses|typing)\s+import|import\s+(dataclasses|typing))", line), line + assert line.lstrip().startswith(allowed_prefixes), line diff --git a/scripts/test_runtime_monitor.py b/scripts/test_runtime_monitor.py index 605163d..ea00bf5 100644 --- a/scripts/test_runtime_monitor.py +++ b/scripts/test_runtime_monitor.py @@ -416,3 +416,21 @@ def test_cli_exit_zero_when_terminal(tmp_path): # A finished loop is a clean exit, not an intervention. assert runtime_monitor.main([str(loop_dir)]) == 0 + + +def test_orphan_terminal_record_is_reported_done(tmp_path): + # A crash between the immutable terminal_state.json creation and the + # state.json stamp must not fool the monitor into recommending more work: + # the terminal record itself is authoritative. + state = {"active_task": "T3", "state": "execute", "iteration_id": 3} + runlog = _runlog([(1, "T3", 0.5), (2, "T3", 0.5), (3, "T3", 0.5)]) + loop_dir = _write_loop(tmp_path, state, runlog) + (loop_dir / "terminal_state.json").write_text( + json.dumps({"schema": "loop-engineer/terminal@1", "state": "Succeeded"}), + encoding="utf-8", + ) + + report = runtime_monitor.health_report(loop_dir) + + assert report["recommendation"] == "done" + assert report.get("terminal_state") == "Succeeded" diff --git a/templates/state.json.tmpl b/templates/state.json.tmpl index e0ed087..6c1a6fb 100644 --- a/templates/state.json.tmpl +++ b/templates/state.json.tmpl @@ -1,7 +1,7 @@ { "schema": "loop-engineer/state@1", "project": "{{PROJECT_NAME}}", - "iteration_id": "{{ITERATION_ID}}", + "iteration_id": {{ITERATION_ID}}, "plan_version": {{PLAN_VERSION}}, "active_task": "{{ACTIVE_TASK_ID}}", "state": "{{STATE}}",