feat(metrics): ST1 — loop metrics + adversarially-hardened FCR/RP baseline

README.md

@@ -72,8 +72,8 @@ opinions.
 What this suite owns:
 
 - **7 typed terminal states** — a contract primitive, so no run ends in a silent "completed."
-- **`false-completion-rate`** — measurable with the bundled held-out gate and anti-cheat scan (computed from real runs; no baseline ships yet).
-- **`repair-productivity`** — the fraction of repair attempts that measurably move verification forward.
+- **`false-completion-rate`** — measurable with the bundled held-out gate and anti-cheat scan; **0.0** on the shipped gate-backed example (see [Measured baseline](#measured-baseline)).
+- **`repair-productivity`** — the fraction of repair attempts that measurably move verification forward; **1.0** on that example.
 - **Repo-native loop state** — survives compaction, crashes, and handoff.
 - **Deterministic-gate-before-rubric ordering** — model judges are advisory, not the first line of proof.
 
@@ -83,6 +83,33 @@ Engineer's claim is the proof-of-done framing plus the typed termination and
 loop-health metrics on top. It composes with those tools; it does not replace
 their execution engines.
 
+### Measured baseline
+
+The two metrics are **derived by a tool, not quoted from prose.** The checked-in
+scorecard [`docs/metrics-baseline.json`](docs/metrics-baseline.json) is computed
+by `python3 -m loop metrics` over the gate-backed `examples/coverage-repair` run —
+its `false_completion:false` is backed by a real `holdout_gate.py` verdict, and
+its `productive` flag is recomputed from the repair record's own score delta, not
+trusted:
+
+| Metric | Baseline | Source |
+|---|---:|---|
+| `false-completion-rate` | **0.0** | RUNLOG success-claims × verify bundles, cross-checked against the held-out gate flag (both agree) |
+| `repair-productivity` | **1.0** | one repair pass, `verification_after.score` 0.83 > `before` 0.74 (recomputed, agreed) |
+
+The number ships with a `provenance` block naming every input file (including the
+held-out verdict's sha256), so a skeptic can re-derive it. That committed verdict
+is *evidence, not proof*: it is validated structurally, but a fully-fabricated,
+internally-consistent artifact defeats offline shape-checking by construction —
+tamper detection of the artifact itself belongs to the anti-cheat layer; the tool
+does not claim the verdict is tamper-proof. Reproduce (and refuse to publish over a
+non-gate-backed, inconsistent, vacuous, or unanchored run):
+
+```bash
+python3 -m loop metrics examples/coverage-repair            # print the scorecard
+python3 -m loop metrics --baseline examples/coverage-repair # rewrite docs/metrics-baseline.json
+```
+
 ---
 
 ## Proof-of-done, not self-assertion

docs/metrics-baseline.json

@@ -0,0 +1,52 @@
+{
+  "schema": "loop-engineer/metrics@1",
+  "loop": "examples/coverage-repair",
+  "false_completion_rate": 0.0,
+  "repair_productivity": 1.0,
+  "iterations_claiming_success": 2,
+  "false_completions": 0,
+  "repair_passes": 1,
+  "productive_repairs": 1,
+  "cost_per_success_usd": null,
+  "evidence_backed": true,
+  "provenance": {
+    "fcr_source": [
+      ".loop/artifacts/verify-T1.json",
+      ".loop/artifacts/verify-T2-iter1.json",
+      ".loop/artifacts/verify-T2.json",
+      "RUNLOG.md"
+    ],
+    "rp_source": [
+      ".loop/repair/iter-002.json"
+    ],
+    "rejected_records": [],
+    "unanchored_records": [],
+    "unrecognized_outcomes": [],
+    "false_completion_rate_holdout": 0.0,
+    "fcr_methods_agree": true,
+    "holdout_source": [
+      ".loop/artifacts/holdout-verdict.json"
+    ],
+    "holdout_verdicts": [
+      {
+        "source": ".loop/artifacts/holdout-verdict.json",
+        "sha256": "b203cf5bb6b05f15172209e067d9f4fa0aaf60cada33fe98172a8279051c1000"
+      }
+    ],
+    "unmatched_verify": []
+  },
+  "baseline": {
+    "source_example": "examples/coverage-repair",
+    "commit": "2344314b5f0ce06c121baf51c4f17688186d42fb",
+    "inputs": [
+      "examples/coverage-repair/.loop/artifacts/holdout-verdict.json",
+      "examples/coverage-repair/.loop/artifacts/verify-T1.json",
+      "examples/coverage-repair/.loop/artifacts/verify-T2-iter1.json",
+      "examples/coverage-repair/.loop/artifacts/verify-T2.json",
+      "examples/coverage-repair/.loop/repair/iter-002.json",
+      "examples/coverage-repair/RUNLOG.md",
+      "examples/coverage-repair/TASKS.json",
+      "examples/coverage-repair/terminal_state.json"
+    ]
+  }
+}

examples/coverage-repair/.loop/manifest.yaml

@@ -24,7 +24,7 @@ outputs:
   task_queue: TASKS.json
   current_state: .loop/state.json
   verification_bundle: .loop/artifacts/
-  repair_actions: .loop/artifacts/repair-record.json
+  repair_actions: .loop/repair/iter-002.json
   terminal_state: .loop/terminal_state.json
   lessons_learned: .loop/memory/lessons.md
 

examples/coverage-repair/ADR.md

@@ -74,7 +74,7 @@ still gives resumable, evidence-backed verification. We do **not** reach for mul
   coverage), each with its own `verify` gate; progress is the count of `done` tasks with non-null
   `evidence`.
 - **Patch-and-repair:** on a red gate, hand to `[[loop-repair]]` for one bounded hypothesis →
-  one change → re-verify, capped at N=2 (see `repair-record.json`).
+  one change → re-verify, capped at N=2 (see `.loop/repair/iter-002.json`).
 
 ## Hand-off
 

examples/coverage-repair/README.md

@@ -29,7 +29,7 @@ Succeeded`.
 | `TASKS.json` | `[[loop-contract]]` → updated by `[[loop-run]]` | The **machine-readable queue** — `T1` (validation) and `T2` (coverage), each with its `verify` command, `criterion_ref`, `attempts`, and `evidence`. Both end `done`. |
 | `RUNLOG.md` | `[[loop-run]]` (+ repair blocks from `[[loop-repair]]`) | The **append-only history** — two dated iterations: iteration 1 verify **FAIL** → repair → iteration 2 verify **PASS** → terminal. |
 | `.loop/state.json` | `[[loop-run]]` | The **live FSM cursor** — serialized after every transition; here at the terminal snapshot (`state: terminal`, `best_score: 0.83`, `terminal_state: "Succeeded"`). This is what makes the loop resumable across sessions. |
-| `repair-record.json` | `[[loop-repair]]` | One **structured repair record** — `failure_mode`, `hypothesis`, `repair_action`, `verification_before`, `verification_after`, `remaining_delta` (+ `productive: true`). The verification delta proves the repair moved the score (`repair-productivity`). |
+| `.loop/repair/iter-002.json` | `[[loop-repair]]` | One **structured repair record** at its canonical path (`loop-engineer/repair@1`) — `failure_mode`, `hypothesis`, `repair_action`, `verification_before`, `verification_after`, `remaining_delta` (+ `productive: true`). The verification delta proves the repair moved the score, and is the record `python3 -m loop metrics` reads for `repair-productivity`. |
 | `terminal_state.json` | `[[loop-run]]` | The **single end record** — `state == "Succeeded"`, `criteria_met` both true, `evidence` paths, and `false_completion: false`. No silent "completed." |
 
 > The `scripts/verify-*` gates and `EVALS/` rubrics referenced here are designed by `[[loop-evals]]`
@@ -76,7 +76,7 @@ the scenario:
 
 ```bash
 # All JSON in this example parses:
-for f in TASKS.json .loop/state.json repair-record.json terminal_state.json; do
+for f in TASKS.json .loop/state.json .loop/repair/iter-002.json terminal_state.json; do
   uv run --with pyyaml python3 -c "import json,sys; json.load(open('$f')); print('ok:', '$f')"
 done
 

examples/coverage-repair/RUNLOG.md

@@ -39,7 +39,7 @@ Then advanced to `T2` (coverage) in the same iteration's verify step:
   exist. **No** test-harness assertion edits or fixture relaxations to manufacture the pass.
 - **attempt:** 1 of 2
 - **measurable improvement:** pending re-verify (see Iteration 2) — full record in
-  `repair-record.json`
+  `.loop/repair/iter-002.json`
 
 ---
 
@@ -61,7 +61,7 @@ Then advanced to `T2` (coverage) in the same iteration's verify step:
 - **repair action:** the two added production branches lifted coverage past the gate.
 - **attempt:** 1 of 2 (cap not reached — one productive pass)
 - **measurable improvement:** YES — `verification_after.score` 0.83 > `verification_before.score`
-  0.74 → `productive: true`. See `repair-record.json`.
+  0.74 → `productive: true`. See `.loop/repair/iter-002.json`.
 
 ### Terminal
 

examples/coverage-repair/WORKFLOW.md

@@ -42,7 +42,7 @@ same `.loop/state.json` checkpoint; they never spawn a fresh untracked attempt.
 - **Max repair attempts per task:** `2` (default).
 - After exceeding the cap: replan / revert / approve / terminate — never silently retry.
 - Each repair attempt produces a structured repair record (see `[[loop-repair]]` and
-  `repair-record.json` in this example).
+  `.loop/repair/iter-002.json` in this example).
 - A repair that does not measurably improve the score is churn → replan.
 - Detected verifier-gaming → hard-terminate `FailedSafety` immediately.
 

loop/__main__.py

@@ -9,17 +9,18 @@
 
 _PROG = "python3 -m loop"
 
-_COMMANDS = ("scaffold", "doctor", "validate", "verify", "inspect")
+_COMMANDS = ("scaffold", "doctor", "validate", "verify", "inspect", "metrics")
 
 # Read commands operate on an EXISTING contract dir; scaffold CREATES one, so it
 # is exempt from the "target must exist" guard.
-_READ_COMMANDS = ("doctor", "validate", "verify", "inspect")
+_READ_COMMANDS = ("doctor", "validate", "verify", "inspect", "metrics")
 
-_USAGE = f"usage: {_PROG} <scaffold|doctor|validate|verify|inspect> <workspace-or-.loop>"
+_USAGE = f"usage: {_PROG} <scaffold|doctor|validate|verify|inspect|metrics> <workspace-or-.loop>"
 
-_HELP = f"""{_PROG} — validate and inspect a portable repo-OS loop contract.
+_HELP = f"""{_PROG} — validate, inspect, and measure a portable repo-OS loop contract.
 
 {_USAGE}
+       {_PROG} metrics [--baseline] <workspace-or-.loop>
 
 commands:
   scaffold   Write a fresh, doctor-clean loop contract into <target>.
@@ -28,11 +29,17 @@
   verify     Alias for doctor — check the contract's state.
   inspect    Score an existing loop against the prime-directive checklist
              (emits a weak/strong verdict and a gap report).
+  metrics    Derive false-completion-rate + repair-productivity from the loop's
+             real .loop/ evidence (RUNLOG, verify bundles, held-out gate, repair
+             records) and emit a JSON scorecard. With --baseline, write a
+             checked-in baseline scorecard — refused unless the run is gate-backed.
 
 arguments:
   <target>   A workspace root or its .loop/ directory.
 
 options:
+  --baseline    (metrics only) write docs/metrics-baseline.json over a gate-backed
+                run; exits non-zero and writes nothing otherwise.
   -h, --help    Show this help and exit.
   --version     Show the version and exit.
 """
@@ -68,6 +75,34 @@ def _print_json(report: dict) -> int:
     return 0 if report.get("ok") else 1
 
 
+def _run_metrics(argv: list[str]) -> int:
+    """`metrics [--baseline] <target>` — parses its own flag, then delegates to
+    scripts/metrics.py (imported repo-relative, the QW8 editable-install path)."""
+    unknown = [a for a in argv if a.startswith("-") and a != "--baseline"]
+    if unknown:
+        print(f"metrics: unknown option: {unknown[0]}", file=sys.stderr)
+        print(_USAGE, file=sys.stderr)
+        return 2
+    positional = [a for a in argv if not a.startswith("-")]
+    if not positional:
+        print("metrics: missing target argument", file=sys.stderr)
+        print(_USAGE, file=sys.stderr)
+        return 2
+    target = Path(positional[0])
+    if not target.exists():
+        print(
+            f"metrics: target path does not exist: {target}\n"
+            f"       pass an existing loop workspace or its .loop/ directory.",
+            file=sys.stderr,
+        )
+        return 2
+    scripts_dir = Path(__file__).resolve().parent.parent / "scripts"
+    sys.path.insert(0, str(scripts_dir))
+    import metrics  # type: ignore
+
+    return metrics.run(argv)
+
+
 def main(argv: list[str] | None = None) -> int:
     argv = list(sys.argv[1:] if argv is None else argv)
 
@@ -87,6 +122,11 @@ def main(argv: list[str] | None = None) -> int:
         print(_USAGE, file=sys.stderr)
         return 2
 
+    # metrics carries its own optional --baseline flag, so it parses its own args
+    # before the generic single-target guards below.
+    if command == "metrics":
+        return _run_metrics(argv)
+
     if not argv:
         print(f"{command}: missing target argument", file=sys.stderr)
         print(_USAGE, file=sys.stderr)

loop/contract.py

@@ -290,6 +290,98 @@ def _jsonschema_validate(data: dict[str, Any], name: str, path: Path, issues: li
         issues.append(ContractIssue("schema_violation", f"{path.name}: {location}: {err.message}", path))
 
 
+# The FCR/RP evidentiary trail (M5): repair records and receipt/rollout ledgers.
+# Validated OPTIONALLY — only when the files exist — so an in-flight loop that has
+# not emitted them yet still passes, while a loop that ships malformed metric
+# inputs can no longer pass validation with them unchecked.
+_RECORD_SCHEMA_FILES = {
+    "repair": "repair-record.schema.json",
+    "rollout": "rollout-record.schema.json",
+    "receipt": "receipt.schema.json",
+}
+
+# The $id each record schema publishes, reported under schemas_checked when the
+# corresponding record files were present and validated (deterministic order).
+_RECORD_SCHEMA_IDS = (
+    ("repair", "loop-engineer/repair@1"),
+    ("rollout", "loop-engineer/rollout@1"),
+    ("receipt", "loop-engineer/receipt@1"),
+)
+
+
+def _load_schema_file(filename: str) -> dict[str, Any]:
+    return json.loads((_schemas_dir() / filename).read_text(encoding="utf-8"))
+
+
+def _structural_record_check(data: dict[str, Any], schema: dict[str, Any], path: Path, issues: list[dict]) -> None:
+    props = schema.get("properties", {})
+    for key in schema.get("required", []):
+        if key not in data:
+            issues.append(ContractIssue("invalid_record", f"{path.name}: missing {key}", path))
+    for key, sub in props.items():
+        if key in data and isinstance(sub, dict) and "const" in sub and data[key] != sub["const"]:
+            issues.append(ContractIssue("schema_mismatch", f"{path.name}: {key} != {sub['const']!r}", path))
+    for vk in ("verification_before", "verification_after"):
+        sub = props.get(vk)
+        if isinstance(sub, dict) and "score" in sub.get("required", []) and vk in data:
+            value = data[vk]
+            score = value.get("score") if isinstance(value, dict) else None
+            if isinstance(score, bool) or not isinstance(score, (int, float)):
+                issues.append(ContractIssue("invalid_record", f"{path.name}: {vk}.score must be numeric", path))
+
+
+def _validate_record(data: dict[str, Any], schema_key: str, path: Path, mode: str, issues: list[dict]) -> None:
+    filename = _RECORD_SCHEMA_FILES[schema_key]
+    if mode == "jsonschema":
+        import jsonschema  # type: ignore
+
+        validator = jsonschema.Draft202012Validator(_load_schema_file(filename))
+        for err in validator.iter_errors(data):
+            location = "/".join(str(p) for p in err.absolute_path) or "<root>"
+            issues.append(ContractIssue("schema_violation", f"{path.name}: {location}: {err.message}", path))
+    else:
+        _structural_record_check(data, _load_schema_file(filename), path, issues)
+
+
+def _validate_jsonl(path: Path, schema_key: str, mode: str, issues: list[dict]) -> None:
+    for lineno, line in enumerate(path.read_text(encoding="utf-8", errors="ignore").splitlines(), start=1):
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            data = json.loads(line)
+        except json.JSONDecodeError as exc:
+            issues.append(ContractIssue("invalid_json", f"{path.name}:{lineno}: {exc}", path))
+            continue
+        if not isinstance(data, dict):
+            issues.append(ContractIssue("invalid_record", f"{path.name}:{lineno}: expected object", path))
+            continue
+        _validate_record(data, schema_key, path, mode, issues)
+
+
+def _validate_optional_records(paths: LoopPaths, mode: str, issues: list[dict]) -> set[str]:
+    """Validate record files that are present; return the set of record schema
+    keys actually checked (``repair``/``rollout``/``receipt``) so ``doctor`` can
+    report them under ``schemas_checked`` instead of under-counting its coverage."""
+    checked: set[str] = set()
+    repair_dir = paths.loop_dir / "repair"
+    if repair_dir.is_dir():
+        for record_path in sorted(repair_dir.glob("*.json")):
+            data = _read_json(record_path, issues)
+            if data is not None:
+                _validate_record(data, "repair", record_path, mode, issues)
+                checked.add("repair")
+    for ledger_path in sorted(paths.loop_dir.glob("*.jsonl")):
+        _validate_jsonl(ledger_path, "rollout", mode, issues)
+        checked.add("rollout")
+    receipts_dir = paths.loop_dir / "receipts"
+    if receipts_dir.is_dir():
+        for receipt_path in sorted(receipts_dir.glob("*.jsonl")):
+            _validate_jsonl(receipt_path, "receipt", mode, issues)
+            checked.add("receipt")
+    return checked
+
+
 def validate_contract(target: str | Path) -> dict[str, Any]:
     paths = resolve_loop_paths(target)
     issues: list[dict] = []
@@ -328,12 +420,17 @@ def validate_contract(target: str | Path) -> dict[str, Any]:
     if not paths.runlog.exists():
         issues.append(ContractIssue("missing_file", "missing RUNLOG.md", paths.runlog))
     _check_stub_verify_scripts(paths, issues)
+    records_checked = _validate_optional_records(paths, mode, issues)
+
+    schemas_checked = list(SCHEMA_IDS) + [
+        schema_id for key, schema_id in _RECORD_SCHEMA_IDS if key in records_checked
+    ]
 
     return {
         "ok": not issues,
         "paths": paths.to_json(),
         "validation_mode": mode,
-        "schemas_checked": list(SCHEMA_IDS),
+        "schemas_checked": schemas_checked,
         "issues": issues,
     }
 

pyproject.toml

@@ -22,6 +22,14 @@ keywords = ["agent", "loop", "agentic", "verification", "harness", "orchestratio
 yaml = ["pyyaml>=6"]
 schemas = ["jsonschema>=4"]
 
+# Console entry point. `loop.__main__:main` resolves the bundled scripts/ dir
+# relative to its own __file__, so this is EDITABLE-INSTALL ONLY: `pip install -e .`
+# keeps loop/ pointing at the repo (where scripts/ lives). A non-editable wheel
+# does not ship scripts/, so `inspect`/`metrics` would not resolve — see the
+# [tool.hatch.build.targets.wheel] note below.
+[project.scripts]
+loop = "loop.__main__:main"
+
 [project.urls]
 Homepage = "https://github.com/SollanSystems/loop-engineer"
 Repository = "https://github.com/SollanSystems/loop-engineer"

reference/eval-suite.md

@@ -56,6 +56,7 @@ RP = (repair passes where verification_after > verification_before)
 ```
 
 - "Improvement" is read straight off the structured repair-record schema [[loop-repair]] emits: `verification_before` vs `verification_after` on the same metric (a layer-1 pass count, a layer-2 rubric dimension, or a closed `remaining_delta`). A pass that leaves `remaining_delta` unchanged is churn.
+- **The canonical repair record is RP's only input.** RP reads *the repair record* — `schemas/repair-record.schema.json` (`loop-engineer/repair@1`), on disk at `.loop/repair/<iteration_id>.json`, the `verification_before`/`verification_after.score` pass [[loop-repair]] emits. The append-only **rollout / candidate ledger** (`schemas/rollout-record.schema.json`, `loop-engineer/rollout@1`, `scripts/rollout_ledger.py`) is a *separate* artifact that adjudicates rollout candidates; its `productive` is a rollout-productivity flywheel view, **not** the RP baseline. Neither is trusted verbatim: `scripts/metrics.py` recomputes `productive` (`recheck_productive`) from the record's own evidence and rejects any record whose stored flag disagrees, so RP is aggregated only over validated records.
 - **Target: high and trending up.** Low RP means the repair loop is thrashing — it is hitting its max-N cap (default N=2, per `WORKFLOW.md`) and burning budget without converging. Low RP is the data-driven trigger for the escalation ladder's "same failure mode repeats without measurable improvement → re-plan" rung (see `reference/safety-and-approvals.md`).
 - RP and the repair max-N cap are complementary: the cap bounds *how many* repairs run; RP measures whether those repairs were *worth* running. A loop with a healthy cap but RP≈0 is still broken.