diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 92341b8..cc6cea8 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "loop-engineer", - "version": "0.3.4", + "version": "0.5.0", "description": "Design, launch, verify, repair, and improve agent loops. A Claude-Code-native architect+operator for long-running, verifiable, self-improving agentic-coding systems.", "author": { "name": "Sollan Systems", "url": "https://github.com/SollanSystems" }, "homepage": "https://github.com/SollanSystems/loop-engineer", diff --git a/CHANGELOG.md b/CHANGELOG.md index 22b32bc..4d81626 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,58 @@ All notable changes to `loop-engineer` are documented here. `WORKFLOW.md` and `README.md` are reworded to describe the mechanism; the 0.3.4 history is left intact. +## 0.5.0 — 2026-07-03 + +The two pre-launch milestones of the v1.0 roadmap landed together: **"enforce the +wedge"** (false-completion defense is now enforced by validators, not asserted by +docs) and **"first screen"** (the README/demo surface rebuilt for a stranger's +first 30 seconds). PRs #7–#13. The version jumps 0.3.4 → 0.5.0 to match the +roadmap's milestone numbering (`docs/superpowers/plans/2026-06-30-loop-engineer-v1.0-roadmap.md`); +there is no 0.4.x tag. + +### Added +- **Gate-backed flagship example.** `examples/coverage-repair` now runs + end-to-end through the real held-out gate; its `false_completion: false` is + backed by a committed gate verdict (`.loop/artifacts/holdout-verdict.json`), + not a hand-set flag (#9). +- **Weak→strong demo, filmed live.** `docs/demo.gif` + `docs/demo.cast`: the + inspector scores a self-asserted DIY loop (committed as `examples/naive-loop`) + 0/weak, then the gate-backed example 90/strong — 100% live tool output. + Social card at `docs/social-card.png` (#13). +- **`loop scaffold`** command + JSON Schemas for the contract artifacts + (`schemas/*.schema.json`), with templates reconciled to what the validator + actually checks (#8). +- **Promised templates shipped:** `templates/verify-safety.sh`, + `templates/extract-trace-metrics.sh`, `templates/judge-rubric.sh`; central + model-routing doctrine at `reference/model-routing.md` (#11). +- **v1.0 master roadmap + four strategic specs** committed under + `docs/superpowers/` — credibility enforcement, ST1 metrics baseline, ST2 + portable contract spec, ST3 integration adapters (this release). + +### Changed +- **Validator cross-checks.** A `Succeeded` terminal no longer validates with + `false_completion: true` or an empty/false `criteria_met`; the inspector + grades false-completion defense on *invocation evidence* (the gate/scan + actually ran), never on a self-asserted flag (#8). +- **Held-out gate + scanner hardening.** An empty visible set can no longer + certify (`test_empty_visible_set_returns_not_ready`); the anti-cheat scanner + detects edits that neuter its own gate-decision functions + (`test_self_neuter_of_gate_matcher_is_detected`) and reports gate tampering + with a distinct exit code (#8). +- **README first screen** rebuilt for launch conversion: tagline, concrete + failure modes, zero-install first command, stack diagram, comparison table, + demo embed (#7, #12). +- **Skill trigger surface:** diagnostic spokes (loop-inspector, + loop-runtime-monitor) named at the router and marketplace, trigger-phrase + batch, path anchoring and neutral framing across all 9 skills (#11). + +### Fixed +- **CLI:** `--help`/`--version`/usage text, distinct operational-error messages, + explicit exit codes, ledger tolerance for foreign receipt lines (#10). +- **The repo's own live contract passes its own gate:** `python3 -m loop doctor + .loop` → `ok: true` — the release-blocking exit criterion of the + wedge-enforcement milestone (#8). + ## 0.3.4 — 2026-06-29 Dogfood-driven hardening: ran `loop-inspector` + `loop-runtime-monitor` against 9 real diff --git a/README.md b/README.md index d370154..96d672c 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ [![CI](https://github.com/SollanSystems/loop-engineer/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/SollanSystems/loop-engineer/actions/workflows/ci.yml) [![Python 3.10–3.12](https://img.shields.io/badge/python-3.10%E2%80%933.12-blue)](https://www.python.org/downloads/) [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE) -[![Release](https://img.shields.io/badge/release-0.3.4-blue)](https://github.com/SollanSystems/loop-engineer/tags) +[![Release](https://img.shields.io/badge/release-0.5.0-blue)](https://github.com/SollanSystems/loop-engineer/tags) Long-running agents commit **false completion**. After context compaction they forget what "done" meant, optimize to the visible test, patch in circles, and @@ -335,8 +335,8 @@ license, and README differentiation. ## Status -- Version: `0.3.4` -- Release tag: `loop-engineer--v0.3.4` (cut at publish) +- Version: `0.5.0` +- Release tag: `loop-engineer--v0.5.0` - License: MIT - Primary interface: Claude Code plugin - Portable core: Python CLI + JSON schemas diff --git a/docs/superpowers/plans/2026-06-30-loop-engineer-v1.0-roadmap.md b/docs/superpowers/plans/2026-06-30-loop-engineer-v1.0-roadmap.md new file mode 100644 index 0000000..8214500 --- /dev/null +++ b/docs/superpowers/plans/2026-06-30-loop-engineer-v1.0-roadmap.md @@ -0,0 +1,444 @@ +# Loop Engineer — Master Roadmap: v0.3.4 → v1.0 + +*Plan/docs deliverable. Authored 2026-06-30. Source of truth: `review/REVIEW.md`, +`review/IMPROVEMENT-BACKLOG.md`, `review/POSITIONING.md`, `review/COMPETITIVE-ANALYSIS.md`. +Nothing here is invented; every task cites a finding ID (G/M/T/O), a backlog item ID +(QW/HI/ST), or is marked `[NEW — beyond review]` with a one-line justification.* + +This roadmap is itself dogfooded: `roadmap/v1.0/` is a **validated loop-engineer +contract** (`python3 -m loop doctor roadmap/v1.0` → `ok: true`, 0 issues) that models +this planning milestone as an honest `Succeeded` — see §7. The four strategic design +specs it references (§5) carry the detailed mechanism; this file is the arc, the +sequencing law, and the finding→milestone ledger. + +--- + +## 1. Definition of best — what v1.0 means + +The wedge (verified in `COMPETITIVE-ANALYSIS.md` §Whitespace) is that Loop Engineer +owns an essentially empty quadrant: a **portable, typed-termination operating contract +with evidence-before-completion and first-class false-completion defense**, sitting +*above* the execution runtimes. v1.0 is reached when that wedge is not merely narrated +but **true under a skeptic's own tooling**. Four north-star exit criteria: + +**N1 — The differentiator is ENFORCED, not self-asserted.** +The single theme of the three HIGH findings (G1, M1, M2) is that false-completion +defense is taken on trust in load-bearing paths. v1.0 requires that: +- No `Succeeded` terminal validates while `false_completion:true` or `criteria_met` + is empty (closes **G1/QW2**). +- Inspector false-completion-defense credit is granted only on evidence a held-out / + anti-cheat gate actually ran (closes **M1/HI4**). +- The shipped flagship example actually runs the held-out gate end-to-end (closes + **M2/HI1**). +- The anti-cheat scanner cannot be silently defanged by rewriting its own gate + functions (closes **G3/HI6**); an empty visible set cannot certify (closes + **G2/QW6**). +- **Verify (whole-suite gate):** `uv run --with pytest --with pyyaml python3 -B -m pytest tests scripts -q` + green with the new regression tests, and `python3 -m loop doctor .loop` → `ok:true` + on the repo's own live contract. + +**N2 — The adoption surface is complete.** +A stranger gets it in 10 seconds and reaches a visible score in 30 with zero install. +Requires the README hero (**QW1**), a weak→strong demo asset (**O2/HI2**), the +three-tier stack + comparison table (**HI3**), a runnable flagship example (**O1/HI1**), +the diagnostic spokes exposed at the router (**T1/QW3**), and the trigger-phrase and +onboarding-friction fixes (**T2-T4/QW9, O3/QW7, O4/QW8, QW4, QW5**). +- **Verify:** `python3 scripts/validate_frontmatter.py` and `python3 scripts/self_eval.py` + green; `test -f docs/demo.gif`; `python3 -m loop inspect examples/coverage-repair` + runs clean with no install on a fresh clone. + +**N3 — The metrics are real.** +False-completion-rate (FCR) and repair-productivity (RP) are the primitives +`COMPETITIVE-ANALYSIS.md` names as having no competitor; their provenance must be +airtight. Requires one canonical repair record (**M4/QW11**), a recomputed-not-trusted +`productive` flag (**M3/HI5**), and a published baseline derived from a *gate-backed* +run, never a self-asserted one (**ST1**). +- **Verify:** `python3 scripts/self_eval.py` green; the `metrics` command's determinism + covered by a pytest case; README numbers traceable to a computation over real + `.loop/` receipts, not prose. + +**N4 — The contract is a portable standard.** +A versioned, tool-agnostic spec + published schemas that other harnesses can emit and +consume, closing the last integrity gap (receipts/repair records unvalidated — **M5/ST2**) +and turning every Q3/Q2 engine into a complement via integration recipes (**ST3**). +- **Verify:** `python3 -m loop doctor roadmap/v1.0` → `ok:true` in CI; the schema + conformance checklist passes; ≥2 integration recipes each produce a valid on-disk + contract that `validate_contract` accepts. + +v1.0 ships only when N1–N4 all hold **and** the launch front door (ST5) is built on top +of a finished N1–N3. + +--- + +## 2. Release arc v0.4.0 → v1.0.0 + +Each release is a milestone with a theme, the finding/backlog IDs it closes, entry/exit +criteria, and the **one honest sequencing rule** it respects (drawn from +`IMPROVEMENT-BACKLOG.md` §Notes on sequencing). The suggested arc from the review was +followed; refinements are flagged `[refined]` with justification. + +### v0.4.0 — "Enforce the wedge" +*The differentiator must survive a skeptic running `doctor`/`inspect`/reading the scanner.* + +- **Closes:** G1/QW2, G2/QW6, G3/HI6, G4/QW10, M1/HI4, M2/HI1 (+ onboarding half O1). + `[refined]` The review's v0.4 seed listed G1,M1,M2,HI4,HI6,HI1; **G2/QW6 and G4/QW10 + are pulled forward into this milestone** because they are gate-integrity/honesty-of-gate + fixes and this milestone's whole theme is gate integrity — a skeptic reading + `holdout_gate.py`/`self_eval.py` finds them in the same pass they find G1/G3. +- **Entry:** v0.3.4 baseline green (105 passed / self_eval 13/13 per MEMORY.md). +- **Tasks & Verify:** + 1. **QW2 (G1)** — `_validate_terminal` cross-checks: `Succeeded` requires + `false_completion is False` **and** ≥1 true `criteria_met` entry. + *Verify:* `uv run --with pytest --with pyyaml python3 -B -m pytest scripts -q -k terminal_cross` — the contradictory-terminal JSON from REVIEW.md G1 now fails; `doctor` flags it. + 2. **HI4 (M1)** — inspector scores `false_completion_defense` only when + `holdout_gate.py`/`anticheat_scan.py` invocation is detectable in verify scripts/RUNLOG. + *Verify:* pytest case "flag set but gate never run → no credit" passes. + 3. **HI1 (M2/O1)** — flagship `examples/coverage-repair` actually runs the held-out + gate to a real 1-of-7 terminal state; example-local `verify-*` paths resolve from + the example dir. *Verify:* `python3 -m loop inspect examples/coverage-repair` shows + gate output as the evidence behind `false_completion:false`; example TASKS verify + paths resolve (`cd examples/coverage-repair && ls scripts/verify-fast` or relabeled). + 4. **HI6 (G3)** — AST/hash structural invariant over the anticheat gate-decision + functions; the `return False`-first-line repro now produces a finding. + *Verify:* pytest case pins the REVIEW.md G3 repro → finding emitted. + 5. **QW6 (G2)** — `holdout_gate.decide`: `if not visible: return NotReady`. + *Verify:* pytest `decide([], [...passed...])` → NotReady, not Succeeded. + 6. **QW10 (G4)** — rename/comment `self_eval` substring checks as + documentation-completeness, not behavioral enforcement, wherever `self_eval` is + described as "the gate." *Verify:* `python3 scripts/self_eval.py` green; grep confirms + the label change. + 7. **`[NEW — beyond review]` live-contract exit gate** — `python3 -m loop doctor .loop` + must return `ok:true`. Justification: dogfood gap #4 (§6) found the repo's own + flagship on-disk contract fails the product's own validator; N1 is not met while + that is true. *Verify:* `python3 -m loop doctor .loop` → `ok:true`. +- **Exit:** N1 holds. Full suite + self_eval green; the three HIGH findings each have a + pinned regression; live `.loop` passes doctor. +- **Sequencing rule:** *Wedge-credibility before launch* — close the holes a skeptic + finds by running `doctor`/`inspect` before any traffic (ST5) is invited. + +### v0.5.0 — "First screen" +*The entire first impression: a stranger gets it in 10s, scores a loop in 30s.* + +- **Closes:** T1/QW3, T2-T4/QW9, O2/HI2, O3/QW7, O4/QW8, plus positioning-sourced + QW1, HI3, QW4, QW5 (no finding ID — sourced from POSITIONING.md). +- **Entry:** v0.4.0 shipped (HI1 done — required to film an honest demo). +- **Tasks & Verify:** + 1. **QW1** — README hero: tagline line 1, 3 concrete failure modes, zero-install + first command. *Verify:* README line 1–3 = tagline + 2-line pain; first fenced cmd + is a no-install `inspect`; `python3 scripts/self_eval.py` README checks green. + 2. **HI2 (O2)** — `docs/demo.gif` (+ `docs/demo.cast`) of the weak→strong `inspect` + arc (score ~36 `weak` → ~90 `strong`), embedded at README top + set as social card. + *Verify:* `test -f docs/demo.gif && test -f docs/demo.cast`; grep README for the embed. + 3. **HI3** — CONTRACT/ORCHESTRATE/EXECUTE stack diagram + verified comparison table + (7 competitors + LE; columns per POSITIONING §5), honest table-stakes caveat kept. + *Verify:* grep README for the three tiers + the table header; competitor rows match + `COMPETITIVE-ANALYSIS.md`. + 4. **QW3 (T1)** — router `SKILL.md:3` + `marketplace.json:5` name the 2 diagnostic + spokes (loop-inspector, loop-runtime-monitor) + a diagnostic verb. + *Verify:* `python3 scripts/validate_frontmatter.py` + `python3 scripts/self_eval.py` + green; grep frontmatter for `inspect`/`audit`/`watch`. + 5. **QW9 (T2/T3/T4)** — trigger-phrase batch (shared "grade" disambiguated, + `loop-evals` description trimmed into peer length band, `loop-run` first example + qualified). Ride QW3's frontmatter edit. *Verify:* `validate_frontmatter.py` green; + `loop-evals` description back in the ~400–510-char band. + 6. **QW7 (O3)** — CONTRIBUTING gains the local-marketplace staleness-trap note. + *Verify:* grep CONTRIBUTING.md for `marketplace`/`refresh`/`stale`. + 7. **QW8 (O4)** — `[project.scripts] loop = "loop.__main__:main"`, preserving relative + `scripts/` resolution. *Verify:* pytest confirms relative resolution when invoked + as `loop`; `python3 -m loop` still works. + 8. **QW4** — GitHub About + topics (add 8 discovery/category terms), own + "false completion" + "loop engineering". *Verify:* metadata-only; checklist in PR. + 9. **QW5** — PRs to ≥2 community discovery lists (awesome-claude-code / + -plugins). *Verify:* PR links tracked in the release notes. +- **Exit:** N2 holds. The QW1+HI2+HI3 batch (POSITIONING §8 "the one thing") ships + together; frontmatter/self_eval green. +- **Sequencing rule:** *HI1 gates HI2* — you cannot honestly film the money-shot demo + over an example that self-asserts completion; the runnable, gate-backed example (v0.4) + is the prerequisite for the GIF. + +### v0.6.0 — "Metrics real" +*Turn FCR and repair-productivity from claims into derivations with a published baseline.* + +- **Closes:** M4/QW11, M3/HI5, ST1. +- **Entry:** v0.4.0 (HI1 + HI4) shipped — a real gated run and honest credit exist. +- **Tasks & Verify:** + 1. **QW11 (M4)** — canonicalize "the repair record": one 7-field schema (or + `eval-suite.md` states which is canonical for RP and how the two relate). + *Verify:* `python3 scripts/self_eval.py` structural checks green; schema referenced + from both `rollout_ledger.py` and `evals/cases/structural.json`. + 2. **HI5 (M3)** — repair-record validator recomputes `productive` from + `verification_before/after` and rejects mismatches; `summarize()` aggregates only + validated records. *Verify:* pytest — a disagreeing record is rejected. + 3. **ST1** — `metrics` command derives FCR + RP from real `.loop/` receipts/RUNLOG; + a checked-in baseline computed over the gate-backed HI1 example; README numbers + sourced from that computation. *Verify:* pytest determinism case; + `python3 -m loop metrics examples/coverage-repair` reproduces the checked-in baseline. +- **Exit:** N3 holds. Baseline is reproducible and traceable to a gated run. +- **Sequencing rule:** *No FCR baseline over a self-asserted example, and QW11 gates ST1* + — the canonical repair schema must exist before RP is derived, and the baseline must + post-date HI1/HI4 or it would itself be a false completion. + +### v0.7.0 — "Portable standard" +*The contract becomes a documented, versioned, tool-agnostic format — and the repo's own +templates/contract stop lying to it.* + +- **Closes:** M5/ST2, plus the dogfood template/validator drift (§6 gaps #1,#2,#3,#5). +- **Entry:** v0.6.0 (QW11) shipped — a canonical repair record exists to schematize. +- **Tasks & Verify:** + 1. **ST2 (M5)** — versioned spec doc + published JSON schemas for + `state.json`/`TASKS.json`/receipt/repair-record + stability/version note + + conformance checklist; `validate_contract` optionally checks `.loop/receipts/*.jsonl` + and repair-record files when present. *Verify:* `python3 -m loop doctor roadmap/v1.0` + → `ok:true` in CI; pytest over the conformance checklist. + 2. **`[NEW — beyond review]` template↔validator reconciliation** (dogfood gaps #1,#3) — + templates emit `schema` (not `schema_version`); manifest + JSON companions use one + consistent schema key. Justification: dogfood found a contract scaffolded straight + from `templates/` fails `doctor` with `schema_mismatch`. *Verify:* `python3 -m loop + doctor` on a freshly-scaffolded-from-template contract → `ok:true`. + 3. **`[NEW — beyond review]` terminal-template completeness** (dogfood gap #2) — + `templates/terminal_state.json.tmpl` carries top-level `criteria_met`, + `false_completion`, and a top-level `evidence` list. Justification: template + scaffolding currently yields 3 `invalid_terminal` issues. *Verify:* scaffold → + `doctor` → 0 terminal issues. + 4. **`[NEW — beyond review]` `Planned`/`NotStarted`/`Deferred` state design** (dogfood + gap #5) — spec the first-class "planned but not yet started" representation into + ST2's versioned contract. Justification: a roadmap-as-contract had to express this + indirectly via a `Succeeded` planning milestone + `pending_approval`. *Verify:* + design section present in the ST2 spec; no runtime change required this milestone. +- **Exit:** N4's standard-half holds. The spec is published; templates, the live + contract, and doctor agree. +- **Sequencing rule:** *You cannot publish a portable standard while your own reference + contract and templates fail your own validator* — reconcile the dogfood drift before + inviting other harnesses to emit the format. + +### v0.8.0 — "Composes the field" +*Every 50K-star engine becomes a complement that needs you.* + +- **Closes:** ST3, ST4. +- **Entry:** v0.7.0 (ST2) shipped — a stable format exists for adapters to target. +- **Tasks & Verify:** + 1. **ST3** — integration recipes for ≥2 of {LangGraph terminal-node → typed terminal + state, Temporal workflow → repo-OS contract, OpenHands run → FCR gate, ruflo swarm → + acceptance gate}, each a runnable snippet + resulting terminal-state/evidence + mapping. *Verify:* each recipe produces an on-disk contract that + `python3 -m loop doctor ` accepts. + 2. **ST4** — contributor funnel: ≥4 `good first issue`/`help wanted` (the QW3/QW9 + trigger fixes as labeled issues once shipped are examples of the *class*), a 2nd + runnable example, and a read-only foreign-harness `inspect` adapter + checked-in + gap report. *Verify:* `python3 -m loop inspect ` produces a + scored report; the 2nd example passes `doctor`. +- **Exit:** N4 fully holds — the standard is not just published but demonstrably ridden + by external engines. +- **Sequencing rule:** *Compose, don't compete* — never claim to replace an execution + engine; every recipe wraps, never swaps, the underlying runtime (POSITIONING §7 DON'T). + +### v1.0.0 — "Launch" +*The front door is finished; open it.* + +- **Closes:** ST5. +- **Entry:** N1–N3 hold and the launch prerequisites (HI1 v0.4, HI2 + HI3 v0.5, ST1 v0.6) + are all shipped. +- **Tasks & Verify:** + 1. **ST5** — Show HN / r/ClaudeAI / r/LocalLLaMA post led by the HI2 GIF; a + `false completion` blog/dev.to post embedding the HI3 diagram; two-way cross-link + with `claude-code-orchestration`'s `/verify-slice`. *Verify:* the demo (HI2), hero + (QW1), table (HI3), and baseline (ST1) all present on `main` before posting — + a checklist gate, not a code test. +- **Exit:** v1.0 tagged; N1–N4 all hold; launch surfaces live. +- **Sequencing rule:** *Launch last* — a launch without the weak→strong GIF and the + honest, gate-backed example is a wasted first impression at 0★ + (`IMPROVEMENT-BACKLOG.md` §Notes: ST5 gated on HI1+HI2+HI3). + +--- + +## 3. Dependency graph + +The four EXACT invariants from `IMPROVEMENT-BACKLOG.md` §Notes on sequencing are +preserved and shown as hard edges: + +``` + v0.4 ENFORCE THE WEDGE v0.5 FIRST SCREEN + ┌─────────────────────────────┐ ┌───────────────────────────┐ + │ QW2(G1) QW6(G2) HI6(G3) │ │ QW1 QW3(T1) QW4 QW5 │ + │ QW10(G4) HI4(M1) HI1(M2/O1) │ │ QW7(O3) QW8(O4) QW9(T2-4) │ + │ ── wedge-credibility ── │ │ HI2(O2) HI3 │ + │ {QW2,HI1,HI4,HI6} ───────┼───┐ └───────────────────────────┘ + └──────────┬───────────┬───────┘ │ ▲ │ │ + HI1 ──┤ └── HI4 ──┐ │ │ HI1 │ HI2 │ HI3 + │ │ │ (gates) │ gates │ │ + ▼ (gates HI2) ▼ │ │ HI2 ▼ ▼ + ┌──────────────┐ v0.6 METRICS REAL │ v1.0 LAUNCH (ST5) + │ HI2 (v0.5) │ ┌──────────────────┐ │ gated on HI1+HI2+HI3 + └──────────────┘ │ QW11(M4) ─gates─▶ │ │ (+ ST1 baseline) + │ HI5(M3) ST1 │◀┘ + │ ST1 ◀─ needs HI1+HI4 (no baseline over + └────────┬─────────┘ a self-asserted example) + │ ST1 needs QW11 + HI1 + ▼ + v0.7 PORTABLE STANDARD + ┌──────────────────────────┐ + │ ST2(M5) + template/ │ needs QW11 (canonical + │ validator drift + states │ repair record) to schematize + └───────────┬──────────────┘ + │ ST2 = the format adapters target + ▼ + v0.8 COMPOSES THE FIELD + ┌──────────────────────────┐ + │ ST3 recipes · ST4 funnel │ + └───────────┬──────────────┘ + ▼ (all N1–N3 + launch assets in place) + v1.0 LAUNCH · ST5 +``` + +**The four preserved invariants, called out explicitly:** +1. **Wedge-credibility (QW2/HI1/HI4/HI6) before launch/GIF.** All four land in v0.4, + strictly before HI2 (v0.5 GIF) and ST5 (v1.0 launch). +2. **HI1 gates HI2 and ST1.** HI1 is in v0.4; HI2 (v0.5) and ST1 (v0.6) both come after. +3. **QW11 gates ST1.** Both are in v0.6; QW11 is sequenced first within the milestone. +4. **No FCR baseline over a self-asserted example.** ST1 (v0.6) depends on HI1 **and** + HI4 (both v0.4) so the baseline is computed only over a gate-backed run. + +--- + +## 4. Mapping table — every finding & every backlog item + +Nothing is dropped. Milestone column is the release that closes it. + +### 4a. Review findings (§3 of REVIEW.md) + +| ID | Dimension / severity | Backlog item | Milestone | +|---|---|---|---| +| G1 | gate-integrity HIGH | QW2 | v0.4 | +| G2 | gate-integrity MEDIUM | QW6 | v0.4 | +| G3 | gate-integrity MEDIUM | HI6 | v0.4 | +| G4 | gate-integrity LOW | QW10 | v0.4 | +| M1 | metric-honesty HIGH | HI4 | v0.4 | +| M2 | metric-honesty HIGH | HI1 | v0.4 | +| M3 | metric-honesty MEDIUM | HI5 | v0.6 | +| M4 | metric-honesty MEDIUM | QW11 | v0.6 | +| M5 | metric-honesty LOW | ST2 | v0.7 | +| T1 | trigger-quality MEDIUM | QW3 | v0.5 | +| T2 | trigger-quality LOW | QW9 (batch) | v0.5 | +| T3 | trigger-quality LOW | QW9 (batch) | v0.5 | +| T4 | trigger-quality LOW | QW9 (batch) | v0.5 | +| O1 | onboarding-dx MEDIUM | HI1 | v0.4 | +| O2 | onboarding-dx MEDIUM | HI2 | v0.5 | +| O3 | onboarding-dx MEDIUM | QW7 | v0.5 | +| O4 | onboarding-dx LOW | QW8 | v0.5 | +| docs-honesty | **none surfaced** | — | — (nothing to close; audited clean) | + +### 4b. Backlog items (IMPROVEMENT-BACKLOG.md) + +| ID | Title (abbrev) | Finding(s) | Milestone | +|---|---|---|---| +| QW1 | README hero | (positioning) | v0.5 | +| QW2 | terminal cross-check | G1 | v0.4 | +| QW3 | diagnostic spokes to router/marketplace | T1 | v0.5 | +| QW4 | GitHub metadata/SEO | (positioning) | v0.5 | +| QW5 | awesome-list PRs | (positioning) | v0.5 | +| QW6 | empty visible → NotReady | G2 | v0.4 | +| QW7 | staleness-trap doc | O3 | v0.5 | +| QW8 | console-script `loop` | O4 | v0.5 | +| QW9 | trigger-phrase batch | T2, T3, T4 | v0.5 | +| QW10 | honest self_eval labels | G4 | v0.4 | +| QW11 | canonical repair schema | M4 | v0.6 | +| HI1 | runnable flagship example | M2, O1 | v0.4 | +| HI2 | weak→strong demo GIF | O2 | v0.5 | +| HI3 | three-tier stack + table | (positioning) | v0.5 | +| HI4 | honest inspector credit | M1 | v0.4 | +| HI5 | enforce `productive` | M3 | v0.6 | +| HI6 | anticheat structural invariant | G3 | v0.4 | +| ST1 | published FCR/RP baseline + metrics CLI | (metric-honesty + competitive) | v0.6 | +| ST2 | versioned portable contract spec + schemas | M5 | v0.7 | +| ST3 | integration recipes | (positioning + competitive) | v0.8 | +| ST4 | contributor funnel | (positioning) | v0.8 | +| ST5 | launch surfaces | (positioning) | v1.0 | + +**Nothing is deferred past v1.0.** All 17 enumerable findings and all 22 backlog items +route to a milestone in the v0.4→v1.0 arc. + +### 4c. Note on REVIEW.md's internal count inconsistency + +REVIEW.md is internally inconsistent about its own finding count and this roadmap sides +with the enumerable/backlog-consistent number: +- REVIEW.md §1 and §4 (the count table) both claim **18 total** and **6 metric-honesty** + findings (2 HIGH / 2 MEDIUM / 2 LOW). +- But REVIEW.md §3.2 only **enumerates 5 metric-honesty findings** (M1–M5 = 2 HIGH / + 2 MEDIUM / **1 LOW**), giving **17 enumerable findings**. +- `IMPROVEMENT-BACKLOG.md` §"Dimension coverage this pass" agrees with the *enumerated* + count: metric-honesty = **5** (2 HIGH + 2 MEDIUM + 1 LOW). + +There is no M6 anywhere in the text, so the "6 / 18" figures appear to be a +tally error; the mapping above covers the 17 findings that actually exist (M1–M5) plus +docs-honesty=none. If a sixth metric-honesty finding is ever recovered, it would route to +v0.6 with the other metric work. + +--- + +## 5. Strategic design specs (mechanism lives here) + +The four strategic specs authored 2026-06-30 carry the detailed enforcement/format +design; each milestone above delegates its mechanism to one of them: + +| Spec | Drives milestone | Closes | +|---|---|---| +| [`docs/superpowers/specs/2026-06-30-v04-credibility-enforcement.md`](../specs/2026-06-30-v04-credibility-enforcement.md) | v0.4 | G1/QW2, G2/QW6, G3/HI6, M1/HI4, M2/HI1 + the dogfood live-contract gate | +| [`docs/superpowers/specs/2026-06-30-st1-metrics-baseline.md`](../specs/2026-06-30-st1-metrics-baseline.md) | v0.6 | ST1, M3/HI5, M4/QW11 | +| [`docs/superpowers/specs/2026-06-30-st2-portable-contract-spec.md`](../specs/2026-06-30-st2-portable-contract-spec.md) | v0.7 | ST2/M5 + template/validator drift + the `Planned` state design | +| [`docs/superpowers/specs/2026-06-30-st3-integration-adapters.md`](../specs/2026-06-30-st3-integration-adapters.md) | v0.8 | ST3, ST4 | + +*(The contract in `roadmap/v1.0/specs/spec-1..4-*.md` mirrors these four as the +contract's own internal spec set; the canonical, dependency-free copies are the +`docs/superpowers/specs/` files listed above.)* + +--- + +## 6. Dogfood — gaps found by self-hosting + +These are **real defects surfaced by authoring this roadmap as a loop-engineer contract +against the live validator (`loop/contract.py`)**, logged in `roadmap/v1.0/RUNLOG.md`. +They are not in the original review; each is routed to a milestone (mostly v0.7, the +template/validator-drift milestone). Marked `[NEW — beyond review]` throughout. + +| # | Dogfood gap | Route | How it's closed / Verify | +|---|---|---|---| +| 1 | **Template `schema_version` vs validator `schema` drift** — `templates/state.json.tmpl` & `terminal_state.json.tmpl` emit `"schema_version":"1.0"` but the validator checks key `schema` (`loop-engineer/state@1`, `…/terminal@1`); a from-template scaffold fails `doctor` with `schema_mismatch (got None)`. | v0.7 (task 2) | Templates emit `schema`. *Verify:* scaffold-from-template → `python3 -m loop doctor ` → `ok:true`. | +| 2 | **Terminal template missing required fields** — no `criteria_met`, no `false_completion`; `evidence` nested under `verification_evidence` instead of a top-level list. `_validate_terminal` requires all three at top level → 3 `invalid_terminal` issues. | v0.7 (task 3) | Template carries the three top-level fields. *Verify:* scaffold → `doctor` → 0 terminal issues. | +| 3 | **Manifest template uses `schema:` while its JSON companions use `schema_version:`** — the shipped template set is internally inconsistent on the very field that names the schema. | v0.7 (task 2) | One consistent schema key across manifest + companions. *Verify:* grep templates for a single `schema` key convention. | +| 4 | **The repo's OWN live contract fails doctor** — `python3 -m loop doctor .loop` → `ok:false` (state/TASKS/terminal carry `schema_version`; terminal lacks `criteria_met`/`false_completion`). The flagship on-disk contract does not pass the product's own gate. | **v0.4** (task 7, release-blocking exit gate) | Migrate the live `.loop` to the validated shape. *Verify:* `python3 -m loop doctor .loop` → `ok:true`. | +| 5 | **No `Planned`/`NotStarted`/`Deferred` state exists** — the 7 states are all terminal and the FSM starts at `intake`; there is no first-class way to mark a milestone "planned but not started" (a roadmap-as-contract needs exactly this). Had to express it via a `Succeeded` planning milestone + a `pending_approval` block. | v0.7 (task 4) | Design the planned/deferred representation into the ST2 versioned spec. *Verify:* design section present in the ST2 spec. | +| 6 | **`doctor` has no `--strict` cross-field mode** — because G1 is an un-enforced hole, dogfooding cannot ask doctor to enforce the honesty this contract practices voluntarily. | **v0.4** (folds into QW2/G1) | The QW2 cross-field rule makes the honesty mandatory (a `--strict` surface is optional once the rule is unconditional). *Verify:* the G1 regression test (§2 v0.4 task 1). | + +Routing summary: gaps #4 and #6 are pulled into **v0.4** (they are wedge-enforcement / +live-contract-integrity, and #4 is an explicit N1 exit blocker); gaps #1, #2, #3, #5 land +in **v0.7** where the template/validator drift is reconciled as part of publishing the +portable standard. + +--- + +## 7. `roadmap/v1.0/` is a validated contract — the positive contrast to G1/M2 + +`roadmap/v1.0/` is not just documentation; it is a **loop-engineer contract that passes +the product's own gate**: + +- `python3 -m loop doctor roadmap/v1.0` → **`ok: true`, 0 issues** (passed on the first + run and again with the full file set). +- Its terminal is an **honest `Succeeded`** for the *planning* milestone only: + `false_completion:false`, all-true `criteria_met` — every criterion is a real artifact + (this roadmap, the four specs, the mapping table) that exists on disk. +- The v1.0 **release** is deliberately **withheld as human-gated** + (`state.json.pending_approval`), not silently marked done. + +This is the deliberate inverse of findings **G1** (a `Succeeded` terminal with +`false_completion:true` validating clean) and **M2** (a flagship example asserting +`false_completion:false` with no gate ever run): here the success terminal is +internally consistent *and* evidence-backed, and the un-earned part of the goal (shipping +v1.0) is honestly parked in `pending_approval` instead of claimed. The roadmap practices +the enforcement that v0.4 will make mandatory for everyone. +*Verify:* `python3 -m loop doctor roadmap/v1.0` → `ok:true`. + +--- + +*End of roadmap. Every milestone task above carries a concrete `Verify:` command +(pytest / self_eval / doctor / inspect / file check). Canonical test invocation per +project memory: `uv run --with pytest --with pyyaml python3 -B -m pytest tests scripts -q`.* diff --git a/docs/superpowers/specs/2026-06-30-st1-metrics-baseline.md b/docs/superpowers/specs/2026-06-30-st1-metrics-baseline.md new file mode 100644 index 0000000..820fb4d --- /dev/null +++ b/docs/superpowers/specs/2026-06-30-st1-metrics-baseline.md @@ -0,0 +1,176 @@ +# ST1 — Published FCR / Repair-Productivity Baseline + Metrics CLI + +*Design spec. Report-only planning artifact — no code in this document. Compiled 2026-06-30.* + +**Backlog item:** ST1 (rank 15, Impact H / Effort H, Strategic) — *"Publish a real FCR / repair-productivity baseline + metrics CLI."* +**Grounded in:** metric-honesty findings **M3** (`productive` flag unenforced), **M4** (two disjoint 7-field repair-record shapes), **M5** (validator covers only 4 of 5 schemas; never checks repair records / receipts); backlog **QW11** (canonicalize the repair record), **HI5** (enforce `productive` — recompute + reject), **ST1**. +**Sources of truth:** `reference/eval-suite.md` §2 (FCR/RP definitions, "derived not self-reported"), `scripts/rollout_ledger.py`, `evals/cases/structural.json` (`repair_record_fields`), `skills/loop-repair/SKILL.md` (§ "The structured repair record"), `schemas/receipt.schema.json`, `examples/coverage-repair/repair-record.json`, `templates/RUNLOG.md.tmpl`, `templates/terminal_state.json.tmpl`. + +--- + +## 1. Problem statement + +false-completion-rate (FCR) and repair-productivity (RP) are the two first-class metrics the competitive analysis names as having no rival — they *are* the wedge. Today they are **claimed but not derivable from a tool the repo ships**, and their inputs are **self-asserted, not enforced**: + +- **No metrics command exists.** `reference/eval-suite.md` §2 defines FCR and RP as formulas and asserts both are "derived, not self-reported," but nothing in `scripts/` or `loop/` computes them from a real `.loop/` run. The numbers live only in prose. +- **The `productive` input to RP is trusted verbatim.** `scripts/rollout_ledger.py` `summarize()` sums whatever `productive` boolean the caller supplies (M3); `skills/loop-repair/SKILL.md` prescribes `productive = verification_after.score > verification_before.score` but nothing recomputes or validates it. +- **"The repair record" is two different shapes** (M4 / QW11): `rollout_ledger.py RECORD_FIELDS` (`id`/`parent`/`verdict`/`score`/`score_delta`/`coherent_with_prior_winner`/`productive`) vs `evals/cases/structural.json repair_record_fields` (`failure_mode`/`hypothesis`/`repair_action`/`verification_before`/`verification_after`/`remaining_delta`/`productive`). They share only the count "7" and the field `productive`. RP's provenance is ambiguous. +- **The contract validator never checks the FCR/RP evidentiary trail** (M5): `validate_contract()` covers manifest/state/tasks/terminal only; `schemas/receipt.schema.json` is never referenced and no repair-record schema exists. + +A metric with no published, tool-derived baseline reads as a claim. Worse: a baseline computed over the repo's *current* flagship example would be a baseline over a **self-asserted** run (the coverage-repair example asserts `false_completion:false` with no held-out gate ever invoked — findings M1/M2). Publishing that number would itself be a false completion of ST1. + +## 2. Goal + +Ship a repo-native **`metrics` command** that derives FCR + RP (plus supporting cost/efficiency figures) from a loop's *real* `.loop/` receipts / RUNLOG / repair records / verify bundles, with the `productive` input **recomputed and cross-checked** (never trusted); and publish a **checked-in baseline** computed by that command over a **genuinely gate-backed** example, with README numbers sourced from the computation rather than prose. + +## 3. Non-goals + +- Not building a new verification engine. Deterministic evidence is read from the contract's own `scripts/verify-*` bundles (eval-suite.md §5 "reuse, do not reimplement"). +- Not implementing the gate-enforcement fixes ST1 depends on (QW2 / HI1 / HI4 / G1 / G2 / M1 / M2). Those live in the companion **v0.4 credibility / gate-enforcement spec** (see §7). This spec *consumes* their output; it does not duplicate it. +- Not shipping a comparative A/B win/loss number. `scripts/benchmark_harness.py` + eval-suite.md §7 already own the A/B protocol and deliberately bake in no product claim; ST1 scores *one* loop, not a harness-vs-harness swing. +- Not a live-telemetry or dashboard product. Output is a machine-readable JSON scorecard from committed files, offline, no network. + +--- + +## 4. Design + +### 4.1 Resolve the repair-record divergence (QW11 / M4) — one canonical repair record + +There are two genuinely different artifacts today, both mislabelled "7-field repair record." They are **not** the same thing and unifying their *fields* would be wrong. The resolution is to canonicalize the **name** and the **derivation rule**, not to merge the shapes: + +| Artifact | Canonical shape | Schema tag | On-disk location | Role | Feeds | +|---|---|---|---|---|---| +| **Repair record** *(the canonical "repair record")* | `evals/cases/structural.json repair_record_fields`: `failure_mode`, `hypothesis`, `repair_action`, `verification_before`, `verification_after`, `remaining_delta`, `productive` (+ envelope `schema`/`iteration_id`/`attempt` per `examples/coverage-repair/repair-record.json`) | `loop-engineer/repair@1` | `.loop/repair/.json` | one bounded repair pass, emitted by `loop-repair` | **RP** (canonical input) | +| **Rollout / candidate ledger record** | `rollout_ledger.py RECORD_FIELDS`: `id`, `parent`, `verdict`, `score`, `score_delta`, `coherent_with_prior_winner`, `productive` | `loop-engineer/rollout@1` (new) | `.loop/*.jsonl` (append-only) | one *candidate adjudication* in a rollout / genetic-hardening loop | rollout-productivity (a flywheel view), **not** the RP baseline | + +**Decisions:** +1. The **repair record** (structural.json shape, `loop-engineer/repair@1`) is the single artifact that may be called "the repair record." It is RP's canonical input — consistent with `eval-suite.md` §2.2, which already anchors RP on the repair record's `verification_before`/`verification_after` fields. +2. The `rollout_ledger.py` record is **renamed in documentation** to the "rollout ledger / candidate record" (`loop-engineer/rollout@1`). It is a distinct artifact for candidate adjudication; it must stop being branded "the repair record." Its `productive` field is the *rollout*-productivity signal, not the RP baseline. +3. **Both derive `productive` by the same underlying rule** — *did this pass measurably improve the score?* — expressed against each shape's own evidence: + - repair record → `productive == (verification_after.score > verification_before.score)` + - rollout record → `productive == (score_delta > 0)` (equivalently `score > parent-winner score`) +4. `reference/eval-suite.md` gains one sentence naming the repair record as canonical for RP and pointing at the rollout ledger as the separate candidate artifact. `evals/cases/structural.json` remains the structural source of the 7 repair fields (no field change). + +*This closes M4 (two schemas branded "the" repair record) and satisfies QW11 (one canonical 7-field repair schema). QW11 is a hard predecessor of ST1: the baseline cannot be derived until it is unambiguous which record RP reads.* + +### 4.2 Canonical schemas (closes M5 for the RP/FCR trail) + +Publish two JSON Schema files under `schemas/` (companion to the existing `schemas/receipt.schema.json`): + +- **`schemas/repair-record.schema.json`** (`$id: loop-engineer/repair@1`): the 7 canonical fields as `required`, plus envelope `schema`/`iteration_id`/`attempt`; `verification_before`/`verification_after` each require a numeric `score`; `productive` is `boolean`. `additionalProperties: true` (repair records carry loop-specific evidence keys like `metric`/`failing`, per the coverage-repair example). +- **`schemas/rollout-record.schema.json`** (`$id: loop-engineer/rollout@1`): the 7 `rollout_ledger.py` fields as `required`; `score`/`score_delta` numeric-or-null; `productive` boolean. + +`validate_contract()` is extended (in the companion enforcement spec's scope, but specified here because it is the M5 fix) to **optionally** validate `.loop/repair/*.json` and `.loop/*.jsonl` receipt/rollout files against these schemas *when present*, so a loop can no longer "pass validation" while its metric inputs are missing or malformed. + +### 4.3 Recompute-and-reject `productive` (HI5 / M3) + +A shared validator function — call it `recheck_productive(record)` — is the single point of truth used by **both** the metrics command and (in the enforcement spec) `rollout_ledger.summarize()`: + +- **Repair record:** recompute `expected = verification_after.score > verification_before.score`. If the stored `productive` disagrees with `expected`, the record is **rejected** (not silently coerced) — the metrics command excludes it from the RP numerator/denominator and reports it under a `rejected_records` provenance list with the reason. A record missing `verification_before`/`verification_after.score` is likewise rejected (it "cannot demonstrate productivity" — loop-repair SKILL.md already states such a record is invalid). +- **Rollout record:** recompute `expected = (score_delta is not None and score_delta > 0)`; same reject-on-disagreement rule. +- **No caller-supplied boolean is ever summed verbatim.** RP is computed only over records whose `productive` was *recomputed and agreed*. This converts RP from a self-report into a derivation — exactly the property `eval-suite.md` §2.2 claims and M3 shows is currently false. + +Rejection is a first-class output, not a crash: the command exits non-zero **only** when asked to emit a *baseline* (see §4.5) over a run containing rejected records; in plain `metrics` mode it reports them and continues, so an operator can see *which* records are dishonest. + +### 4.4 The `metrics` command + +A new `scripts/metrics.py`, wired as a `python3 -m loop metrics ` subcommand alongside `doctor` / `inspect` (same editable-install, repo-relative `scripts/` resolution constraint noted in `pyproject.toml`; see QW8). Pure stdlib, offline, deterministic. + +**Inputs (all read from the target loop dir — never the agent's narration):** + +| Input | File(s) | Used for | +|---|---|---| +| Success claims + per-iteration outcome | `RUNLOG.md` (`### Outcome` = `task_passed`/`terminal`/…), `terminal_state.json` (`state`, `succeeded`) | FCR numerator/denominator (claims of "done") | +| Deterministic evidence per iteration | `.loop/` verify bundles (`verify-*.json`: `verify_fast`/`verify_full` outcome, `score`, `failing`), the layer-1 gate output | FCR cross-join (did deterministic verify agree?) | +| Held-out gate output | `scripts/holdout_gate.py` result / `anticheat_scan.py` sweep recorded in the verify trail or RUNLOG | `evidence_backed` flag (§4.5); FCR via aggregated `false_completion` flag | +| Repair records | `.loop/repair/.json` (`loop-engineer/repair@1`) | RP (after recompute-and-reject) | +| Receipts | `.loop/receipts/*.jsonl` (`loop-engineer/receipt@1`) | cost-per-success, iteration/dispatch counts (layer 7) | + +**Computation (exactly the eval-suite.md §2 formulas):** + +``` +FCR = (iterations claiming success AND failing deterministic verify) + / (iterations claiming success) # target 0 + +RP = (repair passes where verification_after.score > verification_before.score, recomputed) + / (total repair passes attempted) # target high, trending up +``` + +- FCR is anchored to the **deterministic layer**, per eval-suite.md §3 — a "Succeeded"/`task_passed` claim not backed by a green `verify-full` for the same `iteration_id` is a false completion. The command computes FCR **two ways** and asserts they agree: (a) the RUNLOG-claim × verify-bundle cross-join, and (b) the aggregated `holdout_gate` `false_completion` flag. Disagreement is surfaced (it means the loop's own gate output and its logged claims are inconsistent) rather than silently picking one. +- RP is computed only over validated (recomputed-and-agreed) repair records (§4.3). + +**Output:** a JSON scorecard to stdout, e.g. + +```json +{ + "schema": "loop-engineer/metrics@1", + "loop": "", + "false_completion_rate": 0.0, + "repair_productivity": 0.5, + "iterations_claiming_success": 3, + "false_completions": 0, + "repair_passes": 2, + "productive_repairs": 1, + "cost_per_success_usd": null, + "evidence_backed": true, + "provenance": { + "fcr_source": ["RUNLOG.md", ".loop/"], + "rp_source": [".loop/repair/iter-002.json"], + "rejected_records": [] + } +} +``` + +- `evidence_backed` is **true only when a held-out / anti-cheat gate invocation is detectable** in the verify trail or RUNLOG for the success-claiming iterations — the same honesty rule HI4 puts on the inspector. A run that self-asserts `false_completion:false` with no gate ever invoked reports `evidence_backed:false` and an `FCR` derived purely from the deterministic cross-join (never from the untrusted flag). +- Every headline number ships with a `provenance` block naming the files it came from, so a skeptic can re-derive it by hand. + +### 4.5 The published baseline — and why it is gated + +A `--baseline` mode writes a checked-in scorecard (e.g. `docs/metrics-baseline.json`) that the README's FCR/RP numbers cite by reference (not prose). This mode enforces the honesty preconditions ST1 exists to protect: + +1. It **refuses** (exits non-zero, writes nothing) if `evidence_backed` is false for the run — a baseline may only be computed over a **genuinely gate-backed** run. *Do not baseline over a self-asserted run — that would itself be a false completion of ST1.* +2. It **refuses** if any repair/rollout record was rejected by §4.3 (a baseline must not contain a record whose `productive` disagrees with its own evidence). +3. It stamps the source example, the commit, and the exact input file list so the number is reproducible and diffable in PRs. + +The baseline is computed over the **HI1-fixed** `examples/coverage-repair` — the version that actually runs `holdout_gate.py` end-to-end and records the gate's output as the evidence behind `false_completion:false`. Until HI1 lands, the example is not eligible and `--baseline` will (correctly) refuse. + +--- + +## 5. Acceptance criteria + +- **AC1 (QW11/M4):** `schemas/repair-record.schema.json` (`loop-engineer/repair@1`) and `schemas/rollout-record.schema.json` (`loop-engineer/rollout@1`) exist; `reference/eval-suite.md` names the repair record as RP's canonical input and the rollout ledger as the separate candidate artifact; `evals/cases/structural.json repair_record_fields` remains the structural source and validates against the repair schema. `self_eval.py` structural checks stay green. +- **AC2 (HI5/M3):** a `recheck_productive` validator recomputes `productive` from `verification_before`/`verification_after.score` (repair) and `score_delta` (rollout) and **rejects** disagreements; the metrics command aggregates RP only over validated records; a regression test pins a disagreeing repair record → rejected + excluded from RP. +- **AC3 (metrics CLI):** `python3 -m loop metrics ` derives FCR + RP from real `.loop/` receipts / RUNLOG / repair records / verify bundles — never from agent narration — and emits a JSON scorecard with a `provenance` block; a pytest case pins determinism on a fixture loop dir. +- **AC4 (FCR honesty):** FCR is computed from the deterministic cross-join and cross-checked against the aggregated `holdout_gate` `false_completion` flag; `evidence_backed` is true only when a gate invocation is detectable; a test covers "claim set but gate never run" → `evidence_backed:false`. +- **AC5 (baseline gating):** `--baseline` refuses to emit over a run that is not `evidence_backed` or that contains rejected records; on the HI1-fixed example it writes `docs/metrics-baseline.json`; README FCR/RP numbers are sourced from that file, not prose. +- **AC6 (M5, optional-when-present):** `validate_contract()` validates `.loop/repair/*.json` and `.loop/*.jsonl` against their schemas when present (may land with the companion enforcement spec; noted here for traceability). + +## 6. Test plan + +- Unit: `recheck_productive` over agree / disagree / missing-score cases (repair + rollout shapes). +- Unit: FCR cross-join over a fixture with a "claimed done + verify red" iteration (FCR = 1) and an all-green fixture (FCR = 0). +- Unit: RP over a fixture with one productive + one churn repair record (RP = 0.5), and a fixture whose stored `productive` lies (rejected → excluded). +- Determinism: same fixture dir → byte-identical scorecard across runs. +- Guard: `--baseline` over a non-`evidence_backed` fixture exits non-zero and writes nothing. +- Schema: `schemas/repair-record.schema.json` / `rollout-record.schema.json` validate the shipped `examples/coverage-repair/repair-record.json` and a rollout-ledger fixture. + +## 7. Dependency ordering (READ BEFORE SCHEDULING) + +ST1 is Effort-H precisely because it has hard predecessors. Two classes of dependency: + +**A. Schema predecessor (can land now, independent of the gate work):** +- **QW11 (§4.1)** — canonicalize the repair record. *Must* land before RP is derived; otherwise it is ambiguous which record RP reads. This is pure documentation + schema work and is the natural first slice of ST1 itself. + +**B. Credibility predecessor — the v0.4 gate-enforcement spec (companion; do NOT baseline before it lands):** +The *metrics CLI (the tool)* can be built and tested against fixtures independently. The **published baseline number** must wait for the enforcement work that makes a run genuinely gate-backed, because a baseline over a self-asserted run is a false completion of ST1: +- **HI1** — make the flagship example actually run the held-out gate end-to-end. Gates the baseline's *input* (§4.5). Also gates HI2 (demo GIF) and this baseline per the backlog's own sequencing note. +- **HI4** — score false-completion-defense only on real gate invocation. Same honesty rule this spec's `evidence_backed` flag depends on. +- **QW2 / G1** — cross-check terminal state vs `false_completion`/`criteria_met`; **G2 / QW6** — empty-visible → NotReady; **M1 / M2** — inspector/example self-assertion holes. These are the enforcement fixes that make the gate output the metrics command reads trustworthy. + +**Sequencing:** `QW11 (schema)` → `metrics.py + recheck_productive + tests (this spec, against fixtures)` → *(companion v0.4 enforcement: QW2, G1, G2, M1, M2, HI1, HI4)* → `python3 -m loop metrics --baseline` over the HI1-fixed example → commit `docs/metrics-baseline.json` → wire README numbers. Do **not** run the final baseline step until the companion enforcement spec has landed. + +## 8. Risks + +- **Baselining too early.** The single biggest risk is publishing a number over a self-asserted run. Mitigated structurally by §4.5's `--baseline` refusal — the tool itself will not let you. +- **RUNLOG ↔ verify-bundle join fragility.** The FCR cross-join keys on `iteration_id`; if a loop's RUNLOG and verify bundles disagree on iteration ids the join is lossy. Mitigation: the command reports unmatched iterations under provenance and treats an unmatched success-claim as a false completion (fail-closed, per eval-suite.md §2.1 "not backed by a green verify … is a false completion, full stop"). +- **Two-way FCR disagreement.** If the deterministic cross-join and the aggregated `holdout_gate` flag disagree, that is itself a defect signal; the command surfaces it rather than picking a winner, so it can't launder an inconsistent run into a clean number. diff --git a/docs/superpowers/specs/2026-06-30-st2-portable-contract-spec.md b/docs/superpowers/specs/2026-06-30-st2-portable-contract-spec.md new file mode 100644 index 0000000..875ada4 --- /dev/null +++ b/docs/superpowers/specs/2026-06-30-st2-portable-contract-spec.md @@ -0,0 +1,438 @@ +# ST2 — Portable Loop-Engineer Contract: Versioned, Tool-Agnostic Standard + +*Design spec. Report-only planning artifact — defines the target, does not implement it.* +*Compiled 2026-06-30. Backlog ID: ST2 (rank 17, Impact H / Effort H, Strategic).* +*Scope boundary: this document specifies. No code, schema, template, or `.loop/` runtime is +touched by writing it.* + +--- + +## 1. Why this exists + +Loop Engineer's defensible claim (verified in `review/COMPETITIVE-ANALYSIS.md` §Whitespace, +primitive #4) is that it ships an **engine-neutral repo-OS contract** — files on disk, +resumable from `.loop/state.json`, additive on top of any Q3/Q2 runtime. No verified rival +(Superpowers 242k★, ECC 224k★, OpenHands 79k★, ruflo 62k★, AutoGen, CrewAI, LangGraph, +Temporal) ships a portable on-disk evidence contract. That is the deepest moat available: if +the on-disk format is a *documented, versioned, tool-agnostic standard*, other harnesses can +emit and consume it, and Loop Engineer becomes the reference implementation rather than one +more harness. + +Today the contract is **not** a published standard — it is an implicit format enforced by one +validator that is provably incomplete. The gate this spec closes is the LOW gate-integrity / +metric-honesty finding **M5**: + +> `loop/contract.py:219-240` — `validate_contract()` calls only +> `_validate_manifest` / `_validate_state` / `_validate_tasks` / `_validate_terminal`. +> `schemas/receipt.schema.json` exists but is **never referenced** in `contract.py`, and **no +> repair-record schema exists at all**. So the contract-integrity gate is silent on whether +> the FCR/repair-productivity evidentiary trail (receipts, repair records) exists or is +> well-formed — a loop can "pass validation" while its metric inputs are missing or malformed. + +The published schemas today are 5 (`schemas/*.schema.json`: manifest, state, tasks, terminal, +receipt); the validator checks 4 of them and no repair-record schema is published at all. +**Covering 4 of 5 while claiming a portable contract is the exact gap an external evaluator +finds first** when they open `contract.py` to decide whether the "layer above" claim is real. + +--- + +## 2. The dogfood gaps this spec must fix first + +Before ST2 can publish the contract as a standard, the standard must be internally +self-consistent. **Self-hosting this very roadmap surfaced three concrete inconsistencies** +between what the `templates/` scaffold emits and what `loop/contract.py` (and the published +`schemas/`) demand. A "portable standard" whose own templates fail its own validator is not a +standard — it is a bug. All three are confirmed against source below and are **in scope for +ST2 as the reconciliation layer**. + +### DG-1 — Templates emit `schema_version`; the validator and schemas require `schema` + +- **Evidence:** + - `templates/state.json.tmpl:2` → `"schema_version": "1.0"`. + - `templates/terminal_state.json.tmpl:2` → `"schema_version": "1.0"`. + - But `loop/contract.py:134` (`_validate_state`) calls `_require_schema(data, + "loop-engineer/state@1", …)` which checks `data.get("schema")`; `contract.py:170` + (`_validate_terminal`) checks `data.get("schema") == "loop-engineer/terminal@1"`. + - `schemas/state.schema.json` and `schemas/terminal.schema.json` both `required` the key + `"schema"` with a `const` value, not `schema_version`. +- **Consequence:** A state or terminal file scaffolded verbatim from the shipped template + fails `validate_contract`/`doctor` with a `schema_mismatch` (got `None`). The scaffold the + suite ships cannot pass the gate the suite ships. +- **Note the split:** `templates/TASKS.json.tmpl:2` (`"schema": "loop-engineer/tasks@1"`) and + `templates/manifest.yaml.tmpl:4` (`schema: loop-engineer/manifest@1`) are **already + correct** — the drift is isolated to the two JSON state templates. The fix is a one-key + rename in each, plus a version note (see §6). + +### DG-2 — The `terminal_state` template omits `criteria_met`, `evidence`, and `false_completion` + +- **Evidence:** `templates/terminal_state.json.tmpl` carries `verification_evidence` (object), + `lessons_learned` (array), `succeeded`, `failure_reason` — but **none** of the three fields + the validator and schema make load-bearing: + - `loop/contract.py:175-180` requires `criteria_met` (dict), `evidence` (list), + `false_completion` (bool). + - `schemas/terminal.schema.json:6` `required`: `schema`, `state`, `iteration_id`, + `criteria_met`, `evidence`, `false_completion`, `reason`, `lessons_ref`. +- **Consequence:** The terminal template is missing the *entire proof surface* — the exact + fields (`criteria_met`, `evidence`, `false_completion`) that carry the differentiator. A + terminal file scaffolded from it fails validation on three `invalid_terminal` issues and, if + hand-patched to pass, still cannot express the false-completion signal the product sells. +- **Interaction with the wedge:** `false_completion` is the field HI4/QW2 are hardening. If the + template does not even emit it, every downstream honesty fix has nothing to bind to. **DG-2 + is a prerequisite for the metric-honesty work, not merely cosmetic.** + +### DG-3 — No non-terminal lifecycle state; `doctor` forces a terminal file onto an unstarted loop + +- **Evidence:** `loop/contract.py:225` reads `paths.terminal` unconditionally, and + `_read_json` (`contract.py:34-36`) appends a `missing_file` issue whenever the file is + absent. There is no contract-level concept of "this loop has not terminated yet, and that is + a valid, conformant state." +- **Consequence:** A freshly-scaffolded or mid-flight loop (`.loop/state.json` present, + `terminal_state` = `null`, no `terminal_state.json` yet — the *correct* representation of an + unstarted or in-progress run) **fails `doctor`** with `missing RUNLOG.md`-class + `missing_file` on the terminal path. To make `doctor` green, an operator is pushed to write a + terminal file onto a loop that has not terminated — i.e. to fabricate a terminal state. The + gate incentivizes exactly the false completion the product exists to prevent. +- **Root shape:** the state schema already models non-terminal life correctly + (`schemas/state.schema.json:18-21`: `terminal_state` may be `null`). The gap is that + `validate_contract` treats the *terminal file* as unconditionally required rather than + "required iff `state.json.terminal_state != null`." + +--- + +## 3. Goals / non-goals + +### Goals + +1. **Publish the on-disk contract as a versioned, tool-agnostic standard** — a single + normative spec doc plus a complete set of published JSON Schemas that any harness (not just + Loop Engineer) can emit and validate against. +2. **Close M5** — extend `validate_contract` to check receipts and repair records against + published schemas when present, so the FCR/repair-productivity evidentiary trail is covered. +3. **Reconcile templates ↔ validator ↔ schemas** — fix DG-1, DG-2, DG-3 so the shipped + scaffold passes the shipped gate, and add regression coverage so the drift cannot silently + return. +4. **Add a non-terminal lifecycle state** so an unstarted / in-progress loop is a first-class + *conformant* state, not a validation failure. +5. **Publish a conformance checklist + version/stability note** so a third party can claim + "emits a Loop-Engineer-conformant contract v1" and know exactly what that means. + +### Non-goals + +- **Not** a runtime rewrite. The state machine, the 7 terminal states, and the file layout are + unchanged. This is a *format-standardization + validator-completeness* effort. +- **Not** the FCR/RP baseline itself (that is ST1, which depends on this). +- **Not** the integration recipes (ST3). +- **Not** re-litigating the 7 terminal states — they are canonical and frozen + (`loop/contract.py:9-17`). +- **Not** the honesty-enforcement fixes themselves (QW2 terminal cross-check, HI4 inspector + credit, HI5 `productive` recompute) — this spec makes the *schemas and validator surface* + those fixes bind to, and declares the dependency, but does not implement the cross-field + semantic rules (those are their own backlog items). + +--- + +## 4. The contract standard — versioning & stability model + +### 4.1 One normative document + +Publish a single canonical spec, `reference/repo-os-contract.md` already partially exists (the +manifest template at `templates/manifest.yaml.tmpl:2` cites `reference/repo-os-contract.md §10` +as the canonical schema source). ST2 promotes it to the **normative standard**: it must +enumerate every file in the contract, its schema `$id`, its required keys, and its lifecycle +role. The spec is the human-readable companion to the machine-readable `schemas/`. + +### 4.2 Schema identity and versioning + +The contract already uses embedded, versioned schema identifiers of the form +`loop-engineer/@` (`loop/contract.py:19-24`, and each `schemas/*.schema.json` +`$id`). ST2 formalizes this into a stability contract: + +- **Version is a single integer major**, embedded in the `$id` and in the `schema` key of every + artifact (`loop-engineer/state@1`, …). This is the version an external emitter targets. +- **Additive changes are minor and MUST NOT bump the major.** Adding an *optional* key, or a + new *optional* file, is backward compatible: every artifact schema already sets + `"additionalProperties": true` (confirmed in all five `schemas/*.schema.json`), so a v1 + consumer tolerates unknown keys from a newer emitter. +- **Breaking changes (removing/renaming a required key, changing a type, tightening an enum) + bump the major** to `@2` and get a new `$id`. Both majors may be published side by side. +- **A `SPEC_VERSION` / stability table** in the normative doc records, per artifact, the + current major and its stability tier (`stable` | `provisional`). Recommendation for v1: + manifest / state / tasks / terminal = **stable**; receipt / repair-record = **provisional** + (they are the newest surfaces and ST1 may still shape them). + +### 4.3 Stability note (normative text to publish) + +> The Loop-Engineer contract is a **portable, tool-agnostic on-disk standard**. Any surface +> that can read a repo, run a shell command, and write files can emit or consume it. Conformance +> is defined by the published JSON Schemas at `schemas/*.schema.json` (schema `$id` +> `loop-engineer/@`). Within a major version, changes are strictly additive +> and optional; a validator for major *N* accepts any artifact whose required keys and types +> match major *N*, ignoring unknown keys. Breaking changes ship as a new major with a new `$id`. + +--- + +## 5. Schema set — publish all six; canonicalize the repair record + +ST2's published schema set is **six** artifacts. Five exist; two need work (receipt gets wired +in; repair-record is net-new), and every existing schema must be reconciled with its template. + +| Artifact | Schema file | Status today | ST2 action | +|---|---|---|---| +| manifest | `schemas/manifest.schema.json` | published, validated | keep; confirm template parity (already correct) | +| state | `schemas/state.schema.json` | published, validated (partial) | **DG-1** template rename; widen validator to the schema's full `required` set (§6.3) | +| tasks | `schemas/tasks.schema.json` | published, validated | keep; template already correct | +| terminal | `schemas/terminal.schema.json` | published, validated | **DG-2** template completion; keep | +| receipt | `schemas/receipt.schema.json` | published but **never referenced** | **wire into `validate_contract`** (§7) — this is M5's core | +| **repair-record** | *(none)* | **does not exist** | **author + publish** (§5.1) — this is M5's other half | + +### 5.1 Repair-record schema — resolve the two-shapes conflict (QW11 / M4) first + +There are currently **two disjoint "7-field" shapes both branded "the" repair record** +(finding M4 / backlog QW11): + +- `scripts/rollout_ledger.py` `RECORD_FIELDS`: + `id` / `parent` / `verdict` / `score` / `score_delta` / `coherent_with_prior_winner` / + `productive` — a rollout/ledger shape. +- `evals/cases/structural.json` `repair_record_fields`: + `failure_mode` / `hypothesis` / `repair_action` / `verification_before` / + `verification_after` / `remaining_delta` / `productive` — a repair-diagnosis shape + (and the shape `skills/loop-repair/SKILL.md:71` prescribes). + +They share only the count "7" and the field `productive`. **ST2 cannot publish a repair-record +schema until this is resolved — QW11 is a hard dependency.** Recommendation for the spec author +to ratify: + +- **Canonical `repair-record@1` = the `structural.json` diagnosis shape** + (`failure_mode` / `hypothesis` / `repair_action` / `verification_before` / + `verification_after` / `remaining_delta` / `productive`). Rationale: it is the shape the + repair *skill* prescribes, the shape the eval structural-invariant already pins, and the one + the manifest template points at + (`templates/manifest.yaml.tmpl:24` → `.loop/artifacts/repair-record.json`). The + `rollout_ledger` shape is a *rollout-ledger* record (genome/rollout bookkeeping), which the + standard should name distinctly (`rollout-record`), not conflate with the repair record. +- **`verification_before` / `verification_after`** are objects carrying at least `score` + (number); `remaining_delta` is a number; `productive` is a boolean. +- **`productive` is derived, and the schema documents that it MUST equal + `verification_after.score > verification_before.score`** (per `eval-suite.md:62` "derived, + not self-reported"). The *schema* documents the invariant; the *recompute-and-reject + enforcement* is HI5 (declared dependency, not implemented here). + +Publishing this schema is what lets `validate_contract` check the repair trail (M5) and what +lets ST1 derive repair-productivity from a well-formed, single-shape record. + +### 5.2 Receipt schema — already publishable + +`schemas/receipt.schema.json` is well-formed and correctly describes the append-one-JSON-object- +per-line `.loop/receipts/*.jsonl` trail (role ∈ {read,reason,write,orchestrate}, model, +outcome ∈ {ok,fail,escalated}). ST2 does **not** reshape it; it wires it into the validator +(§7). Its `description` already states the portability intent ("Any JSONL receipt source that +carries these keys … works") — that sentence is the seed of the standard's cross-tool claim. + +--- + +## 6. Template ↔ validator ↔ schema reconciliation + +The single acceptance invariant for this section: **every file emitted verbatim from +`templates/` (placeholders filled with schema-valid values) passes `validate_contract` with +zero issues, and validates against its published JSON Schema.** A round-trip regression test +must pin this. + +### 6.1 DG-1 fix — rename `schema_version` → `schema` in the two JSON state templates + +- `templates/state.json.tmpl:2`: `"schema_version": "1.0"` → `"schema": "loop-engineer/state@1"`. +- `templates/terminal_state.json.tmpl:2`: `"schema_version": "1.0"` → + `"schema": "loop-engineer/terminal@1"`. +- The human-facing "version" is now carried by the schema `$id` major, not a parallel + `schema_version` key. If a scaffold timestamp/version is still wanted, keep it under a + distinct optional key (e.g. `scaffold_version`) that no validator keys on. + +### 6.2 DG-2 fix — complete the `terminal_state` template to the terminal schema + +`templates/terminal_state.json.tmpl` must emit **all** `terminal.schema.json:6` required keys: +`schema`, `state`, `iteration_id`, `criteria_met`, `evidence`, `false_completion`, `reason`, +`lessons_ref`. Concretely add: + +- `"criteria_met": { "{{CRITERION_1}}": {{CRITERION_1_MET}} }` (object of criterion→bool), +- `"evidence": ["{{EVIDENCE_PATH_1}}"]` (list of evidence paths/handles), +- `"false_completion": {{FALSE_COMPLETION}}` (bool), +- `"reason": "{{TERMINAL_REASON}}"`, +- `"lessons_ref": "{{LESSONS_REF}}"`. + +The existing `verification_evidence` / `lessons_learned` / `succeeded` keys may remain as +optional convenience fields (they are tolerated by `additionalProperties: true`), but the +**canonical proof surface is `criteria_met` + `evidence` + `false_completion`**, and the +template must lead with them so a scaffolded terminal file is proof-complete by construction. + +### 6.3 DG-1 corollary — widen `_validate_state` to the state schema's full required set, and fix `iteration_id` type + +Two latent template↔schema mismatches surface while fixing DG-1: + +- **`iteration_id` type.** `templates/state.json.tmpl:4` emits `"{{ITERATION_ID}}"` (a JSON + *string*), but `schemas/state.schema.json:9` requires `integer, minimum 0` + (`terminal.schema.json:10` likewise). ST2 must make the templates emit an unquoted integer + placeholder (`"iteration_id": {{ITERATION_ID}}`) so the scaffold validates against the schema. +- **Validator/schema `required` divergence.** `_validate_state` (`contract.py:138`) checks only + `iteration_id` / `state` / `plan_version` / `budget_remaining`, whereas + `state.schema.json:6` requires eleven keys (adds `active_task`, `best_score`, `failure_mode`, + `pending_approval`, `checkpoint_path`, `terminal_state`). ST2's standard resolves this by + making the **published JSON Schema the single source of truth** and having `_validate_state` + either (a) validate against the schema directly (preferred, see §7.2) or (b) check the same + eleven keys. Either way, template, validator, and schema converge on one required set. + +### 6.4 DG-3 fix — a non-terminal lifecycle state + +Introduce a first-class **non-terminal / pre-terminal** lifecycle concept so an unstarted or +in-flight loop is conformant without a terminal file: + +- **Contract rule:** `terminal_state.json` is **required iff `state.json.terminal_state != + null`.** When `state.json` reports a live (non-terminal) `state` and `terminal_state: null`, + the *absence* of `terminal_state.json` is **conformant**, not a `missing_file` issue. + `validate_contract` must gate the terminal read on the state's `terminal_state` field. +- **Lifecycle vocabulary (documented, not a new file schema):** the standard names the + non-terminal lifecycle values a loop's `state` field may hold before termination — at minimum + a **`Planned`** (contract scaffolded, not yet running) and **`Ready`/`Running`** (executing) + band — and states that these are *not* terminal states and never appear in the frozen + 7-member `terminal_state` enum. This gives `doctor` a way to report "conformant, not yet + terminated" instead of forcing a fabricated terminal file. +- **`doctor` output:** add a lifecycle line to the report (e.g. `"lifecycle": "planned"` | + `"running"` | `"terminated:"`) so an operator sees *why* no terminal file is + expected, closing the incentive to fabricate one. +- **Regression:** a test that a scaffolded loop with `terminal_state: null` and **no** + `terminal_state.json` passes `doctor` clean; and that a loop claiming a non-null + `terminal_state` **without** the terminal file still fails. + +--- + +## 7. Extending `validate_contract` (closing M5) + +### 7.1 Receipts + repair records, checked when present + +`validate_contract` (`contract.py:219-240`) gains two optional checks, invoked after the four +existing validators: + +- **`_validate_receipts(paths, issues)`** — if `.loop/receipts/*.jsonl` exists, parse each line + as JSON and validate against `receipt@1`. Malformed lines, wrong `schema`, or an out-of-enum + `role`/`outcome` emit an issue. Absent receipts are **not** an error (optional trail). +- **`_validate_repair_records(paths, issues)`** — if the repair-record file(s) exist + (`.loop/artifacts/repair-record.json` per manifest, and/or `.loop/artifacts/*repair*.json`), + validate each against the canonical `repair-record@1` (§5.1). Absent records are **not** an + error. +- **`schemas_checked`** in the return value (`contract.py:238`) must grow from the current 4 + (`SCHEMA_IDS`) to **all six** so the report honestly states its coverage. The "covers only 4 + of 5 shipped schemas" evidence for M5 is retired by this single change plus the new + repair-record schema. + +**"Optional when present" is the right severity:** a portable contract should validate a +mid-flight loop that has not yet produced receipts. ST2 makes malformed-trail a hard issue and +missing-trail a non-issue. (A future ST1 may layer a *stricter* profile that requires the trail +for a `Succeeded` terminal — noted as a hook, out of scope here.) + +### 7.2 Recommended: validate against the published JSON Schemas directly + +Today the validators are hand-rolled key checks that have already drifted from the published +schemas (§6.3). ST2 should make the published `schemas/*.schema.json` the **single source of +truth** and drive validation from them (a small pure-stdlib subset validator, or an optional +`jsonschema` dependency guarded like the optional-`yaml` import at `contract.py:106`). This +guarantees template↔validator↔schema can never diverge again, because there is exactly one +authority. If a full schema engine is undesirable in the stdlib core, the fallback is a +generated key/type check emitted *from* the schemas so drift is still mechanically impossible. +Either way, **the JSON Schema is normative; the validator conforms to it, not vice versa.** + +--- + +## 8. Conformance checklist (publishable) + +The standard ships a checklist a third party (or CI) can run to claim "Loop-Engineer contract +v1 conformant." Draft: + +**A. Artifacts present & well-formed** +- [ ] `.loop/manifest.yaml` validates against `loop-engineer/manifest@1` (incl. the canonical + 7 `terminal_states`, verbatim and in order). +- [ ] `.loop/state.json` validates against `loop-engineer/state@1` (`schema` key present, + `iteration_id` integer, all required keys present). +- [ ] `TASKS.json` validates against `loop-engineer/tasks@1`; no duplicate task ids; no task + `status:"done"` without `evidence`. +- [ ] `RUNLOG.md` present. + +**B. Lifecycle honesty** +- [ ] Exactly one of: (`terminal_state == null` **and** no `terminal_state.json`) **or** + (`terminal_state ∈` canonical 7 **and** `terminal_state.json` present & valid). No + terminal file on a non-terminated loop; no non-null `terminal_state` without the file. +- [ ] `terminal_state.json` (when present) validates against `loop-engineer/terminal@1` with a + real `criteria_met`, non-empty `evidence`, and an explicit `false_completion` boolean. + +**C. Evidentiary trail (checked when present)** +- [ ] Every `.loop/receipts/*.jsonl` line validates against `loop-engineer/receipt@1`. +- [ ] Every repair record validates against `loop-engineer/repair-record@1`, and `productive` + equals `verification_after.score > verification_before.score` (the derivation invariant; + hard-enforced by HI5). + +**D. Versioning** +- [ ] Every artifact's `schema` / `$id` names a *published, current-major* schema. +- [ ] Unknown keys are tolerated (a v1 validator does not reject a newer emitter's additive + fields). + +**E. Cross-field integrity (declared, enforced by sibling items)** +- [ ] A `Succeeded` terminal has `false_completion == false` **and** ≥1 true entry in + `criteria_met` (**QW2** — this spec surfaces the field; QW2 enforces the rule). + +--- + +## 9. Acceptance criteria (for the ST2 implementation that follows this spec) + +1. A single normative spec doc (`reference/repo-os-contract.md`, promoted) enumerates all six + artifacts, their `$id`s, required keys, lifecycle roles, and the §4.3 stability note. +2. Six published JSON Schemas exist and are current: manifest, state, tasks, terminal, receipt, + **repair-record** (net-new, canonical shape per §5.1, QW11 resolved). +3. `validate_contract` checks receipts and repair records against their schemas when present; + `schemas_checked` reports all six; M5's "4 of 5" evidence no longer reproduces. +4. **DG-1**: state & terminal templates emit `schema` (not `schema_version`) and validate clean. +5. **DG-2**: the terminal template emits `criteria_met` + `evidence` + `false_completion` + (+ `reason`, `lessons_ref`, `iteration_id`) and validates against `terminal@1`. +6. **DG-3**: a scaffolded loop with `terminal_state: null` and no `terminal_state.json` passes + `doctor`; a non-null `terminal_state` without the file still fails; `doctor` reports a + lifecycle line. +7. A round-trip regression test: **every `templates/*` artifact, filled and scaffolded, passes + `validate_contract` and its JSON Schema** — the drift class of DG-1/DG-2/DG-3 cannot silently + return. +8. A published conformance checklist (§8) exists and is runnable in CI against + `examples/coverage-repair` (and any second example). +9. `loop-contract` scaffold output validates in CI (per the ST2 backlog acceptance). + +--- + +## 10. Dependencies, risks, sequencing + +- **QW11 (canonical repair record) is a hard prerequisite for §5.1.** Do not publish + `repair-record@1` until the two-shapes conflict is ratified; publishing the wrong shape would + bake ambiguity into a *versioned standard*, the most expensive place to be wrong. +- **HI5 (recompute & reject `productive`)** consumes this spec's repair-record schema. §5.1 + *documents* the `productive` invariant; HI5 *enforces* it. Ship the schema first, then HI5. +- **QW2 / HI4 (false-completion enforcement)** consume DG-2. This spec makes the terminal + template *emit* `false_completion`; QW2 makes the validator *cross-check* it; HI4 makes the + inspector *credit only on real gate invocation*. Order: DG-2 → QW2 → HI4. +- **ST1 (published FCR/RP baseline) depends on this entire spec** — a baseline requires a + single well-formed repair-record shape (§5.1) and a validated receipt/repair trail (§7). Per + the backlog: *do not publish a baseline until the inputs are schema-checked and gate-backed.* +- **Risk — validator/schema authority drift (§7.2).** If the hand-rolled validators are kept + instead of driving from the schemas, the DG-class drift returns the next time a schema + changes. Mitigation: make the JSON Schema normative and the validator generated-from / driven- + by it, or at minimum add the §9.7 round-trip test as a CI gate. +- **Risk — over-scoping the receipt/repair check to "required."** Requiring the trail on every + loop would break mid-flight and inspect-only conformance. Mitigation: "checked when present" + (§7.1); a stricter "required-for-Succeeded" profile is a future, opt-in tier. +- **Competitive payoff is realized only if the standard is *documented for outsiders*.** The + moat (COMPETITIVE §Whitespace #4) is other harnesses emitting the format. The normative doc + + conformance checklist are what make that possible; schemas alone are necessary but not + sufficient. + +--- + +## 11. Out of scope (explicit) + +- The FCR/RP metrics computation and baseline (ST1). +- Integration recipes for LangGraph/Temporal/OpenHands/ruflo (ST3). +- The cross-field *enforcement* of `Succeeded ⇒ ¬false_completion ∧ criteria_met` (QW2), the + inspector-credit fix (HI4), the `productive` recompute (HI5), and the anticheat structural + invariant (HI6) — all declared as dependents, none implemented here. +- Any change to the 7 terminal states or the file layout. diff --git a/docs/superpowers/specs/2026-06-30-st3-integration-adapters.md b/docs/superpowers/specs/2026-06-30-st3-integration-adapters.md new file mode 100644 index 0000000..c12ad36 --- /dev/null +++ b/docs/superpowers/specs/2026-06-30-st3-integration-adapters.md @@ -0,0 +1,400 @@ +# ST3 — Integration Recipes & Optional Installable Adapters + +> **Spec type:** design (plan/docs deliverable — no implementation in this doc). +> **Backlog item:** ST3 (`review/IMPROVEMENT-BACKLOG.md` rank 16, Impact H / Effort H, Strategic). +> **Source of truth:** `review/POSITIONING.md` §5 (three-tier stack) · `review/COMPETITIVE-ANALYSIS.md` (positioning map + whitespace) · `review/IMPROVEMENT-BACKLOG.md` ST3. +> **Date:** 2026-06-30 · **Status:** proposed, unimplemented. + +--- + +## 1. Problem & wedge + +The competitive scan is unambiguous (`COMPETITIVE-ANALYSIS.md`): the entire +agent-tooling market — **LangGraph (36k★), ruflo (62k★), OpenHands (79k★), +Temporal (21k★)**, plus AutoGen/CrewAI/native `/loop` — lives in the bottom two +tiers of the stack. They **execute** and **orchestrate** loops extremely well, +then hand *"is it actually finished?"* back to the agent's own say-so. None ships +a typed terminal-state taxonomy, an evidence-before-completion gate, or first-class +`false-completion-rate` / `repair-productivity`. + +Loop Engineer owns the missing top tier (`POSITIONING.md` §5): + +``` +┌──────────────────────────────────────────────────────────────┐ +│ CONTRACT ← Loop Engineer: what "done" means, what proves │ +│ it, when repair is allowed, how it must end. │ +├──────────────────────────────────────────────────────────────┤ +│ ORCHESTRATE ← LangGraph · AutoGen · CrewAI · ruflo │ +├──────────────────────────────────────────────────────────────┤ +│ EXECUTE ← OpenHands · Temporal · /loop · /goal │ +└──────────────────────────────────────────────────────────────┘ +``` + +**The wedge for ST3: "composes, it doesn't compete."** The layer claim only +converts to adoption if a stranger already on LangGraph/ruflo/OpenHands/Temporal +can *bolt Loop Engineer on top in one file* without swapping their runtime. ST3 +delivers exactly that: concrete recipes that wrap an existing engine's terminal +node in Loop Engineer's typed-termination + evidence gate, turning each 36k–79k★ +competitor into **a complement that needs you**, not a rival that buries you. + +**Non-goal (positioning discipline, `POSITIONING.md` §5, §7):** never re-implement +or replace an execution engine. Every recipe keeps the host engine as the executor +and adds only the contract/proof layer above it. + +--- + +## 2. Objective & scope + +**Objective.** Ship integration recipes for **≥2** of the four target engines, +each showing the *same two-part mapping*: + +1. **engine terminal → one of Loop Engineer's 7 typed terminal states**, and +2. **engine run artifacts → a Loop Engineer evidence record** (`terminal_state.json` + conforming to `schemas/terminal.schema.json`, plus `.loop/receipts/*.jsonl` + per `schemas/receipt.schema.json`). + +**In scope.** +- Four recipes (this spec designs all four; the acceptance bar requires ≥2 to ship + runnable): LangGraph, Temporal, OpenHands, ruflo. +- A shared **adapter contract** (§4) so every recipe maps through one code path + (no per-engine drift in how a terminal state or an evidence record is produced). +- An optional, **BYO-friendly** installable helper module (§6) — pure-stdlib, + additive, zero framework lock-in — that each recipe imports; recipes must also + degrade to a copy-paste-able snippet with no install. +- Docs home: `reference/integrations/.md` (per-recipe) + this spec. + +**Out of scope (defer / other backlog items).** +- Publishing an FCR/RP *baseline* (ST1) — recipes emit the evidence trail ST1 + later aggregates, but ST3 ships **no headline numbers**. +- The versioned portable-contract spec + schema conformance CLI (ST2) — ST3 + *consumes* the existing `schemas/*.json` as-is and flags any gap it hits back + to ST2. +- Any change to `loop/`, `scripts/`, `schemas/`, `templates/`, `evals/` — recipes + are additive `reference/` docs + one optional helper module; they must not + modify the core gate logic. + +--- + +## 3. The mapping (the load-bearing design decision) + +Every recipe is a projection from an engine's native "the run ended" signal onto +the two Loop Engineer artifacts. The taxonomy is fixed (`loop/contract.py` +`TERMINAL_STATES`); the recipe's whole job is the projection function. + +### 3.1 Engine terminal → 1 of 7 typed terminal states + +Canonical target set (verbatim from `loop/contract.py:9-17`): +`Succeeded`, `FailedUnverifiable`, `FailedBlocked`, `FailedBudget`, +`FailedSafety`, `FailedSpecGap`, `AbortedByHuman`. + +The projection is **never** "the engine said done → `Succeeded`." A raw engine +terminal maps to `Succeeded` **only** through Loop Engineer's own gate +(`scripts/holdout_gate.py` `decide()` + `scripts/anticheat_scan.py`). This is the +entire wedge: the engine's say-so is demoted to an *input*, not the verdict. + +| Engine terminal signal | Gate result | → Typed terminal state | +|---|---|---| +| run reached its end node / workflow returned | holdout green + anticheat clean | **`Succeeded`** | +| run reached its end node | visible green, **holdout red** | **`FailedUnverifiable`** (the false-completion event) | +| run reached its end node | holdout not defined / not runnable | **`FailedUnverifiable`** (cannot certify) | +| anticheat: gate-tampering (CRITICAL) | — | **`FailedSafety`** | +| anticheat: hidden-answer / skip-injection (HIGH) | — | **`FailedUnverifiable`** | +| engine raised an unrecoverable external error / missing credential / locked resource | — | **`FailedBlocked`** | +| engine hit step/token/wall-clock/cost budget cap | — | **`FailedBudget`** | +| run finished but a SPEC criterion has no mapped check at all | — | **`FailedSpecGap`** | +| operator interrupt / human abort signal | — | **`AbortedByHuman`** | + +`holdout_gate.decide()` already returns exactly the three verdicts the top rows +need (`Succeeded` / `FailedUnverifiable` / `NotReady`) plus a `false_completion` +boolean. The adapter's projection table is the *only* new logic; the gate is +reused verbatim. + +### 3.2 Engine artifacts → evidence record + +The recipe writes a `terminal_state.json` matching `schemas/terminal.schema.json` +(`terminal@1`), required keys: `schema, state, iteration_id, criteria_met, +evidence, false_completion, reason, lessons_ref`. Mapping: + +- `state` ← §3.1 projection. +- `criteria_met` ← `{criterion_id: bool}` from mapping each SPEC criterion to a + gate check id (a criterion with no check → `FailedSpecGap`, per §3.1). +- `evidence` ← list of on-disk paths the engine produced that the gate consumed + (engine run log, test output, holdout gate JSON). Recipes **copy or reference** + the engine's native artifacts into `.loop/artifacts/` rather than fabricating. +- `false_completion` ← `holdout_gate` result's `false_completion` (never a hand-set + literal — this is the exact hole HI4/QW2 flag; recipes must source it from the + gate call). +- Plus one `.loop/receipts/*.jsonl` line per dispatch (`receipt@1`: `role`, `model`, + `outcome`, optional `tokens`/`cost_usd`) so ST1/`loop-flywheel` can later compute + cost-per-success from the engine's own model calls. + +> **Cross-check invariant (QW2 / gate-integrity HIGH).** A recipe MUST NOT emit +> `state: "Succeeded"` with `false_completion: true` or empty `criteria_met`. The +> adapter derives `Succeeded` only from a green gate, so this is structurally +> unreachable — but each recipe's test pins it. + +--- + +## 4. Adapter contract (one code path for all recipes) + +To prevent per-engine drift, every recipe funnels through one small, engine-neutral +projection surface. Conceptually (Python-shaped, pure-stdlib, no engine import): + +``` +# reference/integrations/_adapter.py (optional installable helper — §6) + +@dataclass(frozen=True) +class EngineOutcome: + """Engine-agnostic description of how a host run ended.""" + reached_end: bool # engine's own terminal signal + external_error: str | None # unrecoverable env/credential/resource block + budget_exhausted: bool # step/token/wall-clock/cost cap hit + human_abort: bool # operator interrupt + artifacts: list[str] # on-disk paths the run produced + +def to_terminal_state( + outcome: EngineOutcome, + gate_verdict: dict, # from holdout_gate.decide(...) + anticheat: dict, # from anticheat_scan (findings, clean) + criteria_met: dict[str, bool], # SPEC-criterion -> mapped-check pass/fail +) -> dict: # a terminal_state.json body (terminal@1) + ... +``` + +`to_terminal_state` implements §3.1 precedence **safety → human → blocked → +budget → spec-gap → gate verdict** and assembles the §3.2 evidence record. It +imports nothing from any engine; the recipes adapt each engine's native result +object into an `EngineOutcome` and pass the gate outputs through. This keeps the +"how a terminal state is decided" logic in exactly one tested place — recipes only +translate *shapes*, never *policy*. + +Precedence rationale: `FailedSafety` (tampering) and `AbortedByHuman` must win over +any green gate so a gamed or human-killed run can never launder itself into +`Succeeded`. + +--- + +## 5. Recipes + +Each recipe ships as `reference/integrations/.md` with: (a) a 3-line +"what this composes" header naming the tier, (b) the mapping table specialized to +the engine's terminal signal, (c) a runnable snippet outline, (d) the resulting +`terminal_state.json` + one receipt line, (e) a copy-paste test asserting the +false-completion invariant. + +### 5.1 LangGraph terminal-node → typed terminal state *(flagship, ship first)* + +**Composes:** the ORCHESTRATE tier. LangGraph's graph runs to a terminal node and +returns whatever the node returns; Loop Engineer replaces "return the state" with +"gate the state, then emit a typed terminal." + +Snippet outline: + +```python +from langgraph.graph import StateGraph, END +from loop_engineer.integrations import EngineOutcome, to_terminal_state +from scripts.holdout_gate import decide +from scripts import anticheat_scan # trajectory + diff sweep + +def certify_node(state: dict) -> dict: + # 1. run the SAME visible/holdout split the loop optimized against + gate = decide(visible=state["visible_results"], holdout=state["holdout_results"]) + ac = anticheat_scan.scan(diff=state["diff"], trajectory=state["tool_trail"]) + # 2. project the graph's terminal into a typed state + evidence + terminal = to_terminal_state( + outcome=EngineOutcome(reached_end=True, external_error=None, + budget_exhausted=state["step"] >= state["max_steps"], + human_abort=False, artifacts=state["artifacts"]), + gate_verdict=gate, anticheat=ac, criteria_met=state["criteria_met"], + ) + write_terminal_state(terminal) # -> terminal_state.json (terminal@1) + append_receipt(role="orchestrate", model=state["model"], outcome="ok") + return {**state, "terminal": terminal} + +graph.add_node("certify", certify_node) +graph.add_edge("certify", END) # certify IS the only path to END +``` + +Mapping specialization: LangGraph's `END` is reachable **only** through +`certify_node`; the graph can no longer terminate on the agent's own return. A +`GraphRecursionError` (LangGraph's own step cap) maps to `FailedBudget`; a caught +tool/credential exception maps to `FailedBlocked`. + +### 5.2 Temporal workflow → repo-OS contract *(flagship, ship first)* + +**Composes:** the EXECUTE tier (durable execution). Temporal guarantees the run +*survives crashes*; it says nothing about whether the work is *correct* +(`COMPETITIVE-ANALYSIS.md`: low on the verification axis by design). Loop Engineer +adds the correctness/termination contract on top. + +Snippet outline: a final **`certify` activity** at the end of the workflow (activities +can do I/O; the workflow stays deterministic): + +```python +@activity.defn +async def certify_activity(run: RunArtifacts) -> dict: + gate = decide(visible=run.visible, holdout=run.holdout) + ac = anticheat_scan.scan(diff=run.diff, trajectory=run.trail) + terminal = to_terminal_state( + outcome=EngineOutcome(reached_end=True, external_error=run.error, + budget_exhausted=run.attempts >= run.cap, + human_abort=run.cancelled, artifacts=run.artifacts), + gate_verdict=gate, anticheat=ac, criteria_met=run.criteria_met, + ) + write_terminal_state(terminal); append_receipt(...) + return terminal + +@workflow.defn +class GoalWorkflow: + @workflow.run + async def run(self, spec) -> dict: + artifacts = await workflow.execute_activity(do_work, spec, ...) + return await workflow.execute_activity(certify_activity, artifacts, ...) +``` + +Mapping specialization: a Temporal `CancelledError` (workflow cancellation) → +`AbortedByHuman`; an activity that exhausts its `RetryPolicy` on an external +dependency → `FailedBlocked`; the workflow's own timeout → `FailedBudget`. The +`.loop/state.json` (`state@1`) is written from the workflow's durable state so a +resumed workflow resumes the *same* contract — this is the "repo-OS contract" +mapping: Temporal owns durability, Loop Engineer owns the on-disk success/evidence +truth. + +### 5.3 OpenHands run → FCR gate *(alternate)* + +**Composes:** the EXECUTE tier (autonomous coding runtime). OpenHands writes, runs, +and tests code in a sandbox — incidental verification, but "done" is still the +agent stopping. Loop Engineer wraps the run's exit in the false-completion gate. + +Snippet outline: a post-run hook that reads the OpenHands trajectory (event +stream / final state) as the `anticheat` trajectory input, runs the holdout split +against the sandbox, and projects: + +```python +result = openhands.run(task=spec) # sandboxed agent run +gate = decide(visible=run_visible(result), holdout=run_holdout(result)) +ac = anticheat_scan.scan(diff=result.git_diff, trajectory=result.event_paths) +terminal = to_terminal_state( + outcome=EngineOutcome(reached_end=result.finished, external_error=result.fatal, + budget_exhausted=result.iterations >= result.max_iterations, + human_abort=False, artifacts=[result.log_path]), + gate_verdict=gate, anticheat=ac, criteria_met=map_criteria(result), +) +``` + +Mapping specialization: OpenHands' `AgentStuckError` / max-iteration stop → +`FailedBudget`; a sandbox that touched a holdout/answer-key path (HIGH anticheat +finding) → `FailedUnverifiable` — the exact "the runtime ran tests but the agent +peeked" case OpenHands can't itself catch. + +### 5.4 ruflo swarm → acceptance gate *(alternate)* + +**Composes:** the ORCHESTRATE tier (multi-agent swarm). A swarm's terminal is +"the coordinator decided the objective is met" — pure self-report across N agents. +Loop Engineer adds a single acceptance gate the swarm must pass *as a whole*. + +Snippet outline: register the gate as the swarm's terminal hook (ruflo exposes +hooks / an MCP coordination server), so no individual agent can declare the swarm +done — the acceptance gate does: + +```python +swarm_result = ruflo.orchestrate(objective=spec, agents=[...]) +gate = decide(visible=swarm_result.visible, holdout=swarm_result.holdout) +ac = anticheat_scan.scan(diff=swarm_result.merged_diff, + trajectory=swarm_result.agent_trails) +terminal = to_terminal_state( + outcome=EngineOutcome(reached_end=swarm_result.converged, external_error=None, + budget_exhausted=swarm_result.rounds >= swarm_result.max_rounds, + human_abort=False, artifacts=swarm_result.artifacts), + gate_verdict=gate, anticheat=ac, criteria_met=swarm_result.criteria_met, +) +``` + +Mapping specialization: swarm non-convergence within max rounds → `FailedBudget`; +a criterion no agent was assigned → `FailedSpecGap` (the swarm literally never +worked on it — a failure mode a self-reporting coordinator hides). + +--- + +## 6. Optional installable adapter (BYO, additive) + +Per `POSITIONING.md` §3 (P2 persona: "Bring your own runtime. Loop Engineer is the +contract above it"), each recipe should work in two modes: + +1. **Zero-install copy-paste** — the snippet inlines the ~15-line `to_terminal_state` + projection; a reader on any stack pastes it. This is the default and the wedge + demo (no framework lock-in, pure-stdlib). +2. **Installable helper** — `loop_engineer.integrations` (shipped by the existing + editable `pyproject.toml`, no new dependency) exporting `EngineOutcome` + + `to_terminal_state` + thin `write_terminal_state` / `append_receipt` writers. + Optional convenience only; imports nothing from any engine, so installing it + never pulls LangGraph/Temporal/etc. + +**Discipline:** the helper is a *projection + writer*, never an executor. It must +not import, wrap, or vendor any engine — that would forfeit the "composes, doesn't +compete" claim. Engine packages stay the host app's own dependency. + +--- + +## 7. Acceptance criteria + +Mirrors `IMPROVEMENT-BACKLOG.md` ST3 ("recipes for ≥2 of {…}, each with a runnable +snippet and the resulting terminal-state/evidence mapping"): + +1. `reference/integrations/` carries recipes for **≥2** target engines (LangGraph + and Temporal are the flagship pair; OpenHands/ruflo may ship in the same PR or + follow). +2. Each shipped recipe has: the specialized mapping table (engine terminal → 1 of + 7 states), a runnable snippet outline, and a concrete resulting + `terminal_state.json` (valid against `terminal@1`) + one `receipt@1` line. +3. Each recipe includes a test (or worked example) asserting the **false-completion + invariant**: a run whose visible checks pass but holdout fails maps to + `FailedUnverifiable` with `false_completion: true`, and **never** to + `Succeeded` — pinning QW2/HI4 at the integration boundary. +4. `Succeeded` is emitted only when `holdout_gate.decide()` returns `Succeeded` + **and** anticheat is clean; `criteria_met` has ≥1 true entry. (Structurally + guaranteed by §4; test-pinned.) +5. The optional helper imports zero engine packages; each recipe also works as a + pure copy-paste snippet with no install. `self_eval.py` stays green; no file + under `loop/ scripts/ schemas/ templates/ evals/` is modified. +6. Every recipe's prose frames the engine as a **complement** and never claims to + replace it (`POSITIONING.md` §5/§7 discipline). + +--- + +## 8. Risks & open questions + +- **Engine API drift.** LangGraph/Temporal/OpenHands/ruflo APIs move fast; snippets + are *outlines*, and per-engine version pins + a "verified against vX" note belong + in each recipe (library-research discipline: confirm current API via Context7 / + primary docs before finalizing a snippet). This spec does not pin versions. +- **Trajectory availability.** The anticheat sweep needs the engine's tool/path + trail; engines expose it differently (LangGraph state, Temporal history, + OpenHands event stream, ruflo agent trails). Where a trail is unavailable, the + recipe must map to `FailedUnverifiable` (cannot certify), not silently skip the + anticheat step — the same fail-closed posture as `holdout_gate` on an empty + holdout set. +- **Depends-on / feeds-into.** ST3 is standalone-shippable but is strongest after + QW2 + HI4 (so the gate the recipes call is itself honest) and feeds ST1 (the + receipts/evidence trail it emits is what a published FCR baseline aggregates). + It also surfaces the ST2 need: recipes want a *versioned* on-disk contract to + target; any schema gap they hit is an ST2 input. +- **Which alternate ships.** OpenHands (79k★, largest EXECUTE community) vs ruflo + (62k★, largest CC-native swarm) as the third recipe — decide by whichever + community the launch (ST5) targets first. + +--- + +## 9. Traceability + +| This spec | Grounded in | +|---|---| +| Three-tier stack framing | `POSITIONING.md` §5 | +| "composes, doesn't compete" wedge | `POSITIONING.md` §1, §5, §7 · `COMPETITIVE-ANALYSIS.md` whitespace §4 | +| Target engines (LangGraph/Temporal/OpenHands/ruflo) as layers-below | `COMPETITIVE-ANALYSIS.md` ADJACENT table + positioning map Q3 | +| 7 typed terminal states | `loop/contract.py:9-17` · `schemas/terminal.schema.json` | +| holdout/false-completion gate reuse | `scripts/holdout_gate.py` `decide()` | +| anticheat → FailedSafety/FailedUnverifiable | `scripts/anticheat_scan.py` docstring | +| evidence + receipt record | `schemas/terminal.schema.json` · `schemas/receipt.schema.json` | +| false-completion invariant at the boundary | `IMPROVEMENT-BACKLOG.md` QW2, HI4 | +| Acceptance bar (≥2 recipes, runnable snippet + mapping) | `IMPROVEMENT-BACKLOG.md` ST3 | diff --git a/docs/superpowers/specs/2026-06-30-v04-credibility-enforcement.md b/docs/superpowers/specs/2026-06-30-v04-credibility-enforcement.md new file mode 100644 index 0000000..a888974 --- /dev/null +++ b/docs/superpowers/specs/2026-06-30-v04-credibility-enforcement.md @@ -0,0 +1,547 @@ +# v0.4 — Enforce the Wedge: Credibility-Enforcement Spec + +**Status:** SPEC (design + tests-to-write). Not an implementation. +**Date:** 2026-06-30 +**Sources of truth:** `review/REVIEW.md`, `review/IMPROVEMENT-BACKLOG.md`, +`review/POSITIONING.md`, `review/COMPETITIVE-ANALYSIS.md`. +**Scope of this spec:** the four fixes that convert Loop Engineer's headline +differentiator — *"they execute the loop; Loop Engineer proves when it's done"* — +from **self-asserted** to **enforced**. Covers findings **G1**, **M1 + HI4**, +**M2 + HI1**, **HI6**. + +--- + +## 0. Why these four, and why together + +The review's central finding (REVIEW §1, §3.5 note, IMPROVEMENT-BACKLOG "Honesty +note on the wedge") is that **false-completion defense is, in load-bearing paths, +narrated rather than enforced**: a loop can self-assert `false_completion:false` +and receive full credit — from the contract validator, from the inspector score, +and in the flagship example — without any held-out or anti-cheat gate ever +running. And the one script whose job is to catch a gamed gate cannot detect +being gamed itself. + +Each fix closes one leak in the same pipe: + +| Fix | Finding(s) | Severity | The leak it plugs | +|---|---|---|---| +| G1 | REVIEW G1 / QW2 | HIGH | Contract validator accepts a self-contradictory `Succeeded` terminal. | +| M1 + HI4 | REVIEW M1 / HI4 | HIGH | Inspector awards anti-cheat credit for a bare terminal flag. | +| M2 + HI1 | REVIEW M2, O1 / HI1 | HIGH | Flagship example's `false_completion:false` is gate-less. | +| HI6 | REVIEW G3 / HI6 | MEDIUM | Anti-cheat scanner cannot detect a logic rewrite of its own gate functions. | + +They interlock: **M2/HI1 makes the example gate-backed**, which is a precondition +for **M1/HI4's tightened inspector rule** to still grant the example credit +(otherwise tightening M1 would make the repo's own demo fail its own check). +Ship them in one v0.4 batch. Every fix below ships with a regression test that is +**RED on today's code and GREEN after the fix**, pinning the exact repro quoted +in REVIEW.md. + +**Test invocation (canonical for this repo):** +``` +uv run --with pytest --with pyyaml python -B -m pytest -q scripts loop +``` +Green baseline before this work: 105 passed / `self_eval` 13/13. + +--- + +## 1. G1 — Contract validator must cross-check the terminal state + +### 1.1 Problem (file:line + repro) + +`loop/contract.py:169` `_validate_terminal()` validates only *field types* — that +`state` is one of the 7 canonical states, `criteria_met` is a dict, `evidence` is +a list, `false_completion` is a bool. It never checks that a **`Succeeded`** +terminal is internally consistent. (REVIEW **G1**, HIGH, CONFIRMED; +IMPROVEMENT-BACKLOG **QW2**.) + +Verified repro (REVIEW G1, `loop/contract.py:169-180`): + +```python +_validate_terminal( + {'schema': 'loop-engineer/terminal@1', 'state': 'Succeeded', + 'criteria_met': {}, 'evidence': [], 'false_completion': True}, + path, issues, +) +# -> issues == [] (validates clean; doctor/validate_contract report ok) +``` + +A `terminal_state.json` can claim victory (`Succeeded`) while simultaneously +flagging itself a false completion (`false_completion:true`) and listing **zero** +met criteria — and `validate_contract()` (`loop/contract.py:219`) / +`doctor_report()` (`loop/contract.py:243`) pass it clean. For the P3 evaluator +who runs `doctor` to decide whether the proof layer is real, this single repro +refutes the entire wedge. + +### 1.2 Enforcement design + +Add a **cross-field consistency rule** to `_validate_terminal`, after the +existing type checks, gated on `state == "Succeeded"` so failure-state terminals +(which legitimately carry `false_completion:true` and/or empty `criteria_met`) +are never touched: + +```python +# after the four existing type checks in _validate_terminal: +if data.get("state") == "Succeeded": + if data.get("false_completion") is not False: + issues.append(ContractIssue( + "contradictory_terminal", + "Succeeded terminal requires false_completion == false", + path, + )) + criteria = data.get("criteria_met") + if not (isinstance(criteria, dict) and any(v is True for v in criteria.values())): + issues.append(ContractIssue( + "contradictory_terminal", + "Succeeded terminal requires >=1 true entry in criteria_met", + path, + )) +``` + +Rules, stated normatively: + +- **R1** — `state == "Succeeded"` ⇒ `false_completion` **is** `False` (exactly + `False`, not merely falsy — `None`/missing must fail, since the type check + already requires bool but a self-contradictory file must not slip through on a + missing key). +- **R2** — `state == "Succeeded"` ⇒ `criteria_met` is a dict with **≥1** entry + whose value is exactly `True`. +- **R3** — Non-`Succeeded` terminals are **exempt** from R1/R2 (a `FailedBudget` + run may honestly report `false_completion:true` and `criteria_met:{}`). + +Both R1 and R2 emit the same issue code `contradictory_terminal` so +`doctor`/`validate_contract` surface it as one class. `SCHEMA_IDS` / +`schemas_checked` are unaffected (still the same 4 validators). + +### 1.3 Tests — RED then GREEN + +Add to `scripts/test_loop_contract_core.py`: + +1. **`test_validate_terminal_rejects_succeeded_with_false_completion_true`** + — pins the exact REVIEW G1 repro: `Succeeded` + `false_completion:True` + + `criteria_met:{}`. Asserts `validate_contract`/`_validate_terminal` returns + **≥1** issue whose code is `contradictory_terminal`. + *RED today* (repro returns `[]`) → *GREEN after fix*. +2. **`test_validate_terminal_rejects_succeeded_with_empty_criteria_met`** + — `Succeeded` + `false_completion:False` + `criteria_met:{}` (isolates R2). + Asserts a `contradictory_terminal` issue. *RED → GREEN.* +3. **`test_validate_terminal_rejects_succeeded_with_all_false_criteria`** + — `Succeeded` + `false_completion:False` + `criteria_met:{"1": false}` + (no *true* entry). Asserts a `contradictory_terminal` issue. *RED → GREEN.* +4. **`test_validate_terminal_accepts_consistent_succeeded`** (guard, must be + GREEN after) + — `Succeeded` + `false_completion:False` + `criteria_met:{"1": true}` ⇒ no + `contradictory_terminal` issue. Prevents over-blocking. +5. **`test_validate_terminal_allows_failure_state_with_false_completion_true`** + (guard for R3, must be GREEN after) + — `state:"FailedBudget"` + `false_completion:True` + `criteria_met:{}` ⇒ no + `contradictory_terminal` issue. Proves failure terminals are exempt. +6. **`test_doctor_flags_contradictory_terminal_from_disk`** + — write the repro `terminal_state.json` into a `tmp_path` contract fixture, + run `doctor_report(tmp_path)`, assert `ok is False` and the issue surfaces + end-to-end. *RED → GREEN.* + +The existing `test_loop_doctor_accepts_valid_contract_*` tests +(`scripts/test_loop_contract_core.py:137,154`) must **stay green** — their +terminal fixtures already carry `false_completion:false` + a true criterion. + +### 1.4 Acceptance + +- The REVIEW G1 repro now yields ≥1 `contradictory_terminal` issue; `doctor` / + `validate_contract` report `ok:false` for it. +- Consistent `Succeeded` terminals and all non-`Succeeded` terminals validate + unchanged. +- Full suite green including the two pre-existing valid-contract doctor tests. + +--- + +## 2. M1 + HI4 — Inspector grants false-completion-defense credit only on detectable gate invocation + +### 2.1 Problem (file:line + repro) + +`scripts/inspect_loop.py:187-193` computes the `false_completion_defense` +checklist item (worth 14 of 100 points, `_CHECKS` in `inspect_loop.py`): + +```python +has_false_completion = ( + _script_exists(paths.workspace, "holdout_gate.py", "anticheat_scan.py", "anti_cheat.py") + or terminal.get("false_completion") is False # <-- the leak (line 189) + or "verifier_gaming" in str(manifest).lower() + or "false-completion" in workflow + or "false_completion" in workflow +) +``` + +The `terminal.get("false_completion") is False` disjunct awards **full** +anti-cheat credit whenever a `terminal_state.json` merely *states* the flag — with +no check that a held-out / anti-cheat gate ever ran. That is exactly the +self-reported signal `reference/eval-suite.md:62` says the metric is meant to +*replace*. (REVIEW **M1**, HIGH, CONFIRMED; IMPROVEMENT-BACKLOG **HI4**.) + +The mere-`_script_exists` disjunct is the same class of leak one step weaker: +credit for a gate file **existing** in `scripts/`, not for it being **invoked**. +Per the finding's own wording — *"only when … actually invoked … not merely +exists"* — both are tightened. + +### 2.2 Enforcement design + +Replace the bare-flag and mere-existence disjuncts with an **invocation-detection** +signal. Add a helper `_gate_invoked(paths) -> bool` that returns `True` iff a +reference to a held-out / anti-cheat gate (`holdout_gate`, `anticheat_scan`, +`anti_cheat`) is textually present in **any** of the loop's *execution-trail* +artifacts — the surfaces that only get a gate name written into them when the +gate was actually wired into a run: + +- each existing verify-* script's text + (`verify-fast[.sh]`, `verify-full[.sh]`, `verify-safety[.sh]`), and +- `RUNLOG.md` (the append-only iteration trail; the example writes the gate into + its Terminal block), and +- any `TASKS.json` task `verify` command string. + +```python +_GATE_TOKENS = ("holdout_gate", "anticheat_scan", "anti_cheat") + +def _gate_invoked(paths) -> bool: + haystacks = [_read_text(s) for s in _verify_script_paths(paths.workspace)] + haystacks.append(_read_text(paths.runlog)) + haystacks.extend( + str(row.get("verify", "")) + for row in _read_json_object(paths.tasks).get("tasks", []) + if isinstance(row, dict) + ) + blob = "\n".join(haystacks).lower() + return any(tok in blob for tok in _GATE_TOKENS) +``` + +New credit rule for `false_completion_defense`: + +```python +has_false_completion = ( + _gate_invoked(paths) # invocation, not existence + or "verifier_gaming" in str(manifest).lower() # manifest declares the policy + or "false-completion" in workflow # WORKFLOW documents the defense + or "false_completion" in workflow +) +``` + +Normative rules: + +- **R1** — A bare `terminal_state.json.false_completion == false` grants **no** + credit on its own (the `terminal.get(...) is False` disjunct is **removed**). +- **R2** — Credit is granted when a gate token is detectable in a verify script, + `RUNLOG.md`, or a `TASKS.json` verify command (evidence the gate was run). +- **R3** — The manifest `verifier_gaming` policy and `WORKFLOW.md` + false-completion prose remain valid credit sources (they are contract-owned + declarations of the defense, not per-run self-report), matching REVIEW M1/HI4 + which scope the fix to the *terminal flag*. + +Design note: mere `_script_exists(... holdout_gate.py ...)` is intentionally +**dropped** as a standalone credit source per the finding's "actually invoked … +not merely exists" wording; a gate file that is never referenced by any verify +script / RUNLOG / task earns nothing. + +### 2.3 Tests — RED then GREEN + +Add to `scripts/test_inspect_loop.py` (build minimal loop dirs under `tmp_path` +exercising `_evaluate_contract_checks(loop_dir)["false_completion_defense"]`): + +1. **`test_false_completion_defense_denied_for_bare_terminal_flag`** — pins the + HI4 "flag set but gate never run → no credit" repro: dir has + `terminal_state.json {"false_completion": false}` and **no** gate reference in + any verify script / RUNLOG / TASKS / manifest / WORKFLOW. Assert credit is + `False`. *RED today* (line 189 grants `True`) → *GREEN after fix*. +2. **`test_false_completion_defense_denied_when_gate_only_exists_never_invoked`** + — `scripts/holdout_gate.py` file present but referenced nowhere. Assert credit + `False`. *RED today* (`_script_exists` grants `True`) → *GREEN after fix*. +3. **`test_false_completion_defense_granted_when_gate_invoked_in_runlog`** — + `RUNLOG.md` mentions `holdout_gate.py`; no terminal flag, no script file. + Assert credit `True`. *RED today* (no current disjunct matches) → + *GREEN after fix* (proves the RUNLOG invocation path is wired). +4. **`test_false_completion_defense_granted_when_gate_invoked_in_verify_script`** + — `scripts/verify-full` text calls `anticheat_scan.py`. Assert credit `True`. + *GREEN after fix.* +5. **`test_false_completion_defense_granted_from_manifest_policy`** (guard, R3) + — manifest contains `verifier_gaming`, no terminal flag. Assert credit `True` + (unchanged behavior preserved). *GREEN before and after.* + +### 2.4 Acceptance + +- A bare `false_completion:false` flag, and a merely-present-but-uninvoked gate + file, both earn **no** `false_completion_defense` credit. +- Credit is earned when a gate token is detectable in a verify script, `RUNLOG.md`, + or a task verify command (or via the manifest/WORKFLOW declaration). +- Only the `false_completion_defense` check changes; the other four checklist + items and the terminal-coverage score are byte-for-byte unaffected (existing + `scripts/test_inspect_loop.py` cases for those stay green). +- **Cross-dependency:** after §3 (M2/HI1) lands, the flagship example must earn + this credit via real invocation — see §3.3 test 4. + +--- + +## 3. M2 + HI1 — Flagship example must actually run the held-out gate + +### 3.1 Problem (file:line + repro) + +`examples/coverage-repair/terminal_state.json:10` asserts +`"false_completion": false`, but: + +- the only cited evidence files (`.loop/artifacts/verify-T1.json`, + `.loop/artifacts/verify-T2.json`) contain plain pass/score data with **no** + `holdout_gate` / `anticheat` reference **anywhere** under + `examples/coverage-repair/` (verified: `grep -rl 'holdout\|anticheat' + examples/coverage-repair` → empty); and +- `examples/coverage-repair/TASKS.json:12,20` reference `scripts/verify-fast` / + `scripts/verify-full`, which exist only at **repo root**, not under the example + dir — so the paths are broken relative to the example itself + (`examples/coverage-repair/README.md:44` admits it "ships the loop artifacts, + not the target repo"). + +So the repo's own flagship demo never wires the held-out check the suite claims +makes FCR *measured, not narrated*, undermining `README.md:277` **by example**, +and a newcomer never watches the FSM iterate/repair/verify live. (REVIEW **M2** +HIGH + **O1** MEDIUM, CONFIRMED; IMPROVEMENT-BACKLOG **HI1**.) + +> This spec prescribes the **adoption-max path** (make it genuinely run). The +> honest fallback (relabel "inspect a finished run" + document the skip) is +> recorded in §3.5, but the fallback voids §2.3 test 4 and the HI2 demo GIF, so +> the runnable path is preferred. + +### 3.2 Enforcement design (changes to the example — implemented in v0.4, not here) + +Make `false_completion:false` **gate-backed** and the example locally replayable: + +1. **Ship a minimal real target under the example** so a gate can actually run: + `examples/coverage-repair/pricing.py` (the under-test module), a `tests/` + split into a **visible** set (what the loop optimizes against) and a + **holdout** set (withheld), and example-local `scripts/verify-fast`, + `scripts/verify-full`, `scripts/verify-safety` that resolve **relative to the + example dir**. +2. **Add a held-out manifest** `examples/coverage-repair/.loop/holdout-manifest.json` + with `visible` + `holdout` command splits (the schema in + `scripts/holdout_gate.py`), runnable via + `python3 scripts/holdout_gate.py examples/coverage-repair/.loop/holdout-manifest.json + --cwd examples/coverage-repair`. +3. **Record the gate's verdict as the evidence** behind the flag: write + `.loop/artifacts/holdout-verdict.json` = the `decide(...)` output + (`{"verdict":"Succeeded","false_completion":false,"passed_visible":true, + "passed_holdout":true, ...}`), add it to `terminal_state.json.evidence`, and + set `terminal_state.json.false_completion` **from** that artifact — never + hand-authored. +4. **Reference the gate in RUNLOG.md**: the Terminal block records that + `scripts/holdout_gate.py` produced the `Succeeded` verdict (this is what makes + the example earn §2's tightened inspector credit, and it is honest — the gate + really ran). +5. **Fix TASKS.json verify-path resolution** so each `task.verify` resolves from + the example dir (ship the example-local verify scripts from step 1). + +### 3.3 Tests — RED then GREEN + +Add a new test module `scripts/test_example_coverage_repair.py` (runs against the +checked-in example, so it is a durable regression that the demo stays honest): + +1. **`test_flagship_holdout_manifest_runs_end_to_end`** — load + `examples/coverage-repair/.loop/holdout-manifest.json`, run + `holdout_gate.run_manifest(manifest, cwd=)`; assert + `verdict == "Succeeded"`, `false_completion is False`, `passed_holdout is True`. + *RED today* (manifest does not exist) → *GREEN after fix*. +2. **`test_flagship_false_completion_is_gate_backed`** — assert + `terminal_state.json.evidence` contains the holdout-verdict artifact **and** + that artifact's `false_completion` equals `terminal_state.json.false_completion`. + *RED today* (no artifact) → *GREEN after fix*. +3. **`test_flagship_task_verify_paths_resolve_from_example_dir`** — for every + `task.verify` in the example `TASKS.json`, assert the referenced script exists + **relative to `examples/coverage-repair/`**. *RED today* (they point at repo + root) → *GREEN after fix*. +4. **`test_inspect_flagship_grants_false_completion_defense_via_invocation`** + (ties M2/HI1 to M1/HI4) — run `_evaluate_contract_checks("examples/coverage-repair")`; + assert `false_completion_defense is True` **and** that it is earned via + `_gate_invoked` (assert `_gate_invoked(resolve_loop_paths(example)) is True`), + not the bare terminal flag. *RED after §2 alone* (tightened rule would drop the + example's credit) → *GREEN once §3 wires the gate into RUNLOG/verify*. +5. **`test_flagship_terminal_validates_clean`** (ties to §1) — `doctor_report` + over the example is `ok:true` (consistent `Succeeded`: `false_completion:false` + + a true `criteria_met` entry). Guards that §3 edits don't trip §1's new rule. + +### 3.4 Acceptance + +- The example reaches a real 1-of-7 terminal state with `holdout_gate.py`'s + recorded output as the evidence behind `false_completion:false`. +- Example-local `verify-*` paths resolve from the example dir. +- `python3 -m loop inspect examples/coverage-repair` still grants + `false_completion_defense` credit **under §2's tightened rule**, via real + invocation. +- All five tests green; `doctor` over the example is `ok:true`. + +### 3.5 Documented fallback (only if the runnable path is dropped) + +Relabel the `README.md:38-44` "30-second demo" as "inspect a **finished** run," +state in `examples/coverage-repair/README.md` that the example intentionally does +not execute the held-out gate and why, and **remove** §2.3 test 4 (the example +would no longer earn invocation-based credit). This fallback forfeits the HI2 +weak→strong demo GIF's honest subject and is **not** the recommended path. + +--- + +## 4. HI6 — Structural invariant on the anti-cheat gate-decision functions + +### 4.1 Problem (file:line + repro) + +`scripts/anticheat_scan.py:52` `_SELF_FILES` blanket-exempts `anticheat_scan.py` +from gate-tampering detection, and the only self-edit detection is +`_gate_evisceration_findings` (collection-literal net-shrink, lines 214-256) and +`_semantic_gate_weakening_findings` (two narrow `_RANK`/downgrade regexes, lines +259-293). None of these inspect the **bodies** of the functions that *decide* +whether a tamper finding is produced. (REVIEW **G3**, MEDIUM, CONFIRMED; +IMPROVEMENT-BACKLOG **HI6**.) + +Verified repro (REVIEW G3 / HI6) — a diff inserting `return False` as the first +line of `_is_gate_path`'s body: + +```python +scan(diff_text=) +# -> {'findings': [], 'clean': True, 'downgrade_to': None} +``` + +A function-body rewrite of `_is_gate_path` / `_is_verify_gate` (an early +`return False`) silently disables gate-tampering detection **and** is invisible to +the scanner, because textual matching catches only three narrow tamper shapes, not +arbitrary logic edits. For a product whose differentiator is catching gamed gates, +a scanner that cannot detect being gamed is the suite's highest-irony credibility +risk. + +### 4.2 Enforcement design + +Add a **structural (AST-span) invariant** over the scanner's own gate-decision +functions, wired into the diff path (to pin the exact `scan(diff_text=...)` +repro) and backed by a runtime AST-signature check (defense-in-depth for the +running module). + +**Registry.** Name the functions whose logic determines whether a tamper finding +is produced: + +```python +_GATE_DECISION_FNS = ("_basename", "_is_verify_gate", "_is_gate_path", "scan") +``` + +**(a) Diff-based logic-edit detection — pins the repro.** Add +`_gate_logic_edit_findings(diff_text)`: + +- Reuse the existing `in_self` hunk-tracking (as `_gate_evisceration_findings` + does) to consider only `anticheat_scan.py` hunks. This check is **not** + self-exempt. +- Parse each hunk's `@@ -a,b +c,d @@` header to assign line numbers to added + (`+`, new-file numbering) and removed (`-`, old-file numbering) content lines. +- Skip blank-only, comment-only, and the collection-entry lines already handled + by `_gate_evisceration_findings` (avoid double-flagging). +- Compute each `_GATE_DECISION_FNS` member's source line-span + (`ast.FunctionDef.lineno .. end_lineno`) from the **current module source** + (`inspect.getsource` / `ast.parse`). +- If any surviving added/removed line's number falls inside a gate function's + span, emit `{"signature": "gate-logic-edit", "severity": "critical", + "evidence": [:120]}`. A `critical` finding downgrades to `FailedSafety` + via the existing severity roll-up in `scan()` (lines 348-354). + +Wire it in alongside the existing self-edit checks: + +```python +findings.extend(_gate_evisceration_findings(diff_text)) +findings.extend(_semantic_gate_weakening_findings(diff_text)) +findings.extend(_gate_logic_edit_findings(diff_text)) # NEW +``` + +**(b) Runtime AST-signature invariant — defense-in-depth.** Add +`check_self_integrity() -> list[dict]` that recomputes a normalized-AST hash for +each `_GATE_DECISION_FNS` member from the live module source and compares it to a +pinned baseline constant `_GATE_DECISION_SIGNATURES: dict[str, str]`; a mismatch +yields a `critical` `gate-self-mutation` finding. A companion test (§4.3 test 4) +asserts the pinned baseline equals the live source, so a **legitimate** refactor +that forgets to re-pin fails CI, and a **malicious** logic edit that changes a +body without re-pinning is caught at runtime. + +Normative rules: + +- **R1** — Any added/removed non-trivial line inside the AST span of a + `_GATE_DECISION_FNS` member within an `anticheat_scan.py` hunk ⇒ a `critical` + `gate-logic-edit` finding ⇒ `downgrade_to == "FailedSafety"`. +- **R2** — Comment-only / blank-only / pure-reorder edits inside those functions + ⇒ **no** finding (no false positives on cosmetic changes). +- **R3** — The pinned `_GATE_DECISION_SIGNATURES` must equal + `check_self_integrity()` over the live source (baseline stays honest). + +### 4.3 Tests — RED then GREEN + +Add to `scripts/test_anticheat_scan.py`: + +1. **`test_return_false_rewrite_of_is_gate_path_is_flagged`** — pins the exact + REVIEW G3/HI6 repro: construct a unified diff adding `+ return False` as + the first body line of `_is_gate_path` in `scripts/anticheat_scan.py`; assert + `scan(diff_text=diff)["clean"] is False`, a finding with `signature == + "gate-logic-edit"` and `severity == "critical"`, and + `downgrade_to == "FailedSafety"`. *RED today* (`{'findings': [], 'clean': + True}`) → *GREEN after fix*. +2. **`test_body_rewrite_of_is_verify_gate_is_flagged`** — a diff editing a body + line of `_is_verify_gate` ⇒ `gate-logic-edit` critical. *RED → GREEN.* +3. **`test_comment_or_reorder_in_gate_fn_stays_clean`** (guard, R2) — a diff + adding only a `# comment` line (or reordering two existing lines) inside + `_is_gate_path` ⇒ `clean is True`. *GREEN after fix* (must not false-positive). +4. **`test_gate_decision_signatures_match_live_source`** (guard, R3) — assert + `check_self_integrity()` over the live module returns **no** finding, i.e. the + pinned `_GATE_DECISION_SIGNATURES` equals the current source's normalized-AST + hashes. *GREEN after the baseline is pinned; fails loudly on any un-re-pinned + edit.* +5. **`test_existing_evisceration_and_semantic_checks_unchanged`** (regression) — + the current `_gate_evisceration_findings` / `_semantic_gate_weakening_findings` + fixtures in `scripts/test_anticheat_scan.py` still produce their existing + verdicts; and the scanner's own regression FIXTURES (e.g. `+ assert 1 == 1`, + exempt via `_SELF_FILES` in `scan()`'s added-line loop) are **not** newly + flagged by `_gate_logic_edit_findings` because they live outside the gate-fn + spans. *GREEN before and after.* + +### 4.4 Acceptance + +- The `return False` repro yields a `critical` `gate-logic-edit` finding → + `downgrade_to == "FailedSafety"`. +- Body rewrites of any `_GATE_DECISION_FNS` member inside `anticheat_scan.py` + hunks are flagged; comment/reorder-only edits stay clean. +- `check_self_integrity()` passes against the live source, and the pinned baseline + test fails if a future edit changes a gate body without re-pinning. +- All pre-existing `scripts/test_anticheat_scan.py` cases stay green. + +--- + +## 5. Landing order, gates, and out-of-scope + +**Order (single v0.4 batch):** +1. **G1** (§1) — smallest, self-contained; unblocks a clean example terminal. +2. **HI6** (§4) — self-contained; no cross-file coupling. +3. **M2 / HI1** (§3) — wires the example gate (needed before M1 tightens). +4. **M1 / HI4** (§2) — tighten inspector credit **last**, after the example earns + credit via real invocation (else §2.3-adjacent example credit would regress). + +**Definition of done (all four):** +- Every "RED → GREEN" test above is added and passes after its fix; each was + confirmed RED against pre-fix code first (TDD). +- `uv run --with pytest --with pyyaml python -B -m pytest -q scripts loop` green + (≥ 105 prior + the new cases). +- `python3 -m loop doctor examples/coverage-repair` → `ok:true`; + `python3 -m loop inspect examples/coverage-repair` → `false_completion_defense` + present and gate-backed. +- `self_eval` structural checks stay green (no SKILL.md wording regressions). + +**Explicitly out of scope for this spec** (tracked elsewhere in +IMPROVEMENT-BACKLOG, not part of the four wedge-enforcement fixes): QW6 +(empty-visible `NotReady`, gate-integrity G2), HI5 (recompute `productive`, M3), +QW11 (canonical repair record, M4), ST1/ST2 (published FCR baseline + versioned +schema, M5), and all trigger-quality (T1-T4) / onboarding-DX (O2-O4) items. This +spec is strictly the **HIGH-severity + HI6 wedge-enforcement** cut. + +--- + +## 6. Traceability + +| Spec § | REVIEW finding | Backlog item | Repro pinned by test | +|---|---|---|---| +| §1 | G1 (HIGH) | QW2 | `test_validate_terminal_rejects_succeeded_with_false_completion_true` | +| §2 | M1 (HIGH) | HI4 | `test_false_completion_defense_denied_for_bare_terminal_flag` | +| §3 | M2 (HIGH) + O1 (MED) | HI1 | `test_flagship_holdout_manifest_runs_end_to_end` + `test_flagship_false_completion_is_gate_backed` | +| §4 | G3 (MED) | HI6 | `test_return_false_rewrite_of_is_gate_path_is_flagged` | diff --git a/pyproject.toml b/pyproject.toml index 051ea47..9f16148 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "loop-engineer" -version = "0.3.4" +version = "0.5.0" description = "Portable Loop Contract Core: validate and inspect repo-native operating contracts for agent loops." readme = "README.md" requires-python = ">=3.10" diff --git a/scripts/test_docs_version.py b/scripts/test_docs_version.py index f8a8a33..4afae04 100644 --- a/scripts/test_docs_version.py +++ b/scripts/test_docs_version.py @@ -15,9 +15,9 @@ def test_readme_has_no_stale_seven_skills(): assert "all 9 skills" in readme -def test_plugin_version_is_0_3_4(): +def test_plugin_version_is_0_5_0(): plugin = json.loads(_read(".claude-plugin/plugin.json")) - assert plugin["version"] == "0.3.4" + assert plugin["version"] == "0.5.0" def test_pyproject_version_matches_plugin(): @@ -30,6 +30,7 @@ def test_pyproject_version_matches_plugin(): def test_changelog_has_current_and_historical_entries(): changelog = _read("CHANGELOG.md") + assert "## 0.5.0" in changelog assert "## 0.3.4" in changelog assert "## 0.3.3" in changelog assert "## 0.3.2" in changelog