Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,23 @@ jobs:
- name: Install recipe dependencies
run: python -m pip install --upgrade pip pyyaml pytest jsonschema langgraph
- name: LangGraph recipe end-to-end
run: python -B -m pytest -q -p no:cacheprovider scripts/test_langgraph_recipe.py
run: python -B -m pytest -q -p no:cacheprovider scripts/test_langgraph_recipe.py scripts/test_langgraph_recipe_st3.py

recipe-temporal:
name: recipe (temporal)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install recipe dependencies
run: python -m pip install --upgrade pip pyyaml pytest jsonschema temporalio
- name: Temporal recipe end-to-end
# start_local() downloads the Temporal dev-server binary (~20MB) to
# $TMPDIR/temporal-sdk-python-<version> — ephemeral and quick, so (like
# recipe-langgraph) this job carries no cache step.
run: python -B -m pytest -q -p no:cacheprovider scripts/test_temporal_recipe.py

action-dogfood:
name: action (dogfood on flagship example)
Expand Down
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,24 @@ All notable changes to `loop-engineer` are documented here.
`WORKFLOW.md` and `README.md` are reworded to describe the mechanism; the 0.3.4
history is left intact.

## Unreleased

**ST3 — integration adapters.** `loop/integrations.py`: an engine-neutral,
pure-stdlib projection (`EngineOutcome` + `to_terminal_state`) from any
engine's "the run ended" signal onto the 7 typed terminal states, with the
fixed precedence safety → human → blocked → budget → spec-gap → gate verdict.
`Succeeded` is reachable only through a green `holdout_gate.decide` verdict,
a clean anticheat sweep, a met criterion, and evidence; `false_completion` is
copied from the gate, never synthesized; missing gate/anticheat input fails
closed to `FailedUnverifiable`. The LangGraph recipe is upgraded in place to
this bar (its run now scores clean under `loop metrics` — closing the
recorded FCR-1.0 follow-up) and a Temporal recipe lands
(`examples/temporal-certify/`, certify-activity pattern, cancellation →
`AbortedByHuman`, retry exhaustion → `FailedBlocked`, timeout →
`FailedBudget`). Both recipes pin the false-completion invariant
(visible-green/holdout-red → `FailedUnverifiable` with
`false_completion: true`, never `Succeeded`) and pass the doctor round-trip.

## 0.7.0 — 2026-07-08

**ST2 — the portable standard.** The on-disk contract is now a documented,
Expand Down
125 changes: 125 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# CLAUDE.md — loop-engineer

Read this first. It replaces the "continue where we left off" cold-start that has been
booting this repo for 9+ days. Nested git repo (`SollanSystems/loop-engineer`, MIT) —
do NOT commit CLAUDE.md or treat it as plugin content.

## What this is

A Claude-Code plugin that designs, launches, verifies, repairs, and improves **agent
loops**. It is an **architect + operator**, not a task-doer: it stands up the operating
contract for a long-running agentic-coding run, gates it against false completion, and
mines its own history for compounding improvement.

Two shipping surfaces from one repo:
- **Plugin** — hub-and-spoke skill suite (router + 8 spokes) invoked inside Claude Code.
- **Portable Loop Contract Core** — a pure-stdlib Python package `loop` + CLI
(`loop` / `loop-engineer`, also `uvx loop-engineer`) that validates and inspects
repo-native operating contracts on any stack. `pip install -e .` pulls **zero** runtime
deps; two optional extras enrich validation: `yaml` (PyYAML) and `schemas` (jsonschema).

## Skill architecture (9 skills = router + 8 spokes)

| Skill | Role |
|---|---|
| `loop-engineer` | Router — dispatches broad agent-loop intent to the right spoke. |
| `loop-architect` | The brain — classify the task, choose architecture + Claude-Code realization, emit a structured ADR (architecture, loop patterns, risk profile, terminal-state plan, next spokes). |
| `loop-contract` | Scaffold the repo-OS contract (SPEC / WORKFLOW / TASKS.json / RUNLOG / `.loop/state.json` + `verify-*` skeletons) from an ADR, then run the pre-execution reflection. |
| `loop-run` | The operator — run/resume the state machine one bounded transition at a time; every run ends in exactly one of the 7 terminal states. |
| `loop-repair` | Patch-and-repair loop — classify the failure mode, make the smallest bounded repair, write a 7-field repair record, enforce a max-attempt cap; refuses to widen scope or edit tests to fake a pass. |
| `loop-evals` | Design the eval harness — the 7-layer suite, FCR + repair-productivity as first-class metrics, deterministic-first-then-rubric, judge calibration. Delegates the deterministic gate to the contract's own `verify-*` scripts. |
| `loop-flywheel` | Turn run history (RUNLOG / traces / receipts) into new eval cases; drive baseline→harden→regression→freeze; compact memory. |
| `loop-inspector` | Read-only score of a **foreign** loop against the prime-directive checklist + 7 terminal states → a scored gap report (advisory heuristic, not a gate). |
| `loop-runtime-monitor` | Watch a running loop from outside — detect stall / repair-churn / budget-overrun and surface one intervention. |

## Version + state

- **Current release: v0.7.0** (`.claude-plugin/plugin.json` + `pyproject.toml`), main at `4647820`.
- **Active branch: `feat/v0.8.0-composes-the-field`** (v0.8.0 NOT yet released — release cut
is the last commit of PR-B). Working tree has in-progress Temporal recipe files (Task 3).
- **Shipped through v0.7.0:**
- ST2 "Portable standard" — `reference/repo-os-contract.md` promoted to the normative,
versioned, tool-agnostic spec (PR #31).
- External-review credibility patch set **PRs #27–#30** (10 findings): empty-evidence
terminal now fails doctor; terminal write-once + atomic writes; strict UTF-8 + declared
ledger discovery; strict-by-install on the Action + pre-commit hook; verify-script
existence checks; **inspector honesty** (#30) — keyword-stuffed fakes no longer score
100/strong, `loop inspect` documented as advisory while `loop doctor` is the hard gate.
- Adoption slices (PRs #19–#23): PyPI wheel substrate, `loop/emit.py` writer API +
LangGraph recipe, `hooks/stop_firewall.py` Stop-hook false-completion firewall,
composite `action.yml` + `.pre-commit-hooks.yaml`.
- **v0.8.0 "composes-the-field" (in progress):** ST3 integration adapters
(`loop/integrations.py` EngineOutcome→terminal projection, LangGraph recipe upgraded to
gate+adapter+metrics-clean, Temporal recipe) + ST4 contributor funnel (foreign-harness
inspect adapter `loop/foreign.py`, gap report, `flaky-test-triage` example, contributor
issue drafts + CONTRIBUTING). Resume ledger: `.superpowers/sdd/` (progress.md + briefs).
- **Test baseline: 395 tests** collected on the v0.8.0 branch (372/10 at v0.7.0; ~10 skip
in structural-fallback mode without jsonschema).
- **Dogfood:** the suite has been run on its own v1.0 launch (self-hosted contract at
`roadmap/launch/`) and on `examples/coverage-repair` (doctor-clean, inspect 90/strong).

## Load-bearing invariants (enforced by `scripts/self_eval.py` vs `evals/cases/structural.json`)

- **7 canonical terminal states:** `Succeeded`, `FailedUnverifiable`, `FailedBlocked`,
`FailedBudget`, `FailedSafety`, `FailedSpecGap`, `AbortedByHuman`.
- **7-field repair record:** `failure_mode`, `hypothesis`, `repair_action`,
`verification_before`, `verification_after`, `remaining_delta`, `productive`.
- **7-layer eval suite:** deterministic-correctness, artifact-quality, human-calibration,
loop-behavior, security/governance, regression-resistance, cost/efficiency.
- **2 first-class metrics:** `false-completion-rate` (FCR), `repair-productivity` (RP).
- Also pinned: 6-item `failure_mode_taxonomy`, `repair_cap_default` 2, `rubric_target_mean`
9.5 (advisory), 9 skill names, 8 reference filenames, 14 template filenames, MIT license.
- **When the suite changes** (new skill/template/reference/terminal state), update
`evals/cases/structural.json` — the self-eval checks compare live repo state against it.

## Verify commands (run from the repo root; this env has no system pytest — use `uv run`)

```bash
uv run --with pyyaml python3 -B scripts/validate_frontmatter.py # 9 SKILL.md frontmatter blocks
uv run --with pyyaml python3 -B scripts/self_eval.py # structural invariants (self-locates root)
uv run --with pyyaml --with jsonschema --with pytest python3 -B -m pytest -q -p no:cacheprovider scripts # full suite
```

CLI subcommands (`python3 -m loop <cmd>` — commands: `scaffold doctor validate verify
inspect metrics`): `loop doctor` is the **hard gate**, `loop inspect .` is the **advisory
scorecard**. CI (`.github/workflows/ci.yml`) installs `pyyaml pytest jsonschema` and runs
`python -B -m pytest -q -p no:cacheprovider scripts`; without jsonschema the core falls back
to structural hand-checks (a deliberate design, not a bug).

## Install / refresh

Installed **user-scope** from the local marketplace `loop-engineer-local`. The plugin-cache
copy is a **static COPY** (nested under `cache/<mkt>/loop-engineer/<ver>/`), NOT a symlink —
it goes **stale after any post-install commit**. Refresh with:

```bash
git -C /mnt/c/Dev/projects/loop-engineer archive HEAD | tar -x -C <cache-dir>
diff -rq /mnt/c/Dev/projects/loop-engineer <cache-dir> # verify
```

Then **restart Claude Code** to reload the skills.

## Roadmap / open work

- **Human gates (blocking PyPI):** register the PyPI pending trusted publisher
(project `loop-engineer`, owner `SollanSystems`, workflow `publish.yml`, environment
`pypi`), then `git tag v0.7.0 4647820 && git push origin v0.7.0`; verify the funnel
`uvx loop-engineer@0.7.0 inspect .` from a scratch dir.
- **Finish v0.8.0:** Temporal recipe (Task 3) → ship PR-A → PR-B (foreign adapter, gap
report, flaky example, contributor funnel, 0.8.0 release cut) → file the 6 contributor
issues at merge.
- **Then:** "inspect N public harnesses" scoreboard post; Show HN launch (human-gated).
- **Positioning (verified):** never claim to coin "loop engineering" — cobusgreyling's
`loop-engineering` (~4.6k stars) owns the term. Own **"false completion"** /
**"proof-of-done contract"** instead.

## Repo conventions

- Specs → `docs/superpowers/specs/` (dated); plans → `docs/superpowers/plans/` (dated).
- Session narratives → `memory/session-summaries/YYYY-MM-DD-*.md` (memory/ is gitignored).
- **Gitignored workbench/telemetry** (not plugin content): `.loop/`, `review/`, `roadmap/`,
`.claude/`, `.gsd/`, `memory/`, `.tmp/`. CI runs on a fresh checkout where these are
absent — never point a CI/dogfood job at the live gitignored `.loop/`; target a tracked
`examples/*` contract instead.
- Env quirks: the Bash deny-list blocks `rm`, bare `cd`, `VAR=` assignments, `timeout`,
`printf`, `source` — use `git -C <path>`, literal absolute paths, and `bash -c`.
90 changes: 74 additions & 16 deletions docs/integrations/langgraph.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,88 @@
# LangGraph — proof-of-done in 10 lines
# LangGraph — gate the graph, then emit proof-of-done

`loop.emit` is a pure-stdlib writer: your graph keeps its own runtime, and the
terminal node records evidence-backed state the `loop` CLI can independently
validate. `pip install loop-engineer` (LangGraph itself stays your dependency).
LangGraph owns the ORCHESTRATE tier: it stays your runtime — the state machine
that routes nodes, holds state, and decides what runs next. Loop Engineer adds
the tier *above* it — a contract-and-proof layer that turns "the graph reached
`END`" into evidence-backed, independently-checkable proof-of-done. It never
replaces LangGraph; it certifies what LangGraph produced.

## The pattern

Make a `certify` node the **only** edge into `END`. It runs the same visible +
withheld-holdout split the loop optimized against through the real gate
(`holdout_gate.decide`) and the trajectory sweep (`anticheat_scan.scan`),
projects the result through `to_terminal_state`, and records it via `loop.emit`
— which refuses a dishonest `Succeeded` before anything hits disk.

```python
from loop import emit
from loop.integrations import EngineOutcome, to_terminal_state

def certify(state): # the ONLY node wired to END
gate = holdout_gate.decide(visible, holdout) # visible green + holdout green?
ac = anticheat_scan.scan(diff_text="", trajectory=[...])
terminal = to_terminal_state(
outcome=EngineOutcome(reached_end=True, artifacts=[...]),
gate_verdict=gate, anticheat=ac,
criteria_met={"1": gate["verdict"] == "Succeeded"},
)
emit.terminate(ws, state=terminal["state"], criteria_met=terminal["criteria_met"],
evidence=terminal["evidence"], false_completion=terminal["false_completion"],
reason=terminal["reason"], iteration_id=1)
return {"terminal_state": terminal["state"]}
```

emit.open_contract("run/") # once, before the graph runs
Wire it so nothing else reaches `END`:

def conclude(state): # your graph's terminal node
emit.append_iteration("run/", iteration_id=1, outcome="task_passed",
task_id="T1", verify_cmd="pytest -q", verify_outcome="pass")
emit.terminate("run/", state="Succeeded",
criteria_met={"tests": True}, evidence=["reports/pytest.txt"])
return {}
```python
graph.add_edge("do_work", "certify").add_edge("certify", END)
```

`emit.terminate` **refuses an evidence-free `Succeeded`** (raises `EmitError`) —
the same cross-check `loop doctor` enforces, applied before the file exists.
## LangGraph signal → typed terminal state

| LangGraph signal | Typed terminal state |
|---|---|
| graph reached `END`, holdout green + anticheat clean | `Succeeded` |
| graph reached `END`, visible green / holdout red | `FailedUnverifiable` (`false_completion: true`) |
| `GraphRecursionError` (LangGraph's own step cap) | `FailedBudget` |
| caught tool/credential exception | `FailedBlocked` |
| operator interrupt | `AbortedByHuman` |

## Zero-install mode

Gate it in CI:
The `loop.integrations` module is convenience, not a requirement — the whole
projection is ~15 lines you can paste into any graph with no dependency:

```python
def to_terminal(gate, anticheat, criteria_met, evidence,
*, human_abort=False, blocked=None, over_budget=False):
fc = gate.get("false_completion") is True
if anticheat.get("downgrade_to") == "FailedSafety": state = "FailedSafety"
elif human_abort: state = "AbortedByHuman"
elif blocked: state = "FailedBlocked"
elif over_budget: state = "FailedBudget"
elif any(v is None for v in criteria_met.values()): state = "FailedSpecGap"
elif (not gate or not anticheat or anticheat.get("downgrade_to")
or gate.get("verdict") != "Succeeded" or fc
or not any(criteria_met.values()) or not evidence): state = "FailedUnverifiable"
else: state = "Succeeded"
return {"schema": "loop-engineer/terminal@1", "state": state,
"criteria_met": {k: v is True for k, v in criteria_met.items()},
"evidence": list(evidence), "false_completion": fc}
```

## Gate it in CI

```yaml
- run: pip install loop-engineer
- run: loop doctor run/
- run: loop doctor run/ # -> {"ok": true}: the contract is structurally honest
- run: loop metrics run/ # -> false_completion_rate + evidence-backed scorecard
```

Full runnable example: [`examples/langgraph-emit/`](../../examples/langgraph-emit/).
`loop metrics` scores the run from its on-disk evidence (RUNLOG success claims,
the verify bundle, the held-out verdict) — not from the graph's narration.

Verified against `langgraph` 1.2.8 (2026-07-08).

Full runnable example (happy path + `--sabotage-holdout` false-completion demo):
[`examples/langgraph-emit/`](../../examples/langgraph-emit/).
Loading
Loading