Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

- **Two PRs:** PR-A = ST3 (Tasks 1–4), PR-B = ST4 + release cut (Tasks 5–10). PR-B branches from main after PR-A merges. The 0.8.0 release commit is the **last** commit of PR-B.
- **Paths are the shipped reality, not the 2026-06-30 spec's:** helper = `loop/integrations.py` (NOT `loop_engineer.integrations`); docs = `docs/integrations/<engine>.md` (NOT `reference/integrations/`).
- **All disk writes go through `loop.emit`** (`open_contract` / `append_iteration` / `append_receipt` / `terminate`). No second write path for contract artifacts. (Recipes may write their own verify-bundle/verdict JSONs under `.loop/artifacts/` — those are evidence artifacts, not contract objects.) *Adjudicated carve-out (PR-A whole-branch review, 2026-07-08):* both recipes reset `<ws>/RUNLOG.md` to emit's own fresh header immediately after `open_contract`, because the scaffold seeds an unfilled `{{ITERATION_OUTCOME}}` placeholder that `loop metrics` flags as an unrecognized outcome token and the root fix (templates/scaffold/emit) is outside PR-A's file discipline. The reset strips a placeholder, never fabricates state. Root-fix affordance (e.g. `emit.open_contract(seed_runlog=False)`) is tracked as a PR-B contributor issue.
- **All disk writes go through `loop.emit`** (`open_contract` / `append_iteration` / `append_receipt` / `terminate`). No second write path for contract artifacts. (Recipes may write their own verify-bundle/verdict JSONs under `.loop/artifacts/` — those are evidence artifacts, not contract objects.) *Adjudicated carve-out (PR-A whole-branch review, 2026-07-08):* both recipes reset `<ws>/RUNLOG.md` to emit's own fresh header immediately after `open_contract`, because the scaffold seeds an unfilled `{{ITERATION_OUTCOME}}` placeholder that `loop metrics` flags as an unrecognized outcome token and the root fix (templates/scaffold/emit) is outside PR-A's file discipline. The reset strips a placeholder, never fabricates state. Root-fix affordance (e.g. `emit.open_contract(seed_runlog=False)`) is tracked as a PR-B contributor issue. **[Retired 2026-07-09 by the #40 root fix — `templates/RUNLOG.md.tmpl` seeds no `## Iteration` block, so a fresh scaffold is metrics-clean by construction; both recipe resets are removed and no `seed_runlog` kwarg was needed.]**
- **PR-A file discipline:** `loop/` gains ONLY `integrations.py`. No modification to `scripts/holdout_gate.py`, `scripts/anticheat_scan.py`, `scripts/inspect_loop.py`, `scripts/metrics.py`, `scripts/self_eval.py`, `scripts/validate_frontmatter.py`, or anything under `schemas/`, `templates/`, `evals/`. `scripts/test_langgraph_recipe.py` stays **byte-identical** (new assertions go in a NEW file `scripts/test_langgraph_recipe_st3.py`). New test files under `scripts/` are additions, allowed.
- **PR-B scorer discipline:** `scripts/inspect_loop.py` may change ONLY (a) the path-resolution seam (3 `resolve_loop_paths(...)` call sites → `_resolve_paths(...)`) and (b) additive report labeling (`foreign_layout`, `advisory`). Zero changes to `_CHECKS` weights, credit tiers, signal regexes, the score cap, or `_verdict`. A regression test pins that the fixture earns no invoked credit.
- **Fixed precedence** in `to_terminal_state`: safety → human → blocked → budget → spec-gap → gate verdict. `FailedSafety` and `AbortedByHuman` beat a green gate.
Expand Down
6 changes: 0 additions & 6 deletions examples/langgraph-emit/graph_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,6 @@ def certify(state: State) -> dict:

def main(workspace: str, sabotage: bool) -> int:
emit.open_contract(workspace)
# The fresh scaffold's RUNLOG carries a placeholder example iteration (its
# unfilled outcome renders as "REPLACE"); start the demo's run history clean
# so `loop metrics` scores only the graph's own iterations.
(Path(workspace) / "RUNLOG.md").write_text(
f"# RUNLOG.md — {Path(workspace).name}\n", encoding="utf-8"
)
graph = (
StateGraph(State)
.add_node(do_work)
Expand Down
6 changes: 0 additions & 6 deletions examples/temporal-certify/workflow_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,12 +153,6 @@ def certify_workflow_failure(workspace: str, cause: BaseException | None) -> dic

async def run_and_certify(client: Client, workspace: str, *, sabotage: bool, wf_id: str) -> dict:
emit.open_contract(workspace)
# The fresh scaffold's RUNLOG carries a placeholder example iteration (its
# unfilled outcome renders as "REPLACE"); start the demo's run history clean
# so `loop metrics` scores only the workflow's own iterations.
(Path(workspace) / "RUNLOG.md").write_text(
f"# RUNLOG.md — {Path(workspace).name}\n", encoding="utf-8"
)
try:
return await client.execute_workflow(
CertifiedGoalWorkflow.run, WorkArgs(workspace=workspace, sabotage=sabotage),
Expand Down
9 changes: 9 additions & 0 deletions scripts/test_emit.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

import pytest

import metrics
from loop import emit
from loop.contract import validate_contract

Expand All @@ -24,6 +25,14 @@ def test_open_contract_is_doctor_clean(workspace):
assert validate_contract(workspace)["ok"] is True


def test_open_contract_is_metrics_clean(workspace):
# The seeded RUNLOG is reference-only prose — no `## Iteration` block, so a
# fresh scaffold must score zero iterations and no outcome tokens.
scorecard = metrics.compute_metrics(workspace)
assert scorecard["provenance"]["unrecognized_outcomes"] == []
assert scorecard["iterations_claiming_success"] == 0


def test_append_iteration_writes_parseable_runlog_and_updates_state(workspace):
runlog = emit.append_iteration(
workspace, iteration_id=1, outcome="task_passed", task_id="T1",
Expand Down
17 changes: 2 additions & 15 deletions scripts/test_template_roundtrip.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,21 +90,8 @@ def _fill_values() -> dict[str, str]:
"ARTIFACT_PATH": ".loop/artifacts/verify-T1.json",
"FALSE_COMPLETION": "false", # boolean
"TERMINAL_REASON": "fast gate passed with evidence",
# --- RUNLOG.md prose ---
"ITERATION_DATE": "2026-06-30",
"STATE_NAME": "running",
"ACTION_1": "ran the fast gate",
"ACTION_2": "recorded evidence",
"VERIFY_GATE": "verify-fast",
"VERIFY_OUTCOME": "PASS",
"VERIFY_CMD": "scripts/verify-fast",
"SCORE_BEFORE": "0.0",
"SCORE_AFTER": "1.0",
"ITERATION_OUTCOME": "task_passed",
"REPAIR_ACTION": "n/a",
"REPAIR_ATTEMPT": "0",
"IMPROVEMENT_FLAG": "n/a",
"ITERATION_NOTES": "clean first pass",
# RUNLOG.md seeds a reference-only preamble that declares only
# {{PROJECT_NAME}}, so it needs no per-iteration prose fill.
# --- SPEC / WORKFLOW / AGENTS / EVALS prose ---
"NON_GOAL_1": "no production writes",
"NON_GOAL_2": "no schema changes",
Expand Down
49 changes: 19 additions & 30 deletions templates/RUNLOG.md.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -2,41 +2,30 @@

> Human-readable iteration history. Append-only — one entry per loop iteration.
> Machine state lives in `.loop/state.json`; this is the audit trail.
>
> This file starts with no iterations. `loop metrics` reads only `## Iteration`
> blocks, so a fresh RUNLOG scores clean by construction. The entry format below
> is a reference, not an iteration — `loop.emit.append_iteration()` writes real
> blocks for you (or copy the shape by hand) beneath the marker at the bottom.

---

## Iteration {{ITERATION_ID}} — {{ITERATION_DATE}}
## Entry format (reference — not an iteration, not scored)

**State entered:** `{{STATE_NAME}}`
**Active task:** `{{ACTIVE_TASK_ID}}` — {{TASK_TITLE}}
**Plan version:** {{PLAN_VERSION}}
Each iteration is a section headed `## Iteration <id> — <date>` carrying:

### Actions taken

- {{ACTION_1}}
- {{ACTION_2}}

### Verification result

- **Gate:** `{{VERIFY_GATE}}` — {{VERIFY_OUTCOME}} (`{{VERIFY_CMD}}`)
- **Score before:** {{SCORE_BEFORE}}
- **Score after:** {{SCORE_AFTER}}

### Outcome

`{{ITERATION_OUTCOME}}` — one of: `task_passed`, `task_failed`, `repair_triggered`, `approval_requested`, `replanned`, `terminal`

### Repair record (if repair triggered)

- **Failure mode:** {{FAILURE_MODE}}
- **Repair action:** {{REPAIR_ACTION}}
- **Attempt:** {{REPAIR_ATTEMPT}} of {{REPAIR_CAP}}
- **Measurable improvement:** {{IMPROVEMENT_FLAG}}

### Notes

{{ITERATION_NOTES}}
- **State entered / Active task / Plan version** — the loop state, the task
worked, and the plan version in force.
- **Actions taken** — a bullet list of what the agent did.
- **Verification result** — the gate (`scripts/verify-*`), its outcome, and the
score before/after.
- **Outcome** — exactly one backticked token, one of: `task_passed`,
`task_failed`, `repair_triggered`, `approval_requested`, `replanned`,
`terminal`.
- **Repair record** — failure mode, repair action, attempt N of the cap, and the
measurable-improvement flag (only when a repair was triggered).
- **Notes** — free-form context.

---

<!-- Add new iterations above this line. Do not edit past entries. -->
<!-- Append new `## Iteration <id>` blocks below this line. Do not edit past entries. -->
Loading