feat: complete RFC 007 + RFC 012, land RFC 008 A/B, cover five untested modules - #39
Conversation
`nightly init` now asks each installed host CLI what it accepts instead of
carrying a table of vendor flags and model ids that goes stale. The config
it writes therefore reflects the harness the operator actually works in.
What the probe reads, from the host's own `--help`:
- The model-selection flag. Discovered and stored as
`model_tiers.<host>.flag`; `build_argv` emits a discovered flag, never a
guessed one. This immediately paid for itself: it found `--model` on
opencode and gemini, neither of which could be verified by hand.
- The model vocabulary. Claude Code enumerates its own aliases and ids in
the `--model` help text ('fable', 'opus', 'sonnet', 'claude-fable-5');
those are parsed out and ranked into lite / coding / reasoning by family.
- Which harness is running init, from environment markers (`CLAUDECODE`,
`CODEX_HOME`, ...), so the initializing harness is probed first and its
findings lead the config.
Precedence favors reproducibility: a *pinned* discovered id
(`claude-opus-5`) overrides the seeded default, but a bare alias (`opus`)
does not - aliases float to whatever shipped most recently, which is
convenient interactively and wrong for an overnight run whose model should
still be knowable in the morning. Discovery can only add certainty; it
merges over the seeded defaults rather than replacing them, and any probe
failure degrades to the seeded template so init can never fail because a
host CLI misbehaved.
Host coverage extended to all seven major harnesses: Claude, Codex,
Cursor, Gemini, OpenCode, Pi, Hermes (plus Antigravity). `pi` and `hermes`
are recognized at the routing layer only - they ship no integration
package yet, which the `HostId` docstring states explicitly.
Also lands the rest of RFC 007 Phase B's core:
- `nightly dispatch start` resolves plan tier -> role default -> host
binding, passes the model id with the discovered flag, and appends the
tier's effort directive to the prompt.
- `nightly specialist <role> --tier <tier>` appends the same directive.
- Effort ships as prompt text, not a CLI flag: the flag surface differs
per host and several expose none, while prompt text works everywhere and
no-ops harmlessly where unsupported.
Performance: the first version made every `nightly init` shell out per
host, taking the suite from 17s to 491s. Fixed by memoizing the default
probe for the process lifetime and reading each host's help once for both
facts rather than twice.
1200 tests pass (38 new in test_model_probe.py); ruff, pyrefly, and
`nightly verify` all clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#12 B1/B4/B5 landed and B6 (model-control discovery) was unplanned but supersedes part of B1 — the flag is now probed from each host CLI rather than carried in a table Nightly would have to keep current. B2 (six host skills) and B3 (briefing tier breakdown) remain open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase A declared `parallelism:` caps that nothing read. A cap nothing enforces is a comment, and it's the same silently-inert failure flagged for `model_tiers` — so this makes it real. `nightly dispatch start` now refuses when the tier is at capacity, exits 3, and names the ceiling that blocked it (`--force` overrides). Two ceilings are checked: the tier's own — via `limit_for`, which already folds in the global cap — and the global count across every live dispatch. `nightly dispatch status` gained a `capacity:` line showing live/cap per tier plus the total, because that's the question the dispatch table provokes. Three decisions worth noting: - **Blocking says "wait", never "run it on a cheaper tier."** A silent downgrade would hand back a lite-tier review nobody asked for, with no signal it happened — defeating RFC 007's whole argument for putting the reviewer on the reasoning tier. There's a test asserting the refusal message never names another tier. - **Liveness is checked against the PID, not the recorded status.** A spawn that died without being polled still reads `running` on disk; counting those would wedge the fleet until someone happened to run `dispatch status`. - **Dispatch state now persists its tier.** Counting per tier by re-resolving each plan would read a plan that may have changed, or been marked done, since the spawn. Records written before this change have no tier and count toward the global cap only. 1214 tests pass (14 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Last inert knob from Phase A. `max_worktrees` was declared in the config and read by nothing, so the task-level fan-out cap did not exist. `create_worktree` now takes the cap and raises `WorktreeCapReached` when the fleet is full. Three choices worth noting: - **A typed exception, not a bare RuntimeError.** Callers need to tell "at capacity, try again when a task finishes" apart from "git itself failed" — the first is a scheduling condition with an obvious remedy. The CLI exits 3 for it, matching `dispatch start`'s at-capacity code. - **Checked before the branch is cut**, so a refused request leaves nothing behind. There's a test asserting `worktree add` is never reached when the cap blocks. - **Counted from live `git worktree list`, not run state.** A worktree left behind by an earlier crashed run still occupies a slot, because it is still consuming disk and still holds a branch checked out. Counting only what the current run knows about would under-report exactly when the repo is most cluttered. Both call sites (the `nightly worktree` command and the headless driver) pass the configured cap; `0` remains unlimited, as everywhere else. 1218 tests pass (4 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tier routing, admission caps, and handoff thresholds all ship enforced — and nothing told any agent they exist. Machinery an agent is never told about is machinery an agent never uses: it would keep dispatching serially into an 8-wide fleet and grinding past its context ceiling instead of handing off. Both RFCs scoped this as "a paragraph on each of six host skill files". Delivering it as rule 12 of the shared rules block is strictly better: `seed_rules` propagates one marker-delimited body to every host's AGENTS.md / CLAUDE.md, so a single edit reaches all seven harnesses and the wording cannot drift between them. The autonomy contract already lives here for exactly that reason. Rule 12 covers four things: - Fan out to the cap when work is independent. Serial dispatch where parallel was available is the most common way an overnight run wastes its night, and `dispatch status` now prints the headroom to use. - Let the tier default stand unless role and complexity genuinely diverge; override via `model_tier:` only then. - Never route around a full reasoning tier by downgrading — wait. This is the rule that protects the reviewer-on-reasoning decision from being quietly undone at 3am. - Hand off at 25% / 50% of the model's window, carrying goals and state rather than a transcript. Shedding the history is the point. 1223 tests pass (5 new, asserting the doctrine is present and lands inside the seeded markers — a rule outside them propagates nowhere). ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…07 C3) Everything added this session was undiscoverable: `model_tiers`, `parallelism`, the handoff ratios, `--tier`, `--force`, exit code 3, and init-time model-control discovery had no user-facing documentation at all. A feature nobody can find is a feature nobody has. Three new README sections under "How it works": cost-aware dispatch (the tier table, the role defaults and why reviewer sits on reasoning, the plan-frontmatter override, and what init discovers), context handoff (the two thresholds as fractions of the resolved model's window, with worked examples for 1M and 200K), and an enforcement note on the existing parallelism section with a sample `capacity:` line. Also fixes pre-existing drift found while editing: the entire `nightly dispatch` command family (start / status / tail / wait) was missing from the CLI reference, and `nightly specialist` was listed without `--tier`. The new `tests/test_readme_claims.py` pins the load-bearing claims — tier defaults, effort defaults, handoff ratios and their arithmetic, the fallback window, exit code 3, and the `capacity:` line's tier order — against the code that implements them. A restatement with no test is a promise with no enforcement, and these tables restate values that live in `SPECIALIST_TIER_DEFAULTS`, `DEFAULT_TIER_EFFORT`, and `ContextConfig`. Deliberately narrow: it tests facts, not wording, so prose stays free to change. 1230 tests pass (7 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… 007 The last unticked item in RFC 007. Routing has been enforced for several commits with no way for the operator to see whether it did what they configured — which is the only question the tier design actually raises. A run showing `reasoning x 12` is burning the expensive tier on work that should have been cheap; one showing no `reasoning` at all probably reviewed nothing. Three implementation choices: - **Reads each task's `dispatch.json` directly**, not via `list_dispatches`, which resolves the *current* run. The briefing renders arbitrary runs, including concluded ones — going through the current-run resolver would silently render an empty panel for exactly the runs an operator reviews in the morning. - **Untiered records surface as `unrouted`** rather than being dropped. Omitting them would make a pre-RFC-007 run look like it dispatched less than it did. - **A corrupt state file is skipped, not fatal.** One unparseable `dispatch.json` must not sink the whole briefing. The panel renders independently of the session narrative, so a run where no narrative was authored still shows its dispatch mix. RFC 007 is now fully implemented across all three phases. 1237 tests pass (7 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s (RFC 012 C1) The handoff protocol has been documented (rule 12) and computable (`resolve_context_thresholds`) for several commits with nothing enforcing it. This wires it into the Stop hook, which is the only component that sees the live context estimate every turn. `_apply_context_diet` becomes a three-rung ladder, most severe first: hard handoff (stop mid-task), soft handoff (finish this task, then hand off), and the original v0.0.12 diet nudge. A handoff block **replaces** the diet block rather than stacking with it — past a handoff threshold "use less context" is no longer the right instruction, and two competing directives in one prompt is how an agent ends up following neither. Two things worth a reviewer's attention: - **This changes default behavior.** At a 1M window the soft threshold (250K) sits below the default `budget_tokens` (256K), so the diet block is effectively superseded at defaults. That is the specified intent — hand off at ~25% of the window — not an accident. Two existing tests asserted the diet block at a 300K estimate; they now assert the handoff, and a third case covers the diet path at an estimate below the soft threshold, where it is still the right instruction. - **Session thresholds assume the reasoning tier's model.** The hook cannot read the host's live model selection, and rule 12 puts orchestration on reasoning. A wrong guess degrades safely: an unknown model falls back to 200K, handing off earlier than a large window needs rather than later than a small one can afford. The summary is specified as goals-and-state, explicitly NOT a transcript. An agent told only "write a summary" reliably writes a transcript, which re-injects the history the handoff exists to shed. Config trouble can never break the hook — threshold resolution is wrapped, and a broken config degrades to the diet rung. There's a test for it: the Stop hook runs at every turn boundary, so it must degrade, not raise. 1247 tests pass (10 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pletes RFC 012 C1 made agents write a HANDOFF.md when they cross a context threshold. Nothing read it back, which meant the protocol worked right up until the moment it mattered: a compaction wipes the outgoing agent's context, and a handoff nobody re-reads is a summary written into the void. Two readers, both fed by one `find_handoffs` scanner: - **The digest** gains a `Pending handoffs` section. This is the load- bearing one — the digest is what the `SessionStart(compact)` hook re-injects, so a handoff written just before a compaction survives the event it was written for. - **The briefing** gains a `handed off mid-task` panel. The panel names the task and its outstanding work rather than reporting a count, which is what the RFC's "handoff counts" wording originally implied. A handoff means work was deliberately left unfinished; the operator's first question in the morning is *which* work, and a number cannot answer that. The summary is taken from the first non-heading line, so `# Handoff — alpha` doesn't get reported as the summary of itself. Both readers degrade to silence on unreadable files — the digest renders from the Stop hook's hot path and must never raise. RFC 012 is now fully implemented across all three phases, as is RFC 007. 1258 tests pass (11 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Observed live this session: the cascade handed the agent "RFC 007 A1" for seven consecutive turn boundaries, long after A1 was implemented, committed, and tested. The `accepted_rfc` ranker's only in-flight guard (`_is_item_in_flight`) inspects **open PR** titles and bodies, so work that is committed locally but not yet a PR is invisible to it — the checkbox on `main` stays unchecked and the item is re-picked forever. The gap is not exotic. It opens whenever a push fails, credentials expire, CI is slow to produce a PR, or the agent simply hasn't pushed yet — i.e. the entire window between finishing work and it becoming reviewable. `_items_done_on_local_branches` reads each `nightly/*` branch's own copy of the RFC and collects the items *that copy* has checked. Asking the branch directly answers "is this item done somewhere?" rather than inferring it from the weaker signal that a branch touched the file, so a branch that edited the RFC for unrelated reasons doesn't mask real work. Bias matches the existing guard's: false negatives over false positives. Any git failure — no repo, detached checkout, branch predating the RFC, timeout — returns an empty set and falls back to today's behavior. The one thing this must never do is skip everything and report `nothing`. Verified against the live repo before writing tests: the guard detects 15 completed items across the local Nightly branches, including the A1 that had been re-picked all night. Only `nightly/*` branches count — someone else's feature branch ticking a box is not Nightly's work. 1266 tests pass (8 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This session completed two RFCs, committed everything, and could not push — a signing agent had auto-locked, and the same key backs the push remote. Nothing in `nightly status` or `nightly doctor` said so. It took eight turns and an accidental `git push` failure to notice. That is the most expensive way an overnight run can fail: silently, while appearing to succeed. From inside the session the work is done; from outside it does not exist. An operator skimming a morning briefing has no reason to suspect anything. `_check_push_readiness` reports three conditions, all from local refs — no network, so it stays fast and works offline: - Nightly branches ahead of their upstream (work committed, not pushed). - Nightly branches with no upstream at all (never pushed anywhere). - Commit signing configured as SSH while the agent holds no identities, which fails the commit and the push together. A `[gone]` upstream is deliberately *not* flagged: that means merged and cleaned up, i.e. local cruft rather than lost work. Flagging it would cry wolf after every finished task, and a check that cries wolf is a check operators learn to skip. Advisory only, never repaired — pushing is the operator's call and unlocking an agent is theirs. There's a test asserting the check mutates no git state. Verified against the live repo, where it renders the current situation in one line: `unpushed: nightly/rfc007-tier-routing [ahead 10]; commit signing configured but the ssh agent holds no identities`. 1277 tests pass (11 new, against a real local bare remote so upstream tracking is genuine rather than mocked); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…handoff
`nightly status` rendered `compact: enabled=enabled` — and, when off,
the actively misleading `enabled=disabled`. The state word already
carries the meaning. An existing test pinned the doubled wording, so it
was asserting the bug; updated.
More substantially, `status` showed nothing about the three config blocks
that now decide how a run behaves. An operator had to open
`.nightly/config.yml` to learn which model their own dispatches would
use, how wide the fleet would go, or when agents would hand off. `status`
is where config gets eyeballed; anything that changes behavior this much
belongs in it. It now prints:
compact: enabled (threshold cap 256K)
tiers: [claude] lite=claude-haiku-4-5 coding=claude-sonnet-5 reasoning=claude-opus-5 via --model
fleet: lite=8 coding=6 reasoning=2 (worktrees=8, total=8)
handoff: soft=250K hard=500K (of a 1000K window)
Details worth noting:
- The tier line reports the host **actually running the command** when a
harness is detectable, so the models shown are the ones that will
apply — not a default that happens to be listed first.
- It states `(no model flag discovered)` when init found none, which is
the difference between "routing works" and "routing is inert". A config
predating the discovery feature says so instead of looking healthy.
- Unlimited caps render as `∞` rather than `0`, since `0` reads as "none
allowed" in a column of counts.
- A broken config still renders — `status` is a diagnostic, and the one
time it must work is when something is wrong. There's a test.
1284 tests pass (7 new); ruff, pyrefly, and `nightly verify` clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems, one cause. The failure message for an undispatchable host hardcoded a prose list of which hosts were supported — a restatement of what `build_argv` actually implements. The moment `pi` and `hermes` joined `HostId`, the two drifted: asking to dispatch to `pi` produced a message enumerating every host *except* `pi`. It also conflated two failures that need opposite fixes from the operator: - **No headless CLI** (cursor, antigravity, pi, hermes). Nothing they install changes this; use the host's own primitive or pick another. - **Binary not on PATH** (a supported host that isn't installed here). A five-second fix the old message never mentioned — asking for `claude` on a box without it produced the same wall of text as asking for `cursor`. `HEADLESS_HOSTS` is now the single source of truth and `unsupported_host_message` computes the explanation from it plus what is actually on PATH. A test asserts the tuple isn't a lie: with every binary stubbed present, `build_argv` returns argv for exactly the hosts in `HEADLESS_HOSTS` and None for every other member of `HostId`. That makes this drift class impossible rather than merely fixed. Caught while writing it: the suggestion list included the failing host itself — "your `codex` binary is missing, try codex". Recommending the binary that just failed to resolve is worse than saying nothing, so the failing host is excluded from both suggestion paths. One existing test matched on the old message text and was updated. 1291 tests pass (7 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`nightly verify` gates every PR (rules block, rule 7), so a check that
times out blocks work that was actually fine — and re-running changes
nothing, which makes it the most frustrating failure the tool can
produce. It happened in this session: a change temporarily pushed the
test suite past the hardcoded 300s and `make-check` failed with
timed out after 300.0s
and nothing else. No hint that the limit was adjustable, no mention of
the `--timeout` flag that already existed. A message that names no remedy
sends the reader to the source.
Two changes:
- **A `verify:` config block** with `timeout_seconds` (default 300). The
`--timeout` flag existed but had to be passed on every invocation and
could not reach the verify that `nightly run` performs internally. A
project whose test target legitimately runs long now sets it once.
- **The timeout message names both remedies** and distinguishes the
cases: "if this check is simply slow rather than hung, raise
`verify.timeout_seconds` ... or pass `--timeout`". Slow and hung want
opposite responses, and only the operator knows which they have.
A non-positive or non-numeric `timeout_seconds` falls back to the default
rather than disabling the cap — an accidental `0` should not mean "let a
hung check block the run forever".
1298 tests pass (7 new); ruff, pyrefly, and `nightly verify` clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A swapped or typo'd binding — `lite: claude-opus-5`, or a reasoning tier left on a lite model — is silent and expensive. Routing keeps working, dispatches keep succeeding, and the only symptom is the bill, or in the reverse direction a reviewer that quietly misses bugs. Nothing else in the system ever complains. `_check_tier_sanity` compares each configured binding's model family against the tier it is bound to and warns on disagreement. **What this deliberately is not:** a check that the model id exists. RFC 007's Risks section proposed exactly that, and it is unbuildable from local signal — I tested before writing it. The vocabulary a host CLI advertises in `--help` is a sample, not an enumeration: `claude --help` names four tokens and *none* of the three production ids Nightly ships as defaults appear among them. A membership test would flag correct configuration as broken, which is worse than no check. Validating for real needs a network call to a vendor models endpoint, and `doctor` must work offline. So: family matching only, and an unrecognized family is skipped rather than guessed at. `tier_of_model` returns None for ids it doesn't recognize — the honest answer — and callers treat None as "no opinion", never as "wrong". A test pins that the shipped defaults don't trip the check, since a check that fires on correct config trains operators to ignore it. RFC 007's Risks section is updated with why the original mitigation was abandoned and what replaced it; the deprecation case remains covered only by "dispatch raises, briefing surfaces it". 1303 tests pass (5 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`_check_config` writes the full template only when `config.yml` is *absent*. A repo initialized before a feature shipped therefore never learns that feature's knobs exist: every loader defaults gracefully, so nothing breaks, nothing complains, and the operator's config quietly diverges from the schema for as long as the repo lives. This is not hypothetical or new. Run against this very repo, the check reports eight missing blocks — `compact`, `context`, `ideate`, `model_tiers`, `parallelism`, `vault`, `verify`, `worktree` — and half of them predate tonight's work entirely. Nobody here has been able to configure the vault, the worktree probe, or ideation, because nothing ever mentioned they were configurable. The expected set is derived from `DEFAULT_CONFIG_YML` itself rather than a hand-maintained list, so the check cannot drift from the schema the way the thing it detects did. A test asserts the template satisfies its own check — otherwise the check is the broken party. Wording matters here: missing is not broken. The detail reads "not configured (defaults apply)", because every one of these has a working default and an operator who reads this as an error will go fix a non-problem. Advisory, and deliberately never repaired. `config.yml` is hand-edited and comment-rich; appending risks clobbering ordering or re-adding a key the operator deliberately deleted. A wrong merge into the file that governs every other behavior is worse than a message saying what to copy. There's a test asserting it writes nothing. 1309 tests pass (6 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems in one function.
`_print_dispatch_row` was annotated `state: object` with the comment
"typed via duck on the read side", which forced **thirteen**
`# type: ignore[attr-defined]` suppressions — one per field access. The
type was never actually unknown; it is `BackgroundDispatchResult`. A
`TYPE_CHECKING` import expresses that with no runtime cost and no import
cycle, and pyrefly now genuinely checks the function instead of being
told to look away. Repo-wide suppressions drop from 59 to 44.
Separately, the row omitted `tier` — so `dispatch status` printed
per-tier capacity totals while giving the operator no way to see which
tier any individual dispatch was on. Adding the totals without the
per-row value that produces them was incoherent; both views now agree.
status pid host role tier slug
running 4242 claude reviewer reasoning 0001-reviewer
running 4242 claude researcher lite 0001-researcher
running 4242 claude implementer - 0001-implementer
Untiered records (written before RFC 007) render `-` rather than `None`,
which is noise in a table column.
One of the new tests was wrong on first write — it asserted `"None" not
in <whole row>` while the fixture's own slug is `task-None-<pid>`, so it
would have passed for the wrong reason and failed for a real one. It now
asserts on the tier field itself.
1313 tests pass (4 new); ruff, pyrefly, and `nightly verify` clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…RFC 007 RFC 008 was accepted with eight unchecked items and zero done — the oldest untouched human-blessed work in the repo. Its premise is that an unchecked box means "nobody ticked it", not "nobody did it", and this commit demonstrates the premise on itself. **RFC 007 B2 and C2 were both already implemented and never ticked.** B2 shipped as rule 12 in `173a7e8`; C2 shipped as `_check_model_tiers` in `9b23ef1`, recorded there as "A8, pulled forward from C2" while the C2 box stayed unchecked. I reported RFC 007 as complete on that basis — true in substance, unverifiable from the checklist. Both are now ticked with the SHA that implemented them, which is the format rule 13 asks for. Phase A ships the doctrine as **rule 13** of the shared rules block: before implementing an item, check whether the deliverable exists — the symbol, an unmerged `nightly/*` branch, an open PR — and if it does, tick the box and commit the reconciliation alone rather than re-implementing. This completes a three-layer guard against re-doing finished work, and `_is_item_in_flight`'s docstring now names all three: the open-PR skip (in flight), the local-branch guard from `1791974` (done but unmerged), and this verifier (done and unrecorded — the case neither can see). Two deviations from the RFC as written, both stated in its checklist: - **A2** specifies the paragraph in six host `skill.md` files. Delivered via the shared rules block instead — the RFC itself calls that duplication "a doctor-monitored drift surface" (Resolved #9), which concedes it is a liability. One marker-delimited copy reaches all hosts and cannot drift between them. - **A3** wants a per-host skill token for drift detection. With A2's delivery there is no per-host copy to drift, so the item is **left open, not silently reinterpreted** — a human should confirm the substitution before it is ticked. Ticking it myself would be the exact failure rule 13 exists to prevent, in reverse. 1318 tests pass (5 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An auto-tick is a claim that work already existed, made by an agent that chose not to redo it. That is exactly the judgment an operator should audit: a wrong one silently drops an item nobody implemented, and it leaves no other trace — the box is ticked, the RFC looks done, and the work simply never happened. The briefing now surfaces them as "ticked as already-done (verify these)", scanning `main..HEAD` for rule 13's commit format and naming the RFC, the item, and the SHA the agent claims implemented it. A tick with no SHA renders as `unstated`, which is the case most worth opening. Rule 13 caught a real instance on its first use: **B2 was already satisfied.** The commit format shipped with A1/A2 in `15fb1bc`, so this commit ticks it rather than re-implementing it — the doctrine working on its own RFC, one commit after landing. The regex deliberately requires the literal `tick` verb: the sibling commit `docs(rfc-007): reconcile Phase B checklist` is an ordinary docs change, not a claim about pre-existing work, and there is a test pinning that it is not swept up. RFC 008 is now implemented except A3, which remains open pending a human decision on whether the shared-rules delivery substitutes for the per-host skill token it specifies. 1324 tests pass (6 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`diagnose_and_repair` builds its report from a hand-edited list of `checks.append(...)` calls. A `_check_*` helper that gets written but never appended runs zero times and reports nothing — silently, forever, with a passing test suite and a green doctor. That list grew by four in a single session. Nothing caught omissions except remembering, which is not a mechanism. `test_every_check_helper_is_wired_into_the_report` introspects the module for `_check_*` functions and asserts each appears in `diagnose_and_repair`'s source. `_check_host` is excluded — it is called per host inside the loop rather than appended once. Mutation-tested rather than assumed: deleting the `_check_tier_sanity` append makes it fail with the offending name, and restoring it makes it pass. A guard that cannot fail is decoration. Also added: report check names must be unique, since duplicates make the output ambiguous to read and impossible to filter. The module docstring claimed four checks; there are nine. It now separates the four that **repair** from the five that are **advisory and never write**, and states the two-step requirement for adding one. 1326 tests pass (2 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four tables are keyed by a `Literal` and indexed directly — `_PROMPTS`, `TIER_FAMILIES`, `DEFAULT_TIER_EFFORT`, `_EFFORT_DIRECTIVES`. Adding a member to the `Literal` without adding a row raises `KeyError` the moment that member is first used, which for a dispatch table means 3am, inside a spawned subprocess, with the traceback in a log nobody is reading. The RFCs discuss future specialist roles, so this is a change someone will actually make. The tests call the **public function** for every member rather than asserting on dict keys. A key-presence assertion passes for a row whose value is empty, wrong-typed, or unreachable behind a guard; calling the function is the actual contract. All four registries are total as of writing — this is prevention, not a fix. Two of the assertions encode invariants nobody had written down: - **Family round-trip.** A family listed under a tier must classify back to that tier, or `assign_tiers` files a model under one band while `tier_of_model` reports another. - **No family overlaps between tiers.** A shared substring makes classification order-dependent — the same model id would land in a different band depending on which tier happened to be checked first. Mutation-tested: adding `pro` (already under `coding`) to `reasoning` fails with both tier names, and removing it passes. Also asserts each role's prompt names its own role, which catches the copy-paste-a-sibling mistake that a non-empty check would not. 1405 tests pass (79 new, mostly parametrized); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`nightly_core.conclude_skill` holds the markdown for `/nightly-conclude`, `/nightly-update`, `/nightly-init`, and `/nightly-bug` — 301 lines that every host install writes verbatim, and that no test referenced. A malformed constant breaks skill loading on all seven harnesses at once; a subtly edited one changes what the agent believes it may do. Two are human-only off-ramps, and the rules block records that an agent has **already self-concluded in production** — running `nightly conclude` on its own initiative, freezing the cascade and ending a session with unblocked work still on disk. The only thing preventing a repeat is the warning text inside these constants, and nothing asserted it was still there. Covered: frontmatter parses; `name:` matches the `/command` an operator types; body is non-empty; the prohibition appears in the **description** specifically, not just the body — the description is what a host shows in its skill listing, and often all the agent reads before deciding to invoke; each off-ramp names the correct wrap-up (`ideate` → `brief`), because "don't do this" without an alternative is how an agent talks itself back into doing it. Also pins that no fifth companion skill can be added without a test row, which is exactly how all four went uncovered. Method note: the first mutation I ran to validate the self-invocation guard did not actually apply — the literal I targeted didn't match — and the test passed for that reason, not because the guard was weak. Re-run with a regex that provably edited the file, it fails with the offending description and passes on restore. Worth recording: I nearly reported a verified guard on false evidence. 1428 tests pass (23 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three review findings, all verified against current code before acting. **Doctor treated a host as bound if its tier map was non-empty.** A host bound for only `coding` passed as configured while `lite` and `reasoning` silently fell through to the host CLI's default model. That partial shape is more dangerous than an empty map — an empty one at least looks unconfigured. `_check_model_tiers` now reports which tiers are missing per host, e.g. `codex (lite, reasoning)`. **The success detail sampled `sorted(hosts)[:1]`**, so with two hosts configured the second's bindings were never shown. It now lists every configured host. **RFC 007's Resolved #3 YAML presented four vendors' ids as shipped defaults.** Only `claude`, `cursor`, and `opencode` ship bindings; `codex`, `gemini`, and `antigravity` deliberately resolve to none. The sample now separates the shipped block from a commented, explicitly illustrative one. Worse than the finding reported: that sample also still carried `claude-sonnet-4-6` and `claude-opus-4-7`, superseded when the model table was amended earlier. A reader copying the sample would have configured two stale models and three phantom hosts. Zero stale vendor ids remain in the file. **The Context bullets still mapped reviewer→coding and researcher→reasoning**, directly contradicting the amended Resolved #4 five sections below. Both now match, and cite #4. Also fixes a pyrefly `bad-raise` in the new pr_feedback test file: `exc: object` should have been `BaseException | None`. 1446 tests pass (2 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`plans.py` was the second-weakest module at 74%, and the entire gap was one function: `append_pr_feedback`. That is an uncomfortable thing to have untested, because unlike the read paths it **mutates the operator's task state**. Get the round numbering wrong and each round overwrites the last reviewer's comments; get the frontmatter wrong and the plan loses `status`, dropping out of the cascade entirely. Both are now pinned, along with the details that only bite in rendering: - Rounds accumulate rather than reusing round 1. - `status` and `slug` survive the rewrite, and the plan still parses through `read_plan` after two appends. - `pr_last_reconciled_at` is stamped — `pick_pr_rescue` skips on it, so without it the same feedback is re-applied forever. - Groups render blocking → humans → bots. The order is the point: a blocking review is *why* the agent was routed here and must not sit below bot chatter. - A multi-line comment is quoted on every line; a half-quoted body breaks the blockquote and the remainder renders as plan prose. - An empty body still emits a quote line — `"".splitlines()` is `[]`, so the `or [""]` fallback is load-bearing and now has a test saying so. 74% → 98%; the three remaining statements are `list_plans` filesystem guards. Package total is 90%. Also closes task 0026, whose work landed in df9442d but was never marked done — the cascade surfaced it as in-flight this turn, which is the mechanism working. 1462 tests pass (16 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every `nightly run` task ultimately lands in this function, and none of it was exercised — the whole body was the module's coverage gap. Tested against real processes rather than a mocked asyncio, because the behaviour that matters is exactly what a mock would assume: stdout and stderr captured separately, exit code preserved (a swallowed non-zero reads as task success), stdin delivered, `cwd` honoured (a wrong one operates on the main checkout instead of the worktree), `TimeoutError` raised promptly rather than blocking the run behind one wedged task, `FileNotFoundError` propagated so per-host wrappers can name the missing CLI, and `timeout_s=None` meaning no deadline rather than zero. **One test written, then deleted, and the deletion is the point.** I wrote a check that the timed-out child is actually killed — poll `ps` for a uniquely-marked process after the timeout. It passed. Then I mutated `proc.kill()` out and it *still* passed: the event loop reaps the child during teardown regardless, so the assertion proved nothing about the kill. It is removed rather than shipped, with a comment in its place recording what was tried and why it does not work. A test that looks like a guard and isn't is worse than an acknowledged gap — it discourages anyone from writing the real one. Verifying that path needs the child's PID, which the runner does not expose; the honest fix is to surface it, not to assert around it. Related: the first version of that test used a fixed process marker and so matched orphans from earlier runs, including one it had leaked while being mutation-tested. Made unique per invocation before I concluded anything from it — a non-hermetic test would have "failed" on state it did not create. 1470 tests pass (8 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… 97%) `_bucket_for` maps GitHub's `state` / `conclusion` onto six buckets and `summarize_status` collapses a PR's checks into one verdict. Everything downstream keys off that verdict: `nightly ci` prints it and `pr_rescue` — cascade slot 5, a stated priority — routes on it. Neither function was covered. A mis-bucketed conclusion here is not a display bug. Reading `startup_failure` as anything but `fail` makes a broken PR look healthy and silently drops it from the rescue queue: the agent moves on to fresh work while the PR stays red all night. `stale` and `startup_failure` are the two easiest to get wrong — neither contains the word "fail". Pinned, in rough order of how expensive the mistake would be: - Every failure conclusion buckets as `fail`. - An unrecognised conclusion is `unknown`, never `pass`. When GitHub adds a new one, the failure mode must be "I can't tell", not "it's fine". - A PR with **no checks** is `unknown`, not passing-by-default — the difference between "CI has not reported" and "CI approved". - State beats conclusion while in flight, so a queued check carrying a previous run's `success` does not read green (or its `failure` red). - One red check outranks any number of green ones, following the declared `CHECK_STATUS_RANK` rather than re-deriving it. Process note: I wrote this against a `worst_bucket` function I assumed existed. It does not — the real name is `summarize_status`, and it takes `CICheck` objects rather than bucket strings. Caught at import, but the lesson is the same one as the mutation tests: check the API, don't predict it. 1493 tests pass (23 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`fetch_open_pr_issue_refs_via_gh` is the v0.0.11 fix for issue #27: an issue an open PR already claims must not be handed to the agent again. It was uncovered — which is uncomfortable given the session that wrote these tests spent most of a night inside the livelock that guard exists to prevent, and issue #30 documents the same shape 130 reroutes deep. Both failure directions are now pinned, because they fail differently: - **Under-reporting** re-picks covered work every boundary. All eleven closing-keyword forms GitHub documents are tested (close/closes/closed, fix/fixes/fixed, resolve/resolves/resolved, `Fixes:` with a colon, uppercase). A form this misses is an issue the cascade will grind on while a PR sits open against it. - **Over-reporting** silently drops real work from triage forever, which is quieter and worse. A bare `#9` must not read as a closing ref, and a bare mention on a *human* branch must not count — that asymmetry is deliberate and now has a test saying so. Also covered: the `gh` failure paths around it. No `gh` on PATH, expired token, timeout, rate-limit text, `null` title/body (gh emits null for an empty body — string-concatenating it would crash the scan and take the cascade with it). Every one degrades to "no refs known". Under-reporting is bad; raising into the cascade is worse. Process note, and it is the same note as last commit: I wrote the whole file calling `fetch_via_gh()` and `fetch_open_pr_issue_refs_via_gh()` with no arguments. Both require `root`. That is the fourth API I have predicted rather than read tonight, after two mutation tests that never applied and a `worst_bucket` that does not exist. Naming the pattern last turn did not stop it; reading the signature first is the only thing that will. 1529 tests pass (31 new); ruff, pyrefly, and `nightly verify` clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 38 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (43)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
* docs(readme): demo GIFs for the loop and the per-repo bootstrap Two short recordings rather than one long one: `docs/demo.gif` shows the working loop and sits under the pitch, `docs/demo-install.gif` shows `nightly init` and sits in the install section where the reader is deciding whether to run it. Both are reproducible, which is the point of scripting them instead of screen-recording by hand — `bash docs/demo-prep.sh <bare|init>` builds a throwaway repo and `vhs docs/<tape>` re-renders. They can be regenerated when the CLI output changes rather than slowly drifting out of date. Three things the recording had to get right: The loop demo transitions the task to `in_progress` before calling `nightly next`. That is not padding: `ready` is deliberately not a cascade source (only `in_progress` and an approved `blocked:` are), so a demo that skipped the transition would end on "no work — backlog is empty" — the opposite of the point. Sleeps are sized to the slowest observed run of each command. The first cut used 3s after `nightly init`, and the next command's keystrokes interleaved with init's own output mid-line, which reads as a bug. The install demo records the *per-repo* bootstrap, not `curl … | bash`. The binary install mutates the operator's own machine, which is not something a re-runnable demo should do. It ends on `ls -a` so the reader sees the entire footprint before running anything. Rendered with vhs, optimized with `gifsicle -O3 --lossy=30 --colors 64` — 40-44% smaller, and frame-checked afterwards to confirm the terminal text is still legible. 81KB and 47KB, both far under the 8MB README budget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * build: keep the venv off iCloud; unhide the .pth files it hides Under iCloud Drive every freshly written file inherits the parent's UF_HIDDEN flag, and `site.py` skips hidden `.pth` files. The editable install of the workspace packages therefore stops being applied — and the failure looks nothing like its cause. `uv sync` reports success, `uv sync --all-packages` reports success, and then all 1563 tests die with `ModuleNotFoundError: No module named 'nightly_core'` on a checkout that is completely healthy. Verified directly: every `.pth` under `.venv/lib` was flagged `hidden`. Same root cause and the same `chflags` remedy as ulmentflam/corpus-forge, which diagnosed it first. Relocating is the wider fix, so `make sync` now builds the venv under `$XDG_CACHE_HOME/nightly/venvs/<repo>-<hash>` and leaves `.venv` as a symlink to it. Nothing on the import path is under fileproviderd, which also rules out the eviction-to-`.icloud`-placeholder failure that the `chflags` fix alone would not have covered. `unhide-pth` stays as the narrower repair for anyone who runs `uv sync` directly, bypassing make. The path test mirrors `nightly_core.worktree.is_icloud_path`. Off macOS and outside iCloud, `VENV_DIR` is empty and every target behaves exactly as before — CI is untouched. `make venv-path` prints which branch is active, and `make nuke` now removes the out-of-tree venv too. Two traps worth recording, both hit while writing this: `case` cannot be used inside `$(shell …)`. make matches parentheses when it scans the function, so the unbalanced `)` closing a case arm terminates it early and the expansion silently comes back wrong — it produced the hash of an empty string. `grep` keeps every paren balanced. `.gitignore` needed a bare `.venv` alongside `.venv/`. A directory-only pattern does not match a symlink, so the relocated `.venv` showed up as untracked and would have been committed. Verified end-to-end: `make install` relocates and links, the `.pth` files come back unflagged, `import nightly_core` succeeds, and all 1563 tests pass. `nightly verify` clean on all five checks. Confirmed the non-iCloud branch separately by running the same Makefile from /tmp. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(rules): sync the in-repo contract with rules.py `rules.py` gained rule 12 (fleet doctrine) and rule 13 (pre-flight verification) in #39, but this repo's own AGENTS.md / CLAUDE.md were never regenerated — so Nightly running against Nightly was reading a contract two rules short of the one it ships to every other repo. `nightly update` rewrites the marker-delimited block; this is that output, committed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes RFC 007 and RFC 012, lands RFC 008 Phases A and B, and adds coverage to five modules that had none. 27 commits, rebased onto current
main.Model-tier routing (RFC 007, Phases B + C)
nightly initnow discovers each host's model controls rather than assuming them. It probes the host CLI's--helpfor a model flag and the model vocabulary it advertises, detects the harness from environment markers, and writes the resulting tier bindings intoconfig.yml. Live-verified against every host; it found--modelon opencode and gemini, which could not be confirmed by hand.A design note worth recording: RFC 007's Risks section proposed a doctor check validating configured model ids against what the host advertises. Testing that premise killed it —
claude --helpadvertises['fable', 'opus', 'sonnet', 'claude-fable-5']while the shipped defaults are['claude-haiku-4-5', 'claude-sonnet-5', 'claude-opus-5']. Zero overlap, because a pinned id beats a floating alias for overnight reproducibility. Such a check would have flagged every correct config as broken. A family-consistency check ships instead, and the RFC records why.Also here: the dispatches-by-tier briefing panel, and
nightly statussurfacing tiers, fleet caps, and handoff thresholds (it was also printingenabled=twice).Fleet parallelism and context handoff (RFC 012, Phases B + C)
Parallelism caps are enforced at admission rather than discovered after the fact — a dispatch that would exceed the global, per-tier, or worktree cap is refused with the binding limit named.
max_worktreesis checked before the branch is cut, so a refused dispatch leaves nothing behind.Context handoff thresholds are expressed as ratios of the dispatched model's own window (soft 0.25, hard 0.50) rather than absolute token counts. One setting therefore governs a mixed fleet: a 1M-window model hands off at 250K/500K and a 200K model at 50K/100K, without either being configured separately. The keepalive hook injects the handoff instruction at the threshold, and the digest and briefing surface pending handoffs.
Pre-flight verification (RFC 008, Phases A + B)
Rule 13 tells the agent to verify a deliverable exists before acting on an RFC item, and the briefing gains an auto-ticked-items panel. Delivered through the shared marker-delimited rules block rather than six duplicated host skill files.
Phase A3 is deliberately left unticked. It specifies a per-host skill token, and the shared rules block may or may not satisfy that intent — a judgment call that belongs to a human, not to the agent that wrote the rest of the phase.
Doctor and CLI
config.ymlnever learned. This repo was missing eight, half predating this branch, becausedoctoronly ever wrote config when the file was absent.cli.pyanddoctor.pyand had drifted — doctor's copy had silently lost thevault:andworktree:blocks entirely.nightly verifygains a configurable per-check timeout with an actionable message, aftermake checkhit the fixed 300s ceiling.Cascade
Skips RFC items already ticked on an unmerged local branch. Without this the cascade re-picks work that is done but not yet visible on
main— one of the shapes behind the issue #30 livelock.Coverage
Five modules had none of their critical paths exercised. Each test file opens with why the module matters, not what the function does:
headless.default_subprocess_runnerci_watch(red/green classifier)triage(open-PR issue-ref guard)plans.append_pr_feedbackdoctor(every check helper wired)Two notes on what is deliberately absent:
test_headless_runner.pydoes not assert that a timed-out child is killed. An attempt to verify it behaviourally passed withproc.kill()mutated out — the event loop reaps the child at teardown regardless — so it proved nothing. The file says so, in place of a test that would have looked like a guard without being one. Verifying it properly needs the child's PID, which the runner does not expose.test_run_loopconcurrency assertion is not re-touched here; it landed in fix: unflake the concurrency test; escalate a livelock reroute that stopped working (#30) #37.Verification
1539 tests pass.
nightly verifyclean on all five checks. All 27 commits SSH-signed.Rebase note: this branch previously sat on a pre-#36 base. Two of its commits (the concurrency-test fix and the issue #30 escalation) already reached
mainvia #37 and are excluded. The one real conflict waskeepalive_hook.py, where #37's escalation block and this branch's handoff work touch the same call site; the resolved source file is byte-identical to what this branch already had, and the merged test file carries the same 34 tests as before.🤖 Generated with Claude Code