From 9929dac8b9d0a8942df760a76e97c55ca2264388 Mon Sep 17 00:00:00 2001 From: letur Date: Thu, 10 Sep 2026 08:27:12 +0200 Subject: [PATCH] feat(reconcile): wake the lead when ready work has no actor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A task whose blocker clears becomes ready and nothing dispatches it. The reconciler may not — it reconciles and does not decide — and the lead, which may, is an interactive session that acts when spoken to. On 2026-09-10 a blocker cleared at 01:37 and the task went out at 08:05, because the operator typed "Status". So the pass after `collect` reads `plan`, and when the ready set has grown it types one line into the lead's terminal: which tasks are ready, and the command that sends them. Notifying is not deciding — nothing moves, and the choice is still the lead's. Three rules keep it from being worse than the silence it replaces, and `scripts/lib/notify_lead.py` owns all three: it fires on the transition and not on the pass, it refuses to type into a lead that is not at rest (the wake waits rather than being dropped), and it is silent when no lead session is running. What has been told is held in the reconciler's own runtime directory, so the loop still writes no record. `plan` makes five verbs where the selftest asserted four. The assertion is widened deliberately and argues in place why: it lists what the loop may WANT, and `plan` is a read that moves nothing, while `dispatch` and its neighbours change what runs and never may join it. Claude-Session: https://claude.ai/code/session_01Jff2KWdWmVMhCFAdZS6iRP --- .agents/skills/fleet-onboarding/SKILL.md | 7 +- .agents/skills/fleet-queue/SKILL.md | 8 +- .agents/skills/update-fleet/SKILL.md | 11 +- AGENTS.md | 31 ++- FLEET.md | 15 +- scripts/check.sh | 11 +- scripts/lib/notify_lead.py | 298 +++++++++++++++++++++++ scripts/reconcile-selftest.sh | 212 +++++++++++++++- scripts/reconcile.sh | 74 +++++- 9 files changed, 630 insertions(+), 37 deletions(-) create mode 100644 scripts/lib/notify_lead.py diff --git a/.agents/skills/fleet-onboarding/SKILL.md b/.agents/skills/fleet-onboarding/SKILL.md index b3317c0..26fd3fd 100644 --- a/.agents/skills/fleet-onboarding/SKILL.md +++ b/.agents/skills/fleet-onboarding/SKILL.md @@ -277,11 +277,16 @@ down because the user asked. ./scripts/reconcile.sh status ``` -Two things to pass on, once: +Three things to pass on, once: - It **reconciles and does not decide**. No dispatch, no cancel, no reorder, and it writes no record itself — `scripts/queue.sh` stays the only writer, which is what keeps the queue single-writer. +- It **will occasionally type one line into Mission Control**, and only ever + the same one: that N tasks are ready and nothing will dispatch them. That is + the loop telling the actor who may act; an unprompted line there is this and + not a bug. It arrives once per transition and never while the lead is + mid-turn. - To switch it off for good: `./scripts/reconcile.sh stop`. To bring it back: `./scripts/reconcile.sh start`. diff --git a/.agents/skills/fleet-queue/SKILL.md b/.agents/skills/fleet-queue/SKILL.md index f705c83..435862d 100644 --- a/.agents/skills/fleet-queue/SKILL.md +++ b/.agents/skills/fleet-queue/SKILL.md @@ -655,7 +655,7 @@ spotted. ``` It folds `watch` continuously and runs `collect`, `shepherd` and `refuel` on -their own intervals. Three things to know and nothing else: +their own intervals. Four things to know and nothing else: - **It changes nothing about how you work.** You still plan, still write briefs, still `dispatch`. It reconciles the RECORDS with the world; deciding @@ -663,6 +663,12 @@ their own intervals. Three things to know and nothing else: - **`queue.sh` is still the only writer.** The loop shells out and never touches a record. So `list` and the TUI pane cannot start disagreeing with it. +- **It will type one line at you, and only ever this one:** that N tasks are + ready and nothing will dispatch them. A blocker cleared, the loop may not + act on it, and you were not looking — on 2026-09-10 that sat for six and a + half hours until the operator asked for status. Treat the line as `plan` + already run: `dispatch`. It arrives once per transition and never mid-turn, + so a second one means the ready set has grown again. - **Run the commands anyway when you want an answer NOW.** `collect` is idempotent and reading it yourself is always allowed; the loop only means you are rarely the first to notice. diff --git a/.agents/skills/update-fleet/SKILL.md b/.agents/skills/update-fleet/SKILL.md index b349ecc..5f50489 100644 --- a/.agents/skills/update-fleet/SKILL.md +++ b/.agents/skills/update-fleet/SKILL.md @@ -108,11 +108,12 @@ lead's standing context — so it needs both the reinstall in §3 and the hand-over in §8. A change to `orchestration/voice.example.conf` (or your own `voice.conf`) needs the same two: it moves what the rendered payload calls you. -`scripts/lib/queue.py` is deliberately absent from this table. The -reconciler's loop never sources it — every pass calls `./scripts/queue.sh` as -a fresh subprocess, so a change there reaches the loop on its very next call, -with no restart needed. §6 covers only `scripts/reconcile.sh` itself, which -the running loop does hold in memory. +`scripts/lib/queue.py` is deliberately absent from this table, and so is +`scripts/lib/notify_lead.py`. The reconciler's loop never sources either — every +pass shells out to `./scripts/queue.sh` and to `python3 +scripts/lib/notify_lead.py` as fresh subprocesses, so a change to either +reaches the loop on its very next call, with no restart needed. §6 covers only +`scripts/reconcile.sh` itself, which the running loop does hold in memory. Run §3–§6 in any order, then §7, then §8 last — §8 is the one that cannot be automated, and everything else should already be done when you raise it. diff --git a/AGENTS.md b/AGENTS.md index a92150f..e0743bc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,8 +60,13 @@ names every path and the reason for each. what keeps the seam honest rather than merely asserted. - `orchestration/reconcile/` — the reconciler's runtime state: its supervisor's pid, the heartbeat proving its loop is ticking, its log, the advisory `nudge` - flag and the `down` flag. Written by `./scripts/reconcile.sh` and created on - first start. The loop's code is tracked; nothing it writes is. + flag, the `down` flag, and `notified.json` — which ready tasks the lead has + already been woken about, so a transition is told once. That last one is + runtime state and not a record for the same reason as all the others: "the + lead has been told" is true of one machine's loop and one conversation, and + writing it onto a task would make the loop a second writer over the queue. + Written by `./scripts/reconcile.sh` and created on first start. The loop's + code is tracked; nothing it writes is. - `interface/fleet_queue.lua` — the TUI queue pane, and the fleet's only live view of the queue, drawn in a thurbox column over the same records `queue.sh list` reads. `scripts/install-extension.sh` installs it @@ -179,14 +184,26 @@ The loop, driven by `./scripts/queue.sh`: that `ensure` honours the flag and `start` clears it. It consumes `queue.sh watch` continuously and calls `collect`, `shepherd` and `refuel` on separate intervals; its header argues every number -and is the full usage. Three things about it are load-bearing: +and is the full usage. Four things about it are load-bearing: -- **It writes nothing.** Every effect goes through `./scripts/queue.sh`, which - stays the only writer over the records. It calls exactly `watch`, - `collect`, `shepherd` and `refuel`, and - `scripts/reconcile-selftest.sh` asserts that the set is those four. +- **It writes no record.** Every effect on the queue goes through + `./scripts/queue.sh`, which + stays the only writer over the records; its own runtime directory above holds + the rest. It calls exactly `watch`, `collect`, + `shepherd`, `refuel` and the read-only `plan`, and + `scripts/reconcile-selftest.sh` asserts that the set is those five and argues + in place why a READ may join it while `dispatch` never may. - **It reconciles; it does not decide.** No dispatch, no cancel, no reorder, and it does not re-decide `refuel`'s rule about a spent quota window. +- **It tells the lead when the ready set grows, which is the one thing it says + out loud.** A task whose blocker clears is ready and has no actor: the loop + may not dispatch, and the lead only acts when spoken to — on 2026-09-10 that + cost six and a half hours. So after `collect` it reads `plan` and, when the + ready set has grown, types one line into the lead's terminal naming what is + ready and the command that sends it. Once per transition, never into a lead + mid-turn, and silent when no lead session is running. + `scripts/lib/notify_lead.py` owns those three rules. Notifying is not + deciding: nothing moves, and the choice is still the lead's. - **`nudge` is the accelerator and never the guarantee.** A worker's Claude Code `Stop` hook can call `./scripts/reconcile.sh nudge` to bring the periodic pass forward; a worker that died on a token limit fires no hook at diff --git a/FLEET.md b/FLEET.md index 0caf236..d122cf6 100644 --- a/FLEET.md +++ b/FLEET.md @@ -106,6 +106,11 @@ YAML by hand. Nothing to push — the map is gitignored. restarts nothing while that window is spent; see `AGENTS.md` and `fleet-queue` §5c. +**A landing is not an ending — it is what unblocks the next task.** So `plan` +is the last thing you run before you go quiet, not the first thing you run when +the operator asks. `collect` prints that advice every time something lands; it +is an instruction, and a ready set left sitting is work with no actor at all. + **Steps 4, 6 and 8 do not have to wait for you to remember them.** `./scripts/reconcile.sh ensure` runs a supervised loop that folds the event stream continuously and calls `collect`, `shepherd` and `refuel` on their own @@ -113,7 +118,9 @@ intervals — see `## What you are not`, which owns why an automation exists her at all. It reconciles and never decides: you still plan, still write briefs, still dispatch. When something is unexpectedly current, that is why; `./scripts/reconcile.sh status` says whether -it is up, and `logs` says what it has been doing. +it is up, and `logs` says what it has been doing. It is also the one thing that +will speak to you unprompted, and only ever to say that the ready set has grown +— read that line as `plan` already run for you, and dispatch. The operator watches all of that in the TUI queue pane rather than by asking you: `interface/fleet_queue.lua` draws the queue in a thurbox column, `F3` @@ -311,6 +318,12 @@ the rule protects is still true of it — - **It observes; it does not decide.** It folds `watch`, and it runs `collect`, `shepherd` and `refuel` on their own clocks. It never dispatches, cancels or reorders anything. Choosing what runs is still yours. +- **It wakes you when the ready set grows, and that is not it deciding.** A + task whose blocker clears has no actor — the loop may not dispatch it and you + are not looking — so it reads `plan` and types one line into your terminal + naming what is ready and the command that sends it. Once per transition, and + never while you are mid-turn. The decision it hands you is still yours to + make; what it took away was the six hours before you knew there was one. - **It writes no record.** Every effect goes through `./scripts/queue.sh`, which stays the only writer, exactly as the pane stays a pure reader. - **It is stoppable, and a stop stays stopped.** `orchestration/reconcile/down` diff --git a/scripts/check.sh b/scripts/check.sh index fc3aec1..44e921e 100755 --- a/scripts/check.sh +++ b/scripts/check.sh @@ -151,14 +151,15 @@ check_queue() { # The reconciler's lifecycle, the part of it that can break silently: it runs # `collect`, so a second instance or a stop that does not stop costs closed # tasks and reaped sessions. -# reconcile-selftest.sh proves adoption, a durable stop, and the two claims +# reconcile-selftest.sh proves adoption, a durable stop, and the three claims # that are specific to it — that the four cadences are four separate clocks, -# and that the ONLY thing it ever asks the queue to do is watch, collect, -# shepherd and refuel. It stubs the queue command, so it needs no thurbox, no -# `gh` and no network. +# that the ONLY things it ever asks the queue to do are watch, collect, +# shepherd, refuel and the read-only plan, and that the one line it sends the +# lead goes out on a transition rather than on every pass. It stubs the queue +# command and thurbox-cli, so it needs no thurbox, no `gh` and no network. check_reconcile() { if ./scripts/reconcile-selftest.sh >/dev/null; then - ok "reconcile: adopts rather than duplicates, a stop stays stopped, and it writes nothing" + ok "reconcile: adopts rather than duplicates, a stop stays stopped, it writes no record, and it wakes the lead once per transition" else # Re-run visibly: a failing claim is the whole message. ./scripts/reconcile-selftest.sh diff --git a/scripts/lib/notify_lead.py b/scripts/lib/notify_lead.py new file mode 100644 index 0000000..26b8d4c --- /dev/null +++ b/scripts/lib/notify_lead.py @@ -0,0 +1,298 @@ +#!/usr/bin/env python3 +"""Tell the lead that ready work exists which nothing will dispatch. + +WHY THIS EXISTS, measured. On 2026-09-10 at 01:37 a pull request merged, +`forge-agnostic/01-01-forge-seam` landed, and the `semantic-dependency` blocker +on `forge-agnostic/02-02-gitlab-adapter` cleared. That task was dispatched at +08:05, because the operator typed "Status" and the lead looked. Six hours and +twenty-eight minutes of work that was ready, unclaimed, and had no actor. + +Nothing was broken. The reconciler may not dispatch — `AGENTS.md` says it +reconciles and does not decide, and `reconcile-selftest.sh` holds it to the +verbs it is allowed to want. The lead is the only actor that may dispatch, and +it is an interactive session that acts when someone speaks to it. +`reconcile.sh nudge` wakes the LOOP; there was no path in the other direction. + +NOTIFYING IS NOT DECIDING, and that is the whole seam. This changes no record, +moves no task and launches nothing. It reads the ready set and puts it in front +of the one actor that may act on it, along with the command that acts. The loop +keeps its constraint; the lead keeps the decision. + +FOUR THINGS MAKE IT SURVIVABLE, and each is a way this could have been worse +than silence: + + IT FIRES ON THE TRANSITION. A loop that says "one task is ready" every + pass is turned off within the day, and then the six hours come back with + the notification disabled on top. So the delivered ready set is + remembered, and a pass whose ready set holds nothing new says nothing. + IT DOES NOT INTERRUPT A TURN. `session send` types into the lead's + terminal. Typed into a session mid-turn that is an interjection in the + middle of somebody's work — `shepherd` declines to touch a working + session for exactly this reason. Only a lead that has SAID it is at rest + is woken; anything else and the wake waits, which is not the same as + being dropped. + IT IS ONE LINE. The lead is a token budget. Which tasks, and the command + that sends them. No table, no narration. + IT SURVIVES THE LEAD NOT EXISTING. No lead session is an ordinary fleet — + the operator closed it, or never installed the extension. That produces + no send, no error and, because the note is deduplicated, one log line + rather than one per pass. + +WHERE THE STATE LIVES. In the reconciler's own runtime directory, beside its +pid, heartbeat and flags — never on the task. "The lead has been told" is a +fact about one machine's loop and one conversation; it is not part of what a +task IS, and writing it onto a record would make this the second writer over +the queue. `scripts/reconcile.sh` still writes no record. + +Usage (it is `scripts/reconcile.sh`'s, and nothing else's): + + ./scripts/queue.sh plan --json | python3 scripts/lib/notify_lead.py --state-dir DIR + +It prints a line worth logging, or nothing, and it exits 0 whatever happens. +A notifier that can fail the pass it rides on would cost the reconciler the +`collect` it just did, which is a real loss traded for a message. + +Environment: + FLEET_LEAD_SESSION the lead session's name, overriding the rendered + extension.toml. The gate uses it, since a rendered + manifest belongs to the operator's checkout and is not + something a test may write. + +Requires: python3. `thurbox-cli` for the send itself — without it there is +nothing to wake and this says so once. +""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import os +import shutil +import subprocess +import sys + +# THE ONLY TWO STATES A WAKE IS ALLOWED INTO, and the list is an allowlist +# because the failure modes are asymmetric. `idle` and `done` are the agent's +# own hook saying it is at rest — at rest between turns, and typing at it is +# how the operator would speak to it anyway. +# +# Everything else is refused, and `blocked` is the one to understand: it means +# the agent is sitting on a permission or input prompt, so text sent to it does +# not start a turn, it ANSWERS the dialog. `working` is a turn in flight. +# `running`, `uncovered` and `unreported` are not the agent saying anything at +# all — see the state table in `.agents/skills/thurbox-session/SKILL.md`, whose +# whole point is that the absence of a word is not `idle`. +AT_REST = ("idle", "done") + +# How many refs the one line spells before it stops naming them. Six is about +# what stays readable in a terminal; the count is always exact, and `dispatch` +# sends the whole set regardless of what was named. +NAMED = 6 + +STATE_FILE = "notified.json" + + +def _load_queue(): + """Load scripts/lib/queue.py under a name that is not `queue`. + + For `manifest_session` and `checkout_root`, so that "what is the lead + session called" has one answer here and in queue.py's control-plane guard + rather than two parsers that agree until someone renames the session. The + alias is what keeps this directory on sys.path from shadowing the standard + library's `queue`, and it is the same load fleet_status.py does. + """ + path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "queue.py") + spec = importlib.util.spec_from_file_location("fleet_queue", path) + module = importlib.util.module_from_spec(spec) + sys.modules["fleet_queue"] = module + spec.loader.exec_module(module) + return module + + +fleetqueue = _load_queue() + + +# --- what the loop remembers ------------------------------------------------- + + +def read_state(state_dir: str) -> dict: + try: + with open(os.path.join(state_dir, STATE_FILE)) as fh: + doc = json.load(fh) + except (OSError, json.JSONDecodeError): + return {} + return doc if isinstance(doc, dict) else {} + + +def write_state(state_dir: str, told: list, note: str) -> None: + """Remember what the lead has been told, and what was last logged about it. + + Best effort on purpose: a runtime directory that cannot be written is worth + a repeated notification, and is not worth failing a reconciler pass over. + """ + try: + os.makedirs(state_dir, exist_ok=True) + with open(os.path.join(state_dir, STATE_FILE), "w") as fh: + json.dump({"told": told, "note": note}, fh) + except OSError: + pass + + +def say(state_dir: str, told: list, note: str) -> int: + """Print `note` only if it is not the one already standing. + + The deduplication is the difference between a diagnosis and a wall. A lead + that is away for an hour, or busy for one, is one fact; at the collect + cadence it would otherwise be thirty log lines saying it again. + """ + if note and read_state(state_dir).get("note") != note: + print(note) + write_state(state_dir, told, note) + return 0 + + +# --- who to wake ------------------------------------------------------------- + + +def lead_name() -> str: + """What the lead session is called, or "" when nothing here can tell. + + The RENDERED extension.toml only. `install-extension.sh` writes it into the + control-plane clone, so its presence is the claim; the tracked + `extension.toml.in` beside it carries a `__LEAD_GLYPH__` placeholder, which + names a session that does not exist. Unknown stays silent — a fleet driven + without the extension is legitimate and may not be nagged by a guess. + """ + override = os.environ.get("FLEET_LEAD_SESSION", "").strip() + if override: + return override + root = fleetqueue.checkout_root() + name, _repo = fleetqueue.manifest_session(os.path.join(root, "extension.toml")) + if not name or "__" in name: + return "" + return name + + +def lead_session(name: str) -> tuple[str | None, str]: + """(session id, state) for the named session, or (None, why not). + + `session list` and not `session get`: the list carries `id` and `state` for + every session in one call and probes no pane, which is the cheaper of the + two answers and the one that does not touch the lead to ask about it. + """ + if not shutil.which("thurbox-cli"): + return None, "no thurbox-cli here, so there is nothing to wake" + try: + out = subprocess.run( + ["thurbox-cli", "session", "list", "--json"], + capture_output=True, + text=True, + timeout=10, + ) + except (OSError, subprocess.SubprocessError) as exc: + return None, f"thurbox-cli session list failed: {exc}" + if out.returncode != 0: + return None, "thurbox-cli session list failed" + try: + rows = json.loads(out.stdout) + except json.JSONDecodeError: + return None, "thurbox-cli session list did not answer JSON" + if not isinstance(rows, list): + return None, "thurbox-cli session list did not answer a list" + for row in rows: + if isinstance(row, dict) and row.get("name") == name and row.get("id"): + return str(row["id"]), str(row.get("state") or "unknown") + return None, f"no session named {name!r} is running" + + +def wake(sid: str, text: str) -> tuple[bool, str]: + """Type one line into the lead's terminal. + + `session send`, not `message send`. The mailbox is what POLICY.md forbids + workers, because an arriving worker message interrupts whoever is talking + to the lead and routine completion is not worth that. This is the case the + same rule leaves open: it goes only to a lead that has said it is at rest, + so there is nobody to interrupt, and the thing it carries is the one fact + the lead cannot learn any other way. + """ + try: + out = subprocess.run( + ["thurbox-cli", "session", "send", sid, text], + capture_output=True, + text=True, + timeout=20, + ) + except (OSError, subprocess.SubprocessError) as exc: + return False, f"session send failed: {exc}" + if out.returncode != 0: + detail = (out.stderr or out.stdout or "").strip().splitlines() + return False, "session send failed: " + (detail[-1] if detail else "no output") + return True, "" + + +def message(ready: list) -> str: + """One line: what is ready, and the command that sends it.""" + shown = ", ".join(ready[:NAMED]) + more = "" if len(ready) <= NAMED else f" and {len(ready) - NAMED} more" + return ( + f"fleet reconciler: {len(ready)} task(s) ready and nothing will dispatch " + f"them — {shown}{more}. Run ./scripts/queue.sh dispatch" + ) + + +# --- the pass ---------------------------------------------------------------- + + +def main(argv: list) -> int: + ap = argparse.ArgumentParser( + description="wake the lead when ready work has no actor", + ) + ap.add_argument( + "--state-dir", + required=True, + help="the reconciler's runtime directory, where what-was-told lives", + ) + args = ap.parse_args(argv) + + try: + plan = json.loads(sys.stdin.read() or "{}") + except (json.JSONDecodeError, ValueError): + return 0 + if not isinstance(plan, dict): + return 0 + ready = sorted(str(r) for r in (plan.get("ready") or [])) + + # PRUNE, THEN COMPARE, and the prune is what makes a re-entry news. A task + # that leaves the ready set — dispatched, blocked again, abandoned — is + # forgotten, so if it comes back it is a transition again rather than + # something the lead was already told about weeks ago. + told = [r for r in read_state(args.state_dir).get("told", []) if r in ready] + fresh = [r for r in ready if r not in told] + if not fresh: + return say(args.state_dir, told, "") + + name = lead_name() + if not name: + return say(args.state_dir, told, "ready work, but no lead session is configured") + sid, state = lead_session(name) + if not sid: + return say(args.state_dir, told, f"ready work, but {state}") + if state not in AT_REST: + return say(args.state_dir, told, f"ready work; {name} is {state} — the wake waits") + + sent, why = wake(sid, message(ready)) + if not sent: + return say(args.state_dir, told, f"could not wake {name}: {why}") + + # Only a delivered wake is remembered, so a send that failed is retried on + # the next pass rather than swallowed. + return say( + args.state_dir, + ready, + f"woke {name}: {len(ready)} task(s) ready — {', '.join(ready[:NAMED])}", + ) + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/reconcile-selftest.sh b/scripts/reconcile-selftest.sh index 6c02b05..c36e922 100755 --- a/scripts/reconcile-selftest.sh +++ b/scripts/reconcile-selftest.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Prove the reconciler's claims, rather than assert them. # -# `scripts/reconcile.sh` makes seven promises. Most are invisible until the day +# `scripts/reconcile.sh` makes eight promises. Most are invisible until the day # they cost something — a second loop closing tasks under the first, a stop the # next onboarding run undoes, a reconciler that decided to dispatch. Each gets a # test here, against a throwaway queue, a throwaway runtime directory and a @@ -27,21 +27,31 @@ # 7. A PHANTOM IS NEVER "UP". A supervisor restarting a tick loop that cannot # run at all is a live process and not a running reconciler — never # adopted, never reported healthy. +# 8. IT WAKES THE LEAD, ON THE TRANSITION AND NOT ON THE PASS. Ready work +# that nothing will dispatch reaches the lead ONCE, when the ready set +# becomes non-empty or grows; a lead mid-turn is not interrupted and the +# wake waits for it; and a fleet with no lead session at all produces no +# send and no error. # # Tests 2 and 5 are the ones to read first. 2 is the operator's stop actually # meaning stop; 5 is the rule that keeps this a reconciler and not a second -# control plane. +# control plane. 8 is the one that reads oddly beside 5 and does not break it: +# telling the lead is not deciding, and `plan` is a read. # # HOW IT RUNS OFFLINE. `FLEET_RECONCILE_QUEUE_CMD` is the seam — the same shape # as `FLEET_QUEUE_WATCH_CMD` in queue.sh — and this replaces the real # `queue.sh` with a recorder that appends its own argv to a file and prints -# what the loop expects to read. So nothing here needs thurbox, `gh`, -# `quota-axi` or a network, and the intervals are compressed to seconds so a -# whole day of cadence fits in a few of them. +# what the loop expects to read. `thurbox-cli` is stubbed on PATH beside it, so +# test 8's lead is a state the test writes to a file rather than a session on +# this machine. Nothing here needs thurbox, `gh`, `quota-axi` or a network, and +# the intervals are compressed to seconds so a whole day of cadence fits in a +# few of them. # # Usage: scripts/reconcile-selftest.sh (also: ./scripts/check.sh reconcile) # -# Requires: bash. Nothing else — the queue command is a stub. +# Requires: bash and python3 — the latter because the notifier test drives +# `scripts/lib/notify_lead.py`, which is real code and not a stub. The queue +# command and thurbox-cli are both stubs. set -uo pipefail @@ -124,6 +134,14 @@ watch) collect) echo "collect: nothing new" ;; shepherd) echo "shepherd: 0 open pull request(s)" ;; refuel) echo "refuel: no task here is holding a session" ;; +plan) + # The FIFTH verb, and the only read among them. $READY is the ready set + # the test is driving; the real `plan --json` carries `waiting` and + # `overlaps` beside it, and the notifier reads neither. + refs="" + [ -s "$READY" ] && refs="$(sed 's/.*/"&"/' "$READY" | paste -sd, -)" + printf '{"ready":[%s],"waiting":[],"overlaps":[]}\n' "$refs" + ;; *) echo "queue-stub: REFUSING an unexpected subcommand: $*" >&2 exit 3 @@ -133,8 +151,54 @@ exit 0 STUB chmod +x "$tmp/queue-stub.sh" +# --- thurbox-cli, stubbed on PATH ------------------------------------------- +# +# The lead is a thurbox session, so waking it needs two answers from +# thurbox-cli and nothing else: which session carries the lead's name and what +# state it is in (`session list`), and then the send itself. Both are stubbed +# here so the test drives the lead's state directly. +# +# $LEAD_STATE ABSENT MEANS NO LEAD. Not an error, not an empty state — the +# session list simply does not have that row, which is the shape of a fleet +# whose lead is not running, and of one that never had the extension +# installed. +tbxbin="$tmp/bin" +mkdir -p "$tbxbin" +cat >"$tbxbin/thurbox-cli" <<'SH' +#!/usr/bin/env bash +case "${1:-} ${2:-}" in +"session list") + if [ -s "$LEAD_STATE" ]; then + printf '[{"id":"lead-uuid","name":"%s","state":"%s"}]\n' \ + "$LEAD_NAME" "$(cat "$LEAD_STATE")" + else + printf '[]\n' + fi + ;; +"session send") + printf '%s\n' "$4" >>"$SENDS" + ;; +*) exit 0 ;; +esac +exit 0 +SH +chmod +x "$tbxbin/thurbox-cli" +PATH="$tbxbin:$PATH" +export PATH + export CALLS="$calls" export MOVED="$tmp/moved" +export READY="$tmp/ready" +export SENDS="$tmp/sends" +export LEAD_STATE="$tmp/lead-state" +# The lead's name is normally read out of the rendered extension.toml, which +# belongs to the operator's own checkout and is not something a gate may write. +# FLEET_LEAD_SESSION is the override that seam exists for. +export LEAD_NAME="Gate Control" +export FLEET_LEAD_SESSION="$LEAD_NAME" +: >"$READY" +: >"$SENDS" +printf 'idle\n' >"$LEAD_STATE" export FLEET_QUEUE_DIR="$tmp/queue" mkdir -p "$FLEET_QUEUE_DIR" export FLEET_RECONCILE_DIR="$tmp/rt" @@ -233,17 +297,145 @@ else fi echo 0 >"$MOVED" +# --- 8. it wakes the lead on the transition, and only then ------------------- +# +# The failure this exists for, measured: on 2026-09-10 a blocker cleared at +# 01:37 and the task it freed was dispatched at 08:05, because the reconciler +# may not dispatch and nothing told the lead there was anything to dispatch. +# Six and a half hours of ready, unclaimed work with no actor. +# +# What is proved here is the shape that makes the fix survivable rather than +# the fix itself. A loop that says "one task is ready" every twenty seconds +# gets turned off within the day, so the assertion is that it fires on the +# TRANSITION and is silent on every pass in between. + +count_sends() { grep -c . "$SENDS" 2>/dev/null || true; } +# shellcheck disable=SC2317,SC2329 # invoked indirectly, as wait_for's predicate +sends_atleast() { [ "$(count_sends)" -ge "$1" ]; } + +if [ "$(count_sends)" = "0" ]; then + pass "an empty ready set wakes nobody" +else + fail "an empty ready set wakes nobody" "$(cat "$SENDS")" +fi + +# The transition: the ready set becomes non-empty. +printf 'alpha/01-first\n' >"$READY" +if wait_for 20 sends_atleast 1; then + pass "ready work reaches the lead" +else + fail "ready work reaches the lead" "sends: $(cat "$SENDS")${nl}$(cat "$FLEET_RECONCILE_DIR/reconcile.log")" +fi + +woke="$(cat "$SENDS")" +expect "the message names the task that is ready" "alpha/01-first" "$woke" +expect "the message carries the command that sends it" "queue.sh dispatch" "$woke" +if [ "$(wc -l <"$SENDS")" = "1" ]; then + pass "the wake is one line, not a report" +else + fail "the wake is one line" "$woke" +fi + +# AND THEN IT STOPS. Several collect intervals pass with the same task still +# ready and still undispatched; the lead hears nothing more about it. +sleep 5 +if [ "$(count_sends)" = "1" ]; then + pass "it stays quiet while the same set stays ready" +else + fail "it stays quiet while the same set stays ready" "$(cat "$SENDS")" +fi + +# A GROWING SET IS A NEW TRANSITION. A second task becoming ready is news, and +# the message names the whole ready set rather than only the new arrival — +# `dispatch` is going to send all of it. +printf 'alpha/01-first\nbeta/02-second\n' >"$READY" +if wait_for 20 sends_atleast 2; then + pass "a growing ready set is a fresh transition" +else + fail "a growing ready set is a fresh transition" "$(cat "$SENDS")" +fi +expect "the second message names both ready tasks" "beta/02-second" "$(tail -1 "$SENDS")" +expect "and names the one that was already ready" "alpha/01-first" "$(tail -1 "$SENDS")" + +# A WORKING LEAD IS NOT INTERRUPTED, and the wake is not lost either. Typing +# into a session mid-turn is how a fix gets half-applied — `shepherd` declines +# to touch a working session for the same reason — so the notice waits for the +# lead to come to rest instead of being dropped or forced through. +printf 'working\n' >"$LEAD_STATE" +sent_before="$(count_sends)" +printf 'alpha/01-first\nbeta/02-second\ngamma/03-third\n' >"$READY" +sleep 5 +if [ "$(count_sends)" = "$sent_before" ]; then + pass "a lead mid-turn is not interrupted" +else + fail "a lead mid-turn is not interrupted" "$(tail -1 "$SENDS")" +fi +expect "and the log says the wake is waiting rather than gone" "the wake waits" \ + "$(cat "$FLEET_RECONCILE_DIR/reconcile.log")" + +printf 'idle\n' >"$LEAD_STATE" +if wait_for 20 sends_atleast $((sent_before + 1)); then + pass "the held wake lands once the lead is at rest" +else + fail "the held wake lands once the lead is at rest" "$(cat "$SENDS")" +fi +expect "and it names the task that arrived while the lead was busy" "gamma/03-third" "$(tail -1 "$SENDS")" + +# NO LEAD, NO ERROR STORM. A fleet whose lead session is not running is a +# normal fleet — the operator closed it, or the extension was never installed. +# The loop must carry on folding and say nothing it cannot act on twice. +: >"$LEAD_STATE" +sent_before="$(count_sends)" +watch_before="$(count_calls watch)" +printf 'alpha/01-first\nbeta/02-second\ngamma/03-third\ndelta/04-fourth\n' >"$READY" +sleep 5 +if [ "$(count_sends)" = "$sent_before" ]; then + pass "no lead session means no send" +else + fail "no lead session means no send" "$(tail -1 "$SENDS")" +fi +if [ "$(count_calls watch)" -gt "$watch_before" ]; then + pass "and the loop keeps folding regardless" +else + fail "and the loop keeps folding regardless" "watch stuck at $watch_before" +fi +# One line about it, not one per pass: the note is deduplicated against the +# last one, so a lead that is away for an hour costs the log a single entry. +absent="$(grep -c "no session named" "$FLEET_RECONCILE_DIR/reconcile.log" 2>/dev/null || true)" +if [ "${absent:-0}" -le 1 ]; then + pass "an absent lead is reported once, not once per pass" +else + fail "an absent lead is reported once" "$absent log lines say so" +fi + +printf 'idle\n' >"$LEAD_STATE" +: >"$READY" + # --- 5. it is not a writer --------------------------------------------------- # # The stub refuses any subcommand it was not taught, so a `dispatch` would have # failed the loop already. This states the claim positively as well: over the # whole run, the set of things the reconciler asked the queue to do is exactly -# the four reconciling ones plus the `root` precondition check. +# the four reconciling ones, plus `plan`, plus the `root` precondition check. +# +# WHY `plan` IS IN THIS SET AND `dispatch` NEVER CAN BE. This list is not a +# list of harmless commands; it is the list of things the loop is allowed to +# want. `plan` READS — it prints the ready set, the waiting set and why each +# one waits, and it writes nothing and moves nothing. The loop asks it so that +# test 8 above has something to tell the lead about, and telling the lead is +# not deciding: the ready set goes to the actor that may act on it, and that +# actor is still the lead. +# +# `dispatch`, `add`, `block`, `archive` and `reap` all CHANGE what runs, and a +# loop that could call one of them would be a second control plane with no +# operator in it. Widening this set again is a decision about power, not about +# convenience — a read that answers a question the loop already needs may join +# it, and nothing that acts ever may. verbs="$(awk '{print $1}' "$calls" | sort -u | tr '\n' ' ')" -if [ "$verbs" = "collect refuel root shepherd watch " ]; then - pass "it calls only watch/collect/shepherd/refuel — never dispatch, add or reap" +if [ "$verbs" = "collect plan refuel root shepherd watch " ]; then + pass "it calls watch/collect/shepherd/refuel and the read-only plan — never dispatch, add or reap" else - fail "it calls only the reconciling subcommands" "called: $verbs" + fail "it calls only the reconciling subcommands, plus the read-only plan" "called: $verbs" fi # It writes to its own runtime directory and to nothing else. The queue diff --git a/scripts/reconcile.sh b/scripts/reconcile.sh index d3dec8b..ae00ee8 100755 --- a/scripts/reconcile.sh +++ b/scripts/reconcile.sh @@ -48,16 +48,31 @@ # `start` the operator asking for it back. It CLEARS the flag. That is the # whole difference between the two. # -# IT WRITES NOTHING ITSELF. Every effect it has goes through +# IT WRITES NO RECORD. Every effect it has on the queue goes through # `./scripts/queue.sh`, which stays the only writer over the records — the # rule that keeps the queue single-writer. Grep this file for a write to a -# task and you will not find one. +# task and you will not find one. Its own runtime directory is the exception +# and is not one: a pid, a heartbeat, a log, the flags and `notified.json` are +# facts about this loop on this machine, not about any task. # # IT DOES NOT DECIDE WHAT RUNS. No dispatch, no cancel, no reorder. It # reconciles recorded state with observed state; choosing the work stays the # lead's. It does not second-guess `refuel` either — the rule that nothing is # restarted into a spent quota window lives there, and this calls the command. # +# BUT IT DOES SAY WHEN THERE IS SOMETHING TO DECIDE, which is the one thing it +# tells anybody. A task whose blocker clears becomes READY and has no actor: +# this loop may not dispatch it, and the lead, which may, only acts when spoken +# to. On 2026-09-10 that gap ran from 01:37 to 08:05 and the operator closed it +# by typing "Status". So the pass that follows `collect` reads the ready set and +# wakes the lead when it has grown — once per transition, never into a turn in +# flight, and silently when there is no lead session at all. +# +# Notifying is not deciding. Nothing is dispatched, no record moves, and the +# choice is still the lead's; all that changed is that it knows there is one to +# make. `scripts/lib/notify_lead.py` owns the rules that keep it from becoming +# noise, and reconcile-selftest.sh's test 8 holds it to them. +# # Usage: # scripts/reconcile.sh ensure # start unless running or asked down # scripts/reconcile.sh start # start, and clear a previous `stop` @@ -97,6 +112,11 @@ # pull request's own CI does not change state faster than that, so # a tighter interval would ask the forge the same question several # times for one answer. +# notify collect's clock, and no clock of its own. What makes a task ready +# is a blocker clearing, and what clears a blocker is a landing +# `collect` has just recorded — so a fifth interval would only ask +# the same question at a second, worse moment. It reads local +# records and one local socket, so it costs nothing to ride along. # # Every one is overridable for a test or an unusual fleet; see Environment. # @@ -111,12 +131,16 @@ # FLEET_RECONCILE_SHEPHERD_SECS seconds between shepherds (default 900) # FLEET_QUEUE_DIR the queue to reconcile (default: THIS # CHECKOUT's — see scripts/queue.sh root) +# FLEET_LEAD_SESSION the lead session to wake (default: the name in +# the rendered extension.toml). Read by +# scripts/lib/notify_lead.py, which is where it +# is argued. # # Requires: python3 and whatever the pass it is running needs — thurbox-cli for -# `watch` and `refuel`, `gh` for `collect` and `shepherd`, `quota-axi` for the -# fuel reading. Every one of those degrades to "could not check" inside -# queue.sh rather than to a guess, so a missing tool costs its own pass and -# never the loop. +# `watch`, `refuel` and the notification, `gh` for `collect` and `shepherd`, +# `quota-axi` for the fuel reading. Every one of those degrades to "could not +# check" inside queue.sh rather than to a guess, so a missing tool costs its own +# pass and never the loop. set -uo pipefail @@ -234,6 +258,32 @@ run_pass() { return 0 } +# THE ONE THING THIS LOOP SAYS OUT LOUD, and the one place it asks the queue a +# question rather than telling it to reconcile something. +# +# A task whose blocker clears becomes ready and has NO ACTOR. This loop may not +# dispatch — that is the constraint the whole design rests on — and the lead, +# which may, is an interactive session that acts when someone speaks to it. On +# 2026-09-10 that gap was six and a half hours long and the operator closed it +# by hand. `scripts/lib/notify_lead.py` owns the rest: the transition test that +# keeps this from being a stream of the same sentence, the refusal to type into +# a lead mid-turn, and the silence when there is no lead at all. +# +# `plan` IS THE FIFTH VERB and it is a READ. It prints the ready set and writes +# nothing; the assertion in reconcile-selftest.sh names it and argues why it may +# be there while `dispatch` never can be. Telling the lead is not deciding. +# +# It cannot fail the pass. Every branch inside the notifier exits 0, and this +# returns 0 regardless — a message is not worth the `collect` the loop just did. +notify_lead() { + local plan out + plan="$($QUEUE_CMD plan --json 2>/dev/null)" || return 0 + [ -n "$plan" ] || return 0 + out="$(printf '%s\n' "$plan" | python3 scripts/lib/notify_lead.py --state-dir "$RT" 2>&1)" + [ -n "$out" ] && log "notify: $out" + return 0 +} + # The loop. Read the cadences in the header before changing a number here. tick() { # Validated BEFORE the first heartbeat, so a reconciler that cannot run the @@ -245,7 +295,7 @@ tick() { return 2 fi - local last_collect=0 last_refuel=0 last_shepherd=0 nudged stamp + local last_collect=0 last_refuel=0 last_shepherd=0 nudged did_collect stamp while :; do # The down flag is checked at the TOP of every pass as well as by the # supervisor, so a `stop` that lands mid-pass is honoured at the next @@ -269,9 +319,11 @@ tick() { # collect first, and on its own: it is the one that CLOSES tasks, and # shepherd's view of which pull requests still matter is better for # running after it. + did_collect=0 if [ "$nudged" -eq 1 ] || [ $((stamp - last_collect)) -ge "$COLLECT_SECS" ]; then run_pass collect always collect last_collect="$stamp" + did_collect=1 fi if [ $((stamp - last_shepherd)) -ge "$SHEPHERD_SECS" ]; then run_pass shepherd always shepherd @@ -282,6 +334,14 @@ tick() { last_refuel="$stamp" fi + # LAST, and on collect's clock, because it is the only pass that + # looks OUTWARD. The three above reconcile records with the world; + # this one reads the result and, when the ready set has just grown, + # tells the one actor that may act on it. Running it after them means + # it sees the landings collect just recorded rather than the ones it + # recorded a pass ago — which is the whole latency this exists to cut. + [ "$did_collect" -eq 1 ] && notify_lead + trim_log # And then the continuous half, which is also what paces the loop: this