diff --git a/.agents/skills/fleet-onboarding/SKILL.md b/.agents/skills/fleet-onboarding/SKILL.md index b3317c0..26fd3fd 100644 --- a/.agents/skills/fleet-onboarding/SKILL.md +++ b/.agents/skills/fleet-onboarding/SKILL.md @@ -277,11 +277,16 @@ down because the user asked. ./scripts/reconcile.sh status ``` -Two things to pass on, once: +Three things to pass on, once: - It **reconciles and does not decide**. No dispatch, no cancel, no reorder, and it writes no record itself — `scripts/queue.sh` stays the only writer, which is what keeps the queue single-writer. +- It **will occasionally type one line into Mission Control**, and only ever + the same one: that N tasks are ready and nothing will dispatch them. That is + the loop telling the actor who may act; an unprompted line there is this and + not a bug. It arrives once per transition and never while the lead is + mid-turn. - To switch it off for good: `./scripts/reconcile.sh stop`. To bring it back: `./scripts/reconcile.sh start`. diff --git a/.agents/skills/fleet-queue/SKILL.md b/.agents/skills/fleet-queue/SKILL.md index f705c83..435862d 100644 --- a/.agents/skills/fleet-queue/SKILL.md +++ b/.agents/skills/fleet-queue/SKILL.md @@ -655,7 +655,7 @@ spotted. ``` It folds `watch` continuously and runs `collect`, `shepherd` and `refuel` on -their own intervals. Three things to know and nothing else: +their own intervals. Four things to know and nothing else: - **It changes nothing about how you work.** You still plan, still write briefs, still `dispatch`. It reconciles the RECORDS with the world; deciding @@ -663,6 +663,12 @@ their own intervals. Three things to know and nothing else: - **`queue.sh` is still the only writer.** The loop shells out and never touches a record. So `list` and the TUI pane cannot start disagreeing with it. +- **It will type one line at you, and only ever this one:** that N tasks are + ready and nothing will dispatch them. A blocker cleared, the loop may not + act on it, and you were not looking — on 2026-09-10 that sat for six and a + half hours until the operator asked for status. Treat the line as `plan` + already run: `dispatch`. It arrives once per transition and never mid-turn, + so a second one means the ready set has grown again. - **Run the commands anyway when you want an answer NOW.** `collect` is idempotent and reading it yourself is always allowed; the loop only means you are rarely the first to notice. diff --git a/.agents/skills/update-fleet/SKILL.md b/.agents/skills/update-fleet/SKILL.md index b349ecc..5f50489 100644 --- a/.agents/skills/update-fleet/SKILL.md +++ b/.agents/skills/update-fleet/SKILL.md @@ -108,11 +108,12 @@ lead's standing context — so it needs both the reinstall in §3 and the hand-over in §8. A change to `orchestration/voice.example.conf` (or your own `voice.conf`) needs the same two: it moves what the rendered payload calls you. -`scripts/lib/queue.py` is deliberately absent from this table. The -reconciler's loop never sources it — every pass calls `./scripts/queue.sh` as -a fresh subprocess, so a change there reaches the loop on its very next call, -with no restart needed. §6 covers only `scripts/reconcile.sh` itself, which -the running loop does hold in memory. +`scripts/lib/queue.py` is deliberately absent from this table, and so is +`scripts/lib/notify_lead.py`. The reconciler's loop never sources either — every +pass shells out to `./scripts/queue.sh` and to `python3 +scripts/lib/notify_lead.py` as fresh subprocesses, so a change to either +reaches the loop on its very next call, with no restart needed. §6 covers only +`scripts/reconcile.sh` itself, which the running loop does hold in memory. Run §3–§6 in any order, then §7, then §8 last — §8 is the one that cannot be automated, and everything else should already be done when you raise it. diff --git a/AGENTS.md b/AGENTS.md index a92150f..e0743bc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,8 +60,13 @@ names every path and the reason for each. what keeps the seam honest rather than merely asserted. - `orchestration/reconcile/` — the reconciler's runtime state: its supervisor's pid, the heartbeat proving its loop is ticking, its log, the advisory `nudge` - flag and the `down` flag. Written by `./scripts/reconcile.sh` and created on - first start. The loop's code is tracked; nothing it writes is. + flag, the `down` flag, and `notified.json` — which ready tasks the lead has + already been woken about, so a transition is told once. That last one is + runtime state and not a record for the same reason as all the others: "the + lead has been told" is true of one machine's loop and one conversation, and + writing it onto a task would make the loop a second writer over the queue. + Written by `./scripts/reconcile.sh` and created on first start. The loop's + code is tracked; nothing it writes is. - `interface/fleet_queue.lua` — the TUI queue pane, and the fleet's only live view of the queue, drawn in a thurbox column over the same records `queue.sh list` reads. `scripts/install-extension.sh` installs it @@ -179,14 +184,26 @@ The loop, driven by `./scripts/queue.sh`: that `ensure` honours the flag and `start` clears it. It consumes `queue.sh watch` continuously and calls `collect`, `shepherd` and `refuel` on separate intervals; its header argues every number -and is the full usage. Three things about it are load-bearing: +and is the full usage. Four things about it are load-bearing: -- **It writes nothing.** Every effect goes through `./scripts/queue.sh`, which - stays the only writer over the records. It calls exactly `watch`, - `collect`, `shepherd` and `refuel`, and - `scripts/reconcile-selftest.sh` asserts that the set is those four. +- **It writes no record.** Every effect on the queue goes through + `./scripts/queue.sh`, which + stays the only writer over the records; its own runtime directory above holds + the rest. It calls exactly `watch`, `collect`, + `shepherd`, `refuel` and the read-only `plan`, and + `scripts/reconcile-selftest.sh` asserts that the set is those five and argues + in place why a READ may join it while `dispatch` never may. - **It reconciles; it does not decide.** No dispatch, no cancel, no reorder, and it does not re-decide `refuel`'s rule about a spent quota window. +- **It tells the lead when the ready set grows, which is the one thing it says + out loud.** A task whose blocker clears is ready and has no actor: the loop + may not dispatch, and the lead only acts when spoken to — on 2026-09-10 that + cost six and a half hours. So after `collect` it reads `plan` and, when the + ready set has grown, types one line into the lead's terminal naming what is + ready and the command that sends it. Once per transition, never into a lead + mid-turn, and silent when no lead session is running. + `scripts/lib/notify_lead.py` owns those three rules. Notifying is not + deciding: nothing moves, and the choice is still the lead's. - **`nudge` is the accelerator and never the guarantee.** A worker's Claude Code `Stop` hook can call `./scripts/reconcile.sh nudge` to bring the periodic pass forward; a worker that died on a token limit fires no hook at diff --git a/FLEET.md b/FLEET.md index 0caf236..d122cf6 100644 --- a/FLEET.md +++ b/FLEET.md @@ -106,6 +106,11 @@ YAML by hand. Nothing to push — the map is gitignored. restarts nothing while that window is spent; see `AGENTS.md` and `fleet-queue` §5c. +**A landing is not an ending — it is what unblocks the next task.** So `plan` +is the last thing you run before you go quiet, not the first thing you run when +the operator asks. `collect` prints that advice every time something lands; it +is an instruction, and a ready set left sitting is work with no actor at all. + **Steps 4, 6 and 8 do not have to wait for you to remember them.** `./scripts/reconcile.sh ensure` runs a supervised loop that folds the event stream continuously and calls `collect`, `shepherd` and `refuel` on their own @@ -113,7 +118,9 @@ intervals — see `## What you are not`, which owns why an automation exists her at all. It reconciles and never decides: you still plan, still write briefs, still dispatch. When something is unexpectedly current, that is why; `./scripts/reconcile.sh status` says whether -it is up, and `logs` says what it has been doing. +it is up, and `logs` says what it has been doing. It is also the one thing that +will speak to you unprompted, and only ever to say that the ready set has grown +— read that line as `plan` already run for you, and dispatch. The operator watches all of that in the TUI queue pane rather than by asking you: `interface/fleet_queue.lua` draws the queue in a thurbox column, `F3` @@ -311,6 +318,12 @@ the rule protects is still true of it — - **It observes; it does not decide.** It folds `watch`, and it runs `collect`, `shepherd` and `refuel` on their own clocks. It never dispatches, cancels or reorders anything. Choosing what runs is still yours. +- **It wakes you when the ready set grows, and that is not it deciding.** A + task whose blocker clears has no actor — the loop may not dispatch it and you + are not looking — so it reads `plan` and types one line into your terminal + naming what is ready and the command that sends it. Once per transition, and + never while you are mid-turn. The decision it hands you is still yours to + make; what it took away was the six hours before you knew there was one. - **It writes no record.** Every effect goes through `./scripts/queue.sh`, which stays the only writer, exactly as the pane stays a pure reader. - **It is stoppable, and a stop stays stopped.** `orchestration/reconcile/down` diff --git a/scripts/check.sh b/scripts/check.sh index fc3aec1..44e921e 100755 --- a/scripts/check.sh +++ b/scripts/check.sh @@ -151,14 +151,15 @@ check_queue() { # The reconciler's lifecycle, the part of it that can break silently: it runs # `collect`, so a second instance or a stop that does not stop costs closed # tasks and reaped sessions. -# reconcile-selftest.sh proves adoption, a durable stop, and the two claims +# reconcile-selftest.sh proves adoption, a durable stop, and the three claims # that are specific to it — that the four cadences are four separate clocks, -# and that the ONLY thing it ever asks the queue to do is watch, collect, -# shepherd and refuel. It stubs the queue command, so it needs no thurbox, no -# `gh` and no network. +# that the ONLY things it ever asks the queue to do are watch, collect, +# shepherd, refuel and the read-only plan, and that the one line it sends the +# lead goes out on a transition rather than on every pass. It stubs the queue +# command and thurbox-cli, so it needs no thurbox, no `gh` and no network. check_reconcile() { if ./scripts/reconcile-selftest.sh >/dev/null; then - ok "reconcile: adopts rather than duplicates, a stop stays stopped, and it writes nothing" + ok "reconcile: adopts rather than duplicates, a stop stays stopped, it writes no record, and it wakes the lead once per transition" else # Re-run visibly: a failing claim is the whole message. ./scripts/reconcile-selftest.sh diff --git a/scripts/lib/notify_lead.py b/scripts/lib/notify_lead.py new file mode 100644 index 0000000..26b8d4c --- /dev/null +++ b/scripts/lib/notify_lead.py @@ -0,0 +1,298 @@ +#!/usr/bin/env python3 +"""Tell the lead that ready work exists which nothing will dispatch. + +WHY THIS EXISTS, measured. On 2026-09-10 at 01:37 a pull request merged, +`forge-agnostic/01-01-forge-seam` landed, and the `semantic-dependency` blocker +on `forge-agnostic/02-02-gitlab-adapter` cleared. That task was dispatched at +08:05, because the operator typed "Status" and the lead looked. Six hours and +twenty-eight minutes of work that was ready, unclaimed, and had no actor. + +Nothing was broken. The reconciler may not dispatch — `AGENTS.md` says it +reconciles and does not decide, and `reconcile-selftest.sh` holds it to the +verbs it is allowed to want. The lead is the only actor that may dispatch, and +it is an interactive session that acts when someone speaks to it. +`reconcile.sh nudge` wakes the LOOP; there was no path in the other direction. + +NOTIFYING IS NOT DECIDING, and that is the whole seam. This changes no record, +moves no task and launches nothing. It reads the ready set and puts it in front +of the one actor that may act on it, along with the command that acts. The loop +keeps its constraint; the lead keeps the decision. + +FOUR THINGS MAKE IT SURVIVABLE, and each is a way this could have been worse +than silence: + + IT FIRES ON THE TRANSITION. A loop that says "one task is ready" every + pass is turned off within the day, and then the six hours come back with + the notification disabled on top. So the delivered ready set is + remembered, and a pass whose ready set holds nothing new says nothing. + IT DOES NOT INTERRUPT A TURN. `session send` types into the lead's + terminal. Typed into a session mid-turn that is an interjection in the + middle of somebody's work — `shepherd` declines to touch a working + session for exactly this reason. Only a lead that has SAID it is at rest + is woken; anything else and the wake waits, which is not the same as + being dropped. + IT IS ONE LINE. The lead is a token budget. Which tasks, and the command + that sends them. No table, no narration. + IT SURVIVES THE LEAD NOT EXISTING. No lead session is an ordinary fleet — + the operator closed it, or never installed the extension. That produces + no send, no error and, because the note is deduplicated, one log line + rather than one per pass. + +WHERE THE STATE LIVES. In the reconciler's own runtime directory, beside its +pid, heartbeat and flags — never on the task. "The lead has been told" is a +fact about one machine's loop and one conversation; it is not part of what a +task IS, and writing it onto a record would make this the second writer over +the queue. `scripts/reconcile.sh` still writes no record. + +Usage (it is `scripts/reconcile.sh`'s, and nothing else's): + + ./scripts/queue.sh plan --json | python3 scripts/lib/notify_lead.py --state-dir DIR + +It prints a line worth logging, or nothing, and it exits 0 whatever happens. +A notifier that can fail the pass it rides on would cost the reconciler the +`collect` it just did, which is a real loss traded for a message. + +Environment: + FLEET_LEAD_SESSION the lead session's name, overriding the rendered + extension.toml. The gate uses it, since a rendered + manifest belongs to the operator's checkout and is not + something a test may write. + +Requires: python3. `thurbox-cli` for the send itself — without it there is +nothing to wake and this says so once. +""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import os +import shutil +import subprocess +import sys + +# THE ONLY TWO STATES A WAKE IS ALLOWED INTO, and the list is an allowlist +# because the failure modes are asymmetric. `idle` and `done` are the agent's +# own hook saying it is at rest — at rest between turns, and typing at it is +# how the operator would speak to it anyway. +# +# Everything else is refused, and `blocked` is the one to understand: it means +# the agent is sitting on a permission or input prompt, so text sent to it does +# not start a turn, it ANSWERS the dialog. `working` is a turn in flight. +# `running`, `uncovered` and `unreported` are not the agent saying anything at +# all — see the state table in `.agents/skills/thurbox-session/SKILL.md`, whose +# whole point is that the absence of a word is not `idle`. +AT_REST = ("idle", "done") + +# How many refs the one line spells before it stops naming them. Six is about +# what stays readable in a terminal; the count is always exact, and `dispatch` +# sends the whole set regardless of what was named. +NAMED = 6 + +STATE_FILE = "notified.json" + + +def _load_queue(): + """Load scripts/lib/queue.py under a name that is not `queue`. + + For `manifest_session` and `checkout_root`, so that "what is the lead + session called" has one answer here and in queue.py's control-plane guard + rather than two parsers that agree until someone renames the session. The + alias is what keeps this directory on sys.path from shadowing the standard + library's `queue`, and it is the same load fleet_status.py does. + """ + path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "queue.py") + spec = importlib.util.spec_from_file_location("fleet_queue", path) + module = importlib.util.module_from_spec(spec) + sys.modules["fleet_queue"] = module + spec.loader.exec_module(module) + return module + + +fleetqueue = _load_queue() + + +# --- what the loop remembers ------------------------------------------------- + + +def read_state(state_dir: str) -> dict: + try: + with open(os.path.join(state_dir, STATE_FILE)) as fh: + doc = json.load(fh) + except (OSError, json.JSONDecodeError): + return {} + return doc if isinstance(doc, dict) else {} + + +def write_state(state_dir: str, told: list, note: str) -> None: + """Remember what the lead has been told, and what was last logged about it. + + Best effort on purpose: a runtime directory that cannot be written is worth + a repeated notification, and is not worth failing a reconciler pass over. + """ + try: + os.makedirs(state_dir, exist_ok=True) + with open(os.path.join(state_dir, STATE_FILE), "w") as fh: + json.dump({"told": told, "note": note}, fh) + except OSError: + pass + + +def say(state_dir: str, told: list, note: str) -> int: + """Print `note` only if it is not the one already standing. + + The deduplication is the difference between a diagnosis and a wall. A lead + that is away for an hour, or busy for one, is one fact; at the collect + cadence it would otherwise be thirty log lines saying it again. + """ + if note and read_state(state_dir).get("note") != note: + print(note) + write_state(state_dir, told, note) + return 0 + + +# --- who to wake ------------------------------------------------------------- + + +def lead_name() -> str: + """What the lead session is called, or "" when nothing here can tell. + + The RENDERED extension.toml only. `install-extension.sh` writes it into the + control-plane clone, so its presence is the claim; the tracked + `extension.toml.in` beside it carries a `__LEAD_GLYPH__` placeholder, which + names a session that does not exist. Unknown stays silent — a fleet driven + without the extension is legitimate and may not be nagged by a guess. + """ + override = os.environ.get("FLEET_LEAD_SESSION", "").strip() + if override: + return override + root = fleetqueue.checkout_root() + name, _repo = fleetqueue.manifest_session(os.path.join(root, "extension.toml")) + if not name or "__" in name: + return "" + return name + + +def lead_session(name: str) -> tuple[str | None, str]: + """(session id, state) for the named session, or (None, why not). + + `session list` and not `session get`: the list carries `id` and `state` for + every session in one call and probes no pane, which is the cheaper of the + two answers and the one that does not touch the lead to ask about it. + """ + if not shutil.which("thurbox-cli"): + return None, "no thurbox-cli here, so there is nothing to wake" + try: + out = subprocess.run( + ["thurbox-cli", "session", "list", "--json"], + capture_output=True, + text=True, + timeout=10, + ) + except (OSError, subprocess.SubprocessError) as exc: + return None, f"thurbox-cli session list failed: {exc}" + if out.returncode != 0: + return None, "thurbox-cli session list failed" + try: + rows = json.loads(out.stdout) + except json.JSONDecodeError: + return None, "thurbox-cli session list did not answer JSON" + if not isinstance(rows, list): + return None, "thurbox-cli session list did not answer a list" + for row in rows: + if isinstance(row, dict) and row.get("name") == name and row.get("id"): + return str(row["id"]), str(row.get("state") or "unknown") + return None, f"no session named {name!r} is running" + + +def wake(sid: str, text: str) -> tuple[bool, str]: + """Type one line into the lead's terminal. + + `session send`, not `message send`. The mailbox is what POLICY.md forbids + workers, because an arriving worker message interrupts whoever is talking + to the lead and routine completion is not worth that. This is the case the + same rule leaves open: it goes only to a lead that has said it is at rest, + so there is nobody to interrupt, and the thing it carries is the one fact + the lead cannot learn any other way. + """ + try: + out = subprocess.run( + ["thurbox-cli", "session", "send", sid, text], + capture_output=True, + text=True, + timeout=20, + ) + except (OSError, subprocess.SubprocessError) as exc: + return False, f"session send failed: {exc}" + if out.returncode != 0: + detail = (out.stderr or out.stdout or "").strip().splitlines() + return False, "session send failed: " + (detail[-1] if detail else "no output") + return True, "" + + +def message(ready: list) -> str: + """One line: what is ready, and the command that sends it.""" + shown = ", ".join(ready[:NAMED]) + more = "" if len(ready) <= NAMED else f" and {len(ready) - NAMED} more" + return ( + f"fleet reconciler: {len(ready)} task(s) ready and nothing will dispatch " + f"them — {shown}{more}. Run ./scripts/queue.sh dispatch" + ) + + +# --- the pass ---------------------------------------------------------------- + + +def main(argv: list) -> int: + ap = argparse.ArgumentParser( + description="wake the lead when ready work has no actor", + ) + ap.add_argument( + "--state-dir", + required=True, + help="the reconciler's runtime directory, where what-was-told lives", + ) + args = ap.parse_args(argv) + + try: + plan = json.loads(sys.stdin.read() or "{}") + except (json.JSONDecodeError, ValueError): + return 0 + if not isinstance(plan, dict): + return 0 + ready = sorted(str(r) for r in (plan.get("ready") or [])) + + # PRUNE, THEN COMPARE, and the prune is what makes a re-entry news. A task + # that leaves the ready set — dispatched, blocked again, abandoned — is + # forgotten, so if it comes back it is a transition again rather than + # something the lead was already told about weeks ago. + told = [r for r in read_state(args.state_dir).get("told", []) if r in ready] + fresh = [r for r in ready if r not in told] + if not fresh: + return say(args.state_dir, told, "") + + name = lead_name() + if not name: + return say(args.state_dir, told, "ready work, but no lead session is configured") + sid, state = lead_session(name) + if not sid: + return say(args.state_dir, told, f"ready work, but {state}") + if state not in AT_REST: + return say(args.state_dir, told, f"ready work; {name} is {state} — the wake waits") + + sent, why = wake(sid, message(ready)) + if not sent: + return say(args.state_dir, told, f"could not wake {name}: {why}") + + # Only a delivered wake is remembered, so a send that failed is retried on + # the next pass rather than swallowed. + return say( + args.state_dir, + ready, + f"woke {name}: {len(ready)} task(s) ready — {', '.join(ready[:NAMED])}", + ) + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/scripts/reconcile-selftest.sh b/scripts/reconcile-selftest.sh index 6c02b05..c36e922 100755 --- a/scripts/reconcile-selftest.sh +++ b/scripts/reconcile-selftest.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Prove the reconciler's claims, rather than assert them. # -# `scripts/reconcile.sh` makes seven promises. Most are invisible until the day +# `scripts/reconcile.sh` makes eight promises. Most are invisible until the day # they cost something — a second loop closing tasks under the first, a stop the # next onboarding run undoes, a reconciler that decided to dispatch. Each gets a # test here, against a throwaway queue, a throwaway runtime directory and a @@ -27,21 +27,31 @@ # 7. A PHANTOM IS NEVER "UP". A supervisor restarting a tick loop that cannot # run at all is a live process and not a running reconciler — never # adopted, never reported healthy. +# 8. IT WAKES THE LEAD, ON THE TRANSITION AND NOT ON THE PASS. Ready work +# that nothing will dispatch reaches the lead ONCE, when the ready set +# becomes non-empty or grows; a lead mid-turn is not interrupted and the +# wake waits for it; and a fleet with no lead session at all produces no +# send and no error. # # Tests 2 and 5 are the ones to read first. 2 is the operator's stop actually # meaning stop; 5 is the rule that keeps this a reconciler and not a second -# control plane. +# control plane. 8 is the one that reads oddly beside 5 and does not break it: +# telling the lead is not deciding, and `plan` is a read. # # HOW IT RUNS OFFLINE. `FLEET_RECONCILE_QUEUE_CMD` is the seam — the same shape # as `FLEET_QUEUE_WATCH_CMD` in queue.sh — and this replaces the real # `queue.sh` with a recorder that appends its own argv to a file and prints -# what the loop expects to read. So nothing here needs thurbox, `gh`, -# `quota-axi` or a network, and the intervals are compressed to seconds so a -# whole day of cadence fits in a few of them. +# what the loop expects to read. `thurbox-cli` is stubbed on PATH beside it, so +# test 8's lead is a state the test writes to a file rather than a session on +# this machine. Nothing here needs thurbox, `gh`, `quota-axi` or a network, and +# the intervals are compressed to seconds so a whole day of cadence fits in a +# few of them. # # Usage: scripts/reconcile-selftest.sh (also: ./scripts/check.sh reconcile) # -# Requires: bash. Nothing else — the queue command is a stub. +# Requires: bash and python3 — the latter because the notifier test drives +# `scripts/lib/notify_lead.py`, which is real code and not a stub. The queue +# command and thurbox-cli are both stubs. set -uo pipefail @@ -124,6 +134,14 @@ watch) collect) echo "collect: nothing new" ;; shepherd) echo "shepherd: 0 open pull request(s)" ;; refuel) echo "refuel: no task here is holding a session" ;; +plan) + # The FIFTH verb, and the only read among them. $READY is the ready set + # the test is driving; the real `plan --json` carries `waiting` and + # `overlaps` beside it, and the notifier reads neither. + refs="" + [ -s "$READY" ] && refs="$(sed 's/.*/"&"/' "$READY" | paste -sd, -)" + printf '{"ready":[%s],"waiting":[],"overlaps":[]}\n' "$refs" + ;; *) echo "queue-stub: REFUSING an unexpected subcommand: $*" >&2 exit 3 @@ -133,8 +151,54 @@ exit 0 STUB chmod +x "$tmp/queue-stub.sh" +# --- thurbox-cli, stubbed on PATH ------------------------------------------- +# +# The lead is a thurbox session, so waking it needs two answers from +# thurbox-cli and nothing else: which session carries the lead's name and what +# state it is in (`session list`), and then the send itself. Both are stubbed +# here so the test drives the lead's state directly. +# +# $LEAD_STATE ABSENT MEANS NO LEAD. Not an error, not an empty state — the +# session list simply does not have that row, which is the shape of a fleet +# whose lead is not running, and of one that never had the extension +# installed. +tbxbin="$tmp/bin" +mkdir -p "$tbxbin" +cat >"$tbxbin/thurbox-cli" <<'SH' +#!/usr/bin/env bash +case "${1:-} ${2:-}" in +"session list") + if [ -s "$LEAD_STATE" ]; then + printf '[{"id":"lead-uuid","name":"%s","state":"%s"}]\n' \ + "$LEAD_NAME" "$(cat "$LEAD_STATE")" + else + printf '[]\n' + fi + ;; +"session send") + printf '%s\n' "$4" >>"$SENDS" + ;; +*) exit 0 ;; +esac +exit 0 +SH +chmod +x "$tbxbin/thurbox-cli" +PATH="$tbxbin:$PATH" +export PATH + export CALLS="$calls" export MOVED="$tmp/moved" +export READY="$tmp/ready" +export SENDS="$tmp/sends" +export LEAD_STATE="$tmp/lead-state" +# The lead's name is normally read out of the rendered extension.toml, which +# belongs to the operator's own checkout and is not something a gate may write. +# FLEET_LEAD_SESSION is the override that seam exists for. +export LEAD_NAME="Gate Control" +export FLEET_LEAD_SESSION="$LEAD_NAME" +: >"$READY" +: >"$SENDS" +printf 'idle\n' >"$LEAD_STATE" export FLEET_QUEUE_DIR="$tmp/queue" mkdir -p "$FLEET_QUEUE_DIR" export FLEET_RECONCILE_DIR="$tmp/rt" @@ -233,17 +297,145 @@ else fi echo 0 >"$MOVED" +# --- 8. it wakes the lead on the transition, and only then ------------------- +# +# The failure this exists for, measured: on 2026-09-10 a blocker cleared at +# 01:37 and the task it freed was dispatched at 08:05, because the reconciler +# may not dispatch and nothing told the lead there was anything to dispatch. +# Six and a half hours of ready, unclaimed work with no actor. +# +# What is proved here is the shape that makes the fix survivable rather than +# the fix itself. A loop that says "one task is ready" every twenty seconds +# gets turned off within the day, so the assertion is that it fires on the +# TRANSITION and is silent on every pass in between. + +count_sends() { grep -c . "$SENDS" 2>/dev/null || true; } +# shellcheck disable=SC2317,SC2329 # invoked indirectly, as wait_for's predicate +sends_atleast() { [ "$(count_sends)" -ge "$1" ]; } + +if [ "$(count_sends)" = "0" ]; then + pass "an empty ready set wakes nobody" +else + fail "an empty ready set wakes nobody" "$(cat "$SENDS")" +fi + +# The transition: the ready set becomes non-empty. +printf 'alpha/01-first\n' >"$READY" +if wait_for 20 sends_atleast 1; then + pass "ready work reaches the lead" +else + fail "ready work reaches the lead" "sends: $(cat "$SENDS")${nl}$(cat "$FLEET_RECONCILE_DIR/reconcile.log")" +fi + +woke="$(cat "$SENDS")" +expect "the message names the task that is ready" "alpha/01-first" "$woke" +expect "the message carries the command that sends it" "queue.sh dispatch" "$woke" +if [ "$(wc -l <"$SENDS")" = "1" ]; then + pass "the wake is one line, not a report" +else + fail "the wake is one line" "$woke" +fi + +# AND THEN IT STOPS. Several collect intervals pass with the same task still +# ready and still undispatched; the lead hears nothing more about it. +sleep 5 +if [ "$(count_sends)" = "1" ]; then + pass "it stays quiet while the same set stays ready" +else + fail "it stays quiet while the same set stays ready" "$(cat "$SENDS")" +fi + +# A GROWING SET IS A NEW TRANSITION. A second task becoming ready is news, and +# the message names the whole ready set rather than only the new arrival — +# `dispatch` is going to send all of it. +printf 'alpha/01-first\nbeta/02-second\n' >"$READY" +if wait_for 20 sends_atleast 2; then + pass "a growing ready set is a fresh transition" +else + fail "a growing ready set is a fresh transition" "$(cat "$SENDS")" +fi +expect "the second message names both ready tasks" "beta/02-second" "$(tail -1 "$SENDS")" +expect "and names the one that was already ready" "alpha/01-first" "$(tail -1 "$SENDS")" + +# A WORKING LEAD IS NOT INTERRUPTED, and the wake is not lost either. Typing +# into a session mid-turn is how a fix gets half-applied — `shepherd` declines +# to touch a working session for the same reason — so the notice waits for the +# lead to come to rest instead of being dropped or forced through. +printf 'working\n' >"$LEAD_STATE" +sent_before="$(count_sends)" +printf 'alpha/01-first\nbeta/02-second\ngamma/03-third\n' >"$READY" +sleep 5 +if [ "$(count_sends)" = "$sent_before" ]; then + pass "a lead mid-turn is not interrupted" +else + fail "a lead mid-turn is not interrupted" "$(tail -1 "$SENDS")" +fi +expect "and the log says the wake is waiting rather than gone" "the wake waits" \ + "$(cat "$FLEET_RECONCILE_DIR/reconcile.log")" + +printf 'idle\n' >"$LEAD_STATE" +if wait_for 20 sends_atleast $((sent_before + 1)); then + pass "the held wake lands once the lead is at rest" +else + fail "the held wake lands once the lead is at rest" "$(cat "$SENDS")" +fi +expect "and it names the task that arrived while the lead was busy" "gamma/03-third" "$(tail -1 "$SENDS")" + +# NO LEAD, NO ERROR STORM. A fleet whose lead session is not running is a +# normal fleet — the operator closed it, or the extension was never installed. +# The loop must carry on folding and say nothing it cannot act on twice. +: >"$LEAD_STATE" +sent_before="$(count_sends)" +watch_before="$(count_calls watch)" +printf 'alpha/01-first\nbeta/02-second\ngamma/03-third\ndelta/04-fourth\n' >"$READY" +sleep 5 +if [ "$(count_sends)" = "$sent_before" ]; then + pass "no lead session means no send" +else + fail "no lead session means no send" "$(tail -1 "$SENDS")" +fi +if [ "$(count_calls watch)" -gt "$watch_before" ]; then + pass "and the loop keeps folding regardless" +else + fail "and the loop keeps folding regardless" "watch stuck at $watch_before" +fi +# One line about it, not one per pass: the note is deduplicated against the +# last one, so a lead that is away for an hour costs the log a single entry. +absent="$(grep -c "no session named" "$FLEET_RECONCILE_DIR/reconcile.log" 2>/dev/null || true)" +if [ "${absent:-0}" -le 1 ]; then + pass "an absent lead is reported once, not once per pass" +else + fail "an absent lead is reported once" "$absent log lines say so" +fi + +printf 'idle\n' >"$LEAD_STATE" +: >"$READY" + # --- 5. it is not a writer --------------------------------------------------- # # The stub refuses any subcommand it was not taught, so a `dispatch` would have # failed the loop already. This states the claim positively as well: over the # whole run, the set of things the reconciler asked the queue to do is exactly -# the four reconciling ones plus the `root` precondition check. +# the four reconciling ones, plus `plan`, plus the `root` precondition check. +# +# WHY `plan` IS IN THIS SET AND `dispatch` NEVER CAN BE. This list is not a +# list of harmless commands; it is the list of things the loop is allowed to +# want. `plan` READS — it prints the ready set, the waiting set and why each +# one waits, and it writes nothing and moves nothing. The loop asks it so that +# test 8 above has something to tell the lead about, and telling the lead is +# not deciding: the ready set goes to the actor that may act on it, and that +# actor is still the lead. +# +# `dispatch`, `add`, `block`, `archive` and `reap` all CHANGE what runs, and a +# loop that could call one of them would be a second control plane with no +# operator in it. Widening this set again is a decision about power, not about +# convenience — a read that answers a question the loop already needs may join +# it, and nothing that acts ever may. verbs="$(awk '{print $1}' "$calls" | sort -u | tr '\n' ' ')" -if [ "$verbs" = "collect refuel root shepherd watch " ]; then - pass "it calls only watch/collect/shepherd/refuel — never dispatch, add or reap" +if [ "$verbs" = "collect plan refuel root shepherd watch " ]; then + pass "it calls watch/collect/shepherd/refuel and the read-only plan — never dispatch, add or reap" else - fail "it calls only the reconciling subcommands" "called: $verbs" + fail "it calls only the reconciling subcommands, plus the read-only plan" "called: $verbs" fi # It writes to its own runtime directory and to nothing else. The queue diff --git a/scripts/reconcile.sh b/scripts/reconcile.sh index d3dec8b..ae00ee8 100755 --- a/scripts/reconcile.sh +++ b/scripts/reconcile.sh @@ -48,16 +48,31 @@ # `start` the operator asking for it back. It CLEARS the flag. That is the # whole difference between the two. # -# IT WRITES NOTHING ITSELF. Every effect it has goes through +# IT WRITES NO RECORD. Every effect it has on the queue goes through # `./scripts/queue.sh`, which stays the only writer over the records — the # rule that keeps the queue single-writer. Grep this file for a write to a -# task and you will not find one. +# task and you will not find one. Its own runtime directory is the exception +# and is not one: a pid, a heartbeat, a log, the flags and `notified.json` are +# facts about this loop on this machine, not about any task. # # IT DOES NOT DECIDE WHAT RUNS. No dispatch, no cancel, no reorder. It # reconciles recorded state with observed state; choosing the work stays the # lead's. It does not second-guess `refuel` either — the rule that nothing is # restarted into a spent quota window lives there, and this calls the command. # +# BUT IT DOES SAY WHEN THERE IS SOMETHING TO DECIDE, which is the one thing it +# tells anybody. A task whose blocker clears becomes READY and has no actor: +# this loop may not dispatch it, and the lead, which may, only acts when spoken +# to. On 2026-09-10 that gap ran from 01:37 to 08:05 and the operator closed it +# by typing "Status". So the pass that follows `collect` reads the ready set and +# wakes the lead when it has grown — once per transition, never into a turn in +# flight, and silently when there is no lead session at all. +# +# Notifying is not deciding. Nothing is dispatched, no record moves, and the +# choice is still the lead's; all that changed is that it knows there is one to +# make. `scripts/lib/notify_lead.py` owns the rules that keep it from becoming +# noise, and reconcile-selftest.sh's test 8 holds it to them. +# # Usage: # scripts/reconcile.sh ensure # start unless running or asked down # scripts/reconcile.sh start # start, and clear a previous `stop` @@ -97,6 +112,11 @@ # pull request's own CI does not change state faster than that, so # a tighter interval would ask the forge the same question several # times for one answer. +# notify collect's clock, and no clock of its own. What makes a task ready +# is a blocker clearing, and what clears a blocker is a landing +# `collect` has just recorded — so a fifth interval would only ask +# the same question at a second, worse moment. It reads local +# records and one local socket, so it costs nothing to ride along. # # Every one is overridable for a test or an unusual fleet; see Environment. # @@ -111,12 +131,16 @@ # FLEET_RECONCILE_SHEPHERD_SECS seconds between shepherds (default 900) # FLEET_QUEUE_DIR the queue to reconcile (default: THIS # CHECKOUT's — see scripts/queue.sh root) +# FLEET_LEAD_SESSION the lead session to wake (default: the name in +# the rendered extension.toml). Read by +# scripts/lib/notify_lead.py, which is where it +# is argued. # # Requires: python3 and whatever the pass it is running needs — thurbox-cli for -# `watch` and `refuel`, `gh` for `collect` and `shepherd`, `quota-axi` for the -# fuel reading. Every one of those degrades to "could not check" inside -# queue.sh rather than to a guess, so a missing tool costs its own pass and -# never the loop. +# `watch`, `refuel` and the notification, `gh` for `collect` and `shepherd`, +# `quota-axi` for the fuel reading. Every one of those degrades to "could not +# check" inside queue.sh rather than to a guess, so a missing tool costs its own +# pass and never the loop. set -uo pipefail @@ -234,6 +258,32 @@ run_pass() { return 0 } +# THE ONE THING THIS LOOP SAYS OUT LOUD, and the one place it asks the queue a +# question rather than telling it to reconcile something. +# +# A task whose blocker clears becomes ready and has NO ACTOR. This loop may not +# dispatch — that is the constraint the whole design rests on — and the lead, +# which may, is an interactive session that acts when someone speaks to it. On +# 2026-09-10 that gap was six and a half hours long and the operator closed it +# by hand. `scripts/lib/notify_lead.py` owns the rest: the transition test that +# keeps this from being a stream of the same sentence, the refusal to type into +# a lead mid-turn, and the silence when there is no lead at all. +# +# `plan` IS THE FIFTH VERB and it is a READ. It prints the ready set and writes +# nothing; the assertion in reconcile-selftest.sh names it and argues why it may +# be there while `dispatch` never can be. Telling the lead is not deciding. +# +# It cannot fail the pass. Every branch inside the notifier exits 0, and this +# returns 0 regardless — a message is not worth the `collect` the loop just did. +notify_lead() { + local plan out + plan="$($QUEUE_CMD plan --json 2>/dev/null)" || return 0 + [ -n "$plan" ] || return 0 + out="$(printf '%s\n' "$plan" | python3 scripts/lib/notify_lead.py --state-dir "$RT" 2>&1)" + [ -n "$out" ] && log "notify: $out" + return 0 +} + # The loop. Read the cadences in the header before changing a number here. tick() { # Validated BEFORE the first heartbeat, so a reconciler that cannot run the @@ -245,7 +295,7 @@ tick() { return 2 fi - local last_collect=0 last_refuel=0 last_shepherd=0 nudged stamp + local last_collect=0 last_refuel=0 last_shepherd=0 nudged did_collect stamp while :; do # The down flag is checked at the TOP of every pass as well as by the # supervisor, so a `stop` that lands mid-pass is honoured at the next @@ -269,9 +319,11 @@ tick() { # collect first, and on its own: it is the one that CLOSES tasks, and # shepherd's view of which pull requests still matter is better for # running after it. + did_collect=0 if [ "$nudged" -eq 1 ] || [ $((stamp - last_collect)) -ge "$COLLECT_SECS" ]; then run_pass collect always collect last_collect="$stamp" + did_collect=1 fi if [ $((stamp - last_shepherd)) -ge "$SHEPHERD_SECS" ]; then run_pass shepherd always shepherd @@ -282,6 +334,14 @@ tick() { last_refuel="$stamp" fi + # LAST, and on collect's clock, because it is the only pass that + # looks OUTWARD. The three above reconcile records with the world; + # this one reads the result and, when the ready set has just grown, + # tells the one actor that may act on it. Running it after them means + # it sees the landings collect just recorded rather than the ones it + # recorded a pass ago — which is the whole latency this exists to cut. + [ "$did_collect" -eq 1 ] && notify_lead + trim_log # And then the continuous half, which is also what paces the loop: this