From e9f8270781e7a362b138a9fe9e2078aa5cc8213e Mon Sep 17 00:00:00 2001 From: Zook Bot <275398848+zook-bot@users.noreply.github.com> Date: Wed, 22 Jul 2026 08:09:58 +0000 Subject: [PATCH] =?UTF-8?q?feat(witness):=20quiesce=20completed=20mol-pole?= =?UTF-8?q?cat-work=20husks=20=E2=80=94=20clear=20gc.routed=5Fto=20AND=20a?= =?UTF-8?q?ssignee=20so=20the=20pool=20stops=20re-offering=20dead=20steps?= =?UTF-8?q?=20(tk-p9ji9)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mol-polecat-work runs its 7 step beads INLINE in one session and no step closes its own bead. The graph is chained, so while load-context stays open it is the ONLY ready step — and it still carries gc.routed_to=/ polecat. Open + ready + routed is exactly the pool's offer predicate, so every idle polecat is handed the same dead step, forever. Each burns a wisp to re-derive "already done" and drain. The molecule cannot finalize itself either: under close-on-land its anchor stays OPEN until the refinery lands the PR, so the churn runs for the whole human-approval wait (~1 wisp/4-5min, ~200 overnight on a single anchor). The witness had been containing this BY HAND, molecule by molecule — ten of them by 2026-07-22. Automates that containment as a witness-patrol pass. Two re-offer shapes need two different levers, and clearing gc.routed_to fixes only the first. Verified live: the pool work query returns open, UNASSIGNED, routed, ready beads only, so an assigned step never rides the pool path at all. unassigned assignee empty + gc.routed_to set -> the pool offers it; clearing gc.routed_to removes it. assigned assignee= + gc.session_affinity=require -> already invisible to the pool; handed back on the ASSIGNED path, keyed on the assignee. Clearing gc.routed_to here is a NO-OP — which is why the by-hand sweeps kept leaving a residual of sessions still holding their in_progress step. Both keys go in ONE gc bd update. A two-call sequence is unsafe: clearing the assignee first would briefly leave the bead open+unassigned+routed — the exact pool-offer shape — racing a fresh polecat into the husk being retired. DANGER clause preserved: the pass NEVER closes a step bead. Closing load-context unblocks workspace-setup and walks the next polecat onto a branch that is already green-gated and PR'd; any push there stales the anchor's check.=green@ marker and BLOCKS the open PR. There is no close path in the script and the test asserts its absence. It also never writes the anchor, never rewrites status, and never touches workflow-finalize (routed to the control dispatcher — the molecule's only escape path). An anchor it cannot resolve is SKIPPED: quiescing a live molecule would strip the assignee off steps a running polecat still has to claim, draining it mid-implementation, whereas an un-quiesced husk only wastes wisps. Quiescing is containment, not finalization — the molecule is left stranded but quiet, matching what the manual sweep achieves. Finalizing the step graph at submit-and-exit time is the durable upstream fix (gascity core / gastown formula) and is deliberately out of scope. Validated against live rig state, not just a compile check: - dry-run selected exactly the completed molecules, skipping 3 live ones (including this bead's own in-flight molecule) - real pass quiesced 12 steps across 2 completed workflows, one of which (tk-c9ivi) crossed into pre_open_gate mid-session and was caught the same pass - after: the pool offer no longer contains any quiesced step, and both churning sessions hold 0 assigned beads, while genuine work still flows - anchors untouched (PR #203 still pull_request, codex green at head); step status unchanged, nothing closed; finalize routes intact Tests: assets/scripts/quiesce-completed-workflows.test.sh — 20 assertions, hermetic. Full suite 11/11 green (264 assertions). shellcheck clean at severity>=warning. --- assets/scripts/quiesce-completed-workflows.sh | 192 +++++++++++++++ .../quiesce-completed-workflows.test.sh | 233 ++++++++++++++++++ doctor/check-base-artifact-collision/run.sh | 3 +- formulas/mol-witness-patrol.toml | 96 +++++++- 4 files changed, 519 insertions(+), 5 deletions(-) create mode 100755 assets/scripts/quiesce-completed-workflows.sh create mode 100755 assets/scripts/quiesce-completed-workflows.test.sh diff --git a/assets/scripts/quiesce-completed-workflows.sh b/assets/scripts/quiesce-completed-workflows.sh new file mode 100755 index 0000000..6a19ab0 --- /dev/null +++ b/assets/scripts/quiesce-completed-workflows.sh @@ -0,0 +1,192 @@ +#!/usr/bin/env bash +# quiesce-completed-workflows — stop the pool (and the affine hand-back) from +# re-offering the dead step beads of a mol-polecat-work molecule whose inline +# execution has already finished (tk-p9ji9). +# +# Background. mol-polecat-work (graph.v2) materializes 7 step beads, but the +# polecat executes them INLINE in one session and no step closes its own bead. +# The step graph is chained (load-context blocks workspace-setup blocks ...), so +# while load-context stays open it is the ONLY ready step — and it still carries +# `gc.routed_to=/polecat`. Open + ready + routed is exactly the +# pool's offer predicate, so every idle polecat is handed the same dead step, +# forever: ~1 wisp per 4-5 min for the entire human-approval wait on the PR. The +# molecule cannot finalize itself either, because under close-on-land its anchor +# stays OPEN until the refinery lands the PR. +# +# The witness has been containing this BY HAND, molecule by molecule (ten of them +# as of 2026-07-22). This pass is that containment, automated. +# +# TWO re-offer shapes, two different levers — clearing `gc.routed_to` alone fixes +# only the first (verified live: `gc hook ` returns open, UNASSIGNED, +# routed, ready beads only, so an assigned step never rides the pool path): +# +# unassigned shape assignee empty + gc.routed_to set +# -> the POOL offers it. Clearing gc.routed_to removes it. +# assigned shape assignee= + gc.session_affinity=require +# -> already invisible to the pool; it is handed back on the +# ASSIGNED-work path, keyed on the assignee. Clearing +# gc.routed_to here is a NO-OP; the assignee must go too. +# +# Both keys are cleared in ONE `gc bd update`. Order matters and a two-call +# sequence is unsafe: clearing the assignee first would briefly leave the bead +# open + unassigned + routed — the exact pool-offer shape — racing a fresh +# polecat into the husk we are trying to retire. +# +# WHAT THIS PASS NEVER DOES — closing a step bead is the footgun this bug exists +# to prevent. Closing load-context unblocks workspace-setup and walks the next +# polecat forward onto a branch that is ALREADY green-gated and PR'd; any push +# there moves the head, stales the anchor's `check.=green@` marker and +# BLOCKS the open PR from merging. There is deliberately no close path in this +# script. It also never touches the anchor, never touches `status`, and never +# touches the `workflow-finalize` step (routed to the control dispatcher — that +# is the path that finalizes the graph, and it must keep its route). +# +# Quiescing is containment, not finalization: the molecule is left stranded-but- +# quiet, which is what the witness's manual sweep achieves today. Finalizing the +# step graph at submit-and-exit time is the durable upstream fix (gascity core / +# gastown formula) and is deliberately out of scope here. +# +# NOT set -e: best-effort, must never abort the witness patrol. Any tool error +# skips that root and retries next patrol cycle. +set -uo pipefail + +DRY_RUN=0 +while [ $# -gt 0 ]; do + case "$1" in + --dry-run) DRY_RUN=1; shift ;; + *) shift ;; + esac +done + +# Anchor states that mean "the workflow's inline execution is DONE". +# +# pre_open_gate polecat handed off; codex is reviewing the BRANCH, no PR yet +# pull_request PR open, parked in the merge gate awaiting human approval +# merged PR landed; the anchor closes on the refinery's reconcile pass +# +# `merged` and a CLOSED anchor (handled separately below) are strictly-later +# lifecycle states than `pull_request`: if the steps are dead at pull_request they +# are dead afterwards too. A closed anchor is the safest case of all — the work +# bead itself is finished. +is_terminal_anchor() { + case "$1" in # $1 = anchor status + closed) return 0 ;; + esac + case "$2" in # $2 = anchor metadata.merge_result + pre_open_gate|pull_request|merged) return 0 ;; + esac + return 1 +} + +STEPS=$(gc bd list --status=open,in_progress --json --limit=0 2>/dev/null) +[ -n "$STEPS" ] && [ "$STEPS" != "[]" ] \ + || { echo "quiesce-completed-workflows: no open work beads"; exit 0; } + +# One compact row per live mol-polecat-work step bead. Built into a variable (not +# piped into the loop) so the loop runs in THIS shell and the counters survive. +ROWS=$(printf '%s' "$STEPS" | jq -c ' + .[] + | select((.metadata["gc.step_ref"] // "") | startswith("mol-polecat-work.")) + | { + id, + step: (.metadata["gc.step_ref"] // ""), + root: (.metadata["gc.root_bead_id"] // ""), + routed: (.metadata["gc.routed_to"] // ""), + assignee: (.assignee // "") + }' 2>/dev/null) +[ -n "$ROWS" ] \ + || { echo "quiesce-completed-workflows: no live mol-polecat-work steps"; exit 0; } + +ROOTS=$(printf '%s\n' "$ROWS" | jq -r -s 'map(.root) | map(select(. != "")) | unique | .[]' 2>/dev/null) +[ -n "$ROOTS" ] \ + || { echo "quiesce-completed-workflows: no resolvable workflow roots"; exit 0; } + +quiesced=0; roots_done=0; roots_live=0; already=0; unresolved=0 + +# Batched per ROOT, deliberately: a rig with several husks is ~6 `gc bd update` +# calls per root, and sweeping every bead in one flat pass has blown a 2-minute +# tool timeout in practice. Per-root batching also makes a partial pass coherent — +# a molecule is either quiesced or untouched, never half-swept. +while IFS= read -r root; do + [ -n "${root:-}" ] || continue + + # Resolve the anchor the way the formula itself does: root -> input convoy -> + # its single tracked member. mol-polecat-base requires exactly one member, so + # anything else is a shape we do not understand. + convoy=$(gc bd show "$root" --json 2>/dev/null \ + | jq -r '.[0].metadata["gc.input_convoy_id"] // empty' 2>/dev/null) + anchor="" + [ -n "$convoy" ] && anchor=$(gc convoy status "$convoy" --json 2>/dev/null \ + | jq -r 'if ((.children // []) | length) == 1 then (.children[0].id // empty) else empty end' 2>/dev/null) + + # FAIL CLOSED on an unresolved anchor. Quiescing a LIVE molecule would strip the + # assignee off the steps a running polecat still has to claim, draining it + # mid-implementation and stranding real work. An un-quiesced husk only wastes + # wisps — the cheaper failure by far, and the witness still catches it by hand. + if [ -z "$anchor" ]; then + echo "quiesce-completed-workflows: root $root — anchor unresolved (convoy '${convoy:-none}'); skipped" >&2 + unresolved=$((unresolved + 1)); continue + fi + + ainfo=$(gc bd show "$anchor" --json 2>/dev/null \ + | jq -r '.[0] | "\(.status // "")|\(.metadata.merge_result // "")"' 2>/dev/null) + if [ -z "$ainfo" ] || [ "$ainfo" = "|" ]; then + echo "quiesce-completed-workflows: root $root — anchor $anchor unreadable; skipped" >&2 + unresolved=$((unresolved + 1)); continue + fi + astatus=${ainfo%%|*}; amerge=${ainfo##*|} + + if ! is_terminal_anchor "$astatus" "$amerge"; then + echo "quiesce-completed-workflows: root $root — anchor $anchor still live (status=$astatus merge_result=${amerge:-none}); left alone" + roots_live=$((roots_live + 1)); continue + fi + + echo "quiesce-completed-workflows: root $root — anchor $anchor DONE (status=$astatus merge_result=${amerge:-none}); quiescing steps" + roots_done=$((roots_done + 1)) + + while IFS= read -r row; do + [ -n "${row:-}" ] || continue + sid=$(printf '%s' "$row" | jq -r '.id // empty') + step=$(printf '%s' "$row" | jq -r '.step // empty') + routed=$(printf '%s' "$row" | jq -r '.routed // empty') + who=$(printf '%s' "$row" | jq -r '.assignee // empty') + [ -n "$sid" ] || continue + + # NEVER touch the finalize step: it is routed to the control dispatcher, which + # is the machinery that actually closes the graph out. De-routing it would + # remove the molecule's only escape path. Guarded twice — by step id and by + # route — because losing this one is unrecoverable without a hand repair. + case "$step" in *.workflow-finalize) continue ;; esac + case "$routed" in *control-dispatcher*) continue ;; esac + + # Idempotent: nothing left to clear means a previous pass (or the witness by + # hand) already quiesced this step. + if [ -z "$routed" ] && [ -z "$who" ]; then + already=$((already + 1)); continue + fi + + # Snapshot the prior values into the patrol log — this action is meant to be + # reversible by hand, so the log has to say what was there. + echo " $sid ($step): routed='${routed:-none}' assignee='${who:-none}' -> cleared" + if [ "$DRY_RUN" -eq 1 ]; then + quiesced=$((quiesced + 1)); continue + fi + + # Both keys in ONE update (see the header note on the two-call race). Only the + # keys actually present are touched, so sibling metadata stays intact and the + # bead's status is never rewritten. + UPDATE_ARGS=("$sid") + [ -n "$routed" ] && UPDATE_ARGS+=(--unset-metadata gc.routed_to) + [ -n "$who" ] && UPDATE_ARGS+=(--assignee "") + if gc bd update "${UPDATE_ARGS[@]}" >/dev/null 2>&1; then + quiesced=$((quiesced + 1)) + else + echo "quiesce-completed-workflows: $sid update failed; retries next patrol" >&2 + fi + done <<< "$(printf '%s\n' "$ROWS" | jq -c --arg r "$root" 'select(.root == $r)' 2>/dev/null)" +done <<< "$ROOTS" + +MODE="" +[ "$DRY_RUN" -eq 1 ] && MODE="(dry-run) " +echo "quiesce-completed-workflows: ${MODE}${quiesced} steps quiesced across $roots_done completed workflow(s); $roots_live still live, $already already quiet, $unresolved unresolved" +exit 0 diff --git a/assets/scripts/quiesce-completed-workflows.test.sh b/assets/scripts/quiesce-completed-workflows.test.sh new file mode 100755 index 0000000..0ff81d0 --- /dev/null +++ b/assets/scripts/quiesce-completed-workflows.test.sh @@ -0,0 +1,233 @@ +#!/usr/bin/env bash +# Hermetic test for quiesce-completed-workflows.sh (tk-p9ji9). Stubs `gc` (bd +# list/show/update, convoy status) on PATH. No live city, Dolt, or network. +# +# The pass retires the dead step beads of a mol-polecat-work molecule whose inline +# execution has finished, so the pool stops re-offering them. Covered: +# (POOL) unassigned + routed step under a DONE anchor -> gc.routed_to cleared +# (AFFINE) assigned + routed step under a DONE anchor -> assignee cleared TOO +# (clearing only routed_to is a no-op for this shape — it rides the +# assigned-work path, which is keyed on the assignee) +# (ATOMIC) both keys are cleared in ONE `gc bd update`, never two — a two-call +# sequence briefly leaves the bead open+unassigned+routed, the exact +# pool-offer shape, racing a fresh polecat into the husk +# (LIVE) anchor NOT terminal -> molecule untouched (a running polecat still +# needs its assignee to claim the next step) +# (NOCLOSE) no step bead is ever closed, and `status` is never rewritten — the +# DANGER clause: closing load-context walks a polecat onto an already +# green-gated branch and stales check., blocking the open PR +# (FINAL) the workflow-finalize step keeps its control-dispatcher route — it is +# the molecule's only escape path +# (CLOSED) a CLOSED anchor also counts as done (strictly later than pull_request) +# (FAILSAFE) unresolvable anchor -> skipped, never quiesced +# (IDEM) a second pass is a no-op; already-quiet steps are not re-updated +# (DRY) --dry-run reports the same selection but issues no update at all +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SCRIPT="$HERE/quiesce-completed-workflows.sh" +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +PASS=0; FAIL=0 +ok() { PASS=$((PASS + 1)); echo "ok - $1"; } +bad() { FAIL=$((FAIL + 1)); echo "FAIL - $1"; } +eq() { [ "$1" = "$2" ] && ok "$3" || bad "$3 (got '$1' want '$2')"; } + +mkdir -p "$TMP/bin" + +# --- Fixture ------------------------------------------------------------------ +# Step beads: id|step_ref|root|routed_to|assignee|status +# +# root-DONE : anchor parked in the merge gate (merge_result=pull_request). +# Carries BOTH re-offer shapes plus its finalize step. +# root-LIVE : anchor still open, no merge_result -> a live molecule, hands off. +# root-CLOSED: anchor CLOSED (landed) -> also done. +# root-ORPHAN: root has no input convoy -> anchor unresolvable -> fail closed. +# root-QUIET : already quiesced by an earlier pass -> counted, not re-updated. +cat > "$TMP/steps" <<'S' +s-pool|mol-polecat-work.workspace-setup|root-DONE|gc-toolkit/gc-toolkit.polecat||open +s-affine|mol-polecat-work.load-context|root-DONE|gc-toolkit/gc-toolkit.polecat|gc-toolkit__polecat-lx-dead|in_progress +s-final|mol-polecat-work.workflow-finalize|root-DONE|gc-toolkit/core.control-dispatcher||open +s-live|mol-polecat-work.load-context|root-LIVE|gc-toolkit/gc-toolkit.polecat|gc-toolkit__polecat-lx-busy|in_progress +s-closed|mol-polecat-work.implement|root-CLOSED|gc-toolkit/gc-toolkit.polecat|gc-toolkit__polecat-lx-gone|open +s-orphan|mol-polecat-work.implement|root-ORPHAN|gc-toolkit/gc-toolkit.polecat|gc-toolkit__polecat-lx-x|open +s-quiet|mol-polecat-work.implement|root-QUIET|||open +S + +# Roots: root_id|convoy_id (root-ORPHAN deliberately absent -> no convoy) +cat > "$TMP/roots" <<'R' +root-DONE|convoy-DONE +root-LIVE|convoy-LIVE +root-CLOSED|convoy-CLOSED +root-QUIET|convoy-QUIET +R + +# Convoys: convoy_id|anchor_id +cat > "$TMP/convoys" <<'C' +convoy-DONE|anchor-DONE +convoy-LIVE|anchor-LIVE +convoy-CLOSED|anchor-CLOSED +convoy-QUIET|anchor-QUIET +C + +# Anchors: anchor_id|status|merge_result +cat > "$TMP/anchors" <<'A' +anchor-DONE|open|pull_request +anchor-LIVE|open| +anchor-CLOSED|closed| +anchor-QUIET|open|pre_open_gate +A + +: > "$TMP/updates" # one line per `gc bd update` invocation (the full argv) +: > "$TMP/cleared" # id -> which keys this pass cleared + +# --- gc stub ------------------------------------------------------------------ +cat > "$TMP/bin/gc" <<'GC' +#!/usr/bin/env bash +case "$1 ${2:-}" in + "convoy status") + anchor=$(awk -F'|' -v c="$3" '$1==c{print $2; exit}' "$FAKE_CONVOYS") + if [ -n "$anchor" ]; then jq -n --arg a "$anchor" '{children:[{id:$a}]}' + else printf '{"children":[]}\n'; fi ;; + "bd list") + out="" + while IFS='|' read -r id step root routed assignee status; do + [ -n "$id" ] || continue + # Re-read live (post-update) routed/assignee so pass 2 sees pass 1's writes. + cur=$(awk -F'\t' -v i="$id" '$1==i{print $2"|"$3}' "$FAKE_STATE" 2>/dev/null) + [ -n "$cur" ] && { routed="${cur%%|*}"; assignee="${cur##*|}"; } + obj=$(jq -n --arg id "$id" --arg st "$step" --arg rt "$root" \ + --arg ro "$routed" --arg as "$assignee" --arg s "$status" \ + '{id:$id, status:$s, assignee:$as, + metadata:{"gc.step_ref":$st, "gc.root_bead_id":$rt, "gc.routed_to":$ro}}') + if [ -z "$out" ]; then out="$obj"; else out="$out,$obj"; fi + done < "$FAKE_STEPS" + printf '[%s]\n' "$out" ;; + "bd show") + id="$3" + convoy=$(awk -F'|' -v r="$id" '$1==r{print $2; exit}' "$FAKE_ROOTS") + arow=$(awk -F'|' -v a="$id" '$1==a{print; exit}' "$FAKE_ANCHORS") + if [ -n "$arow" ]; then + st=$(printf '%s' "$arow" | cut -d'|' -f2); mr=$(printf '%s' "$arow" | cut -d'|' -f3) + jq -n --arg s "$st" --arg m "$mr" '[{status:$s, metadata:{merge_result:$m}}]' + elif [ -n "$convoy" ]; then + jq -n --arg c "$convoy" '[{metadata:{"gc.input_convoy_id":$c}}]' + else printf '[{"metadata":{}}]\n'; fi ;; + "bd update") + printf '%s\n' "$*" >> "$FAKE_UPDATES" + id="$3"; keys="" + case "$*" in *"--unset-metadata gc.routed_to"*) keys="routed" ;; esac + case "$*" in *"--assignee"*) keys="${keys:+$keys+}assignee" ;; esac + printf '%s\t%s\n' "$id" "$keys" >> "$FAKE_CLEARED" + # Apply to live state so a second pass observes the result. + routed=""; assignee="" + case "$keys" in + *routed*) : ;; + esac + old=$(awk -F'|' -v i="$id" '$1==i{print $4"|"$5}' "$FAKE_STEPS") + routed="${old%%|*}"; assignee="${old##*|}" + case "$*" in *"--unset-metadata gc.routed_to"*) routed="" ;; esac + case "$*" in *"--assignee"*) assignee="" ;; esac + printf '%s\t%s\t%s\n' "$id" "$routed" "$assignee" >> "$FAKE_STATE" ;; +esac +exit 0 +GC +chmod +x "$TMP/bin/gc" + +export PATH="$TMP/bin:$PATH" +export FAKE_STEPS="$TMP/steps" FAKE_ROOTS="$TMP/roots" FAKE_CONVOYS="$TMP/convoys" \ + FAKE_ANCHORS="$TMP/anchors" FAKE_UPDATES="$TMP/updates" \ + FAKE_CLEARED="$TMP/cleared" FAKE_STATE="$TMP/state" +: > "$TMP/state" + +# --- Run 0: --dry-run must select the same work but write nothing. ------------ +OUT0="$(bash "$SCRIPT" --dry-run)" +eq "$(wc -l < "$TMP/updates" | tr -d ' ')" "0" "(DRY) --dry-run issues no gc bd update at all" +printf '%s\n' "$OUT0" | grep -q '(dry-run)' \ + && ok "(DRY) summary marks the pass as a dry run" || bad "(DRY) summary marks dry run (got: $OUT0)" +printf '%s\n' "$OUT0" | grep -q 's-affine' \ + && ok "(DRY) dry run still reports the steps it would quiesce" || bad "(DRY) dry run reports selection" + +# --- Run 1: the real pass. ---------------------------------------------------- +: > "$TMP/updates"; : > "$TMP/cleared"; : > "$TMP/state" +OUT1="$(bash "$SCRIPT" 2>"$TMP/err1")" +ERR1="$(cat "$TMP/err1")" + +# (POOL) unassigned+routed under a done anchor -> routed_to cleared. +grep -q '^s-pool routed$' "$TMP/cleared" \ + && ok "(POOL) unassigned+routed step -> gc.routed_to cleared (leaves the pool query)" \ + || bad "(POOL) routed_to cleared (got: $(grep '^s-pool' "$TMP/cleared" || echo none))" + +# (AFFINE) assigned shape -> the assignee must go too, else the hand-back survives. +grep -q '^s-affine routed+assignee$' "$TMP/cleared" \ + && ok "(AFFINE) assigned+affine step -> assignee cleared too (kills the existing_assignment hand-back)" \ + || bad "(AFFINE) assignee must also be cleared (got: $(grep '^s-affine' "$TMP/cleared" || echo none))" + +# (ATOMIC) one update per bead — never a clear-assignee-then-clear-route sequence. +eq "$(grep -c '^s-affine' "$TMP/cleared")" "1" \ + "(ATOMIC) both keys cleared in a SINGLE update (no open+unassigned+routed race window)" +grep 'bd update s-affine' "$TMP/updates" | grep -q -- '--unset-metadata gc.routed_to' \ + && grep 'bd update s-affine' "$TMP/updates" | grep -q -- '--assignee' \ + && ok "(ATOMIC) the single update carries BOTH flags" || bad "(ATOMIC) one update carries both flags" + +# (LIVE) a molecule whose anchor is still live is left completely alone. +grep -q '^s-live' "$TMP/cleared" \ + && bad "(LIVE) must NOT touch a live molecule's steps" \ + || ok "(LIVE) live anchor -> steps untouched (running polecat keeps its assignee)" +printf '%s\n' "$OUT1" | grep -q 'anchor anchor-LIVE still live' \ + && ok "(LIVE) summary explains why the live root was skipped" || bad "(LIVE) live-skip reason" + +# (CLOSED) a closed anchor counts as done. +grep -q '^s-closed routed+assignee$' "$TMP/cleared" \ + && ok "(CLOSED) closed anchor -> steps quiesced (landed is strictly past pull_request)" \ + || bad "(CLOSED) closed anchor treated as done" + +# (FINAL) the finalize step keeps its control-dispatcher route. +grep -q '^s-final' "$TMP/cleared" \ + && bad "(FINAL) must NOT de-route workflow-finalize — it is the escape path" \ + || ok "(FINAL) workflow-finalize keeps its control-dispatcher route" + +# (FAILSAFE) unresolvable anchor -> skipped, not quiesced. +grep -q '^s-orphan' "$TMP/cleared" \ + && bad "(FAILSAFE) must NOT quiesce a root whose anchor cannot be resolved" \ + || ok "(FAILSAFE) unresolved anchor -> skipped (fail closed)" +# The warning is a diagnostic, so it goes to stderr (matching the other passes); +# capture both streams to assert on it. +printf '%s\n' "$ERR1" | grep -q 'anchor unresolved' \ + && ok "(FAILSAFE) unresolved root is reported on stderr" || bad "(FAILSAFE) unresolved root reported" + +# (NOCLOSE) the DANGER clause: nothing is ever closed and status is never written. +grep -qE -- '--status|--close|bd close' "$TMP/updates" \ + && bad "(NOCLOSE) pass must never close a step bead or rewrite status" \ + || ok "(NOCLOSE) no step bead closed, no status rewritten (DANGER clause honored)" +# Static guard: no close/status-write COMMAND may exist in the script at all. +# Matches invocations only — the header comments legitimately discuss closing, +# since explaining why this pass must never close is half the point of the file. +grep -qE -- 'bd close|--status[ =]+closed|--close([ =]|$)' "$SCRIPT" \ + && bad "(NOCLOSE) script must contain no close/status-write command" \ + || ok "(NOCLOSE) script contains no bead-close command whatsoever" + +# (ANCHOR) the anchor bead itself is never updated. +grep -qE 'bd update anchor-' "$TMP/updates" \ + && bad "(ANCHOR) must never write to the anchor" || ok "(ANCHOR) anchor never modified" + +# (QUIET) an already-quiesced step is counted, not re-updated. +grep -q '^s-quiet' "$TMP/cleared" \ + && bad "(QUIET) already-quiet step must not be re-updated" \ + || ok "(QUIET) already-quiet step skipped (idempotent)" + +printf '%s\n' "$OUT1" | grep -q '3 steps quiesced across 3 completed workflow(s); 1 still live, 1 already quiet, 1 unresolved' \ + && ok "run 1 summary counts are exact" || bad "run 1 summary (got: $(printf '%s' "$OUT1" | tail -1))" + +# --- Run 2: convergence — a swept molecule stays swept. ----------------------- +: > "$TMP/cleared"; : > "$TMP/updates" +OUT2="$(bash "$SCRIPT")" +eq "$(wc -l < "$TMP/updates" | tr -d ' ')" "0" \ + "(IDEM) second pass issues no updates — quiesced steps stay quiesced" +printf '%s\n' "$OUT2" | grep -q '0 steps quiesced' \ + && ok "(IDEM) second pass reports nothing left to do" || bad "(IDEM) second-pass summary (got: $(printf '%s' "$OUT2" | tail -1))" + +echo "---" +echo "$PASS passed, $FAIL failed" +[ "$FAIL" -eq 0 ] diff --git a/doctor/check-base-artifact-collision/run.sh b/doctor/check-base-artifact-collision/run.sh index af203cc..73beb95 100755 --- a/doctor/check-base-artifact-collision/run.sh +++ b/doctor/check-base-artifact-collision/run.sh @@ -54,7 +54,8 @@ # signoff_head) + protected-branch auto-promote + # integration-branch INFO local deltas. # - formulas/mol-witness-patrol.toml — base + cycle-recycle + -# snake_case session-list jq + .work_dir metadata local deltas. +# snake_case session-list jq + .work_dir metadata + completed-workflow +# quiesce step (tk-p9ji9) local deltas. # # The allowlist is intentionally narrow. Adding a new entry means the # rig is taking on the maintenance cost of re-reconciling that artifact diff --git a/formulas/mol-witness-patrol.toml b/formulas/mol-witness-patrol.toml index a118787..93463c4 100644 --- a/formulas/mol-witness-patrol.toml +++ b/formulas/mol-witness-patrol.toml @@ -28,6 +28,8 @@ monitors the WORK layer: 2. **Refinery queue health** — work beads assigned to refinery, staleness. 3. **Polecat health** — detect stuck polecats, file warrants for dog pool. 4. **Help mail** — triage HELP/escalation requests from polecats. +5. **Completed-workflow quiesce** — retire the dead step beads of finished + mol-polecat-work molecules so the pool stops re-offering them (tk-p9ji9). Gate checks and convoy/swarm completion are town-wide concerns handled by the deacon, not the per-rig witness. @@ -58,7 +60,7 @@ the worktree. This makes the work schedulable again. Read each step's description before acting — Config values override defaults.""" formula = "mol-witness-patrol" -version = 11 +version = 12 [vars] [vars.binding_prefix] @@ -499,14 +501,100 @@ Recommendation: " ``` **Exit criteria:** Queue health assessed; nudge sent or escalation -filed if needed. Continue to `check-polecat-health` — do NOT exit the -wisp here. The `next-iteration` step is the only place that pours the +filed if needed. Continue to `quiesce-completed-workflows` — do NOT exit +the wisp here. The `next-iteration` step is the only place that pours the next wisp and burns this one.""" +[[steps]] +id = "quiesce-completed-workflows" +title = "Quiesce completed workflows" +needs = ["check-refinery"] +description = """ +Retire the dead step beads of `mol-polecat-work` molecules whose inline +execution has already finished, so the pool stops re-offering them (tk-p9ji9). + +**The failure this prevents.** mol-polecat-work materializes 7 step beads, but +the polecat runs them INLINE in one session and no step closes its own bead. +The graph is chained, so while `load-context` stays open it is the ONLY ready +step — and it still carries `gc.routed_to=/{{binding_prefix}}polecat`. +Open + ready + routed is exactly the pool's offer predicate, so every idle +polecat gets handed the same dead step, forever. Each one burns a wisp to +re-derive "already done, nothing to do" and drain. The molecule cannot finalize +itself either: under close-on-land its anchor stays OPEN until the refinery +lands the PR, so the churn runs for the whole human-approval wait — measured at +~1 wisp / 4-5 min, i.e. ~200 wisps overnight on a single anchor. + +This step is the containment the witness previously performed BY HAND, molecule +by molecule (ten of them by 2026-07-22), now automated. + +**Two re-offer shapes, two different levers.** Clearing `gc.routed_to` fixes +only the first — verified live: the pool work query returns open, UNASSIGNED, +routed, ready beads only, so an assigned step never rides the pool path. + +| Shape | State | Lever | +|---|---|---| +| unassigned | assignee empty + `gc.routed_to` set | clear `gc.routed_to` | +| assigned | assignee=`` + `gc.session_affinity=require` | clear `gc.routed_to` **and** the assignee | + +For the assigned shape the bead is already invisible to the pool; it is handed +back on the ASSIGNED-work path, which is keyed on the assignee. Clearing only +`gc.routed_to` there is a no-op — that is why the by-hand sweeps kept leaving a +residual of sessions still holding their in_progress step. + +**DANGER — never "fix" this by closing the step bead.** Closing `load-context` +unblocks `workspace-setup` and walks the next polecat forward onto a branch that +is ALREADY green-gated and PR'd; any push there moves the head, stales the +anchor's `check.=green@` marker, and BLOCKS the open PR from merging. +The script has no close path at all, and its regression test asserts that. +Clearing the routing is the safe lever; closing is not. + +Quiescing is containment, not finalization — the molecule is left stranded but +quiet. Finalizing the step graph at submit-and-exit time is the durable upstream +fix (gascity core / gastown formula) and is deliberately out of scope here. + +```bash +# Resolve the pack scripts the same way mol-refinery-patrol does: from agent-env +# paths, so the idle loop runs the CURRENT script even when a worktree copy lags. +SCRIPTS_DIR="" +for cand in "${GC_RIG_ROOT:-}/assets/scripts" \ + "$(git rev-parse --show-toplevel 2>/dev/null)/assets/scripts" \ + "${GC_CITY_PATH:-}/rigs/gc-toolkit/assets/scripts"; do + if [ -x "$cand/quiesce-completed-workflows.sh" ]; then SCRIPTS_DIR="$cand"; break; fi +done +# Best-effort, like every other patrol pass: a failed pass is retried next cycle +# and must never abort the wisp. The -x guard means a rig that has not synced +# this script simply keeps the old by-hand behavior. +if [ -n "$SCRIPTS_DIR" ]; then + "$SCRIPTS_DIR/quiesce-completed-workflows.sh" \ + || echo "quiesce-completed-workflows: pass failed (non-fatal); retries next cycle" >&2 +fi +``` + +Run it with `--dry-run` first if you want to see the selection without writing. + +**What it touches, and what it must not.** For each molecule it resolves the +anchor the way the formula itself does (root -> `gc.input_convoy_id` -> the +convoy's single tracked member) and acts only when that anchor is DONE: +`merge_result` of `pre_open_gate`, `pull_request`, or `merged`, or the anchor is +closed. It never writes to the anchor, never rewrites `status`, and never +touches the `workflow-finalize` step — that one keeps its control-dispatcher +route because it is the molecule's only escape path. An anchor it cannot resolve +is SKIPPED, deliberately: quiescing a live molecule would strip the assignee off +steps a running polecat still has to claim, draining it mid-implementation. An +un-quiesced husk only wastes wisps, which is much the cheaper failure. + +Regression test: `assets/scripts/quiesce-completed-workflows.test.sh` (hermetic, +stubs `gc`; covers both shapes, the single-update atomicity, the no-close rule, +the finalize-route exemption, fail-closed, and idempotence). + +**Exit criteria:** Completed-workflow husks quiesced, or none found. Continue to +`check-polecat-health` — do NOT exit the wisp here. The `next-iteration` step is +the only place that pours the next wisp and burns this one.""" + [[steps]] id = "check-polecat-health" title = "Check polecat work progress" -needs = ["check-refinery"] +needs = ["quiesce-completed-workflows"] description = """ Detect polecats that are alive but not making progress on their work.