Pool re-offers a completed mol-polecat-work's never-closed load-context step FOREVER — permanent per-molecule polecat burn, does NOT stop when the anchor lands (tk-p9ji9)#205
Merged
Conversation
…uted_to AND assignee so the pool stops re-offering dead steps (tk-p9ji9)
mol-polecat-work runs its 7 step beads INLINE in one session and no step
closes its own bead. The graph is chained, so while load-context stays open
it is the ONLY ready step — and it still carries gc.routed_to=<rig>/<prefix>
polecat. Open + ready + routed is exactly the pool's offer predicate, so every
idle polecat is handed the same dead step, forever. Each burns a wisp to
re-derive "already done" and drain. The molecule cannot finalize itself
either: under close-on-land its anchor stays OPEN until the refinery lands the
PR, so the churn runs for the whole human-approval wait (~1 wisp/4-5min,
~200 overnight on a single anchor). The witness had been containing this BY
HAND, molecule by molecule — ten of them by 2026-07-22.
Automates that containment as a witness-patrol pass.
Two re-offer shapes need two different levers, and clearing gc.routed_to
fixes only the first. Verified live: the pool work query returns open,
UNASSIGNED, routed, ready beads only, so an assigned step never rides the
pool path at all.
unassigned assignee empty + gc.routed_to set
-> the pool offers it; clearing gc.routed_to removes it.
assigned assignee=<polecat session> + gc.session_affinity=require
-> already invisible to the pool; handed back on the ASSIGNED
path, keyed on the assignee. Clearing gc.routed_to here is a
NO-OP — which is why the by-hand sweeps kept leaving a
residual of sessions still holding their in_progress step.
Both keys go in ONE gc bd update. A two-call sequence is unsafe: clearing the
assignee first would briefly leave the bead open+unassigned+routed — the exact
pool-offer shape — racing a fresh polecat into the husk being retired.
DANGER clause preserved: the pass NEVER closes a step bead. Closing
load-context unblocks workspace-setup and walks the next polecat onto a branch
that is already green-gated and PR'd; any push there stales the anchor's
check.<gate>=green@<oid> marker and BLOCKS the open PR. There is no close path
in the script and the test asserts its absence. It also never writes the
anchor, never rewrites status, and never touches workflow-finalize (routed to
the control dispatcher — the molecule's only escape path). An anchor it cannot
resolve is SKIPPED: quiescing a live molecule would strip the assignee off
steps a running polecat still has to claim, draining it mid-implementation,
whereas an un-quiesced husk only wastes wisps.
Quiescing is containment, not finalization — the molecule is left stranded but
quiet, matching what the manual sweep achieves. Finalizing the step graph at
submit-and-exit time is the durable upstream fix (gascity core / gastown
formula) and is deliberately out of scope.
Validated against live rig state, not just a compile check:
- dry-run selected exactly the completed molecules, skipping 3 live ones
(including this bead's own in-flight molecule)
- real pass quiesced 12 steps across 2 completed workflows, one of which
(tk-c9ivi) crossed into pre_open_gate mid-session and was caught the
same pass
- after: the pool offer no longer contains any quiesced step, and both
churning sessions hold 0 assigned beads, while genuine work still flows
- anchors untouched (PR #203 still pull_request, codex green at head);
step status unchanged, nothing closed; finalize routes intact
Tests: assets/scripts/quiesce-completed-workflows.test.sh — 20 assertions,
hermetic. Full suite 11/11 green (264 assertions). shellcheck clean at
severity>=warning.
Contributor
Author
|
Codex signoff (pre-open, comment-only — not an approval): COMMENT (pass): pre-open branch review completed for polecat/tk-p9ji9 -> main at e9f8270. No blocking findings. Verification:
Coverage note: no PR exists yet, so no GitHub review was posted. Harness policy did not allow sub-agent spawning without explicit delegation, so this was a direct sequential branch review. Review artifact: /tmp/compound-engineering/ce-code-review/20260722-082626-55d30e16/report.md |
johnzook
approved these changes
Jul 22, 2026
zook-bot
added a commit
that referenced
this pull request
Jul 23, 2026
…ke-reason-aware teardown (tk-z130v.3) (#213) ## Summary # Ground gc-toolkit bead-hosts in core (assigned-work wake reason) + wake-reason-aware teardown Parent decision: tk-z130v (read its notes — the A/B evidence and the accepted trade are there). Repo: `zookanalytics/gc-toolkit` (INTERNAL fork; standard bead→PR→Codex→ refinery flow — open a PR, do NOT self-merge). This IS an implementation bead. ## Goal (one sentence) A gc-toolkit bead-host must survive an INVOLUNTARY config-drift drain by being brought back automatically by the reconciler with its conversation intact — achieved by making the host the **assignee of its hosted bead** (the assigned-work wake reason survives the `drained` state), NOT by any fork exemption. ## Proven facts you are building on (do not re-derive; verify only where noted) - gascity core revives a stopped session that is the assignee of a bead with awake-demand. A/B proven on this city: assignee set ⇒ revived ~20s, no operator action; assignee cleared ⇒ stayed `drained`. (Mechanism: `compute_awake_set.go` assigned-work loop has no `Drained` gate; `workBeadHasAwakeDemand` = in_progress, or open+Ready.) - Assignment MUST use the session **name** (`s-<id>`), not the session id — passing a session id to `gc bd update <tk-bead> --assignee` flips the resolver to the HQ store and fails "no issue found". `s-<id>` is also what the awake-set matches on. - A grounded host does NOT idle-sleep and cannot be stopped by plain `drain-ack` while the assignment + awake-demand hold (revives ~20s). Teardown must clear the wake reason FIRST. ## The change (core of it) The dual-link already written on host bring-up is the natural home. In `tools/gc-bead-host.sh`: 1. **`cmd_link` (bring-up)** — in addition to the existing reverse/forward/lineage writes, set the **work bead's `assignee` to the session name** (`$name`, already resolved in that function). This is the grounding. Keep it idempotent (re-linking the same session must not thrash). Note the existing convention (see `gc-helm.sh` header comment: "a child bead's `assignee` is its OWNING session — the session_name"), so assignee=session_name is already an established meaning here — confirm the host bead is consistent with it. 2. **`cmd_unlink` (teardown / re-bind)** — clear the work bead's `assignee` (alongside the existing metadata clears) so the host can actually be stopped. Without this, a re-bind or cleanup leaves the old session grounded and un-drainable. 3. **`gc-helm.sh takeaway --release`** — ALREADY clears assignee (`--assignee=`) as its closing step (verified). Confirm it still does after your change and that release = ungrounded-then-stoppable. If any other operator-facing "done with this bead" path exists, it must also clear the wake reason before drain. ## The prompt/model change (do not skip — the old guidance is now wrong) `agents/bead-host/prompt.template.md` currently tells hosts to `gc runtime drain-ack` "between visits" for token economy. For a GROUNDED host that is a no-op-with-restart (revives ~20s). Update the model to ONE coherent story and make the prompt match it: - **DEFAULT (recommend): warm-while-live.** A grounded host stays warm as long as its bead has awake-demand; it is put down by ENDING the work — close the bead, or park it (defer/block so it is no longer `ready`), or `takeaway --release`. Replace the "drain-ack between visits" instruction accordingly: intentional teardown clears the wake reason (release / close / park) FIRST, then drains. - **ALTERNATIVE (only if you verify it): hold-to-sleep.** Keep the assignment but use a bounded `gc session suspend`/`held_until` to sleep between visits — the hold beats every wake reason (spike claim, `compute_awake_set.go` "Hold suppression — overrides everything"). BEFORE proposing this, VERIFY EMPIRICALLY that a config-drift occurring DURING the hold still results in eventual revival (not permanent death). This interaction is UNVERIFIED. If you cannot verify it cleanly, do NOT adopt it — ship the warm-while-live model. ## Guardrails to verify (must not regress) - **Assignee=session_name must not make the host bead look like pool/handback work.** Confirm scale_check, keeper prime-sweep (assignee-based), witness reconcile, and any pool router do NOT pick up a host bead just because it now has an assignee. (The child-bead convention suggests safe, but a HOST bead is often a decision/open bead — check it isn't treated as an unassigned handback or a routable root.) - **Migration / live hosts.** After deploy, existing live bead-hosts are still ungrounded. Provide a one-time backfill (set assignee=session_name on every live host bead from its hosts_bead link) OR document that they ground on next `open`. This matters: the exemption drop (tk-z130v.4) is gated on all live hosts being grounded, and THIS conversation's host (tk-z130v) is one of them. - **Idempotency / cross-rig.** `up` runs cross-rig with `BEADS_DIR` pinned; the assignee write must land in the bead's own rig ledger (same pinning as the existing link writes). ## Tests / verification (TDD where practical) - Offline-testable: `link` sets assignee=session_name on the work bead; `unlink` clears it; both idempotent. (gc-bead-host.sh already has offline-testable link/unlink — extend those.) - Live smoke on a THROWAWAY host (mirror tk-z130v's A/B; do not touch real hosts): open → assignee set → drain/idle → auto-revive; release/unlink → assignee cleared → drain sticks. Tear the throwaway down cleanly. - Do NOT drift real city config to test the involuntary path; the assigned-work grounding is the tested invariant and the idle-revive A/B already stands in for it. ## Deliverable A PR on `zookanalytics/gc-toolkit` implementing the above, with the prompt/model updated to the shipped sleep story, guardrails verified, and a backfill (or a documented ground-on-next-open) for live hosts. Report the PR number back on this bead. Do NOT open any upstream (gastownhall) PR. Do NOT touch the gascity fork or gc-8yr6px — that is tk-z130v.4, deliberately separate and blocked on this. ## Implementation notes Implemented (branch polecat/tk-z130v.3 @ ca40c4d, target main). Grounding + warm-while-live teardown. CORE (tools/gc-bead-host.sh): `link` sets work-bead assignee=session_name — the assigned-work wake reason compute_awake_set revives across an INVOLUNTARY drain (no Drained gate); `unlink` clears it (ungrounds → stoppable); new `backfill` grounds already-linked hosts city-wide, rig-pinned per bead. New/resumed hosts ground automatically via up→link. gc-helm `takeaway --release` already clears assignee — verified + noted it ungrounds. MODEL (agents/bead-host/{prompt.template.md,agent.toml}): warm-while-live — a grounded host stays warm while its bead has awake-demand; teardown clears the wake reason (close/park/release) FIRST, then drains. hold-to-sleep NOT adopted (unverified; a polecat must not drift live config). GUARDRAILS (verified, must-not-regress): scale-check/keeper/pool-router/refinery/reconcile never act on a session-name assignee (they key on gc.routed_to/branch/merge_result/own-identity; grounding also fails --unassigned). The witness recover-orphaned-beads scan newly sees grounded host beads, but drained/asleep already classify not-orphaned (revival scenario safe); closed the dead-host edge safe-by-code (cf #206/#210): recover-orphaned-beads drops beads grounded to their own host (assignee==host_session_name). TESTS: bead-host-binding-fixture.sh grounding/unground/idempotent/backfill (31/31 live); host-bead-skip.test.sh (5/5, extracts the witness filter verbatim). bash -n + shellcheck clean. Rebased onto origin/main incl. #205 — quiesce is orthogonal (keys gc.step_ref=mol-polecat-work.*, never host beads). DEFERRED (operator; needs a live LLM session — a polecat must not spawn/reset live sessions): live smoke on a throwaway host (open→assignee set→drain→auto-revive; release/unlink→cleared→drain sticks). Parent tk-z130v already A/B-proved revival; this ships the metadata contract that triggers it. Scope: gc-toolkit only (no upstream/gascity; tk-z130v.4 separate). PR opens via the refinery pre-open codex gate — PR number to be recorded then. Rework tk-5bygy landed on polecat/tk-z130v.3 at 358e0b7; re-review tk-knb1t dispatched (one-anchor-per-PR, tk-ynz4b). Rework tk-v369i landed on polecat/tk-z130v.3 at 9fd8e47; re-review tk-wc3gf dispatched (one-anchor-per-PR, tk-ynz4b). Rework tk-0gnq9 landed on polecat/tk-z130v.3 at cc62091 (rebased onto origin/main 4b09bd5 — resolved the assets/scripts/gc-helm.sh cmd_clear conflict with landed tk-xypcy #211, preserving the codex signoff fixes); re-review tk-q6zwf dispatched (one-anchor-per-PR, tk-ynz4b). Gates at cc62091: bash -n clean; shellcheck clean at warning+ (info-only SC2015 x41 on the A && ok || bad assertion idiom, SC2016 x4). ## Refinery handoff - Issue: `tk-z130v.3` (task, P1) - Source branch: `polecat/tk-z130v.3` - Target: `main` - Codex signed off pre-open at `cc620913`; PR opened codex-green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OBSERVED (gc-toolkit, 2026-07-17T09:21Z-09:32Z): four polecat sessions
(lx-xz3a6, lx-94hvo, lx-8tznd, lx-r2nhg) were spawned ~5min apart and each
claimed the SAME step bead tk-cnt3i (mol-polecat-work.load-context) for work
that was already complete, shipped, and parked in the merge gate. Each session
burned a wisp to re-derive "nothing to do" and drain. See tk-3b5hv notes for the
per-incarnation record.
MECHANISM:
them INLINE in one session; no step contains
bd close. Steps are meant to befinalized by machinery.
while load-context stays open it is the ONLY ready step.
pool-routed = the pool re-offers it to every idle polecat, forever.
complete. The convoy's single member is the work bead tk-3b5hv, which stays OPEN
by design under close-on-land until the refinery lands the PR. So the workflow
cannot finalize while the PR sits in the gate, and the loop runs for the whole
duration of the HUMAN approval wait (potentially hours/overnight).
SCOPE: likely affects EVERY mr-mode PR awaiting operator approval, not just this
bead. Corroborating: six "Finalize workflow" step beads are currently open in this
rig (tk-nzuv8, tk-smqq0, tk-sbhp9, tk-f372s, tk-tdey1, tk-x82ck).
DANGER — DO NOT "FIX" BY CLOSING THE STEP BEAD:
Closing load-context to stop the churn would unblock workspace-setup, and the pool
would walk the next polecat forward into implement/self-review on a branch that is
ALREADY green-gated and PR'd. Any push there moves the head, which stales the
anchor's
check.codex=green@<oid>marker and BLOCKS the open PR from merging. The"polecat never closes mol-polecat-work step beads" rule exists for this reason.
The correct fix is at the machinery level, not by hand on live workflow state.
CANDIDATE FIXES (for triage, not prescriptive):
e.g. clear/suppress gc.routed_to on step beads once submit-and-exit has run
(the anchor carries merge_result=pre_open_gate|pull_request, a reliable signal
that the workflow's inline execution is DONE).
carries a terminal merge_result.
completion, so close-on-land's intentionally-open anchor stops holding the
whole step graph open.
RELATED: tk-qnsqhv (double-dispatch race — distinct: that is a bead re-pooled
while its original polecat is still live; this is a completed workflow's dead step
re-offered indefinitely).
EVIDENCE ANCHOR: tk-3b5hv / PR #192 (open, non-draft, headRefOid=846c24a,
check.codex=green@846c24a, reviewDecision=REVIEW_REQUIRED — normal awaiting-
external-approval, not a stall).
Implementation notes
Implemented: witness-patrol now auto-quiesces completed mol-polecat-work husks (clears gc.routed_to AND assignee; never closes a step). Validated against live rig state — 12 steps quiesced across 2 done molecules, both re-offer paths confirmed dead, anchors/PR #203 undisturbed. 11/11 test suites green.
RE-OFFER #1 (resumed session gc-toolkit__polecat-lx-vg082, 2026-07-22T08:14Z): load-context husk tk-fgkgk was re-offered on the hook. Verified NO action needed — this is the bug's own symptom, not new work. Anchor state: merge_result=pre_open_gate, branch polecat/tk-p9ji9 pushed at e9f8270, review bead tk-ghlvg OPEN and routed to gc-toolkit/gc-toolkit.polecat-codex (pre-open codex gate queued, in flight). Deliverable re-verified against the required scope in the 2026-07-22 comment: quiesce-completed-workflows.sh handles BOTH shapes (unassigned -> unset gc.routed_to; affine assignee+session_affinity=require -> clears assignee too), gated on anchor merge_result in {pre_open_gate,pull_request,merged}, never closes a step (DANGER clause honored), wired into mol-witness-patrol.toml with a hermetic test. No re-handoff, no re-implement, no close. Drained.
RE-OFFER #2 (same session slot gc-toolkit__polecat-lx-vg082, 2026-07-22T08:19Z): husk tk-fgkgk re-offered via the ASSIGNED-WORK path (gc hook reason=existing_assignment — the assigned shape, so clearing gc.routed_to alone is a no-op). Re-verified: branch polecat/tk-p9ji9 @ e9f8270, local HEAD == origin; commit carries quiesce-completed-workflows.sh + .test.sh + mol-witness-patrol.toml wiring; no PR (correct for pre_open_gate); review bead tk-ghlvg in_progress with codex polecat lx-07t9t, gate in flight and healthy. No action needed. ESCALATED to gc-toolkit.witness (mail lx-wisp-ousk1) to de-route the husk (clear assignee AND gc.routed_to) and sweep siblings of root tk-zleg5; DANGER clause restated — no step bead closed. Future incarnations: escalation is already filed, verify silently and drain; do NOT re-escalate or re-append.
Refinery handoff
tk-p9ji9(bug, P2)polecat/tk-p9ji9maine9f82707; PR opened codex-green.