fix: dedup herdr stale wakes for settled panes

[Bre77]

Intent

Fix the herdr-backend watcher stale-echo churn: idle and done crews' herdr panes re-wake the supervisor every few minutes because bin/fm-watch.sh hashes a volatile bounded pane capture and claude's idle UI repaints timer-driven rows (the recap line toggling, rotating tips, the animated 'for Xm Ys' summary, transient slash-completion menus), so each repaint mints a fresh stable hash. Decision: dedup the stale signal on the native SEMANTIC agent state (herdr idle/done/blocked map to a constant 'agentstate:idle') instead of the pane content, because content line-stripping cannot converge - those volatile rows also shift a variable-length prefix through the tail-N hash window (measured). Constraints held: the tmux path stays byte-identical (busy state is always unknown there, so it keeps the pane-content hash); the resolved backend busy state is shared between this new stale signal and the existing busy-suppression check so herdr's agent.get is read once per pane per poll, not twice; the now-unused window_is_busy wrapper is removed in favor of the extracted busy_state_is_busy. Verified live against real herdr 0.7.1 on 2026-07-05 (a done pane churned 4 distinct hashes and an idle pane 8 over ~7 minutes, both collapsing to 1 under the semantic signal; a plain post-exit shell pane stayed at 1 and is intentionally left on the pane-hash fallback). The related herdr away-mode injection wedge is already fixed upstream by PR #251, so this change is deliberately scoped to only the watcher stale-echo churn #251 did not touch. Adds a colocated behavioral test in tests/fm-watch-triage.test.sh and a backend-verification section in docs/herdr-backend.md.

What Changed

• Captain, bin/fm-watch.sh now deduplicates settled herdr stale panes using the semantic agent state instead of volatile pane hashes, while keeping tmux and busy-looking herdr panes on the existing content-hash path.
• Resets stale suppression when the signal changes and reuses the resolved backend busy state for both stale signaling and busy suppression.
• Adds herdr stale-dedup regression coverage, stabilizes the session-start missing-tool fixture, and documents the live herdr verification notes.

Risk Assessment

✅ Low: Captain, the change is narrow, the semantic herdr stale signal matches the documented backend contract, and the added tests cover the prior regressions without introducing a material merge risk.

Testing

The configured full baseline had already passed; I reran the targeted watcher triage and session-start regression suites, then exercised a fake herdr idle pane through the real watcher and drain path, confirming semantic stale dedup wakes once, stores agentstate:idle, and does not re-wake after volatile pane repaint.

Evidence: fm-watch-triage transcript

ok - signal_reason_is_actionable: benign absorbed, captain verbs and coalesced batches surfaced
ok - stale_is_terminal: terminal status surfaces, non-terminal and no-status are benign
ok - scan_captain_relevant_statuses lists only captain-relevant statuses
ok - classifier primitives: last line, captain-relevance, window->task, FM_CAPTAIN_RE override
ok - crew_is_provably_working: only working+run-step/pane is provable; idle/finished/parked/failed/unknown surface
ok - signal_crew_provably_working: benign only when every referenced crew is provably working
ok - a no-verb signal whose crew is provably working is absorbed (no exit, no queue, suppressor advanced, beacon present)
ok - a bare turn-end whose crew is provably working (busy pane) is absorbed
ok - a bare turn-end whose crew is not provably working is surfaced (the swallowed-finish fix)
ok - a no-verb working: note whose crew is idle with no running pipeline is surfaced
ok - captain-relevant signal is surfaced (queue + exit) and marked surfaced
ok - a stale pane sitting on a terminal status is surfaced (queue + exit)
ok - herdr settled pane: stale-dedup signal is the semantic agent state, not the volatile pane hash
ok - herdr idle pane: busy banner keeps stale signal on pane-content hash
ok - herdr settled pane: stale suppressor resets after intervening work
ok - a stale terminal-looking status is overridden and absorbed while a run is actively working, then wedge-escalated
ok - provably-working non-terminal stale is absorbed on first sight, then wedge-escalated past the threshold
ok - a not-provably-working non-terminal stale is surfaced immediately (never left to wait out the timer)
ok - matching non-terminal stale suppressors repair missing or corrupt stale-since timers
ok - triage log capping handles wc byte counts with leading spaces
ok - a heartbeat with no captain-relevant change is absorbed and backs off the cadence
ok - heartbeat backstop fail-safe surfaces a captain-relevant status the per-wake path missed
ok - the liveness beacon stays fresh while the watcher absorbs benign wakes (fm-guard never false-alarms)
ok - with .afk present the watcher reverts to one-shot so the daemon owns triage (no double-triage)

Evidence: fm-session-start transcript

ok - context digest distinguishes ABSENT, empty-but-present, and populated files
ok - a lock refusal prints a loud read-only banner, skips every mutating step, and still completes the digest
ok - digest sections are ordered diagnostics-first, bulk-context-last
ok - status tail is bounded to the configured line count, with the full log path always printed
ok - orphan status logs are printed once with bounded tails
ok - tmux endpoint liveness is reported per task: alive for a live window, dead for a gone one
ok - herdr endpoint liveness is reported per task: alive for a live pane, dead for a gone one
ok - fm-session-start.sh composes the real fm-lock.sh, fm-bootstrap.sh, and fm-wake-drain.sh output verbatim
ok - an empty fleet reports (none) for in-flight tasks and an absent AFK flag
ok - next step sources the X-mode cadence before arming the watcher
ok - next step delegates watcher ownership to the AFK daemon

Evidence: Manual fake-herdr semantic stale evidence

Shows first watcher poll emitted stale: default:w1:p2, recorded hash=agentstate:idle, drained one stale wake, then stayed running with empty stdout and empty wake queue after volatile pane repaint.

## Manual fake-herdr stale-dedup evidence
Scenario: a settled herdr pane reports native idle while its visible pane text repaints.

# first watcher poll: pre-seeded semantic signal reaches n>=2 and wakes once
stale: default:w1:p2
hash=agentstate:idle
stale_suppressor=agentstate:idle
# drained durable wake
drain: 1783243072	1	stale	default:w1:p2	stale: default:w1:p2

# repaint pane text but keep native agent state idle; watcher should stay quiet
second_poll=still-running-no-duplicate-wake
second_stdout=<empty>
wake_queue_after_repaint=<empty>

# herdr command calls observed
status --json --session default
pane read w1:p2 --source recent --lines 200 --session default
status --json --session default
agent get w1:p2 --session default
status --json --session default
pane read w1:p2 --source recent --lines 200 --session default
status --json --session default
agent get w1:p2 --session default
status --json --session default
pane read w1:p2 --source recent --lines 200 --session default
status --json --session default
agent get w1:p2 --session default
status --json --session default
pane read w1:p2 --source recent --lines 200 --session default
status --json --session default
agent get w1:p2 --session default

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (19m19s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (2) ✅

• 🚨 bin/fm-watch.sh:195 - Using one constant stale key for every settled herdr period means .stale-$key remains agentstate:idle after the first surfaced stale pane. If the same pane later does more work and returns to idle/done without a status/turn-ended signal, the stale path treats it as already surfaced and can suppress or delay the new completion wake. Clear the stale suppressor when the signal leaves settled-idle, or include a transition/generation component in the semantic key.

🔧 Fix: Captain, reset stale suppressor on signal change
1 error still open:

• 🚨 bin/fm-watch.sh:195 - Captain, idle is not always a settled herdr pane: the existing docs note that long foreground tool calls can report agent_status=idle while the pane still shows the busy banner. Collapsing every idle verdict to agentstate:idle means changing tool output no longer resets the stale generation; in afk mode it can queue a stale wake after two polls, and in normal mode it can keep or suppress the stale marker across real intervening work. Corroborate idle with the pane busy regex before using the semantic settled signal, or keep the pane-content hash while the idle pane still looks busy.

🔧 Fix: Captain, corroborate idle stale signal
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed ✅

• 🚨 tests failed with exit code 1
• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: Captain, stabilize session-start missing-tool fixture
✅ Re-checked - no issues remain.

• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• Baseline already ran successfully before this validation: command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• bash tests/fm-watch-triage.test.sh | tee /tmp/no-mistakes-evidence/01KWRQ2K32MZPGK20MYN5K2X1D/fm-watch-triage.out
• bash tests/fm-session-start.test.sh | tee /tmp/no-mistakes-evidence/01KWRQ2K32MZPGK20MYN5K2X1D/fm-session-start.out
• Manual fake-herdr watcher/drain scenario saved with tee /tmp/no-mistakes-evidence/01KWRQ2K32MZPGK20MYN5K2X1D/herdr-semantic-stale-manual.out

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 874289c1c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Increase the resurface test timeout

Captain, this new test races the watcher’s own polling cadence: after flipping the fake herdr pane back to idle, the watcher may need one poll to record the new agentstate:idle signal and two more polls to reach the n>=2 stale threshold, so the 40×0.1s budget is too tight once shell/herdr-stub overhead is included. I reproduced bash tests/fm-watch-triage.test.sh failing here with not ok - watcher did not resurface a settled herdr pane after intervening work; use a larger timeout or pre-seed the post-transition count so CI does not flake/fail on slower hosts.

Useful? React with 👍 / 👎.