Skip to content

feat: add safe secondmate lane restart helper#235

Open
mielyemitchell wants to merge 13 commits into
kunchenguid:mainfrom
mielyemitchell:fm/fm-lane-restart-x1
Open

feat: add safe secondmate lane restart helper#235
mielyemitchell wants to merge 13 commits into
kunchenguid:mainfrom
mielyemitchell:fm/fm-lane-restart-x1

Conversation

@mielyemitchell

Copy link
Copy Markdown

Intent

Add a one-command secondmate lane context-refresh helper at bin/fm-restart.sh . It must only accept existing kind=secondmate state records, refuse disposable crewmates, ask the lane to stow durable context through the normal marked fm-send secondmate path, then require a bounded lane status-line signal confirming stow completed before any restart. If that stow completion signal does not arrive, abort clearly without sending raw exit or respawning. Provide --skip-stow as the explicit escape hatch for lanes already stowed or dead. After stow completion or --skip-stow, send the harness exit command itself directly and unmarked to the resolved raw backend endpoint so /exit or /quit is handled by the harness rather than converted into chat. Verify the harness process/pane is actually gone using backend process/current-state primitives, never chat acknowledgement. If the lane is already dead it should skip nudge/wait and respawn cleanly. On herdr it must verify the recorded pane still belongs to the expected fm- tab before control actions, avoid last-tab refusal by renaming the old tab, respawn through bin/fm-spawn.sh --secondmate, then close the old tab with cleanup verification. On tmux it should kill the old window after exit and respawn through fm-spawn. --force should interrupt/exit with the harness-appropriate sequence after stow completion if the lane does not exit in time. Keep bash 3.2 compatibility, use fm-backend.sh primitives instead of raw tmux/herdr when primitives exist, update AGENTS.md briefly, and add focused bin tests for live, dead, force, refusal, required stow completion, --skip-stow, raw-exit routing, and backend cleanup behavior.

What Changed

  • Captain, added bin/fm-restart.sh to safely restart registered secondmate lanes, with stow gating, --skip-stow, --force, dead-lane handling, and raw harness exit routing.
  • Extended backend primitives for tmux and herdr so restart can verify lane liveness, validate herdr tab ownership, respawn through fm-spawn.sh, and clean up old backend surfaces.
  • Updated firstmate docs and added focused restart/backend tests covering stow refusal, forced exit, dead respawn, raw exit routing, and cleanup behavior.

Risk Assessment

⚠️ Medium: Captain, this is a clean review pass, but the change is still medium risk because it adds a new stateful restart path that sends control input, kills endpoints, and respawns persistent lanes.

Testing

Baseline full tests had already passed; I then ran the focused restart, herdr backend, and session-start tests, plus a CLI-level fake tmux lane transcript showing stow confirmation before raw /quit, abort before exit/respawn when stow is missing, and clean respawn when the lane is already dead. No UI screenshot was applicable because this is a shell/backend lane-control change.

Evidence: Targeted shell test log
== tests/fm-restart.test.sh ==
ok - fm-restart: refuses non-secondmate ids
ok - fm-restart: dead lane respawns without nudge/wait
ok - fm-restart: refuses unsupported backends instead of guessing process state
ok - fm-restart: live lane with unknown process state aborts before stow
ok - fm-restart: live lane with mismatched process aborts before cleanup
ok - fm-restart: live tmux lane stows/exits, kills old window, respawns
ok - fm-restart: tmux detects node-wrapped codex from pane process argv
ok - fm-restart: live herdr lane renames old tab, respawns, then closes old tab
ok - fm-restart: herdr cleanup verifies the old endpoint is gone before success
ok - fm-restart: stow timeout aborts before raw exit
ok - fm-restart: --skip-stow is the explicit stow escape hatch
ok - fm-restart: exit timeout aborts honestly without --force after stow
ok - fm-restart: --force uses harness exit sequence, then closes and respawns
ok - fm-restart: stale herdr pane labels are not controlled
== tests/fm-backend-herdr.test.sh ==
ok - fm_backend_herdr_version_check: accepts the current protocol (14)
ok - fm_backend_herdr_version_check: refuses an old protocol loudly
ok - fm_backend_herdr_version_check: refuses loudly when herdr is not installed
ok - fm_backend_herdr_workspace_label: a primary home (no marker) resolves to 'firstmate'
ok - fm_backend_herdr_workspace_label: a secondmate home (.fm-secondmate-home) resolves to '2ndmate-<id>'
ok - fm_backend_herdr_workspace_label: trims whitespace around the marker's secondmate id
ok - fm_backend_herdr_workspace_label: an empty marker file falls back to the primary label 'firstmate'
ok - fm_backend_herdr_workspace_label: two different secondmate homes get two different, non-colliding labels
ok - fm_backend_herdr_cli: sets HERDR_SESSION AND appends a trailing --session flag on every call
ok - fm_backend_target_exists: herdr liveness checks use explicit session targeting without autostart
ok - fm_backend_herdr_container_ensure: version-gates, starts the server, ensures the firstmate workspace, echoes session:workspace_id + the seeded default tab id
ok - fm_backend_herdr_container_ensure: reuses an existing firstmate workspace without recreating it, and reports no seeded default tab (adopted, not created)
ok - fm_backend_herdr_container_ensure: workspace create passes --no-focus
ok - fm_backend_herdr_container_ensure: creates the workspace under the SECONDMATE home's own label, not 'firstmate'
ok - fm_backend_herdr_create_task: prunes exactly the seeded default tab container_ensure identified, once the first real task tab exists
ok - herdr repeated spawn/teardown: one persistent firstmate workspace reused, zero orphans, default tab pruned, create ran once
ok - fm_backend_herdr_create_task: an ADOPTED workspace's pre-existing tab is never pruned (the created-vs-adopted gate)
ok - fm_backend_herdr_create_task: the label-collision startup-workspace scenario (2026-07-02 incident) leaves the captain's live tab untouched
ok - fm_backend_herdr_workspace_prune_seeded_default_tab: refuses to close the seeded default tab when its pane reports a working agent (defense in depth)
ok - no bin/ jq filter names a --arg/--argjson variable after a jq reserved keyword
ok - fm_backend_herdr_create_task: refuses a duplicate tab label (herdr's own tab create has no uniqueness check)
ok - fm_backend_herdr_create_task: a same-labeled tab with a live (even idle) registered agent still refuses exactly as before
ok - fm_backend_herdr_create_task: scans every same-labeled tab and refuses if any duplicate is live
ok - fm_backend_herdr_create_task: closes and replaces a same-labeled tab whose pane is dead (pane_not_found)
ok - fm_backend_herdr_create_task: closes and replaces a same-labeled tab whose pane is alive but hosts no registered agent (a restored plain shell)
ok - fm_backend_herdr_create_task: closes every confirmed same-labeled husk only after creating the replacement
ok - fm_backend_herdr_create_task: refuses success when a preexisting husk tab remains after replacement
ok - fm_backend_herdr_create_task: refuses (fail-safe) rather than guessing when the duplicate's agent state cannot be classified confidently
ok - fm_backend_herdr_create_task: creates the replacement tab BEFORE closing the husk tab, never the reverse
ok - fm_backend_herdr_create_task: creates a tab and parses tab_id/pane_id from the JSON response, prunes nothing when no seeded tab id is given
ok - fm_backend_herdr_create_task: tab create passes --no-focus
ok - fm_backend_herdr_workspace_find: matches only THIS home's own label among several coexisting workspaces
ok - fm_backend_herdr_list_live: scoped to this home's own workspace, never a sibling home's
ok - fm_backend_herdr_parse_target: splits '<session>:<pane_id>' on the FIRST colon (pane_id itself contains one)
ok - fm_backend_herdr_normalize_key: Enter/Escape/C-c/C-q map to herdr's verified enter/escape/ctrl+c/ctrl+q
ok - fm_backend_herdr_capture: calls 'pane read <pane> --source recent --lines N' with the session set
ok - fm_backend_herdr_capture: works around the verified small-N '--lines' bug by over-fetching and trimming locally
ok - fm_backend_herdr_capture: ensures the session and preserves pane read failure
ok - fm_backend_herdr_send_key: normalizes the key and targets the right pane
ok - fm_backend_herdr_kill: calls pane close and stays best-effort on failure
ok - fm_backend_herdr_current_path: reads pane foreground_cwd (the live running process), not the frozen creation-time cwd
ok - fm_backend_herdr_busy_state: working -> busy
ok - fm_backend_herdr_busy_state: done -> idle, blocked -> idle (surfaced like a stale pane, not suppressed as busy)
ok - fm_backend_herdr_busy_state: unparseable/absent agent state reports unknown, the regex-fallback cue
ok - fm_backend_herdr_composer_state: a bare '❯' composer row reads empty
ok - fm_backend_herdr_composer_state: the ghost placeholder text reads empty, not pending
ok - fm_backend_herdr_composer_state: real composer text reads pending
ok - fm_backend_herdr_composer_state: a slash-command popup's argument-hint placeholder still reads pending (the incident fix)
ok - fm_backend_herdr_composer_state: reports unknown when the pane cannot be captured
ok - fm_backend_herdr_composer_state: reports unknown when no border-delimited composer row is found
ok - fm_backend_herdr_send_text_submit: reports 'empty' once the composer row reads empty after one Enter
ok - fm_backend_herdr_send_text_submit: reports 'pending' when the composer never clears after retried Enters (swallowed)
ok - fm_backend_herdr_send_text_submit: a slash-command popup's placeholder fill on Enter #1 does not short-circuit as submitted; Enter #2 is retried and lands it
ok - fm_backend_herdr_send_text_submit: reports 'send-failed' when the literal send-text call itself errors
ok - fm_backend_herdr_send_text_submit: reports 'unknown' when the post-Enter capture fails (never retries past an unreadable pane)
ok - fm_backend_validate: herdr is a known backend (P2)
ok - fm_backend_busy_state: tmux (no native primitive) always reports unknown, preserving the P1 regex-only path
ok - fm-peek/fm-send: explicit stale targets matching metadata use the recorded backend
== tests/fm-session-start.test.sh ==
ok - context digest distinguishes ABSENT, empty-but-present, and populated files
ok - a lock refusal prints a loud read-only banner, skips every mutating step, and still completes the digest
ok - digest sections are ordered diagnostics-first, bulk-context-last
ok - status tail is bounded to the configured line count, with the full log path always printed
ok - orphan status logs are printed once with bounded tails
ok - tmux endpoint liveness is reported per task: alive for a live window, dead for a gone one
ok - herdr endpoint liveness is reported per task: alive for a live pane, dead for a gone one
ok - fm-session-start.sh composes the real fm-lock.sh, fm-bootstrap.sh, and fm-wake-drain.sh output verbatim
ok - an empty fleet reports (none) for in-flight tasks and an absent AFK flag
ok - next step sources the X-mode cadence before arming the watcher
ok - next step delegates watcher ownership to the AFK daemon
Evidence: fm-restart end-to-end behavior transcript

## live lane: marked stow, raw /quit, kill old window, respawn
restart: asking lane to stow; waiting for status 'stowed: restart-ready' (timeout 0s)
restart: lane refreshed
exit=0
-- event log --
event tmux args=display-message -p -t firstmate:fm-lane #{pane_id}
event tmux args=display-message -p -t firstmate:fm-lane #{pane_pid}
event send target=fm-lane payload=Stow any lane-local durable context now. Do not start new work; firstmate will refresh this lane. When stow is complete, append exactly 'stowed: restart-ready' to your status.
event tmux args=display-message -p -t firstmate:fm-lane #{pane_id}
event send target=firstmate:fm-lane payload=/quit
event tmux args=display-message -p -t firstmate:fm-lane #{pane_id}
event tmux args=display-message -p -t firstmate:fm-lane #{pane_pid}
event tmux args=display-message -p -t firstmate:fm-lane #{pane_id}
event tmux args=kill-window -t firstmate:fm-lane
event spawn args=lane --backend tmux --secondmate

## missing stow confirmation: aborts before raw exit/respawn
restart: asking lane to stow; waiting for status 'stowed: restart-ready' (timeout 0s)
exit=1
-- event log --
event tmux args=display-message -p -t firstmate:fm-lane #{pane_id}
event tmux args=display-message -p -t firstmate:fm-lane #{pane_pid}
event send target=fm-lane payload=Stow any lane-local durable context now. Do not start new work; firstmate will refresh this lane. When stow is complete, append exactly 'stowed: restart-ready' to your status.

## dead lane: no stow nudge, clean respawn
restart: lane endpoint is already dead; respawning
restart: lane refreshed
exit=0
-- event log --
event tmux args=display-message -p -t firstmate:fm-lane #{pane_id}
event tmux args=display-message -p -t firstmate:fm-lane #{pane_id}
event spawn args=lane --backend tmux --secondmate

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • 🚨 /Users/m_mit/.no-mistakes/worktrees/9a08938f86b4/01KWNJ25XZATCMCWW2A40FSHK8/bin/fm-restart.sh:251 - fm-restart.sh treats a live endpoint with an unreadable or unrecognized foreground process as safe to restart without stow. harness_running collapses process-read failures, label mismatches, and unknown harness values into the same false result, and the main branch then skips the stow request and goes straight to cleanup/respawn. That violates the restart safety contract for any live secondmate whose process primitive temporarily fails or whose harness value is missing/unrecognized: it can kill a running lane without the required stowed: restart-ready status. Split this into explicit states, e.g. running, not-running, and unknown, and abort on unknown unless --skip-stow is provided.

🔧 Fix: Captain, guard uncertain restart process state
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
  • Baseline already completed successfully before this run: command -v tmux &gt;/dev/null || { echo &#34;tmux is required for e2e tests&#34; &gt;&amp;2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo &#34;== $t ==&#34;; bash &#34;$t&#34; || rc=1; done; exit &#34;$rc&#34;
  • bash tests/fm-restart.test.sh
  • bash tests/fm-backend-herdr.test.sh
  • bash tests/fm-session-start.test.sh
  • Created and ran /var/folders/ql/vwkrq94j4mq90s39twg74z7c0000gn/T/no-mistakes-evidence/01KWNJ25XZATCMCWW2A40FSHK8/fm-restart-e2e.sh, a fake tmux-backed secondmate lane exercise that recorded successful stow-gated restart, missing-stow abort, and dead-lane respawn behavior.
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@kunchenguid

Copy link
Copy Markdown
Owner

Thanks for the PR! It looks like this branch has a merge conflict with the base branch right now. When you get a chance, could you rebase onto (or merge in) the latest base branch, resolve the conflict, and push? Once GitHub shows the PR as mergeable again, it'll be picked back up for review.

Noted for firstmate#235 at 809d5a79.

@mielyemitchell mielyemitchell force-pushed the fm/fm-lane-restart-x1 branch from 809d5a7 to 137a228 Compare July 5, 2026 18:07
@mielyemitchell

Copy link
Copy Markdown
Author

Rebased onto current main — mergeable now at 137a228 (workflows just need maintainer approval to run). Thanks for the nudge!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants