Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,8 @@ tests/fm-tangle-guard.test.sh # primary-checkout tangle detection, r
tests/fm-brief.test.sh # fm-brief.sh bash -n parse regression guard (issue #166) and clean no-mistakes/direct-PR/local-only brief generation tests
tests/fm-spawn-batch.test.sh # batch dispatch and FM_HOME project-path scoping tests
tests/fm-spawn-dispatch-profile.test.sh # concrete dispatch profile flags: active-profile backstop, harness/model/effort meta, launch templates, batch forwarding, and secondmate exemption
tests/fm-spawn-worktree-symlink.test.sh # worktree wait-loop path canonicalization: a symlinked project path (e.g. /tmp -> /private/tmp) must not misdetect the project as the worktree
tests/fm-gotmp.test.sh # per-task GOTMPDIR temp root: fm-spawn tasktmp= meta contract and fm-teardown cleanup of the recorded temp root
tests/fm-update.test.sh # fast-forward-only self-update, reread, nudge, dedup, and skip-safety tests
tests/fm-secondmate-sync.test.sh # local-HEAD secondmate sync, no-fetch, bootstrap nudge gating, and spawn hook tests
tests/fm-secondmate-harness.test.sh # secondmate-vs-crewmate harness resolution, optional secondmate model/effort pins, primary-to-secondmate config inheritance, and config-push tests
Expand All @@ -99,9 +101,9 @@ tests/fm-secondmate-safety.test.sh # secondmate home safety, idle charter
tests/fm-teardown.test.sh # fm-teardown.sh landed-work safety and reminder checks: fork-remote allow, squash/content landings, dirty and unlanded refusals, PR-head metadata, no-pr= branch discovery, tasks-axi/manual backlog reminder, --force override
tests/fm-review-diff.test.sh # fm-review-diff.sh authoritative review diff coverage: recorded pr_head=, fetched refs/pull/<n>/head, no-pr local branch behavior, and warning fallback
tests/fm-pr-merge.test.sh # fm-pr-merge.sh records pr= and available pr_head= before merging, parses PR URLs into gh-axi number/--repo calls, defaults to squash, preserves explicit merge methods, rejects malformed URLs and repo overrides, and propagates real merge failures
tests/fm-crew-state.test.sh # fm-crew-state.sh current-state reconciliation: run-step authority including closed panes, stale needs-decision/blocked superseded by a resumed run, genuine-parked, cross-branch runs-list attribution, pane/status-log fallback, scout skip, torn-down/missing-meta graceful
tests/fm-crew-state.test.sh # fm-crew-state.sh current-state reconciliation: run-step authority including closed panes, stale needs-decision/blocked superseded by a resumed run, genuine-parked, cross-branch runs-list attribution, bounded run-lookup retry/backoff recovering a raced `axi status` or coarse runs-list lookup, pane/status-log fallback, scout skip, torn-down/missing-meta graceful
tests/fm-backend.test.sh # runtime-backend abstraction: fm-backend.sh selection/meta/dispatch helpers, shell-portable sourced backend matching, and old-vs-new fake-tool command-log conformance for fm-send/fm-peek/fm-spawn/fm-teardown
tests/fm-backend-tmux-smoke.test.sh # real (private-socket) tmux smoke test for the tmux adapter: create/duplicate-refuse, send text + Enter, send literal + key, bounded capture, live-window resolve, kill
tests/fm-backend-tmux-smoke.test.sh # real (private-socket) tmux smoke test for the tmux adapter: create/duplicate-refuse, new-window trailing-colon session target (renumber-windows index-in-use guard), send text + Enter, send literal + key, bounded capture, live-window resolve, kill
tests/fm-backend-herdr.test.sh # fake herdr CLI unit tests for the experimental herdr adapter, including version/tool gates, target parsing, send/capture, structural composer-state verification, slash-submit retry regression coverage, native busy state, per-home workspace-label resolution, default-tab prune safety, restored-layout husk replacement, and verified CLI bug workarounds
tests/fm-backend-herdr-smoke.test.sh # real herdr adapter smoke test, skipped when herdr or jq is unavailable, using an isolated throwaway HERDR_SESSION and guarded session cleanup, including live-agent duplicate refusal and no-agent husk replacement
tests/fm-backend-autodetect-smoke.test.sh # real herdr auto-detection smoke test, skipped when herdr, jq, or treehouse is unavailable, using the same guarded session cleanup
Expand Down
11 changes: 10 additions & 1 deletion bin/backends/tmux.sh
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,22 @@ fm_backend_tmux_container_ensure() {
# refusing an existing <window-name> in <session>. Mirrors fm-spawn.sh's
# duplicate-check-then-new-window sequence, including the exact error text
# (session:window, matching how fm-spawn.sh composed its own $T).
#
# The new-window target is "$ses:" (trailing colon), NOT the bare "$ses". A bare
# session target resolves to that session's CURRENT window, so tmux inserts
# next-up-from-current; with `renumber-windows on` (tmux >= 3.7) that can land on
# an occupied index and fail "index in use". The trailing colon is the documented
# "pick the next free window index in this session" form and is unambiguous across
# tmux versions/configs, so it replaces the external PATH shim that used to rewrite
# bare session targets. Behavior is identical to the bare form when the bare form
# succeeds (both append at the next free index).
fm_backend_tmux_create_task() { # <session> <window-name> <proj-abs>
local ses=$1 wname=$2 proj_abs=$3
if tmux list-windows -t "$ses" -F '#{window_name}' | grep -qx "$wname"; then
echo "error: window $ses:$wname already exists" >&2
return 1
fi
tmux new-window -d -t "$ses" -n "$wname" -c "$proj_abs"
tmux new-window -d -t "$ses:" -n "$wname" -c "$proj_abs"
}

# fm_backend_tmux_current_path: the live pane's current working directory, or
Expand Down
91 changes: 70 additions & 21 deletions bin/fm-crew-state.sh
Original file line number Diff line number Diff line change
Expand Up @@ -307,13 +307,22 @@ log_reports_ci_ready() {
# "<status> <branch> <short-sha> <date> [<pr-url>]" separated by runs of
# spaces (verified: no quoting, so splitting on the first two whitespace runs
# is exact) - but branch + coarse status is exactly what this predicate needs:
# is a run for THIS branch active right now. Echoes the first (most recent)
# matching row's status word (running/completed/cancelled/failed), or empty
# when the branch has no run within FM_CREW_STATE_RUNS_LIMIT rows.
nm_runs_status_for_branch() { # <branch>
# is a run for THIS branch active right now. Sets COARSE_STATUS to the first
# (most recent) matching row's status word (running/completed/cancelled/failed),
# or empty when the branch has no run within FM_CREW_STATE_RUNS_LIMIT rows.
# Results come back through globals rather than stdout because the retry loop
# below also needs NM_RUNS_RESPONDED - whether the bounded `no-mistakes runs`
# call produced any output at all - and a command-substitution subshell could
# not report that second fact. `no-mistakes runs` always emits text when the
# CLI is alive (even with zero runs it prints a "no runs" line), so an empty
# capture reliably means the call timed out, never a legitimate empty answer.
nm_runs_status_for_branch() { # <branch>; sets COARSE_STATUS + NM_RUNS_RESPONDED
local branch=$1 out row st rest br
COARSE_STATUS=""
NM_RUNS_RESPONDED=0
out=$(nm_run runs --limit "$FM_CREW_STATE_RUNS_LIMIT")
[ -n "$out" ] || return 0
NM_RUNS_RESPONDED=1
while IFS= read -r row; do
row=$(trim "$row")
[ -n "$row" ] || continue
Expand All @@ -322,7 +331,7 @@ nm_runs_status_for_branch() { # <branch>
rest=$(trim "$rest")
br=${rest%% *}
if [ "$br" = "$branch" ]; then
printf '%s' "$st"
COARSE_STATUS=$st
return 0
fi
done <<< "$out"
Expand All @@ -340,28 +349,68 @@ HAVE_RUN=0
# run-step block below skips the TOON field parsing entirely for this crew.
RUN_SOURCE=full
COARSE_STATUS=""
NM_RUNS_RESPONDED=0
# Bounded retry/backoff around the run-attribution lookup. During an actively
# running pipeline the bounded no-mistakes calls can lose a race - time out to
# empty (the CLI is busy serving the run) - and leave HAVE_RUN=0. The caller
# would then fall through to the possibly-stale status log, which defeats the
# watcher's provably-working absorption and surfaces a validating crew as
# stale every poll (per-minute false stale wakes during long validations).
# Re-attempt a few times with a short backoff before accepting a
# non-authoritative verdict. Only an unresponsive attempt is retried, and an
# empty result from EITHER bounded call - `axi status` or the coarse
# `no-mistakes runs` list - is that timeout/race signature, because both
# commands emit non-empty text whenever the CLI is alive (`runs` prints a "no
# runs" line even with zero runs). An attempt is authoritative (no retry) only
# when every call that left HAVE_RUN=0 actually answered: `axi status`
# reported another branch's run AND the coarse runs list answered with no row
# for this branch. That definitive "no run for this branch" breaks the loop
# without retrying, because retrying cannot change an authoritative answer and
# would only add hot-path latency for the steady-state implementing crew
# (branch created, validation not yet started).
# This is the upstream replacement for the local state/.crew-state-retry.sh
# (FM_CREW_STATE_BIN) production mitigation, which wrapped this whole helper
# with the same retry-until-run-step/pane semantics.
# With FM_CREW_STATE_RETRIES=0 the loop runs exactly once, byte-identical to the
# original single-attempt behavior (used by the test suite to stay fast).
CREW_STATE_RETRIES=${FM_CREW_STATE_RETRIES:-2}
case "$CREW_STATE_RETRIES" in ''|*[!0-9]*) CREW_STATE_RETRIES=2 ;; esac
CREW_STATE_RETRY_DELAY=${FM_CREW_STATE_RETRY_DELAY:-2}
case "$CREW_STATE_RETRY_DELAY" in ''|*[!0-9]*) CREW_STATE_RETRY_DELAY=2 ;; esac
# Scouts and secondmates never drive a no-mistakes validation of their own
# worktree, so skip the lookup for them and read state from pane/log directly.
if [ "$KIND" = ship ] && [ -n "$CREW_BRANCH" ] && command -v no-mistakes >/dev/null 2>&1; then
RUN_OUT=$(nm_run axi status)
if [ -n "$RUN_OUT" ]; then
run_branch=$(strip_quotes "$(nm_field branch)")
if [ -n "$run_branch" ] && [ "$run_branch" = "$CREW_BRANCH" ]; then
HAVE_RUN=1
else
# The active-or-most-recent run is for another branch (the CLI is alive
# and answered; only the attribution missed) - try the coarse fallback.
# Deliberately nested inside `[ -n "$RUN_OUT" ]`: an empty/timed-out
# primary call means the CLI itself did not respond, so retrying it
# immediately with a second bounded call would just double the wait
# for no better answer.
COARSE_STATUS=$(nm_runs_status_for_branch "$CREW_BRANCH")
if [ -n "$COARSE_STATUS" ]; then
attempt=0
while : ; do
cli_responded=0
RUN_OUT=$(nm_run axi status)
if [ -n "$RUN_OUT" ]; then
cli_responded=1
run_branch=$(strip_quotes "$(nm_field branch)")
if [ -n "$run_branch" ] && [ "$run_branch" = "$CREW_BRANCH" ]; then
HAVE_RUN=1
RUN_SOURCE=coarse
else
# The active-or-most-recent run is for another branch (the CLI is alive
# and answered; only the attribution missed) - try the coarse fallback.
# Deliberately nested inside `[ -n "$RUN_OUT" ]`: an empty/timed-out
# primary call means the CLI itself did not respond within this attempt,
# so the coarse re-query would just double the wait; the retry loop below
# is what gives the CLI another chance after a backoff instead.
nm_runs_status_for_branch "$CREW_BRANCH"
if [ -n "$COARSE_STATUS" ]; then
HAVE_RUN=1
RUN_SOURCE=coarse
elif [ "$NM_RUNS_RESPONDED" = 0 ]; then
cli_responded=0
fi
fi
fi
fi
[ "$HAVE_RUN" = 1 ] && break
[ "$cli_responded" = 0 ] || break
[ "$attempt" -lt "$CREW_STATE_RETRIES" ] || break
attempt=$((attempt + 1))
[ "$CREW_STATE_RETRY_DELAY" -gt 0 ] && sleep "$CREW_STATE_RETRY_DELAY"
done
fi

# --- run-step authoritative path -------------------------------------------
Expand Down
16 changes: 13 additions & 3 deletions bin/fm-spawn.sh
Original file line number Diff line number Diff line change
Expand Up @@ -785,11 +785,21 @@ if [ "$KIND" != secondmate ] && [ "$BACKEND" != orca ]; then
spawn_send_text_line "$T" 'treehouse get'

# Wait for the treehouse subshell: the pane's cwd moves from the project to the worktree.
# Compare CANONICALIZED paths: the backend reports the pane's real cwd (e.g. tmux's
# #{pane_current_path} resolves symlinks), while PROJ_ABS was built with `pwd` and can
# still carry a symlink component (on macOS /tmp -> /private/tmp, /var -> /private/var).
# Comparing the raw strings would then read as "already moved" while the pane is still
# at the project, setting WT to the project dir and tripping validate_spawn_worktree's
# isolation check. Canonicalize both sides so the loop tracks a genuine cwd change.
PROJ_ABS_REAL=$(cd "$PROJ_ABS" 2>/dev/null && pwd -P) || PROJ_ABS_REAL=$PROJ_ABS
for _ in $(seq 1 60); do
p=$(spawn_current_path "$T" || true)
if [ -n "$p" ] && [ "$p" != "$PROJ_ABS" ]; then
WT="$p"
break
if [ -n "$p" ]; then
p_real=$(cd "$p" 2>/dev/null && pwd -P) || p_real=$p
if [ "$p_real" != "$PROJ_ABS_REAL" ]; then
WT="$p"
break
fi
fi
sleep 1
done
Expand Down
2 changes: 2 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,8 @@ FM_CHECK_INTERVAL=300 # seconds between slow checks (merge polls or the X-mode
FM_CHECK_TIMEOUT=30 # seconds allowed per slow check script
FM_CREW_STATE_NM_TIMEOUT=10 # seconds allowed per no-mistakes query inside fm-crew-state.sh
FM_CREW_STATE_RUNS_LIMIT=200 # recent no-mistakes runs rows scanned when cross-branch attribution falls back from axi status
FM_CREW_STATE_RETRIES=2 # extra run-lookup attempts after a bounded no-mistakes call (axi status or the coarse runs list) times out to empty; 0 = single attempt (fully answered no-run verdicts never retry)
FM_CREW_STATE_RETRY_DELAY=2 # seconds slept between those run-lookup retry attempts
FM_CREW_STATE_BIN=bin/fm-crew-state.sh # test override for the current-state reader used by provably-working watcher triage
FMX_PAIRING_TOKEN= # X mode pairing token; .env opt-in authorizes replies and eligible lifecycle actions
FMX_RELAY_URL=https://myfirstmate.io # optional X relay override, mainly for local relay development
Expand Down
6 changes: 3 additions & 3 deletions docs/herdr-backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ You do not need to attach for routine supervision: `bin/fm-peek.sh fm-<id>` read

Verify it works by spawning a trivial task with `--backend herdr` and confirming the task's meta records `backend=herdr` plus `herdr_session=`, `herdr_workspace_id=`, `herdr_tab_id=`, and `herdr_pane_id=`; the workspace for your home should show the new `fm-<id>` tab.

Limitations: herdr is experimental, not yet used for `bin/fm-bootstrap.sh`'s required-tools list (the version/tool gate happens at spawn time instead), and carries the known gaps documented below (a small-`--lines` capture bug with a built-in workaround, and a `pane_cwd`-adjacent worktree-discovery symlink fragility) - see "Known gaps and follow-up notes" at the end of this document.
Limitations: herdr is experimental, not yet used for `bin/fm-bootstrap.sh`'s required-tools list (the version/tool gate happens at spawn time instead), and carries the known gaps documented below (a small-`--lines` capture bug with a built-in workaround) - see "Known gaps and follow-up notes" at the end of this document.

## Status: experimental

Expand Down Expand Up @@ -376,8 +376,8 @@ This is a test-harness-only concern - `fm_backend_herdr_composer_state` and `fm_
A genuine `events.subscribe`-driven push is a reasonable follow-up, not implemented here.
- **`bin/fm-bootstrap.sh`'s required-tools list is unchanged.** It still unconditionally requires `tmux`, and does not yet conditionally add `herdr` and `jq` when a backend selection resolves to herdr.
The version/tool gate happens at spawn time instead and refuses loudly, so this is bootstrap-detection polish, not a functional gap.
- **Worktree-discovery isolation guard is symlink-fragile for a project path under a symlinked prefix (e.g. macOS's `/tmp` -> `/private/tmp`).** Discovered while building the runtime-backend-auto-detection real smoke test (`tests/fm-backend-autodetect-smoke.test.sh`), which needed a scratch project. `fm-spawn.sh`'s `PROJ_ABS` is a LOGICAL `cd && pwd` (symlink components kept), while herdr's `foreground_cwd` (and real tmux's `pane_current_path`, on the same OS-level cwd primitive) report the PHYSICALLY resolved path.
When the project itself lives under a symlinked directory, the very first worktree-discovery poll sees two different strings for the identical starting directory and the isolation guard false-refuses the spawn as "not isolated" before `treehouse get` ever moves the pane - backend-agnostic, not specific to herdr. Worked around in the test by resolving its scratch `TMP_ROOT` through `pwd -P` before use; the underlying `fm-spawn.sh` path-comparison gap (worth resolving `PROJ_ABS` physically, or comparing physically-resolved forms in the isolation guard) is unfixed and worth a dedicated follow-up.
- **RESOLVED: worktree-discovery isolation guard was symlink-fragile for a project path under a symlinked prefix (e.g. macOS's `/tmp` -> `/private/tmp`).** Discovered while building the runtime-backend-auto-detection real smoke test (`tests/fm-backend-autodetect-smoke.test.sh`), which needed a scratch project. `fm-spawn.sh`'s `PROJ_ABS` is a LOGICAL `cd && pwd` (symlink components kept), while herdr's `foreground_cwd` (and real tmux's `pane_current_path`, on the same OS-level cwd primitive) report the PHYSICALLY resolved path.
When the project itself lived under a symlinked directory, the very first worktree-discovery poll saw two different strings for the identical starting directory and the isolation guard false-refused the spawn as "not isolated" before `treehouse get` ever moved the pane - backend-agnostic, not specific to herdr. Fixed upstream: `fm-spawn.sh`'s wait loop now canonicalizes both sides (`pwd -P`) before comparing, so the loop only breaks on a genuine cwd move; regression-covered by `tests/fm-spawn-worktree-symlink.test.sh`. The autodetect smoke test's own `pwd -P` resolution of its scratch `TMP_ROOT` remains as belt-and-braces.
- **RESOLVED: a restart's restored-layout husk no longer needs a manual pane close before respawn.** See "Respawn idempotency: a restored task tab is a husk, not a duplicate" above for the fix (`fm_backend_herdr_pane_agent_state`, `fm_backend_herdr_create_task`'s close-and-replace).
Left over from that fix: the `dead` (`pane_not_found`) husk classification is exercised only at the unit level, never against the real binary - killing a pane's process on a live server was observed to make herdr reap the whole tab immediately (never leaving a dead-but-still-listed pane for the duplicate check to find), and a real session restart was never observed to produce one either.
It remains a conservative, defensively-coded path for a herdr failure mode (e.g. a restored process that fails to start) nobody has reproduced against the real binary yet.
Loading