diff --git a/.agents/skills/bootstrap-diagnostics/SKILL.md b/.agents/skills/bootstrap-diagnostics/SKILL.md index ddd0f26fc5..5f517a49a2 100644 --- a/.agents/skills/bootstrap-diagnostics/SKILL.md +++ b/.agents/skills/bootstrap-diagnostics/SKILL.md @@ -46,8 +46,9 @@ When any diagnostic needs captain attention, report the plain consequence and re Keep each affected poll unavailable, inspect the named private state path, and do not bypass the migration or execute a quarantined artifact; a completed safe-scan marker allows unrelated authenticated polls to continue while private repair remains pending. - `SCRATCH_SWEEP: : skipped: ` - the temp-root scratch sweep found a stale orphaned scratch directory but deliberately left it in place, either because removal failed or because it does not look like scratch at all (`looks like a firstmate home` means a home is sitting in the temp root and must be moved or removed by hand, never swept). The shared temp filesystem is small and every home on the host writes into it, so a leak that keeps being skipped eventually fills it and makes every command on the machine fail silently on temp writes; clear the named directory yourself once you have confirmed nothing is using it. -- `SCRATCH_SWEEP: : skipped: ` - the sweep could not run to completion for the whole root: either no open-handle check was available (install `lsof`, or check why procfs or the temp root is unreadable) or the sweep's time budget ran out with candidates left. +- `SCRATCH_SWEEP: : skipped: ` - the sweep could not run to completion for the whole root: no open-handle check was available (install `lsof`, or check why procfs or the temp root is unreadable), the sweep's time budget ran out with candidates left, or a firstmate home's records could not be read so the sweep refused before examining anything. A budget line is self-healing across sessions and needs attention only if the deferred count keeps growing; a missing open-handle check disables reclamation entirely and does need fixing. + An unreadable-records line means nothing at all was reclaimed this session, so never report a completed sweep after one: repair the records the reason names (a home's task records under `state/`, or the home directory or secondmate registry that decides which homes are read) rather than clearing the temp root by hand, because a live task's recorded scratch is exactly what could not be ruled out. - `SECONDMATE_SYNC: secondmate : skipped: ` - the local-HEAD secondmate sync left a live secondmate home on its existing checkout because the home was dirty, diverged, unsafe, on the wrong branch, missing the primary target commit, or otherwise not fast-forwardable, or because inherited local-material propagation failed; bootstrap continued, but inspect the reason because the secondmate's tracked instructions, inherited settings, or shared captain preferences may be stale after a primary update. - `SECONDMATE_LIVENESS: secondmate : skipped: |respawn failed after : ` - the session-start liveness sweep could not guarantee that the registered secondmate is running a real agent process. Investigate the reason because that secondmate is not guaranteed live. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7876f7cbdb..10f48709b7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -97,6 +97,9 @@ Family selection is the ordinary local path; `--all` is deliberate full regressi CI owns broad regression across required portable parallel shards, the portable serial lane, the Herdr lane, lint, invariants, the coverage guard, and stock macOS Bash compatibility in [`.github/workflows/ci.yml`](.github/workflows/ci.yml). Use `bin/fm-test-run.sh --help` for lane names, `--jobs` rules, and required gate-skip flags when reproducing a lane locally. Discover tests by listing `tests/*.test.sh`: each is a self-contained bash script named `.test.sh`, and its header comment describes what it covers, so pass one to `bin/fm-test-run.sh` to focus on a subject with canonical timing output. +The runner is also the single owner of fixture cleanup: it gives every executed script a private `TMPDIR` and removes it when that script finishes, on an interrupted run as well as a normal one, and it reaps fixture directories orphaned by earlier killed runs, so a new test needs no temp-root `EXIT` trap of its own. +That matters because `/tmp` is a tmpfs on some hosts, where a leaked fixture is leaked RAM rather than disk. +`bin/fm-tmp-sweep.sh` is the single owner of which orphaned scratch directories are safe to remove, for both the runner's reap and the session-start sweep, and the runner's header owns its own test-only environment seams. Tests that need a real optional backend or an explicit opt-in (real herdr/zellij/cmux smoke tests, the live Pi regression) skip themselves and print the tool or environment gate needed to enable them, so the portable suite remains safe on machines without those tools. The [Herdr backend guide](docs/herdr-backend.md#destructive-lab-safety) owns the lane's isolation boundary, while [runtime backend verification](docs/verification/runtime-backends.md#herdr) owns active empirical evidence; live harness credential tests remain opt-in. diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 6cdf9879ac..eb5ccaac53 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -1325,7 +1325,20 @@ fi # Nested (not a bare /tmp/fm-/gotmp) so other per-task temp can live alongside # later, and teardown cleans one deterministic path. GOTMPDIR (not TMPDIR) is the # targeted knob: TMPDIR is too broad (affects every program's temp, not just Go's). -TASK_TMP="/tmp/fm-$ID" +# +# FM_TASK_TMP_ROOT relocates that root and is unset in normal operation, so the +# real path is unchanged. bin/fm-test-run.sh sets it to the executed script's +# private temp root: a test that drives a real spawn would otherwise leave a +# /tmp/fm-/ behind with no task and no teardown to remove it, and on +# a tmpfs /tmp that residue is held in RAM. It must be absolute: this path is +# recorded as tasktmp= and reaches an unguarded rm -rf in bin/fm-teardown.sh, +# which would resolve a relative value against whatever directory teardown +# happens to run in. +case "${FM_TASK_TMP_ROOT:-/tmp}" in + /*) ;; + *) echo "error: FM_TASK_TMP_ROOT must be an absolute path, got '$FM_TASK_TMP_ROOT'" >&2; exit 1 ;; +esac +TASK_TMP="${FM_TASK_TMP_ROOT:-/tmp}/fm-$ID" mkdir -p "$TASK_TMP/gotmp" # Per-harness turn-end hook where enabled: a file that touches diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 973032d43d..db4181fb89 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -43,10 +43,54 @@ # families never schedule under --jobs. # -h, --help print this header # +# Fixture containment: +# Every executed script runs with TMPDIR and TMP pointed at a private +# directory inside this run's own root, so a fixture built the way the suite +# builds them - mktemp -d "${TMPDIR:-/tmp}/fm-.XXXXXX", directly or +# through tests/lib.sh fm_test_tmproot - lands inside the run instead of +# beside it. Each script's private root is removed as soon as that script +# finishes, and the run root is removed on normal exit and on INT, TERM, HUP, +# or QUIT. This is the single owner of fixture cleanup: no test file needs its +# own EXIT trap for temp roots, and an interrupted run cleans up too. /tmp is +# a tmpfs on some hosts, where a leaked fixture is leaked RAM, not disk. +# FM_TASK_TMP_ROOT is pointed at the same private directory, so a test that +# drives a real bin/fm-spawn.sh keeps its per-task scratch root inside the run +# instead of leaving a /tmp/fm-/ that no teardown will ever claim. +# +# Orphan reap: +# An executing run first reaps fixture directories orphaned by earlier runs +# that were killed outright (SIGKILL leaves no trap to run), so a host that +# already leaked heals on the next test run. bin/fm-tmp-sweep.sh is the single +# owner of which directories are safe to remove and of every refusal that +# protects a live task, an in-use directory, or an operational home; this +# runner supplies only the candidate cap a test host wants, the homes this +# checkout can see, and the horizons below. Read that script's header for the +# full contract. +# +# The reap runs as two passes. The first is scoped by name to fm-test-run.*, +# the run roots this runner alone creates, on a short window: SIGKILL runs no +# trap, so the reap is the only thing that reclaims a killed run, and every +# fixture is nested inside that root. The second uses the sweep's own +# conservative window over everything else, because the shared root also holds +# scratch belonging to live firstmate tooling that can sit unwritten far +# longer than a fixture. No tool has to register itself to stay out of the +# short window; it cannot reach a name this runner did not create. +# FM_TEST_REAP_ROOT (default /tmp), FM_TEST_REAP_MIN_AGE_SECONDS (default 900, +# rounded up to whole minutes, and applied only to the first pass), +# FM_TEST_REAP_MAX (default 500), and FM_TEST_REAP_HOMES (extra +# colon-separated Firstmate homes whose state/*.meta to read) exist so this +# behavior is testable without touching the host's real /tmp. +# FM_TEST_RUN_ACTIVE=1 is exported to every executed script and suppresses the +# reap in a nested run. +# # Per-script machine-parseable markers (stdout): # FM_TEST_BEGIN