From 631e7ca66038295d12a37dfddf840ca94bb10bd8 Mon Sep 17 00:00:00 2001 From: Bchue Date: Wed, 5 Aug 2026 21:47:16 -0400 Subject: [PATCH 1/8] fix(tests): contain and reap test fixtures instead of leaking them into /tmp /tmp is a tmpfs on some hosts, so a fixture the suite leaves behind is leaked RAM. Fixtures accumulated without bound: most test files installed no EXIT trap, and the ones that did still lost their fixtures whenever a run was killed. On one host that reached 3.0 GB of RAM held by dead fixtures. Fix it in the runner, which is the one place that already sits at both the start and the end of every run, rather than in each test file. - Every executed script now runs with TMPDIR, TMP, and FM_TASK_TMP_ROOT pointed at a private directory inside the run's own root, so fixtures built the way the suite builds them land inside the run. Each script's root is removed as soon as that script finishes, so peak usage stays at one script's fixtures. - The run root is removed on a normal exit and on INT, TERM, HUP, and QUIT, and the runner then dies from the same signal so callers still see a signal death. - An executing run first reaps fixture directories orphaned by runs that were killed outright, so an already-leaked host heals on the next run. The reap keeps anything it cannot establish: only direct children of the reap root, only mktemp-shaped fm- names, only directories this user owns, only entries past a minimum age, never a recorded tasktmp= path or a per-task scratch root, never a directory a process still holds open, and nothing at all when the in-use inventory or a home's task records cannot be read. - bin/fm-spawn.sh roots per-task scratch at FM_TASK_TMP_ROOT, which is unset in normal operation so the real path is unchanged. A test that drives a real spawn previously stranded a /tmp/fm-/ that no teardown would ever claim. Behavioural coverage in tests/fm-test-run.test.sh proves a completed run, an interrupted run, and a killed run each leave no fixture behind, that a later run reaps pre-existing orphans, and that a live task's recorded scratch directory and a directory another process holds open are never removed. --- CONTRIBUTING.md | 2 + bin/fm-spawn.sh | 8 +- bin/fm-test-run.sh | 342 ++++++++++++++++++++++++++++++++- tests/fm-backend-orca.test.sh | 4 +- tests/fm-backend.test.sh | 8 +- tests/fm-gotmp.test.sh | 2 +- tests/fm-kimi-harness.test.sh | 2 +- tests/fm-session-start.test.sh | 4 +- tests/fm-test-run.test.sh | 331 +++++++++++++++++++++++++++++++ tests/lib.sh | 29 ++- 10 files changed, 701 insertions(+), 31 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7876f7cbdb..c6da486673 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -97,6 +97,8 @@ Family selection is the ordinary local path; `--all` is deliberate full regressi CI owns broad regression across required portable parallel shards, the portable serial lane, the Herdr lane, lint, invariants, the coverage guard, and stock macOS Bash compatibility in [`.github/workflows/ci.yml`](.github/workflows/ci.yml). Use `bin/fm-test-run.sh --help` for lane names, `--jobs` rules, and required gate-skip flags when reproducing a lane locally. Discover tests by listing `tests/*.test.sh`: each is a self-contained bash script named `.test.sh`, and its header comment describes what it covers, so pass one to `bin/fm-test-run.sh` to focus on a subject with canonical timing output. +The runner is also the single owner of fixture cleanup: it gives every executed script a private `TMPDIR` and removes it when that script finishes, on an interrupted run as well as a normal one, and it reaps fixture directories orphaned by earlier killed runs, so a new test needs no temp-root `EXIT` trap of its own. +That matters because `/tmp` is a tmpfs on some hosts, where a leaked fixture is leaked RAM rather than disk; the runner's header owns the reap's safety rules and its test-only environment seams. Tests that need a real optional backend or an explicit opt-in (real herdr/zellij/cmux smoke tests, the live Pi regression) skip themselves and print the tool or environment gate needed to enable them, so the portable suite remains safe on machines without those tools. The [Herdr backend guide](docs/herdr-backend.md#destructive-lab-safety) owns the lane's isolation boundary, while [runtime backend verification](docs/verification/runtime-backends.md#herdr) owns active empirical evidence; live harness credential tests remain opt-in. diff --git a/bin/fm-spawn.sh b/bin/fm-spawn.sh index 6cdf9879ac..4d70237013 100755 --- a/bin/fm-spawn.sh +++ b/bin/fm-spawn.sh @@ -1325,7 +1325,13 @@ fi # Nested (not a bare /tmp/fm-/gotmp) so other per-task temp can live alongside # later, and teardown cleans one deterministic path. GOTMPDIR (not TMPDIR) is the # targeted knob: TMPDIR is too broad (affects every program's temp, not just Go's). -TASK_TMP="/tmp/fm-$ID" +# +# FM_TASK_TMP_ROOT relocates that root and is unset in normal operation, so the +# real path is unchanged. bin/fm-test-run.sh sets it to the executed script's +# private temp root: a test that drives a real spawn would otherwise leave a +# /tmp/fm-/ behind with no task and no teardown to remove it, and on +# a tmpfs /tmp that residue is held in RAM. +TASK_TMP="${FM_TASK_TMP_ROOT:-/tmp}/fm-$ID" mkdir -p "$TASK_TMP/gotmp" # Per-harness turn-end hook where enabled: a file that touches diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index 973032d43d..bec9ef6c1c 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -43,10 +43,48 @@ # families never schedule under --jobs. # -h, --help print this header # +# Fixture containment: +# Every executed script runs with TMPDIR and TMP pointed at a private +# directory inside this run's own root, so a fixture built the way the suite +# builds them - mktemp -d "${TMPDIR:-/tmp}/fm-.XXXXXX", directly or +# through tests/lib.sh fm_test_tmproot - lands inside the run instead of +# beside it. Each script's private root is removed as soon as that script +# finishes, and the run root is removed on normal exit and on INT, TERM, HUP, +# or QUIT. This is the single owner of fixture cleanup: no test file needs its +# own EXIT trap for temp roots, and an interrupted run cleans up too. /tmp is +# a tmpfs on some hosts, where a leaked fixture is leaked RAM, not disk. +# FM_TASK_TMP_ROOT is pointed at the same private directory, so a test that +# drives a real bin/fm-spawn.sh keeps its per-task scratch root inside the run +# instead of leaving a /tmp/fm-/ that no teardown will ever claim. +# +# Orphan reap: +# An executing run first reaps fixture directories orphaned by earlier runs +# that were killed outright (SIGKILL leaves no trap to run), so a host that +# already leaked heals on the next test run. The reap is deliberately narrow +# and keeps anything it cannot establish: +# - only direct children of the reap root, never a wider glob; +# - only names shaped like an mktemp fixture, fm-.<6+ alphanumerics>; +# - only directories (never symlinks) owned by the current user; +# - only entries older than the minimum age, so a concurrent run is safe; +# - never a path recorded as tasktmp= by any discoverable Firstmate home, +# and never a per-task scratch root (identified by its gotmp/ child); +# - never a directory any process still holds open; +# - nothing at all if the in-use inventory or a home's task records cannot +# be read. +# FM_TEST_REAP_ROOT (default /tmp), FM_TEST_REAP_MIN_AGE_SECONDS (default +# 900), FM_TEST_REAP_MAX (default 500), and FM_TEST_REAP_HOMES (extra +# colon-separated Firstmate homes whose state/*.meta to read) exist so this +# behavior is testable without touching the host's real /tmp. +# FM_TEST_RUN_ACTIVE=1 is exported to every executed script and suppresses the +# reap in a nested run. +# # Per-script machine-parseable markers (stdout): # FM_TEST_BEGIN