feat(heavy-run): land host-global ledger, release nudge, and cached-critical guard - #69
Open
yjuyjuy wants to merge 2 commits into
Open
feat(heavy-run): land host-global ledger, release nudge, and cached-critical guard#69yjuyjuy wants to merge 2 commits into
yjuyjuy wants to merge 2 commits into
Conversation
added 2 commits
August 5, 2026 08:27
…ritical guard
The heavy-run serialization point capped concurrent heavy runs per operational
home, but the machine is the resource being protected and a fleet runs a primary
home plus one or more secondmate homes on one host. A per-home queue gave each
home its own N slots, so the host-wide count was N times the number of homes -
the exact multiplication the cap exists to prevent. This lands the three ratified
deltas on top of the already-landed base wrapper and refuse-by-default posture.
DELTA 2, host-global ledger. The default queue moves to a fixed path outside any
home, ${TMPDIR:-/tmp}/fm-heavy-runs-<uid>, one directory per operating user so a
shared host neither cross-contaminates nor can be hijacked through a world-
writable /tmp. Every home's runs share one running count. Each lease record gains
a home= attribution field, surfaced by --status, so firstmate can see which home
owns a shared-ledger run. The ceiling value resolves from FM_HEAVY_SLOTS_FILE
(the primary home's config/heavy-run-slots) when set and readable, so homes
sharing the ledger share one cap, falling back to the local config then to 1.
FM_HEAVY_RUN_DIR still overrides the path, which is how the tests isolate.
DELTA 3, release-wake nudge. When a run that held a slot releases it and a waiter
is still queued, the releasing process enqueues one check heavy-run-slot-free
wake, so firstmate nudges a crewmate parked on `paused: awaiting test slot` to
retry. Firstmate is the nudger, never the granter: the waiter re-acquires through
ordinary admission, and there is no FIFO ticket queue. The nudge runs from the
exit trap, lock-free to avoid deadlocking against a lock the same process may
still hold, and a failed enqueue is swallowed so it never turns a clean command
exit into a failure.
DELTA 4, cached-critical guard. Admission reads the resource monitor's cached
sustained verdict (state/.resource-status) and refuses a free slot (exit 76)
while that verdict is a fresh critical, because starting new heavy work into a
thrashing host is the second failure mode this control prevents. It reads the
published cache only and never samples afresh, honouring the sustained-sampling
rule, and fails open on an absent, stale, unreadable, or non-critical verdict so
a home with no resource monitor is never wedged.
ADRs 0001 and 0002 record why admission refuses by default through a lease
wrapper and why the ledger deliberately lives outside home isolation.
Tests: 7 new cases cover the host-global default path, home attribution, the
authoritative ceiling pointer and its fallback, the release nudge firing and not
firing, the critical refusal, and the fail-open behaviour. The suite pins
FM_HEAVY_RUN_DIR so it isolates from the real host-global ledger. 21/21 pass;
bin/fm-lint.sh clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Land the three outstanding ratified deltas of the heavy-run admission controller (bin/fm-heavy-run.sh). The base lease-wrapper and DELTA 1 (refuse-by-default with the max-wait opt-in) already landed; do NOT re-add them.
DELTA 2 host-global ledger: the default queue moves from a per-home path to a fixed host-global path outside any home (TMPDIR fallback slash-tmp, one directory per operating uid so a shared host cannot cross-contaminate or be hijacked via a world-writable tmp), because the machine is the protected resource and a per-home queue multiplies slots across secondmate homes. Lease records gain a home attribution field shown by --status. The ceiling resolves from FM_HEAVY_SLOTS_FILE (the primary home's config/heavy-run-slots) when set and readable so homes sharing the ledger share one cap, falling back to the local config then to 1.
DELTA 3 release-wake nudge: freeing a held slot with a waiter queued enqueues one check heavy-run-slot-free wake so firstmate nudges a paused waiter to retry; firstmate is the nudger never the granter; runs lock-free from the exit trap and swallows a failed enqueue so it never fails a clean command exit.
DELTA 4 cached-critical guard: admission reads the watcher's already-published cached sustained verdict state/.resource-status and refuses a free slot with exit 76 on a fresh critical, reading the cache only (no fresh sampling, honouring the sustained-sampling rule) and failing OPEN on absent, stale, unreadable, or non-critical so a home with no monitor is never wedged.
Two ADRs added under docs/adr/ (refuse-by-default lease-wrapper admission; host-global ledger exception to home isolation). docs/configuration.md owns the config knobs and the new FM_HEAVY env vars; AGENTS.md carries pointers only, no duplicated contract.
7 new colocated tests in tests/fm-heavy-run.test.sh; the suite pins FM_HEAVY_RUN_DIR to isolate from the real host-global ledger; 21 of 21 pass and bin/fm-lint.sh is clean.
Deliberately out of scope: retiring the stagger-decision-releases stopgap and relabelling the 6-plus-6 crew cap live only in the primary home's gitignored data/captain.md (already relabelled there) and have no tracked-material representation.
What Changed
/tmp, one directory per operating uid), with ownership/symlink hardening ([ -O "$HEAVY_DIR" ]and no-symlink checks) added on top after review flagged a hijack risk on shared hosts with a world-writable tmp.--status; the slot ceiling resolves fromFM_HEAVY_SLOTS_FILE(primary home'sconfig/heavy-run-slots) when set and readable, falling back to the local config, then to 1.check heavy-run-slot-freewake from the exit trap, swallowing enqueue failures so it never fails a clean command exit.state/.resource-status) and refuses a free slot with exit 76 on a fresh critical, using cache-only reads and failing open on absent/stale/unreadable/non-critical state.docs/adr/0001-heavy-run-refuse-by-default-admission.md,docs/adr/0002-heavy-run-host-global-ledger.md), updateddocs/configuration.mdwith the new FM_HEAVY env vars, and added pointer-only references inAGENTS.md.tests/fm-heavy-run.test.sh(pinningFM_HEAVY_RUN_DIRfor isolation), bringing the suite to 22 passing.Risk Assessment
✅ Low: The prior ownership/symlink hijack gap is now fully closed with -O/!-L/-w assertions on both ledger root and queue subdir plus a colocated refusal test; remaining deltas were already verified correct.
Testing
Executed the full tests/fm-heavy-run.test.sh suite (self-isolating via FM_HEAVY_RUN_DIR) which exercises DELTA 2/3/4 behavior end-to-end via subprocess invocations of bin/fm-heavy-run.sh; all 22 tests passed (0 failures). Lint was intentionally skipped per this phase's scope rules (linters/static analysis are excluded from the test phase and owned elsewhere).
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed ✅
bin/fm-heavy-run.sh:320- ensure_queue applies umask 077 only when it creates $HEAVY_DIR; if the dir already exists it verifies only -d/!-L(on $QUEUE)/-w, never ownership. On a shared host with sticky world-writable /tmp, another uid can pre-create /tmp/fm-heavy-runs-<victim-uid> (name is the guessable victim uid) and the victim then runs against an attacker-owned ledger — reading lease records, forging/starving the queue, and tampering with admit.lock/seq. $HEAVY_DIR itself is also never !-L-checked, so a symlinked HEAVY_DIR redirects LOCK/SEQ_FILE. This defeats the DELTA 2 intent claim that per-uid naming prevents hijack via a world-writable tmp. Fix: assert[ -O "$HEAVY_DIR" ] && [ ! -L "$HEAVY_DIR" ](and refuse otherwise) before using the ledger.🔧 Fix: harden heavy-run ledger against unowned/symlinked dir hijack
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
bash tests/fm-heavy-run.test.sh (full run, isolated via FM_HEAVY_RUN_DIR pinning) — 22 ok, 0 not ok, exit 0Confirmed coverage of DELTA 2: 'the default ledger is host-global, not per-home', 'a lease record carries its owning home for attribution', 'the authoritative ceiling pointer wins, and falls back safely'Confirmed coverage of DELTA 3: 'slot release nudges a queued waiter with a check wake', 'no waiter means no release nudge'Confirmed coverage of DELTA 4: 'a free slot is refused under sustained critical pressure', 'the pressure guard fails open on an absent or stale verdict'✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.