From 3e62ae8b258e94aca5afe73cb68f6f16c4e394da Mon Sep 17 00:00:00 2001 From: Simon Keimer Date: Fri, 25 Sep 2026 08:57:16 +0200 Subject: [PATCH] chore(gate): queue behind other heavy builds on the host, not race them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing coordinates two heavy cargo runs on one machine. On 2026-09-25 a neighbouring project's gate (fnec-rust) was OOM-killed three times while this repo's `cargo test --workspace` ran alongside it — and every kill read as a failure of the code under test, which is the misattribution `GATE: INVALID` exists to prevent, arriving by a channel this gate cannot see. `scripts/lib/host-build-lock.sh` takes a HOST-WIDE flock, held until the calling script exits. `scripts/gate.sh` and the pre-push hook each source it once. The path is shared with fnec-rust's `scripts/host-build-lock.sh`, so the two projects' gates queue instead of contending for RAM; `HEAVY_BUILD_LOCK` overrides it, and every project must agree. Placement is deliberate: - gate.sh takes it right after argument parsing and BEFORE `START_HEAD`, so time spent waiting is not part of the run and the drift guard's baseline is taken after the wait. - `--fingerprint` is exempt — the seconds-long primitive builds nothing. - the pre-push hook takes it AFTER the `GATE_SKIP` escape hatch, so a deliberate skip never waits. - neither calls the other, so both may take it without deadlocking. Verified without building (a build in this worktree would put a target dir inside it): with a test lock held, `scripts/gate.sh` and the pre-push hook each printed "waiting for it to finish" and were stopped by `timeout` before any cargo step (exit 124); `--fingerprint` returned in 1.2 s against a 6 s hold. NOT verified: this repo's rule to sabotage-verify gate.sh through a full run after every edit. A full gate is ~2 h and this checkout's sibling was running the `ota_channel_adaptation` acceptance suite at the time; running a second suite beside it is the exact contention this change exists to stop. Run `scripts/gate.sh` on a quiet machine before merging. Verification-objective: two heavy cargo runs on one host serialize instead of racing for RAM — observed as a gate and a pre-push hook each blocking on a held host lock, and the exempt fingerprint mode not blocking Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_018p7FxX7QMWNJVNp9LbaLkB --- .cargo-husky/hooks/pre-push | 5 +++++ scripts/gate.sh | 9 +++++++++ scripts/lib/host-build-lock.sh | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+) create mode 100644 scripts/lib/host-build-lock.sh diff --git a/.cargo-husky/hooks/pre-push b/.cargo-husky/hooks/pre-push index 60da928f..bdcc15e1 100755 --- a/.cargo-husky/hooks/pre-push +++ b/.cargo-husky/hooks/pre-push @@ -32,6 +32,11 @@ if [ "${GATE_SKIP:-0}" = "1" ]; then exit 0 fi +# Queue behind any other heavy build on this machine, in any project, rather than +# race it for RAM. After the escape hatch, so a deliberate skip never waits. +# shellcheck source=scripts/lib/host-build-lock.sh +source "$(git rev-parse --show-toplevel)/scripts/lib/host-build-lock.sh" "pre-push" + cargo fmt --all -- --check # Crates owning the changed files. Falls back to the whole workspace when the upstream ref is diff --git a/scripts/gate.sh b/scripts/gate.sh index 19a6b684..9295ee65 100755 --- a/scripts/gate.sh +++ b/scripts/gate.sh @@ -62,6 +62,15 @@ case "${1:-}" in *) echo "unknown argument: $1" >&2; exit 2 ;; esac +# Queue behind any other heavy build on this machine, in any project, before +# anything is timed or snapshotted — so time spent waiting is not part of the run, +# and START_HEAD below is taken after the wait. `--fingerprint` is exempt: it is +# the seconds-long primitive and builds nothing. +if [ "$MODE" != "fingerprint" ]; then + # shellcheck source=scripts/lib/host-build-lock.sh + source "$REPO_ROOT/scripts/lib/host-build-lock.sh" "gate" +fi + COMMIT=$(git rev-parse HEAD 2>/dev/null || echo "unknown") if [ -n "$(git status --porcelain 2>/dev/null)" ]; then DIRTY="dirty"; else DIRTY="clean"; fi diff --git a/scripts/lib/host-build-lock.sh b/scripts/lib/host-build-lock.sh new file mode 100644 index 00000000..99d02b6b --- /dev/null +++ b/scripts/lib/host-build-lock.sh @@ -0,0 +1,33 @@ +# Take the HOST-WIDE heavy-build lock, held until the calling script exits. +# Sourced, not executed: `source scripts/lib/host-build-lock.sh ""`. +# +# Why host-wide and not per repo. Nothing coordinates two heavy cargo runs on one +# machine. On 2026-09-25 another project's gate (fnec-rust) was OOM-killed three +# times while this repo's `cargo test --workspace` ran alongside it, and each kill +# read as a failure of the code under test — the same misattribution +# `GATE: INVALID` exists to prevent, arriving by a channel the gate cannot see. +# The lock file is shared across projects: fnec-rust's `scripts/host-build-lock.sh` +# takes the SAME path, so the two gates queue instead of racing for RAM. +# +# Two rules that keep it from deadlocking: +# - a script takes it ONCE, near the top; nothing it calls takes it again +# (the pre-push hook does not call gate.sh, so each may take it); +# - do not wrap a script that takes it in `flock` yourself — the inner take +# would wait forever on the lock the outer one holds. +# +# Override the path with HEAVY_BUILD_LOCK (every project must agree on it). If +# `flock` is not installed, the caller runs unlocked and says so. + +_hbl_who="${1:-gate}" +_hbl_path="${HEAVY_BUILD_LOCK:-${XDG_RUNTIME_DIR:-/tmp}/heavy-build.lock}" + +if command -v flock >/dev/null 2>&1; then + exec 9>"$_hbl_path" + if ! flock -n 9; then + echo "$_hbl_who: another heavy build holds $_hbl_path — waiting for it to finish" >&2 + flock 9 + echo "$_hbl_who: lock acquired, continuing" >&2 + fi +else + echo "$_hbl_who: flock not installed — running WITHOUT the host-wide build lock" >&2 +fi