From 074e0718e3a53992ab5a32d547b057fdb8d8bfb3 Mon Sep 17 00:00:00 2001 From: Simon Keimer Date: Fri, 25 Sep 2026 08:58:16 +0200 Subject: [PATCH] chore(gates): queue behind other heavy builds on the host, not race them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing coordinates two heavy cargo runs on one machine. On 2026-09-25 this repo's gate was OOM-killed three times while another project's `cargo test --workspace` (OpenPulseHF) ran alongside it, and each kill looked like a failure of the code under test. `scripts/host-build-lock.sh` takes a HOST-WIDE flock, held until the calling script exits. `scripts/check-all.sh` and `.githooks/pre-push` each source it once, near the top. The path is shared with OpenPulseHF's `scripts/lib/host-build-lock.sh` (dc0sk/OpenPulseHF#1440), so the two projects' gates queue instead of contending for RAM. `HEAVY_BUILD_LOCK` overrides it, and every project must agree on the value. One helper, sourced by both, rather than the snippet pasted twice — two copies of one decision drifting apart is this repo's dominant defect class. Deadlock rules, stated in the helper: a script takes the lock once; nothing it calls takes it again (pre-push does not call check-all.sh); and nobody wraps a locking script in `flock` by hand, since the inner take would wait forever on the lock the outer one holds. Verified against the real helper: with the lock held by another process, a second take printed "waiting for it to finish" and acquired it 3.5 s later, exactly when the holder released; uncontended, it acquired in 0.01 s. Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_018p7FxX7QMWNJVNp9LbaLkB --- .githooks/pre-push | 7 +++++++ scripts/check-all.sh | 5 +++++ scripts/host-build-lock.sh | 35 +++++++++++++++++++++++++++++++++++ 3 files changed, 47 insertions(+) create mode 100644 scripts/host-build-lock.sh diff --git a/.githooks/pre-push b/.githooks/pre-push index bfaa0a3a..3d743e0d 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -1,6 +1,13 @@ #!/usr/bin/env bash set -euo pipefail +# Queue behind any other heavy build on this machine, in any project — this hook +# runs the whole suite, and a second suite running alongside it OOM-killed the +# gate three times on 2026-09-25. See scripts/host-build-lock.sh. +ROOT="$(git rev-parse --show-toplevel)" +# shellcheck source=scripts/host-build-lock.sh +source "$ROOT/scripts/host-build-lock.sh" "pre-push" + cargo test --workspace cargo audit # License / bans / sources policy (advisories are already covered by cargo audit diff --git a/scripts/check-all.sh b/scripts/check-all.sh index c8ff540e..5bd5f2da 100755 --- a/scripts/check-all.sh +++ b/scripts/check-all.sh @@ -28,6 +28,11 @@ ROOT="$PWD" FAST=0 [[ "${1:-}" == "--fast" ]] && FAST=1 +# Queue behind any other heavy build on this machine, in any project, rather +# than race it for RAM. See the helper for why this is host-wide. +# shellcheck source=scripts/host-build-lock.sh +source "$ROOT/scripts/host-build-lock.sh" "check-all" + FAILED=() run() { local name="$1"; shift diff --git a/scripts/host-build-lock.sh b/scripts/host-build-lock.sh new file mode 100644 index 00000000..c3eba5aa --- /dev/null +++ b/scripts/host-build-lock.sh @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: GPL-3.0-only +# Copyright (C) 2026 Simon Keimer (DC0SK) +# +# Take the HOST-WIDE heavy-build lock, held until the calling script exits. +# Sourced, not executed: `source scripts/host-build-lock.sh ""`. +# +# Why host-wide and not per repo. A full `cargo test --workspace` here peaks at +# several GB while it links, and nothing coordinates two of them on one machine. +# On 2026-09-25 this repo's gate was OOM-killed three times while another +# project's `cargo test --workspace` ran alongside it, and each kill looked like +# a failure of the code under test. A lock per repo would not have helped, since +# the two runs were in different repositories. The lock file is shared by every +# project whose gate sources an equivalent of this, so they queue instead. +# +# Two rules that keep it from deadlocking: +# - a script takes it ONCE, near the top; nothing it calls takes it again; +# - do not wrap a script that takes it in `flock` yourself — the inner take +# would wait forever on the lock the outer one holds. +# +# Override the path with HEAVY_BUILD_LOCK (all projects must agree on it). If +# `flock` is not installed, the gate runs unlocked and says so. + +_hbl_who="${1:-gate}" +_hbl_path="${HEAVY_BUILD_LOCK:-${XDG_RUNTIME_DIR:-/tmp}/heavy-build.lock}" + +if command -v flock >/dev/null 2>&1; then + exec 9>"$_hbl_path" + if ! flock -n 9; then + echo "$_hbl_who: another heavy build holds $_hbl_path — waiting for it to finish" >&2 + flock 9 + echo "$_hbl_who: lock acquired, continuing" >&2 + fi +else + echo "$_hbl_who: flock not installed — running WITHOUT the host-wide build lock" >&2 +fi