Skip to content

chore(gates): queue behind other heavy builds on the host, not race them - #455

Merged
dc0sk merged 1 commit into
mainfrom
chore/host-wide-build-lock
Sep 25, 2026
Merged

dc0sk merged 1 commit into
mainfrom
chore/host-wide-build-lock

Conversation

@dc0sk

@dc0sk dc0sk commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Why

Nothing coordinates two heavy cargo runs on one machine. On 2026-09-25 this repo's gate was OOM-killed three times while another project's cargo test --workspace (OpenPulseHF) was running at the same time. Each kill looked like a failure of the code under test.

What

scripts/host-build-lock.sh takes a host-wide flock and holds it until the calling script exits. scripts/check-all.sh and .githooks/pre-push each source it once, near the top.

The lock path is shared with OpenPulseHF's scripts/lib/host-build-lock.sh (dc0sk/OpenPulseHF#1440), so the two projects' gates queue instead of competing for RAM. HEAVY_BUILD_LOCK overrides the path, and every project has to use the same value.

There's one helper, sourced by both scripts. Pasting the snippet into each would create two copies of the same decision that can drift apart, which is this repo's most common defect.

Deadlock rules (stated in the helper):

  • A script takes the lock once, and nothing it calls takes it again. pre-push does not call check-all.sh.
  • Nobody wraps a script that takes the lock in flock by hand. The inner take would wait forever on the lock the outer one already holds.

Verification

  • Contended: with the lock held by another process, a second take printed "waiting for it to finish". It got the lock 3.5 s later, exactly when the holder released it.
  • Uncontended: it got the lock in 0.01 s.
  • This PR's own push: the pre-push hook took the lock and ran the full suite, after I'd waited for OpenPulseHF's gate to finish (it did at 11:36).

Context

The same session also:

  • replaced the "don't run two at once" advice in the global CLAUDE.md with a rule that requires this lock;
  • added one line to the code-quality-gates skill's pre-push template.

🤖 Generated with Claude Code

https://claude.ai/code/session_018p7FxX7QMWNJVNp9LbaLkB

Nothing coordinates two heavy cargo runs on one machine. On 2026-09-25 this
repo's gate was OOM-killed three times while another project's
`cargo test --workspace` (OpenPulseHF) ran alongside it, and each kill looked
like a failure of the code under test.

`scripts/host-build-lock.sh` takes a HOST-WIDE flock, held until the calling
script exits. `scripts/check-all.sh` and `.githooks/pre-push` each source it
once, near the top. The path is shared with OpenPulseHF's
`scripts/lib/host-build-lock.sh` (dc0sk/OpenPulseHF#1440), so the two projects'
gates queue instead of contending for RAM. `HEAVY_BUILD_LOCK` overrides it, and
every project must agree on the value.

One helper, sourced by both, rather than the snippet pasted twice — two copies
of one decision drifting apart is this repo's dominant defect class.

Deadlock rules, stated in the helper: a script takes the lock once; nothing it
calls takes it again (pre-push does not call check-all.sh); and nobody wraps a
locking script in `flock` by hand, since the inner take would wait forever on
the lock the outer one holds.

Verified against the real helper: with the lock held by another process, a
second take printed "waiting for it to finish" and acquired it 3.5 s later,
exactly when the holder released; uncontended, it acquired in 0.01 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018p7FxX7QMWNJVNp9LbaLkB
@dc0sk
dc0sk merged commit 2064801 into main Sep 25, 2026
8 checks passed
@dc0sk
dc0sk deleted the chore/host-wide-build-lock branch September 25, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant