Skip to content

chore(gate): queue behind other heavy builds on the host, not race them - #1440

Open
dc0sk wants to merge 1 commit into
mainfrom
chore/host-wide-build-lock
Open

dc0sk wants to merge 1 commit into
mainfrom
chore/host-wide-build-lock

Conversation

@dc0sk

@dc0sk dc0sk commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Nothing coordinates two heavy cargo runs on one machine. On 2026-09-25 a
neighbouring project's gate (fnec-rust) was OOM-killed three times while this
repo's cargo test --workspace ran alongside it — and every kill read as a
failure of the code under test, which is the misattribution GATE: INVALID
exists to prevent, arriving by a channel this gate cannot see.

scripts/lib/host-build-lock.sh takes a HOST-WIDE flock, held until the
calling script exits. scripts/gate.sh and the pre-push hook each source it
once. The path is shared with fnec-rust's scripts/host-build-lock.sh, so the
two projects' gates queue instead of contending for RAM; HEAVY_BUILD_LOCK
overrides it, and every project must agree.

Placement is deliberate:

  • gate.sh takes it right after argument parsing and BEFORE START_HEAD, so
    time spent waiting is not part of the run and the drift guard's baseline is
    taken after the wait.
  • --fingerprint is exempt — the seconds-long primitive builds nothing.
  • the pre-push hook takes it AFTER the GATE_SKIP escape hatch, so a
    deliberate skip never waits.
  • neither calls the other, so both may take it without deadlocking.

Verified without building (a build in this worktree would put a target dir
inside it): with a test lock held, scripts/gate.sh and the pre-push hook each
printed "waiting for it to finish" and were stopped by timeout before any
cargo step (exit 124); --fingerprint returned in 1.2 s against a 6 s hold.

NOT verified: this repo's rule to sabotage-verify gate.sh through a full run
after every edit. A full gate is ~2 h and this checkout's sibling was running
the ota_channel_adaptation acceptance suite at the time; running a second
suite beside it is the exact contention this change exists to stop. Run
scripts/gate.sh on a quiet machine before merging.

Verification-objective: two heavy cargo runs on one host serialize instead of
racing for RAM — observed as a gate and a pre-push hook each blocking on a
held host lock, and the exempt fingerprint mode not blocking

Pushed with GATE_SKIP=1, visibly and deliberately: the diff contains no Rust, the hook would have run workspace clippy inside a worktree, and the sibling checkout was mid-way through the ota_channel_adaptation acceptance suite. Do not merge until scripts/gate.sh has run on a quiet machine (this repo's own sabotage-after-every-edit rule).

The lock only protects a checkout once cargo-husky reinstalls the hook from .cargo-husky/hooks/pre-push — the installed copy in .git/hooks currently differs from the versioned source.

🤖 Generated with Claude Code

https://claude.ai/code/session_018p7FxX7QMWNJVNp9LbaLkB

Nothing coordinates two heavy cargo runs on one machine. On 2026-09-25 a
neighbouring project's gate (fnec-rust) was OOM-killed three times while this
repo's `cargo test --workspace` ran alongside it — and every kill read as a
failure of the code under test, which is the misattribution `GATE: INVALID`
exists to prevent, arriving by a channel this gate cannot see.

`scripts/lib/host-build-lock.sh` takes a HOST-WIDE flock, held until the
calling script exits. `scripts/gate.sh` and the pre-push hook each source it
once. The path is shared with fnec-rust's `scripts/host-build-lock.sh`, so the
two projects' gates queue instead of contending for RAM; `HEAVY_BUILD_LOCK`
overrides it, and every project must agree.

Placement is deliberate:
  - gate.sh takes it right after argument parsing and BEFORE `START_HEAD`, so
    time spent waiting is not part of the run and the drift guard's baseline is
    taken after the wait.
  - `--fingerprint` is exempt — the seconds-long primitive builds nothing.
  - the pre-push hook takes it AFTER the `GATE_SKIP` escape hatch, so a
    deliberate skip never waits.
  - neither calls the other, so both may take it without deadlocking.

Verified without building (a build in this worktree would put a target dir
inside it): with a test lock held, `scripts/gate.sh` and the pre-push hook each
printed "waiting for it to finish" and were stopped by `timeout` before any
cargo step (exit 124); `--fingerprint` returned in 1.2 s against a 6 s hold.

NOT verified: this repo's rule to sabotage-verify gate.sh through a full run
after every edit. A full gate is ~2 h and this checkout's sibling was running
the `ota_channel_adaptation` acceptance suite at the time; running a second
suite beside it is the exact contention this change exists to stop. Run
`scripts/gate.sh` on a quiet machine before merging.

Verification-objective: two heavy cargo runs on one host serialize instead of
racing for RAM — observed as a gate and a pre-push hook each blocking on a
held host lock, and the exempt fingerprint mode not blocking
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018p7FxX7QMWNJVNp9LbaLkB
dc0sk added a commit to dc0sk/fnec-rust that referenced this pull request Sep 25, 2026
…hem (#455)

Nothing coordinates two heavy cargo runs on one machine. On 2026-09-25 this
repo's gate was OOM-killed three times while another project's
`cargo test --workspace` (OpenPulseHF) ran alongside it, and each kill looked
like a failure of the code under test.

`scripts/host-build-lock.sh` takes a HOST-WIDE flock, held until the calling
script exits. `scripts/check-all.sh` and `.githooks/pre-push` each source it
once, near the top. The path is shared with OpenPulseHF's
`scripts/lib/host-build-lock.sh` (dc0sk/OpenPulseHF#1440), so the two projects'
gates queue instead of contending for RAM. `HEAVY_BUILD_LOCK` overrides it, and
every project must agree on the value.

One helper, sourced by both, rather than the snippet pasted twice — two copies
of one decision drifting apart is this repo's dominant defect class.

Deadlock rules, stated in the helper: a script takes the lock once; nothing it
calls takes it again (pre-push does not call check-all.sh); and nobody wraps a
locking script in `flock` by hand, since the inner take would wait forever on
the lock the outer one holds.

Verified against the real helper: with the lock held by another process, a
second take printed "waiting for it to finish" and acquired it 3.5 s later,
exactly when the holder released; uncontended, it acquired in 0.01 s.


Claude-Session: https://claude.ai/code/session_018p7FxX7QMWNJVNp9LbaLkB

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant