ci: shut down the Bazel server after each test leg by jeffpignataro · Pull Request #1547 · aspect-build/rules_py

jeffpignataro · 2026-09-17T18:36:33Z

The bazel-8 leg of test (e2e/rules-python-protobuf, bazel-8) in run 35181795460 failed with

ERROR: Error computing the main repository mapping: The version of MODULE.bazel.lock is not supported by this version of Bazel.

although actions/checkout had run git clean -ffdx and git reset --hard, the committed lockfile is lockFileVersion: 24 (Bazel 8.6.0) at every commit involved, and the job's BEP records build_tool_version: 8.6.0. The retry passed on a different runner.

Cause. The runner's previous job was the bazel-9 leg of the same workspace, from the run that cancel-in-progress killed when the PR was force-pushed. Its Bazel 9.2.0 server stayed alive in the output base, which every leg on a persistent runner shares. When the next job's Bazel 8.6 client found a server from another install base it killed it the way blaze.cc always does: by sending it a shutdown command. BazelLockFileModule.afterCommand runs on shutdown too, and Bazel 9.2 had parsed the committed v24 file as "old version, treat as empty", so its recomputed lockfile always differed from what it last read and it wrote a lockFileVersion: 28 file into the workspace. That happened after the checkout restored the file and one second before the test command read it with --lockfile_mode=error.

Reproduced locally with bazelisk 8.6.0 and 9.2.0 on one output base:

1. Bazel 8.6 build --lockfile_mode=update      -> lock v24 (the "committed" file)
2. Bazel 9.2 build --lockfile_mode=update      -> lock v28, server left running
3. cp committed lock back                      -> lock v24   (simulates git reset)
4. Bazel 8.6 info server_pid                   -> lock v28   (rewritten by the dying 9.2 server)
5. Bazel 8.6 build --lockfile_mode=error       -> ERROR: The version of MODULE.bazel.lock is not supported

Change. Every test leg ends with bazel shutdown on self-hosted runners, including on failure and cancellation. The lockfile rewrite then happens inside the job that owns the server, before the next job's checkout restores the committed file. USE_BAZEL_VERSION is job-level, so the shutdown uses the leg's own Bazel. GitHub-hosted legs get a fresh VM per job and skip the step.

A job that is cancelled or crashes before this step still leaves its server behind.

Changes are visible to end-users: no

Test plan

CI on this PR: the self-hosted test legs show the new step and pass. Reproducing the original failure needs the bazel-8 leg to land on a runner that just ran the bazel-9 leg of the same workspace, which the scheduler does not guarantee; the local reproduction above covers the mechanism.

🤖 Generated with Claude Code

https://claude.ai/code/session_011fhMDeQAA118gVquCvqVda

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fhMDeQAA118gVquCvqVda

coderabbitai · 2026-09-17T18:36:40Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 8d27f2c5-b727-4e7a-a594-92ca69fd0b31

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

aspect-workflows · 2026-09-17T18:36:56Z

✨ Aspect Workflows Tasks

📅 Thu Sep 17 18:37:00 UTC 2026

✅ 44 successful tasks

✅ buildifier · ⏱ 18.2s · 🐙 GitHub Actions · ☑️ Check
💬 Format complete (clean)
✅ gazelle · ⏱ 20.3s · 🐙 GitHub Actions · ☑️ Check
💬 Gazelle complete (clean)
✅ test-e2e-bazel-8 [test] · ⏱ 1m 1s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (324/324 passed · 322 cached)
✅ test-e2e-bazel-9 [test] · ⏱ 59.2s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (314/314 passed · 312 cached)
✅ test-e2e-crossbuild-bazel-8 [test] · ⏱ 55.9s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (39/39 passed · 39 cached)
✅ test-e2e-crossbuild-bazel-9 [test] · ⏱ 43.4s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (33/33 passed · 33 cached)
✅ test-e2e-interpreter-build-config-bazel-8 [test] · ⏱ 28.1s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-e2e-interpreter-build-config-bazel-9 [test] · ⏱ 37.9s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-e2e-interpreter-input-validation-bazel-8 [test] · ⏱ 16.4s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-e2e-interpreter-input-validation-bazel-9 [test] · ⏱ 38.8s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-e2e-interpreter-runtime-metadata-bazel-8 [test] · ⏱ 23.9s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (2/2 passed)
✅ test-e2e-interpreter-runtime-metadata-bazel-9 [test] · ⏱ 53.7s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (2/2 passed)
✅ test-e2e-interpreter-toolchain-settings-bazel-8 [test] · ⏱ 18.5s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-e2e-interpreter-toolchain-settings-bazel-9 [test] · ⏱ 37.2s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-e2e-rules-python-interop-bazel-8 [test] · ⏱ 54.6s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (16/16 passed)
✅ test-e2e-rules-python-interop-bazel-9 [test] · ⏱ 1m 19s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (16/16 passed)
✅ test-e2e-rules-python-protobuf-bazel-8 [test] · ⏱ 39.8s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-e2e-rules-python-protobuf-bazel-9 [test] · ⏱ 37.5s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-e2e-rules-python-provider-compat-bazel-8 [test] · ⏱ 27.3s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (4/4 passed)
✅ test-e2e-rules-python-provider-compat-bazel-9 [test] · ⏱ 43.7s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (4/4 passed)
✅ test-examples-debugger-bazel-8 [test] · ⏱ 33.4s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-examples-debugger-bazel-9 [test] · ⏱ 51.4s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-examples-dev_deps-bazel-8 [test] · ⏱ 27.6s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-examples-dev_deps-bazel-9 [test] · ⏱ 1m 6s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-examples-django-bazel-8 [test] · ⏱ 28.5s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-examples-django-bazel-9 [test] · ⏱ 43.6s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-examples-multi_version-bazel-8 [test] · ⏱ 26.4s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (2/2 passed)
✅ test-examples-multi_version-bazel-9 [test] · ⏱ 1m 4s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (2/2 passed)
✅ test-examples-protobuf-bazel-8 [test] · ⏱ 21.1s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-examples-protobuf-bazel-9 [test] · ⏱ 23.1s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-examples-py_binary-bazel-8 [test] · ⏱ 20.9s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-examples-py_binary-bazel-9 [test] · ⏱ 50.9s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-examples-py_pex_binary-bazel-8 [test] · ⏱ 24.3s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-examples-py_pex_binary-bazel-9 [test] · ⏱ 59.9s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed · 1 cached)
✅ test-examples-py_venv-bazel-8 [test] · ⏱ 22.8s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (3/3 passed)
✅ test-examples-py_venv-bazel-9 [test] · ⏱ 58.8s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (3/3 passed)
✅ test-examples-pytest-bazel-8 [test] · ⏱ 37.8s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (11/11 passed)
✅ test-examples-pytest-bazel-9 [test] · ⏱ 1m 10s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (11/11 passed)
✅ test-examples-uv_pip_compile-bazel-8 [test] · ⏱ 24.4s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-examples-uv_pip_compile-bazel-9 [test] · ⏱ 41.5s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-examples-virtual_deps-bazel-8 [test] · ⏱ 24.4s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-examples-virtual_deps-bazel-9 [test] · ⏱ 44.7s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (1/1 passed)
✅ test-root-bazel-8 [test] · ⏱ 10m 26s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (374/374 passed)
✅ test-root-bazel-9 [test] · ⏱ 13m 22s · 🐙 GitHub Actions · ☑️ Check
💬 Bazel test complete (373/373 passed)

_{⏱ Last updated Thu Sep 17 18:52:48 UTC 2026 · 📊 GitHub API quota 1,098/7,700 (14% used, resets in 9m)}
_{🚀 Powered by Aspect CLI (v2026.28.2) | Aspect Build · X · LinkedIn · YouTube}

github-actions · 2026-09-17T18:43:24Z

py_binary startup benchmark

Version	Time (ms)	vs BCR	vs main	Build (s)
BCR 1.11.7	156.0/155.9 ±0.8	—	—	1.50
main	54.7/54.7 ±0.7	-64.9%	—	1.67
PR	54.6/54.5 ±0.5	-65.0%	-0.3%	1.43

Time = mean/median ±stddev.
Measured with hyperfine --warmup 5 --runs 50 --shell=none on Linux
Gate: PR vs HEAD main median (threshold: 10%, and must exceed the 2×SE noise floor, here 0.4%). BCR is shown only as a historical baseline.
Build time: cold bazel build //:bench with isolated output base, no disk cache; external repos prefetched so network is excluded.

sys.path quality

Version	entries	sp roots
BCR 1.11.7	6	1
main	7	2
PR	7	2

sys.path quality measured by bench_syspath inside the assembled venv: sys.path entries, distinct site-packages roots, duplicate realpaths. Duplicates indicate symlink redundancy; many roots suggest an inefficient venv layout.

Bazel analysis benchmark

Version	Time (ms)	vs BCR	vs main	Targets	Actions	Configured (ext)	Cfg/target	Actions/cfg
BCR 2.0.0-alpha.6	1592/1557 ±228	—	—	404	20613	7380 (6675)	18.3	2.79
main	1556/1500 ±276	-3.6%	—	404	18363	7379 (6674)	18.3	2.49
PR	1641/1572 ±299	+1.0%	+4.8%	404	18363	7379 (6674)	18.3	2.49

Measured with hyperfine --warmup 1 --runs 10 on Linux
Time = mean/median ±stddev. Cfg = configured targets; (ext) = the count in external repos (the @pypi hub machinery).
Gate: PR vs HEAD main median (threshold: 10%, and must exceed the 2×SE noise floor, here 17.2%). BCR is shown only as a historical baseline.
Command: warm-server bazel build --nobuild //workspace/..., analysis cache discarded each run via a fresh --action_env value; no disk cache.
Workload: 100 py_test targets — 2 test files/package in file generation mode (one venv per test file); a single dep_group.

py_image_layer benchmark

Scenario	Version	Time (s)	Actions	vs BCR	vs main
analysis	BCR 2.0.0-alpha.6	0.68/0.65 ±0.11	747	—	—
analysis	main	0.65/0.63 ±0.12	668	-3.7%	—
analysis	PR	0.66/0.62 ±0.14	668	-5.4%	-1.8%
1p source	BCR 2.0.0-alpha.6	0.74/0.72 ±0.09	4	—	—
1p source	main	0.68/0.68 ±0.06	5	-5.3%	—
1p source	PR	0.69/0.68 ±0.05	5	-5.3%	-0.0%
3p wheel	BCR 2.0.0-alpha.6	0.85/0.86 ±0.06	5	—	—
3p wheel	main	0.80/0.80 ±0.07	6	-7.1%	—
3p wheel	PR	0.82/0.82 ±0.05	6	-4.1%	+3.3%

Time = mean/median ±stddev.
Measured with hyperfine on Linux, building //workspace:image_layers (10 binaries, ~30-wheel dep pool, grouped first-party/pip/interpreter tier) with isolated output base, no disk cache.
Scenarios: analysis = warm-server bazel build --nobuild, re-analyzed each run via a fresh --action_env value; incrementals run against a built state with warm analysis: source = append to the last package's lib.py, wheel = rewrite click post_install_patches content.
Actions: for Analysis, the total action count behind the image target from aquery deps(...); for incrementals, actions re-executed for the mutation, from a single instrumented run's BEP build metrics (deterministic; per-mnemonic breakdown in the *-actions.json artifacts). Informational only, not gated.
Gate: PR vs HEAD main median per scenario (threshold: 10%, and must exceed the 2×SE noise floor). BCR is shown only as a historical baseline.

ci: shut down the Bazel server after each test leg

454cce4

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fhMDeQAA118gVquCvqVda

jeffpignataro requested a review from jbedard September 17, 2026 18:38

jeffpignataro marked this pull request as ready for review September 17, 2026 18:58

jbedard approved these changes Sep 17, 2026

View reviewed changes

jeffpignataro merged commit 3ebd1dd into main Sep 17, 2026
150 of 156 checks passed

jeffpignataro deleted the jeff/fix/ci-bazel-shutdown branch September 17, 2026 19:00

Navigation Menu

Uh oh!

ci: shut down the Bazel server after each test leg - #1547

ci: shut down the Bazel server after each test leg#1547
jeffpignataro merged 1 commit into
mainfrom
jeff/fix/ci-bazel-shutdown

jeffpignataro commented Sep 17, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 17, 2026 •

edited

Loading

Review skipped

Uh oh!

aspect-workflows Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jeffpignataro commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes are visible to end-users: no

Test plan

Uh oh!

coderabbitai Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

aspect-workflows Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✨ Aspect Workflows Tasks

✅ 44 successful tasks

Uh oh!

github-actions Bot commented Sep 17, 2026

py_binary startup benchmark

sys.path quality

Bazel analysis benchmark

py_image_layer benchmark

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jeffpignataro commented Sep 17, 2026 •

edited

Loading

coderabbitai Bot commented Sep 17, 2026 •

edited

Loading

aspect-workflows Bot commented Sep 17, 2026 •

edited

Loading