Skip to content

ci: run bazel-9 jobs with --lockfile_mode=off - #1545

Closed
xangcastle wants to merge 1 commit into
mainfrom
xangcastle/ci-bazel9-lockfile-off
Closed

xangcastle wants to merge 1 commit into
mainfrom
xangcastle/ci-bazel9-lockfile-off

Conversation

@xangcastle

Copy link
Copy Markdown
Member

The bazel-9 matrix entry ran with --lockfile_mode=update. The committed lockfiles are Bazel 8 format (lockFileVersion 24); Bazel 9.2 rejects them in error mode and, in update mode, rewrites them to version 28 inside the checkout. update never validated anything, it only wrote.

Since the large workspaces moved to Aspect Workflows runners (#1542) with a persistent workdir shared across jobs, that write leaks. A bazel-8 job scheduled on a runner whose previous job was the bazel-9 job of the same workspace fails with:

ERROR: Error computing the main repository mapping: The version of MODULE.bazel.lock is not supported by this version of Bazel.

even though the checkout carries version 24. Seen on #1544: the same commit failed on one runner and passed on re-run on another. The failing job's runner log lists test-e2e-rules-python-protobuf-bazel-9 as its previous build.

off makes Bazel 9 neither read nor write the lockfile, which is what the e2e test.sh scripts already do for their own invocations.

Changes are visible to end-users: no

Test plan

  • Locally with Bazel 9.2 on e2e/rules-python-protobuf: bazel mod graph --lockfile_mode=off resolves and the lockfile is untouched, while --lockfile_mode=update rewrites it to version 28 and --lockfile_mode=error rejects version 24.
  • CI on this PR runs every workspace under both matrix entries.

The bazel-9 matrix entry ran with `--lockfile_mode=update`. The
committed lockfiles are Bazel 8 format (lockFileVersion 24); Bazel 9.2
rejects them in error mode and, in update mode, rewrites them to
version 28 inside the checkout. `update` validated nothing, it only
wrote.

Since the large workspaces moved to Aspect Workflows runners with a
persistent workdir shared across jobs, that write leaks: a bazel-8 job
scheduled on a runner whose previous job was the bazel-9 job of the
same workspace fails with "The version of MODULE.bazel.lock is not
supported by this version of Bazel", even though the checkout carries
version 24. Seen on #1544, where the same commit failed and then passed
on re-run on a different runner.

`off` makes Bazel 9 neither read nor write the lockfile, which is what
the e2e test.sh scripts already do for their own invocations. Verified
locally with Bazel 9.2 on e2e/rules-python-protobuf: module resolution
succeeds and the lockfile is untouched.
@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: af118c8c-61d5-4cc9-bd2b-35ee2142abd7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@aspect-workflows

aspect-workflows Bot commented Sep 17, 2026

Copy link
Copy Markdown

✨ Aspect Workflows Tasks

📅 Thu Sep 17 05:18:20 UTC 2026

✅ 44 successful tasks

  • ✅ buildifier · ⏱ 16.1s · 🐙 GitHub Actions · ☑️ Check
    💬 Format complete (clean)
  • ✅ gazelle · ⏱ 18.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Gazelle complete (clean)
  • ✅ test-e2e-bazel-8 [test] · ⏱ 50.3s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (324/324 passed · 322 cached)
  • ✅ test-e2e-bazel-9 [test] · ⏱ 47.3s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (314/314 passed · 312 cached)
  • ✅ test-e2e-crossbuild-bazel-8 [test] · ⏱ 41.4s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (39/39 passed · 39 cached)
  • ✅ test-e2e-crossbuild-bazel-9 [test] · ⏱ 40.6s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (39/39 passed · 39 cached)
  • ✅ test-e2e-interpreter-build-config-bazel-8 [test] · ⏱ 24.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-build-config-bazel-9 [test] · ⏱ 1m 14s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-input-validation-bazel-8 [test] · ⏱ 16.1s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-input-validation-bazel-9 [test] · ⏱ 46.3s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-runtime-metadata-bazel-8 [test] · ⏱ 23.5s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (2/2 passed)
  • ✅ test-e2e-interpreter-runtime-metadata-bazel-9 [test] · ⏱ 44.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (2/2 passed)
  • ✅ test-e2e-interpreter-toolchain-settings-bazel-8 [test] · ⏱ 19.7s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-interpreter-toolchain-settings-bazel-9 [test] · ⏱ 49.7s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-e2e-rules-python-interop-bazel-8 [test] · ⏱ 53.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (16/16 passed)
  • ✅ test-e2e-rules-python-interop-bazel-9 [test] · ⏱ 1m 21s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (16/16 passed)
  • ✅ test-e2e-rules-python-protobuf-bazel-8 [test] · ⏱ 34.7s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-e2e-rules-python-protobuf-bazel-9 [test] · ⏱ 37.1s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-e2e-rules-python-provider-compat-bazel-8 [test] · ⏱ 22s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (4/4 passed)
  • ✅ test-e2e-rules-python-provider-compat-bazel-9 [test] · ⏱ 1m 20s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (4/4 passed)
  • ✅ test-examples-debugger-bazel-8 [test] · ⏱ 24.1s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-debugger-bazel-9 [test] · ⏱ 46.9s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-dev_deps-bazel-8 [test] · ⏱ 23s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-dev_deps-bazel-9 [test] · ⏱ 43.5s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-django-bazel-8 [test] · ⏱ 23.9s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-django-bazel-9 [test] · ⏱ 57.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-multi_version-bazel-8 [test] · ⏱ 31.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (2/2 passed)
  • ✅ test-examples-multi_version-bazel-9 [test] · ⏱ 52s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (2/2 passed)
  • ✅ test-examples-protobuf-bazel-8 [test] · ⏱ 22.6s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-protobuf-bazel-9 [test] · ⏱ 21.1s · ✨ Aspect · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-py_binary-bazel-8 [test] · ⏱ 21.6s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-py_binary-bazel-9 [test] · ⏱ 43.7s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-py_pex_binary-bazel-8 [test] · ⏱ 26.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-py_pex_binary-bazel-9 [test] · ⏱ 54.3s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed · 1 cached)
  • ✅ test-examples-py_venv-bazel-8 [test] · ⏱ 29.4s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (3/3 passed)
  • ✅ test-examples-py_venv-bazel-9 [test] · ⏱ 1m 17s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (3/3 passed)
  • ✅ test-examples-pytest-bazel-8 [test] · ⏱ 37.8s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (11/11 passed)
  • ✅ test-examples-pytest-bazel-9 [test] · ⏱ 1m 7s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (11/11 passed)
  • ✅ test-examples-uv_pip_compile-bazel-8 [test] · ⏱ 33.5s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-uv_pip_compile-bazel-9 [test] · ⏱ 35.3s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-virtual_deps-bazel-8 [test] · ⏱ 27.2s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-examples-virtual_deps-bazel-9 [test] · ⏱ 1m 18s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (1/1 passed)
  • ✅ test-root-bazel-8 [test] · ⏱ 15m 39s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (374/374 passed)
  • ✅ test-root-bazel-9 [test] · ⏱ 17m 40s · 🐙 GitHub Actions · ☑️ Check
    💬 Bazel test complete (373/373 passed)

⏱ Last updated Thu Sep 17 05:36:32 UTC 2026 · 📊 GitHub API quota 2,748/7,700 (36% used, resets in 48s)
🚀 Powered by Aspect CLI (v2026.28.2)  |  Aspect Build · X · LinkedIn · YouTube

@github-actions

Copy link
Copy Markdown

py_binary startup benchmark

Version Time (ms) vs BCR vs main Build (s)
BCR 1.11.7 166.8/166.5 ±3.0 1.56
main 57.2/57.0 ±1.1 -65.7% 1.78
PR 56.4/56.4 ±1.3 -66.1% -1.1% 1.55

Time = mean/median ±stddev.
Measured with hyperfine --warmup 5 --runs 50 --shell=none on Linux
Gate: PR vs HEAD main median (threshold: 10%, and must exceed the 2×SE noise floor, here 0.9%). BCR is shown only as a historical baseline.
Build time: cold bazel build //:bench with isolated output base, no disk cache; external repos prefetched so network is excluded.

sys.path quality

Version entries sp roots dupes
BCR 1.11.7 6 1 0
main 7 2 0
PR 7 2 0

sys.path quality measured by bench_syspath inside the assembled venv: sys.path entries, distinct site-packages roots, duplicate realpaths. Duplicates indicate symlink redundancy; many roots suggest an inefficient venv layout.

Bazel analysis benchmark

Version Time (ms) vs BCR vs main Targets Actions Configured (ext) Cfg/target Actions/cfg
BCR 2.0.0-alpha.6 1558/1509 ±246 404 20613 7380 (6675) 18.3 2.79
main 1447/1408 ±201 -6.7% 404 18363 7379 (6674) 18.3 2.49
PR 1513/1460 ±226 -3.3% +3.7% 404 18363 7379 (6674) 18.3 2.49

Measured with hyperfine --warmup 1 --runs 10 on Linux
Time = mean/median ±stddev. Cfg = configured targets; (ext) = the count in external repos (the @pypi hub machinery).
Gate: PR vs HEAD main median (threshold: 10%, and must exceed the 2×SE noise floor, here 13.6%). BCR is shown only as a historical baseline.
Command: warm-server bazel build --nobuild //workspace/..., analysis cache discarded each run via a fresh --action_env value; no disk cache.
Workload: 100 py_test targets — 2 test files/package in file generation mode (one venv per test file); a single dep_group.

py_image_layer benchmark

Scenario Version Time (s) Actions vs BCR vs main
analysis BCR 2.0.0-alpha.6 0.75/0.72 ±0.11 747
analysis main 0.76/0.72 ±0.12 668 +0.6%
analysis PR 0.75/0.71 ±0.13 668 -1.1% -1.7%
1p source BCR 2.0.0-alpha.6 0.76/0.73 ±0.09 4
1p source main 0.76/0.75 ±0.04 5 +2.2%
1p source PR 0.78/0.74 ±0.22 5 +1.3% -0.9%
3p wheel BCR 2.0.0-alpha.6 0.88/0.88 ±0.03 5
3p wheel main 0.91/0.90 ±0.02 6 +2.4%
3p wheel PR 0.91/0.90 ±0.03 6 +2.8% +0.4%

Time = mean/median ±stddev.
Measured with hyperfine on Linux, building //workspace:image_layers (10 binaries, ~30-wheel dep pool, grouped first-party/pip/interpreter tier) with isolated output base, no disk cache.
Scenarios: analysis = warm-server bazel build --nobuild, re-analyzed each run via a fresh --action_env value; incrementals run against a built state with warm analysis: source = append to the last package's lib.py, wheel = rewrite click post_install_patches content.
Actions: for Analysis, the total action count behind the image target from aquery deps(...); for incrementals, actions re-executed for the mutation, from a single instrumented run's BEP build metrics (deterministic; per-mnemonic breakdown in the *-actions.json artifacts). Informational only, not gated.
Gate: PR vs HEAD main median per scenario (threshold: 10%, and must exceed the 2×SE noise floor). BCR is shown only as a historical baseline.

@jbedard jbedard closed this Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants