ci: connect builds to Aspect Cloud by joshcbarnes · Pull Request #1546 · aspect-build/rules_py

joshcbarnes · 2026-09-17T15:20:16Z

Points CI at Aspect Cloud's remote cache and BES. The jobs run on stock GitHub-hosted runners, which have no cache of their own, and nothing here named an endpoint.

Draft: opened to exercise the one thing that cannot be validated locally — the ASPECT_OSS_API_TOKEN → JWT exchange.

Follows aspect-build/rules_lint#1003, adapted to this repo's two differences: a self-hosted runner pool, and ~20 sibling Bazel workspaces that CI drives through the Aspect CLI rather than vanilla bazel.

What it takes

setup-aspect already pipes its aspect-api-token input into aspect auth login --with-api-token. That records a credential but no endpoints, and Aspect Cloud's built-in entry ships with none, so a token alone leaves --remote with nothing to wire. The endpoints come from a committed .aspect/config.json, found through $ASPECT_WORKSPACE — pinned to the checkout root, since the test matrix builds from subdirectories. That variable selects only the deployment list, never which .aspect/config.axl a task loads, nor which workspace is built.

REMOTE_FLAG and BAZEL_ASPECT_FLAGS are empty when the secret is absent, which is every fork PR: forks get no org secrets, the login never runs, and either flag would reach Aspect Cloud with no credential.

Why `test` computes its own `REMOTE_FLAG`

--remote is a hard failure on a self-hosted Aspect Workflows runner — REMOTE_REFUSED_BY_ENV: the ASPECT_WORKFLOWS_* environment already names that machine's deployment, and the CLI refuses to route the build off it rather than wire a second one. The job's REMOTE_FLAG therefore reads its own runs-on condition back, so the four workspaces that ask for the 32-core pool keep the runner's wiring. Without the gate they would fail outright.

Why the version bump

The pin was 2026.28.2, which predates the deployment feature — no --remote, no aspect auth configure. --task-key is gone at 2026.38.14, and an unrecognized flag is forwarded to Bazel rather than rejected, so the three call sites move to --task:name or the build fails. Only the root pins a version; the launcher walks up, so the sibling workspaces follow it.

One rc, two workspaces

e2e/crossbuild is the only sibling workspace CI builds with vanilla bazel. It imports the root's rc through %workspace%/../.. instead of carrying a copy — %workspace% expands to an absolute path before Bazel opens the file, so that reaches the root from any working directory, unlike a bare relative path. Inside the imported file %workspace% re-expands to the importing workspace, which is why .aspect/generated/ is committed there too: that is where its execution log lands, and Bazel fails the build rather than create the directory.

Verified locally

aspect build --remote from the repo root and from e2e/crossbuild with ASPECT_WORKSPACE at the root — the CI shape — both print Using deployment 'aspect-cloud', Streaming build events to grpcs://bes.aspect.build, and an app.aspect.build/i/… link.
With ~/.aspect/config.json moved aside, --remote fails no default is set without ASPECT_WORKSPACE and resolves with it, confirming the committed file is what CI will read.
bazel build --config=aspect-cloud streams to Aspect Cloud from the root, and --config=aspect-cloud resolves in e2e/crossbuild and from a subdirectory of it.
The execution log from a e2e/crossbuild build lands in e2e/crossbuild/.aspect/generated/, leaving the root's untouched.
Removing .aspect/generated/ fails the build, confirming the committed directory is load-bearing.

What this PR is meant to prove

Local runs used a browser login, not the token, so CI is the first exercise of:

ASPECT_OSS_API_TOKEN exchanges for a JWT. Note: setup-aspect core.warnings rather than fails on a bad login, so check that step's output explicitly — a green setup step does not mean it worked.
That account has cache/BES write access for the org.
The CLI jobs print Using deployment 'aspect-cloud'; the vanilla-bazel jobs print Streaming build results to.
The CLI bump from 2026.28.2 holds across all 21 workspaces on both Bazel 8 and 9.
The 32-core matrix entries still run green on the self-hosted pool with REMOTE_FLAG empty.

Deliberately out of scope

The test.sh scripts in e2e/* invoke plain bazel in ~20 nested workspaces. They stay off Aspect Cloud: an rc import alone would not connect them, since those scripts pass no --config=aspect-cloud and $BAZEL_ASPECT_FLAGS is not threaded into them.

🤖 Generated with Claude Code

CI runs on stock GitHub-hosted runners, which have no remote cache of their own, and nothing here named an endpoint: `aspect auth login --with-api-token`, which setup-aspect already runs from its `aspect-api-token` input, records a credential but no endpoints, and Aspect Cloud's built-in entry ships with none. So the deployment is declared in a committed `.aspect/config.json` and found through `$ASPECT_WORKSPACE`, pinned to the checkout root because the test matrix builds from `e2e/*` and `examples/*`. That variable selects only the deployment list, not which `.aspect/config.axl` a task loads, and not which workspace is built. Aspect CLI tasks then take `--remote`; vanilla `bazel` calls take `--config=aspect-cloud` from the committed `.aspect/bazelrc`, plus a `--workspace_status_command` so they are attributed in the Web UI rather than arriving anonymous (a stopgap for `aspect setup workspace-data`, aspect-build/aspect-cli#1465). Both flags resolve to nothing without the token, which is every fork PR: those get no org secrets, so the login never runs and either one would reach Aspect Cloud with no credential. The token is `ASPECT_OSS_API_TOKEN`, matching aspect-build/rules_lint#1002 — the org-wide `ASPECT_API_TOKEN` authenticates as a tenant the OSS team cannot see, so uploads would succeed while the UI stayed empty. There is deliberately no fallback to it. `test` computes its own `REMOTE_FLAG` by reading `runs-on` back, because `--remote` is a hard failure on a self-hosted Aspect Workflows runner (`REMOTE_REFUSED_BY_ENV`): the `ASPECT_WORKFLOWS_*` environment already names that machine's deployment, and the CLI refuses to route the build off it rather than wire a second one. Without the gate the four workspaces that ask for the 32-core pool would fail outright. Requires the CLI pin to move off 2026.28.2, which predates the deployment feature and has no `--remote`. `--task-key` is gone there, and an unrecognized flag is forwarded to Bazel rather than rejected, so the three call sites move to `--task:name` or the build fails. `e2e/crossbuild` is the one sibling workspace CI drives with vanilla `bazel`. It imports the root's rc through `%workspace%/../..` rather than carrying a copy: `%workspace%` expands to an absolute path before Bazel opens the file, so that reaches the root from any working directory, unlike a bare relative path. Inside the imported file `%workspace%` re-expands to the importing workspace, which is why `.aspect/generated/` is committed there too — that is where its execution log lands, and Bazel fails the build outright rather than create the directory. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-09-17T15:20:24Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 41bc24c1-220f-4e8a-b9f7-59d1701da831

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

aspect-workflows · 2026-09-17T15:20:44Z

Aspect Workflows Tasks

📅 Thu Sep 17 20:18:46 UTC 2026

Task Results

✅ buildifier · complete (clean) · 21.7s · GitHub Actions · Check
✅ gazelle · complete (clean) · 20.9s · GitHub Actions · Check
✅ test-e2e-bazel-8 [test] · complete (324/324 passed · 322 cached) · 52.1s · Aspect · GitHub Actions · Check
✅ test-e2e-bazel-9 [test] · complete (314/314 passed · 312 cached) · 45.4s · Aspect · GitHub Actions · Check
✅ test-e2e-crossbuild-bazel-8 [test] · complete (39/39 passed · 39 cached) · 41s · Aspect · GitHub Actions · Check
✅ test-e2e-crossbuild-bazel-9 [test] · complete (39/39 passed · 39 cached) · 40.2s · Aspect · GitHub Actions · Check
✅ test-e2e-interpreter-build-config-bazel-8 [test] · complete (1/1 passed · 1 cached) · 21.2s · Aspect · GitHub Actions · Check
✅ test-e2e-interpreter-build-config-bazel-9 [test] · complete (1/1 passed · 1 cached) · 54.1s · Aspect · GitHub Actions · Check
✅ test-e2e-interpreter-input-validation-bazel-8 [test] · complete (1/1 passed · 1 cached) · 20.1s · Aspect · GitHub Actions · Check
✅ test-e2e-interpreter-input-validation-bazel-9 [test] · complete (1/1 passed · 1 cached) · 46.6s · Aspect · GitHub Actions · Check
✅ test-e2e-interpreter-runtime-metadata-bazel-8 [test] · complete (2/2 passed · 2 cached) · 27.9s · Aspect · GitHub Actions · Check
✅ test-e2e-interpreter-runtime-metadata-bazel-9 [test] · complete (2/2 passed · 2 cached) · 25s · Aspect · GitHub Actions · Check
✅ test-e2e-interpreter-toolchain-settings-bazel-8 [test] · complete (1/1 passed · 1 cached) · 21.4s · Aspect · GitHub Actions · Check
✅ test-e2e-interpreter-toolchain-settings-bazel-9 [test] · complete (1/1 passed · 1 cached) · 20.5s · Aspect · GitHub Actions · Check
✅ test-e2e-rules-python-interop-bazel-8 [test] · complete (16/16 passed · 16 cached) · 1m 41s · Aspect · GitHub Actions · Check
✅ test-e2e-rules-python-interop-bazel-9 [test] · complete (16/16 passed · 16 cached) · 1m 59s · Aspect · GitHub Actions · Check
✅ test-e2e-rules-python-protobuf-bazel-8 [test] · complete (1/1 passed · 1 cached) · 35.1s · Aspect · GitHub Actions · Check
✅ test-e2e-rules-python-protobuf-bazel-9 [test] · complete (1/1 passed · 1 cached) · 35.3s · Aspect · GitHub Actions · Check
✅ test-e2e-rules-python-provider-compat-bazel-8 [test] · complete (4/4 passed · 4 cached) · 27.3s · Aspect · GitHub Actions · Check
✅ test-e2e-rules-python-provider-compat-bazel-9 [test] · complete (4/4 passed · 4 cached) · 51.4s · Aspect · GitHub Actions · Check
✅ test-examples-debugger-bazel-8 [test] · complete (1/1 passed · 1 cached) · 21.8s · Aspect · GitHub Actions · Check
✅ test-examples-debugger-bazel-9 [test] · complete (1/1 passed · 1 cached) · 42.1s · Aspect · GitHub Actions · Check
✅ test-examples-dev_deps-bazel-8 [test] · complete (1/1 passed · 1 cached) · 22.5s · Aspect · GitHub Actions · Check
✅ test-examples-dev_deps-bazel-9 [test] · complete (1/1 passed · 1 cached) · 47.7s · Aspect · GitHub Actions · Check
✅ test-examples-django-bazel-8 [test] · complete (1/1 passed · 1 cached) · 25.6s · Aspect · GitHub Actions · Check
✅ test-examples-django-bazel-9 [test] · complete (1/1 passed · 1 cached) · 48.6s · Aspect · GitHub Actions · Check
✅ test-examples-multi_version-bazel-8 [test] · complete (2/2 passed · 2 cached) · 22.3s · Aspect · GitHub Actions · Check
✅ test-examples-multi_version-bazel-9 [test] · complete (2/2 passed · 2 cached) · 31.2s · Aspect · GitHub Actions · Check
✅ test-examples-protobuf-bazel-8 [test] · complete (1/1 passed · 1 cached) · 18.2s · Aspect · GitHub Actions · Check
✅ test-examples-protobuf-bazel-9 [test] · complete (1/1 passed · 1 cached) · 19.6s · Aspect · GitHub Actions · Check
✅ test-examples-py_binary-bazel-8 [test] · complete (1/1 passed · 1 cached) · 21.8s · Aspect · GitHub Actions · Check
✅ test-examples-py_binary-bazel-9 [test] · complete (1/1 passed · 1 cached) · 37.8s · Aspect · GitHub Actions · Check
✅ test-examples-py_pex_binary-bazel-8 [test] · complete (1/1 passed · 1 cached) · 24.7s · Aspect · GitHub Actions · Check
✅ test-examples-py_pex_binary-bazel-9 [test] · complete (1/1 passed · 1 cached) · 43.4s · Aspect · GitHub Actions · Check
✅ test-examples-py_venv-bazel-8 [test] · complete (3/3 passed · 3 cached) · 23.6s · Aspect · GitHub Actions · Check
✅ test-examples-py_venv-bazel-9 [test] · complete (3/3 passed · 3 cached) · 45.2s · Aspect · GitHub Actions · Check
✅ test-examples-pytest-bazel-8 [test] · complete (11/11 passed · 11 cached) · 27s · Aspect · GitHub Actions · Check
✅ test-examples-pytest-bazel-9 [test] · complete (11/11 passed · 11 cached) · 57.7s · Aspect · GitHub Actions · Check
✅ test-examples-uv_pip_compile-bazel-8 [test] · complete (1/1 passed · 1 cached) · 31.5s · Aspect · GitHub Actions · Check
✅ test-examples-uv_pip_compile-bazel-9 [test] · complete (1/1 passed · 1 cached) · 47.4s · Aspect · GitHub Actions · Check
✅ test-examples-virtual_deps-bazel-8 [test] · complete (1/1 passed · 1 cached) · 22.1s · Aspect · GitHub Actions · Check
✅ test-examples-virtual_deps-bazel-9 [test] · complete (1/1 passed · 1 cached) · 43.5s · Aspect · GitHub Actions · Check
✅ test-root-bazel-8 [test] · complete (374/374 passed · 374 cached) · 5m 10s · Aspect · GitHub Actions · Check
✅ test-root-bazel-9 [test] · complete (373/373 passed · 373 cached) · 3m 24s · Aspect · GitHub Actions · Check

_{⏱ Last updated Thu Sep 17 20:24:04 UTC 2026 · 📊 GitHub API quota 0/7,700 (0% used, resets in 59m)}
_{🚀 Powered by Aspect CLI (v2026.38.14) | Aspect Build · X · LinkedIn · YouTube}

github-actions · 2026-09-17T15:26:38Z

py_binary startup benchmark

Version	Time (ms)	vs BCR	vs main	Build (s)
BCR 1.11.7	173.7/174.0 ±1.9	—	—	1.75
main	59.9/59.5 ±2.4	-65.8%	—	1.75
PR	60.4/59.9 ±2.6	-65.6%	+0.7%	1.57

Time = mean/median ±stddev.
Measured with hyperfine --warmup 5 --runs 50 --shell=none on Linux
Gate: PR vs HEAD main median (threshold: 10%, and must exceed the 2×SE noise floor, here 1.7%). BCR is shown only as a historical baseline.
Build time: cold bazel build //:bench with isolated output base, no disk cache; external repos prefetched so network is excluded.

sys.path quality

Version	entries	sp roots
BCR 1.11.7	6	1
main	7	2
PR	7	2

sys.path quality measured by bench_syspath inside the assembled venv: sys.path entries, distinct site-packages roots, duplicate realpaths. Duplicates indicate symlink redundancy; many roots suggest an inefficient venv layout.

Bazel analysis benchmark

Version	Time (ms)	vs BCR	vs main	Targets	Actions	Configured (ext)	Cfg/target	Actions/cfg
BCR 2.0.0-alpha.6	1526/1527 ±242	—	—	404	20613	7380 (6675)	18.3	2.79
main	1508/1443 ±261	-5.5%	—	404	18363	7379 (6674)	18.3	2.49
PR	1534/1410 ±277	-7.7%	-2.3%	404	18363	7379 (6674)	18.3	2.49

Measured with hyperfine --warmup 1 --runs 10 on Linux
Time = mean/median ±stddev. Cfg = configured targets; (ext) = the count in external repos (the @pypi hub machinery).
Gate: PR vs HEAD main median (threshold: 10%, and must exceed the 2×SE noise floor, here 16.7%). BCR is shown only as a historical baseline.
Command: warm-server bazel build --nobuild //workspace/..., analysis cache discarded each run via a fresh --action_env value; no disk cache.
Workload: 100 py_test targets — 2 test files/package in file generation mode (one venv per test file); a single dep_group.

py_image_layer benchmark

Scenario	Version	Time (s)	Actions	vs BCR	vs main
analysis	BCR 2.0.0-alpha.6	0.60/0.57 ±0.11	747	—	—
analysis	main	0.59/0.57 ±0.12	668	+0.5%	—
analysis	PR	0.56/0.53 ±0.12	668	-7.1%	-7.6%
1p source	BCR 2.0.0-alpha.6	0.62/0.60 ±0.06	4	—	—
1p source	main	0.62/0.61 ±0.07	5	+1.3%	—
1p source	PR	0.63/0.61 ±0.07	5	+1.2%	-0.1%
3p wheel	BCR 2.0.0-alpha.6	0.71/0.71 ±0.04	5	—	—
3p wheel	main	0.76/0.74 ±0.06	6	+5.5%	—
3p wheel	PR	0.78/0.77 ±0.06	6	+9.1%	+3.4%

Time = mean/median ±stddev.
Measured with hyperfine on Linux, building //workspace:image_layers (10 binaries, ~30-wheel dep pool, grouped first-party/pip/interpreter tier) with isolated output base, no disk cache.
Scenarios: analysis = warm-server bazel build --nobuild, re-analyzed each run via a fresh --action_env value; incrementals run against a built state with warm analysis: source = append to the last package's lib.py, wheel = rewrite click post_install_patches content.
Actions: for Analysis, the total action count behind the image target from aquery deps(...); for incrementals, actions re-executed for the mutation, from a single instrumented run's BEP build metrics (deterministic; per-mnemonic breakdown in the *-actions.json artifacts). Informational only, not gated.
Gate: PR vs HEAD main median per scenario (threshold: 10%, and must exceed the 2×SE noise floor). BCR is shown only as a historical baseline.

`aspect ci bazelrc` regenerates the Workflows runner's ~/.bazelrc on every job using the CLI the checkout pins, and at 2026.38.14 that rc gained an enabled `aspect-github-actions` group carrying `--color=yes` — GitHub Actions renders ANSI but gives the job no TTY, so Bazel's `--color=auto` misdetects. It applies to every vanilla `bazel` call on the runner, including the ones these e2e scripts make. This test greps the captured output for the line `hello` exactly, and forced color prefixes it with an ANSI reset, so the match failed on output that was otherwise correct. Only this script compares a whole line; the rest match substrings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Names the PR that replaces it, aspect-build/aspect-cli#1465, at the top of the file rather than mid-paragraph, so whoever opens it next sees that the whole script is temporary before reading what it does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jbedard reviewed Sep 17, 2026

View reviewed changes

Comment thread .bazelrc

jbedard reviewed Sep 17, 2026

View reviewed changes

Comment thread e2e/crossbuild/.bazelrc

jbedard reviewed Sep 17, 2026

View reviewed changes

Comment thread .github/workflows/workspace_status.sh Outdated

jbedard approved these changes Sep 17, 2026

View reviewed changes

joshcbarnes marked this pull request as ready for review September 17, 2026 19:54

joshcbarnes and others added 2 commits September 17, 2026 13:01

joshcbarnes merged commit 6556cf0 into main Sep 17, 2026
103 checks passed

joshcbarnes deleted the feat/aspect-cloud-saas branch September 17, 2026 20:26

Navigation Menu

Uh oh!

ci: connect builds to Aspect Cloud - #1546

ci: connect builds to Aspect Cloud#1546
joshcbarnes merged 3 commits into
mainfrom
feat/aspect-cloud-saas

joshcbarnes commented Sep 17, 2026

Uh oh!

coderabbitai Bot commented Sep 17, 2026 •

edited

Loading

Review skipped

Uh oh!

aspect-workflows Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

joshcbarnes commented Sep 17, 2026

What it takes

Why test computes its own REMOTE_FLAG

Why the version bump

One rc, two workspaces

Verified locally

What this PR is meant to prove

Deliberately out of scope

Uh oh!

coderabbitai Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

aspect-workflows Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Aspect Workflows Tasks

Task Results

Uh oh!

github-actions Bot commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

py_binary startup benchmark

sys.path quality

Bazel analysis benchmark

py_image_layer benchmark

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Why `test` computes its own `REMOTE_FLAG`

coderabbitai Bot commented Sep 17, 2026 •

edited

Loading

aspect-workflows Bot commented Sep 17, 2026 •

edited

Loading

github-actions Bot commented Sep 17, 2026 •

edited

Loading