Skip to content

local-ci.md

Latest commit

 

History

History
306 lines (266 loc) · 17.2 KB

local-ci.md

File metadata and controls

306 lines (266 loc) · 17.2 KB
Raw

Local CI gate

pnpm check:local-ci proves what the hosted CI gate proves, on the development machine, in one command — including the full Node matrix. It exists because the hosted Verify work for one Node version takes ~13 minutes of runner time by current measurement (hosted CI splits it into a fast leg and two integration shards that run in parallel, see below). PR CI runs every leg on Node 24 plus the fast leg on Node 26, while pushes to main and workflow_dispatch run every leg on the full 22.19/24/26 matrix. The local gate keeps all three Node versions because local green is used to merge and must prove what the post-merge main run will prove. A many-core development machine can run all three legs plus the release gates concurrently. The local-merge workflow it enables:

  1. Run pnpm check:local-ci on the branch's HEAD commit.
  2. If the gate is green, the branch is mergeable — merge it.
  3. Hosted CI still runs on the merged commit (push to main) and stays the asynchronous post-merge safety net; if it disagrees with the local run, the hosted result wins and the merge gets a follow-up fix.

For quick iteration, pnpm check:local-ci --current-node-only runs a single Verify-equivalent leg on whatever Node is currently active, with the repo's normal local worker derivation. It skips the Node matrix and the examples/release/micro-eval gates, so it is a fast signal, not a merge gate.

Docs-only PRs skip the hosted Verify, examples, release-gates, micro-eval, host-install-proofs, and host-filesystem jobs. Docs-only means changes under docs/ or agent-patterns/, changeset markdown (.changeset/*.md), or top-level markdown. Nested markdown elsewhere is treated as code. Pushes to main never use this skip. The allowlist and fail-open listing checks are implemented by scripts/classify-docs-only.mjs and covered by packages/agent-bundle/tests/classify-docs-only.test.ts.

What it runs

Every leg is an isolated git worktree pinned to the HEAD commit (uncommitted changes are not covered — the runner warns), with its own node_modules and its own TMPDIR (<system tmp>/abci-<hash8>-<leg>, where <hash8> is derived from the repo root path; recreated every run). The temp roots live under the short system temp directory rather than the repo worktree because Chrome creates AF_UNIX sockets inside TMPDIR and the kernel caps socket paths at 108 bytes; the hash keeps concurrent runs from different checkouts from colliding. The private temp root keeps concurrent legs from observing each other's temp traffic: suites that assert temp-root hygiene (for example cli.test.ts scans os.tmpdir() for leaked agent-bundle-artifact-* directories) only ever see their own leg's directories, so a sibling leg's in-flight work cannot fail them — while a directory the leg itself leaks still fails its own scan. Rstest re-hashes that leg directory, worker ID, and invocation identity to /tmp/ab-rstest-<hash16> before exposing its worker TMPDIR; this leaves headroom below Linux's 108-byte sun_path cap for nested socket fixtures without sacrificing per-leg, per-worker, or concurrent-run isolation. Because those hashed roots live beside the leg directory rather than inside it, each one carries an owner marker (.ab-rstest-owner.json) naming the leg TMPDIR and process it was derived from; the runner removes the roots owned by a leg's TMPDIR — and only those, once their creating process has exited — before the leg starts (leftovers of an interrupted run) and after it finishes (scripts/rstest-worker-roots.mjs), so reruns cannot accumulate worker caches or interrupted-test fixtures under /tmp. Legs live under .worktrees/local-ci/ (gitignored), are reused across runs for warm caches, and can be recreated with --fresh.

The three Verify legs below mirror the hosted main-push matrix. Hosted CI runs each Node version as three parallel jobs — Verify (fast, Node N) (build, typecheck, lint, unit, route-unit, projection) and Verify (integration-1|2, Node N) (build, then one --shard N/2 of the integration pool) — fanned into the required Verify gate check; a local leg runs the same pools serially in one worktree, which proves the same union. On PRs, hosted CI runs every leg on Node 24 and the fast leg on Node 26.

Local leg Node Steps Mirrors hosted job
verify-node22 22.19.x install, playwright install chrome, build (publint runs inside each package's rslib build), typecheck, lint, test:unit, test:route-unit, test:projection, test:integration Verify (fast | integration-1 | integration-2, Node 22.19.0)
verify-node24 24.x same Verify (…, Node 24)
verify-node26 26.x same Verify (…, Node 26)
gates-node22 22.19.x install, examples:check, check:release, eval:spot examples-check, release-gates, rsc-runtime-micro-eval

The three hosted Node-22.19 jobs fold into one gates-node22 worktree because each of their entry scripts starts from pnpm build in a fresh install, which one worktree provides just as well as three.

gates-node22 runs the full check:release, a strict superset of the hosted per-PR release-gates job (check:release:ci): it additionally runs the scaffolder template matrix that the hosted side defers to the nightly packed-matrix job and to release-candidate.yml (Version Packages candidates and workflow_dispatch), so local green covers the per-PR, nightly, and release-candidate packed pools.

All four legs run concurrently. The summary table (leg × step × status × duration × test census) is printed and written to .worktrees/local-ci/summary.md (plus summary.json); per-step logs land in .worktrees/local-ci/logs/. The command exits non-zero if any step fails.

Real-host install proofs

The host-install proof suites (host-install-proof.test.ts, packed-host-install-proof.test.ts, dev-host-install.test.ts, dev-live-host.test.ts, and the packed Claude plugin validation in packed-native-smoke.test.ts) skip their Claude and Codex legs whenever the claude or codex CLI is not on PATH. That skip is what let #364 change the Codex interface.logo emission and break both proofs on main without CI noticing. Hosted CI therefore runs a dedicated host-install-proofs job (Node 22.19) on every PR and main push:

  1. node scripts/host-cli-pins.mjs print reads the pins — the hostCli block in packages/agent-bundle/src/adapters/schemas/claude/PROVENANCE.json (@anthropic-ai/claude-code) and .../schemas/codex/PROVENANCE.json (@openai/codex). Each pin must equal that file's observedCliVersion, so bumping the CLI CI runs is the same deliberate edit as re-pinning the schemas observed against it.
  2. node scripts/host-cli-pins.mjs install --prefix "$RUNNER_TEMP/host-cli" installs exactly those versions into a global npm prefix (cached by OS, architecture, and pin), then re-runs Claude Code's install.cjs when npm blocked its postinstall, which is what links the native binary.
  3. node scripts/host-cli-pins.mjs verify fails closed with one diagnostic line per host when claude --version or codex --version on PATH is not the pin.
  4. pnpm build, pnpm test:host-install, pnpm test:host-install:packed, and AGENT_BUNDLE_PACKAGE_PREBUILT=1 pnpm test:packed:native.

None of it needs a login or a secret: every proof runs against an isolated HOME / CLAUDE_CONFIG_DIR / CODEX_HOME, and the subcommands involved (claude plugin marketplace add, plugin install, plugin list --json, plugin details, plugin validate --strict; codex plugin marketplace add, plugin add, plugin list) work signed out. The job passes no workflow secrets.

To run the same proofs locally:

pnpm check:host-cli           # your PATH claude/codex must match the pins
pnpm build
pnpm test:host-install
pnpm test:host-install:packed
AGENT_BUNDLE_PACKAGE_PREBUILT=1 pnpm test:packed:native

If your machine tracks a newer CLI, install the pinned versions into a throwaway prefix and put it first on PATH for the run:

node scripts/host-cli-pins.mjs install --prefix /tmp/host-cli
PATH=/tmp/host-cli/bin:$PATH pnpm check:host-cli
PATH=/tmp/host-cli/bin:$PATH pnpm test:host-install

What stays binary-gated and login-gated, and therefore outside this job: the claude -p session proofs (AGENT_BUNDLE_HOST_INSTALL_CLAUDE_SESSION, AGENT_BUNDLE_NATIVE_CLAUDE_SMOKE), the Codex exec smoke (AGENT_BUNDLE_NATIVE_CODEX_SMOKE), and the packed Eval smokes (AGENT_BUNDLE_PACKED_NATIVE_{CLAUDE,CODEX}_SMOKE). Those still run only in the opt-in native-host-smoke workflow on a signed-in runner.

Host-install filesystem OS matrix

The published agent-bundle package has no os field: it is a platform-generic npm package, and dev --install-host already has Windows junction publish, atomic rename/rollback, ownership manifests, path-casing comparisons, and per-OS host config locations. Primary CI still runs Verify, examples, release-gates, and the pinned-CLI host-install proofs on ubuntu-latest only. Cloning that suite onto three OSes would multiply the PR critical path.

Hosted CI therefore adds one extra job, host-filesystem, on ubuntu-latest, macos-latest, and windows-latest (Node 22.19, engines floor — three cells, not a Node × OS product). It builds once and runs pnpm test:host-filesystem (rstest.host-filesystem.config.ts, which does not build the Workbench e2e example payload):

  • dev-host-install.test.ts (Claude/Codex legs skip when those CLIs are absent; this job does not install them)
  • dev-host-install-manager.test.ts (rollback and stale ownership)
  • install.test.ts / uninstall.test.ts (receipt ownership)
  • durable-fs.test.ts (atomic publish, Windows directory fsync)
  • internal-child-resolution-policy.test.ts (packaged child resolution, #769)
  • npm-cli-resolution.test.ts (Windows official + Unix/nvm + PATH + split-prefix/pnpm npm_execpath)
  • packed-install-bin.test.ts (packaged installer bin from a consumer cwd)
  • rstest-worker-isolation.test.ts (canonical TMPDIR; macOS /tmp → /private/tmp)

That is the slice that can actually diverge by OS. macOS matters because Claude, Codex, and Cursor authors commonly develop there; Windows is in the matrix because the code already has win32 branches, not because the rest of the suite is claimed green there. A green Host filesystem (windows-latest) job is evidence for this slice only.

To run the same slice locally after pnpm build:

pnpm test:host-filesystem

The local gate does not fan this across OSes — one machine can prove one OS. Hosted macOS and Windows results are the qualification for those runners.

Node provisioning

The runner introduces no new tooling. For each hosted runtime line (22.19.x, 24.x, 26.x) it resolves a Node binary from, in order:

  1. AGENT_BUNDLE_LOCAL_CI_NODE_22 / _24 / _26 — a Node binary or bin directory, for machines with bespoke layouts;
  2. mise where node@<line>;
  3. ~/.nvm/versions/node/*;
  4. the current process's Node, if it matches the line.

Every resolved binary is version-checked against the hosted line before use. If a line is missing, the runner fails with the exact install command (e.g. mise install node@22.19). pnpm itself is pinned by reusing the entrypoint that launched the runner, executed on each leg's own Node, so pnpm, its lifecycle children, and pnpm exec node all agree on the leg's runtime — node_modules trees (native modules such as the rspack bindings) are never shared across Node ABIs, while the content-addressed pnpm store is shared safely.

Parallelism and time budgets

The integration pool derives workers from cores (rstest.integration.config.ts), tuned for a leg that owns the machine. The runner instead slices the machine: with N concurrent Verify legs each leg gets min(4, cores / (2 N)) integration workers (AGENT_BUNDLE_INTEGRATION_MAX_WORKERS) and cores / N unit workers (--pool.maxWorkers), and full runs pin AGENT_BUNDLE_TEST_TIME_SCALE=4 — the same polling-budget scale hosted CI uses — because four legs sharing a machine is exactly the contention that scale exists for. Exporting AGENT_BUNDLE_TEST_TIME_SCALE yourself (e.g. when the machine is also running other heavy work) overrides the default; the integration config never lets it drop below what its own pool shape requires.

Load-sensitive failures are fixed at their cause, never absorbed with a per-test retry. The recurring shape is a test that acts before the product has published the state it is about to assert on; the fix is to wait on the product's own readiness signal. Precedents: the dev watcher's stat-signature dedupe (#122/#329), content-identity reload announcements (#200/#332), and the examples-real.e2e source edits, which used to pair a file write with an immediate manual rebuild and so raced the watcher's own rebuild of the same write for a second epoch. Those edits now go through replaceWatchedSourceAndAwaitRebuild (packages/agent-bundle/tests/support/watched-files.ts): one atomic replacement, then a wait on the coordinator's published build attempt, so one edit is exactly one build.

Infrastructure failures and their retry policy

One failure shape in hosted CI is registry or runner infrastructure, not the tree under test. It is never a reason to weaken a gate; where the failure happens before any repository code runs it gets a bounded retry, and everywhere else the policy is to re-run the job once the cause has cleared, then treat a repeat as a real signal.

  • Registry 5xx while setting up the job. Every hosted job installs pnpm, Node, and dependencies through the shared .github/actions/setup-workspace action. pnpm/setup@v2 downloads the pnpm executable from the npm registry as its first network call and exposes no retry input, so the action retries it once after 15 s, then runs pnpm install --frozen-lockfile up to three times (10 s, then 20 s back-off). A persistent outage still fails the job within about a minute of extra wall time. The trigger was 2026-09-04, when @pnpm/exe returned 504 Gateway Timeout on eight CI runs that were all green on rerun (#576).
  • Runner network stalls during npm install in the packed pool. The pool's consumer installs are cache-backed per worker (rstest.worker-isolation.ts): each worker pays for one cold download of the packed dependency tree (about 180 MB), and public-api-packed warms that cache in a beforeAll with its own budget so no per-test budget spans a cold network. A remaining timeout inside that beforeAll on a hosted runner is a registry or runner-network stall; re-run the job. A timeout in a test body after the warm-up is not: read the failure text, it names the step (install, tsc, CLI) that overran.

What is deliberately not covered

  • dependency-review runs as a GitHub-side action against the GitHub advisory database on the PR diff; it has no local equivalent and stays a hosted-only, PR-time check.
  • package-preview (pkg.pr.new) and the Release packages workflow have hosted publish-side effects. The latter also verifies the four remote commit URLs, which a local gate cannot reproduce.
  • host-install-proofs needs the pinned claude and codex CLIs on PATH (see Real-host install proofs). The local gate does not install host CLIs into its legs, so run those proofs by hand with the commands above when a change touches adapter emission, the installers, or the proof suites.
  • host-filesystem is the three-OS host-install/filesystem slice (see Host-install filesystem OS matrix). The local gate can run pnpm test:host-filesystem on the current OS after pnpm build; it cannot prove macOS or Windows from a Linux machine.
  • native-host-smoke needs signed-in Claude/Codex CLIs and is opt-in even on hosted CI.
  • Environment skew: Verify, examples, release-gates, micro-eval, and the pinned-CLI host-install proofs stay on ubuntu-latest. The host-filesystem job is the exception: ubuntu-latest, macos-latest, and windows-latest. Hosted Workbench browser suites launch Playwright's bundled Chromium — pinned by the Playwright version in the lockfile and selected with AGENT_BUNDLE_PLAYWRIGHT_CHANNEL=chromium (read by packages/workbench/tests/support/workbench-e2e.ts) — so the browser under test only changes with a commit. The local gate keeps the developer default, branded Google Chrome (playwright install chrome in its browsers step; the OS dependencies, --with-deps, are one-time machine setup and may need root). Export AGENT_BUNDLE_PLAYWRIGHT_CHANNEL=chromium after pnpm exec playwright install chromium to run the hosted browser locally. The one hosted suite still on the image's Chrome is examples/mcp-app's browser-app pool, because the shipped agentBundleBrowserRstest helper targets branded Chrome; the examples-check job records that Chrome version in its step summary. A green local run on a different distro, glibc, or browser build is strong but not identical evidence — this is the main reason hosted CI remains the post-merge safety net.
  • Job isolation: hosted gives every job a fresh VM; local legs reuse worktrees for speed. --fresh restores cold-start fidelity when staleness is suspected.