Skip to content

Commit 1195041

Browse files
ci(host-proofs): run the binary-gated real-host install proofs on every PR with pinned CLIs
The host-install, packed host-install, and packed Claude plugin-validation proofs skip their Claude/Codex legs when the CLIs are absent, which let #364 break both proofs on main unnoticed (#367/#368 repaired them from local runs). - Pin @anthropic-ai/claude-code@2.1.250 and @openai/codex@0.147.0 as `hostCli` in each adapter's schema PROVENANCE.json; scripts/host-cli-pins.mjs reads the pins, refuses a pin that differs from observedCliVersion, installs them (re-running Claude's postinstall when npm blocks it), and fails closed with one diagnostic line when the CLI on PATH is not the pin. - Add the host-install-proofs CI job (cached CLI prefix, version gate, build, test:host-install, test:host-install:packed, test:packed:native) honoring the docs-only skip. No secrets or login are needed. - Harden the Codex interface proof: one shared pinned snapshot for both proof suites, installed manifest must equal the built artifact, validate against the pinned plugin schema, and reject fields outside the adapter's exported codexInterfaceFields. - Document the job and the local commands in docs/local-ci.md and README.
1 parent fcda871 commit 1195041

16 files changed

Lines changed: 704 additions & 31 deletions

File tree

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
"agent-bundle": patch
3+
---
4+
5+
Pin the Claude Code and Codex CLI versions the real-host install proofs run
6+
against beside each adapter's schema provenance (`hostCli` in
7+
`src/adapters/schemas/{claude,codex}/PROVENANCE.json`, kept equal to
8+
`observedCliVersion`), and export the Codex adapter's declared
9+
`codexInterfaceFields` so the host-install proofs can reject an undeclared
10+
`interface` emission before comparing against their single pinned snapshot.
11+
Repository CI now installs the pinned CLIs and runs the host-install, packed
12+
host-install, and packed Claude plugin-validation proofs on every change,
13+
signed out and without secrets. No runtime behavior changes.

‎.github/workflows/ci.yml‎

Lines changed: 59 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,63 @@ jobs:
175175
# the nightly packed-matrix job and in pre-publish `pnpm check:release`.
176176
- run: pnpm check:release:ci
177177

178+
# Binary-gated real-host install proofs. The proof suites skip their Claude
179+
# and Codex legs when the CLIs are absent, which is how #364 changed the
180+
# Codex `interface.logo` emission and broke both proofs on main without CI
181+
# noticing (#367/#368 repaired them from local runs). This job installs the
182+
# exact CLI versions pinned in each adapter's schema PROVENANCE.json
183+
# (`hostCli`, kept equal to `observedCliVersion` by scripts/host-cli-pins.mjs),
184+
# fails closed if `claude`/`codex --version` differs from the pin, and runs
185+
# the source-built proofs, the packed-tarball proofs, and the packed Claude
186+
# plugin validation. Nothing here needs a login: every proof runs against an
187+
# isolated HOME / CLAUDE_CONFIG_DIR / CODEX_HOME and no secrets are passed.
188+
# Signed-in smokes (`claude -p`, `codex exec`, the Eval harnesses) stay in
189+
# the opt-in native-host-smoke workflow.
190+
host-install-proofs:
191+
needs: changes
192+
if: >-
193+
${{ !cancelled() && github.event_name != 'schedule' &&
194+
(github.event_name != 'pull_request' || needs.changes.outputs.docs_only != 'true') }}
195+
name: Host install proofs (Node 22.19)
196+
runs-on: ubuntu-latest
197+
timeout-minutes: 30
198+
steps:
199+
- uses: actions/checkout@v7
200+
- uses: pnpm/setup@v2
201+
with:
202+
cache: true
203+
install: false
204+
runtime: node@22.19.0
205+
- run: pnpm install --frozen-lockfile
206+
- name: Read pinned host CLI versions
207+
id: pins
208+
run: node scripts/host-cli-pins.mjs print
209+
# The global npm prefix holding both CLIs (Claude Code and Codex ship
210+
# platform-native binaries), keyed by OS, arch, and the exact pins so a
211+
# deliberate re-pin misses the cache and installs fresh.
212+
- name: Restore pinned host CLIs
213+
id: host-cli-cache
214+
uses: actions/cache@v6
215+
with:
216+
path: ${{ runner.temp }}/host-cli
217+
key: host-cli-${{ runner.os }}-${{ runner.arch }}-${{ steps.pins.outputs.pins }}
218+
- name: Install pinned host CLIs
219+
if: steps.host-cli-cache.outputs.cache-hit != 'true'
220+
run: node scripts/host-cli-pins.mjs install --prefix "$RUNNER_TEMP/host-cli"
221+
- name: Expose host CLIs on PATH
222+
run: echo "$RUNNER_TEMP/host-cli/bin" >> "$GITHUB_PATH"
223+
- name: Verify host CLI versions match the schema pins
224+
run: node scripts/host-cli-pins.mjs verify
225+
- run: pnpm build
226+
- name: Host install proofs (source-built bundle)
227+
run: pnpm test:host-install
228+
- name: Host install proofs (packed tarball)
229+
run: pnpm test:host-install:packed
230+
- name: Packed Claude plugin validation proof
231+
env:
232+
AGENT_BUNDLE_PACKAGE_PREBUILT: '1'
233+
run: pnpm test:packed:native
234+
178235
# Release-boundary scaffolder template matrix (mcp-server, cli-tool) plus
179236
# the full packed pool — the nightly form of pre-publish `check:release`.
180237
packed-matrix:
@@ -207,7 +264,8 @@ jobs:
207264
rsc-runtime-micro-eval:
208265
# Deterministic end-to-end spot-check of the built RSC runtime artifacts
209266
# (hook -> RSC worker -> shared kernel state -> MCP tool lowering) without
210-
# any real Claude/Codex host. Real native-host smokes stay skip-gated in
267+
# any real Claude/Codex host. Login-free real-host install proofs run in
268+
# the host-install-proofs job; signed-in native smokes stay skip-gated in
211269
# the manually dispatched native-host-smoke workflow on purpose.
212270
needs: changes
213271
if: >-

‎.github/workflows/native-host-smoke.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,10 @@ concurrency:
2020
jobs:
2121
native-host-smoke:
2222
# This runs only on a trusted runner with existing CLI login state; no workflow secrets are passed.
23+
# The login-free subset (host install proofs, packed host install proofs, packed Claude plugin
24+
# validation) also runs on every PR and main push in ci.yml's host-install-proofs job against the
25+
# CLI versions pinned in the adapter schema PROVENANCE files; this workflow adds the signed-in
26+
# `claude -p` / `codex exec` evidence on top, against whatever CLI the trusted runner has.
2327
runs-on: self-hosted
2428
strategy:
2529
fail-fast: false

‎docs/local-ci.md‎

Lines changed: 66 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@ Verify-equivalent leg on whatever Node is currently active, with the repo's
2121
normal local worker derivation. It skips the Node matrix and the
2222
examples/release/micro-eval gates, so it is a fast signal, not a merge gate.
2323

24-
Docs-only PRs skip the hosted Verify, examples, release-gates, and micro-eval
25-
jobs. Docs-only means changes under `docs/` or `agent-patterns/`, changeset
24+
Docs-only PRs skip the hosted Verify, examples, release-gates, micro-eval, and
25+
host-install-proofs jobs. Docs-only means changes under `docs/` or `agent-patterns/`, changeset
2626
markdown (`.changeset/*.md`), or top-level markdown. Nested markdown elsewhere
2727
is treated as code. Pushes to `main` never use this skip. The allowlist and
2828
fail-open listing checks are implemented by `scripts/classify-docs-only.mjs`
@@ -75,6 +75,65 @@ duration × test census) is printed and written to
7575
`.worktrees/local-ci/summary.md` (plus `summary.json`); per-step logs land in
7676
`.worktrees/local-ci/logs/`. The command exits non-zero if any step fails.
7777

78+
## Real-host install proofs
79+
80+
The host-install proof suites (`host-install-proof.test.ts`,
81+
`packed-host-install-proof.test.ts`, `dev-host-install.test.ts`, and the
82+
packed Claude plugin validation in `packed-native-smoke.test.ts`) skip their
83+
Claude and Codex legs whenever the `claude` or `codex` CLI is not on PATH.
84+
That skip is what let #364 change the Codex `interface.logo` emission and
85+
break both proofs on `main` without CI noticing. Hosted CI therefore runs a
86+
dedicated `host-install-proofs` job (Node 22.19) on every PR and `main` push:
87+
88+
1. `node scripts/host-cli-pins.mjs print` reads the pins — the `hostCli`
89+
block in `packages/agent-bundle/src/adapters/schemas/claude/PROVENANCE.json`
90+
(`@anthropic-ai/claude-code`) and `.../schemas/codex/PROVENANCE.json`
91+
(`@openai/codex`). Each pin must equal that file's `observedCliVersion`,
92+
so bumping the CLI CI runs is the same deliberate edit as re-pinning the
93+
schemas observed against it.
94+
2. `node scripts/host-cli-pins.mjs install --prefix "$RUNNER_TEMP/host-cli"`
95+
installs exactly those versions into a global npm prefix (cached by OS,
96+
architecture, and pin), then re-runs Claude Code's `install.cjs` when npm
97+
blocked its postinstall, which is what links the native binary.
98+
3. `node scripts/host-cli-pins.mjs verify` fails closed with one diagnostic
99+
line per host when `claude --version` or `codex --version` on PATH is not
100+
the pin.
101+
4. `pnpm build`, `pnpm test:host-install`, `pnpm test:host-install:packed`,
102+
and `AGENT_BUNDLE_PACKAGE_PREBUILT=1 pnpm test:packed:native`.
103+
104+
None of it needs a login or a secret: every proof runs against an isolated
105+
`HOME` / `CLAUDE_CONFIG_DIR` / `CODEX_HOME`, and the subcommands involved
106+
(`claude plugin marketplace add`, `plugin install`, `plugin list --json`,
107+
`plugin details`, `plugin validate --strict`; `codex plugin marketplace add`,
108+
`plugin add`, `plugin list`) work signed out. The job passes no workflow
109+
secrets.
110+
111+
To run the same proofs locally:
112+
113+
```sh
114+
pnpm check:host-cli # your PATH claude/codex must match the pins
115+
pnpm build
116+
pnpm test:host-install
117+
pnpm test:host-install:packed
118+
AGENT_BUNDLE_PACKAGE_PREBUILT=1 pnpm test:packed:native
119+
```
120+
121+
If your machine tracks a newer CLI, install the pinned versions into a
122+
throwaway prefix and put it first on PATH for the run:
123+
124+
```sh
125+
node scripts/host-cli-pins.mjs install --prefix /tmp/host-cli
126+
PATH=/tmp/host-cli/bin:$PATH pnpm check:host-cli
127+
PATH=/tmp/host-cli/bin:$PATH pnpm test:host-install
128+
```
129+
130+
What stays binary-gated *and* login-gated, and therefore outside this job:
131+
the `claude -p` session proofs (`AGENT_BUNDLE_HOST_INSTALL_CLAUDE_SESSION`,
132+
`AGENT_BUNDLE_NATIVE_CLAUDE_SMOKE`), the Codex `exec` smoke
133+
(`AGENT_BUNDLE_NATIVE_CODEX_SMOKE`), and the packed Eval smokes
134+
(`AGENT_BUNDLE_PACKED_NATIVE_{CLAUDE,CODEX}_SMOKE`). Those still run only in
135+
the opt-in `native-host-smoke` workflow on a signed-in runner.
136+
78137
## Node provisioning
79138

80139
The runner introduces no new tooling. For each hosted runtime line
@@ -116,6 +175,11 @@ never lets it drop below what its own pool shape requires.
116175
hosted-only, PR-time check.
117176
- **package-preview** (pkg.pr.new) and the **release publish** workflow are
118177
publish-side effects, not checks; nothing about them gates a merge.
178+
- **host-install-proofs** needs the pinned `claude` and `codex` CLIs on PATH
179+
(see [Real-host install proofs](#real-host-install-proofs)). The local gate
180+
does not install host CLIs into its legs, so run those proofs by hand with
181+
the commands above when a change touches adapter emission, the installers,
182+
or the proof suites.
119183
- **native-host-smoke** needs signed-in Claude/Codex CLIs and is opt-in even
120184
on hosted CI.
121185
- **Environment skew**: hosted runners are `ubuntu-latest`, and hosted jobs

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
"typecheck": "tsc --noEmit && tsc --project packages/workbench/tsconfig.json && tsc --project packages/create-agent-bundle/tsconfig.json",
2525
"check": "pnpm build && pnpm test:unit && pnpm test:route-unit && pnpm test:projection && pnpm test:integration:run && pnpm lint && pnpm typecheck",
2626
"check:local-ci": "node scripts/local-ci.mjs",
27+
"check:host-cli": "node scripts/host-cli-pins.mjs verify",
2728
"docs:runtime-topology": "node scripts/rsc-runtime-topology.mjs --root . --output docs/architecture/rsc-runtime-workbench.md",
2829
"eval:spot": "pnpm build && pnpm --filter @agent-bundle/rsc-agent-runtime-demo build && pnpm --filter @agent-bundle/rsc-agent-runtime-demo exec rstest run tests/micro-eval.spot.test.ts --config rstest.config.ts",
2930
"check:runtime-topology": "node scripts/rsc-runtime-topology.mjs --root . --output docs/architecture/rsc-runtime-workbench.md --check",

‎packages/agent-bundle/README.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -384,7 +384,12 @@ prove native-host install or dispatch, or an install mode that copies the
384384
artifact elsewhere. `host-install` is separate installed-layout process
385385
evidence: its deterministic adapter-simulator lane is unconditional, available
386386
Claude and Codex binaries also prove their public install paths, and Cursor
387-
records its unavailable non-interactive host-session surface explicitly.
387+
records its unavailable non-interactive host-session surface explicitly. The
388+
Claude and Codex legs skip when those binaries are absent, so the repository's
389+
CI runs them on every change against the exact CLI versions pinned beside each
390+
adapter's schema provenance (`hostCli` in `src/adapters/schemas/*/PROVENANCE.json`),
391+
signed out and with no secrets; only the `claude -p` session and Eval smokes
392+
remain login-gated.
388393

389394
### Contract matrix (`runContractMatrix` / `runPackedContractMatrix` / `runInstalledHostContractMatrix`)
390395

‎packages/agent-bundle/src/adapters/codex.ts‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,12 @@ const isAbsoluteUrl = (value: unknown): value is string => {
212212
}
213213
};
214214

215-
const codexInterfaceFields = Object.freeze([
215+
/**
216+
* Every `interface` field the adapter can emit — generated or authored through
217+
* the `codex.interface` extension. The host-install proofs check installed
218+
* manifests against this declared set so an undeclared emission cannot land.
219+
*/
220+
export const codexInterfaceFields = Object.freeze([
216221
'brandColor',
217222
'capabilities',
218223
'category',

0 commit comments

Comments
 (0)