Toolchain A/B: nightly-2026-07-23 (control) vs nightly-2026-08-25 (candidate), via paired RCs

[bgonz808]

First real exercise of the #301 bump factory, using release candidates as the comparison vehicle rather than throwaway builds. Maintainer's proposal, and it is a better design than what #301 sketched.

Why RCs are the right vehicle

#301 described building the same source twice and diffing posture. Doing that with actual RCs buys several things a scratch build cannot:

• both artifacts go through the real release path — signed, attested, BinSkim-gated, caps-gated — so the comparison is between two things we would actually ship, not two lab specimens
• both are published and durable, so the diff is reproducible later by anyone, not a transient log
• it exercises the release pipeline against a new toolchain before a real release depends on it, which is the cheapest possible place to discover a codegen or hardening regression

The pair

	toolchain	purpose
v0.3.2-rc.2	nightly-2026-07-23 (current pin, age 33d)	control — isolates the toolchain as the only variable
v0.3.2-rc.3	latest nightly (2026-08-25 at time of writing)	candidate

v0.3.2-rc.1 (2026-08-21) is NOT the control: it predates the tool-fleet hardening (#306) and the caps/hardening ratchet, so it differs in more than the toolchain. A fresh rc.2 on the current pin is what makes this a clean A/B.

What gets diffed (all machinery already exists)

1. Advisories — xtask toolchain-advisories --channel <each> (harden(#267): toolchain vuln axis — Rust Security Response WG advisory watcher #298). Fail-closed; the candidate must be clean before anything else matters.
2. Caps manifest — import surface AND hardening (harden(#267): ratchet codegen hardening posture in the caps manifest #299). A toolchain that silently drops CFG or changes the CRT surface shows up here as a manifest violation, which is the entire point of the ratchet.
3. BinSkim rule dispositions — full --kind Fail Pass Review Open NotApplicable SARIF is already produced for both; BA2006 BuildWithSecureTools encodes the toolchain/linker version and the security features it emitted, so it is the single most direct evidence of a codegen posture change.
4. Binary facts — size, and VT panel. The VT half is worth watching given Sign the build tools — SEPARATE Azure identity from the hp-thermal release identity #307: if the ML heuristic reacts differently to the two toolchains, that is a real data point on the hardening/reputation question rather than speculation.

Gate

Per #301's governing rule: auto-fail on any axis, but never auto-promote. A fully-green candidate stops for maintainer sign-off, because "no axis objected" is the precondition for a decision, not the decision. Adopting the new toolchain means editing app/rust-toolchain.toml in a reviewed PR — merge is the promotion mechanism.

Prerequisite worth respecting

#301 argues TOOLCHAIN.lock (manifest-digest verification before rustup install) should land first, so the candidate channel is bytes rather than a name. Doing the A/B before that is still useful — it exercises the comparison machinery — but the candidate's provenance is weaker than the control's until it lands, and that limitation should be stated in whatever evidence this produces rather than glossed.

Sequencing

1. cut v0.3.2-rc.2 on the current pin (control)
2. TOOLCHAIN.lock for the candidate channel (Toolchain supply-chain: authenticity pinning (TOOLCHAIN.lock), bump discovery/automation, advisory monitoring #267) — or proceed and record the gap explicitly
3. cut v0.3.2-rc.3 on the latest nightly (candidate)
4. diff advisories / caps+hardening / BinSkim / VT
5. maintainer sign-off, then a reviewed rust-toolchain.toml bump

Related: #301 (factory), #267 (TOOLCHAIN.lock + advisories), #299 (hardening ratchet), #245 (caps), #307 (signing/reputation), #209 (REPRODUCIBLE.md records exact toolchain per release — the natural place to write the comparison down).

[bgonz808]

Corrected plan — by toolchain DATE, and the permutation matrix completed

Replacing the "old/new" language from the original write-up: those labels stop being readable the moment a second bump happens, and they hide which pin a given artifact was actually built with.

• nightly-2026-07-23 — the current pin (tools/TOOLCHAIN.lock, app/rust-toolchain.toml). The blessed tool fleet was built with it on 2026-08-25.
• nightly-2026-08-25 — the candidate (today's nightly, 33 days newer).

Two variables, four permutations

	tools built with	app built with	vehicle	isolates
A	07-23	07-23	v0.3.2-rc.2	control — built, awaiting sign-off
B	07-23	08-25	v0.3.2-rc.3	app codegen — the measurement that matters
C	08-25	07-23	dry-run, not an RC	tool-toolchain effect on the app binary
D	08-25	08-25	v0.3.2-rc.4 (optional)	end state

The original write-up only specced A, B and D. C is the missing intermediate and completes the 2x2.

Why C should NOT be a published RC

1. It is theoretically null. cargo-auditable is the only tool that touches release bytes (it performs the auditable build and injects .dep-v0); deny/audit/cyclonedx only gate or emit the SBOM. What cargo-auditable injects is a function of the LOCKFILE plus its own source, not of the compiler that built cargo-auditable.
2. The obvious test cannot answer the question. A byte-diff between A and C would be dominated by build non-determinism (build-tools: non-reproducible tool builds + clobber-republish re-digest ALL tools on any bump #176) — indistinguishable from "the tools made no difference".

But C is still worth running, because the stable properties are NOT noise-dominated: hp-thermal.exe's caps manifest (imports + hardening), BinSkim rule dispositions, and SBOM contents. If those are identical between A and C, the independence claim is confirmed by measurement rather than by argument.

Cheap vehicle: release.yml via workflow_dispatch, which defaults to dry_run: true — artifact, BinSkim and caps without publishing a release nobody needs.

Revised sequence

1. A — v0.3.2-rc.2 on 07-23 (built; paused at the release environment approval gate)
2. B — v0.3.2-rc.3 with app/rust-toolchain.toml + tools/TOOLCHAIN.lock moved to 08-25. Note both files must move together; the install action fails on drift between them (harden(#267): TOOLCHAIN.lock — pin the toolchain by content, and by one declaration #309).
3. producer run on 08-25 -> tool posture A/B via the caps ratchet -> bless
4. C — dry-run release on 08-25 tools + 07-23 app; diff caps / BinSkim / SBOM against A
5. D only if an end-to-end smoke test of the final state is wanted; B plus step 3 already cover the substance

Note from cutting A

The first tag push failed source-gate with 'ci' conclusion is 'null' — the tag was pushed before CI finished on the squash-merge commit, and the gate refused to release from a commit without provenance-verified green CI. Correct behaviour, and worth remembering for B: tag after CI completes on the merge commit, not immediately after merging.

[bgonz808]

Plan revised: the candidate was unsoaked, and the A/B cannot run as specced

rc.3 (#313) was cut on nightly-2026-08-25 — published the same day. Closed unmerged. A soak gate now enforces a minimum channel age (#315), and it blocks that channel outright.

Two findings that change this issue's shape

1. The 7-day crate soak was never the right bar for a toolchain, and 30 days is. Of Rust minor releases 1.80+, 8 of 19 (42%) needed a point release; fixes for miscompilation/unsoundness landed at +4, 7, 14, 14, 19, 19, 26, 28, 28, 29, 33 days. Residual exposure by age: 0d ~42%, 7d ~37%, 14d ~26%, 21d ~10%, 30d ~5%. Full table and sources are in tools/TOOLCHAIN.lock.

2. Under that policy the A/B has no useful candidate today. Our current pin nightly-2026-07-23 is 33 days old and already compliant. The newest adoptable nightly right now is roughly 2026-07-26 — three days newer than what we run, which is not a comparison worth cutting a release for. The A/B is therefore deferred until a meaningfully-newer nightly has soaked 30 days (from today, a channel dated on or after ~2026-07-26 becomes eligible as it ages; a genuinely different toolchain means waiting for the 08-xx series to reach 30 days, i.e. late September).

Worth preserving from the aborted attempt

rc.3's first CI run found a real toolchain-attributable difference: under 1.100.0-nightly, verify-artifact failed with pdb-link: cannot read hp-thermal.pdb (os error 2) — same workflow, same source, and 1.99.0-nightly emitted the PDB fine. Every hardening check passed; only the PDB was missing. That is exactly the class of change this A/B exists to surface, and it should be re-checked when a soaked 1.100-series candidate is evaluated.

Also: the matrix needs re-cutting for a different reason

/CETCOMPAT and -Z deny-partial-mitigations landed on main (#314), so rc.2 is no longer a valid control — it predates both. The A/B control must be re-cut once the tool fleet is rebuilt with the same hardening. Sequence is now:

1. producer run with /CETCOMPAT added to the tool RUSTFLAGS -> caps ratchet fires on CET_COMPAT -> review -> bless
2. re-cut the control RC on nightly-2026-07-23 with hardened tools AND hardened app
3. wait for a 30-day-soaked candidate toolchain
4. cut the candidate RC; diff caps+hardening / BinSkim / size / VT

Steps 1 and 2 are actionable now; 3 gates the rest.