Make any release failure turn the run red

[MelbourneDeveloper]

TL;DR

The release pipeline had five ways to publish nothing and still report success. v0.17.0 went out through all of them intact. This makes every one of them a red run, and adds a backstop for the failure mode none of them covered: a job that never runs at all.

Details

A release is either published or it is not. These each downgraded "did not publish" to something quieter:

Where	What it swallowed
build / vsix matrices	continue-on-error on win32 — a Windows binary that never built was a warning inside a green run
Package step	cp compiler/lib/lib*.a ... || true — a build producing no runtime archives would package, publish and reach Homebrew as a compiler that can link nothing
preflight	a missing publish token skipped its channel with a warning, so a release reaching neither Homebrew, Scoop nor Open VSX passed while brew install served the previous version
brew / scoop	git commit ... || exit 0 swallowed every commit failure along with the nothing-to-commit case, and skipped the push with it
deploy-webcompiler	non-fatal by design — which is how the live playground served a stale build for weeks behind green releases

All five now fail the run. Nothing cancels: fail-fast: false stays on both matrices, so a broken leg still lets its siblings finish and one run shows every platform's result. It just ends red.

The backstop. Removing the swallows does not address the failure mode that survives: GitHub scores a skipped job as green, so a wrong if:, an unset scope output or a cancelled dependency publishes nothing and still succeeds. The new release-complete job runs if: always() after every other job and fails unless each channel this tag requires actually succeeded. It is driven by the same four scope outputs the jobs' own if: conditions use, so "ran" and "should have run" cannot drift apart.

Two same-class defects outside the release path are fixed too: the || true hiding a failed cargo install cargo-llvm-cov behind a confusing error several steps later, and the one on the Android sdkmanager lookup, now a directory test with a real diagnostic.

docs/RELEASING.md gains a section stating the policy, and its secrets table is corrected — it named VSCE_PAT, TAP_TOKEN and SCOOP_BUCKET_TOKEN, none of which exist. The real credentials are BREW_SCOOP_PAT, OPEN_VSX_PAT, AZURE_CLIENT_ID/AZURE_TENANT_ID and FLY_API_TOKEN. It also still advertised a darwin-x64 build leg that was dropped, and a SKIP_VSCE_PUBLISH variable nothing reads.

How Do The Automated Tests Prove It Works?

The backstop's logic is executed, not eyeballed. scripts/test-release-gate.py extracts the release-complete shell body out of release.yml — the real one, so there is no second copy to drift — and runs it under bash against 13 fabricated sets of job outcomes:

• a full release with everything published → passes
• Homebrew silently skipped, a build leg failed, a job cancelled, the web compiler deploy failed → each fails
• a website-only tag whose site deployed → passes; the same tag with the site silently skipped → fails
• a vsix-only tag whose extension published → passes; the same with the Marketplace publish skipped → fails
• a prerelease correctly leaving the live site alone → passes; the same prerelease with Open VSX skipped → fails
• a job with no if: at all that did not run → fails

Six mutations of the gate were each confirmed to turn the suite red: dropping the unconditional jobs from required, no longer distinguishing skipped from success, no longer counting cancelled as broken, ignoring the website output, always exiting 0, and renaming the step out from under the test. The cancelled mutation initially slipped through, which exposed a real gap — a cancelled job the tag does not require is caught only by the failure sweep, and no case covered that. Case 7 now pins it, and the mutation is caught.

The swallows cannot come back. scripts/verify-release-gates.mjs scans every workflow and fails on continue-on-error, || true or || exit 0 in a run step, on any release job missing from release-complete's needs:, on a renamed or deleted backstop, on a backstop without if: always(), and on a reviewed tolerance that no longer matches anything. Eight regressions were each confirmed caught — including adding a plausible new publish-snap job and forgetting to wire it in — with the workflow tree checksummed back to identical after every probe. The single surviving || true, the first-release tag lookup in release-change-detection.yml, is in the script's reviewed list with the reason it is safe; deleting the tolerance without deleting the entry fails the build.

Both run in make lint and as a step in the already-required Build, Format & Analyse job, so the pinned context list in verify-branch-protection.mjs and the ruleset stay in agreement — no new required check, no ruleset edit.

actionlint is clean across all six workflow files.

🤖 Generated with Claude Code