Skip to content

Commit a5beac1

Browse files
committed
fix(release): the Releases backfill skips a version whose publish is in flight in another run
release-integrity's audit now asks `release-pending-publish.mjs in-flight` before it sets releases-missing. npm settles before the publish job reaches its own "Create GitHub Releases" step, so a landing audited in that window used to write the same Releases beside it (17.6.0: run 36958423332 against the publish of run 36955885276). An in-flight publish, or an answer that cannot be read, leaves releases-missing unset with a notice or warning; the step stays green and the image request is untouched. The job gains `actions: read`. release-verify-npm.mjs batteries 12 and 13 run the audit step's own text, so their stubs now answer the Actions read; battery 12 gains seven cases pinning the branch (in flight, unreadable, another version, the control, and no read on the common path). Claude-Session: https://claude.ai/code/session_01T9u38rswFp5Rw8DswRUReJ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4f2be9a commit a5beac1

2 files changed

Lines changed: 125 additions & 7 deletions

File tree

‎.github/workflows/release.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -790,6 +790,10 @@ jobs:
790790
# `contents: write` is for GitHub Releases, never for refs: this job runs
791791
# no `git push` of any kind.
792792
contents: write
793+
# `actions: read` is the Releases backfill's guard: the audit reads this
794+
# workflow's runs and their jobs to ask whether the same version's
795+
# publish is in flight in another run. Reads only, never a cancel.
796+
actions: read
793797
outputs:
794798
# "the docker job must build" — set only when npm ALREADY has this
795799
# version's WHOLE fixed group and its runtime image is missing.
@@ -1066,9 +1070,33 @@ jobs:
10661070
;;
10671071
esac
10681072
1073+
# ── the publish in flight, before any Releases backfill ───────────
1074+
# npm settles BEFORE the publish job reaches its own "Create GitHub
1075+
# Releases" step, so a landing audited in that window found the group
1076+
# complete and wrote the same Releases beside it: on 17.6.0 the
1077+
# publish of run 36955885276 wrote them 03:03:47Z -> 03:05:42Z, this
1078+
# backfill in run 36958423332 started at 03:04:37Z, and five tags got
1079+
# two Release objects each. The two writers are in different runs,
1080+
# so no `needs:` can order them; the publish's own run creates its
1081+
# Releases and D4 asset, and a landing after it finishes backfills
1082+
# whatever it did not. scripts/release-pending-publish.mjs `in-flight`
1083+
# (its --self-test, run by lint.yml, holds the rule) answers
1084+
# `in-flight` on anything it cannot read, so nothing is backfilled off
1085+
# a guess. It only ever leaves `releases-missing` unset: this step
1086+
# stays green and the image request below is not its business.
10691087
if [ "$releases_ok" = true ]; then
10701088
echo "GitHub Releases + ADR-0087 D4 asset are present for ${version}."
1089+
elif ! flight=$(GITHUB_TOKEN="$GH_TOKEN" node scripts/release-pending-publish.mjs in-flight --version "$version" --version-commit "$version_commit" --head "$SHA" --workflow release.yml); then
1090+
echo "::warning::${version}'s GitHub Releases or ADR-0087 D4 asset are incomplete, and whether its publish is still in flight could not be asked (reason above). Nothing is backfilled off a guess; the next landing reads again."
1091+
elif [ "$(jq -r '.state' <<<"$flight")" != 'clear' ]; then
1092+
jq -c . <<<"$flight"
1093+
if [ "$(jq -r '.reason' <<<"$flight")" = 'publish-in-flight' ]; then
1094+
echo "::notice::${version}'s GitHub Releases or ADR-0087 D4 asset are incomplete, but its publish is still in flight ($(jq -r '.detail' <<<"$flight")). That run creates them; nothing is backfilled beside it. A landing after it finishes backfills whatever it did not."
1095+
else
1096+
echo "::warning::${version}'s GitHub Releases or ADR-0087 D4 asset are incomplete, and whether its publish is still in flight could not be read ($(jq -r '.detail' <<<"$flight")). Nothing is backfilled off a guess; the next landing reads again."
1097+
fi
10711098
else
1099+
jq -c . <<<"$flight"
10721100
echo "::warning::${version} is on npm but its GitHub Releases or the ADR-0087 D4 asset are incomplete (#4900) — backfilling."
10731101
echo "releases-missing=true" >> "$GITHUB_OUTPUT"
10741102
fi

‎scripts/release-verify-npm.mjs‎

Lines changed: 97 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -142,8 +142,10 @@
142142
*
143143
* The self-test's battery 12 runs the audit step itself — its text, read out of
144144
* `.github/workflows/release.yml` — against a throwaway repository, a stub
145-
* registry and stub `npm` / `gh` / `curl`, so the wiring is pinned where it
146-
* lives rather than described here.
145+
* registry, a stub Actions API (the audit asks whether the version's publish
146+
* is in flight in another run before it backfills any Release) and stub
147+
* `npm` / `gh` / `curl`, so the wiring is pinned where it lives rather than
148+
* described here.
147149
*
148150
* Battery 13 pins WHICH TREE that backfill builds from (#20982). The audit runs
149151
* on `github.sha`, the head of whichever push is audited, while the publish job
@@ -203,7 +205,7 @@ const SELF_TEST_BATTERIES = Object.freeze({
203205
'9. The job summary is written SYNCHRONOUSLY, before process.exit': 4,
204206
'10. The probe: the whole group in one read, three answers': 9,
205207
'11. The 17.5.0 publish window, replayed on npm\'s own clock': 7,
206-
'12. The audit step backfills only a version whose whole group is on npm': 17,
208+
'12. The audit step backfills only a version whose whole group is on npm, and no publish of it is in flight': 24,
207209
'13. The backfill builds from the version commit\'s tree, as the publish does': 11,
208210
});
209211

@@ -936,6 +938,47 @@ async function stubRegistryServer() {
936938
};
937939
}
938940

941+
/**
942+
* The Actions API the audit's in-flight read asks (`release-pending-publish.mjs
943+
* in-flight`), on 127.0.0.1. `set({ publishing, broken })`: `publishing` names
944+
* a version whose publish job is `in_progress` in run 4242, which only the
945+
* `in_progress` run-list filter lists; `broken` answers every request 503.
946+
* Records every request, so a case can assert the audit asked, or did not.
947+
*/
948+
async function stubActionsApi() {
949+
let state = { publishing: null, broken: false };
950+
const asked = [];
951+
const server = createServer((req, res) => {
952+
const url = new URL(String(req.url), 'http://stub.invalid');
953+
asked.push(`${req.method} ${url.pathname}${url.search}`);
954+
const send = (status, body) => {
955+
res.writeHead(status, { 'content-type': 'application/json' });
956+
res.end(JSON.stringify(body));
957+
};
958+
if (state.broken || req.method !== 'GET') return send(503, { message: 'unavailable' });
959+
const run = { id: 4242, event: 'push', status: 'in_progress' };
960+
if (url.pathname.endsWith('/actions/workflows/release.yml/runs')) {
961+
const runs = state.publishing && url.searchParams.get('status') === 'in_progress' ? [run] : [];
962+
return send(200, { total_count: runs.length, workflow_runs: runs });
963+
}
964+
if (state.publishing && url.pathname.endsWith('/actions/runs/4242')) return send(200, run);
965+
if (state.publishing && url.pathname.endsWith('/actions/runs/4242/jobs')) {
966+
return send(200, { total_count: 1, jobs: [{ name: `Publish ${state.publishing} to npm (awaiting approval)`, status: 'in_progress' }] });
967+
}
968+
return send(404, { message: 'Not Found' });
969+
});
970+
await new Promise((resolve_) => { server.listen(0, '127.0.0.1', resolve_); });
971+
return {
972+
url: `http://127.0.0.1:${server.address().port}`,
973+
asked,
974+
set(next) {
975+
state = { publishing: next.publishing ?? null, broken: next.broken ?? false };
976+
asked.length = 0;
977+
},
978+
close: () => new Promise((resolve_) => { server.close(resolve_); }),
979+
};
980+
}
981+
939982
/**
940983
* Stub `npm` / `gh` / `curl` for the audit step, each answering from env:
941984
* `npm view NAME@V version` from STUB_NPM_PRESENT (the same set the registry
@@ -1227,6 +1270,14 @@ async function stubReleasesApi() {
12271270
req.on('data', (chunk) => { data += chunk; });
12281271
req.on('end', () => {
12291272
const url = String(req.url);
1273+
// The audit's in-flight read (`release-pending-publish.mjs in-flight`):
1274+
// this battery's release has no publish in flight, so every run list is
1275+
// empty and the audit goes on to request the backfill.
1276+
if (req.method === 'GET' && /\/actions\/workflows\/release\.yml\/runs\?/.test(url)) {
1277+
res.writeHead(200, { 'content-type': 'application/json' });
1278+
res.end('{"total_count":0,"workflow_runs":[]}');
1279+
return;
1280+
}
12301281
if (req.method === 'GET' && url.includes('/releases/tags/')) {
12311282
res.writeHead(404, { 'content-type': 'application/json' });
12321283
res.end('{"message":"Not Found"}');
@@ -1599,7 +1650,7 @@ export async function selfTest() {
15991650
}
16001651

16011652
// ── 12. The audit step itself ─────────────────────────────────────────────
1602-
battery('12. The audit step backfills only a version whose whole group is on npm');
1653+
battery('12. The audit step backfills only a version whose whole group is on npm, and no publish of it is in flight');
16031654
{
16041655
// The step's REAL text, read out of release.yml, run by bash the way
16051656
// Actions runs it, in a throwaway repository: base (1.0.0) -> the version
@@ -1624,6 +1675,7 @@ export async function selfTest() {
16241675

16251676
const root = mkdtempSync(join(tmpdir(), 'release-verify-npm-audit-'));
16261677
const registry = await stubRegistryServer();
1678+
const actions = await stubActionsApi();
16271679
try {
16281680
const repo = join(root, 'repo');
16291681
mkdirSync(repo);
@@ -1673,19 +1725,22 @@ export async function selfTest() {
16731725
if (script !== null) writeFileSync(scriptFile, script);
16741726

16751727
let runs = 0;
1676-
const audit = async ({ event = 'push', before, head, present, broken = [], releases = false, image = false }) => {
1728+
const audit = async ({ event = 'push', before, head, present, broken = [], releases = false, image = false, flight = {} }) => {
16771729
runs += 1;
16781730
const temp = join(root, `run-${runs}`);
16791731
mkdirSync(temp);
16801732
g('checkout', '-q', '--detach', head);
16811733
registry.set({ present, broken });
1734+
actions.set(flight);
16821735
const env = {
16831736
PATH: `${bin}:${dirname(process.execPath)}:${process.env.PATH ?? ''}`,
16841737
HOME: process.env.HOME ?? root,
16851738
SHA: head,
16861739
EVENT: event,
16871740
BEFORE: before ?? '',
16881741
GH_TOKEN: 'stub',
1742+
GITHUB_API_URL: actions.url,
1743+
GITHUB_RUN_ID: '9999',
16891744
GITHUB_REPOSITORY: 'objectstack-ai/objectstack',
16901745
GITHUB_OUTPUT: join(temp, 'output'),
16911746
GITHUB_STEP_SUMMARY: join(temp, 'summary.md'),
@@ -1697,7 +1752,7 @@ export async function selfTest() {
16971752
};
16981753
const r = script === null ? { status: -1, stdout: '', stderr: 'no script' } : await runAsActions({ scriptFile, cwd: repo, env });
16991754
const summary = existsSync(env.GITHUB_STEP_SUMMARY) ? readFileSync(env.GITHUB_STEP_SUMMARY, 'utf8') : '';
1700-
return { ...r, outputs: readOutputs(env.GITHUB_OUTPUT), summary, asked: [...registry.asked] };
1755+
return { ...r, outputs: readOutputs(env.GITHUB_OUTPUT), summary, asked: [...registry.asked], actionsAsked: [...actions.asked] };
17011756
};
17021757
const said = (r) => `exit ${r.status}; outputs ${JSON.stringify(r.outputs)}; asked ${JSON.stringify(r.asked)}; ${r.stderr.trim().split('\n').slice(-2).join(' | ')}`;
17031758
const backfills = (r) => r.outputs['image-missing'] === 'true' || r.outputs['releases-missing'] === 'true';
@@ -1764,8 +1819,42 @@ export async function selfTest() {
17641819
const later = await audit({ before: landing, head: freshLanding, present: ['@objectstack/cli@1.1.0', '@objectstack/spec@1.1.0'] });
17651820
t("a later landing's new package does not hold the backfill back: the image is requested", later.status === 0 && later.outputs['image-missing'] === 'true', said(later));
17661821
t('...because the group read is the version commit\'s: the new package is never asked about', later.asked.length === 2 && !later.asked.includes('@objectstack/fresh'), said(later));
1822+
1823+
// The 17.6.0 race: npm settled before the publish job reached its own
1824+
// Releases step, and a landing's audit wrote the same Releases beside it.
1825+
// The two writers are in different runs, so the audit asks the Actions
1826+
// API whether this version's publish is in flight elsewhere.
1827+
const whole = ['@objectstack/cli@1.1.0', '@objectstack/spec@1.1.0'];
1828+
const flying = await audit({ before: vc, head: landing, present: whole, flight: { publishing: '1.1.0' } });
1829+
t('PUBLISH IN FLIGHT in another run (the whole group on npm, Releases missing): the audit stays green', flying.status === 0, said(flying));
1830+
t('...and backfills NO GitHub Release beside it', flying.outputs['releases-missing'] === undefined, said(flying));
1831+
t(
1832+
'...and says why, naming the run the publish is in flight in',
1833+
/::notice::1\.1\.0's GitHub Releases .* its publish is still in flight .*run 4242/.test(flying.stdout),
1834+
said(flying),
1835+
);
1836+
t(
1837+
'CONTROL: the backfill above passed THROUGH the in-flight read, which found nothing in flight',
1838+
done.actionsAsked.some((a) => /\/actions\/workflows\/release\.yml\/runs\?status=in_progress/.test(a)) && done.actionsAsked.every((a) => a.startsWith('GET ')),
1839+
JSON.stringify(done.actionsAsked),
1840+
);
1841+
const other = await audit({ before: vc, head: landing, present: whole, flight: { publishing: '1.2.0' } });
1842+
t('a publish of ANOTHER version in flight does not hold this one back: the Releases are backfilled', other.outputs['releases-missing'] === 'true', said(other));
1843+
const unreadable = await audit({ before: vc, head: landing, present: whole, flight: { broken: true } });
1844+
t(
1845+
'the Actions API unreadable: the audit stays green, backfills no GitHub Release off a guess, and a warning says so',
1846+
unreadable.status === 0 && unreadable.outputs['releases-missing'] === undefined &&
1847+
/::warning::1\.1\.0's GitHub Releases .* could not be read \(.*HTTP 503/.test(unreadable.stdout),
1848+
said(unreadable),
1849+
);
1850+
t(
1851+
'Releases present: the in-flight read is never asked — the common path pays no Actions reads',
1852+
complete.actionsAsked.length === 0,
1853+
JSON.stringify(complete.actionsAsked),
1854+
);
17671855
} finally {
17681856
await registry.close();
1857+
await actions.close();
17691858
rmSync(root, { recursive: true, force: true });
17701859
}
17711860
}
@@ -2029,7 +2118,8 @@ export async function selfTest() {
20292118
`OK release-verify-npm self-test: ${cases.length} cases pass across `
20302119
+ `${Object.keys(SELF_TEST_BATTERIES).length} batteries (the #15321 false red reproduced and absorbed, `
20312120
+ 'the masked partial publish caught, absence still fatal, the release audit backfilling '
2032-
+ 'only a version whose whole group is on npm, and that backfill built from the version commit\'s tree).',
2121+
+ 'only a version whose whole group is on npm and whose publish is not in flight, and that backfill built '
2122+
+ 'from the version commit\'s tree).',
20332123
);
20342124
selfTestReachedVerdict = true;
20352125
return 0;

0 commit comments

Comments
 (0)