Skip to content

Commit 4d31b13

Browse files
committed
fix(release): the image backfill skips a version whose publishing run has not finished building its image
The audit step asks the in-flight reader once, before either backfill, and gates `image-missing` on its `docker` verdict the way it gates `releases-missing` on the publish job's: a notice while the publishing run's docker job is not yet created, queued or running; a warning, and no request, when the read is unreadable or cannot be asked; the existing request otherwise. Battery 12 of release-verify-npm's self-test pins the four triage pins plus the not-yet-created case at the audit's own text. Claude-Session: https://claude.ai/code/session_01T9u38rswFp5Rw8DswRUReJ Co-authored-by: Claude <noreply@anthropic.com>
1 parent cf6ec80 commit 4d31b13

3 files changed

Lines changed: 118 additions & 32 deletions

File tree

‎.github/workflows/release.yml‎

Lines changed: 33 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1070,38 +1070,55 @@ jobs:
10701070
;;
10711071
esac
10721072
1073-
# ── the publish in flight, before any Releases backfill ───────────
1073+
# ── the publish in flight, before any backfill ────────────────────
10741074
# npm settles BEFORE the publish job reaches its own "Create GitHub
1075-
# Releases" step, so a landing audited in that window found the group
1076-
# complete and wrote the same Releases beside it: on 17.6.0 the
1077-
# publish of run 36955885276 wrote them 03:03:47Z -> 03:05:42Z, this
1078-
# backfill in run 36958423332 started at 03:04:37Z, and five tags got
1079-
# two Release objects each. The two writers are in different runs,
1080-
# so no `needs:` can order them; the publish's own run creates its
1081-
# Releases and D4 asset, and a landing after it finishes backfills
1082-
# whatever it did not. scripts/release-pending-publish.mjs `in-flight`
1083-
# (its --self-test, run by lint.yml, holds the rule) answers
1084-
# `in-flight` on anything it cannot read, so nothing is backfilled off
1085-
# a guess. It only ever leaves `releases-missing` unset: this step
1086-
# stays green and the image request below is not its business.
1075+
# Releases" step, and before that run's `docker` job even exists (it
1076+
# needs `publish`, so GitHub creates it when the publish job
1077+
# completes). A landing audited in that window found the group
1078+
# complete and wrote beside the publishing run. On 17.6.0 the publish
1079+
# of run 36955885276 wrote its Releases 03:03:47Z -> 03:05:42Z, this
1080+
# backfill in run 36958423332 started writing them at 03:04:37Z, and
1081+
# five tags got two Release objects each; and both runs built and
1082+
# pushed the 17.6.0 image, 03:05:49Z -> 03:10:02Z beside 03:05:48Z ->
1083+
# 03:09:18Z. The writers are in different runs, so no `needs:` can
1084+
# order them; the publishing run creates its Releases, D4 asset and
1085+
# image, and a landing after it finishes backfills whatever it did
1086+
# not. scripts/release-pending-publish.mjs `in-flight` (its
1087+
# --self-test, run by lint.yml, holds the rule) answers both in one
1088+
# read: `.state` for the publish job, `.docker.state` for that run's
1089+
# image build, in flight from the approval until its docker job
1090+
# completes. Anything it cannot read answers `in-flight`, so nothing
1091+
# is backfilled off a guess. It only ever leaves `releases-missing`
1092+
# and `image-missing` unset: this step stays green.
1093+
flight_asked=true
1094+
flight=$(GITHUB_TOKEN="$GH_TOKEN" node scripts/release-pending-publish.mjs in-flight --version "$version" --version-commit "$version_commit" --head "$SHA" --workflow release.yml) || flight_asked=false
1095+
if [ "$flight_asked" = true ]; then
1096+
jq -c . <<<"$flight"
1097+
fi
10871098
if [ "$releases_ok" = true ]; then
10881099
echo "GitHub Releases + ADR-0087 D4 asset are present for ${version}."
1089-
elif ! flight=$(GITHUB_TOKEN="$GH_TOKEN" node scripts/release-pending-publish.mjs in-flight --version "$version" --version-commit "$version_commit" --head "$SHA" --workflow release.yml); then
1100+
elif [ "$flight_asked" != true ]; then
10901101
echo "::warning::${version}'s GitHub Releases or ADR-0087 D4 asset are incomplete, and whether its publish is still in flight could not be asked (reason above). Nothing is backfilled off a guess; the next landing reads again."
10911102
elif [ "$(jq -r '.state' <<<"$flight")" != 'clear' ]; then
1092-
jq -c . <<<"$flight"
10931103
if [ "$(jq -r '.reason' <<<"$flight")" = 'publish-in-flight' ]; then
10941104
echo "::notice::${version}'s GitHub Releases or ADR-0087 D4 asset are incomplete, but its publish is still in flight ($(jq -r '.detail' <<<"$flight")). That run creates them; nothing is backfilled beside it. A landing after it finishes backfills whatever it did not."
10951105
else
10961106
echo "::warning::${version}'s GitHub Releases or ADR-0087 D4 asset are incomplete, and whether its publish is still in flight could not be read ($(jq -r '.detail' <<<"$flight")). Nothing is backfilled off a guess; the next landing reads again."
10971107
fi
10981108
else
1099-
jq -c . <<<"$flight"
11001109
echo "::warning::${version} is on npm but its GitHub Releases or the ADR-0087 D4 asset are incomplete (#4900) — backfilling."
11011110
echo "releases-missing=true" >> "$GITHUB_OUTPUT"
11021111
fi
11031112
if [ "$image_ok" = true ]; then
11041113
echo "ghcr: image for $version is present."
1114+
elif [ "$flight_asked" != true ]; then
1115+
echo "::warning::No ghcr image for ${version} (or the registry could not be probed), and whether its publishing run is still building it could not be asked (reason above). No Docker job is requested off a guess; the next landing reads again."
1116+
elif [ "$(jq -r '.docker.state' <<<"$flight")" != 'clear' ]; then
1117+
if [ "$(jq -r '.docker.reason' <<<"$flight")" = 'docker-in-flight' ]; then
1118+
echo "::notice::No ghcr image for ${version} yet, but its publishing run has not finished building it ($(jq -r '.docker.detail' <<<"$flight")). That run pushes the image; no Docker job is requested beside it. A landing after it finishes requests one if the image is still missing."
1119+
else
1120+
echo "::warning::No ghcr image for ${version} (or the registry could not be probed), and whether its publishing run is still building it could not be read ($(jq -r '.docker.detail' <<<"$flight")). No Docker job is requested off a guess; the next landing reads again."
1121+
fi
11051122
else
11061123
echo "::warning::No ghcr image for $version (or the registry could not be probed) — requesting the Docker job."
11071124
echo "image-missing=true" >> "$GITHUB_OUTPUT"

‎scripts/release-pending-publish.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -856,7 +856,7 @@ export function judgePublishInFlight({ version, runs, currentRunId }) {
856856
? {
857857
state: 'in-flight',
858858
reason: 'docker-in-flight',
859-
detail: `the image build of ${version}'s publishing run is not finished: ${building
859+
detail: `image build not finished in ${building
860860
.map((b) => `run ${b.run} (${b.event}, publish job ${b.publish}, Docker image ${b.docker})`)
861861
.join(', ')}`,
862862
inFlight: building,

‎scripts/release-verify-npm.mjs‎

Lines changed: 84 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -142,8 +142,9 @@
142142
*
143143
* The self-test's battery 12 runs the audit step itself — its text, read out of
144144
* `.github/workflows/release.yml` — against a throwaway repository, a stub
145-
* registry, a stub Actions API (the audit asks whether the version's publish
146-
* is in flight in another run before it backfills any Release) and stub
145+
* registry, a stub Actions API (the audit asks whether the version's publish,
146+
* or its publishing run's image build, is in flight in another run before it
147+
* backfills any Release or requests the image) and stub
147148
* `npm` / `gh` / `curl`, so the wiring is pinned where it lives rather than
148149
* described here.
149150
*
@@ -205,7 +206,7 @@ const SELF_TEST_BATTERIES = Object.freeze({
205206
'9. The job summary is written SYNCHRONOUSLY, before process.exit': 4,
206207
'10. The probe: the whole group in one read, three answers': 9,
207208
'11. The 17.5.0 publish window, replayed on npm\'s own clock': 7,
208-
'12. The audit step backfills only a version whose whole group is on npm, and no publish of it is in flight': 24,
209+
'12. The audit step backfills only a version whose whole group is on npm, and no publish or image build of it is in flight': 34,
209210
'13. The backfill builds from the version commit\'s tree, as the publish does': 11,
210211
});
211212

@@ -940,13 +941,15 @@ async function stubRegistryServer() {
940941

941942
/**
942943
* The Actions API the audit's in-flight read asks (`release-pending-publish.mjs
943-
* in-flight`), on 127.0.0.1. `set({ publishing, broken })`: `publishing` names
944-
* a version whose publish job is `in_progress` in run 4242, which only the
945-
* `in_progress` run-list filter lists; `broken` answers every request 503.
946-
* Records every request, so a case can assert the audit asked, or did not.
944+
* in-flight`), on 127.0.0.1. `set({ publishing, publishStatus, docker, broken })`:
945+
* `publishing` names a version whose publish job is `publishStatus` (default
946+
* `in_progress`) in run 4242, which only the `in_progress` run-list filter
947+
* lists; `docker` is the status of that run's docker job, absent when unset
948+
* (GitHub has not created it); `broken` answers every request 503. Records
949+
* every request, so a case can assert the audit asked, or did not.
947950
*/
948951
async function stubActionsApi() {
949-
let state = { publishing: null, broken: false };
952+
let state = { publishing: null, publishStatus: 'in_progress', docker: null, broken: false };
950953
const asked = [];
951954
const server = createServer((req, res) => {
952955
const url = new URL(String(req.url), 'http://stub.invalid');
@@ -963,7 +966,9 @@ async function stubActionsApi() {
963966
}
964967
if (state.publishing && url.pathname.endsWith('/actions/runs/4242')) return send(200, run);
965968
if (state.publishing && url.pathname.endsWith('/actions/runs/4242/jobs')) {
966-
return send(200, { total_count: 1, jobs: [{ name: `Publish ${state.publishing} to npm (awaiting approval)`, status: 'in_progress' }] });
969+
const jobs = [{ name: `Publish ${state.publishing} to npm (awaiting approval)`, status: state.publishStatus }];
970+
if (state.docker) jobs.push({ name: 'Docker image / Build & push ghcr.io/objectstack-ai/objectstack', status: state.docker });
971+
return send(200, { total_count: jobs.length, jobs });
967972
}
968973
return send(404, { message: 'Not Found' });
969974
});
@@ -972,7 +977,12 @@ async function stubActionsApi() {
972977
url: `http://127.0.0.1:${server.address().port}`,
973978
asked,
974979
set(next) {
975-
state = { publishing: next.publishing ?? null, broken: next.broken ?? false };
980+
state = {
981+
publishing: next.publishing ?? null,
982+
publishStatus: next.publishStatus ?? 'in_progress',
983+
docker: next.docker ?? null,
984+
broken: next.broken ?? false,
985+
};
976986
asked.length = 0;
977987
},
978988
close: () => new Promise((resolve_) => { server.close(resolve_); }),
@@ -1650,7 +1660,7 @@ export async function selfTest() {
16501660
}
16511661

16521662
// ── 12. The audit step itself ─────────────────────────────────────────────
1653-
battery('12. The audit step backfills only a version whose whole group is on npm, and no publish of it is in flight');
1663+
battery('12. The audit step backfills only a version whose whole group is on npm, and no publish or image build of it is in flight');
16541664
{
16551665
// The step's REAL text, read out of release.yml, run by bash the way
16561666
// Actions runs it, in a throwaway repository: base (1.0.0) -> the version
@@ -1725,7 +1735,7 @@ export async function selfTest() {
17251735
if (script !== null) writeFileSync(scriptFile, script);
17261736

17271737
let runs = 0;
1728-
const audit = async ({ event = 'push', before, head, present, broken = [], releases = false, image = false, flight = {} }) => {
1738+
const audit = async ({ event = 'push', before, head, present, broken = [], releases = false, image = false, flight = {}, token = 'stub' }) => {
17291739
runs += 1;
17301740
const temp = join(root, `run-${runs}`);
17311741
mkdirSync(temp);
@@ -1738,7 +1748,7 @@ export async function selfTest() {
17381748
SHA: head,
17391749
EVENT: event,
17401750
BEFORE: before ?? '',
1741-
GH_TOKEN: 'stub',
1751+
GH_TOKEN: token,
17421752
GITHUB_API_URL: actions.url,
17431753
GITHUB_RUN_ID: '9999',
17441754
GITHUB_REPOSITORY: 'objectstack-ai/objectstack',
@@ -1840,6 +1850,7 @@ export async function selfTest() {
18401850
);
18411851
const other = await audit({ before: vc, head: landing, present: whole, flight: { publishing: '1.2.0' } });
18421852
t('a publish of ANOTHER version in flight does not hold this one back: the Releases are backfilled', other.outputs['releases-missing'] === 'true', said(other));
1853+
t('...and the image is requested', other.outputs['image-missing'] === 'true', said(other));
18431854
const unreadable = await audit({ before: vc, head: landing, present: whole, flight: { broken: true } });
18441855
t(
18451856
'the Actions API unreadable: the audit stays green, backfills no GitHub Release off a guess, and a warning says so',
@@ -1848,10 +1859,68 @@ export async function selfTest() {
18481859
said(unreadable),
18491860
);
18501861
t(
1851-
'Releases present: the in-flight read is never asked — the common path pays no Actions reads',
1862+
'...and requests no image build off a guess either, with a warning that names the unread image build',
1863+
unreadable.outputs['image-missing'] === undefined &&
1864+
/::warning::No ghcr image for 1\.1\.0 .* still building it could not be read \(.*HTTP 503/.test(unreadable.stdout),
1865+
said(unreadable),
1866+
);
1867+
t(
1868+
'nothing missing: the in-flight read is never asked — the common path pays no Actions reads',
18521869
complete.actionsAsked.length === 0,
18531870
JSON.stringify(complete.actionsAsked),
18541871
);
1872+
1873+
// The image half of the 17.6.0 race: npm settled, and the publishing
1874+
// run's `docker` job (needs: publish) had not been created yet, so both
1875+
// runs built and pushed 17.6.0's image. The same read answers it.
1876+
t(
1877+
'PUBLISH IN FLIGHT (its docker job not created yet): no image build is requested beside it',
1878+
flying.outputs['image-missing'] === undefined,
1879+
said(flying),
1880+
);
1881+
t(
1882+
'...and a notice says why, naming the run whose image build is not finished',
1883+
/::notice::No ghcr image for 1\.1\.0 yet, but its publishing run has not finished building it .*run 4242 .*Docker image not yet created/.test(flying.stdout),
1884+
said(flying),
1885+
);
1886+
const published = await audit({ before: vc, head: landing, present: whole, releases: true, flight: { publishing: '1.1.0', publishStatus: 'completed' } });
1887+
t(
1888+
'the publish job completed, its docker job not created yet (Releases present): no image build is requested',
1889+
published.status === 0 && published.outputs['image-missing'] === undefined && published.outputs['releases-missing'] === undefined &&
1890+
/::notice::No ghcr image for 1\.1\.0 yet/.test(published.stdout),
1891+
said(published),
1892+
);
1893+
t(
1894+
'...because Releases present no longer skips the in-flight read: the image asks it',
1895+
published.actionsAsked.some((a) => /\/actions\/runs\/4242\/jobs/.test(a)) && published.actionsAsked.every((a) => a.startsWith('GET ')),
1896+
JSON.stringify(published.actionsAsked),
1897+
);
1898+
const building = await audit({ before: vc, head: landing, present: whole, flight: { publishing: '1.1.0', publishStatus: 'completed', docker: 'in_progress' } });
1899+
t(
1900+
'the publishing run\'s docker job in_progress: no image build is requested, while the Releases are backfilled',
1901+
building.outputs['image-missing'] === undefined && building.outputs['releases-missing'] === 'true',
1902+
said(building),
1903+
);
1904+
const built = await audit({ before: vc, head: landing, present: whole, flight: { publishing: '1.1.0', publishStatus: 'completed', docker: 'completed' } });
1905+
t(
1906+
'its docker job completed and the image is absent: the image is requested (the repair)',
1907+
built.status === 0 && built.outputs['image-missing'] === 'true',
1908+
said(built),
1909+
);
1910+
const shipped = await audit({ before: vc, head: landing, present: whole, image: true, flight: { publishing: '1.1.0', publishStatus: 'completed', docker: 'completed' } });
1911+
t(
1912+
'its docker job completed and the image is present: no image build is requested',
1913+
shipped.status === 0 && shipped.outputs['image-missing'] === undefined && /ghcr: image for 1\.1\.0 is present\./.test(shipped.stdout),
1914+
said(shipped),
1915+
);
1916+
const unasked = await audit({ before: vc, head: landing, present: whole, token: '' });
1917+
t(
1918+
'the in-flight read cannot run at all: the audit stays green, backfills nothing, and warns for the Releases and the image',
1919+
unasked.status === 0 && !backfills(unasked) &&
1920+
/::warning::1\.1\.0's GitHub Releases .* could not be asked/.test(unasked.stdout) &&
1921+
/::warning::No ghcr image for 1\.1\.0 .* could not be asked/.test(unasked.stdout),
1922+
said(unasked),
1923+
);
18551924
} finally {
18561925
await registry.close();
18571926
await actions.close();
@@ -2118,7 +2187,7 @@ export async function selfTest() {
21182187
`OK release-verify-npm self-test: ${cases.length} cases pass across `
21192188
+ `${Object.keys(SELF_TEST_BATTERIES).length} batteries (the #15321 false red reproduced and absorbed, `
21202189
+ 'the masked partial publish caught, absence still fatal, the release audit backfilling '
2121-
+ 'only a version whose whole group is on npm and whose publish is not in flight, and that backfill built '
2190+
+ 'only a version whose whole group is on npm and whose publish and image build are not in flight, and that backfill built '
21222191
+ 'from the version commit\'s tree).',
21232192
);
21242193
selfTestReachedVerdict = true;

0 commit comments

Comments
 (0)