Skip to content

Commit f57627b

Browse files
ci(merge-queue): name a no-runner red as infra:no-runner, with a once-per-head re-queue verdict (#21946)
Part of #21933 — this PR delivers the classification, the comment, and the once-per-head re-queue verdict. It does not deliver the automatic re-queue act: the workflow holds no credential that can enqueue a pull request (see "Blocker" below). The card should stay open for that half. ## What changed `.github/workflows/merge-queue-triage.yml` gains limb ③: 1. **Classify, from the job record only.** A job is `infra:no-runner` when it ended `cancelled` with an empty `runner_name` and no steps. A cancelled job the record does not decide (it did get a runner) is checked for the platform's "The job was not acquired by Runner" check-run annotation. No log text is read. The **build** is `infra:no-runner` only when that explains every red. Every other red job must be an aggregate gate whose failed steps are all `Verify … results`, and whose own shard family (`NAME (k/n)`) has a no-runner member. Anything else makes the build `failure`, read as before: a real failing step, a cancelled job that had a runner, an aggregate from a different family, or an annotation that could not be read. 2. **Re-queue budget: one per PR head.** On an `infra:no-runner` build, the script reads the PR's current head through `pulls.get`. The comment carries a durable per-head marker: a hidden HTML comment named `merge-queue-infra:no-runner` with the PR number, head sha and run id. The next `infra:no-runner` red on the same head finds that marker. Its verdict is then "hand to a person, do not re-queue", and it names the earlier queue build. A `failure` build never gets a verdict and never writes the marker. If the head or the PR's comments cannot be read, the verdict is `unknown`, never `once`. 3. **The comment names the class:** `分类:infra:no-runner` or `分类:failure`. For `infra:no-runner` it lists the no-runner jobs, the aggregates they explain, and the verdict. For a mixed build it still lists the no-runner jobs, and it names the red those jobs cannot explain. ⛔ The attestation rule is unchanged: a shard that never ran still does not count as passing (#6082). The comment says so too. Nothing here changes whether a build is green. The idempotency read of the PR's comments now paginates, because the per-head markers live in those comments. Permission change, declared: the job gains **`checks: read`** (read only) for the annotation leg. No other grant changes. The harness now pins that the job holds no `contents:` grant. ## Blocker — the re-queue act The workflow's verdict cannot be carried out by the workflow. - **No enqueue credential.** The only enqueue path measured in this repo is `enablePullRequestAutoMerge` through the fleet App token. `fleet-write.yml` documents that this needs `contents: write`, which is "ONE consumer". This workflow's `GITHUB_TOKEN` holds `actions: read`, `pull-requests: write`, `issues: write` and now `checks: read`. - **The token's own grant would not be enough either.** Granting `contents: write` to `GITHUB_TOKEN` and calling `enqueuePullRequest` / `enablePullRequestAutoMerge` would widen this job's permissions. Even then, the merge group would be created by `GITHUB_TOKEN`, and GitHub does not start workflow runs for events that `GITHUB_TOKEN` causes. It is not established that CI would ever build such a merge group, and this PR does not measure it. So the comment says the workflow will not re-queue, and asks a person to re-queue once. The harness makes an enqueue call an unmodelled API that fails the battery (mutation M28). Wiring the act needs a decision: either a second consumer of the fleet App token with `contents: write`, or a measured `GITHUB_TOKEN` path. ## Done-when, clause by clause - **A synthetic no-runner cancellation is classified and re-queued once.** - Classified: proven. N1 replays the real job records of queue build 37374282440. N2 replays queue build 37371558473, with eight no-runner jobs and two aggregates. N7 uses the real annotation of check run 111979038621. - Re-queued once: only the verdict is proven. N1 grants one re-queue, and the N3 pair (run 2 reads run 1's own comment) sends a second red on the same head to a person. N4 proves the budget is per head. The act is blocked, as described above. - **A real shard failure is not re-queued.** N5 (real failure), N6 (no-runner beside a real failure) and N9 (an aggregate from another family) all classify as `failure`, with no verdict and no marker. - **The triage comment names the class.** N1 through N10 all assert the class line. - **Test / dry-run fixture for both cases.** - Ten scenarios cover limb ③. The new fixtures in `scripts/fixtures/merge-queue-triage/` are trimmed real records, with provenance in the README: `run-37374282440.jobs.json`, `run-37371558473.jobs.json` and `check-run-111979038621.annotations.json`. - Nine new self-test mutations (M20 to M28) each turn the battery red at the scenario they name, with a named control scenario that stays green. ## Validation (at 499d0ae) - `node scripts/check-merge-queue-triage-outcome.mjs` → `OK (133 assertions over 34 scenarios …)`. The base had 90 assertions over 24 scenarios. - `node scripts/check-merge-queue-triage-outcome.mjs --self-test` → `181 assertions, 32 mutations of the shipped script each driven to red`. - `node scripts/pm/dispatch-gates.mjs --commands` derived 51 commands. 50 ran with exit 0, including `pnpm check:workflow-status-functions`, `check:nul-bytes`, `check:required-contexts` and `check:shard-attestation`. Reconciliation `--ran`: 51 accounted, 0 UNRUN. - NOT MEASURED: `pnpm check:pm-dispatch-gates`. Its `--self-test` half alone exceeded the 10-minute foreground cap (exit 124). Its log showed 1785 pass marks and 1 fail mark. The fail mark is on `packages/qa/dogfood/test/per-file-cwd.setup.ts`, a path this diff does not touch. Declared to CI. - Narrowed lint: `eslint --no-inline-config --format json scripts/check-merge-queue-triage-outcome.mjs` → 1 file, 0 errors, 0 warnings. The repo config enables no type-aware linting (no `parserOptions.project`), so this diff cannot move the verdict on any untouched file. The workflow YAML is not an eslint target. ## Acceptance notes - The triage workflow listens to `workflows: [CI]` only. Run 37371558492 is the **Governed Surface Guard** workflow, and its no-runner red is never triaged. The same holds for the sibling guard runs that ejected #21872 that afternoon. Out of this card's file surface; noted here only. - The budget keys on the PR head read at triage time. A push that lands between the ejection and the triage run would be charged for the earlier head's red. That is the conservative direction: the next red reaches a person sooner. - A run whose conclusion is `cancelled` (not `failure`) still gets no comment. That is the existing eviction rule, unchanged. --- _Generated by [Claude Code](https://claude.ai/code/session_01VDtqoecgES7ScQYGbFVDRv)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent d16b9fb commit f57627b

6 files changed

Lines changed: 3560 additions & 7 deletions

File tree

‎.github/workflows/merge-queue-triage.yml‎

Lines changed: 188 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,45 @@ name: Merge Queue Triage
101101
# `skip`/`only` written anywhere. Weakening a gate stays a human act, and an
102102
# anchor issue is a place for that conversation, not a verdict in it.
103103
#
104+
# ## A shard that never got a runner is not a test failure (limb ③)
105+
#
106+
# On 2026-10-05 one PR was ejected twice with nothing in it failing: queue
107+
# build 37371558473 had eight jobs cancelled before any of them was handed a
108+
# runner, and 37374282440 had one (`Test Core (5/6)`). In both, the only
109+
# `failure` was an AGGREGATE gate (`Test Core`, `Dogfood Regression Gate`)
110+
# refusing because a shard of its own family published no attestation. The
111+
# comment printed those aggregates as the failure and "日志不可读" beside them,
112+
# and a person had to open the jobs to tell a runner outage from a test red.
113+
#
114+
# So every run is now CLASSIFIED, from the job RECORD and never from log text:
115+
#
116+
# infra:no-runner a job ended `cancelled` with an empty `runner_name` and no
117+
# steps, OR its check run carries the "not acquired by
118+
# Runner" annotation (read only for a cancelled job the
119+
# record did not already decide).
120+
#
121+
# The BUILD is `infra:no-runner` only when that explains EVERY red: each other
122+
# red job is an aggregate whose failed steps are all `Verify … results` AND
123+
# whose own shard family (`<name> (k/n)`) has a no-runner member. Anything left
124+
# unexplained — a real failing step, a cancelled job that did get a runner, an
125+
# annotation that could not be read — makes it `failure`, read the way it
126+
# always was. That direction is the safe one: a mis-read outage costs a person
127+
# a look, a mis-read failure would be waved through as weather.
128+
#
129+
# ⛔ The attestation rule does not move (#6082): a shard that never ran still
130+
# does not count as passing, and nothing here changes whether a build is
131+
# green. This changes how a red is NAMED and what the comment advises.
132+
#
133+
# Re-queue budget: ONE per PR head sha. An `infra:no-runner` comment carries a
134+
# per-head marker; the next `infra:no-runner` red on the SAME head finds it and
135+
# says the budget is spent and a person has to look. A `failure` never earns
136+
# the advice. ⚠️ The ACT is not wired: this job holds no credential that can
137+
# enqueue a pull request (GitHub's auto-merge / enqueue mutations need
138+
# `contents: write`, and a merge group created by this run's own GITHUB_TOKEN
139+
# is not one CI is known to build), so the comment states the verdict and a
140+
# person or seat performs the one re-queue. The harness enforces that: an
141+
# enqueue call is an unmodelled API and fails the battery.
142+
#
104143
# Fires on conclusion == failure ONLY. 'cancelled' is the queue evicting an
105144
# entry because something AHEAD of it failed (or a manual cancel) — it says
106145
# nothing about this PR, so it gets no comment (the same run-lifecycle
@@ -110,7 +149,9 @@ name: Merge Queue Triage
110149
# workflow_run executes in the DEFAULT branch's context: this file must be on
111150
# main before it fires, it never checks out or runs PR code, and it holds the
112151
# minimum permissions (actions: read for logs, pull-requests: write for the
113-
# comment, issues: write for the anchor issue limb ② files and refreshes).
152+
# comment and the PR head limb ③ keys its budget on, issues: write for the
153+
# anchor issue limb ② files and refreshes, checks: read for the "not acquired
154+
# by Runner" annotation limb ③ consults).
114155
#
115156
# Delivery is retried, never assumed (#9424). A transient GitHub API failure used
116157
# to kill this job outright — github-script hands any throw from the script to
@@ -156,6 +197,7 @@ jobs:
156197
actions: read
157198
pull-requests: write
158199
issues: write
200+
checks: read
159201
steps:
160202
- name: Post the triage comment
161203
uses: actions/github-script@v9
@@ -244,12 +286,17 @@ jobs:
244286
// redelivered workflow_run stops HERE, before any sighting is recorded
245287
// and before any anchor is touched, so one queue build contributes its
246288
// signature exactly once however many times GitHub delivers it.
289+
//
290+
// The listing is kept (`prComments`) because limb ③ reads its
291+
// per-head re-queue markers out of the same comments; `null` means
292+
// it could not be read, which limb ③ must not mistake for "none".
247293
let alreadyPosted = false;
294+
let prComments = null;
248295
try {
249-
const existing = await github.rest.issues.listComments({
296+
prComments = await github.paginate(github.rest.issues.listComments, {
250297
owner, repo, issue_number: prNumber, per_page: 100,
251298
});
252-
alreadyPosted = existing.data.some((c) => (c.body ?? '').includes(marker));
299+
alreadyPosted = prComments.some((c) => (c.body ?? '').includes(marker));
253300
} catch (error) {
254301
core.warning(
255302
`Could not read #${prNumber}'s comments to check for an existing triage `
@@ -269,6 +316,93 @@ jobs:
269316
});
270317
const failedJobs = jobs.filter((j) => j.conclusion === 'failure');
271318
319+
// ── Limb ③: classify the red — infra:no-runner or failure ──────
320+
//
321+
// Keyed on the job RECORD (see the header), never on log text: a
322+
// job that never got a runner has no log to read. The annotation
323+
// is consulted only for a cancelled job the record did not
324+
// already decide, so the common case costs no extra request.
325+
const NO_RUNNER_ANNOTATION = /not acquired by Runner/i;
326+
const AGGREGATE_STEP = /^Verify .* results$/;
327+
const GREEN = new Set(['success', 'skipped', 'neutral']);
328+
const redJobs = jobs.filter((j) => !GREEN.has(j.conclusion));
329+
const noRunnerByRecord = (j) => j.conclusion === 'cancelled'
330+
&& String(j.runner_name ?? '') === ''
331+
&& (j.steps ?? []).length === 0;
332+
const noRunner = [];
333+
const annotationRefusals = [];
334+
for (const j of redJobs.filter((x) => x.conclusion === 'cancelled')) {
335+
if (noRunnerByRecord(j)) { noRunner.push({ job: j, via: 'record' }); continue; }
336+
const checkRunId = Number(String(j.check_run_url ?? '').split('/').pop()) || j.id;
337+
try {
338+
const res = await github.rest.checks.listAnnotations({
339+
owner, repo, check_run_id: checkRunId, per_page: 100,
340+
});
341+
if ((res.data ?? []).some((a) => NO_RUNNER_ANNOTATION.test(String(a?.message ?? '')))) {
342+
noRunner.push({ job: j, via: 'annotation' });
343+
}
344+
} catch (error) {
345+
annotationRefusals.push(`${j.name}(${describe(error)})`);
346+
}
347+
}
348+
const noRunnerNames = noRunner.map((x) => x.job.name);
349+
// An aggregate red is EXPLAINED only by a no-runner member of its
350+
// OWN shard family. A `Verify … results` failure with no such
351+
// member is a shard that ran and did not attest — unexplained.
352+
const explainedAggregate = (j) => {
353+
const failedSteps = (j.steps ?? []).filter((s) => s.conclusion === 'failure');
354+
return j.conclusion === 'failure'
355+
&& failedSteps.length > 0
356+
&& failedSteps.every((s) => AGGREGATE_STEP.test(s.name))
357+
&& noRunnerNames.some((n) => n.startsWith(`${j.name} (`));
358+
};
359+
const explainedAggregates = redJobs.filter(explainedAggregate);
360+
const unexplained = redJobs.filter((j) =>
361+
!noRunner.some((x) => x.job === j) && !explainedAggregates.includes(j));
362+
const infraOnly = noRunner.length > 0 && unexplained.length === 0;
363+
const buildClass = infraOnly ? 'infra:no-runner' : 'failure';
364+
if (annotationRefusals.length > 0) {
365+
core.warning(
366+
`Could not read the check-run annotations of ${annotationRefusals.length} cancelled job(s), `
367+
+ 'so they were NOT classified as infra:no-runner and the build reads as `failure`.',
368+
{ title: 'infra:no-runner classification incomplete' },
369+
);
370+
}
371+
372+
// The re-queue budget: ONE per PR head sha. The head is read from
373+
// the PR itself (the queue ref names the BASE the build stacked
374+
// on, not the PR's head). Every way of not knowing — head
375+
// unreadable, comments unreadable — is `unknown`, never `once`:
376+
// a second automatic chance is the thing this must not invent.
377+
const NO_RUNNER_MARK = /<!-- merge-queue-infra:no-runner\|pr=(\d+)\|head=([0-9a-f]{40})\|run=(\d+) -->/g;
378+
let prHead = null;
379+
let requeue = null;
380+
if (infraOnly) {
381+
let headRefusal = null;
382+
try {
383+
const pr = await github.rest.pulls.get({ owner, repo, pull_number: prNumber });
384+
const sha = String(pr.data?.head?.sha ?? '');
385+
if (/^[0-9a-f]{40}$/.test(sha)) prHead = sha;
386+
} catch (error) {
387+
headRefusal = describe(error);
388+
}
389+
if (!prHead) {
390+
requeue = { verdict: 'unknown', why: `读不到 PR 当前的 head sha(${headRefusal ?? 'head.sha 缺失'})` };
391+
} else if (prComments === null) {
392+
requeue = { verdict: 'unknown', why: '读不到本 PR 的评论,无法确认这个 head 是否已经用掉过一次 infra:no-runner 重排' };
393+
} else {
394+
const priorRuns = new Set();
395+
for (const c of prComments) {
396+
for (const mm of String(c.body ?? '').matchAll(NO_RUNNER_MARK)) {
397+
if (Number(mm[1]) === prNumber && mm[2] === prHead && mm[3] !== String(run.id)) priorRuns.add(mm[3]);
398+
}
399+
}
400+
requeue = priorRuns.size === 0
401+
? { verdict: 'once' }
402+
: { verdict: 'person', priorRuns: [...priorRuns] };
403+
}
404+
}
405+
272406
// ── Log harvest ────────────────────────────────────────────────
273407
//
274408
// Best-effort: the lines a human would grep for first, plus the line
@@ -834,6 +968,54 @@ jobs:
834968
? `⚠️ **本 PR 过去 24h 已在队列失败 ${priorFailuresThisPr} 次(不含本次)。** 内容未变而反复失败 ⇒ 高度怀疑 flaky 测试或与同组 PR 的语义冲突,重排不解决。`
835969
: '本 PR 过去 24h 无队列失败记录(首次)。';
836970
971+
// ── Limb ③ in the comment: the class, by name, and the verdict ──
972+
const nameList = (list) => {
973+
const shown = list.slice(0, 20).map((n) => `\`${n}\``).join('、');
974+
return list.length > 20 ? `${shown} 等共 ${list.length} 个` : shown;
975+
};
976+
const classLines = [];
977+
if (infraOnly) {
978+
classLines.push(
979+
'**分类:`infra:no-runner`** —— 这次的红**全部**来自没拿到 runner 的 job,**不是测试失败**。',
980+
'判据读的是 job 记录,不是日志:`cancelled` + `runner_name` 为空 + 0 个步骤,或 check run 带 “not acquired by Runner” 注解。',
981+
'',
982+
`- 没拿到 runner 的 job(${noRunner.length} 个):${nameList(noRunnerNames)}`,
983+
);
984+
if (explainedAggregates.length > 0) {
985+
classLines.push(`- 聚合 job 的红只因为同族分片没跑:${nameList(explainedAggregates.map((j) => j.name))}(失败步骤只有 \`Verify … results\`)`);
986+
}
987+
classLines.push('- ⛔ 没跑的分片**仍然不算通过**(证明规则不变):这里只改「这次红怎么归类」,不改「构建算不算绿」。');
988+
if (requeue.verdict === 'once') {
989+
classLines.push(
990+
`- **重排判定:可以重排一次。** 这是 head \`${prHead.slice(0, 12)}\` 第一次因 infra:no-runner 被弹出。`
991+
+ '本 workflow 没有入队所需的凭据,**不会自动重排** —— 请人工重排**一次**。同一个 head 再次 infra:no-runner 红就交给人,不再重排。',
992+
);
993+
} else if (requeue.verdict === 'person') {
994+
classLines.push(
995+
`- ⛔ **重排判定:交给人,不要再重排。** head \`${prHead.slice(0, 12)}\` 已经因 infra:no-runner 被弹出过`
996+
+ `(队列构建 ${requeue.priorRuns.map((id) => `[${id}](https://github.com/${owner}/${repo}/actions/runs/${id})`).join('、')}),`
997+
+ '每个 head 只给一次重排。连续没拿到 runner 需要人去看 runner 状态。',
998+
);
999+
} else {
1000+
classLines.push(`- ⚠️ **重排判定:无法确认**(${requeue.why})⇒ 按「交给人」处理,不要据此重排。`);
1001+
}
1002+
} else {
1003+
classLines.push('**分类:`failure`** —— 按下面的日志分诊。');
1004+
if (noRunner.length > 0) {
1005+
classLines.push(
1006+
'',
1007+
`- 另有 ${noRunner.length} 个 job 没拿到 runner(${nameList(noRunnerNames)}),`
1008+
+ `但还有它们解释不了的红:${nameList(unexplained.map((j) => `${j.name} · ${j.conclusion}`))} ⇒ 不是纯 infra:no-runner,⛔ 不按 runner 故障重排。`,
1009+
);
1010+
}
1011+
}
1012+
if (annotationRefusals.length > 0) {
1013+
classLines.push(`- ⚠️ ${annotationRefusals.length} 个 cancelled job 的注解读不到,未能判定是否 no-runner,已按 failure 计:${annotationRefusals.join('、')}`);
1014+
}
1015+
const noRunnerMarker = infraOnly && prHead
1016+
? `<!-- merge-queue-infra:no-runner|pr=${prNumber}|head=${prHead}|run=${run.id} -->`
1017+
: null;
1018+
8371019
// The sighting markers this comment contributes to the 24 h ledger.
8381020
const sightingMarkers = runKeys.map((k) => `<!-- queue-signature:${k}|pr=${prNumber}|run=${run.id} -->`);
8391021
@@ -843,6 +1025,8 @@ jobs:
8431025
`队列构建 [${run.id}](${run.html_url}) 红了。队列跑的是**全量**套件(PR 侧 CI 只跑 affected 子集),`,
8441026
'所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。',
8451027
'',
1028+
...classLines,
1029+
'',
8461030
'**失败的 job(日志抽取,best effort):**',
8471031
'',
8481032
jobSections || '- (没拿到 job 级信息,点上面的 run 链接看)',
@@ -871,6 +1055,7 @@ jobs:
8711055
'',
8721056
marker,
8731057
...sightingMarkers,
1058+
...(noRunnerMarker ? [noRunnerMarker] : []),
8741059
'---',
8751060
'_Generated by [Claude Code](https://claude.ai/code) · merge-queue-triage workflow (#4859)_',
8761061
].join('\n');

0 commit comments

Comments
 (0)