Skip to content

gates: check-issue-citations retries a transient 5xx or connection reset on a board read before refusing - #23438

Merged
objectstack-fleet[bot] merged 3 commits into
mainfrom
claude/issue-23361-citations-5xx-retry
Oct 14, 2026
Merged

objectstack-fleet[bot] merged 3 commits into
mainfrom
claude/issue-23361-citations-5xx-retry

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #23361
Clause-②: no

What this changes

scripts/check-issue-citations.mjs read ONE HTTP 500 from the board as PREREQUISITE NOT MET (exit 3) with no retry, so a GitHub blip turned the lint lane red and ejected a PR from the merge queue (run 38496246256). Every board read now goes through one helper, readBoard, shared by both strategies (probeBoard's head probe and per-number probes, enumerateBoard's pages), with ONE retry budget per run created in buildBoard.

  • Retried (closed set): an HTTP 5xx; a thrown connection failure whose cause chain carries ECONNRESET or UND_ERR_SOCKET (RETRYABLE_NETWORK_CODES).
  • Not retried: every 4xx (a 404 keeps its meaning, "absent"; a 401/403/429 is an answer about the credential or the rate limit), and every other thrown error (ECONNREFUSED, ENOTFOUND, a proxy refusing the tunnel, a body that is not JSON). Those are rethrown unchanged, so they keep the exit path they had.
  • Never a weaker gate: a read still failing after its ladder is handed back as a failure; run()'s existing catch turns it into prerequisiteRefusal, exit 3, never 0. The refusal now names the read, the answer and the attempt count, for example GET issues?per_page=1 -> HTTP 500 after 3 attempts (every attempt one read is allowed, so not one blip).
  • The retried unit is the request AND its JSON body. A reset can land while the body streams; measured, res.json() then throws TypeError: terminated with the same cause codes.
  • run() and buildBoard take optional fetchImpl / sleep (defaults: the global fetch, a real setTimeout sleep). Only the self-test passes them.
  • makeTransferProbe deliberately does not share the retry. Its 3xx is a verdict and stays one. A failed read there never exits 3: it degrades to a null verdict, which classifies as allocated-but-absent, still a finding. It is only reached under --probe-cause, which no workflow runs.
  • The header docblock and line 204 (dispatch-gates: local-env, pinned by line number in dispatch-gates.self-test.mjs) are untouched. All edits sit below it.

Retry policy and worst-case added wall time

BOARD_READ_RETRY: 3 attempts per read (one read plus two retries), backoff 500 ms then 1000 ms, and 4 retries per run shared by every read the run makes.

board added sleep extra requests then
answers normally 0 0 unchanged
stays down (every read 5xx or reset) 1500 ms 2 exit 3
flaps (every read fails twice, then answers): the worst case at most 3000 ms at most 4 the next 5xx is final, then exit 3
  • Board that stays down: the first read (the head probe, or page 1 under --enumerate) spends its ladder and the board build ends, so the cost does not depend on how many numbers the change cites. The self-test pins it with 400 cited numbers (the most probeBoard takes before switching to enumeration) and with the enumerating strategy: 3 reads and 1500 ms in both.
  • Board that flaps: the per-run budget is what bounds a 400-number probe or a 159-page enumeration. Pinned: sleeps 500, 1000, 500, 1000 (3000 ms total), then ... after 1 attempt (this run's budget of 4 retries is spent).
  • Before this change a board that stays down cost 1 request and 0 ms; now it costs 3 requests and 1.5 s. Request latency itself is not bounded, before or after: no read in this file sets a timeout.

Which connection-failure shapes are retried, as measured

Measured on node v22.22.0 (CI runs node 22) against a local socket server, both directly and through a local CONNECT proxy under --use-env-proxy (the gate's own re-exec route):

what the peer does what fetch throws retried
reset (RST) after the request bytes, or on accept TypeError: fetch failed, cause code ECONNRESET yes
close (FIN) with no response fetch failed, cause code UND_ERR_SOCKET ("other side closed") yes
reset mid-body TypeError: terminated from the body read, cause ECONNRESET yes
close mid-body terminated, cause UND_ERR_SOCKET yes
tunnel reset or closed by the proxy fetch failed, cause ECONNRESET / UND_ERR_SOCKET yes
proxy answers CONNECT with 502/503 fetch failed, cause chain ending in UND_ERR_ABORTED no
connection refused (control) cause ECONNREFUSED no
DNS failure (control) cause ENOTFOUND no

Verification (head 3becc14a)

  • Self-test: node scripts/check-issue-citations.mjs --self-test, exit 0, (215 cases, 10 batteries). It was 173 cases and 9 batteries. The new retry battery has 42 cases (floor 42), and SELF_TEST_BATTERY_FLOOR goes from 9 to 10.
    • The card's cases: one transient 500 then a 200 resolves, on the head probe and on a per-number probe. Persistent 500s are refused after exactly 3 attempts, and exit 3 end to end, read off a real child process. That child is offline: its fetchImpl is a stub, and it judges a real temp-repo diff. A 404 is asked once, never retried, and is still a finding.
    • Also: reset then 200, and body-stream close then 200, resolve. A transient 502 on enumerate page 2 still yields the whole board. 401/403/429 are refused on the first attempt. Unnamed errors are rethrown as the identical error object. The worst-case wall readings are pinned as above. A table of 11 persistent failures (500, 502, 503, 504, ECONNRESET, UND_ERR_SOCKET, UND_ERR_SOCKET mid-body, 401, 403, 429, ECONNREFUSED) is applied at the head probe, at a per-number probe and at an enumerated page: none yields a board.
  • Battery wall: 5 interleaved runs each on this shared box. Before (the base file, blob c732ce2e, read from a checkout at 4f81f315): median 5418 ms. After: median 5722 ms. The +0.3 s is the offline child process and its temp repo. Backoff sleeps are injected, so the new cases add no sleep.
  • Reverse verification (one-time, no permanent file): each mutation went through scripts/ablation-replace.mjs in WRAP mode, at HEAD 3becc14a. Each restore was proven by blob == HEAD (d2a21576) and an empty git diff HEAD. Every leg went red by name:
    • A1 attemptsPerRead: 1 (no retry): red at "one transient 500 on the HEAD probe, then a 200, RESOLVES ... it refused instead: GET issues?per_page=1 -> HTTP 500 after 1 attempt".
    • A2 run()'s board-read catch returns 0 instead of refusing: red at "persistent 500s EXIT 3 end to end, never 0 — got exit 0".
    • A3 4xx made retryable: red at "a 404 keeps its meaning: asked ONCE, never retried".
    • A4 run budget deleted: red at "a board that FLAPS spends at most the run's 4 retries (3000 ms)", which slept 500, 1000 six times.
    • A5 an exhausted read reported as a board: red at "persistent 500s are NOT a board".
  • Real-process readings (not kept as tests):
    • Live board, --base 4f81f315~15: 60 citations across 33 files, probed (16 citations), frontier #23434, exit 0. This is readBoard over real responses.
    • Through a local proxy that resets every tunnel: exit 3 after 3 CONNECTs and 2.7 s of wall, refusal GET issues?per_page=1 -> ECONNRESET after 3 attempts (...).
    • Through a proxy that answers CONNECT with 503: exit 3 after 1 CONNECT, not retried, refusal fetch failed as before.
  • This gate on this diff: node scripts/check-issue-citations.mjs, exit 0: no issue citations added against 4f81f3152 (0 file(s) read). The diff is all under scripts/**, a deferred surface.
  • Derived gate union at 3becc14a: dispatch-gates --commands --repo objectstack-ai/objectstack derived 32 commands. All 32 ran and all exited 0. --ran with exit codes reconciles: "32 derived famil(ies) accounted for — 32 run, 0 NOT-MEASURED (a DERIVED zero)".
    • That includes pnpm check:pm-dispatch-gates (fast tier, 1841 cases), whose census pin scripts/check-issue-citations.mjs:204 local-env and live local-env rows for this script both pass.
    • The battery's slow tier (6 sections) is NOT MEASURED locally. Its home is push and schedule runs.
  • Lint, narrowed: eslint --no-inline-config --format json scripts/check-issue-citations.mjs gave 1 file, 0 errors, 0 warnings.
    • Population: eslint.config.mjs's **/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs} object covers the file.
    • Invariance: the resolved config's parserOptions are ecmaVersion and sourceType only, with no project and no projectService. Type-aware linting is off, so this diff cannot move another file's verdict.
    • Full pnpm lint is CI's run.
  • Declared to CI, not run here: 4 workflow-valued families (--census among them), 11 wide-population families, 55 artifact-roster families, the path-scheduled Test Core job and the type-check lanes. No package is touched.
    • The 15 pending-changeset families do not apply: this PR carries skip-changeset, because root scripts/ publishes nothing.

Acceptance notes

  • Under --probe-cause, a transfer probe that reads a 5xx yields a null verdict, whose detail text says "re-run with --probe-cause" even though the run already passed it. That is report wording, in a mode no workflow runs; it never passes a board. Noted, not filed. Carrier: none.
  • No board read sets a request timeout, so a hung read waits out the job timeout. This predates the change and is unchanged. It was not observed in the incident. Noted, not filed.
  • A thrown error outside the retried set still surfaces as a bare fetch failed, which does not name the read. It is kept byte-identical so those exit paths are unchanged.
  • scripts/check-shard-attestation.mjs carries its own githubFetchJson (one retry on 5xx, 429 or any network error). It was not reused: this ruling excludes 429 and unnamed network errors.

Generated by Claude Code

…re refusing

A 5xx, an ECONNRESET or an UND_ERR_SOCKET from a board read now gets a
bounded ladder (3 attempts per read, 500/1000 ms backoff, 4 retries per
run) before the gate concludes PREREQUISITE NOT MET. A 4xx and every other
thrown error are answers and are not retried; a read still failing after
its ladder keeps its exit-3 path, and the refusal names the attempt count.

Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY
Co-authored-by: Claude <noreply@anthropic.com>
A new `retry` battery (42 cases, floor raised to 10 batteries): one
transient 500 then a 200 resolves on the head probe and on a per-number
probe; persistent 500s are refused after exactly the ladder and exit 3 end
to end; a 404 is asked once and stays a finding; every other 4xx and every
unnamed thrown error is not retried; the per-run budget caps a flapping
board at 3000 ms of backoff; no scripted failure at any read yields a board.

Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY
Co-authored-by: Claude <noreply@anthropic.com>
…en it refuses

Every resolve case settles its promise, so a regression reds that case by
name rather than escaping as an uncaught throw; the per-read refusal reads
"every attempt one read is allowed, so not one blip".

Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 38538817058 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

分类:failure —— 按下面的日志分诊。

失败的 job(日志抽取,best effort):

  • Build Core — 失败步骤: Build packages (excluding docs)(日志不可读,点进 job 看)
  • Test Core (2/6) — 失败步骤: Build this shard's dependency closure(日志不可读,点进 job 看)

↳ 失败原因 是判读的关键:超时(Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言(AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

⚠️ 断言这一侧有一类例外,判据是断言在测什么,不是它是不是 AssertionError。 断言的对象是产品行为(一个值、一个形状、一次拒收)⇒ 照上面读:真实的行为改变,去查,⛔ 不要重排掉;
断言的对象是这次实验自身的有效性前提(跑完的耗时、负载下的先后、任何只在时间预算内才成立的条件)⇒ 它跟超时是同一类,同样对负载敏感,重排一次是合法的判别手段。
识别是机械的:断言的消息或它比较的值本身点名了一段时长、一个时间戳、一个耗时计数。实测过的一对 —— AssertionError: SecurityPlugin.init() ran: expected false to be true 测的是产品行为(真回归);
AssertionError: this run took over a second, so second-precision stamps could have differed too: expected 1006 to be less than 1000 测的是实验前提:它守护的那条不变式当时是绿的,同一个 head 原样重排一次即成功。
穿着 AssertionError 外衣的时间测量,仍然是时间测量。(⛔ 这只改「怎么读一次红」,不改「哪些测试可以重排」——后者由别处管。)

跨 PR 相同签名(24h,按失败测试文件聚合):

  • ⚠️ 本次没有可用的聚合签名(日志里没有能解析出测试文件名的 FAIL 行)—— 这不是「没有同签名的其他 PR」,是这一轮没测到。跨 PR 聚合本次不可用,请手工比对其他 PR 的同类评论。
  • ⚠️ 24h 评论账本没读完(超过 5 页仍未读到窗口尽头),所以上面的「不同 PR 数」是下界,不是全量。

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 5 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Queue red, disposition · domain:devx#2 · session_01B7mrh59UAGffd7s9asUfoY · 2026-10-14T09:13Z. Answers the merge-queue triage comment 6164964851.

Not this PR's. No re-queue. This PR changes only scripts/check-issue-citations.mjs, which is part of no package build.


Generated by Claude Code

Merged via the queue into main with commit 863213a Oct 14, 2026
45 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-23361-citations-5xx-retry branch October 14, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants