Skip to content

Commit 099a94d

Browse files
ci(lint, governed-surface-guard): split each full-history job wall into a checkout budget plus a body budget (#22033)
Fixes #22020 Clause-②: no ## Which route The card's done-when offers two routes. This PR takes the first one, in the split form triage asked for (`6022336293`). Every full-history checkout under a fixed job wall gets its own step-level `timeout-minutes: 20`. Each job wall becomes that checkout budget plus the job's re-measured body budget. No gate, command, fetch depth, job id or check name changes. The seven required contexts are untouched, and the merge-base anchor of the authorable-surface deletion gate keeps its full history (`fetch-depth: 0` is unchanged on every job). | job | file | wall before | checkout step | body budget | wall after | |---|---|---|---|---|---| | Type Check · source gates (the card) | `lint.yml` | 10 | 20 | 10 | **30** | | Type Check · workspace (adjacent) | `lint.yml` | 30 | 20 | 35 | **55** | | Type Check · debt ledger (adjacent) | `lint.yml` | 15 | 20 | 15 | **35** | | Type Check · consumer gates (adjacent) | `lint.yml` | 20 | 20 | 25 | **45** | | Governed Surface Queue Guard (adjacent) | `governed-surface-guard.yml` | 10 | 20 | 10 | **30** | **In-scope adjacent fixes.** All four have the same defect class and get the same mechanical treatment: - The other three Type Check lanes, added by the PM after merge-group run 37509811445. In that run, job 112427734781 ("Type Check · consumer gates") spent 11m05s in the checkout and was cancelled at its 20-minute wall. The aggregate failed and PR #22016 was ejected from the queue. - The Governed Surface Queue Guard, which the claim named (run 37508992265, job 112424902156). The `lint` job is not touched. Open PR #22002 edits its region. ## Measured **Window.** I read the 300 most recent completed runs of `Lint & Type Check` (2026-10-05T15:32Z to 2026-10-06T18:41Z) and of `Governed Surface Guard` (2026-10-05T14:57Z to 2026-10-06T18:59Z). Every attempt was included (`GET /actions/runs/{id}/jobs?filter=all`), using step timestamps. The controls come from the 300 most recent `CI` runs (2026-10-05T15:15Z to 2026-10-06T18:22Z). **Checkout per job.** "Wall hits" counts the runs that the old wall cancelled behind a slow checkout: | job | n | p50 | p90 | p99 | max | wall hits | |---|---|---|---|---|---|---| | Type Check · source gates | 293 | 31 s | 122 s | 599 s | 600 s (cut by the wall) | 8 | | Type Check · workspace | 296 | 31 s | 100 s | 395 s | 421 s | 0 | | Type Check · debt ledger | 293 | 31 s | 123 s | 599 s | 774 s | 2 | | Type Check · consumer gates | 293 | 31 s | 151 s | 574 s | 735 s | 2 (both `merge_group`) | | Governed Surface Queue Guard | 292 | 31 s | 156 s | 595 s | 599 s (cut by the wall) | 4 | **Pooled.** Adding the same fetch from `ci.yml`'s Test Core shards gives n = 3,459: p50 31 s, p90 119 s, p99 553 s, max 895 s. Another 19 samples were cut off by a wall and are lower bounds only. **Control.** The shallow (`fetch-depth` 1) checkouts in `ci.yml` over the same hours (n = 2,743) read p50 15 s, p99 63 s, max 107 s. None went over 180 s. So the slow tail belongs to the full-history fetch, not to the runner pool. **Body budgets.** The body is everything after the checkout, on successful runs: | job | max | 2 × max | budget | |---|---|---|---| | source gates | 3.2 min | 6.4 | 10 (the floor) | | workspace | 17.9 min | 35.8 | 35 | | debt ledger | 7.4 min | 14.8 | 15 | | consumer gates | 11.5 min | 23 | 25 | | guard | 1.1 min | 2.2 | 10 (the floor) | The rule is the `lint` job's: 2× the measured max, on a 5-minute grain, never below 10. **Checkout budget.** 20 minutes is 1.3× the pooled 895 s max. The rule's 2× would give 30, but the queue's window binds first: the workspace lane's 35 plus 30 is 65, past the 55 cap the `lint` job argues, while 35 + 20 = 55. It is one fetch, so every job gets the same budget. ## What the required aggregate reads **Before.** The lane's job wall stopped any checkout that left the body too little time. That covers slow checkouts that would have finished, such as the 555 s and 588 s cases, as well as stuck ones. The lane read `cancelled` and every gate was skipped. `TypeScript Type Check` printed "concluded `cancelled` -- expected `success`" and failed the PR. **After:** - **A checkout that ends inside 20 minutes:** nothing happens. The body keeps its full budget and the lane goes green. In this window, that covers every checkout that completed. - **A checkout still running at 20 minutes:** the step fails with "The action 'Checkout repository' has timed out after 20 minutes." Later steps are skipped, the lane reads `failure` and the aggregate goes red. No layout can keep a truly hung checkout green, because the aggregate correctly refuses a lane that ran no gate. This layout names the step that hung, and it fires only beyond 1.3× anything measured. - **A later step that hangs:** the job wall cancels the job, the lane reads `cancelled`, and the aggregate goes red, as before but at the new wall. **Source check (premise 2).** In actions/runner `main` at `67f01c27`, `StepsRunner.RunStepAsync` evaluates every step's `timeout-minutes`, `uses:` steps included. When it expires, the runner sets `TaskResult.Failed` with that message. A job-level cancellation sets `TaskResult.Canceled` instead. No workflow in this repo had a step-level timeout before this PR. ## Deviation from triage's direction Triage wrote: "The gate steps keep the stall guard's intent through their own step-level timeouts." I did not add those. The PM thread has the reasoning: - Actions has no timeout over a group of steps. Covering the gates would take one step-level timeout per step, about 65 of them across the four lanes, each sized from a single 27-hour window. - Several of those steps are bimodal, for example a turbo cache hit in 4 s against a miss in 5 minutes. Timeouts sized that tightly would be a new source of random reds, which is this card's own defect class. The cost: after a fast checkout, a hung gate now runs up to the new wall (30, 35, 45 or 55 minutes) instead of the old one. Every new wall stays at or under the 55 cap, inside the queue's 60-minute window. ## This PR's own lane run Read from the jobs API by the seat (step timestamps), run `37519814891` (`Lint & Type Check`, head `f5060b251`) and run `37519814823` (`Governed Surface Guard`): | job | conclusion | checkout | job duration | new wall | |---|---|---|---|---| | Type Check · source gates | success | 34 s | 197 s (3.3 of 30 min) | 30 | | Type Check · debt ledger | success | 147 s | 217 s | 35 | | Type Check · workspace | success | 242 s | 280 s | 55 | | Type Check · consumer gates | success | 33 s | 298 s | 45 | | TypeScript Type Check (aggregate) | success | — | 3 s | — | | Governed Surface Queue Guard | success | 38 s | 66 s | 30 | _Section filled in by the `domain:devx` seat 2 PM from the run above, because the body is written before its own run exists._ ## Local gates (head `f5060b251`) `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 44 commands from the actual diff. Exit codes were written to disk before reading: - 42 exit 0. This includes `check:required-contexts`, `check:stall-guard-budget`, `check:stall-guard-headroom`, `check:workflow-status-functions`, `check:workflow-step-name-quoting`, `check:aggregator-roster` (+ `--self-test`) and `check:step-collectors` (+ `--self-test`). The full reconciliation is in the report. - I also ran the guard's own `check-governed-queue-guard.mjs --self-test`: 296 cases passed, including the `fetch-depth: 0` pin. - `check:type-check-debt` ran its `--self-test` half (exit 0). Its `--re-measure` half needs the workspace build the debt lane runs first. It is declared to this PR's CI rather than measured here, because the diff touches no package and no ledger. ## Acceptance notes - **Mechanism, measured locally and not changed here.** `fetch-depth: 0` makes actions/checkout v7 fetch `+refs/heads/*` and `+refs/tags/*`: 1,236 branch heads and about 7,950 tags today. From this container: | fetch | time | pack | |---|---|---| | `main`'s full history alone | 29 s and 32 s | 397 MB | | the all-refs refspec CI uses | 473 s | 598 MB | | all heads without tags | 686 s | 597 MB | | depth 1 plus unshallowing `main` | 120 s and 169 s | not recorded | The last variant kept the anchor exact on PR #22002's merge ref and on PR #6356's old head. The cost is the ref set, not `main`'s depth. Fetching less could make every one of these jobs faster, but the CI tail of any narrower fetch is unmeasured (n = 0), and its semantics need a per-gate audit across all five jobs. It is left as a question for the PM in the report. - **`ci.yml` Test Core shards**, same fetch, 45-minute wall: one shard was cancelled at the wall behind a 508 s checkout in the window (run 37338337323, `Test Core (2/6)`). This is outside this PR's file surface, and those steps run under the stall guard, whose budget gate reads that wall. The report names it for the seat. - **`Lint & Repo Gates`** is not exposed in the window: body max 34.5 min plus checkout max 730 s is 46.7, under its 55. --- _Generated by [Claude Code](https://claude.ai/code/session_01VF48aw8RPG6wzDnMgp6rtw)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 2301e17 commit 099a94d

2 files changed

Lines changed: 86 additions & 28 deletions

File tree

‎.github/workflows/governed-surface-guard.yml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,10 +75,22 @@ jobs:
7575
# live required set`, which is #5617's own defect.
7676
name: Governed Surface Queue Guard
7777
runs-on: ubuntu-latest
78-
timeout-minutes: 10
78+
# JOB WALL = CHECKOUT BUDGET (20) + BODY BUDGET, the split lint.yml's four
79+
# Type Check lanes use; `typecheck-source-gates` there argues it, holds the
80+
# pooled numbers for this same full-history fetch, and says what a timed-out
81+
# checkout step reads as (`failure`) against the wall (`cancelled`).
82+
# Window here: the 300 most recent completed runs of this workflow,
83+
# 2026-10-05T14:57Z → 2026-10-06T18:59Z, every attempt. Checkout: p50 31 s
84+
# / p90 156 s / p99 595 s, and the old 10-minute wall cancelled 4 runs
85+
# behind it (3 still inside the checkout), each a required context gone
86+
# red with no guard verdict. Body after the checkout, successful runs
87+
# (n = 288): max 1.1 minutes; 2× is 2.2, and never below 10. 20 + 10 = 30.
88+
timeout-minutes: 30
7989
steps:
8090
- name: Checkout repository
8191
uses: actions/checkout@v7
92+
# CHECKOUT BUDGET: lint.yml's `typecheck-source-gates` sizes it.
93+
timeout-minutes: 20
8294
with:
8395
# A governed-surface question answered over a truncated history
8496
# answers with SILENCE, and silence reads as compliance (#9902). The

‎.github/workflows/lint.yml‎

Lines changed: 73 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -5165,19 +5165,57 @@ jobs:
51655165
typecheck-source-gates:
51665166
name: Type Check · source gates
51675167
runs-on: ubuntu-latest
5168-
# JOB WALL — why every job here has one, the measurement window and the
5169-
# rule: the `lint` job's `timeout-minutes` comment.
5170-
# Measured: p50 3.2 / p99 3.5 / max 3.7 minutes (n = 94); 10 is 2.7×
5171-
# the maximum.
5172-
# A lane stopped by its wall ends `cancelled`, which the required
5173-
# `TypeScript Type Check` aggregate at the bottom of this file fails on.
5174-
timeout-minutes: 10
5168+
# JOB WALL — why every job here has one and the rule (about 2× the measured
5169+
# maximum, a 5-minute grain, never below 10, never above the queue's
5170+
# window): the `lint` job's `timeout-minutes` comment. The four Type Check
5171+
# lanes size their wall as TWO budgets added, argued here once:
5172+
#
5173+
# wall = CHECKOUT BUDGET (20, on the checkout step) + BODY BUDGET
5174+
#
5175+
# The checkout is a full-history fetch (`fetch-depth: 0`, argued at the
5176+
# step) and its time has nothing to do with the change under test. Window
5177+
# for every number in the four lane comments: the 300 most recent completed
5178+
# runs of this workflow, 2026-10-05T15:32Z → 2026-10-06T18:41Z, every
5179+
# attempt, from the jobs API's step timestamps. Pooled with the same fetch
5180+
# in governed-surface-guard.yml and ci.yml's Test Core shards over the same
5181+
# hours, n = 3,459: p50 31 s / p90 119 s / p99 553 s / max 895 s, and 19
5182+
# more cut off by a 10-minute wall. The shallow checkouts in ci.yml over
5183+
# the same hours (n = 2,743) never passed 107 s.
5184+
#
5185+
# The old 10-minute wall was sized when the checkout took ~30 s, so a slow
5186+
# fetch spent the gates' budget: in that window 8 runs of this lane ended
5187+
# `cancelled` at the wall behind a 436–600 s checkout, and the required
5188+
# aggregate failed each one with no gate run.
5189+
#
5190+
# CHECKOUT BUDGET, 20 minutes: 1.3× that 895 s maximum. The rule's 2× would
5191+
# be 30, but the queue's window binds first: the workspace lane's body
5192+
# budget is 35, and 35 + 20 = 55 is the cap the `lint` job argues. One
5193+
# fetch, one budget, the same number on all four lanes.
5194+
#
5195+
# BODY BUDGET, 10 minutes: everything after the checkout, successful runs
5196+
# only (n = 278): p50 2.8 / p99 3.1 / max 3.2 minutes; 2 × 3.2 = 6.4, and
5197+
# never below 10. 20 + 10 = 30.
5198+
#
5199+
# What the required `TypeScript Type Check` aggregate reads; it fails on
5200+
# anything but `success`:
5201+
# - a checkout that ends inside 20 minutes: nothing, the body still has
5202+
# its 10;
5203+
# - a checkout still running at 20: the step FAILS ("The action
5204+
# 'Checkout repository' has timed out after 20 minutes."; actions/runner
5205+
# `StepsRunner.RunStepAsync` sets `TaskResult.Failed` on a step timeout),
5206+
# the gates are skipped, the lane reads `failure`;
5207+
# - a later step that hangs: the wall CANCELS the job, the lane reads
5208+
# `cancelled`. Actions has no timeout over a group of steps, so after a
5209+
# fast checkout a hung gate can run up to the full 30.
5210+
timeout-minutes: 30
51755211
permissions:
51765212
contents: read
51775213

51785214
steps:
51795215
- name: Checkout repository
51805216
uses: actions/checkout@v7
5217+
# CHECKOUT BUDGET: the job's wall comment above sizes it.
5218+
timeout-minutes: 20
51815219
with:
51825220
# The authorable-surface deletion gate (#4650) below anchors on the
51835221
# MERGE BASE of HEAD with origin/main — that is the only anchor under
@@ -5958,19 +5996,22 @@ jobs:
59585996
typecheck-workspace:
59595997
name: Type Check · workspace
59605998
runs-on: ubuntu-latest
5961-
# JOB WALL — why every job here has one, the measurement window and the
5962-
# rule: the `lint` job's `timeout-minutes` comment.
5963-
# Measured: p50 13.0 / p99 16.0 / max 16.0 minutes (n = 86); 2 × 16.0 =
5964-
# 32, 30 on the grain.
5965-
# A lane stopped by its wall ends `cancelled`, which the required
5966-
# `TypeScript Type Check` aggregate at the bottom of this file fails on.
5967-
timeout-minutes: 30
5999+
# JOB WALL = CHECKOUT BUDGET (20) + BODY BUDGET; the split, its window, the
6000+
# checkout numbers and what the aggregate reads: `typecheck-source-gates`.
6001+
# Body after the checkout, successful runs (n = 264): p50 11.0 / p99 17.1 /
6002+
# max 17.9 minutes; 2 × 17.9 = 35.8, 35 on the grain. 20 + 35 = 55, the
6003+
# cap the `lint` job argues, which is what holds the checkout budget at 20.
6004+
# No run of this lane hit the old 30 in the window, but 17.9 plus that
6005+
# 895 s checkout is 32.8: the same exposure, not yet drawn.
6006+
timeout-minutes: 55
59686007
permissions:
59696008
contents: read
59706009

59716010
steps:
59726011
- name: Checkout repository
59736012
uses: actions/checkout@v7
6013+
# CHECKOUT BUDGET: `typecheck-source-gates` sizes it.
6014+
timeout-minutes: 20
59746015
with:
59756016
# The authorable-surface deletion gate (#4650) below anchors on the
59766017
# MERGE BASE of HEAD with origin/main — that is the only anchor under
@@ -6172,18 +6213,21 @@ jobs:
61726213
typecheck-debt:
61736214
name: Type Check · debt ledger
61746215
runs-on: ubuntu-latest
6175-
# JOB WALL — why every job here has one, the measurement window and the
6176-
# rule: the `lint` job's `timeout-minutes` comment.
6177-
# Measured: p50 5.8 / p99 7.4 / max 7.5 minutes (n = 92); 2 × 7.5 = 15.
6178-
# A lane stopped by its wall ends `cancelled`, which the required
6179-
# `TypeScript Type Check` aggregate at the bottom of this file fails on.
6180-
timeout-minutes: 15
6216+
# JOB WALL = CHECKOUT BUDGET (20) + BODY BUDGET; the split, its window, the
6217+
# checkout numbers and what the aggregate reads: `typecheck-source-gates`.
6218+
# Body after the checkout, successful runs (n = 278): p50 4.4 / p99 7.2 /
6219+
# max 7.4 minutes; 2 × 7.4 = 14.8, 15 on the grain. 20 + 15 = 35. In the
6220+
# window the old 15 cancelled this lane twice, behind a 642 s and a 774 s
6221+
# checkout.
6222+
timeout-minutes: 35
61816223
permissions:
61826224
contents: read
61836225

61846226
steps:
61856227
- name: Checkout repository
61866228
uses: actions/checkout@v7
6229+
# CHECKOUT BUDGET: `typecheck-source-gates` sizes it.
6230+
timeout-minutes: 20
61876231
with:
61886232
# The authorable-surface deletion gate (#4650) below anchors on the
61896233
# MERGE BASE of HEAD with origin/main — that is the only anchor under
@@ -6408,19 +6452,21 @@ jobs:
64086452
typecheck-consumers:
64096453
name: Type Check · consumer gates
64106454
runs-on: ubuntu-latest
6411-
# JOB WALL — why every job here has one, the measurement window and the
6412-
# rule: the `lint` job's `timeout-minutes` comment.
6413-
# Measured: p50 7.7 / p99 10.1 / max 10.1 minutes (n = 87); 2 × 10.1 =
6414-
# 20.2, 20 on the grain.
6415-
# A lane stopped by its wall ends `cancelled`, which the required
6416-
# `TypeScript Type Check` aggregate at the bottom of this file fails on.
6417-
timeout-minutes: 20
6455+
# JOB WALL = CHECKOUT BUDGET (20) + BODY BUDGET; the split, its window, the
6456+
# checkout numbers and what the aggregate reads: `typecheck-source-gates`.
6457+
# Body after the checkout, successful runs (n = 271): p50 7.1 / p99 10.5 /
6458+
# max 11.5 minutes; 2 × 11.5 = 23, 25 on the grain. 20 + 25 = 45. In the
6459+
# window the old 20 cancelled this lane twice, both in a merge group,
6460+
# behind a 393 s and a 665 s checkout.
6461+
timeout-minutes: 45
64186462
permissions:
64196463
contents: read
64206464

64216465
steps:
64226466
- name: Checkout repository
64236467
uses: actions/checkout@v7
6468+
# CHECKOUT BUDGET: `typecheck-source-gates` sizes it.
6469+
timeout-minutes: 20
64246470
with:
64256471
# The authorable-surface deletion gate (#4650) below anchors on the
64266472
# MERGE BASE of HEAD with origin/main — that is the only anchor under

0 commit comments

Comments
 (0)