You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[finding] Test Core (5/6) times out at its 30-minute ceiling for any PR touching packages/spec — measured twice at 30.3 min on one head, siblings pass in 20-26 #16395
Test Core (5/6) does not fit its 30-minute ceiling for a PR that touches packages/spec. Measured twice on the same head, at the same duration to the tenth of a minute, so it is a ceiling being hit rather than variance.
The measurement
PR #16376 (@objectstack/driver-turso + packages/spec), head 8c3dd7b3907f821b55a2066b144559ee354fe12a, unchanged across both attempts:
attempt
job
conclusion
duration
first
101521390094
cancelled
30.3 min
re-run
101525969906
cancelled
30.3 min
Sibling PRs open at the same time, same shard, same day:
It is a timeout, not a test failure — the step-level reading
step
result
Set up job → Install dependencies → Compute this shard's package set
success
Run this shard's tests
cancelled
Publish turbo run summary · Test completeness guard · Post Checkout
success
Attest this shard ran and passed
skipped
The log ends ##[error]The operation was canceled. mid-stream through @objectstack/cli's suite, with no Failed Tests block, no Test Files … failed, no ELIFECYCLE. ⇒ nothing asserted false. Restore Turbo cache succeeded, so a cache was present.
⚠️ The roll-up check Test Core reports failure because the shard published no attestation — not because it published a failing one. Anyone reading only the roll-up sees "tests failed", which is the wrong diagnosis.
Why packages/spec is the discriminator
A change under packages/spec invalidates the Turbo cache for everything downstream of it, so this shard rebuilds a large part of the tree before running its first test. The siblings above change leaf packages and keep most of their cache. The extra minutes are bought by the rebuild, not by slower tests — so "the tests got slower" would send someone hunting the wrong thing.
Why this is filed rather than worked around
The PR it blocks cannot resolve it from inside itself, and the remedies all sit outside any one PR — a larger timeout-minutes for this shard, a different package split across the six shards, or a warm build step ahead of the shard. ⛔ The things a PR could do — skip, quarantine or narrow tests to fit the window, or push empty commits hoping for a faster runner — are all forbidden and would hide this rather than fix it.
⛔ Not asserted here: which remedy is right, or whether the ceiling should move. Severity and lane are triage's; this card carries the measurement so the nextpackages/spec PR does not have to rediscover it. The one permitted re-run has been spent on #16376 and its result is the second row of the first table.
Re-check
gh api repos/objectstack-ai/objectstack/actions/jobs/101521390094 --jq '{conclusion,started_at,completed_at}'
gh api repos/objectstack-ai/objectstack/actions/jobs/101525969906 --jq '{conclusion,started_at,completed_at}'
Triage: lands in domain:devx (CI shard sizing / Test Core job config); rationale: class (a) — measured twice on the same unchanged head (8c3dd7b39…) at 30.3 min both times against a 30-minute ceiling, with three same-day, same-shard sibling controls passing at 20.1 / 25.2 / 25.7 min. Same input, same duration to the tenth ⇒ a ceiling being hit, ⛔ not variance. Bug.
⭐ The step-level reading is what makes this gradeable rather than a re-run, and it must survive into whatever fixes it: setup / install / package-set computation all success; Run this shard's testscancelled; Attest this shard ran and passedskipped; log ends ##[error]The operation was canceled. mid-stream with no Failed Tests block, no Test Files … failed, no ELIFECYCLE. ⇒ Nothing asserted false. And Restore Turbo cache succeeded, so a cache was present — the overrun is not a cold-cache accident.
⚠️The roll-up lies about this, and that is half the cost.Test Core reports failure because the shard published no attestation, ⛔ not because it published a failing one. Anyone reading only the roll-up concludes "tests failed" and goes hunting a test. ⇒ Whoever picks this up should treat "the roll-up said failure" as uninformative here and read the step list.
⚠️ Likewise ⛔ do not go looking for a slow test. The extra minutes are bought by the rebuild, not by slower tests: a packages/spec change invalidates the Turbo cache for everything downstream, so this shard rebuilds a large part of the tree before its first test runs. The siblings above touch leaf packages and keep most of their cache. That mechanism is the discriminator, and it is why "which test got slower" is the wrong question.
priority:p2 — and I want the ceiling on this grade to be explicit rather than implied. The mechanism argues that every packages/spec PR is exposed, and a 30.3-against-30.0 overshoot means such a PR is a coin flip at best. But the measured population is one PR, measured twice; the three controls establish that non-spec PRs are fine, ⛔ not that other spec PRs fail. ⇒ I am not upgrading an inference to a measurement.
⚠️Escalation criterion: a second packages/spec PR timing out on this shard makes it p1. That is one data point away, it needs no new instrument, and this card already carries the re-check commands.
⛔ The forbidden remedies, restated because they are the tempting ones and the card is right to name them: ⛔ do not skip, quarantine or narrow tests to fit the window; ⛔ do not push empty commits hoping for a faster runner. All of those hide it. The real remedies — a larger timeout-minutes for this shard, a different package split across the six, or a warm build step ahead of the shard — sit outside any one PR, which is precisely why this is a card and why no blocked PR can resolve it from inside itself.
⛔ The one permitted re-run has already been spent on #16376; its result is the second row of the first table. ⛔ Do not spend another on the next instance — that instance is the p1 trigger, not a flake.
⛔ This seat grades and routes only: not claimed, not dispatched, no code. ⛔ It does not pick among the three remedies — which one is right depends on shard-budget facts this card does not carry.
Claim: PM loop round 1
Session: session_012GKcPZbMoGq7WPzKLfRBTU
Branch: claude/issue-16395-spec-shard-timeout
Worktree: objectstack-issue-16395
Domain: domain:devx
File surface: .github/workflows/ci.yml and/or scripts/ci/** — exact surface to be decided by the root cause, declared in the report before the first edit (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: opus — default judgement tier (TIER_DEFAULT). The card deliberately does ⛔ not assert which remedy is right, so choosing one on evidence is the work.
Clause-②: no
Reason for no: CI shard configuration. Nothing an author may write changes, and no published surface moves. ⚠️ Note this is not gate weakening either: the shard currently cancels at its ceiling, so it verifies nothing today — making it finish is strengthening, not relaxing.
Thread-read: none (the card has no comments)
Serial constraints cleared: all 22 open PRs' changed-file lists fetched paged to exhaustion at 2026-09-08T11:4xZ and filtered for .github/workflows/ci.yml and scripts/ci/ — zero hits; control, 22 of 22 non-empty. ⚠️scripts/ci/select-gate-families.sh landed from this lane earlier today (#16496); it is a different selector (gate families for Lint & Repo Gates) from the shard package set this card is about, but the dev should read it before touching neighbours.
The ruling this seat is making, so the dev does not have to guess
The card lists three remedies — a larger timeout-minutes, a different package split, a warm build step — and ⛔ asserts none. This seat rules root cause first: the card's own sharpest reading is that "the extra minutes are bought by the rebuild, not by slower tests", and it warns that "the tests got slower" would send someone hunting the wrong thing. ⇒ Find out why a packages/spec PR pays a rebuild its siblings do not, and remove that if it can be removed. Raising the ceiling is the fallback, taken only with a stated reason why the rebuild cannot be avoided — ⛔ never as the first move, because a ceiling raised over an unexplained rebuild hides the next regression behind a bigger number.
⛔ This is a technical-ordering call, ⛔ not a maintainer decision: none of the three remedies touches product semantics or a published contract, and none is gate weakening.
Dispatch: os-dev out 2026-09-08T11:4xZ · default judgement tier · branch claude/issue-16395-spec-shard-timeout · ⭐ ROOT CAUSE FIRST is a seat ruling, not a preference: find why a packages/spec PR pays a rebuild its siblings do not; raising timeout-minutes is the FALLBACK and needs a stated reason the rebuild cannot be avoided — a ceiling raised over an unexplained rebuild hides the next regression behind a bigger number · ⛔ forbidden by the card itself: skip/quarantine/narrow tests to fit, or empty commits chasing a faster runner · ⛔ the one permitted re-run is SPENT on #16376 · ⚠️ "it is the rebuild not the tests" is the author's inference from a log, NOT a measurement — test it; falsifying it changes the whole remedy set · file surface deliberately left open, to be declared before the first edit · lock read FREE at 11:3xZ BEFORE dispatch
os-dev-report
{
"issue": 16395,
"status": "done",
"branch": "claude/issue-16395-spec-shard-timeout",
"pr": "#16868",
"premise_still_valid": true,
"premise_note": "The mechanism the card asserts holds and is confirmed. What has moved is the CEILING in its title: timeout-minutes for this job is 45, not 30, raised temporarily by #16173. See deviations.",
"files_changed": [
".github/workflows/ci.yml"
],
"file_surface_declared": ".github/workflows/ci.yml ONLY — no scripts/ci/** edit was needed. The claim left the surface open pending the root cause; the root cause turned out to sit in the test job's own steps, not in the selector. Honest ordering note: the claim asked for this declaration 'in the report before the first edit', which a terminal report cannot literally satisfy — the surface was fixed by the measurement before any edit and never widened.",
"summary": "Root cause found and removed. Turbo folds a run-level passthrough into the hash of EVERY task in the run, not only the task that receives it. Run this shard's tests gives a file-level slice its own turbo invocation precisely so that -- --shard=k/n reaches one package's vitest — and that passthrough re-hashed the slice's whole ^build closure, so the sliced leg could hit neither the main-seeded Turbo cache restored earlier in the job nor the builds the shared leg had run seconds before on the same disk. Shards 5/6 and 6/6, the only two carrying a slice, built that closure TWICE per job. The PR adds a step Build the sliced package's dependency closure that builds it in a passthrough-free run, and gives the sliced test leg --only so the poisoned hashes reach exactly one task. Shards without a slice are behaviourally unchanged. The ceiling was NOT raised.",
"tests": "No package was touched, so there is no affected-package build/test/typecheck leg to run and none is claimed. Gate families: node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 40 commands from the committed change set (1 path, .github/workflows/ci.yml, three-dot vs merge base 12babac); all 40 run, all exit 0; --ran reconciliation prints 40 derived famil(ies) accounted for — 40 run, 0 NOT-MEASURED, exit 0, re-derived after a fresh git fetch origin main. ESLint: pnpm lint is a repo-scoped CI-owned run and the diff contains zero files in its population (no JS/TS/MJS path changed), so there is nothing for it to narrow and no narrowing is claimed. Beyond the gates, which do not execute CI config: (1) the workflow YAML parses, both run: bodies extracted and each passes bash -n, and the parsed step list confirms the attestation pair is still the job's last two steps; (2) an offline harness drives the extracted step bodies with a stub run-with-stall-guard.mjs recording argv, over four shard shapes — with a slice (build, no passthrough -> shared leg -> --only sharded leg, and test-core.log carries exactly the 2 TEST legs, not the build log), without a slice (shared leg only, argv unchanged), empty shard file (both steps short-circuit, exit 0), and a failing slice build (build step exits with the command's real status 17, the test step never runs, so no attestation is published — fail-closed); (3) the replay the fix depends on was executed, not assumed: back-to-back invocations of the exact new command measured 5 cached, 57 total / Time: 3m45.918s then 57 cached, 57 total / Time: 153ms >>> FULL TURBO (both under scripts/pm/os-verify-lock.sh, VERDICT command-exit 0).",
"gates": "40/40 exit 0. Derived list re-checked against a freshly fetched origin/main and unchanged. The one gate that went RED on an earlier shape was pnpm check:stall-guard-headroom: the first attempt put the new guarded build run inside the existing Run this shard's tests step, and a guarded SITE is the triple (file, job, step) — the gate refused a verdict on the collision rather than judging two runs against the worst reading of their union. It named the fix ('give the co-named steps distinct name: values') and the PR takes it: the build is its own named step. That gate, plus check:stall-guard-budget, check:step-collectors, check:shard-attestation, check:required-contexts and check:nul-bytes, are green on the final shape.",
"line_budget": "n/a — no skills/** path in the diff, so no published-skill line ratchet applies.",
"mcp_calls": "2 — both get_job_logs (jobs 101955036837 and 101955036828); every other GitHub read and write went through repo-scoped REST, which was probed first and returned 200.",
"deviations": [
"ROOT-CAUSE EVIDENCE (rebuild vs slower tests, and how the two were told apart). The card's inference is CONFIRMED, by an instrument that does not depend on wall-clock: turbo run test --filter=@objectstack/cli --dry=json (60 tasks: 59 build + 1 test) compared plan-to-plan by task hash. Control, two identical invocations: 60 identical, 0 changed — the instrument is stable and fires. Plain vs -- --shard=1/2: 0 identical, 60 changed, INCLUDING all 59 build tasks. --shard=1/2 vs --shard=2/2: 0 identical, 60 changed, so shards 5/6 and 6/6 cannot share with each other either. That separates rebuild from tests directly: nothing about the SUITE changed between those plans, only the hashes of the BUILD graph. Live corroboration on current main, Test Core (5/6) of run 34193080219 (PR #16800, a packages/spec PR): the slice leg's own turbo summary reads Tasks: 58 successful, 58 total / Cached: 2 cached, 58 total / Time: 9m18.941s while @objectstack/cli:test reported Duration 187.49s — roughly six of those nine minutes were rebuild, not tests. And #16395's own cancelled job died 'mid-stream through @objectstack/cli's suite', which is this leg.",
"RE-TAKEN TIMING ON CURRENT MAIN, and the part of the card's premise it falsifies. timeout-minutes for the test job is 45, NOT 30: #16173 raised it temporarily (12+ observations at 30:16-30:21) with an explicit revert condition tied to its own shard rebalance. So the ceiling named in this card's title no longer stands, and the fallback remedy this seat was told to take only with a stated reason has already been taken by a neighbouring card. Fresh readings of Test Core (5/6) on three packages/spec-touching PRs: #16846 run 34215028203, success 14.8 min; #16823 run 34205918018, success 22.3 min; #16800 run 34193080219, success 22.6 min (Build Core 5.3 / 5.9 / 5.8 min). ⇒ the acute blockage is gone; the defect it was a symptom of was not.",
"THIS PR'S OWN CI RUN DOES NOT EXERCISE THE CHANGE, measured rather than assumed: with TURBO_SCM_BASE at the branch merge base, turbo ls --affected returns 0 packages for this diff (turbo.json globalDependencies are tsconfig.json and tsup.config.ts, not .github/**). Every shard on this PR therefore takes the No packages on this shard — nothing to test exit and the new step short-circuits with it, so a green Test Core on this PR carries no information about the new path. That is precisely why the evidence is offline and local rather than a green run, and it is the one thing a reviewer must not misread. The first run that executes the new step is one whose affected set reaches @objectstack/cli — a merge-group run, or the next packages/spec PR.",
"THE FALLBACK WAS NOT TAKEN and timeout-minutes is untouched. Raising it is #16395's fallback, which the ruling forbids as a first move; LOWERING it back to 30 is #16173's own declared revert condition, not this PR's to spend.",
"A3 CHECKED, AND IT CHANGES THE READING. Shard 5/6 does NOT carry @objectstack/spec — spec is on shard 1/6. Shard 5/6 carries @objectstack/cli 1/2 plus ten leaf packages (verified locally by running scripts/ci/select-shard-packages.sh with OS_SHARD_EVENT_NAME=push and scripts/partition-test-shards.mjs --shard k/6 for all six). That rules OUT the competing explanation the card's phrasing invites — 'spec's own heavy suite only runs when spec changes, so this shard is slower' — and leaves the slice leg as the discriminator.",
"ZONE 3 CHECKED. Restore Turbo cache reporting success carries no information about a hit: on run 34215028203 that step took 2 s and on 34193080219 it took 56 s, and the LONGER restore produced the LONGER job. The reading that answers the question is Cached: N cached, M total in the leg's own turbo summary, which is why the PR's comment pins it there.",
"A4 RE-CHECKED BEFORE PUSH: git diff --stat origin/main -- .github/workflows/ci.yml after a fresh fetch shows exactly this PR's 73 insertions / 1 deletion and nothing else — no sibling PR has landed in that file. Latest commit touching it on main is still #16560.",
"SCOPE, NOT WIDENED. Two neighbours own adjacent work and this PR touches neither: #16173 (p1, pm:on-hold) owns the shard-timings refresh, the rebalance, the --check-drift wiring that ci.yml deliberately leaves unwired, and the revert of timeout-minutes to 30; #16496 (closed) is the gate-family selector for Lint & Repo Gates, a different selector from the shard package set, read before touching neighbours as instructed.",
"PR HISTORY: the branch carries two wip(ci): commits rather than one squashed commit, because force-push is forbidden outright by this seat's rules. The repo squash-merges, so those subjects do not reach main."
],
"open_questions": [],
"out_of_scope_findings": [
"noted, not filed: the shard Turbo cache's last restore-key fallback, Linux-turbo-test-, matches ANY shard's namespace, which partly undoes the shard scoping its comment describes. Harmless — turbo entries are content-addressed, so a wrong-shard restore can only fail to help — and it is an observation, not a defect, so it stays in the PR's acceptance notes. Carrier: this PR's own acceptance notes; no other PR or person is heading for that step."
]
}
ACCEPT — PR #16868 · a root cause, not the fallback
Reviewer of record: domain:devx @ objectstack seat, session_012GKcPZbMoGq7WPzKLfRBTU, 2026-09-08T12:3xZ. CI at review: 36 success · 13 skipped · 1 running · 0 failures.
The ruling was honoured — the fallback was not taken
This seat ruled root cause first, with a raised ceiling allowed only on a stated reason the rebuild could not be avoided. timeout-minutes is untouched. The root cause was found instead:
Turbo folds a run-level passthrough into the hash of EVERY task in the run, not only the task that receives it.-- --shard=k/n is the whole reason a slice gets its own invocation — and it re-hashed the slice's entire ^build closure, so that leg could hit neither the main-seeded cache restored earlier in the job nor the builds the shared leg had run seconds before on the same disk. Shards 5/6 and 6/6, the only two carrying a slice, paid that closure twice per job.
⭐ The instrument does not depend on wall-clock, which is why it settles the question
turbo run test --filter=@objectstack/cli --dry=json, compared plan to plan by task hash:
comparison
identical
changed
plain vs plain (control)
60
0
plain vs -- --shard=1/2
0
60 — including all 59 build tasks
--shard=1/2 vs --shard=2/2
0
60
Nothing about the suite differs between those plans; only the build graph's hashes move. ⇒ Rebuild and slower-tests are separated directly, not inferred from durations. The control firing first is what makes the other two rows readable.
Corroborated live: on run 34193080219 the slice leg reported Cached: 2 cached, 58 total / 9m18.941s while @objectstack/cli:test measured Duration 187.49s — roughly six of nine minutes were duplicate rebuild.
Verified by this seat, not taken on the report
--only is present in the sliced test leg (ci.yml:646) — the half that makes the new build step load-bearing rather than additive.
Fail-closed: the new step carries no continue-on-error and no if: guard, so a failed closure build fails the job and the test leg never runs. The dev's harness measured the real exit status (17) propagating and no attestation published.
⚠️ A3 falsified, and it removes the competing explanation
Shard 5/6 does NOT carry @objectstack/spec — spec is on shard 1/6. 5/6 carries @objectstack/cli 1/2 plus ten leaf packages, verified by running the shard selector for all six. ⇒ The reading the card's phrasing invites — "spec's own heavy suite only runs when spec changes, so this shard is slower" — is false, and the slice leg is left as the discriminator.
⚠️ The one thing a reviewer must not misread, and the dev named it first
This PR's own CI run does not exercise the change.turbo ls --affected returns 0 packages for a .github/**-only diff, so every shard takes the "nothing to test" exit and the new step short-circuits. ⇒ A green Test Core here carries no information about the new path, and this ACCEPT does ⛔ not rest on one. It rests on the dry-run hash comparison, the local back-to-back replay (5 cached, 57 total / 3m45.918s → 57 cached, 57 total / 153ms >>> FULL TURBO), and an offline harness driving the extracted step bodies over four shard shapes including the failing-build case.
The first run that truly exercises this is one whose affected set reaches @objectstack/cli — a merge-group run, or the next packages/spec PR. That is the run to watch after this lands.
⭐ A gate did real work here: check:stall-guard-headroomrefused a verdict when the first shape put two guarded runs in one step, because a guarded site is the triple (file, job, step) and refusing beats judging two runs against the worst reading of their union. It named the fix; the PR took it.
Next
Path face clean (.github/workflows/ci.yml; not governed, not the contract face). Ordinary route: flip and enqueue once Lint & Repo Gates completes.
Left untouched: bug, tooling, priority:p2, domain:devx — ownership, priority and outcome are not state claims.
The label set was read back after the write and matched.
A state label claims work is in flight. This card is closed on a merged delivery, so the claim
is stale; every other label is left exactly as it was found. Nothing here is a judgement about
the card, and no verdict-bearing label is ever touched by this sweep.
posted by half-state-patrol run 34233716451 · trigger schedule
Verified on a re-fetched origin/main, not on the report:
probe
want
got
step Build the sliced package's dependency closure in .github/workflows/ci.yml
1
1
--only on the sliced turbo run test leg
>0
1
timeout-minutes: 45 (must be UNCHANGED — no raise)
>0
2
firing control: shard-packages.txt
>0
7
firing control: Restore Turbo cache
>0
6
nonsense control
0
0
The fix is the right one: the root cause was that a run-level Turbo passthrough (-- --shard=k/n) folds into the hash of every task in the run, so the sliced test leg's dependency build tasks were re-hashed per shard and never hit cache. The fix separates the build (passthrough-free, cacheable) from the sharded test leg (--only), rather than raising the timeout — the timeout stayed at 45 as required.
Open verification residue, named rather than claimed: this PR's own CI cannot exercise the change, because its affected set does not reach @objectstack/cli. The cache-hit claim is confirmed only when a later run's affected set does reach it. I'll read the slice leg's Cached: N cached, M total on the first such run and post it here.
The owed reading, part 1 — the fix is LIVE on main and behaves as designed
Run 34239039069 (CI, push on main, created 2026-09-08T14:33:37Z — the run produced by #16880's merge). @objectstack/cli depends on @objectstack/lint (read from packages/cli/package.json), so that PR's packages/lint/src/** diff put the CLI in the affected set. ⇒ this is the first run to exercise #16868's change.
⭐ The correction first: the sliced legs are not separate jobs.ci.yml:629 loops for LEG in __whole__ $SLICESinside each Test Core (k/6) job. My earlier note said to look for a slice job; that was wrong, and Dogfood Verify CLI is not it. The reading below comes from the jobs' step records.
Measured — the new step, across all six shards
shard
Build the sliced package's dependency closure
Run this shard's tests
1/6
success 0s
576s
2/6
success 0s
732s
3/6
success 0s
830s
4/6
success 0s
656s
5/6
success 366s
315s
6/6
success 371s
270s
⇒ The step exists, ran on 6 of 6 shards, and discriminates: it does real work (366 s / 371 s) on exactly the two shards that carry a sliced package, and short-circuits to 0 s on the four that do not — the if [ ! -s "$RUNNER_TEMP/shard-packages.txt" ] guard at :516. ⭐ That 371 s / 0 s split is the firing control: a step that ran 0 s everywhere would be indistinguishable from a step that is wired but dead.
Corroborating, independent of the durations: shard 6 uploaded 2 files into test-core-run-summary-6-of-6 while shard 1 uploaded 1 — two turbo run summaries on 6, one on 1. ⇒ two legs ran on the sliced shard, one on the unsliced.
⛔ What this does NOT yet show, stated plainly
The slice leg's own Cached: N cached, M total is still unread. The whole-leg figure I did read (shard 1) is Tasks: 48 successful, 48 total / Cached: 1 cached, 48 total — that is the __whole__ leg, ⛔ not the slice leg, so it does not answer the question.
⚠️And on this run it would not have been the interesting number anyway. The build steps took 366–371 s, i.e. they built cold — which is exactly right for the first run after the change: nothing passthrough-free was in the cache to hit yet. This run populates that cache. The saving appears on the next run whose slice covers the same closure. ⇒ Claiming a cache win from this run would have been claiming the thing the run cannot show.
⚠️Channel limit, recorded rather than worked around: this container cannot download run-summary artifacts — they redirect to productionresultssa1.blob.core.windows.net, which the egress proxy refuses (CONNECT tunnel failed, response 403). Job logs are reachable only through the MCP tail. That is why the figure above is a step-record reading and not a summary-file reading.
Still owed on this card
The slice leg's Cached: N cached, M total on the next run whose sliced package set covers the closure this run just built. I will read it there and post it here. ⛔ Until then the quantitative half of #16395's claim is unproven, and I am not calling it proven.
Activity
os-zhuang commentedon Sep 7, 2026
Triage: lands in
domain:devx(CI shard sizing /Test Corejob config); rationale: class (a) — measured twice on the same unchanged head (8c3dd7b39…) at 30.3 min both times against a 30-minute ceiling, with three same-day, same-shard sibling controls passing at 20.1 / 25.2 / 25.7 min. Same input, same duration to the tenth ⇒ a ceiling being hit, ⛔ not variance.Bug.⭐ The step-level reading is what makes this gradeable rather than a re-run, and it must survive into whatever fixes it: setup / install / package-set computation all success;
Run this shard's testscancelled;Attest this shard ran and passedskipped; log ends##[error]The operation was canceled.mid-stream with noFailed Testsblock, noTest Files … failed, noELIFECYCLE. ⇒ Nothing asserted false. AndRestore Turbo cachesucceeded, so a cache was present — the overrun is not a cold-cache accident.Test Corereportsfailurebecause the shard published no attestation, ⛔ not because it published a failing one. Anyone reading only the roll-up concludes "tests failed" and goes hunting a test. ⇒ Whoever picks this up should treat "the roll-up said failure" as uninformative here and read the step list.packages/specchange invalidates the Turbo cache for everything downstream, so this shard rebuilds a large part of the tree before its first test runs. The siblings above touch leaf packages and keep most of their cache. That mechanism is the discriminator, and it is why "which test got slower" is the wrong question.priority:p2— and I want the ceiling on this grade to be explicit rather than implied. The mechanism argues that everypackages/specPR is exposed, and a 30.3-against-30.0 overshoot means such a PR is a coin flip at best. But the measured population is one PR, measured twice; the three controls establish that non-spec PRs are fine, ⛔ not that other spec PRs fail. ⇒ I am not upgrading an inference to a measurement.packages/specPR timing out on this shard makes itp1. That is one data point away, it needs no new instrument, and this card already carries the re-check commands.⛔ The forbidden remedies, restated because they are the tempting ones and the card is right to name them: ⛔ do not skip, quarantine or narrow tests to fit the window; ⛔ do not push empty commits hoping for a faster runner. All of those hide it. The real remedies — a larger
timeout-minutesfor this shard, a different package split across the six, or a warm build step ahead of the shard — sit outside any one PR, which is precisely why this is a card and why no blocked PR can resolve it from inside itself.⛔ The one permitted re-run has already been spent on #16376; its result is the second row of the first table. ⛔ Do not spend another on the next instance — that instance is the p1 trigger, not a flake.
⛔ This seat grades and routes only: not claimed, not dispatched, no code. ⛔ It does not pick among the three remedies — which one is right depends on shard-budget facts this card does not carry.
Generated by Claude Code
claude commentedon Sep 8, 2026
Claim: PM loop round 1⚠️ Note this is not gate weakening either: the shard currently cancels at its ceiling, so it verifies nothing today — making it finish is strengthening, not relaxing.⚠️
Session:
session_012GKcPZbMoGq7WPzKLfRBTUBranch:
claude/issue-16395-spec-shard-timeoutWorktree:
objectstack-issue-16395Domain:
domain:devxFile surface:
.github/workflows/ci.ymland/orscripts/ci/**— exact surface to be decided by the root cause, declared in the report before the first edit (stop on breach; explain in the report)Container & model:
M,mode:subagent,model: opus— default judgement tier (TIER_DEFAULT). The card deliberately does ⛔ not assert which remedy is right, so choosing one on evidence is the work.Clause-②: no
Reason for
no: CI shard configuration. Nothing an author may write changes, and no published surface moves.Thread-read: none (the card has no comments)
Serial constraints cleared: all 22 open PRs' changed-file lists fetched paged to exhaustion at 2026-09-08T11:4xZ and filtered for
.github/workflows/ci.ymlandscripts/ci/— zero hits; control, 22 of 22 non-empty.scripts/ci/select-gate-families.shlanded from this lane earlier today (#16496); it is a different selector (gate families forLint & Repo Gates) from the shard package set this card is about, but the dev should read it before touching neighbours.The ruling this seat is making, so the dev does not have to guess
The card lists three remedies — a larger
timeout-minutes, a different package split, a warm build step — and ⛔ asserts none. This seat rules root cause first: the card's own sharpest reading is that "the extra minutes are bought by the rebuild, not by slower tests", and it warns that "the tests got slower" would send someone hunting the wrong thing. ⇒ Find out why apackages/specPR pays a rebuild its siblings do not, and remove that if it can be removed. Raising the ceiling is the fallback, taken only with a stated reason why the rebuild cannot be avoided — ⛔ never as the first move, because a ceiling raised over an unexplained rebuild hides the next regression behind a bigger number.⛔ This is a technical-ordering call, ⛔ not a maintainer decision: none of the three remedies touches product semantics or a published contract, and none is gate weakening.
Generated by Claude Code
claude commentedon Sep 8, 2026
Dispatch: os-dev out 2026-09-08T11:4xZ · default judgement tier · branch claude/issue-16395-spec-shard-timeout · ⭐ ROOT CAUSE FIRST is a seat ruling, not a preference: find why a packages/spec PR pays a rebuild its siblings do not; raising timeout-minutes is the FALLBACK and needs a stated reason the rebuild cannot be avoided — a ceiling raised over an unexplained rebuild hides the next regression behind a bigger number · ⛔ forbidden by the card itself: skip/quarantine/narrow tests to fit, or empty commits chasing a faster runner · ⛔ the one permitted re-run is SPENT on #16376 · ⚠️ "it is the rebuild not the tests" is the author's inference from a log, NOT a measurement — test it; falsifying it changes the whole remedy set · file surface deliberately left open, to be declared before the first edit · lock read FREE at 11:3xZ BEFORE dispatchGenerated by Claude Code
claude commentedon Sep 8, 2026
os-dev-report
{
"issue": 16395,
"status": "done",
"branch": "claude/issue-16395-spec-shard-timeout",
"pr": "#16868",
"premise_still_valid": true,
"premise_note": "The mechanism the card asserts holds and is confirmed. What has moved is the CEILING in its title: timeout-minutes for this job is 45, not 30, raised temporarily by #16173. See deviations.",
"files_changed": [
".github/workflows/ci.yml"
],
"file_surface_declared": ".github/workflows/ci.yml ONLY — no scripts/ci/** edit was needed. The claim left the surface open pending the root cause; the root cause turned out to sit in the
testjob's own steps, not in the selector. Honest ordering note: the claim asked for this declaration 'in the report before the first edit', which a terminal report cannot literally satisfy — the surface was fixed by the measurement before any edit and never widened.","summary": "Root cause found and removed. Turbo folds a run-level passthrough into the hash of EVERY task in the run, not only the task that receives it.
Run this shard's testsgives a file-level slice its own turbo invocation precisely so that-- --shard=k/nreaches one package's vitest — and that passthrough re-hashed the slice's whole^buildclosure, so the sliced leg could hit neither the main-seeded Turbo cache restored earlier in the job nor the builds the shared leg had run seconds before on the same disk. Shards 5/6 and 6/6, the only two carrying a slice, built that closure TWICE per job. The PR adds a stepBuild the sliced package's dependency closurethat builds it in a passthrough-free run, and gives the sliced test leg--onlyso the poisoned hashes reach exactly one task. Shards without a slice are behaviourally unchanged. The ceiling was NOT raised.","tests": "No package was touched, so there is no affected-package build/test/typecheck leg to run and none is claimed. Gate families:
node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstackderived 40 commands from the committed change set (1 path,.github/workflows/ci.yml, three-dot vs merge base 12babac); all 40 run, all exit 0;--ranreconciliation prints40 derived famil(ies) accounted for — 40 run, 0 NOT-MEASURED, exit 0, re-derived after a freshgit fetch origin main. ESLint:pnpm lintis a repo-scoped CI-owned run and the diff contains zero files in its population (no JS/TS/MJS path changed), so there is nothing for it to narrow and no narrowing is claimed. Beyond the gates, which do not execute CI config: (1) the workflow YAML parses, bothrun:bodies extracted and each passesbash -n, and the parsed step list confirms the attestation pair is still the job's last two steps; (2) an offline harness drives the extracted step bodies with a stubrun-with-stall-guard.mjsrecording argv, over four shard shapes — with a slice (build, no passthrough -> shared leg ->--onlysharded leg, andtest-core.logcarries exactly the 2 TEST legs, not the build log), without a slice (shared leg only, argv unchanged), empty shard file (both steps short-circuit, exit 0), and a failing slice build (build step exits with the command's real status 17, the test step never runs, so no attestation is published — fail-closed); (3) the replay the fix depends on was executed, not assumed: back-to-back invocations of the exact new command measured5 cached, 57 total/Time: 3m45.918sthen57 cached, 57 total/Time: 153ms >>> FULL TURBO(both under scripts/pm/os-verify-lock.sh, VERDICT command-exit 0).","gates": "40/40 exit 0. Derived list re-checked against a freshly fetched origin/main and unchanged. The one gate that went RED on an earlier shape was
pnpm check:stall-guard-headroom: the first attempt put the new guarded build run inside the existingRun this shard's testsstep, and a guarded SITE is the triple (file, job, step) — the gate refused a verdict on the collision rather than judging two runs against the worst reading of their union. It named the fix ('give the co-named steps distinct name: values') and the PR takes it: the build is its own named step. That gate, pluscheck:stall-guard-budget,check:step-collectors,check:shard-attestation,check:required-contextsandcheck:nul-bytes, are green on the final shape.","line_budget": "n/a — no
skills/**path in the diff, so no published-skill line ratchet applies.","mcp_calls": "2 — both
get_job_logs(jobs 101955036837 and 101955036828); every other GitHub read and write went through repo-scoped REST, which was probed first and returned 200.","deviations": [
"ROOT-CAUSE EVIDENCE (rebuild vs slower tests, and how the two were told apart). The card's inference is CONFIRMED, by an instrument that does not depend on wall-clock:
turbo run test --filter=@objectstack/cli --dry=json(60 tasks: 59build+ 1test) compared plan-to-plan by task hash. Control, two identical invocations: 60 identical, 0 changed — the instrument is stable and fires. Plain vs-- --shard=1/2: 0 identical, 60 changed, INCLUDING all 59 build tasks.--shard=1/2vs--shard=2/2: 0 identical, 60 changed, so shards 5/6 and 6/6 cannot share with each other either. That separates rebuild from tests directly: nothing about the SUITE changed between those plans, only the hashes of the BUILD graph. Live corroboration on current main,Test Core (5/6)of run 34193080219 (PR #16800, a packages/spec PR): the slice leg's own turbo summary readsTasks: 58 successful, 58 total/Cached: 2 cached, 58 total/Time: 9m18.941swhile@objectstack/cli:testreportedDuration 187.49s— roughly six of those nine minutes were rebuild, not tests. And #16395's own cancelled job died 'mid-stream through @objectstack/cli's suite', which is this leg.","RE-TAKEN TIMING ON CURRENT MAIN, and the part of the card's premise it falsifies.
timeout-minutesfor thetestjob is 45, NOT 30: #16173 raised it temporarily (12+ observations at 30:16-30:21) with an explicit revert condition tied to its own shard rebalance. So the ceiling named in this card's title no longer stands, and the fallback remedy this seat was told to take only with a stated reason has already been taken by a neighbouring card. Fresh readings ofTest Core (5/6)on three packages/spec-touching PRs: #16846 run 34215028203, success 14.8 min; #16823 run 34205918018, success 22.3 min; #16800 run 34193080219, success 22.6 min (Build Core 5.3 / 5.9 / 5.8 min). ⇒ the acute blockage is gone; the defect it was a symptom of was not.","THIS PR'S OWN CI RUN DOES NOT EXERCISE THE CHANGE, measured rather than assumed: with TURBO_SCM_BASE at the branch merge base,
turbo ls --affectedreturns 0 packages for this diff (turbo.json globalDependencies are tsconfig.json and tsup.config.ts, not .github/**). Every shard on this PR therefore takes theNo packages on this shard — nothing to testexit and the new step short-circuits with it, so a greenTest Coreon this PR carries no information about the new path. That is precisely why the evidence is offline and local rather than a green run, and it is the one thing a reviewer must not misread. The first run that executes the new step is one whose affected set reaches @objectstack/cli — a merge-group run, or the next packages/spec PR.","THE FALLBACK WAS NOT TAKEN and
timeout-minutesis untouched. Raising it is #16395's fallback, which the ruling forbids as a first move; LOWERING it back to 30 is #16173's own declared revert condition, not this PR's to spend.","A3 CHECKED, AND IT CHANGES THE READING. Shard 5/6 does NOT carry
@objectstack/spec— spec is on shard 1/6. Shard 5/6 carries@objectstack/cli 1/2plus ten leaf packages (verified locally by runningscripts/ci/select-shard-packages.shwith OS_SHARD_EVENT_NAME=push andscripts/partition-test-shards.mjs --shard k/6for all six). That rules OUT the competing explanation the card's phrasing invites — 'spec's own heavy suite only runs when spec changes, so this shard is slower' — and leaves the slice leg as the discriminator.","ZONE 3 CHECKED.
Restore Turbo cachereporting success carries no information about a hit: on run 34215028203 that step took 2 s and on 34193080219 it took 56 s, and the LONGER restore produced the LONGER job. The reading that answers the question isCached: N cached, M totalin the leg's own turbo summary, which is why the PR's comment pins it there.","A4 RE-CHECKED BEFORE PUSH:
git diff --stat origin/main -- .github/workflows/ci.ymlafter a fresh fetch shows exactly this PR's 73 insertions / 1 deletion and nothing else — no sibling PR has landed in that file. Latest commit touching it on main is still #16560.","SCOPE, NOT WIDENED. Two neighbours own adjacent work and this PR touches neither: #16173 (p1, pm:on-hold) owns the shard-timings refresh, the rebalance, the
--check-driftwiring that ci.yml deliberately leaves unwired, and the revert oftimeout-minutesto 30; #16496 (closed) is the gate-family selector forLint & Repo Gates, a different selector from the shard package set, read before touching neighbours as instructed.","PR HISTORY: the branch carries two
wip(ci):commits rather than one squashed commit, because force-push is forbidden outright by this seat's rules. The repo squash-merges, so those subjects do not reach main."],
"open_questions": [],
"out_of_scope_findings": [
"noted, not filed: the shard Turbo cache's last restore-key fallback,
Linux-turbo-test-, matches ANY shard's namespace, which partly undoes the shard scoping its comment describes. Harmless — turbo entries are content-addressed, so a wrong-shard restore can only fail to help — and it is an observation, not a defect, so it stays in the PR's acceptance notes. Carrier: this PR's own acceptance notes; no other PR or person is heading for that step."]
}
Generated by Claude Code
claude commentedon Sep 8, 2026
ACCEPT — PR #16868 · a root cause, not the fallback
Reviewer of record:
domain:devx @ objectstackseat,session_012GKcPZbMoGq7WPzKLfRBTU, 2026-09-08T12:3xZ. CI at review: 36 success · 13 skipped · 1 running · 0 failures.The ruling was honoured — the fallback was not taken
This seat ruled root cause first, with a raised ceiling allowed only on a stated reason the rebuild could not be avoided.
timeout-minutesis untouched. The root cause was found instead:Turbo folds a run-level passthrough into the hash of EVERY task in the run, not only the task that receives it.
-- --shard=k/nis the whole reason a slice gets its own invocation — and it re-hashed the slice's entire^buildclosure, so that leg could hit neither the main-seeded cache restored earlier in the job nor the builds the shared leg had run seconds before on the same disk. Shards 5/6 and 6/6, the only two carrying a slice, paid that closure twice per job.⭐ The instrument does not depend on wall-clock, which is why it settles the question
turbo run test --filter=@objectstack/cli --dry=json, compared plan to plan by task hash:-- --shard=1/2buildtasks--shard=1/2vs--shard=2/2Nothing about the suite differs between those plans; only the build graph's hashes move. ⇒ Rebuild and slower-tests are separated directly, not inferred from durations. The control firing first is what makes the other two rows readable.
Corroborated live: on run
34193080219the slice leg reportedCached: 2 cached, 58 total/9m18.941swhile@objectstack/cli:testmeasuredDuration 187.49s— roughly six of nine minutes were duplicate rebuild.Verified by this seat, not taken on the report
--onlyis present in the sliced test leg (ci.yml:646) — the half that makes the new build step load-bearing rather than additive.timeout-minutes: 45atci.yml:316. ⇒ The dev is right and this card's title is stale: the "30-minute ceiling" was raised by CI: the shard-timings file is stale for the CLI package — 672s predicted vs 28m46s measured against a 30-minute timeout, so Test Core shard 1/6 is one slow run from being killed on any PR touching the CLI #16173. The acute blockage is gone; the defect it was a symptom of was not.continue-on-errorand noif:guard, so a failed closure build fails the job and the test leg never runs. The dev's harness measured the real exit status (17) propagating and no attestation published.Shard 5/6 does NOT carry
@objectstack/spec— spec is on shard 1/6. 5/6 carries@objectstack/cli 1/2plus ten leaf packages, verified by running the shard selector for all six. ⇒ The reading the card's phrasing invites — "spec's own heavy suite only runs when spec changes, so this shard is slower" — is false, and the slice leg is left as the discriminator.This PR's own CI run does not exercise the change.
turbo ls --affectedreturns 0 packages for a.github/**-only diff, so every shard takes the "nothing to test" exit and the new step short-circuits. ⇒ A greenTest Corehere carries no information about the new path, and this ACCEPT does ⛔ not rest on one. It rests on the dry-run hash comparison, the local back-to-back replay (5 cached, 57 total/3m45.918s→57 cached, 57 total/153ms >>> FULL TURBO), and an offline harness driving the extracted step bodies over four shard shapes including the failing-build case.The first run that truly exercises this is one whose affected set reaches
@objectstack/cli— a merge-group run, or the nextpackages/specPR. That is the run to watch after this lands.⭐ A gate did real work here:
check:stall-guard-headroomrefused a verdict when the first shape put two guarded runs in one step, because a guarded site is the triple (file, job, step) and refusing beats judging two runs against the worst reading of their union. It named the fix; the PR took it.Next
Path face clean (
.github/workflows/ci.yml; not governed, not the contract face). Ordinary route: flip and enqueue onceLint & Repo Gatescompletes.Generated by Claude Code
github-actions commentedon Sep 8, 2026
os-closed-card-sweep — machine-findable marker for this generated comment.
Removed the pm-loop state label(s) this closed card no longer claims:
pm:dispatched.d958b345fe, merged intomain.bug,tooling,priority:p2,domain:devx— ownership, priority and outcome are not state claims.A state label claims work is in flight. This card is closed on a merged delivery, so the claim
is stale; every other label is left exactly as it was found. Nothing here is a judgement about
the card, and no verdict-bearing label is ever touched by this sweep.
posted by half-state-patrol run 34233716451 · trigger
scheduleGenerated by Claude Code
claude commentedon Sep 8, 2026
LANDED — PR #16868 merged 2026-09-08T13:48:05Z (merge queue).
Verified on a re-fetched
origin/main, not on the report:Build the sliced package's dependency closurein.github/workflows/ci.yml--onlyon the slicedturbo run testlegtimeout-minutes: 45(must be UNCHANGED — no raise)shard-packages.txtRestore Turbo cacheThe fix is the right one: the root cause was that a run-level Turbo passthrough (
-- --shard=k/n) folds into the hash of every task in the run, so the sliced test leg's dependencybuildtasks were re-hashed per shard and never hit cache. The fix separates the build (passthrough-free, cacheable) from the sharded test leg (--only), rather than raising the timeout — the timeout stayed at 45 as required.Open verification residue, named rather than claimed: this PR's own CI cannot exercise the change, because its affected set does not reach
@objectstack/cli. The cache-hit claim is confirmed only when a later run's affected set does reach it. I'll read the slice leg'sCached: N cached, M totalon the first such run and post it here.Generated by Claude Code
claude commentedon Sep 8, 2026
The owed reading, part 1 — the fix is LIVE on
mainand behaves as designedRun 34239039069 (
CI,pushonmain, created 2026-09-08T14:33:37Z — the run produced by #16880's merge).@objectstack/clidepends on@objectstack/lint(read frompackages/cli/package.json), so that PR'spackages/lint/src/**diff put the CLI in the affected set. ⇒ this is the first run to exercise #16868's change.⭐ The correction first: the sliced legs are not separate jobs.
ci.yml:629loopsfor LEG in __whole__ $SLICESinside eachTest Core (k/6)job. My earlier note said to look for a slice job; that was wrong, andDogfood Verify CLIis not it. The reading below comes from the jobs' step records.Measured — the new step, across all six shards
Build the sliced package's dependency closureRun this shard's tests⇒ The step exists, ran on 6 of 6 shards, and discriminates: it does real work (366 s / 371 s) on exactly the two shards that carry a sliced package, and short-circuits to 0 s on the four that do not — the
if [ ! -s "$RUNNER_TEMP/shard-packages.txt" ]guard at:516. ⭐ That 371 s / 0 s split is the firing control: a step that ran 0 s everywhere would be indistinguishable from a step that is wired but dead.Corroborating, independent of the durations: shard 6 uploaded 2 files into
test-core-run-summary-6-of-6while shard 1 uploaded 1 — two turbo run summaries on 6, one on 1. ⇒ two legs ran on the sliced shard, one on the unsliced.⛔ What this does NOT yet show, stated plainly
Cached: N cached, M totalis still unread. The whole-leg figure I did read (shard 1) isTasks: 48 successful, 48 total/Cached: 1 cached, 48 total— that is the__whole__leg, ⛔ not the slice leg, so it does not answer the question.productionresultssa1.blob.core.windows.net, which the egress proxy refuses (CONNECT tunnel failed, response 403). Job logs are reachable only through the MCP tail. That is why the figure above is a step-record reading and not a summary-file reading.Still owed on this card
The slice leg's
Cached: N cached, M totalon the next run whose sliced package set covers the closure this run just built. I will read it there and post it here. ⛔ Until then the quantitative half of #16395's claim is unproven, and I am not calling it proven.Generated by Claude Code
ci(dogfood): the sharded gate builds its closure in a passthrough-fre…