Repository navigation
ci(nightly-tiers): the nightly states its own slicing, one CLI slice per shard - #23444
Merged
objectstack-fleet[bot] merged 2 commits intoOct 14, 2026
Merged
Conversation
…per shard The nightly tiers inherited Test Core's FILE_SHARDED_PACKAGES. 9ff7428 retired the CLI's entry and 5919483 re-sliced it 3 ways, which cannot spread over the nightly's two shards, so since 2026-10-02 the second shard ran nothing and the first carried the whole e2e tier until it was killed at its 45-minute limit. `nightly-tiers.mjs --packages` now names every tier-owning package in a `sliceAcrossShards` key, and the partitioner cuts each named package into as many slices as the run's own `--shard k/N` has shards; a document without the key (every Test Core document) slices by FILE_SHARDED_PACKAGES exactly as before. main()'s planning moves into planRun(), and a new self-test battery drives the nightly's live document through it at the shard count test-nightly-tiers.yml declares. Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY Co-authored-by: Claude <noreply@anthropic.com>
…eader Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY Co-authored-by: Claude <noreply@anthropic.com>
objectstack-fleet
Bot
deleted the
claude/issue-23407-nightly-tiers-shard-balance
branch
October 14, 2026 09:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #23407
Clause-②: no
What was wrong (measured)
The nightly tiers shard the CLI's e2e tier with Test Core's partitioner, and until now with Test Core's slice map,
FILE_SHARDED_PACKAGES. That map has changed twice for Test Core's own reasons, and each change took the nightly's second shard away without any step going red:FILE_SHARDED_PACKAGES--shard 1/2--shard 2/2fd96a847(9ff74285^)@objectstack/cli 1/2@objectstack/cli 2/29ff74285(2026-10-02, retired the entry)@objectstack/cli3ca71b6e(59194834^)@objectstack/cli59194834(2026-10-09, sliced 3 ways)@objectstack/clie4ec6ec1a5(run 38517756588) and4f81f31526(base)@objectstack/cliEach row comes from
node scripts/nightly-tiers.mjs --packagesand thennode scripts/partition-test-shards.mjs DOC --shard k/2, run against that tree's own scripts.The nightly runs agree with that history. This is the "Run this shard's nightly tiers" step's duration, read from the Actions jobs API:
So the workflow header's premise (FILE_SHARDED_PACKAGES gives the CLI two slices, one per shard) stopped holding at
9ff74285, a week before59194834.59194834did not bring the second slice back, because 3 slices cannot spread over 2 shards.The fix
The nightly now states its own slicing. Test Core's map no longer reaches it.
scripts/nightly-tiers.mjs --packageswrites asliceAcrossShardskey naming every tier-owning package. Today that is["@objectstack/cli"].scripts/partition-test-shards.mjscuts each named package into as many slices as the run has shards (the N of the run's own--shard k/N).FILE_SHARDED_PACKAGESdoes not apply to that run.planShards()(NEEDED / SPREADS / WITHIN CAP) and the vitest file floor still apply.turbo lsnever writes the key, so every Test Core document reads it as absent and slices exactly as before (proof below). When the key is present it is checked strictly: it must be an array of distinct names, and each name must be an item of the same document.main()moved into one exported function,planRun(), in the same order.main()prints its result, and the self-test drives that same function.--shard k/N, changing the nightly matrix later re-slices it with no planner edit.New output on the base tree's inputs:
--shard 1/2gives@objectstack/cli 1/2and--shard 2/2gives@objectstack/cli 2/2, at 1102.9s predicted each. A third log line readsslicing map: the document's sliceAcrossShards -- @objectstack/cli cut 2 ways, one slice per shard of this run; FILE_SHARDED_PACKAGES (Test Core's map) does not apply to this run. At--shard k/3the output is@objectstack/cli 1/3,2/3and3/3.Expected wall time per shard: the whole tier ran 2361-2615s on one shard, and the 2-slice runs on 10-01 ran 1196s / 1168s. That points to roughly 1200-1310s of tests per shard (about 20-22 min), plus about 1 min of setup, against the 45-minute limit.
Pin: a planner change that empties a shard again goes red
partition-test-shards.mjs --self-testhas a new battery, "a document states its own slicing: the nightly tiers", with 16 cases. lint.yml runs this self-test on every pull request. The battery covers:planRun().test-nightly-tiers.yml's matrix size equals the N in its--shardargument, and the shards are partitioned fromnightly-tiers.mjs --packages.planRun()at the workflow's shard count: the key names every tier package; no shard is empty; the CLI lands as one distinct slice on each shard.Red shown once, then restored. Both mutations were applied with
scripts/ablation-replace.mjson the committed fix (e11d8343); each restore was proven by blob hash equal to HEAD and an emptygit diff HEAD:const sliced = acrossShards === null ? FILE_SHARDED_PACKAGES : sliceMapForRun(acrossShards, shardCount);was replaced withconst sliced = FILE_SHARDED_PACKAGES;(anchor 1 to 0, blob6bc0cdf7948btoe08f8b5ec718). The self-test exited 1 with:nightly: the live document at the workflow's 2 shards leaves shard(s) 2 with nothing to run, while the split is [@objectstack/cli | ] (@objectstack/cli: whole (... but 3 slices cannot spread across this run's shards)). That is the production failure. Restored to6bc0cdf7948b.packageListDocument()(blobd75822731830to232c14a0299f). The self-test exited 1 withnightly document: sliceAcrossShards is undefined for the tier-owning packages ["@objectstack/cli"] .... Restored tod75822731830.Test Core's partition is byte-identical
Test Core's partition did not change. Check: same command on both trees, base
4f81f315(origin/main) and head448c8745, andcmpon stdout, stderr and exit code. The command isnode scripts/partition-test-shards.mjs DOC --shard k/6 --exclude @objectstack/dogfood, for k = 1..6, over five documents:pnpm turbo ls --output=jsonlist (83 packages);Result: 90 of 90 files identical. The concatenated sha256 is
885ce341b5b2d363...on both trees, and all 30 exits are 0. The full list still puts@objectstack/cli 1/3,2/3and3/3on shards 2, 3 and 4.Gates (all on head
448c8745)node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstackderived 32 commands from the 2-path change set. All 32 ran with exit 0, includingpnpm check:pm-dispatch-gates(fast tier, 1841 cases).--ranreconciliation: 32 derived, 32 run, 0 NOT-MEASURED, 0 UNRUN.nightly-tiers.mjs --self-test(47 cases) andpartition-test-shards.mjs --self-test, both exit 0. The partitioner's module graph gained an import, so the self-tests of the scripts that import it also ran:check-test-completeness,check-test-suite-ceilings,measure-test-shard-timingsandcheck-tier-file-adoption, all exit 0.npx eslint --no-inline-config --format jsonread 2 files with 0 errors and 0 warnings. Both files are in eslint's population (isPathIgnoredis false for each;eslint.config.mjscovers**/*.{...,mjs,...}). Type-aware linting is not enabled (noproject/projectServiceineslint.config.mjs), so this diff cannot change the verdict on any untouched file. Fullpnpm lintis left to CI.skip-changeset: this diff publishes nothing. The rootpackage.jsonis private,scripts/is not shipped, and 0 publicfiles[]entries name these scripts. As a positive control, 72 of 72 public manifests have adistentry.pull_requestpaths filter namesscripts/nightly-tiers.mjs). That run on real runners is the first two-shard reading. It writes nothing to the board.Acceptance notes
vitest list --filesOnly --shard=k/2ignores the shard: both slices listed all 82 tier files. The evidence that the--shard=k/npassthrough partitions the tier is the 2026-10-01 run (1196s / 1168s). This PR's own nightly run is the next reading.timeout-minutescomment ("about 13 min a shard"). The header says the shard count is spelled in three places; it is four, because the artifact namenightly-tiers-reports-...-of-2is the fourth. Correcting these is a governed (Tier H) workflow edit, so it is left out of this scripts-only PR. Carrier: the next PR that touchestest-nightly-tiers.yml.reportjob downloads withpattern: nightly-tiers-reports-*, andcheck-test-completeness.mjs --scheduledreads only its own shard's list. Neither depends on the count. The new self-test reads the matrix, so a mismatch between matrix and--shardnow goes red.Generated by Claude Code