Skip to content

ci(nightly-tiers): the nightly states its own slicing, one CLI slice per shard - #23444

Merged
objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/issue-23407-nightly-tiers-shard-balance
Oct 14, 2026
Merged

objectstack-fleet[bot] merged 2 commits into
mainfrom
claude/issue-23407-nightly-tiers-shard-balance

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #23407
Clause-②: no

What was wrong (measured)

The nightly tiers shard the CLI's e2e tier with Test Core's partitioner, and until now with Test Core's slice map, FILE_SHARDED_PACKAGES. That map has changed twice for Test Core's own reasons, and each change took the nightly's second shard away without any step going red:

tree FILE_SHARDED_PACKAGES --shard 1/2 --shard 2/2
fd96a847 (9ff74285^) cli: 2 @objectstack/cli 1/2 @objectstack/cli 2/2
9ff74285 (2026-10-02, retired the entry) empty @objectstack/cli (nothing)
3ca71b6e (59194834^) empty @objectstack/cli (nothing)
59194834 (2026-10-09, sliced 3 ways) cli: 3 @objectstack/cli (nothing): "3 slices cannot spread across this run's shards"
e4ec6ec1a5 (run 38517756588) and 4f81f31526 (base) cli: 3 @objectstack/cli (nothing): same refusal

Each row comes from node scripts/nightly-tiers.mjs --packages and then node scripts/partition-test-shards.mjs DOC --shard k/2, run against that tree's own scripts.

The nightly runs agree with that history. This is the "Run this shard's nightly tiers" step's duration, read from the Actions jobs API:

  • 2026-09-30 and 2026-10-01: both shards non-empty, 1183s / 1190s and 1196s / 1168s.
  • 2026-10-04 onward: shard 2 ran 0s every night. On seven of eight green nights shard 1 took 2361-2615s; 2026-10-07 read 1589s.
  • 2026-10-14 (run 38517756588): shard 1 was cancelled at the 45-minute limit. Shard 2's log reads "No packages on this shard — nothing to test."

So the workflow header's premise (FILE_SHARDED_PACKAGES gives the CLI two slices, one per shard) stopped holding at 9ff74285, a week before 59194834. 59194834 did not bring the second slice back, because 3 slices cannot spread over 2 shards.

The fix

The nightly now states its own slicing. Test Core's map no longer reaches it.

  • scripts/nightly-tiers.mjs --packages writes a sliceAcrossShards key naming every tier-owning package. Today that is ["@objectstack/cli"].
  • When a document carries that key, scripts/partition-test-shards.mjs cuts each named package into as many slices as the run has shards (the N of the run's own --shard k/N). FILE_SHARDED_PACKAGES does not apply to that run. planShards() (NEEDED / SPREADS / WITHIN CAP) and the vitest file floor still apply.
  • turbo ls never writes the key, so every Test Core document reads it as absent and slices exactly as before (proof below). When the key is present it is checked strictly: it must be an array of distinct names, and each name must be an item of the same document.
  • The planning in main() moved into one exported function, planRun(), in the same order. main() prints its result, and the self-test drives that same function.
  • No workflow change. The nightly keeps 2 shards. Because the count comes from the run's own --shard k/N, changing the nightly matrix later re-slices it with no planner edit.

New output on the base tree's inputs: --shard 1/2 gives @objectstack/cli 1/2 and --shard 2/2 gives @objectstack/cli 2/2, at 1102.9s predicted each. A third log line reads slicing map: the document's sliceAcrossShards -- @objectstack/cli cut 2 ways, one slice per shard of this run; FILE_SHARDED_PACKAGES (Test Core's map) does not apply to this run. At --shard k/3 the output is @objectstack/cli 1/3, 2/3 and 3/3.

Expected wall time per shard: the whole tier ran 2361-2615s on one shard, and the 2-slice runs on 10-01 ran 1196s / 1168s. That points to roughly 1200-1310s of tests per shard (about 20-22 min), plus about 1 min of setup, against the 45-minute limit.

Pin: a planner change that empties a shard again goes red

partition-test-shards.mjs --self-test has a new battery, "a document states its own slicing: the nightly tiers", with 16 cases. lint.yml runs this self-test on every pull request. The battery covers:

  • An absent key slices by FILE_SHARDED_PACKAGES through planRun().
  • Five malformed shapes are refused by name.
  • The run map is N ways, and a one-shard run cuts nothing.
  • One package at 2, 3 and 4 shards lands one distinct slice per shard.
  • A control case: 3 configured slices over 2 shards leaves one shard empty, so the predicate can go red.
  • test-nightly-tiers.yml's matrix size equals the N in its --shard argument, and the shards are partitioned from nightly-tiers.mjs --packages.
  • Three live pins drive the nightly's own document from the live tree through planRun() at the workflow's shard count: the key names every tier package; no shard is empty; the CLI lands as one distinct slice on each shard.

Red shown once, then restored. Both mutations were applied with scripts/ablation-replace.mjs on the committed fix (e11d8343); each restore was proven by blob hash equal to HEAD and an empty git diff HEAD:

  1. Planner-side. const sliced = acrossShards === null ? FILE_SHARDED_PACKAGES : sliceMapForRun(acrossShards, shardCount); was replaced with const sliced = FILE_SHARDED_PACKAGES; (anchor 1 to 0, blob 6bc0cdf7948b to e08f8b5ec718). The self-test exited 1 with: nightly: the live document at the workflow's 2 shards leaves shard(s) 2 with nothing to run, while the split is [@objectstack/cli | ] (@objectstack/cli: whole (... but 3 slices cannot spread across this run's shards)). That is the production failure. Restored to 6bc0cdf7948b.
  2. Producer-side. The key was dropped from packageListDocument() (blob d75822731830 to 232c14a0299f). The self-test exited 1 with nightly document: sliceAcrossShards is undefined for the tier-owning packages ["@objectstack/cli"] .... Restored to d75822731830.

Test Core's partition is byte-identical

Test Core's partition did not change. Check: same command on both trees, base 4f81f315 (origin/main) and head 448c8745, and cmp on stdout, stderr and exit code. The command is node scripts/partition-test-shards.mjs DOC --shard k/6 --exclude @objectstack/dogfood, for k = 1..6, over five documents:

  • the full pnpm turbo ls --output=json list (83 packages);
  • an affected subset with the CLI;
  • an affected subset without it;
  • the CLI alone;
  • an empty list.

Result: 90 of 90 files identical. The concatenated sha256 is 885ce341b5b2d363... on both trees, and all 30 exits are 0. The full list still puts @objectstack/cli 1/3, 2/3 and 3/3 on shards 2, 3 and 4.

Gates (all on head 448c8745)

  • node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 32 commands from the 2-path change set. All 32 ran with exit 0, including pnpm check:pm-dispatch-gates (fast tier, 1841 cases). --ran reconciliation: 32 derived, 32 run, 0 NOT-MEASURED, 0 UNRUN.
  • The edited scripts' own suites: nightly-tiers.mjs --self-test (47 cases) and partition-test-shards.mjs --self-test, both exit 0. The partitioner's module graph gained an import, so the self-tests of the scripts that import it also ran: check-test-completeness, check-test-suite-ceilings, measure-test-shard-timings and check-tier-file-adoption, all exit 0.
  • Lint, narrowed to the 2 changed files: npx eslint --no-inline-config --format json read 2 files with 0 errors and 0 warnings. Both files are in eslint's population (isPathIgnored is false for each; eslint.config.mjs covers **/*.{...,mjs,...}). Type-aware linting is not enabled (no project / projectService in eslint.config.mjs), so this diff cannot change the verdict on any untouched file. Full pnpm lint is left to CI.
  • skip-changeset: this diff publishes nothing. The root package.json is private, scripts/ is not shipped, and 0 public files[] entries name these scripts. As a positive control, 72 of 72 public manifests have a dist entry.
  • This PR triggers the nightly workflow itself (its pull_request paths filter names scripts/nightly-tiers.mjs). That run on real runners is the first two-shard reading. It writes nothing to the board.

Acceptance notes

  • NOT MEASURED locally: whether the two vitest slices are disjoint. vitest list --filesOnly --shard=k/2 ignores the shard: both slices listed all 82 tier files. The evidence that the --shard=k/n passthrough partitions the tier is the 2026-10-01 run (1196s / 1168s). This PR's own nightly run is the next reading.
  • The workflow header's "Sharding — TWO shards" paragraph is stale. So is the timeout-minutes comment ("about 13 min a shard"). The header says the shard count is spelled in three places; it is four, because the artifact name nightly-tiers-reports-...-of-2 is the fourth. Correcting these is a governed (Tier H) workflow edit, so it is left out of this scripts-only PR. Carrier: the next PR that touches test-nightly-tiers.yml.
  • Nothing outside the workflow reads the nightly's shard count. The report job downloads with pattern: nightly-tiers-reports-*, and check-test-completeness.mjs --scheduled reads only its own shard's list. Neither depends on the count. The new self-test reads the matrix, so a mismatch between matrix and --shard now goes red.
  • An empty nightly shard still exits 0 in the workflow ("No packages on this shard"). That green is how this stayed hidden for twelve nights. With this change the planner-side cause is pinned on every pull request.

Generated by Claude Code

…per shard

The nightly tiers inherited Test Core's FILE_SHARDED_PACKAGES. 9ff7428
retired the CLI's entry and 5919483 re-sliced it 3 ways, which cannot
spread over the nightly's two shards, so since 2026-10-02 the second shard
ran nothing and the first carried the whole e2e tier until it was killed
at its 45-minute limit.

`nightly-tiers.mjs --packages` now names every tier-owning package in a
`sliceAcrossShards` key, and the partitioner cuts each named package into
as many slices as the run's own `--shard k/N` has shards; a document
without the key (every Test Core document) slices by FILE_SHARDED_PACKAGES
exactly as before. main()'s planning moves into planRun(), and a new
self-test battery drives the nightly's live document through it at the
shard count test-nightly-tiers.yml declares.

Claude-Session: https://claude.ai/code/session_01B7mrh59UAGffd7s9asUfoY
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet objectstack-fleet Bot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Oct 14, 2026
@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 14, 2026 09:21
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 14, 2026 09:22
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 14, 2026
Merged via the queue into main with commit 2d2e3ce Oct 14, 2026
47 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-23407-nightly-tiers-shard-balance branch October 14, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nightly-tiers: red on main

2 participants