Skip to content

Commit d0f1c5b

Browse files
ci(refresh lane): each refresh diffs its window against the previous dataset, density-guarded, and lists test files over 10 s; the lane's seat files the cards (#23066)
Fixes #23059 Clause-②: no Items 2 and 3 of ruling E on #22949 (the maintainer's word 「22949 同意」, comment 6125775640): regression detection moves from the wall ceiling to the refresh lane. Item 1, both timing steps report-only, is #23058, in flight; this PR touches none of its files. ## What changed 1. **`scripts/measure-test-shard-timings.mjs`** - **The density tag, recorded from this refresh on.** The generator keeps every executed reading per run and writes a new `density` field beside the weights: `runs` (each run's count of EXECUTED packages; cache replays, failed suites and incomplete slice sets do not count, and a run that executed nothing carries no tag) and `shares` (each measured package's median, over the runs that executed it, of its seconds over that run's executed total). No reader of the dataset changes: the partitioner, the drift step, the ceiling check and the timing table read `packages` / `ceilings` only. - **`--diff`**: the refresh PR's regression section, from the previous dataset, the refreshed one and the window's per-shard captures. Rises of 1.25x or more with both numbers, the share before and after, the slowest file and a verdict; downs in one line; every test file over 10 s with its package and slowest run; the packages with no per-file reading and why; and a "What the seat landing this PR does" section with each finding's `Protects:` line ready to paste. Exit 0 on every reading; non-zero only on a refusal (an unknown flag, a malformed generated `density`). - Three constants, two of them the ceiling's own factor: `RISE_FACTOR` = `DENSITY_COMPARABLE_FACTOR` = `CEILING_HEADROOM` (1.25); `FILE_BUDGET_SECONDS` = 10. 2. **`scripts/ci/select-shard-timings-run.mjs`**: `runIsEligible` also returns each run's `test-core-timings-N-of-6` capture ids (`timing_artifact_ids` on `--candidates`). Not an eligibility input: the capture step in `ci.yml` is report-only, so a run is a complete measurement without one. 3. **`.github/workflows/shard-timings-refresh.yml`**: a header section, REGRESSION DETECTION LIVES HERE, AND WHO FILES WHAT; a new step that downloads the window's per-file captures for the body and cannot fail the job; the compose step runs `--diff` against `git show HEAD:scripts/test-shard-timings.json` and puts the section at the top of the body (a refusal's text goes in its place and the PR still opens); and the over-ceiling sentence no longer says "the brake fired on main" (see Deviations). Not touched: the ceiling formula, `CEILING_FLOOR_SECONDS`, `RULED_CEILING_RAISES`, the two timing steps, `ci.yml`, `lint.yml`, the workflow's triggers and its write path. ## The rules, as numbers - **Rise:** this window's weight is at least the previous weight x 1.25. Compared only for a package both datasets weigh and this window EXECUTED; a carried weight is the previous number moved forward by a cache HIT, so it is counted, never compared. - **Comparable density:** two windows are comparable when their mean count of executed packages per run is within 1.25x of each other: 20 and 25 are comparable, 20 and 25.1 are not. The self-test pins both sides of that line. It is the mean and not the median because hourly runs are bimodal (a run after `spec` lands executes nearly everything, the rest replay most of it): the median of a bimodal set jumps from one mode to the other, while the mean moves with the mix, which is what the contention behind a weight follows. - **Density guard (as ruled):** a rise whose share of its runs' executed total moved by less than 1.25x is `density only`, not a finding. - **Verdicts:** `finding` (a card) · `density only` · `provisional` (either weight rests on fewer than 3 executed runs) · `no prior share` (the previous window replayed it) · `baseline` (no comparable density reading on both sides). - **The first diff after this lands** meets a previous dataset with no `density`. Its body says the diff is a BASELINE: the rises are listed as readings, none is filed, and its window is the reference the next comparable window is judged against. The runs it takes are the runs that refresh accumulates (newest first, until every package is executed 3 times), each tagged in the body, and they are taken because they are the first runs a density can be read from at all. - **Files:** strictly over 10 s. Replayed rows are left out, because a cache HIT replays a duration measured outside the window. ## Premises, checked against origin/main (0cbfb7c) - **H1 holds.** The workflow checks out `main` and already reads the previous dataset with `git show HEAD:` (the bins step has copied the refreshed file over the working copy by then). The diff reads it the same way, with no extra fetch. - **H2: the dataset kept no per-run density.** Its keys are `carriedOver, ceilings, note, packages, provenance, provisional, secondsPerTestFileFallback, skippedAsCached, skippedIncompleteSlices, uncapped`; `provenance.runs` names the runs and nothing records a package per run. So the generator records it from this refresh on, as H2 anticipated, and the first diff can tag only its own window's runs. - **The per-file reading is NOT in the dataset**, although the card and ruling E item 3 say it is. `Slowest file:` is printed by `check-test-suite-ceilings.mjs` from the per-shard captures `report-test-timings.mjs --capture` uploads as `test-core-timings-N-of-6`. Those come from the same runs the refresh reads and are retained exactly as long as the run summaries (measured on run 38195222269: all six captures carry the same `expires_at` day as its six run summaries). So the per-file list is still a read of an existing measurement, from those captures. Packages without a per-file reading are named in the body with the reason: carried (every run of the window replayed it) or executed with no vitest file line captured. A run whose captures did not arrive is named too. - **H3 holds**, with one change: the replay over the last two datasets cannot show density or files (below). - **H4 holds.** The body is the compose step's `echo` block; the section is one `cat` of the diff's output. Lists that grow with the workspace are capped: non-finding rises at 25 rows, files at 100, each with a count and the full list printed to the job log. Findings are never capped. ## Replay: what the diff says over the last two datasets on main `--diff --prior` the #22368 dataset (0401847, measured 2026-10-08, 21 runs) `--refreshed` the #22923 dataset (1bd612b, measured 2026-10-12, 12 runs, the first with ceilings). Stderr: `12 rise(s) of 1.25x or more, 0 finding(s) (BASELINE: none judged); 33 down; files over budget NOT MEASURED`. | package | previous s | this window s | rise | verdict | |---|--:|--:|--:|---| | `@objectstack/trigger-api` | 2.18 | 23.85 | 10.94× | baseline | | `@objectstack/example-todo` | 43.17 | 66.07 | 1.53× | baseline | | `@objectstack/example-showcase` | 93.95 | 142.11 | 1.51× | baseline | | `@objectstack/service-realtime` | 10.39 | 14.72 | 1.42× | baseline | | `@objectstack/service-cluster-redis` | 15.30 | 20.59 | 1.35× | baseline | | `@objectstack/plugin-auth` | 359.89 | 465.63 | 1.29× | baseline | | `@objectstack/organizations` | 79.67 | 102.91 | 1.29× | baseline | | `@objectstack/metadata-core` | 18.92 | 24.32 | 1.29× | baseline | | `@objectstack/plugin-dev` | 49.47 | 63.57 | 1.29× | baseline | | `@objectstack/plugin-pinyin-search` | 32.56 | 41.80 | 1.28× | baseline | | `@objectstack/connector-slack` | 33.72 | 42.85 | 1.27× | baseline | | `@objectstack/service-settings` | 73.71 | 92.33 | 1.25× | baseline | Down: 33 lighter, 12 of them to 80 % or less of the previous weight (`cloud-connection` 153.64 to 81.68 s, `service-analytics` 401.06 to 220.50 s, `formula`, `client`, `connector-rest`, `sdui-parser`, `service-cluster`, `driver-sqlite-wasm`, `plugin-hono-server`, `example-crm`, +2). Not compared: 0 carried, 0 new, 0 gone. Read it as: neither dataset carries a `density` reading, so the replay is a BASELINE by construction and none of the 12 rises would have been a card. Twelve packages up and twelve down by the same factor across a 4-day gap is consistent with the density problem the ruling diagnosed (`cloud-connection`, `service-analytics` and `client`, three of the four packages the hourly reds named, are among the downs), and without a tag there is no telling load from code in either list. The first live diff after this lands differs in two ways: its refreshed dataset carries `density`, so the reason reads "the previous dataset carries no density reading" (the self-test pins that text), and its files list is read from that window's captures. Density and files for these two windows: NOT MEASURED. The #22368 window's artifacts are past their 1-day retention; #22923's captures are still retained, but this container cannot download an artifact (measured: `GET /actions/artifacts/11723746884/zip` redirects to the blob host, which answers 403, as the workflow header records). ## Evidence, on head 72443a3 - `node scripts/measure-test-shard-timings.mjs --self-test` → `self-test OK`, exit 0. New battery `refresh diff and per-file budget (#22949 E)`, floor 22 = cases registered (a floor of 23 reds, measured); roster floor 4 → 5. The two acceptance fixtures come first: a density-only rise (a fifth package executes, 4 to 5 per run, every suite 1.3x slower, `a` up 1.3x with its share down to 0.81x → `density only`) and a code rise (same density, `a` alone 2.5x, share 2.17x → `finding`). - `node scripts/ci/select-shard-timings-run.mjs --self-test` → `self-test OK`, exit 0. +2 cases in its first battery (floor 35 → 37 = registered) and a new battery `shard-timings-refresh per-file captures (#22949 E)` (floor 5; roster floor 5 → 6). It drives the new download step's own `run:` block under `bash -e` with stub `curl`/`unzip` (one capture fails to download, one run retained none) and then the real `--diff` over what landed. - **Ablations, one-time, through `scripts/ablation-replace.mjs`, each restored to `git diff HEAD` empty:** - Density guard removed (`else verdict = finding`): generator self-test exit 1, `diff: a rise explained by density alone was read as finding (findings 4)`. - The download step made to `exit 7` on a failed capture: selector self-test exit 1, `captures step: a missing capture failed the step (exit 7)`. The first attempt at this one was a NO-OP, refused by the tool (its replacement contained its anchor, so the anchor count did not move); rerun with a disjoint anchor. - `actionlint` 1.7.7 on the workflow: exit 0. Control leg: a copy with a step `if:` reading an undefined step id is reported (`property "nosuch" is not defined`). No `shellcheck` on this box, so the step scripts were not shellchecked: NOT MEASURED. - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` (no paths; derived from the merge base): 51 commands, all run on 72443a3, all exit 0; `--ran` reconciliation: 51 derived, 51 run, 0 unrun. `check:declared-population-live` was red on an earlier head (the capture schema constant was a quoted, path-shaped module-scope literal that reached no tracked file) and is green after the fix described under Deviations. - **NOT MEASURED: a live refresh.** This lane writes to `main` on its own schedule; the first scheduled refresh after merge is the measurement. The closest live reading is this PR's own `Shard Timings Refresh` check: on `pull_request` it runs the real selection, regeneration, capture download and `--diff` against main's dataset, and renders the would-be body (BASELINE, with real density tags and real files over budget) into its step summary. A job summary has no REST endpoint, so that reading is in the web UI, and the diff's one-line verdict and the captures read are in the job log. ## Deviations - **The selector is touched** for the capture ids. The claim named it "only if the window selection must carry the density tag"; the density tag needed nothing from it (the generator reads it from the run summaries it already gets), but the per-file read needs the capture ids the selector already sees and dropped. - **A new workflow step** ("Download the window's per-file timing captures for the pull request body"). The claim scoped the workflow to the PR body and the instruction lines; this step exists only to feed the body, because the premise that the dataset carries per-file readings did not hold. - **The stale "the brake fired on main" sentence** in the compose step's ceiling line is rewritten as a reading (relayed by the director seat from the #23058 dev): "a reading, not a red -- under #22949 ruling E the ceiling reports on every event, and a regression is judged by the diff at the top of this body". The generator's matching code comment is reworded the same way. - **`scripts/measure-test-shard-timings.mjs` now carries a `dispatch-gates: no-path-population` marker**, and the capture schema is spelled `['test-timing-capture', 1].join('/')`, with a comment saying why, which is the remedy `check:declared-population-live` itself prescribes for a gate that reads no tracked path. The family is still placed by its own file (re-derived: `matched via scripts/measure-test-shard-timings.mjs`). ## Acceptance notes - The workflow header's older paragraph still says `Test Core` "grades every executed package" against the ceilings. Under ruling E it reports; I left it because it is not on this card's lines and it describes #23058's step. Carrier: none. - Interpretation, open to the seat: files over budget are filed as one card per owning package naming its files, not one card per file, and a package or file with an open card gets the new numbers there as a comment instead of a second card. #22998 says "a file over 10 s of wall is a card for the owning lane"; one card per file would be dozens of cards a week on the first lists. - The density reading counts packages, not seconds per shard. A run's contention also depends on how its packages land on the six shards; ruling E names the executed-package count as the tag, and the share guard absorbs a whole run slowing down whatever the cause. ## 维护者速读(草稿) **改了什么** 每周的测试耗时刷新 PR(机器人开的那张)从此在正文最上面给出三张清单:比上一份数据集慢了 25% 以上的包(两个数字、最慢的测试文件、判定)、本窗口每次运行实际执行了多少个包(密度)、以及单个耗时超过 10 秒的测试文件。数据集里新增一个 `density` 字段记录密度。工作流头注释写明:机器人不立卡,落地这张刷新 PR 的席位按清单立卡,永远不让构建变红。 **为什么改** 裁定 E 已把墙钟上限改成只报告,回归检测需要一个新的落点。同一个包在空闲小时和繁忙小时的耗时能差 2.5 倍,所以只比数字会把"整台机器更忙"误判成"代码变慢";密度标签和"占比不变就不算"这道闸门就是用来区分这两者的。 **风险与代价(含回滚)** 不改任何门禁、公式、触发条件和写路径,不会让任何构建变红。代价是每次刷新多下载每次运行的 6 个小文件(各约 15 KB),下载失败只会在正文里点名,不会丢掉这次刷新。合并后的第一次刷新只能当基线(上一份数据集没有密度),从第二次可比较的刷新起才会出现需要立卡的条目。回滚:还原这一个 PR 即可;数据集里多出的 `density` 字段没有任何读者,留着无害。 **席位意见** **你要做的** 批准这张 PR(工作流改动按 Tier H 处理,保持草稿)。合并后看第一次周一刷新 PR 的正文是否带着三张清单和"这是基线"的说明。 --- _Generated by [Claude Code](https://claude.ai/code/session_01LqNgdPZB2Jnr5HpGbaqLeB)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 56986e9 commit d0f1c5b

3 files changed

Lines changed: 1360 additions & 14 deletions

File tree

‎.github/workflows/shard-timings-refresh.yml‎

Lines changed: 131 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,49 @@
154154
# and its output — pass or fail, verbatim — goes into the PR body. The PR is
155155
# opened either way, because a refusal to open it would lose the measurement,
156156
# which is the exact silent-rot failure this workflow exists to end.
157+
#
158+
# REGRESSION DETECTION LIVES HERE, AND WHO FILES WHAT (#22949 ruling E)
159+
# ---------------------------------------------------------------------
160+
# The suite-duration ceilings only REPORT (ruling E item 1): a package's wall on
161+
# a shared runner reads the run's load as much as the package. So this lane is
162+
# the regression detector, and every refresh PR's body opens with what
163+
# `measure-test-shard-timings.mjs --diff` reads (items 2 and 3):
164+
#
165+
# - RISES: every package whose weight is up 1.25x or more against the
166+
# previous dataset -- the one on `main` before this refresh, read with
167+
# `git show HEAD:` -- with both numbers, its share of its runs' executed
168+
# total before and after, its slowest file in the window, and a verdict.
169+
# Only `finding` is a card.
170+
# - DENSITY: each run's count of EXECUTED packages (cache replays excluded),
171+
# which the generator records in the dataset's `density` from the first
172+
# refresh after the ruling on. A rise is judged only against a window of
173+
# COMPARABLE density -- mean executed packages per run within 1.25x of
174+
# each other, a number its self-test pins -- and a rise whose share of its
175+
# runs' executed total moved by less than 1.25x is density alone, not a
176+
# finding. The first refresh after this landed meets a previous dataset
177+
# with no `density`, and its body says the diff is a BASELINE.
178+
# - DOWNS, in one line.
179+
# - FILES OVER BUDGET: every test file over 10 s of wall in the window, with
180+
# its package and its slowest run (#22998). The dataset holds no per-file
181+
# reading; the window's runs do, in the per-shard `test-core-timings-N-of-6`
182+
# captures, which a step before the body downloads: the same runs, retained
183+
# exactly as long as their run summaries.
184+
#
185+
# ⛔ THE BOT FILES NOTHING, AND NONE OF THIS IS EVER A RED BUILD. The seat that
186+
# lands the refresh PR files, after it lands:
187+
#
188+
# - one card per package the diff marks `finding`, for the owning lane (its
189+
# `domain:*`, by the package's path in the pm-dispatch domain lane table),
190+
# with a `Protects:` line naming the package and the numbers -- the body
191+
# prints each line ready to paste;
192+
# - for the files over budget, one card per owning package naming its files
193+
# (#22998: a test file over 10 s of wall is a card for the owning lane);
194+
# - nothing for a rise marked density only, provisional, no prior share or
195+
# baseline, and no second card where an open one already names the package
196+
# or the file: the new numbers go there as a comment.
197+
#
198+
# The body's own "What the seat landing this PR does" section says the same
199+
# with this run's list in it, so the duty is read where the work is.
157200

158201
name: Shard Timings Refresh
159202

@@ -590,6 +633,57 @@ jobs:
590633
echo "BEFORE: $(cat "$RUNNER_TEMP/bins-before.txt")"
591634
echo "AFTER: $(cat "$RUNNER_TEMP/bins-after.txt")"
592635
636+
# The per-file half of the regression readings (#22949 ruling E, item 3).
637+
# The dataset holds no per-file durations; the window's runs do, in the
638+
# per-shard captures `report-test-timings.mjs --capture` uploads as
639+
# `test-core-timings-N-of-6`, retained 1 day like the run summaries the
640+
# generator read. The selector listed their ids beside the summaries'
641+
# (`timing_artifact_ids`), so this is a read of the SAME runs, never a new
642+
# measurement, and it feeds the PR body alone.
643+
#
644+
# ⛔ It cannot fail the job: a capture that did not arrive is a warning
645+
# here and a named run in the PR body, never a lost refresh -- the capture
646+
# step in ci.yml is report-only, so a run can be a complete measurement
647+
# without one. Every fallible command is an `if` condition, which `bash -e`
648+
# does not abort on.
649+
- name: Download the window's per-file timing captures for the pull request body
650+
if: steps.compare.outputs.changed == 'true'
651+
env:
652+
GITHUB_TOKEN: ${{ github.token }}
653+
RUNS: ${{ steps.generate.outputs.runs }}
654+
run: |
655+
FILES="$RUNNER_TEMP/refresh-files"
656+
mkdir -p "$FILES"
657+
for R in $RUNS; do
658+
mkdir -p "$FILES/$R"
659+
# One "SHARD ARTIFACT_ID" line per retained capture of run R.
660+
if ! node -e '
661+
const rs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8"));
662+
const r = rs.find((x) => String(x.run_id) === process.argv[1]);
663+
for (const [n, id] of Object.entries((r && r.timing_artifact_ids) || {})) console.log(n + " " + id);
664+
' "$R" > "$FILES/$R.ids"; then
665+
echo "::warning::Could not read run $R's per-file capture ids; its files are not in the over-budget list."
666+
continue
667+
fi
668+
if [ ! -s "$FILES/$R.ids" ]; then
669+
echo "::warning::Run $R retained no per-file timing capture; its files are not in the over-budget list."
670+
continue
671+
fi
672+
while read -r N ART; do
673+
if curl -sSL --fail -H "Authorization: Bearer $GITHUB_TOKEN" \
674+
-H "Accept: application/vnd.github+json" \
675+
"https://api.github.com/repos/$GITHUB_REPOSITORY/actions/artifacts/$ART/zip" \
676+
-o "$FILES/$R/$N.zip" \
677+
&& unzip -qo "$FILES/$R/$N.zip" -d "$FILES/$R/$N"; then
678+
rm -f "$FILES/$R/$N.zip"
679+
else
680+
echo "::warning::Could not download per-file capture $ART (shard $N) of run $R; its files are not in the over-budget list."
681+
fi
682+
done < "$FILES/$R.ids"
683+
done
684+
echo "Per-file captures read for the PR body:"
685+
find "$FILES" -name '*.json' -type f | sort
686+
593687
# Composition is separated from the WRITE on purpose: a `pull_request` run
594688
# of this lane must exercise the body-building — the shard durations, the
595689
# bins, the conditional blocks — without pushing anything. The write step
@@ -684,13 +778,46 @@ jobs:
684778
lines.push(uncapped.length ? "No ceiling: " + uncapped.join(", ") + "." : "Every package carries a ceiling.");
685779
if (over > 0) {
686780
lines.push(
687-
"⚠️ " + over + " package(s) ran OVER their held ceiling in this window (marked in the table): the brake " +
688-
"fired on main. This refresh does not raise them; that takes a ruling."
781+
"⚠️ " + over + " package(s) ran OVER their held ceiling in this window (marked in the table): a reading, " +
782+
"not a red -- under #22949 ruling E the ceiling reports on every event, and a regression is judged by " +
783+
"the diff at the top of this body. This refresh does not raise them; that takes a ruling."
689784
);
690785
}
691786
console.log(lines.join("\n\n"));
692787
')
693788
789+
# The regression readings (#22949 ruling E): this window diffed against
790+
# the previous dataset -- read from git, like the ceiling line above,
791+
# because the bins step has already copied the refreshed file over the
792+
# working copy -- plus the per-file budget from the captures the step
793+
# above downloaded. A refusal here (a malformed generated field) must
794+
# not cost the measurement, so its text goes into the body in place of
795+
# the section and the PR still opens. A less-than character is spelled
796+
# out first: GitHub's body sanitizer can eat everything after one.
797+
git show HEAD:scripts/test-shard-timings.json > "$RUNNER_TEMP/refresh/previous.json"
798+
DIFF_EXIT=0
799+
node scripts/measure-test-shard-timings.mjs --diff \
800+
--prior "$RUNNER_TEMP/refresh/previous.json" \
801+
--refreshed "$RUNNER_TEMP/refresh/refreshed.json" \
802+
--captures "$RUNNER_TEMP/refresh-files" \
803+
--out "$RUNNER_TEMP/refresh/diff.md" 2> "$RUNNER_TEMP/refresh/diff-stderr.txt" || DIFF_EXIT=$?
804+
cat "$RUNNER_TEMP/refresh/diff-stderr.txt"
805+
if [ "$DIFF_EXIT" -ne 0 ]; then
806+
echo "::warning::The refresh diff refused (exit $DIFF_EXIT); the PR body carries its text in place of the regression readings."
807+
{
808+
echo "## Regression readings: NOT MEASURED -- the diff refused (exit $DIFF_EXIT)"
809+
echo
810+
echo '```'
811+
sed 's/</\&lt;/g' "$RUNNER_TEMP/refresh/diff-stderr.txt"
812+
echo '```'
813+
echo
814+
echo "## What the seat landing this PR does"
815+
echo
816+
echo "File nothing from this refresh's readings: no rise and no file was judged. The refusal above"
817+
echo "is a defect in this lane, and that is the one card to file."
818+
} > "$RUNNER_TEMP/refresh/diff.md"
819+
fi
820+
694821
SHARDS=$(node -e '
695822
const rs = JSON.parse(require("fs").readFileSync(process.env.RUNNER_TEMP + "/candidates.json", "utf8"));
696823
const r = rs.find((x) => String(x.run_id) === process.env.RUN_ID);
@@ -704,6 +831,8 @@ jobs:
704831
echo "Every byte came out of \`scripts/measure-test-shard-timings.mjs\`; nothing here was"
705832
echo "hand-edited, and no bound, timeout or matrix entry was touched."
706833
echo
834+
cat "$RUNNER_TEMP/refresh/diff.md"
835+
echo
707836
echo "## Source"
708837
echo
709838
echo "Measured across $RUN_COUNT accumulated run(s) of the HOURLY \`schedule\` run of CI on"

0 commit comments

Comments
 (0)