Repository navigation
Commit d0f1c5b
ci(refresh lane): each refresh diffs its window against the previous dataset, density-guarded, and lists test files over 10 s; the lane's seat files the cards (#23066)
Fixes #23059
Clause-②: no
Items 2 and 3 of ruling E on #22949 (the maintainer's word 「22949 同意」,
comment 6125775640): regression detection moves from the wall ceiling to
the refresh lane. Item 1, both timing steps report-only, is #23058, in
flight; this PR touches none of its files.
## What changed
1. **`scripts/measure-test-shard-timings.mjs`**
- **The density tag, recorded from this refresh on.** The generator
keeps every executed reading per run and writes a new `density` field
beside the weights: `runs` (each run's count of EXECUTED packages; cache
replays, failed suites and incomplete slice sets do not count, and a run
that executed nothing carries no tag) and `shares` (each measured
package's median, over the runs that executed it, of its seconds over
that run's executed total). No reader of the dataset changes: the
partitioner, the drift step, the ceiling check and the timing table read
`packages` / `ceilings` only.
- **`--diff`**: the refresh PR's regression section, from the previous
dataset, the refreshed one and the window's per-shard captures. Rises of
1.25x or more with both numbers, the share before and after, the slowest
file and a verdict; downs in one line; every test file over 10 s with
its package and slowest run; the packages with no per-file reading and
why; and a "What the seat landing this PR does" section with each
finding's `Protects:` line ready to paste. Exit 0 on every reading;
non-zero only on a refusal (an unknown flag, a malformed generated
`density`).
- Three constants, two of them the ceiling's own factor: `RISE_FACTOR` =
`DENSITY_COMPARABLE_FACTOR` = `CEILING_HEADROOM` (1.25);
`FILE_BUDGET_SECONDS` = 10.
2. **`scripts/ci/select-shard-timings-run.mjs`**: `runIsEligible` also
returns each run's `test-core-timings-N-of-6` capture ids
(`timing_artifact_ids` on `--candidates`). Not an eligibility input: the
capture step in `ci.yml` is report-only, so a run is a complete
measurement without one.
3. **`.github/workflows/shard-timings-refresh.yml`**: a header section,
REGRESSION DETECTION LIVES HERE, AND WHO FILES WHAT; a new step that
downloads the window's per-file captures for the body and cannot fail
the job; the compose step runs `--diff` against `git show
HEAD:scripts/test-shard-timings.json` and puts the section at the top of
the body (a refusal's text goes in its place and the PR still opens);
and the over-ceiling sentence no longer says "the brake fired on main"
(see Deviations).
Not touched: the ceiling formula, `CEILING_FLOOR_SECONDS`,
`RULED_CEILING_RAISES`, the two timing steps, `ci.yml`, `lint.yml`, the
workflow's triggers and its write path.
## The rules, as numbers
- **Rise:** this window's weight is at least the previous weight x 1.25.
Compared only for a package both datasets weigh and this window
EXECUTED; a carried weight is the previous number moved forward by a
cache HIT, so it is counted, never compared.
- **Comparable density:** two windows are comparable when their mean
count of executed packages per run is within 1.25x of each other: 20 and
25 are comparable, 20 and 25.1 are not. The self-test pins both sides of
that line. It is the mean and not the median because hourly runs are
bimodal (a run after `spec` lands executes nearly everything, the rest
replay most of it): the median of a bimodal set jumps from one mode to
the other, while the mean moves with the mix, which is what the
contention behind a weight follows.
- **Density guard (as ruled):** a rise whose share of its runs' executed
total moved by less than 1.25x is `density only`, not a finding.
- **Verdicts:** `finding` (a card) · `density only` · `provisional`
(either weight rests on fewer than 3 executed runs) · `no prior share`
(the previous window replayed it) · `baseline` (no comparable density
reading on both sides).
- **The first diff after this lands** meets a previous dataset with no
`density`. Its body says the diff is a BASELINE: the rises are listed as
readings, none is filed, and its window is the reference the next
comparable window is judged against. The runs it takes are the runs that
refresh accumulates (newest first, until every package is executed 3
times), each tagged in the body, and they are taken because they are the
first runs a density can be read from at all.
- **Files:** strictly over 10 s. Replayed rows are left out, because a
cache HIT replays a duration measured outside the window.
## Premises, checked against origin/main (0cbfb7c)
- **H1 holds.** The workflow checks out `main` and already reads the
previous dataset with `git show HEAD:` (the bins step has copied the
refreshed file over the working copy by then). The diff reads it the
same way, with no extra fetch.
- **H2: the dataset kept no per-run density.** Its keys are
`carriedOver, ceilings, note, packages, provenance, provisional,
secondsPerTestFileFallback, skippedAsCached, skippedIncompleteSlices,
uncapped`; `provenance.runs` names the runs and nothing records a
package per run. So the generator records it from this refresh on, as H2
anticipated, and the first diff can tag only its own window's runs.
- **The per-file reading is NOT in the dataset**, although the card and
ruling E item 3 say it is. `Slowest file:` is printed by
`check-test-suite-ceilings.mjs` from the per-shard captures
`report-test-timings.mjs --capture` uploads as
`test-core-timings-N-of-6`. Those come from the same runs the refresh
reads and are retained exactly as long as the run summaries (measured on
run 38195222269: all six captures carry the same `expires_at` day as its
six run summaries). So the per-file list is still a read of an existing
measurement, from those captures. Packages without a per-file reading
are named in the body with the reason: carried (every run of the window
replayed it) or executed with no vitest file line captured. A run whose
captures did not arrive is named too.
- **H3 holds**, with one change: the replay over the last two datasets
cannot show density or files (below).
- **H4 holds.** The body is the compose step's `echo` block; the section
is one `cat` of the diff's output. Lists that grow with the workspace
are capped: non-finding rises at 25 rows, files at 100, each with a
count and the full list printed to the job log. Findings are never
capped.
## Replay: what the diff says over the last two datasets on main
`--diff --prior` the #22368 dataset (0401847, measured 2026-10-08, 21
runs) `--refreshed` the #22923 dataset (1bd612b, measured 2026-10-12,
12 runs, the first with ceilings). Stderr: `12 rise(s) of 1.25x or more,
0 finding(s) (BASELINE: none judged); 33 down; files over budget NOT
MEASURED`.
| package | previous s | this window s | rise | verdict |
|---|--:|--:|--:|---|
| `@objectstack/trigger-api` | 2.18 | 23.85 | 10.94× | baseline |
| `@objectstack/example-todo` | 43.17 | 66.07 | 1.53× | baseline |
| `@objectstack/example-showcase` | 93.95 | 142.11 | 1.51× | baseline |
| `@objectstack/service-realtime` | 10.39 | 14.72 | 1.42× | baseline |
| `@objectstack/service-cluster-redis` | 15.30 | 20.59 | 1.35× |
baseline |
| `@objectstack/plugin-auth` | 359.89 | 465.63 | 1.29× | baseline |
| `@objectstack/organizations` | 79.67 | 102.91 | 1.29× | baseline |
| `@objectstack/metadata-core` | 18.92 | 24.32 | 1.29× | baseline |
| `@objectstack/plugin-dev` | 49.47 | 63.57 | 1.29× | baseline |
| `@objectstack/plugin-pinyin-search` | 32.56 | 41.80 | 1.28× | baseline
|
| `@objectstack/connector-slack` | 33.72 | 42.85 | 1.27× | baseline |
| `@objectstack/service-settings` | 73.71 | 92.33 | 1.25× | baseline |
Down: 33 lighter, 12 of them to 80 % or less of the previous weight
(`cloud-connection` 153.64 to 81.68 s, `service-analytics` 401.06 to
220.50 s, `formula`, `client`, `connector-rest`, `sdui-parser`,
`service-cluster`, `driver-sqlite-wasm`, `plugin-hono-server`,
`example-crm`, +2). Not compared: 0 carried, 0 new, 0 gone.
Read it as: neither dataset carries a `density` reading, so the replay
is a BASELINE by construction and none of the 12 rises would have been a
card. Twelve packages up and twelve down by the same factor across a
4-day gap is consistent with the density problem the ruling diagnosed
(`cloud-connection`, `service-analytics` and `client`, three of the four
packages the hourly reds named, are among the downs), and without a tag
there is no telling load from code in either list. The first live diff
after this lands differs in two ways: its refreshed dataset carries
`density`, so the reason reads "the previous dataset carries no density
reading" (the self-test pins that text), and its files list is read from
that window's captures. Density and files for these two windows: NOT
MEASURED. The #22368 window's artifacts are past their 1-day retention;
#22923's captures are still retained, but this container cannot download
an artifact (measured: `GET /actions/artifacts/11723746884/zip`
redirects to the blob host, which answers 403, as the workflow header
records).
## Evidence, on head 72443a3
- `node scripts/measure-test-shard-timings.mjs --self-test` → `self-test
OK`, exit 0. New battery `refresh diff and per-file budget (#22949 E)`,
floor 22 = cases registered (a floor of 23 reds, measured); roster floor
4 → 5. The two acceptance fixtures come first: a density-only rise (a
fifth package executes, 4 to 5 per run, every suite 1.3x slower, `a` up
1.3x with its share down to 0.81x → `density only`) and a code rise
(same density, `a` alone 2.5x, share 2.17x → `finding`).
- `node scripts/ci/select-shard-timings-run.mjs --self-test` →
`self-test OK`, exit 0. +2 cases in its first battery (floor 35 → 37 =
registered) and a new battery `shard-timings-refresh per-file captures
(#22949 E)` (floor 5; roster floor 5 → 6). It drives the new download
step's own `run:` block under `bash -e` with stub `curl`/`unzip` (one
capture fails to download, one run retained none) and then the real
`--diff` over what landed.
- **Ablations, one-time, through `scripts/ablation-replace.mjs`, each
restored to `git diff HEAD` empty:**
- Density guard removed (`else verdict = finding`): generator self-test
exit 1, `diff: a rise explained by density alone was read as finding
(findings 4)`.
- The download step made to `exit 7` on a failed capture: selector
self-test exit 1, `captures step: a missing capture failed the step
(exit 7)`. The first attempt at this one was a NO-OP, refused by the
tool (its replacement contained its anchor, so the anchor count did not
move); rerun with a disjoint anchor.
- `actionlint` 1.7.7 on the workflow: exit 0. Control leg: a copy with a
step `if:` reading an undefined step id is reported (`property "nosuch"
is not defined`). No `shellcheck` on this box, so the step scripts were
not shellchecked: NOT MEASURED.
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` (no paths; derived from the merge base): 51
commands, all run on 72443a3, all exit 0; `--ran` reconciliation: 51
derived, 51 run, 0 unrun. `check:declared-population-live` was red on an
earlier head (the capture schema constant was a quoted, path-shaped
module-scope literal that reached no tracked file) and is green after
the fix described under Deviations.
- **NOT MEASURED: a live refresh.** This lane writes to `main` on its
own schedule; the first scheduled refresh after merge is the
measurement. The closest live reading is this PR's own `Shard Timings
Refresh` check: on `pull_request` it runs the real selection,
regeneration, capture download and `--diff` against main's dataset, and
renders the would-be body (BASELINE, with real density tags and real
files over budget) into its step summary. A job summary has no REST
endpoint, so that reading is in the web UI, and the diff's one-line
verdict and the captures read are in the job log.
## Deviations
- **The selector is touched** for the capture ids. The claim named it
"only if the window selection must carry the density tag"; the density
tag needed nothing from it (the generator reads it from the run
summaries it already gets), but the per-file read needs the capture ids
the selector already sees and dropped.
- **A new workflow step** ("Download the window's per-file timing
captures for the pull request body"). The claim scoped the workflow to
the PR body and the instruction lines; this step exists only to feed the
body, because the premise that the dataset carries per-file readings did
not hold.
- **The stale "the brake fired on main" sentence** in the compose step's
ceiling line is rewritten as a reading (relayed by the director seat
from the #23058 dev): "a reading, not a red -- under #22949 ruling E the
ceiling reports on every event, and a regression is judged by the diff
at the top of this body". The generator's matching code comment is
reworded the same way.
- **`scripts/measure-test-shard-timings.mjs` now carries a
`dispatch-gates: no-path-population` marker**, and the capture schema is
spelled `['test-timing-capture', 1].join('/')`, with a comment saying
why, which is the remedy `check:declared-population-live` itself
prescribes for a gate that reads no tracked path. The family is still
placed by its own file (re-derived: `matched via
scripts/measure-test-shard-timings.mjs`).
## Acceptance notes
- The workflow header's older paragraph still says `Test Core` "grades
every executed package" against the ceilings. Under ruling E it reports;
I left it because it is not on this card's lines and it describes
#23058's step. Carrier: none.
- Interpretation, open to the seat: files over budget are filed as one
card per owning package naming its files, not one card per file, and a
package or file with an open card gets the new numbers there as a
comment instead of a second card. #22998 says "a file over 10 s of wall
is a card for the owning lane"; one card per file would be dozens of
cards a week on the first lists.
- The density reading counts packages, not seconds per shard. A run's
contention also depends on how its packages land on the six shards;
ruling E names the executed-package count as the tag, and the share
guard absorbs a whole run slowing down whatever the cause.
## 维护者速读(草稿)
**改了什么**
每周的测试耗时刷新 PR(机器人开的那张)从此在正文最上面给出三张清单:比上一份数据集慢了 25%
以上的包(两个数字、最慢的测试文件、判定)、本窗口每次运行实际执行了多少个包(密度)、以及单个耗时超过 10 秒的测试文件。数据集里新增一个
`density` 字段记录密度。工作流头注释写明:机器人不立卡,落地这张刷新 PR 的席位按清单立卡,永远不让构建变红。
**为什么改**
裁定 E 已把墙钟上限改成只报告,回归检测需要一个新的落点。同一个包在空闲小时和繁忙小时的耗时能差 2.5
倍,所以只比数字会把"整台机器更忙"误判成"代码变慢";密度标签和"占比不变就不算"这道闸门就是用来区分这两者的。
**风险与代价(含回滚)**
不改任何门禁、公式、触发条件和写路径,不会让任何构建变红。代价是每次刷新多下载每次运行的 6 个小文件(各约 15
KB),下载失败只会在正文里点名,不会丢掉这次刷新。合并后的第一次刷新只能当基线(上一份数据集没有密度),从第二次可比较的刷新起才会出现需要立卡的条目。回滚:还原这一个
PR 即可;数据集里多出的 `density` 字段没有任何读者,留着无害。
**席位意见**
**你要做的**
批准这张 PR(工作流改动按 Tier H 处理,保持草稿)。合并后看第一次周一刷新 PR 的正文是否带着三张清单和"这是基线"的说明。
---
_Generated by [Claude
Code](https://claude.ai/code/session_01LqNgdPZB2Jnr5HpGbaqLeB)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 56986e9 commit d0f1c5b
3 files changed
Lines changed: 1360 additions & 14 deletions
File tree
- .github/workflows
- scripts
- ci
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
154 | 154 | | |
155 | 155 | | |
156 | 156 | | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
157 | 200 | | |
158 | 201 | | |
159 | 202 | | |
| |||
590 | 633 | | |
591 | 634 | | |
592 | 635 | | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
593 | 687 | | |
594 | 688 | | |
595 | 689 | | |
| |||
684 | 778 | | |
685 | 779 | | |
686 | 780 | | |
687 | | - | |
688 | | - | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
689 | 784 | | |
690 | 785 | | |
691 | 786 | | |
692 | 787 | | |
693 | 788 | | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
694 | 821 | | |
695 | 822 | | |
696 | 823 | | |
| |||
704 | 831 | | |
705 | 832 | | |
706 | 833 | | |
| 834 | + | |
| 835 | + | |
707 | 836 | | |
708 | 837 | | |
709 | 838 | | |
| |||
0 commit comments