Skip to content

Commit 8113763

Browse files
build(spec): drop the DTS pass's duplicate type check — 93% to 83% of its 6144 MB ceiling (#20483)
Part of #20419 Clause-②: no ## What changes One file, `packages/spec/tsup.config.ts`: - The DTS pass now runs with `compilerOptions: { noCheck: true }`. - The pass's docblock now records today's measurements and the cause of the pass's weight (one TypeScript program per entry). It also records that the pass is not byte-stable. It keeps the ⛔ against raising the ceiling. The `build` script in `packages/spec/package.json` is unchanged, and so is the 6144 ceiling. This PR says `Part of`, not `Fixes`. It is a measured mitigation that stays inside the card's file surface. The root cause has a much larger fix, but that fix moves emitted declaration bytes in about a dozen packages. The dispatch routes that kind of change to a separate decision (last section), so the card stays open for it. ## Measurement: the pass at its ceiling **How it was measured.** - The DTS pass ran alone, spelled exactly as the build script spells it (`NODE_OPTIONS=--max-old-space-size=6144 BUILD_DTS=true tsup`), plus `--trace-gc` on node's argv. - It ran inside a cgroup-v1 memory cgroup capped at 8192 MB. The cap was proven to kill first: a 200 MB allocation in a 64 MB cap exits 137. - **Live heap** is the largest heap V8 kept after a mark-compact, read from `--trace-gc`. - **Peak RSS** is the cgroup's peak anonymous RSS, sampled every 50 ms. - Wall times come from a shared container, so compare them as ratios. | tree | pass | live heap | of 6144 | peak RSS | wall | |---|---|---|---|---|---| | `8cdbe0c6e5` (dispatch base) | as on main, 3 runs | 5633-5658 MB | 92% | 6161-6177 MB | 181-194 s | | `8cdbe0c6e5` | noCheck | 5083 MB | 83% | 5889 MB | 134 s | | `ec6a275177` (base + main at `dc0ab6a2ed`) | as on main (reverse verification) | 5708 MB | 93% | 6252 MB | 183 s | | `ec6a275177` | noCheck (this PR) | 5090 MB | 83% | 5905 MB | 131 s | - **Live heap grew fast.** It went from 5658 to 5708 MB over the 12 main commits between `8cdbe0c6e5` (13:30Z) and `dc0ab6a2ed` (15:13Z). In that last reading, the largest heap before a mark-compact was 5984 MB. The worker's hard heap limit is 6192 MB (6144 old space plus the young generation). That is the margin where V8 gives up with `ERR_WORKER_OUT_OF_MEMORY`. Heap growth mid-pass on the Test Core run was not measured here. - **Reverse verification** was run from the committed state with `scripts/ablation-replace.mjs`: - The mutation removed the `compilerOptions` block (anchor 1 → 0). - The restore returned the file's blob to HEAD's `0bc508b29eab`, and `git diff HEAD` was empty afterwards. - **Full build at the final head `8645ad7e07`.** `pnpm --filter @objectstack/spec build` ran inside the same 8 GB cgroup: - It exited 0, with peak RSS 6000 MB, in 169 s (DTS pass 128 s). - `check-dts-emitted`: 36/36 declared declaration files present. - `check-dts-references`: 130 files, 394/394 relative references resolved. ## Root cause: one `ts.Program` per entry - tsup 8.5.1 runs its DTS pass through a bundled copy of rollup-plugin-dts 6.1.1. - That plugin's `createPrograms` groups entries into programs by a directory key. - tsup always passes the tsconfig path. On that path, `getCompilerOptions` hits its config cache for every entry after the first, and a cache hit keys the entry by its own directory instead of the config's. - Every spec entry lives in its own directory, so the 18 entries get 18 programs. Each program parses, binds and declaration-emits its whole reachable graph again. - The same code is still in rollup-plugin-dts 6.5.1, the latest release. tsup 8.5.1 is also the latest tsup. **Evidence.** - A `ts.createProgram` probe (a `--require` preload in the DTS worker) on a three-entry pass printed three programs, each with one root. They held 193, 123 and 274 non-declaration source files, and did 145, 0 and 69 emits. - Peak live heap grows with the number of entries, not with the graph: - each single entry alone: 330-810 MB, 3-27 s; - the 9-entry half with `src/index.ts`: 3261 MB; - the other 9-entry half: 1442 MB; - all 18 entries: 5658 MB. - For comparison, `tsc --noEmit` over the whole package peaks at 1103 MB in 18 s. - A copy of tsup with the cache-hit key corrected (outside this tree; the store's hard-linked original was not touched) built one program with 18 roots and 394 emits: **1379 MB live, 53 s**. ## Why `noCheck` is safe - rollup-plugin-dts forces `noEmitOnError`. Before each emit, every program therefore ran a full semantic check of the file it was emitting. - The package's `typecheck` script already does that check, as `tsc --noEmit` over the same `tsconfig.json`, in the required `TypeScript Type Check` job. - `noCheck` drops only the duplicate. Syntactic, option, global and declaration diagnostics still fail the pass, so a declaration that cannot be emitted still stops the build. - A plain type error in `src/` is reported by `typecheck`, as it already was. AGENTS.md's Build & Test block already says tsup never type-checks. ## Emitted declarations: the same types. Byte identity was never a property of main. The dispatch asked for a byte-identical tree. That cannot be measured against main, because main does not produce the same bytes twice: - Three runs of `8cdbe0c6e5`, same config, gave three tree digests: `7bf19190…`, `5cf3234c…`, `6fd2cecf…`. The digest is one sha256 over the sorted list of per-file sha256 values. - Each run had the same 128 files and the same 30389539 bytes. The differences are union members, and the members of the object types built from them, printed in type-creation order. The content-hashed chunk names change with them. So the trees were compared in an order-insensitive form. Each file is re-printed through the TypeScript printer with: - union members sorted, - type literals made only of property signatures sorted, - rollup's 8-character chunk hashes stripped from file names and specifiers. Nothing else is normalised. - `8cdbe0c6e5`: main run b, main run c and the noCheck run give **one** normalised digest, with 0 files differing. - `ec6a275177`: main behaviour, noCheck, and the tree from the full build give **one** normalised digest, with 0 files differing. - Negative controls: renaming one union member and deleting one property in a copied tree were both caught. The comparison named exactly the one and then two mutated files. ⇒ noCheck's tree differs from main's only in the ways two builds of main already differ from each other. ## Why not split the pass across entries The split was measured. Two halves peak at 3261 MB and 1442 MB, so memory would fit. But a split redraws rollup's shared chunks. The halves emitted 60 + 32 files against the single pass's 128. A split publishes duplicated declarations across entries, and a class or `unique symbol` declared twice stops being one type. That changes what publishes, which the dispatch puts on a different card. ## What reads the changed file - **The DTS pass.** This is the only place the change takes effect; the JS pass runs with `dts: false`. - **turbo's `build` task.** The file is one of spec's package inputs, so spec and its dependents rebuild once in CI. - **`scripts/check-dev-prereqs.mjs`.** Its build-input hash includes `tsup.config.ts`. A dev dist built before this change reads as stale until it is rebuilt, which is the intended behaviour. - **The docs deploy.** It builds spec through `turbo run build --filter=@objectstack/docs`, so it runs the same pass, now lighter. No consumer of the `build` script invokes anything new. ## Gates, final head `8645ad7e07` - `pnpm --filter @objectstack/spec build`: exit 0 (8 GB cgroup, above). - `pnpm --filter @objectstack/spec check:generated`: exit 0. All 15 generated artifacts are up to date against the dist that build emitted, with a declaration stamp match. - `pnpm check:turbo-task-graph`: exit 0. - `pnpm check:dts-closure`: exit 0 (36/36). - `pnpm check:nul-bytes`: exit 0. - `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands` derived 51 commands. `--ran` reconciled them as 51 accounted: 49 exit 0, and 2 NOT MEASURED (exit 3, prerequisite): - `check:dual-build-cjs-loads` needs every package built. - `check:lean-entry-closure` needs `@objectstack/objectql` built. - This change does not touch the JS pass either of them reads. Both are left to CI. - Spec's dist readers: `check:browser-reachable-entries`, `check:entry-nameability`, `check:dual-source-exports` and `check:exported-any` all exit 0. - Tests: - `pnpm --filter @objectstack/spec exec vitest run --project local --maxWorkers=2`: 568 files, 16691 passed, 1 todo. - The 8 test files that read `tsup.config.ts` as text also passed on their own run (130 tests). - `typecheck` was not run. The changed file is in none of the package's three tsc programs (`tsconfig.json` and `tsconfig.test.json` include `src/**/*`; `tsconfig.scripts.json` includes `scripts/**/*`), so its inputs match main's. Changeset: none, and `skip-changeset`. The JS outputs and every shipped source file are untouched. The declaration tree matches main's in the order-insensitive form above. A byte digest cannot tell this build apart from another build of main. ## Acceptance notes - **The pass is not byte-stable run to run.** This is the observation above, recorded in the docblock. Any gate or review that compares two declaration trees by byte digest will report changes that are not there. - **The previous docblock said every completing ceiling emitted a byte-identical tree.** That no longer holds at `8cdbe0c6e5`. The docblock's measured table was replaced with today's numbers, dated by commit. - **This is a mitigation, not the fix.** noCheck restores about 620 MB of headroom (93% → 83% of the ceiling). At the growth measured above, that headroom is not durable, so the root cause needs the decision below. ## Decision needed: the root cause Cutting 18 programs to one takes the pass from about 5.7 GB to 1.4 GB live, and from about 185 s to 53 s. Measured on spec, the one-program tree matches main's once three more kinds of ordering are also normalised: - top-level statement order, - the order of names inside `import` / `export` braces, - one shared chunk's name (`data-engine` → `analytics.zod`, paired by content). The same kinds of change would reach every other multi-entry tsup package (about eleven, including `core`, `objectql`, `metadata`, `types` and `plugin-auth`). The options are in the dispatch report on the card; the recommendation there is a `pnpm patch` of tsup's bundled rollup-plugin-dts plus an upstream report. --- _Generated by [Claude Code](https://claude.ai/code/session_014EJ1ED8X4MMrT18BhVx4tx)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 851af0c commit 8113763

1 file changed

Lines changed: 49 additions & 15 deletions

File tree

‎packages/spec/tsup.config.ts‎

Lines changed: 49 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -176,26 +176,59 @@ const swapServerOnlyGrammarArm: Plugin = {
176176
* dependency — so this pass meets the container's limit ALONE, with no
177177
* parallelism to cap.
178178
*
179-
* Measured on this package's DTS pass, inside a cgroup capped at 8192 MB
180-
* (peak anonymous RSS of the whole process tree):
179+
* 6144 is the largest ceiling whose WORST case still fits: V8 cannot exceed
180+
* it, and the pass's non-heap overhead measures ~250 MB, so the bound is
181+
* ~6.4 GB inside an 8 GB container.
181182
*
182-
* ceiling result peak RSS wall
183-
* 12288 ok, but only ~0.9 GB spare 7290 MB 132s
184-
* 6144 ok 5794 MB 134s
185-
* 5120 ok 5328 MB 148s
186-
* 4096 ERR_WORKER_OUT_OF_MEMORY — 113s
183+
* WHY THE PASS IS THIS HEAVY: ONE `ts.Program` PER ENTRY. tsup 8.5.1 runs the
184+
* pass through its bundled rollup-plugin-dts 6.1.1, whose `createPrograms`
185+
* groups entries by a directory key. tsup always passes the tsconfig path, and
186+
* on that path every entry after the first is keyed by its OWN directory (the
187+
* same code is in rollup-plugin-dts 6.5.1). So each entry above gets its own
188+
* program, which parses, binds and emits its whole reachable graph again: the
189+
* peak grows with entries × graph, not with the graph. ⇒ Every entry added to
190+
* `entries` adds a program to this pass.
187191
*
188-
* 6144 is chosen as the largest ceiling whose WORST case still fits: V8 cannot
189-
* exceed it, and the pass's non-heap overhead measured ~250 MB, so the bound is
190-
* ~6.4 GB inside an 8 GB container. Every completing ceiling emitted a
191-
* byte-identical declaration tree (122 files, one sha256 over all of them), so
192-
* this number buys headroom and costs nothing but GC time.
192+
* `noCheck`: rollup-plugin-dts forces `noEmitOnError`, which made every one of
193+
* those programs also semantically CHECK each file it emitted — a type check
194+
* the `typecheck` script (`tsc --noEmit` over this same tsconfig, run by the
195+
* required `TypeScript Type Check` job) already performs once. `noCheck` drops
196+
* only that duplicate. Syntactic, option, global and declaration diagnostics
197+
* still fail the pass, so a declaration that cannot be emitted still stops the
198+
* build; a plain type error in `src/` is `typecheck`'s to report, not this
199+
* pass's.
200+
*
201+
* Measured on 8cdbe0c6e5's source, DTS pass alone at the 6144 ceiling, inside a
202+
* cgroup capped at 8192 MB. Live heap is the largest heap V8 kept after a
203+
* mark-compact (`--trace-gc`); peak RSS is the peak anonymous RSS of the whole
204+
* process tree:
205+
*
206+
* pass live heap peak RSS wall
207+
* duplicate check on (before) 5658 MB 6177 MB 181-194s
208+
* noCheck (this config) 5083 MB 5889 MB 134s
209+
* one program, grouping patched 1379 MB 3749 MB 53s
210+
* `tsc --noEmit`, whole package 1103 MB 1149 MB 18s
211+
*
212+
* The third row was measured with the grouping patched in a copy of tsup
213+
* outside this tree. It shows what cutting the program count is worth.
214+
*
215+
* NOT BYTE-STABLE: this pass does not emit the same bytes twice. TypeScript
216+
* prints union members, and the members of object types derived from them, in
217+
* type-creation order, and that order follows emit order. Three runs of the
218+
* same commit gave three different trees, and rollup's content-hashed chunk
219+
* names moved with them. Once union and property-signature order are
220+
* normalised, all three runs and the `noCheck` run are the same tree (128
221+
* files, 30389539 bytes). ⇒ Compare two declaration trees in such an
222+
* order-insensitive form, never by a byte digest.
193223
*
194224
* If this pass starts failing with `ERR_WORKER_OUT_OF_MEMORY`, the live type
195225
* graph has outgrown 6144 — that is a loud, actionable failure and the point of
196226
* the ceiling. ⛔ Do not "fix" it by raising the number past what the build
197-
* container has; that trades this error back for the silent exit 137. Shrink
198-
* the graph, or split the pass across entries.
227+
* container has; that trades this error back for the silent exit 137. Cut the
228+
* program count, shrink the graph, or split the pass across entries. Cutting
229+
* the count moves statement order and one chunk name beyond the noise above. A
230+
* split redraws the shared chunks. Either one changes what publishes, so it
231+
* needs that reviewed first.
199232
*/
200233
const isDts = process.env.BUILD_DTS === 'true';
201234

@@ -204,7 +237,8 @@ const mainConfig: Options = {
204237
splitting: false,
205238
sourcemap: true,
206239
clean: !isDts, // Only clean on main build, not on DTS pass
207-
dts: !isDts ? false : { only: true }, // Only generate DTS on explicit pass, without JS
240+
// Only generate DTS on the explicit pass, without JS; `noCheck` per the docblock above.
241+
dts: !isDts ? false : { only: true, compilerOptions: { noCheck: true } },
208242
format: ['esm', 'cjs'],
209243
target: 'es2020',
210244
treeshake: true,

0 commit comments

Comments
 (0)