Repository navigation
Commit 8113763
build(spec): drop the DTS pass's duplicate type check — 93% to 83% of its 6144 MB ceiling (#20483)
Part of #20419
Clause-②: no
## What changes
One file, `packages/spec/tsup.config.ts`:
- The DTS pass now runs with `compilerOptions: { noCheck: true }`.
- The pass's docblock now records today's measurements and the cause of
the pass's weight (one TypeScript program per entry). It also records
that the pass is not byte-stable. It keeps the ⛔ against raising the
ceiling.
The `build` script in `packages/spec/package.json` is unchanged, and so
is the 6144 ceiling.
This PR says `Part of`, not `Fixes`. It is a measured mitigation that
stays inside the card's file surface. The root cause has a much larger
fix, but that fix moves emitted declaration bytes in about a dozen
packages. The dispatch routes that kind of change to a separate decision
(last section), so the card stays open for it.
## Measurement: the pass at its ceiling
**How it was measured.**
- The DTS pass ran alone, spelled exactly as the build script spells it
(`NODE_OPTIONS=--max-old-space-size=6144 BUILD_DTS=true tsup`), plus
`--trace-gc` on node's argv.
- It ran inside a cgroup-v1 memory cgroup capped at 8192 MB. The cap was
proven to kill first: a 200 MB allocation in a 64 MB cap exits 137.
- **Live heap** is the largest heap V8 kept after a mark-compact, read
from `--trace-gc`.
- **Peak RSS** is the cgroup's peak anonymous RSS, sampled every 50 ms.
- Wall times come from a shared container, so compare them as ratios.
| tree | pass | live heap | of 6144 | peak RSS | wall |
|---|---|---|---|---|---|
| `8cdbe0c6e5` (dispatch base) | as on main, 3 runs | 5633-5658 MB | 92%
| 6161-6177 MB | 181-194 s |
| `8cdbe0c6e5` | noCheck | 5083 MB | 83% | 5889 MB | 134 s |
| `ec6a275177` (base + main at `dc0ab6a2ed`) | as on main (reverse
verification) | 5708 MB | 93% | 6252 MB | 183 s |
| `ec6a275177` | noCheck (this PR) | 5090 MB | 83% | 5905 MB | 131 s |
- **Live heap grew fast.** It went from 5658 to 5708 MB over the 12 main
commits between `8cdbe0c6e5` (13:30Z) and `dc0ab6a2ed` (15:13Z). In that
last reading, the largest heap before a mark-compact was 5984 MB. The
worker's hard heap limit is 6192 MB (6144 old space plus the young
generation). That is the margin where V8 gives up with
`ERR_WORKER_OUT_OF_MEMORY`. Heap growth mid-pass on the Test Core run
was not measured here.
- **Reverse verification** was run from the committed state with
`scripts/ablation-replace.mjs`:
- The mutation removed the `compilerOptions` block (anchor 1 → 0).
- The restore returned the file's blob to HEAD's `0bc508b29eab`, and
`git diff HEAD` was empty afterwards.
- **Full build at the final head `8645ad7e07`.** `pnpm --filter
@objectstack/spec build` ran inside the same 8 GB cgroup:
- It exited 0, with peak RSS 6000 MB, in 169 s (DTS pass 128 s).
- `check-dts-emitted`: 36/36 declared declaration files present.
- `check-dts-references`: 130 files, 394/394 relative references
resolved.
## Root cause: one `ts.Program` per entry
- tsup 8.5.1 runs its DTS pass through a bundled copy of
rollup-plugin-dts 6.1.1.
- That plugin's `createPrograms` groups entries into programs by a
directory key.
- tsup always passes the tsconfig path. On that path,
`getCompilerOptions` hits its config cache for every entry after the
first, and a cache hit keys the entry by its own directory instead of
the config's.
- Every spec entry lives in its own directory, so the 18 entries get 18
programs. Each program parses, binds and declaration-emits its whole
reachable graph again.
- The same code is still in rollup-plugin-dts 6.5.1, the latest release.
tsup 8.5.1 is also the latest tsup.
**Evidence.**
- A `ts.createProgram` probe (a `--require` preload in the DTS worker)
on a three-entry pass printed three programs, each with one root. They
held 193, 123 and 274 non-declaration source files, and did 145, 0 and
69 emits.
- Peak live heap grows with the number of entries, not with the graph:
- each single entry alone: 330-810 MB, 3-27 s;
- the 9-entry half with `src/index.ts`: 3261 MB;
- the other 9-entry half: 1442 MB;
- all 18 entries: 5658 MB.
- For comparison, `tsc --noEmit` over the whole package peaks at 1103 MB
in 18 s.
- A copy of tsup with the cache-hit key corrected (outside this tree;
the store's hard-linked original was not touched) built one program with
18 roots and 394 emits: **1379 MB live, 53 s**.
## Why `noCheck` is safe
- rollup-plugin-dts forces `noEmitOnError`. Before each emit, every
program therefore ran a full semantic check of the file it was emitting.
- The package's `typecheck` script already does that check, as `tsc
--noEmit` over the same `tsconfig.json`, in the required `TypeScript
Type Check` job.
- `noCheck` drops only the duplicate. Syntactic, option, global and
declaration diagnostics still fail the pass, so a declaration that
cannot be emitted still stops the build.
- A plain type error in `src/` is reported by `typecheck`, as it already
was. AGENTS.md's Build & Test block already says tsup never type-checks.
## Emitted declarations: the same types. Byte identity was never a
property of main.
The dispatch asked for a byte-identical tree. That cannot be measured
against main, because main does not produce the same bytes twice:
- Three runs of `8cdbe0c6e5`, same config, gave three tree digests:
`7bf19190…`, `5cf3234c…`, `6fd2cecf…`. The digest is one sha256 over the
sorted list of per-file sha256 values.
- Each run had the same 128 files and the same 30389539 bytes. The
differences are union members, and the members of the object types built
from them, printed in type-creation order. The content-hashed chunk
names change with them.
So the trees were compared in an order-insensitive form. Each file is
re-printed through the TypeScript printer with:
- union members sorted,
- type literals made only of property signatures sorted,
- rollup's 8-character chunk hashes stripped from file names and
specifiers.
Nothing else is normalised.
- `8cdbe0c6e5`: main run b, main run c and the noCheck run give **one**
normalised digest, with 0 files differing.
- `ec6a275177`: main behaviour, noCheck, and the tree from the full
build give **one** normalised digest, with 0 files differing.
- Negative controls: renaming one union member and deleting one property
in a copied tree were both caught. The comparison named exactly the one
and then two mutated files.
⇒ noCheck's tree differs from main's only in the ways two builds of main
already differ from each other.
## Why not split the pass across entries
The split was measured. Two halves peak at 3261 MB and 1442 MB, so
memory would fit.
But a split redraws rollup's shared chunks. The halves emitted 60 + 32
files against the single pass's 128. A split publishes duplicated
declarations across entries, and a class or `unique symbol` declared
twice stops being one type. That changes what publishes, which the
dispatch puts on a different card.
## What reads the changed file
- **The DTS pass.** This is the only place the change takes effect; the
JS pass runs with `dts: false`.
- **turbo's `build` task.** The file is one of spec's package inputs, so
spec and its dependents rebuild once in CI.
- **`scripts/check-dev-prereqs.mjs`.** Its build-input hash includes
`tsup.config.ts`. A dev dist built before this change reads as stale
until it is rebuilt, which is the intended behaviour.
- **The docs deploy.** It builds spec through `turbo run build
--filter=@objectstack/docs`, so it runs the same pass, now lighter.
No consumer of the `build` script invokes anything new.
## Gates, final head `8645ad7e07`
- `pnpm --filter @objectstack/spec build`: exit 0 (8 GB cgroup, above).
- `pnpm --filter @objectstack/spec check:generated`: exit 0. All 15
generated artifacts are up to date against the dist that build emitted,
with a declaration stamp match.
- `pnpm check:turbo-task-graph`: exit 0.
- `pnpm check:dts-closure`: exit 0 (36/36).
- `pnpm check:nul-bytes`: exit 0.
- `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack
--commands` derived 51 commands. `--ran` reconciled them as 51
accounted: 49 exit 0, and 2 NOT MEASURED (exit 3, prerequisite):
- `check:dual-build-cjs-loads` needs every package built.
- `check:lean-entry-closure` needs `@objectstack/objectql` built.
- This change does not touch the JS pass either of them reads. Both are
left to CI.
- Spec's dist readers: `check:browser-reachable-entries`,
`check:entry-nameability`, `check:dual-source-exports` and
`check:exported-any` all exit 0.
- Tests:
- `pnpm --filter @objectstack/spec exec vitest run --project local
--maxWorkers=2`: 568 files, 16691 passed, 1 todo.
- The 8 test files that read `tsup.config.ts` as text also passed on
their own run (130 tests).
- `typecheck` was not run. The changed file is in none of the package's
three tsc programs (`tsconfig.json` and `tsconfig.test.json` include
`src/**/*`; `tsconfig.scripts.json` includes `scripts/**/*`), so its
inputs match main's.
Changeset: none, and `skip-changeset`. The JS outputs and every shipped
source file are untouched. The declaration tree matches main's in the
order-insensitive form above. A byte digest cannot tell this build apart
from another build of main.
## Acceptance notes
- **The pass is not byte-stable run to run.** This is the observation
above, recorded in the docblock. Any gate or review that compares two
declaration trees by byte digest will report changes that are not there.
- **The previous docblock said every completing ceiling emitted a
byte-identical tree.** That no longer holds at `8cdbe0c6e5`. The
docblock's measured table was replaced with today's numbers, dated by
commit.
- **This is a mitigation, not the fix.** noCheck restores about 620 MB
of headroom (93% → 83% of the ceiling). At the growth measured above,
that headroom is not durable, so the root cause needs the decision
below.
## Decision needed: the root cause
Cutting 18 programs to one takes the pass from about 5.7 GB to 1.4 GB
live, and from about 185 s to 53 s. Measured on spec, the one-program
tree matches main's once three more kinds of ordering are also
normalised:
- top-level statement order,
- the order of names inside `import` / `export` braces,
- one shared chunk's name (`data-engine` → `analytics.zod`, paired by
content).
The same kinds of change would reach every other multi-entry tsup
package (about eleven, including `core`, `objectql`, `metadata`, `types`
and `plugin-auth`). The options are in the dispatch report on the card;
the recommendation there is a `pnpm patch` of tsup's bundled
rollup-plugin-dts plus an upstream report.
---
_Generated by [Claude
Code](https://claude.ai/code/session_014EJ1ED8X4MMrT18BhVx4tx)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 851af0c commit 8113763
1 file changed
Lines changed: 49 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
179 | | - | |
180 | | - | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
181 | 182 | | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
187 | 191 | | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
193 | 223 | | |
194 | 224 | | |
195 | 225 | | |
196 | 226 | | |
197 | | - | |
198 | | - | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
199 | 232 | | |
200 | 233 | | |
201 | 234 | | |
| |||
204 | 237 | | |
205 | 238 | | |
206 | 239 | | |
207 | | - | |
| 240 | + | |
| 241 | + | |
208 | 242 | | |
209 | 243 | | |
210 | 244 | | |
| |||
0 commit comments