test(node-compat): refresh the Node-compat corpus to 26.7.0 and score it under named lenses - #785
test(node-compat): refresh the Node-compat corpus to 26.7.0 and score it under named lenses#785colinhacks wants to merge 18 commits into
Conversation
…er named lenses - tests/cross-runtime: corpus = Node v26.7.0 test/ tree; Deno's config.jsonc from main 98f9507a; lenses for Deno's convention, the bun.com universe, the full corpus, and each minus a 718-test engine-specific class; bun now receives the tests' // Flags: on the command line (it ignores NODE_OPTIONS); symmetric one-retry pass; --bin/--files/--merge/--out; failure output kept in results - tests/node-compat-config.jsonc: regenerated by tests/node-compat-regen.mjs from the run — 5,231 entries over six directories, 128 classified divergences - crates/nub-cli/tests/node_compat.rs and tests/run-node-compat.sh: pass each test's // Flags: with NODE_SKIP_FLAG_CHECK=1, so test/common no longer re-spawns flagged tests under plain node - tests/node-suite: v26.7.0 - wiki/research/node-test-suite-leverage.md: measured section and changelog
…currently, classify three gate-only divergences The gate now runs each test from the Node checkout root (flag paths such as --experimental-loader ./test/fixtures/... are relative to it), sets NO_COLOR=1 like the cross-runtime runner, and reads the child's stderr while it runs — reading it only after exit let a chatty test fill the pipe and block, which the gate then reported as a hang. The generator marks the three entries that diverge only inside a full checkout (the repo tsconfig.json maps internal/* onto lib/ through paths, which nub resolves; one fd probe depends on the harness's stdio). Measured: 5,100 / 5,100 non-ignored entries pass.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
ℹ️ Minor suggestions — no correctness blocker found. The lens numbers reconcile exactly against the committed
results.json.
Reviewed changes — the corpus bump to Node 26.7.0 (tests/node-suite → b4f23d36), the named-lens scoring machinery in tests/cross-runtime/run.mjs, the regenerated tests/node-compat-config.jsonc (5,231 entries, 131 ignored) and its new generator tests/node-compat-regen.mjs, and the flag/cwd/stderr changes to crates/nub-cli/tests/node_compat.rs and tests/run-node-compat.sh.
I recomputed every lens figure from the committed results.json and they match the README table exactly (denoExclusions 5,064 files / 4,977 node-pass; nub 98.53 raw / 96.88 node-relative; deno 74.50 / 73.80; bun 68.56 / 67.67; node25 90.42 / 88.92). nubRegressions = 142 decomposes cleanly by directory (es-module 17, module-hooks 14, parallel 89, report 14, test-runner 8), 128 of which fall in the six gate directories; plus the 3 GATE_ENVIRONMENT entries that gives the 131 config ignores. engine-specific.txt has exactly 718 entries with none missing from the results.
Two things I checked specifically and found clean:
- The retry pass is symmetric. 4,878
retried: truemarkers reconcile with themeta.retriedsums, and the 5firstAttemptrecords match the 5 verdict flips exactly — nothing is retried preferentially for one runtime. - No vacuous passes from the new env.
NODE_SKIP_FLAG_CHECKgates only the re-spawn block intest/common/index.js; fixtures andtmpdiranchor to__dirnamerather than cwd, so moving the gate's cwd to the suite root does not change what tests resolve;test_flagsmatchesrun.mjs'sparseFlags(same regex semantics, first match wins); and corpus entries absent from the config count as SKIP and are excluded from the assertion denominator rather than counting as passes.
⚠️ The published site figures now contradict the committed results
site/src/app/(home)/page.tsx (the COMPAT array, ~line 955) and site/content/blog/introducing-nub.mdx:36 still cite Deno's colinhacks/node_test @ node-25.8.1 fork, a 4,367-file denominator, and 98.6 / 78.5 / 66.8. After this PR the corpus is the 26.7.0 tree, the denominator is 5,064, and the numbers are 98.53 / 74.50 / 68.56.
Neither file is in this diff, so nothing here is wrong — but the source comment above COMPAT names tests/cross-runtime/results.json scores.denoExclusions as its provenance, and that provenance no longer produces those numbers. The README even warns that these "drift silently until someone copies them across", which is exactly the state this PR leaves them in.
Worth noting the update is not self-serving: Deno went down (78.5 → 74.50) and Bun went up (66.8 → 68.56), so the refresh moves two competitor figures against the direction anyone would choose.
Technical details
site/src/app/(home)/page.tsx (unchanged by this PR)
/* Source: tests/cross-runtime/results.json `scores.denoExclusions`,
using Deno's Node-compat corpus (colinhacks/node_test @ node-25.8.1). */
{ name: 'Node 25.8', rate: 100, tests: '4,367 / 4,367' }
{ name: 'Nub', rate: 98.6, tests: '4,307 / 4,367' }
{ name: 'Deno 2.9', rate: 78.5, tests: '3,427 / 4,367' }
{ name: 'Bun 1.4', rate: 66.8, tests: '2,916 / 4,367' }
results.json after this PR, scores.denoExclusions (5,064 files, 4,977 node-pass):
nub 98.53 raw / 96.88 node-relative
deno 74.50 / 73.80
bun 68.56 / 67.67
node25 90.42 / 88.92
ℹ️ Nitpicks
-
tests/cross-runtime/README.md— "142 … 8 are Node 26's--enable-source-mapsassert regression" is hard to derive from the artifacts. Only 4 regressions haveassertin the name (es-module/test-esm-import-assertion-warning.mjs,parallel/test-assert-first-line.js,parallel/test-assert.js,parallel/test-runner-assert.js) and 4 havesource-map/sourcemap(parallel/test-compile-cache-typescript-strip-sourcemaps.js,parallel/test-node-output-sourcemaps.mjs,parallel/test-source-map-enable.js,parallel/test-util-getcallsites-sourcemap.js), and the regenerated config classifies only 3 assource-maps. Either the grouping needs a note on how it was counted, or the number needs a recount. -
crates/nub-cli/tests/node_compat.rs:300— the comment still says "single TMPDIR, fork id 0" butNODE_TEST_FORK_IDand thefork_idparameter are gone in this PR. -
Nothing enforces the regenerated config.
.github/workflows/ci.yml:1177records that the heavyweight Node-suite corpus job was removed from CI on 2026-06-03, sotests/node-compat-config.jsoncdrifts until someone reruns the gate by hand. Not a change this PR needs to make, but worth knowing that the 5,100/5,100 in the PR body is a local-run claim rather than a checked one.
Claude Opus ｜ 𝕏
… exact directory set, scrubbed paths, untriaged entries reported apart - The gate exports TEST_SERIAL_ID per worker, which is the variable test/common/tmpdir.js keys its .tmp.<id> dir on; the parallel scan's false positives went from 142 to 0. - run.mjs enumerates exactly Deno's directory set (ffi/ and test426/ back in); the 14 files are merged into results.json, so the deno lens is Deno's own collection. - Captured output and binary paths are scrubbed of the corpus, repo and home directories before results.json is written; the config's reasons inherit that and are sanitized against the line-based comment strippers. - A divergence the generator cannot classify is written as untriaged: true, not ignore; the gate skips it but counts and reports it apart (111 classified, 20 untriaged). Gate: 5,100 / 5,100.
There was a problem hiding this comment.
ℹ️ One minor suggestion inline.
Reviewed changes — 4f9615a1 addresses all four anchored findings from the previous review; I verified each against the committed artifacts rather than the commit message, and resolved those threads.
TEST_SERIAL_IDis correctly plumbed.run_with_timeouttakesserial_idand exports it; the parallel scan passeswid, the sequential re-verify passes0.NODE_TEST_FORK_ID/fork_idare gone repo-wide. This is the keytest/common/tmpdir.jsactually reads, so the 16 workers no longer share.tmp.0.- The Deno lens is now Deno's exact collection.
ffiandtest426are no longer concatenated onto the ignore set. RecomputingdenoExclusionsfrom Deno's 19IGNORED_TEST_DIRSplus the config skips gives 5,078 files / 4,981 node-pass, matchingresults.jsonexactly. - Machine paths are gone.
scrub()maps corpus/repo/home into<corpus>/<repo>/~across tails,firstAttempt.tail,meta.binaries[].binandmeta.corpus./Users/colinmcd94/count went 4,632 → 0 inresults.jsonand 2 → 0 innode-compat-config.jsonc. The one remaining/Users/is inside a Deno panic string produced on Deno's own build machine. untriagedis now distinguishable from a real ignore. The flag is separate in the config, the Rust gate counts and prints it apart ({skipped} skipped as classified divergences, {untriaged} skipped as UNTRIAGED divergences), andrun-node-compat.shhonours it. 111 classified + 20 untriaged + 5,100 expected-pass.- The
--mergeoverlay reconciles. Only the 14 newly-included ffi/test426 verdicts were added (node 4/14, nub 3/14, bun 13/14, deno 13/14, node25 2/14); no pre-existing verdict was perturbed, and all 5,639 files carry all 5 runtime verdicts.
📊 The published site figures are stale, and the correct replacements moved again this round
Not in this diff, so no line to anchor to — but worth flagging before someone acts on the previous review's numbers, which are themselves now out of date.
site/src/app/(home)/page.tsx:955-960 (the COMPAT array) and site/content/blog/introducing-nub.mdx:36 still cite colinhacks/node_test @ node-25.8.1, a 4,367-file denominator, and 98.6 / 78.5 / 66.8. Against 4f9615a1's denoExclusions lens the current values are:
| denominator | pass | % | |
|---|---|---|---|
| nub | 4,981 | 4,907 | 98.51 |
| deno | 4,981 | 3,711 | 74.50 |
| bun | 4,981 | 3,415 | 68.56 |
Reference runtime is Node 26.7.0, not 25.8. Note the prior review quoted 5,064 / 98.53 — the ffi/test426 fix moved both, so those are no longer the numbers to copy.
ℹ️ Nitpicks
results.jsonrecordsmeta.parallelism: 4, but that describes only the 14-file--mergerun; 5,625 of the verdicts came from the earlier parallelism-6 pass. Slightly misleading as provenance for the file as a whole.- The PR body still says "131 classified divergences". With the
untriagedsplit, the config is 111 classified + 20 untriaged.
Claude Opus ｜ 𝕏
…bulk run's parallelism through a merge
…e's (4) in results.json
There was a problem hiding this comment.
ℹ️ No correctness issues — one inline suggestion and one nitpick.
Reviewed changes — the two commits since 4f9615a1, both of which address findings from the previous review. I verified each against the committed artifacts rather than the commit messages.
- Fixed the
sanitize()under-collapse.tests/node-compat-regen.mjs:70now uses/\/{2,}/g. I probed it onfile:///tmp/x,a////b,a//b//c,http://x//yand//— every result is a fixed point with no surviving//, so the gate can no longer be handed a config line that truncates mid-JSON. The comment above it was also corrected to stop listing this generator among the naive readers (its own reader at line 39 is line-anchored). Thread resolved. - Split the merge run's parallelism from the bulk run's.
meta.parallelismnow carries the prior file's value through a--merge(6) andmeta.mergeParallelismrecords the merge's own (4), which is the honest provenance for a file where 5,625 of 5,639 verdicts came from the earlier pass.
Two things I re-checked and found clean:
- The re-merge perturbed nothing.
results.jsonwas regenerated (08:58→09:17), so I recomputed every lens:denoExclusions5,078 / 4,981 with nub 98.51 (96.67 raw), deno 74.50 (73.85), bun 68.56 (67.74), node25 90.38 (88.72), andnubRegressions143. All six lens rows match the README table exactly, unchanged from the prior round. - The generator is idempotent, and the config still parses under the gate's stripper. Re-running
node tests/node-compat-regen.mjsagainst the committedresults.jsonreproducestests/node-compat-config.jsoncbyte-for-byte except its header timestamp — 111 of 111 curated reasons carried forward, 5,231 entries, 111ignore+ 20untriaged+ 5,100 empty, zero//in any reason.
ℹ️ Nitpicks
tests/node-compat-config.jsonc:8citesresults.json generated 2026-08-22T08:58:22.016Z, but the trackedresults.jsonis now09:17:32.862Z—f6401b98regenerated the results without re-running the generator. The content is unaffected (the re-merge only touchedffi/andtest426/, which fall outside the generator's sixDIRS), so a regen rewrites that one line and nothing else. Worth doing so the provenance line names a snapshot that actually exists in the tree.
Claude Opus ｜ 𝕏
… refresh the config's provenance line
There was a problem hiding this comment.
ℹ️ No correctness issues in the code — one inline finding on a committed value.
Reviewed changes — the one commit since f6401b98, which addresses both findings from the previous review. I verified each against the committed artifacts and the run history rather than the commit message.
- The
retryFailuresfix landed verbatim.tests/cross-runtime/run.mjs:725-726now readsretryFailures: prior ? prior.meta.retryFailures : RETRY_FAILURESwithmergeRetryFailuresalongsidemergeParallelism. The code site is correct. Thread replied + resolved. - The config provenance line is current and reproducible.
tests/node-compat-config.jsonc:8now cites09:17:32.862Z, matching the trackedresults.json. I re-ran the generator against a copy —5,231 entries (5,100 pass, 111 ignore with a classified reason — 111 carried forward — and 20 untriaged), anddiffagainst the committed file reports identical, header line included. The generator is byte-for-byte idempotent on this tree.
ℹ️ The meta block is now hand-maintained, which is what let the wrong value in
meta.generatedAt has stayed frozen at 2026-08-22T09:17:32.862Z across both of the last two commits, while the fields beside it moved: f6401b98 changed parallelism 4→6 and mergeParallelism 10→4, and 48f5a2c2 changed retryFailures false→true and added mergeRetryFailures. No run.mjs invocation produced those values — a run would have rewritten generatedAt too.
That contradicts the file's own contract at tests/cross-runtime/run.mjs:551: "every score is then recomputed from the merged record, so a merged file is never a hand edit." It is also the mechanism behind the inline finding below: hand-editing one field at a time is exactly how mergeParallelism and mergeRetryFailures ended up describing two different runs.
The durable fix is to re-derive meta by re-running the merge rather than editing the fields, so generatedAt always names the process that wrote the block. If a full re-merge is too expensive to redo for a metadata correction, the alternative is to stop claiming the file is never hand-edited.
Claude Opus ｜ 𝕏
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes — the one commit since 48f5a2c2, which closes both round-4 findings. I verified the merge was genuinely re-run rather than hand-corrected, since a one-field edit would have closed the inline finding while leaving the body finding fully open.
metais process-written again. Four independent confirmations:generatedAtmoved to2026-08-22T09:34:50.916Z;mergeRetryFailuresis now emitted in the writer's literal key order (… retryFailures, mergeRetryFailures, retried, pertests/cross-runtime/run.mjs:705-743) instead of appended after theretriedblock, which is what a serialized object produces and an edit does not; one failuretailcarries different microsecond timings (140→147µs, 106→75µs, 92→288µs), so a test actually re-executed; and all fivemeta.retriedcounts recompute fromresultswith zero mismatches. The same key-order argument retroactively confirms the prior state was a hand edit, sorun.mjs:551's "a merged file is never a hand edit" holds again.mergeRetryFailuresis nowtrue, matching the 35retried: truemarkers inside the merged scope and themergeParallelism: 4beside it. Thread replied + resolved.- The config header tracks the new artifact.
tests/node-compat-config.jsonc:8cites09:34:50.916Z. Re-running the generator against a copy reproduces the committed file byte-for-byte, header line included —5,231 entries (5,100 pass, 111 ignore with a classified reason — 111 carried forward — and 20 untriaged). - No scores moved. Every lens in
tests/cross-runtime/README.md's table still matchesresults.jsonexactly;denoExclusionsremains 5,078 files / 4,981 nodePass with nub 98.51, deno 74.50, bun 68.56, node25 90.38.
Claude Opus ｜ 𝕏
…heckout, pseudo-terminal for pseudo-tty/, compiled ffi fixture, wpt/ wrappers The corpus default is the tests/node-suite checkout (a full Node tree at v26.7.0), because Node's own tests read doc/api, deps/npm and benchmark/ relative to the root. pseudo-tty/ runs inside a pseudo-terminal through pty-spawn.py (a port of tools/pseudo-tty.py), with the test's // Env: line, its .in file as stdin, and a verdict from line-matching the sibling .out file — Node's own criterion — for every runtime alike. test/ffi/ runs with its fixture library compiled (npx node-gyp rebuild). wpt/ — Node's wrappers over its in-tree Web Platform Tests — is enumerated with a 5-minute budget. Node's own pass rate on the host goes from 97.9% to 99.1% (120 → 50 misses); the config grows to 5,278 entries.
There was a problem hiding this comment.
ℹ️ Minor suggestions inline. No correctness blocker — every published figure reconciles against the committed artifacts, and the generator is still byte-for-byte idempotent.
The pty harness is the substantive addition this round, and it holds up: pty-spawn.py is a faithful port of Node's tools/pseudo-tty.py (same openpty/setsid/dup2 sequence, same ONLCR/ECHOCTL termios clearing, same 128 - code signal encoding), and judgePty compares against the sibling .out with the same [WILDCARD]-style escaping Node's runner uses. The three findings below are cosmetic or documentary.
Re-verification performed against the committed artifacts, all clean:
| Check | Result |
|---|---|
Six lens rows in README.md recomputed from results.json |
Exact match |
perDirectory rows for pseudo-tty/, wpt/, ffi/ |
Exact match |
| Node's own 50 self-failures out of 5,664 eligible | Exact match |
nubRegressions = 164 |
Exact match |
node tests/node-compat-regen.mjs re-run vs committed tests/node-compat-config.jsonc |
Byte-identical (5,278 = 5,133 pass + 117 ignore + 28 untriaged) |
meta key order vs the object literal in run.mjs |
Process-written, no hand edit |
Published site figures moved again and are now stale by a wider margin
Not anchorable — neither file is in this diff — but the numbers this PR publishes are the ones the site quotes, and they changed again in 74817a6.
site/src/app/(home)/page.tsx:955-960 (COMPAT) and site/content/blog/introducing-nub.mdx:36 still cite colinhacks/node_test @ node-25.8.1, a 4,367 denominator, and 98.6 / 78.5 / 66.8. The denoExclusions lens in the committed results.json now gives a 5,044 denominator with nub 98.14, deno 74.19, bun 68.12, against reference Node 26.7.0.
I flagged this in rounds 1 and 2 and then deliberately held it back in rounds 3 through 5 because the numbers had not moved. They have now, so the correct replacements are restated above. A follow-up PR touching those two files is the natural home for it.
Claude Opus ｜ 𝕏
…nd; correct two README counts; document pty-spawn's PATH resolution
…e-maps gate clears (11 of 164)
There was a problem hiding this comment.
Important
tests/cross-runtime/results.json is a tracked artifact in a public repository and carries the maintainer's local absolute paths in 56 places. Everything raised in the previous round is correctly addressed.
Reviewed changes — 74817a6 → adb7199, six edits, all closing round-6 findings.
- Dropped the unreachable
pseudo-tty//wpt/classifier rule fromtests/node-compat-regen.mjs. It was dead by construction rather than by coincidence:DIRS(line 30) excludes both prefixes and the filter at line 82 runs beforecategorize(). Verified by regenerating from a cold config with and without the rule re-inserted — byte-identical both ways — and by counting 0 of 144 divergence candidates matching either prefix. - Removed the dead
exec("")operand from the corpus version detector. The survivingh.match(...)[1]still throws into the enclosingcatch { return "?" }on an unparseablesrc/node_version.h. - README timeout count 3 → 5. Exact: five of the fifty node self-failures carry
timeout: true(internet/test-net-autoselectfamily-timeout-close.js,parallel/test-cluster-dgram-1.js,parallel/test-dns-channel-timeout.js,parallel/test-runner-execution-ordered-bypass.mjs,parallel/test-sqlite-backup.mjs). - README source-maps count 8 → 3, which is what the classifier assigns.
pty-spawn.pynow discloses theexecvpePATH resolution alongside the child-status exit; both confirmed in the code (lines 61 and 81).- The NO_COLOR comment in
run.mjsno longer attributes the absence to a decision by Node's runner.
With the source-maps figure corrected, the whole 164 decomposition reconciles against the committed config: 51 permission-model + 17 module-hooks + 14 report/ + 3 source-maps + 4 output-snapshot + 2 layout-artifact, plus the stated tail. The generator remains byte-for-byte idempotent (5,278 entries = 5,133 pass + 117 ignore + 28 untriaged), and the working tree is clean after a regeneration round-trip.
Why six rounds missed the path leak
The finding itself is inline on tests/cross-runtime/run.mjs. What has no line to anchor to is how it survived this long: every prior round searched for /Users/colinmcd94 with the leading slash, got 0, and closed the risk on that result. The leading slash is exactly the character the truncation eats. Searched unanchored, colinmcd94 returns 56.
Census over the committed results.json, spread across roughly 66 test entries and all five runtimes:
| Pattern | Occurrences |
|---|---|
colinmcd94 |
56 |
Documents/projects/nub |
22 |
sers/colinmcd94 |
5 |
ocuments/projects/nub |
4 |
/Users/colinmcd94 — what prior rounds grepped |
0 |
A second variant carries no username at all: 10 occurrences where the repo prefix is what got bisected, leaving trees/node-compat-26-7/tests/node-suite/… — the local worktree naming scheme rather than the home directory. 213 lines match trees/.
Nothing else leaks. Hostnames, IP and MAC addresses, environment dumps, PIDs, certificates and tokens are all clean, as are non-home macOS paths (/private/var/folders, /opt/homebrew, /Volumes — zero each). node-compat-config.jsonc, config.jsonc and engine-specific.txt are clean. Git history is permanent on a public repo, so the artifact is worth regenerating in the same commit as the ordering fix.
The stale published figures in site/src/app/(home)/page.tsx and site/content/blog/introducing-nub.mdx stand exactly as flagged last round. The numbers did not move again in this commit, so I am not restating them.
Claude Opus ｜ 𝕏
…tail, and scrub the nub checkout's root too A tail cut could bisect an absolute path, leaving a suffix the substring replacement no longer recognised; 56 occurrences of the maintainer's home directory and 10 bare worktree paths reached the tracked results.json that way. The affected (file, runtime) pairs were re-run under the fixed order; an unanchored census of the username and checkout names now finds none. Scores and the regression count are unchanged.
There was a problem hiding this comment.
Important
The scrub-before-truncate fix is correct and closes 60 of the 64 leaked path fragments. Four nub records were not re-judged in the accompanying re-run and still carry the maintainer's checkout path, so results.json and the two README sentences describing it are still not in agreement.
Reviewed changes — the two commits since adb7199, verified against the committed artifacts rather than the commit messages.
- Scrubbed before truncating, at all three sites.
tests/cross-runtime/run.mjs:512,:513and:532now readscrub(raw.out.trim()).slice(-400/-300). Line 511 is untouched, correctly — it never truncates. - Added a
<nub>root.scrub()lazily derives the nub checkout frompath.resolve(BINS.nub)via^(.*)/target/[^/]+/nub$and replaces it before theHOMEpass, so a preload frame naming a second checkout is labelled rather than reduced to~/….meta.binaries.nub.binis now<nub>/target/release/nub. - Regenerated the artifact and its downstream config.
results.jsonmoved to19:42:15.672Z;tests/node-compat-config.jsonc:8cites the same stamp. - Restated the source-maps parenthetical as 11 of 164, with its provenance named (measured against the #784 build).
Re-verification against the committed artifacts:
| Check | Result |
|---|---|
Six lens rows in README.md recomputed from results.json |
Exact match; byte-identical to a6ff5470's scores |
nubVsNode.nubRegressions |
164, unchanged |
node tests/node-compat-regen.mjs vs committed config |
Byte-identical (5,278 = 5,133 + 117 + 28) |
meta key order vs the object literal at run.mjs:795-813 |
Process-written, no hand edit |
Unanchored colinmcd94, sers/colinmcd94, Documents/projects/nub, trees/ |
0 each (were 56 / 5 / 22 / 213) |
| Everything else sensitive (hostnames, IPs, ports, temp dirs, key material, non-home absolute paths) | Clean; the one /Users/ is /Users/runner/.cargo/… from Deno's own build machine |
meta.retried.deno moved 1834 → 1833 retried and 3 → 2 flipped; one Deno test passed first-try this round instead of on retry, so its final verdict is unchanged and no lens moved.
⚠️ Four nub tails were never re-judged, so the checkout path survives the fix
Searched unanchored, ocuments/projects/nub still returns 4 in the committed tests/cross-runtime/results.json — the bisected remains of /Users/colinmcd94/Documents/projects/nub/runtime/preload-common.cjs. No username survives, so this is narrower than last round, but it is the same permanent public-repo artifact and it falsifies both sentences this PR just wrote about it (README.md:89 "so a cut cannot bisect a path", README.md:101 "the file carries nothing machine-specific").
The code fix is not at fault. Seventeen sibling records carry the same preload-common.cjs:1336:30 frame correctly scrubbed to <nub>/runtime/…, and 17 + 4 is exactly the 21 bisected copies in the pre-fix file. For each of the four, 780d3fe1 rewrote the bun and deno tails while leaving the nub tail byte-identical to a6ff5470 — --merge (run.mjs:670) keeps the prior verdict for anything not re-run, and the late scrub() at :783-784 cannot repair a string whose prefix was already cut. These four are simply missing from the re-run's list.
Worth noting why: the fragment contains neither colinmcd94 nor Documents nor trees/, so a file list built from those three patterns cannot see it. That is the same instrument failure as last round, one layer down — round 7's census listed ocuments/projects/nub as its own row for exactly this reason.
Technical details
# Four `nub` verdicts retain a bisected checkout path after the scrub-before-truncate fix
## Affected sites
- `tests/cross-runtime/results.json` — the `nub` record of each of:
- `es-module/test-vm-main-context-default-loader-eval.js`
- `es-module/test-vm-main-context-default-loader.js`
- `parallel/test-util-getcallsites-sourcemap.js`
- `parallel/test-vm-dynamic-import-callback-missing-flag.js`
each `tail` begins `ocuments/projects/nub/runtime/preload-common.cjs:1336:30)` and is byte-identical to the same field at `a6ff5470`, while the sibling `bun`/`deno` tails in the same records were rewritten by `780d3fe1`.
- `tests/cross-runtime/README.md:89` and `:101` — both assert the artifact is free of machine-specific content; both are currently false by four occurrences.
## Required outcome
- No fragment of the maintainer's home or checkout path in any tracked artifact, checked with an unanchored pattern that would fire on a bisected prefix.
- The four `nub` verdicts re-judged under the fixed code path rather than the fragments hand-edited, so the artifact stays process-written.
- `tests/node-compat-config.jsonc` regenerated afterwards so its provenance header names the resulting `results.json`.
## Suggested approach
Re-run just those four files for `nub` and merge into the committed `results.json`, then re-run `tests/node-compat-regen.mjs`. Confirm afterwards with the unanchored census, not a leading-slash path:
```sh
for p in colinmcd94 'ocuments/projects/nub' 'sers/colinmcd94' 'trees/'; do
printf '%-24s %s\n' "$p" "$(grep -o -F "$p" tests/cross-runtime/results.json | wc -l)"
done
```
Optional, and the thing that would end this class of finding: have the writer assert on its own output before `writeFileSync` — serialize, then fail the run if the JSON contains `HOME`, `REPO`, `nubRoot`, or the last two path segments of any of them. That turns a leak into a failed run instead of something a reviewer has to notice, and it would have caught these four regardless of which file list the re-run used.
## Open questions for the human
- Is a targeted `--merge` acceptable here, or would you rather land a full fresh run so the whole artifact comes from one invocation of the fixed code?Claude Opus ｜ 𝕏
…ckout path the previous re-run list missed
There was a problem hiding this comment.
Important
The head commit regenerated tests/cross-runtime/results.json but not the prose that quotes it, so five of the six lens rows in the README table no longer match the artifact they document. Separately, the regeneration promoted one test into the must-pass Rust gate on a verdict that tracks how deep the corpus is checked out rather than runtime behavior.
Reviewed 780d3fe1..80a2a98c — 2 files (tests/cross-runtime/results.json, tests/node-compat-config.jsonc). tests/node-compat-config.jsonc is 5,279 entries = 5,134 pass + 117 ignore + 28 untriaged, matching the PR body, and node tests/node-compat-regen.mjs reproduces it byte-for-byte. The score table recomputed from the new results.json is internally consistent; the drift is entirely in the hand-copied prose, flagged inline.
parallel/test-fs-cp-async-socket.mjs is not safe as an unconditional gate entry
The regeneration added "parallel/test-fs-cp-async-socket.mjs": {} to tests/node-compat-config.jsonc, which makes it a must-pass entry for crates/nub-cli/tests/node_compat.rs. It flipped false → true for node, nub and node25 simultaneously, between two runs on the same host with the same binaries — three runtimes do not change behavior in unison, so the verdict is a property of the run, not of the runtimes.
Mechanism, verified against nodejs/node at v26.7.0:
- The test calls
server.listen(sock)with noerrorhandler, wheresock = join(nextdir(), pid + '.sock'). test/common/tmpdir.jscomputestestRoot = NODE_TEST_DIR ? realpath(NODE_TEST_DIR) : resolve(__dirname, '..')andtmpdirName = '.tmp.' + (TEST_SERIAL_ID || TEST_THREAD_ID || '0').TMPDIRis never read.common/fs.js'snextdir()appendscopy_%<n>.- So the socket path is
<checkoutRoot>/test/.tmp.<serialId>/copy_%1/<pid>.sock. Past macOS's 104-bytesockaddr_un.sun_pathlimit, libuv'suv_pipe_bindreturnsUV_EINVAL— notENAMETOOLONG— which matches every recorded tail (errno: -22, syscall: 'listen'). The unhandled error kills the process before the test's actualERR_FS_CP_SOCKETassertion runs.
Arithmetic on the recorded tails: the before-state corpus root is 80 characters, giving 80 + /test(5) + /.tmp.1663(10) + /copy_%1(8) + /37604.sock(11) = 114 > 104. After the merge the tail shows .tmp.66, which is still 112 > 104 — and the shortest path reachable from an 80-character root is 107. Since node/nub/node25 now pass, the merge run must have used a shorter corpus root (the README's own recipe uses /tmp/node-26.7.0). That is the point: the verdict is a function of checkout depth. It also means Deno's residual EINVAL at a short path is a real Deno limitation, not a length artifact.
For the gate specifically: node_compat.rs:68 sets TEST_SERIAL_ID from the worker id (0..16), node_compat.rs:67 sets TMPDIR — which this test ignores — and nothing sets NODE_TEST_DIR. CI's /home/runner/work/nub/nub/tests/node-suite (~42 chars) is comfortably safe, but the repo's documented trees/<name> worktree convention can push a local cargo test over the limit, and it will fail with an EINVAL that says nothing about path length.
Two ways out, either fine: mark the entry untriaged (or excluded) with the reason recorded, or have the gate export NODE_TEST_DIR to a short scratch path so the socket path stops depending on where the repo lives.
results.json is a --merge composite, not a snapshot
run.mjs's overlay is merged[f] = { ...(merged[f] || {}), ...results[f] }, so anything not re-run keeps its prior verdict. The committed file is now a three-generation composite: every lens is recomputed from the merged record, so the percentages are self-consistent, but "nub 98.14%" aggregates verdicts collected across several runs on a machine under varying load. Worth one sentence under "How to read the results" — a reader currently has no way to know the file is not a single run.
The durable fix for the stale table
Five of six rows drifted this round, and a hand-copied figure has now gone stale in three separate rounds. These numbers are all derivable from results.json, so the answer is to generate the README table (or assert it in a test) rather than retype six rows after every merge. README.md:112 already warns that the site figures are hand-copied from this table — the drift propagates outward from here.
✅ Closed since the last round
The path-scrubbing item is resolved. An unanchored census over results.json at 80a2a98 returns 0 for the username, the project directory names, trees/ and the worktree name. The four previously-leaking records now carry <nub>/runtime/preload-common.cjs:1336:30 and are marked "retried": true, i.e. genuinely re-judged rather than hand-edited. The sole surviving /Users/ string is a .cargo/registry path inside a Rust panic emitted by Deno's own release build. README.md:101's claim is now true; that thread is resolved.
ℹ️ Nitpicks
Three figures in wiki/research/node-test-suite-leverage.md moved with the same regeneration — flagged inline.
Claude Opus ｜ 𝕏
…e the merge composite; classify the socket-path test as a gate-environment artifact readme-table.mjs writes the six lens rows between markers and --check fails when they drift, so the figures are never retyped. results.json is described as the --merge composite it is. test-fs-cp-async-socket.mjs binds a Unix socket under the checkout's test/.tmp.<id>/, which a deep checkout pushes past sun_path's 104 bytes; the generator records that as harness-path rather than letting it into the must-pass set (relocating the scratch via NODE_TEST_DIR broke four tests that expect it under the checkout).
There was a problem hiding this comment.
ℹ️ No critical issues — minor suggestions inline.
Both round-9 findings are closed, and I verified that rather than taking the diff's word for it.
Round-9 finding 1 (stale README figures) — closed. node tests/cross-runtime/readme-table.mjs --check exits 0. Recomputing independently from results.json: all six lens rows match, plus the four figures the generator does not cover — nodeFailCount 49 (README:89), nubRegressions 164 (README:91), retry flips node 1 / nub 1 / bun 4 / deno 2 / node25 0 (README:22), and perDirectory for pseudo-tty 28/31, wpt 24/25, ffi 11/13 (README:87).
Round-9 finding 2 (run-shape-dependent gate entry) — closed. A cold regen (--config at a fresh path, which is the only way to defeat the carry-forward branch at node-compat-regen.mjs:89) yields 5279 entries (5133 pass, 116 ignore, 2 carried forward, 30 untriaged) with the socket entry present — so GATE_ENVIRONMENT genuinely fires rather than being inherited from the prior config. The warm regen is byte-identical to the committed file. The reason string's "(a worktree under ~/.cache)" is accurate: .claude/skills/worktree/SKILL.md:34 uses ~/.cache/nub/worktrees/<slug>.
I also checked whether that socket test is the only path-length-sensitive must-pass entry, and it appears to be. The ~9 sibling unix-socket must-pass tests use common.PIPE, which test/common/index.js defines as path.relative(process.cwd(), tmpdir.path) on non-Windows; the gate sets cwd = suite_root (node_compat.rs:49, :63), so those paths stay short regardless of checkout depth. parallel/test-net-pipe-with-long-path.js builds a 10,000-char path deliberately and accepts EINVAL as its expected outcome. Not exhaustive across all ~5,100 entries, but structurally sound.
On measurement integrity under the multi-generation --merge: no second corrupted verdict and no moved lens percentage. Failing tails' .tmp.<N> ids cluster consistently across runtimes, and the one re-run batch (compile-cache tests) is symmetric across nub and bun with nub's failures persisting post-merge inside the disclosed 164 — which is where laundering would show if it were happening. Three inline notes below; none blocking.
Claude Opus ｜ 𝕏
…ate its header from the measured versions; name path depth in the composite note The pre-push hook now runs readme-table.mjs --check whenever the pushed range touches tests/cross-runtime/** (the push refs are read once and replayed to the site gate). The table's header row, with each runtime's measured version, moves inside the generated region so a stale version string fails the check too. The composite-file note names checkout depth as a second source of cross-generation drift beside host load.
There was a problem hiding this comment.
ℹ️ No critical issues — minor suggestions inline.
All three round-10 findings are closed by 4c30e4e0, and I verified each with a positive control rather than reading the diff.
| Control | Result |
|---|---|
readme-table.mjs --check, clean tree |
0 |
corrupt header bun 1.4.0 → 9.9.9 |
1 (was 0 last round — the header is now inside the guard) |
| corrupt a lens-row digit | 1 |
--write round-trip |
byte-identical, idempotent |
pre-push, range touching tests/cross-runtime/, clean |
0 |
| pre-push, same range, corrupted row | 1, with the --write hint |
| pre-push, range not touching it | 0 (skipped) |
PUSH_REFS replay to the site gate (real shas 18a7fd2^..18a7fd2) |
site_touched=1 — the stdin refactor did not break site detection |
same, with lat:check enabled (the real hook path) |
PUSH_REFS intact — npm does not drain stdin |
The stdin single-read is the part most likely to break silently, so I exercised it on the real hook rather than the truncated copy: git push delivers refs once, and the heredoc replay reaches both consumers with the lat gate's npm invocation in between.
I also re-derived every figure from results.json (unchanged since 2d5bdd40, generatedAt 2026-08-22T20:01:01.407Z): six lens rows, nodeFailCount 49, nubRegressions 164, retry flips 1/1/4/2/0, perDirectory ffi 11/13, pseudo-tty 28/31, wpt 24/25, lens Files 5078/4760/5664/4946/4111/718, corpusNodeVersion 26.7.0. Nothing is stale at HEAD.
Reviewed changes since 2d5bdd40:
- Gated the results table in
.githooks/pre-push, scoped to a pushed range touchingtests/cross-runtime/**, with stdin read once and replayed to the site gate by heredoc. - Generated the table header from
meta.binaries, pullingdeno 2.9.5 / bun 1.4.0 / node 25.9.0inside the guarded markers. - Named checkout-path depth in the merge-composite disclosure at README:110.
- Documented
--writein the reproduce recipe at README:41.
Nitpick. The new block's comment says "Like the site gate, an undecidable range runs the check rather than skipping it" — accurate, but it inherits the site gate's documented caveat at :28-31 without repeating it: the check runs against the working tree while the range inspection looks at the pushed commits, so a dirty tree can fail a push whose commits are clean, and vice versa. Worth a word, because unlike the site gate there is no CI backstop here — .github/workflows/ci.yml:1177 removed the corpus job, so this hook is the only enforcement of the generated table.
Claude Opus ｜ 𝕏
| $PUSH_REFS | ||
| PUSHREFS | ||
| if [ "$table_touched" = 1 ]; then | ||
| if ! (cd "$SKILLS_ROOT" && node tests/cross-runtime/readme-table.mjs --check) >&2; then |
There was a problem hiding this comment.
The gate treats every non-zero exit from the checker as drift, and the checker has three non-drift ways to exit non-zero. I probed each:
| Condition | Exit | Message the user gets |
|---|---|---|
| Table drifted | 1 | correct |
node not on PATH |
127 | "does not match results.json" |
results.json unreadable |
1 (uncaught throw) | "does not match results.json" |
| Markers missing from README | 2 | "does not match results.json" |
The PATH case is the realistic one: a GUI git client runs hooks with a non-login PATH, and an nvm-installed node is absent from it — the push is blocked with a message pointing at a file that is perfectly fine. The marker case is worse in kind, because the suggested remedy is also a dead end: --write exits 2 on missing markers too, so following the hint reproduces the failure.
This is the one behavior in the file that departs from its own stated philosophy. :18-20 says the hook "NEVER BLOCKS ON INFRASTRUCTURE. Missing deps, absent pnpm, a failed install — all warn and exit 0"; the lat gate honors it at :166-169 and :198-201, the site gate at :270-282. This block's comment invokes the site gate by name while diverging from it.
Gating on the drift code specifically keeps the guarantee and lets a broken checker announce itself as broken:
| if ! (cd "$SKILLS_ROOT" && node tests/cross-runtime/readme-table.mjs --check) >&2; then | |
| if [ "$table_touched" = 1 ]; then | |
| (cd "$SKILLS_ROOT" && node tests/cross-runtime/readme-table.mjs --check) >&2 | |
| table_rc=$? | |
| if [ "$table_rc" = 1 ]; then | |
| echo "pre-push: tests/cross-runtime/README.md's results table does not match results.json." >&2 | |
| echo " Regenerate it: node tests/cross-runtime/readme-table.mjs --write" >&2 | |
| exit 1 | |
| elif [ "$table_rc" != 0 ]; then | |
| echo "pre-push: results-table check could not run (exit $table_rc); skipping." >&2 | |
| fi | |
| fi |
There was a problem hiding this comment.
Two of the three rows are closed in be58b18 (127 and 2 now warn and pass, 1 still blocks — exercised on the real hook), but an unreadable results.json still exits 1 through readme-table.mjs:16's uncaught throw, so leaving this open; details in the new review.
Claude Opus ｜ 𝕏
…ate the runtime-versions table too
…(exit 2), not drift
There was a problem hiding this comment.
ℹ️ No critical issues — one residual from the previous round and a doc nit, both inline.
Round-11 finding 2 is closed and I resolved that thread. Round-11 finding 1 is two-thirds closed; the one row it did not reach is the inline comment below, so I left that thread open.
Reviewed changes — 4c30e4e0..be58b18e, 3 files. I exercised the real hook and the generator rather than reading the diff.
- Narrowed the table gate to the drift verdict.
.githooks/pre-push:233-243capturestable_rcand blocks only on1; anything else prints "could not run (exit N); skipping" and falls through. The file has noset -e, so thecmd; rc=$?form is safe. - Pulled the versions table under the generator's guard.
readme-table.mjsgainedversionsTable()and aBLOCKSloop, and README:14-22 now carries<!-- versions-table -->markers. - Moved the nub build provenance to prose. README:24 states the release build of
mainat18a7fd2124and links the generator, sincemeta.binariescarries no commit. - Documented the working-tree caveat in the gate's comment, closing round 11's body nitpick.
| Control | Result |
|---|---|
corrupt deno 2.9.5 inside the new versions table |
1 (was uncovered before this commit) |
corrupt the nub row's on Node v26.7.0 |
1 |
| corrupt a lens-row digit | 1 |
--write round-trip |
byte-identical, idempotent |
hook, drift, range touching tests/cross-runtime/ |
1, with the regenerate hint |
hook, node off PATH |
0 — "could not run (exit 127); skipping" |
| hook, markers absent | 0 — "could not run (exit 2); skipping" |
hook, results.json absent or truncated |
1 — blocks, with the drift message (see inline) |
I also re-derived every figure from results.json, which is unchanged since 2d5bdd40 (generatedAt 2026-08-22T20:01:01.407Z): six lens rows, nodeFailCount 49, nubRegressions 164, retry flips 1/1/4/2/0, perDirectory pseudo-tty 28/31, wpt 24/25, ffi 11/13, corpusNodeVersion 26.7.0. tests/node-compat-config.jsonc is 5,279 = 5,133 pass + 118 ignore + 28 untriaged, matching the PR body. The unanchored path census returns 0 for all four patterns, so README:107's claim still holds. Nothing is stale at HEAD.
ℹ️ Nitpicks
- README:24's build sha is the one provenance fact nothing can check. Moving
18a7fd2124out of the generated table was the right call givenmeta.binaries.nubcarries onlyversionandnode— but it leaves the single figure a reader most wants to trust as hand-typed prose, which is the shape this whole gate exists to eliminate. Havingrun.mjsstamp the nub commit intometa.binaries.nubwould let the generator own it; a follow-up, not this PR.
Claude Opus ｜ 𝕏
| import { fileURLToPath } from "node:url"; | ||
|
|
||
| const HERE = path.dirname(fileURLToPath(import.meta.url)); | ||
| const results = JSON.parse(fs.readFileSync(path.join(HERE, "results.json"), "utf8")); |
There was a problem hiding this comment.
This is the row of round 11's table the fix did not reach. An absent or truncated results.json throws out of readFileSync/JSON.parse and exits 1, which the hook's new = 1 branch reads as drift: the push is blocked with "the results table does not match results.json" and pointed at --write, which throws identically. That is the same dead end the markers case had, and .githooks/pre-push:230 now explicitly names "an unreadable results.json" as a case that warns.
A truncated file is reachable — run.mjs writes this path, so a Ctrl-C mid-write leaves invalid JSON and blocks your next push. Exiting 2 puts it in the hook's warn branch, where the comment already claims it is.
| const results = JSON.parse(fs.readFileSync(path.join(HERE, "results.json"), "utf8")); | |
| let results; | |
| try { | |
| results = JSON.parse(fs.readFileSync(path.join(HERE, "results.json"), "utf8")); | |
| } catch (e) { | |
| console.error(`cannot read results.json: ${e.message}`); | |
| process.exit(2); | |
| } |
Technical details
# An unreadable `results.json` still blocks the push as drift
## Affected sites
- `tests/cross-runtime/readme-table.mjs:16` — uncaught `ENOENT`/`SyntaxError` exits 1, indistinguishable from the drift verdict.
- `.githooks/pre-push:230` — the comment lists "an unreadable results.json" among the cases that warn rather than block; it does not.
- `.githooks/pre-push:238` — the remedy printed in this case is `--write`, which exits 1 the same way.
## Required outcome
- Exit 1 means drift and nothing else. Every non-drift failure of the checker exits non-1 so the hook's warn branch handles it.
- Measured, not reasoned: run the hook with `results.json` moved aside, and again with it truncated to `{"meta":`, and confirm both exit 0 with the "could not run" line.| // node tests/cross-runtime/readme-table.mjs --write # rewrite the rows in README.md | ||
| // | ||
| // The table — header with the measured runtime versions, then one row per | ||
| // lens — lives between the `<!-- results-table -->` markers in README.md. |
There was a problem hiding this comment.
The header comment still describes a single table between the <!-- results-table --> markers, but the file now renders two blocks and --check/--write operate on both. Worth naming the versions table here, since this comment is the only place the second marker pair is documented.
Moves both Node-compat measurement systems to Node 26.7.0, run the way Node's own runner runs it.
tests/cross-runtime: corpus = a full Node v26.7.0 checkout (tests/node-suite);pseudo-tty/under a pty with.outjudging;ffi/with its compiled fixture;wpt/enumerated; Deno's currentconfig.jsonc; lenses (Deno's, bun.com's, full, each minus a 718-test engine-specific class); bun receives// Flags:; symmetric retry;--merge; machine paths scrubbed before truncation; README table generated and checked fromresults.json.tests/node-compat-config.jsonc: regenerated — 5,279 entries: 5,133 pass, 118 classified divergences, 28 untriaged (was 2,554 / 102).// Flags:withNODE_SKIP_FLAG_CHECK=1from the suite root, per-workerTEST_SERIAL_ID, concurrent stderr drain. Gate: 5,133 / 5,133.