perf(validators): replace tr46 with a generated UTS-46 table - #117
Merged
Conversation
|
Total Coverage: 87.97% Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Total Coverage: 87.88% Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
tr46 ships an explicit mapping target for every Unicode code point, which
costs ~230 KB in a consumer's bundle. The SDK uses exactly one of its
entry points, `toUnicode(name, { useSTD3ASCIIRules: true })`, to normalize
domain names.
Almost every one of those mapping targets is reproducible from NFKC plus
context-free case folding, both of which the runtime already implements
natively. So rather than shipping the table, generate and ship only what
cannot be derived:
- run-length encoded ranges of code points UTS-46 disallows
- run-length encoded ranges of combining marks, which UTS-46 allows
inside a label but not at its start
- 526 code points whose mapping differs from NFKC + case folding
That is 11.7 KB of source instead of 225 KB, plus an inline RFC 3492
punycode decoder for `xn--` labels.
Three details are worth calling out, because each was a real bug during
development:
- Case folding must be applied one code point at a time. JavaScript's
`String#toLowerCase` applies the Greek final-sigma rule, so 'ΣΣ'
becomes 'σς'; UTS-46 maps Σ to σ unconditionally.
- Code-point status must be probed in context. Testing a bare combining
mark reports an error that belongs to the label rule, not to the code
point, which would wrongly reject names like "Á".
- The leading-mark rule applies to the mapped label. UTS-46 ignores
variation selectors and the soft hyphen, so a mark that followed one
only becomes label-leading after mapping.
`scripts/generate-idna-table.js` regenerates the table from tr46;
`scripts/verify-idna.js` checks the two implementations against each
other over 3,120,097 inputs (every code point, a multi-label corpus, the
non-ASCII names actually registered on mainnet, and 2,000,000 seeded
fuzz strings). Current result: 0 mismatches. `test/domain/idna.test.ts`
pins the cases that matter for the registry so a regression fails fast
without needing tr46 installed.
Bundled with esbuild (--bundle --minify --platform=node), the published
entry point drops from 1,642,792 to 1,425,317 bytes.
Claude-Session: https://claude.ai/code/session_01GceWCwGXu66D1xEBDxZWBb
yeboster
force-pushed
the
perf/native-idna
branch
from
August 28, 2026 20:24
833d4cc to
ff54eff
Compare
|
Total Coverage: 87.85% Coverage Report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Removes the
tr46runtime dependency and replaces it with a small generated table plus the Unicode operations the runtime already provides.tr46ships an explicit mapping target for every Unicode code point (~230 KB in a consumer's bundle). The SDK uses exactly one entry point from it:toUnicode(name, { useSTD3ASCIIRules: true })inDomainValidator#normalize.Almost all of those mapping targets are reproducible from NFKC + context-free case folding. So
src/validators/idnaderives the common case natively and consults a generated table only for what cannot be derived:DISALLOWED_RANGES— 776 rangesLEADING_MARK_RANGES— 309 rangesMAPPING_EXCEPTIONS— 526 code pointssrc/validators/idna/table.tsPlus an inline RFC 3492 punycode decoder for
xn--labels.Bundle impact
esbuild
--bundle --minify --platform=nodeon the published entry point:mainOwn code grows 43,890 → 58,741 bytes (the table); the dependency side drops ~230 KB.
Note: a second copy of
tr46(260 KB) is still pulled in transitively bynode-fetch. That one disappears with theperf/native-fetchPR, which removesnode-fetchentirely.Correctness
This is a security-relevant path — IDNA mapping decides which names are considered equal — so it is verified rather than spot-checked.
scripts/verify-idna.jsruns the new implementation againsttr46over:test/domain/idna.test.tsis the committed fast regression suite; it pins the registry-relevant cases so a break fails CI without needingtr46installed.Three subtleties, each a real bug caught during development and now covered by tests:
'ΣΣ'.toLowerCase()is'σς'(Greek final-sigma context); UTS-46 maps Σ to σ unconditionally."Á"."︆̈"must be rejected).Also:
。,．and｡are UTS-46 label separators and are folded to.before splitting.Regenerating
tr46is no longer a dependency, so both scripts install it ad hoc:Worth re-running on a Unicode version bump.
Test status
npx tsc --noEmitclean,npx eslint srcclean,npx jest -i— 21 suites / 269 tests, all passing.(Under parallel
jest, the live-testnet mint suites flake independently of this change; they are green serially, and were flaky on unmodifiedmaintoo.)Note on
tsconfig.jsonThe
excludelist gainscoverage,dist,docs,scripts—allowJs: trueotherwise pullsscripts/*.jsand stale coverage output into the program. The same line is touched by thechore/security-hardeningPR; the resulting content is identical, so the merge is clean either way.https://claude.ai/code/session_01GceWCwGXu66D1xEBDxZWBb