-
Notifications
You must be signed in to change notification settings - Fork 0
Normalization
ezicode implements the four Unicode normalization forms (NFC, NFD, NFKC, NFKD) defined in UAX #15. There are two execution modes: a batch API that allocates a single output buffer, and a streaming Normalizer that produces output incrementally with zero allocations.
This is the most subtle subsystem in the library. Read this page if you intend to use normalization on text you don't fully control.
| Form | Decomposes | Composes | Compatibility |
|---|---|---|---|
| NFC | Canonical | Yes | No |
| NFD | Canonical | No | No |
| NFKC | Canonical + compatibility | Yes | Yes |
| NFKD | Canonical + compatibility | No | Yes |
In practice: use NFC for storing and comparing text. Use NFD when you want to strip diacritics or analyze base+marks separately. Use NFKC when you want fi (U+FB01) to compare equal to fi, and ① to compare equal to 1. NFKD is the same idea but without recomposing.
Most applications want NFC. If you don't know which form you want, you want NFC.
const ezicode = @import("ezicode");
const norm = ezicode.unicode.normalization;
const input = "café"; // can be either precomposed or decomposed
const nfc = try norm.normalize(allocator, .nfc, input);
defer allocator.free(nfc);
const nfd = try norm.normalize(allocator, .nfd, input);
defer allocator.free(nfd);
The batch API takes an allocator, allocates one output buffer sized to fit the result, and returns it. It's the right choice when you have the full string in hand and don't care about peak memory.
A quick-check fast path runs first. If the input is already in the requested form (verified by walking *_Quick_Check properties and combining-class adjacency in one pass), normalize returns a copy without doing any reordering or composition work. For text that's already normalized, this is essentially the cost of a memcpy.
The streaming API is for when you can't or don't want to materialize the full string. It maintains a fixed-size internal buffer and emits output incrementally as you feed in codepoints.
var normalizer = norm.Normalizer(.nfc){};
for (input_codepoints) |cp| {
const emitted = normalizer.feed(cp);
// emitted is a []const u21 slice of 0..MAX_FEED_OUTPUT codepoints
for (emitted) |out_cp| {
// write out_cp somewhere
}
}
const final = normalizer.finalize();
// emit any remaining buffered codepoints
The Normalizer does not allocate. Output is returned as a slice into an internal buffer, valid until the next call to feed or finalize.
Two constants matter:
-
MAX_DECOMP_LEN = 32: the internal buffer for accumulating combining marks within a single canonical run. -
MAX_FEED_OUTPUT = 64: the maximum number of codepoints any singlefeedcall will emit.
These are sized to match the Stream-Safe Text Format limit defined in UAX #15 §13, with headroom for compatibility-decomposition expansion.
This is the part that catches people. Read it carefully.
Streaming normalization can only be done losslessly on input that conforms to Stream-Safe Text Format: the standard's recommendation that no canonical sequence of non-starters between two starters exceeds 30 codepoints. Real text essentially never violates this. Zalgo-style adversarial text, where someone has stacked hundreds of combining marks on one base character, does violate it.
ezicode's streaming Normalizer makes two contracts:
For Stream-Safe input: the streaming output is byte-identical to the batch output. Feed any codepoint sequence whose non-starter runs are ≤30 codepoints and Normalizer.feed plus finalize produces exactly the same bytes as normalize(.nfc, ...) would.
For non-Stream-Safe input: the streaming output is well-formed but not byte-identical to batch. Specifically: it terminates, never overflows the internal buffer, never emits more than MAX_FEED_OUTPUT per call, and emits a sequence where combining classes within each canonical run are non-decreasing. It does NOT guarantee that all 150 marks on a single base get reordered together, because doing so would require unbounded buffer space — which is exactly what UAX #15 §13 says streaming cannot guarantee.
If you need lossless normalization of arbitrary adversarial input, use the batch API. If you need streaming and your input might be hostile, either pre-validate that it conforms to Stream-Safe (by checking non-starter run lengths) or accept that streaming output may not match batch output on the hostile cases.
This is documented as part of the streaming contract, with corresponding tests in the conformance suite. The library does not silently degrade.
For all four forms, normalize(form, normalize(form, x)) is byte-equal to normalize(form, x). This is asserted in the conformance suite for every row of NormalizationTest.txt and for every codepoint in isolation.
The Unicode standard defines a set of characters that decompose canonically but are not allowed to be recomposed during NFC (the Composition Exclusion list, derived from DerivedNormalizationProps.txt). ezicode honors this list. If you pass a character in the exclusion list through NFC, its decomposition is reordered canonically but not recomposed back into the original.
This matches the spec and is what NormalizationTest.txt expects.
Rough numbers on the benchmark corpus:
- Quick-check fast path on already-NFC text: bounded by memcpy speed.
- Batch NFC on text that needs work: ~225 MiB/s on hot caches. The bottleneck is the canonical reorder for combining-mark runs.
- Streaming NFC: similar bandwidth on Stream-Safe input. Adversarial Zalgo input degrades, but bounded by
MAX_FEED_OUTPUTper call. - Memory: batch allocates
O(input_size * expansion_factor)once. Streaming allocates zero.
A few failure modes worth knowing about:
-
Passing invalid UTF-8 to a function that expects valid input. ezicode does not validate UTF-8 internally; if you're working from raw bytes, validate first with
ezicode.encoding.utf8.validate. Functions on[]const u21codepoint slices assume the input is valid scalars. -
Out of memory in batch. The batch API returns
error.OutOfMemoryif allocation fails. It does not have a fallback. - Streaming non-equality to batch on hostile input. Covered above. Not a bug. If a test asserts streaming equals batch for a non-Stream-Safe input, the test is wrong, not the library.
ezicode implements the four Unicode normalization forms (NFC, NFD, NFKC, NFKD) defined in UAX #15. There are two execution modes: a batch API that allocates a single output buffer, and a streaming Normalizer that produces output incrementally with zero allocations.
This is the most subtle subsystem in the library. Read this page if you intend to use normalization on text you don't fully control.
| Form | Decomposes | Composes | Compatibility |
|---|---|---|---|
| NFC | Canonical | Yes | No |
| NFD | Canonical | No | No |
| NFKC | Canonical + compatibility | Yes | Yes |
| NFKD | Canonical + compatibility | No | Yes |
In practice: use NFC for storing and comparing text. Use NFD when you want to strip diacritics or analyze base+marks separately. Use NFKC when you want fi (U+FB01) to compare equal to fi, and ① to compare equal to 1. NFKD is the same idea but without recomposing.
Most applications want NFC. If you don't know which form you want, you want NFC.
const ezicode = @import("ezicode");
const norm = ezicode.unicode.normalization;
const input = "café"; // can be either precomposed or decomposed
const nfc = try norm.normalize(allocator, .nfc, input);
defer allocator.free(nfc);
const nfd = try norm.normalize(allocator, .nfd, input);
defer allocator.free(nfd);The batch API takes an allocator, allocates one output buffer sized to fit the result, and returns it. It's the right choice when you have the full string in hand and don't care about peak memory.
A quick-check fast path runs first. If the input is already in the requested form (verified by walking *_Quick_Check properties and combining-class adjacency in one pass), normalize returns a copy without doing any reordering or composition work. For text that's already normalized, this is essentially the cost of a memcpy.
The streaming API is for when you can't or don't want to materialize the full string. It maintains a fixed-size internal buffer and emits output incrementally as you feed in codepoints.
var normalizer = norm.Normalizer(.nfc){};
for (input_codepoints) |cp| {
const emitted = normalizer.feed(cp);
// emitted is a []const u21 slice of 0..MAX_FEED_OUTPUT codepoints
for (emitted) |out_cp| {
// write out_cp somewhere
}
}
const final = normalizer.finalize();
// emit any remaining buffered codepointsThe Normalizer does not allocate. Output is returned as a slice into an internal buffer, valid until the next call to feed or finalize.
Two constants matter:
-
MAX_DECOMP_LEN = 32: the internal buffer for accumulating combining marks within a single canonical run. -
MAX_FEED_OUTPUT = 64: the maximum number of codepoints any singlefeedcall will emit.
These are sized to match the [Stream-Safe Text Format](https://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format) limit defined in UAX #15 §13, with headroom for compatibility-decomposition expansion.
This is the part that catches people. Read it carefully.
Streaming normalization can only be done losslessly on input that conforms to Stream-Safe Text Format: the standard's recommendation that no canonical sequence of non-starters between two starters exceeds 30 codepoints. Real text essentially never violates this. Zalgo-style adversarial text, where someone has stacked hundreds of combining marks on one base character, does violate it.
ezicode's streaming Normalizer makes two contracts:
For Stream-Safe input: the streaming output is byte-identical to the batch output. Feed any codepoint sequence whose non-starter runs are ≤30 codepoints and Normalizer.feed plus finalize produces exactly the same bytes as normalize(.nfc, ...) would.
For non-Stream-Safe input: the streaming output is well-formed but not byte-identical to batch. Specifically: it terminates, never overflows the internal buffer, never emits more than MAX_FEED_OUTPUT per call, and emits a sequence where combining classes within each canonical run are non-decreasing. It does NOT guarantee that all 150 marks on a single base get reordered together, because doing so would require unbounded buffer space — which is exactly what UAX #15 §13 says streaming cannot guarantee.
If you need lossless normalization of arbitrary adversarial input, use the batch API. If you need streaming and your input might be hostile, either pre-validate that it conforms to Stream-Safe (by checking non-starter run lengths) or accept that streaming output may not match batch output on the hostile cases.
This is documented as part of the streaming contract, with corresponding tests in the conformance suite. The library does not silently degrade.
For all four forms, normalize(form, normalize(form, x)) is byte-equal to normalize(form, x). This is asserted in the conformance suite for every row of NormalizationTest.txt and for every codepoint in isolation.
The Unicode standard defines a set of characters that decompose canonically but are not allowed to be recomposed during NFC (the Composition Exclusion list, derived from DerivedNormalizationProps.txt). ezicode honors this list. If you pass a character in the exclusion list through NFC, its decomposition is reordered canonically but not recomposed back into the original.
This matches the spec and is what NormalizationTest.txt expects.
Rough numbers on the benchmark corpus (see Performance for the full table):
- Quick-check fast path on already-NFC text: bounded by memcpy speed.
- Batch NFC on text that needs work: ~225 MiB/s on hot caches. The bottleneck is the canonical reorder for combining-mark runs.
- Streaming NFC: similar bandwidth on Stream-Safe input. Adversarial Zalgo input degrades, but bounded by
MAX_FEED_OUTPUTper call. - Memory: batch allocates
O(input_size * expansion_factor)once. Streaming allocates zero.
A few failure modes worth knowing about:
-
Passing invalid UTF-8 to a function that expects valid input. ezicode does not validate UTF-8 internally; if you're working from raw bytes, validate first with
ezicode.encoding.utf8.validate. Functions on[]const u21codepoint slices assume the input is valid scalars. -
Out of memory in batch. The batch API returns
error.OutOfMemoryif allocation fails. It does not have a fallback. - Streaming non-equality to batch on hostile input. Covered above. Not a bug. If a test asserts streaming equals batch for a non-Stream-Safe input, the test is wrong, not the library.