test(tokenizer): add gigatoken parity experiment

benchmarks/tokenizer-comparison/Cargo.toml

@@ -0,0 +1,19 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+[package]
+name = "dynamo-tokenizer-comparison"
+version = "0.1.0"
+edition = "2024"
+publish = false
+
+# Keep this experiment independent from Dynamo's workspace and its production
+# dependency graph. Gigatoken currently requires nightly Rust.
+[workspace]
+
+[dependencies]
+fastokens = "0.2.0"
+gigatoken_rs = { package = "gigatoken", git = "https://github.com/marcelroed/gigatoken.git", rev = "0d9765fa7312af7534535e6315a5c49d74807b2a" }
+hf-hub = { version = "0.4.3", default-features = false, features = ["rustls-tls", "ureq"] }
+serde_json = "1"
+tokenizers = { version = "0.21.4", default-features = false, features = ["onig", "esaxx_fast", "rustls-tls"] }

benchmarks/tokenizer-comparison/README.md

@@ -0,0 +1,77 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# Tokenizer parity experiment
+
+This standalone Rust experiment compares HuggingFace `tokenizers`, Fastokens,
+and Gigatoken without adding Gigatoken to Dynamo's production workspace or
+runtime dependency graph. It requires nightly Rust because Gigatoken currently
+does.
+
+The simple and input modes validate exact flat token-ID parity:
+
+```bash
+cd benchmarks/tokenizer-comparison
+cargo +nightly -Zprofile-rustflags run -- \
+  --tokenizer /path/to/tokenizer.json \
+  --simple
+```
+
+```bash
+cargo +nightly -Zprofile-rustflags run -- \
+  --tokenizer /path/to/tokenizer.json \
+  --input /path/to/corpus.txt \
+  --documents 8
+```
+
+`--documents` splits the UTF-8 input at newline or character boundaries. Every
+backend receives the same resulting documents.
+
+## Shared dataset benchmark
+
+`--dataset` reuses the dataset loading, batch boundaries, token-ID parity
+checks, and reporting harness from
+`lib/llm/benches/tokenizer_dataset_support.rs`. The in-workspace
+`tokenizer_dataset` benchmark uses that same harness for HuggingFace and
+Fastokens; this standalone runner adds Gigatoken as a third backend.
+
+Run the LongBench-v2 scenario used by Dynamo's dataset bench:
+
+```bash
+cargo +nightly -Zprofile-rustflags run --release -- \
+  --tokenizer /path/to/tokenizer.json \
+  --dataset zai-org/LongBench-v2
+```
+
+Run its batched variant:
+
+```bash
+cargo +nightly -Zprofile-rustflags run --release -- \
+  --tokenizer /path/to/tokenizer.json \
+  --dataset zai-org/LongBench-v2 \
+  --max-samples 503 \
+  --batch-size 64
+```
+
+
+
+To measure one backend memory without constructing the other two, pass both
+`--backend` and `--memory`:
+
+```bash
+cargo +nightly -Zprofile-rustflags run --release -- --tokenizer /path/to/tokenizer.json --dataset zai-org/LongBench-v2 --max-samples 503 --batch-size 64 --backend gigatoken --memory
+```
+
+This reports Linux process `VmRSS` and `VmHWM` after dataset loading, tokenizer
+initialization, warmup, and the benchmark. Deltas use the post-dataset snapshot
+as their baseline; they are process-level measurements, not per-allocation
+attribution.
+
+Supported datasets and their extraction rules match Dynamo's existing bench:
+
+- `zai-org/LongBench-v2`: the `context` field from `data.json`.
+- `RyokoAI/ShareGPT52K`: formatted conversation turns from `sg_90k_part1.json`.
+
+The shared harness warms every backend once, measures each backend separately,
+checks per-document token-ID parity, and reports total time, average latency,
+throughput, and speedup relative to HuggingFace.

benchmarks/tokenizer-comparison/fable5-affinity-results.md

@@ -0,0 +1,12 @@
+# Fable-5 affinity benchmark results
+
+<text>
+
+| Batch size | HuggingFace throughput (Mtok/s) | HuggingFace peak RSS (MiB) | Fastokens throughput (Mtok/s) | Fastokens peak RSS (MiB) | Gigatoken throughput (Mtok/s) | Gigatoken peak RSS (MiB) |
+|---|---:|---:|---:|---:|---:|---:|
+| 1 | 0.69 | 141.8 | 11.72 | 198.5 | 68.49 | 118.0 |
+| 8 | 3.13 | 141.8 | 17.59 | 198.5 | 63.63 | 118.0 |
+| 32 | 3.74 | 144.3 | 32.27 | 198.5 | 71.74 | 118.0 |
+| 128 | 3.59 | 195.0 | 43.52 | 198.5 | 69.86 | 118.0 |
+
+</text>

benchmarks/tokenizer-comparison/longbench-memory-results.md

@@ -0,0 +1,17 @@
+# LongBench-v2 isolated memory results
+
+<text>
+
+| Mode | Backend | Dataset baseline RSS (GiB) | Tokenizer init ΔRSS (GiB) | RSS after benchmark (GiB) | ΔRSS after benchmark (GiB) | Peak process RSS / VmHWM (GiB) | Throughput (MB/s) |
+|---|---|---:|---:|---:|---:|---:|---:|
+| Sequential | HuggingFace | 0.43 | +0.13 | 0.91 | +0.47 | 2.21 | 1.66 |
+| Sequential | Fastokens | 0.43 | +0.15 | 0.95 | +0.51 | 1.27 | 94.97 |
+| Sequential | Gigatoken | 0.43 | +0.09 | 0.77 | +0.34 | 1.27 | 216.83 |
+| Batch=64 | HuggingFace | 0.43 | +0.13 | 18.16 | +17.73 | 19.64 | 5.67 |
+| Batch=64 | Fastokens | 0.43 | +0.15 | 2.81 | +2.38 | 2.97 | 255.06 |
+| Batch=64 | Gigatoken | 0.43 | +0.09 | 1.56 | +1.12 | 1.60 | 718.23 |
+
+</text>
+
+LongBench-v2, 503 contexts / 446.09 MB, Qwen3-14B tokenizer, `taskset -c 0-55`.
+The dataset baseline is the current `VmRSS` after loading the extracted samples. The process high-water mark can retain the higher transient RSS from JSON parsing; this is why the sequential Fastokens and Gigatoken peak remains 1.27 GiB.