test(tokenizer): add gigatoken parity experiment

[Kaonael]

Overview

Adds an experimental Gigatoken token-ID parity comparison alongside Dynamo’s existing tokenizer implementations.

This is a demo-only experiment, not a production integration or DEP. Gigatoken remains outside Dynamo’s production workspace and runtime dependency graph because its current PyO3 dependency conflicts with the version introduced
through AIConfigurator.

Details

• Adds a standalone, nightly-only Rust parity harness for:
  • HuggingFace tokenizers
  • Fastokens
  • Gigatoken
• Verifies exact token-ID parity for the same tokenizer and input.
• Leaves existing router tokenizer benchmarks unchanged.
• Does not add Gigatoken to the router runtime or production dependencies.

Local batch benchmark

For a local batch performance sanity check, prompts were extracted from the public Glint-Research/Fable-5-traces dataset.

The corpus contained 1,000 separate user prompts:

• Input size: 6,927,686 bytes (6.93 MB)
• Output: 2,082,390 token IDs
• Tokenizer: Qwen3-14B tokenizer.json

The temporary benchmark preserved prompt boundaries: it did not concatenate prompts into a single document. Every backend received the same prompt sequence and batch boundaries (1, 8, 32, and 128).

The parity comparison passed for all tokenizer × batch-size configurations.

fn run_batches<F>(docs: &[String], batch_size: usize, mut encode: F) -> Vec<u32>
where
    F: FnMut(&[&str]) -> Vec<u32>,
{
    let mut ids = Vec::new();

    for batch in docs.chunks(batch_size) {
        let texts: Vec<&str> = batch.iter().map(String::as_str).collect();
        ids.extend(encode(&texts));
    }

    ids
}

fn benchmark<F>(docs: &[String], batch_size: usize, encode: F) -> Vec<u32>
where
    F: FnMut(&[&str]) -> Vec<u32>,
{
    let mut encode = encode;

    // Excluded from the throughput measurement.
    let _ = run_batches(docs, batch_size, &mut encode);

    let start = std::time::Instant::now();
    let ids = run_batches(docs, batch_size, &mut encode);
    println!("elapsed: {:.3?}", start.elapsed());

    ids
}

Each backend uses the same docs and batch_size. HuggingFace and Fastokens use
their respective encode_batch APIs; Gigatoken uses encode_docs_ragged. The
resulting flattened token-ID vectors are compared for exact parity.

Benchmark environment

• CPU: 2 × Intel(R) Xeon(R) Platinum 8480+

  • 56 physical cores per socket
  • SMT enabled
  • 224 logical CPUs total

• Memory: 2.0 TiB

• OS: Linux 6.8.0-100-generic (x86_64)

• Toolchain:

  • rustc 1.96.0-nightly (48cc71ee8 2026-03-31)
  • cargo 1.96.0-nightly (888f67534 2026-03-30)

• CPU affinity: each benchmark process ran through taskset -c 0-55, pinning it to the 56 physical cores of the first NUMA socket.

• CPU isolation was not configured: other host processes could still run on these CPUs.

• Backend-internal parallelism was left enabled.

Each backend × batch-size configuration ran in a separate process three times:

• Throughput is the median of the three measured passes.

• Peak RSS is the maximum observed resident-set size across the three processes.

• A warm-up pass occurred before the measured pass and was excluded from throughput timing.

  Batch size	HuggingFace throughput (Mtok/s)	HuggingFace peak RSS (MiB)	Fastokens throughput (Mtok/s)	Fastokens peak RSS (MiB)	Gigatoken throughput (Mtok/s)	Gigatoken peak RSS (MiB)
  1	0.69	141.8	11.72	198.5	68.49	118.0
  8	3.13	141.8	17.59	198.5	63.63	118.0
  32	3.74	144.3	32.27	198.5	71.74	118.0
  128	3.59	195.0	43.52	198.5	69.86	118.0

Peak RSS is process-level memory rather than library-only heap usage. It includes the tokenizer, input corpus, warm-up, tokenizer working buffers, and the output token-ID buffer.

The benchmark harness, downloaded dataset files, and build artifacts were temporary local artifacts and are not included in this PR.

Where should the reviewer start?

• benchmarks/tokenizer-comparison/src/main.rs
• benchmarks/tokenizer-comparison/Cargo.toml
• benchmarks/tokenizer-comparison/README.md

Related Issues

🚫 This PR is NOT linked to an issue:

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information.