tokenizer: pre-size vocabulary for sparse added-token IDs - #13

Draft

pd95 wants to merge 3 commits into

fix/gemma4-tool-call-safetyfrom

fix/tokenizer-sparse-added-tokens

Commits on Sep 18, 2026

fix(tokenizer): bound sparse tokenizer IDs

pd95
and
codex
committed
5f773a5
tokenizer: bound added token IDs during load

pd95
and
codex
committed
c272b36
test(tokenizer): clarify added token limit scope

pd95
and
codex
committed
fe0227a