tokenizer: pre-size vocabulary for sparse added-token IDs

[pd95]

Motivation

Hugging Face tokenizer files may assign added tokens IDs far above the base
vocabulary and leave gaps between IDs. Ollama stores vocabulary entries in an
ID-indexed slice. Growing that slice one token at a time causes repeated
allocation and copying, while trusting an unchecked high ID can trigger an
excessive allocation. Duplicate IDs can also overwrite an existing token
silently.

The complete required extent is knowable before insertion and should be
validated before memory is allocated.

What this changes

• scans base and added-token IDs to determine the final vocabulary extent;
• allocates the ID-indexed vocabulary once instead of repeatedly growing it;
• preserves intentional holes in sparse token ranges;
• detects collisions rather than overwriting an existing token;
• rejects negative, overflowing, or unreasonably large added-token IDs;
• applies the same bounded handling while tokenizer data is loaded during
  model creation;
• adds table-driven coverage for sparse ranges, collisions, limits, and normal
  dense vocabularies.

Dependency and scope

This generic tokenizer change depends on Gemma parser-safety PR #12. It does
not introduce a tokenizer format or model-family-specific behavior.

Compare:
fix/gemma4-tool-call-safety...fix/tokenizer-sparse-added-tokens

Validation

The complete x/tokenizer package passes with the expanded loader tests,
together with the integrated Linux test/lint/build and Darwin MLX build gates.

Refresh and AI assistance

Reconstructed on Ollama v0.34.0; head
e893cdbc39691edc1ccac63d7caa21e135fb7fb6. Codex materially assisted
implementation review, tests, refresh validation, and this PR description.