Register the seven H3 control tokens the vocabulary omits

[Tatlatat]

Problem

MiniMax declares seven H3-specific control tokens in tokenizer_config.json under additional_special_tokens:

<d>  </d>  <|cutoff|>  <|lyrics_start|>  <|lyrics_end|>  <|caption_start|>  <|caption_end|>

They are in neither the BPE vocabulary nor tokenizer.json's added_tokens array, which stops at 151668. h3_tokenizer_load() only reads the latter, so it never learns them and the BPE merges swallow the markup.

The damage reaches past the tag itself. Against the released FL2VA/tokenizer, <d>[English] Hello there.</d> currently encodes as nine pieces:

90707 30768 22574 60 21927 1052 3918 67 29

90707 fuses <d>[ into one token and 3918 fuses .</, so the dialogue marker and the [English] language tag beside it both reach the DiT as ordinary prose. <|cutoff|> becomes six pieces, <|caption_start|> six.

This matters because the official prompting guides (base, ref) build every dialogue line, cross-cut continuation and truncation on exactly these markers.

Fix

Continue the identifiers from the highest declared added token, which is what the reference tokenizer does. For the released vocabulary that yields 151669-151675, comfortably inside the 151936 TEXT_VOCAB embedding bound. Tokens already present in tokenizer.json are skipped, so a future release that declares them properly keeps its own ids rather than getting duplicates.

Verification

transformers 5.16.1 loading the same FL2VA/tokenizer directory:

<d> -> 151669   </d> -> 151670   <|cutoff|> -> 151671   <|lyrics_start|> -> 151672
<|lyrics_end|> -> 151673   <|caption_start|> -> 151674   <|caption_end|> -> 151675

'<d>[English] Hello there.</d>' -> [151669, 58, 22574, 60, 21927, 1052, 13, 151670]

h3.c after this patch, same input, same file:

<d>[English] Hello there.</d>      ->  8 ids: 151669 58 22574 60 21927 1052 13 151670
<|cutoff|>                         ->  1 ids: 151671
<|caption_start|>                  ->  1 ids: 151674
<|im_start|>                       ->  1 ids: 151644          (unchanged)
A red fox walking through snow     ->  6 ids: 32 2518 38835 11435 1526 11794   (unchanged)

tests/test_tokenizer.c gains the three cases above, including the decode round-trip that check_case already performs. make h3_tokenizer_tests passes against both the FL2VA and the Ref2VA tokenizer (83 checks, was 60), and ./h3_tests still reports its 1768 checks. Builds clean under the existing -Wall -Wextra -Wpedantic -Wshadow -Wconversion.

One thing I did not do is an A/B of actual generations. I checked the embedding rows at 151669-151675 and they are statistically indistinguishable from the unused padding rows above them (norm 0.505 +/- 0.006 against 0.503 +/- 0.005 for rows 151676-151935, while the 26 genuinely declared special tokens average 0.662 with a 1.34 maximum), so these embeddings look frozen at initialization rather than trained as language-model tokens. That does not make the mapping wrong, since the text encoder is frozen and the DiT trained on top of whatever fixed vectors it produced, and it is what the reference pipeline feeds. It does mean the argument here is "match the reference tokenizer", not "measured better video".