glm5.2: preserve input marking in prompt tokenization (fix output stalls on stray message tags)

[mp4]

1. Fixed MTMD ordering.
   2&3. Assistant continuation content marked as template text, now handles all specialize parsers.(incl gpt-oss)
2. Token-count endpoints do not use prompt_parts
3. Independent part tokenization changes ordinary tokenization
   common_tokenize_parts() (chat.cpp) now has a fast path: common_chat_parts_have_special_input() (mirroring the tokenizer's own parse_special=false rule for control/unknown tokens) checks whether any is_input part contains special-token text; if not, the whole concatenated prompt is tokenized in a single parse_special=true pass — token IDs identical to legacy. The protection path additionally merges adjacent same-type parts (the jinja runtime merges them at render, but continuation parts appended later didn't) so normal merges across post-render template boundaries are kept.
   Verified: test_normal_prompt_token_ids_unchanged — real multi-turn Qwen3 prompt: parts tokenization == legacy whole-prompt tokenization; plus a synthetic BPE case ("Hel"|"lo" straddling a boundary) that provably fails under naive per-part tokenization and passes here, and the same for the protection path.
4. Remove common/arg.cpp.patch

[Anbeeld]

Please retarget this PR from main to v0.4.7 and rebase or refresh the implementation against that branch.

I re-audited the resulting patch against v0.4.7. It applies without textual conflicts, but the following issues still need to be addressed before merge:

1. MTMD bypasses the new protection. In handle_completions_impl, the mctx != nullptr branch runs before the prompt_parts branch. process_mtmd_prompt() tokenizes the flattened prompt with parse_special=true, so OAI chat requests served by a multimodal-capable model do not receive the intended protection, even when no media is attached.

2. Assistant continuation content is marked as template text. The continuation paths construct a string containing reasoning_content and/or message content, then append the whole string as {false, generation_prompt}. This causes request-provided continuation text to be tokenized with parse_special=true. Please preserve provenance separately for template delimiters and continuation content in the automatic and specialized parser paths.

3. GPT-OSS tokenizes different text from prompt. The GPT-OSS handler replaces the final <|return|> with <|end|> in prompt, but does not make the same replacement in prompt_parts. Since the server tokenizes prompt_parts, the existing compatibility fix is bypassed.

4. Token-count endpoints do not use prompt_parts. handle_count_tokens still tokenizes the flattened prompt with parse_special=true. Its result can differ from actual completion tokenization, and it retains the same input-special-token problem. Please share the provenance-aware tokenization path between inference and token counting.

5. Independent part tokenization can change ordinary tokenization. common_tokenize_parts() tokenizes each Jinja part separately, which prevents normal tokenizer merges across template/input boundaries. Please preserve the existing token IDs for prompts that do not contain injected special-token text, and add a regression test demonstrating that behavior.

6. Remove common/arg.cpp.patch. It is unrelated to prompt tokenization, is unapplied and stale, and contains trailing whitespace that makes git diff --check fail.

Please also add focused regression tests covering:

• special-token text inside ordinary user/tool input;
• operation with an MTMD context;
• assistant continuation content;
• the GPT-OSS <|return|> replacement;
• consistency between completion and token-count paths;
• unchanged token IDs for a normal prompt without injected special tokens.

Once the PR targets v0.4.7 and these paths are covered, I can re-review it.

[mp4]

1. Fixed MTMD ordering.
   2&3. Assistant continuation content marked as template text, now handles all specialize parsers.(incl gpt-oss)
2. Token-count endpoints do not use prompt_parts
3. Independent part tokenization changes ordinary tokenization
   common_tokenize_parts() (chat.cpp) now has a fast path: common_chat_parts_have_special_input() (mirroring the tokenizer's own parse_special=false rule for control/unknown tokens) checks whether any is_input part contains special-token text; if not, the whole concatenated prompt is tokenized in a single parse_special=true pass — token IDs identical to legacy. The protection path additionally merges adjacent same-type parts (the jinja runtime merges them at render, but continuation parts appended later didn't) so normal merges across post-render template boundaries are kept.
   Verified: test_normal_prompt_token_ids_unchanged — real multi-turn Qwen3 prompt: parts tokenization == legacy whole-prompt tokenization; plus a synthetic BPE case ("Hel"|"lo" straddling a boundary) that provably fails under naive per-part tokenization and passes here, and the same for the protection path.
4. Remove common/arg.cpp.patch

[Anbeeld]

Thanks!