Skip to content

Add hunyuan-dense so Hy-MT2 GGUFs can run - #201

Open
sdcb wants to merge 1 commit into
zhongkaifu:mainfrom
sdcb:feature/hunyuan-dense
Open

Add hunyuan-dense so Hy-MT2 GGUFs can run#201
sdcb wants to merge 1 commit into
zhongkaifu:mainfrom
sdcb:feature/hunyuan-dense

Conversation

@sdcb

@sdcb sdcb commented Sep 10, 2026

Copy link
Copy Markdown

Summary

  • Register hunyuan-dense (Hy-MT2 / Hunyuan dense text) through the architecture + chat-protocol seams. Official Q4 GGUFs previously failed at load with Unsupported architecture.
  • Forward matches llama.cpp's hunyuan-vl text graph: RMSNorm → QKV → NeoX RoPE → per-head Q/K RMSNorm → GQA → SwiGLU. QK-norm is after RoPE (the opposite of Qwen 3.5).
  • Chat template follows tencent/Hy-MT2-1.8B chat_template.jinja (BOS + add_generation_prompt), not llama.cpp's Hunyuan-4B-Instruct hardcoded template, which dropped BOS and produced off-task replies.
  • First cut is SingleDevice: no TP, no fused whole-model graph, no continuous batch. CPU ggml is slower than llama.cpp on this path; correctness was checked on Hy-MT2-1.8B-Q4_K_M (Hello你好; official translation prompt produces Chinese).

Test plan

  • HunyuanDenseChatTemplateTests (BOS, system, multi-turn, no-tools protocol)
  • TensorParallelSupportGateTests / SkillCapabilityConsistencyTests still pass with the new registration
  • --backend cpu / ggml_cpu load + translate on Hy-MT2-1.8B-Q4_K_M.gguf
  • GPU path (ggml_cuda / cuda) — not exercised on the authoring machine
  • Long-context / chunked prefill (model advertises 262k; KV grows on demand)

The architecture was unregistered, so official Q4 files failed at load.
Land the text graph (NeoX RoPE then QK-norm) and the Hy-MT2 chat template.
First cut is single-device; no TP or fused whole-model path yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant