Skip to content

Fix Kimi tokenizer and CUDA payload loading - #96

Merged
yyj6666667 merged 5 commits into
kvcache-ai:mainfrom
yyj6666667:fix/kimi-tiktoken-guidance-20260912
Sep 13, 2026
Merged

yyj6666667 merged 5 commits into
kvcache-ai:mainfrom
yyj6666667:fix/kimi-tiktoken-guidance-20260912

Conversation

@yyj6666667

@yyj6666667 yyj6666667 commented Sep 12, 2026

Copy link
Copy Markdown

What does this PR do?

  • Use llguidance's native tiktoken support for Kimi tokenizers; other models keep the existing dispatch.
  • Resolve lazy native binaries using their wheel-relative paths, preserving bundled library lookup. Legacy manifests remain supported.

Two production files, +24/-3 lines, with two focused test files. The packaging counterpart is kvcache-ai/ktransformers#2204.

Tests

  • Focused tokenizer/payload tests: 6 passed in a fresh wheel-only environment.
  • Original Kimi tokenizer: multilingual token IDs, EOS, vocabulary, and JSON grammar acceptance/rejection passed.
  • Fresh native wheels: dependency / architecture audit and eight concurrent cold imports passed.
  • Clean training/serving installation passed. Qwen two-GPU, Kimi four-GPU, and public FP8 GLM-5.3-Flash eight-GPU generation and JSON-schema checks passed.
  • Kimi loaded both ordinary and expert LoRA from a freshly resumed training checkpoint; all converted tensors passed exact comparison. Full Neko style validation remains pending.

No model weights, attention kernels, or training runtime changes. Not a release sign-off.

Keep the Hugging Face tokenizer path unchanged for other models. Preserve Kimi vocabulary, EOS and special token IDs without disabling grammar support.
Allow an explicit binary map in new carrier manifests while retaining legacy manifests. Verify ELF loading with a real sibling-library RPATH fixture and ensure flat extraction fails the negative control.
@yyj6666667 yyj6666667 changed the title Fix Kimi tokenizer support in llguidance Fix Kimi tokenizer and CUDA payload loading Sep 12, 2026
@yyj6666667
yyj6666667 merged commit 541ddc3 into kvcache-ai:main Sep 13, 2026
56 of 65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant