Skip to content

Prepare clean Kimi LoRA release tooling - #2203

Closed
yyj6666667 wants to merge 5 commits into
kvcache-ai:mainfrom
yyj6666667:fix/kimi-clean-release-20260912
Closed

yyj6666667 wants to merge 5 commits into
kvcache-ai:mainfrom
yyj6666667:fix/kimi-clean-release-20260912

Conversation

@yyj6666667

@yyj6666667 yyj6666667 commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Prepare a source-built Kimi LoRA release without checkout-only tools or private runtime patches.

  • Package the existing LoRA converter and deterministic Kimi/NekoQA preprocessing as CLI tools; retain source-script entry points.
  • Document packing-disabled training and full optimizer checkpoint resume.
  • Audit all six KT CPU variants separately from CUDA architecture coverage, validate top-level wheel metadata, and use all five release carriers.
  • Add hash-checked, dependency-only PEFT/TRL repackaging; runtime files and dependency bounds remain unchanged.
  • Preserve wheel-relative dynamic-library paths in the lazy CUDA archive; require matching loader support from SGLang main.
  • Round Kimi RAWINT4 scratch allocations to 64 bytes, preventing misaligned access with odd expert counts.

No optimizer or training math is changed; the native change is confined to Kimi scratch-buffer alignment. LF packaging changes live in a separate user-approved fork, not this PR.

Validation

  • Release tooling: 68 CPU tests passed.
  • Data preparation and converter: 19 CPU tests passed, including fixed public NekoQA data. Five model-directory integration tests were skipped locally.
  • Separate sap4 source probe: 138,850 converted ordinary/expert LoRA tensors exactly match the existing input adapter.
  • SGLang CPU probe: real ELF loading reproduces the flattened-layout failure and passes with wheel-relative paths; legacy payload behavior remains covered.
  • Source-built KT candidate wheel: 24 numerical tests passed with no skips across AMX / AVX512-BF16 and TP1 / TP2, covering RAWINT4 and INT8 forward, input/router/LoRA gradients, accumulation, scaling and frozen base weights. The unfixed main wheel fails both odd-expert cases and passes both even-expert controls.
  • Independent LF standard-profile clean install and KT-profile install/upgrade: 46 dependency/dispatch tests pass in each. All 9,945 prepared train/eval label sequences match the original Kimi template. Tiny non-KT CPU training/resume controls reproduce adapter and optimizer state exactly in both environments; this is not Kimi GPU acceptance.
  • Installed converter: 138,850 ordinary/expert tensors exactly match the preserved older adapter; inputs are unchanged. This is not a newly trained adapter or generation acceptance.
  • Full CUDA source build passes ABI, dynamic-linkage and SASS audits, retaining SM80/86/89/90/120 support and the separate SM90 object. Five-carrier assembly is complete; each wheel is below 104 MB.
  • Two fresh environments install the assembled serving / serving+SFT candidates through normal pip dependency resolution. pip check, native imports and comparison against original wheel bytes pass; adding LF/PEFT/TRL leaves all five core packages unchanged. Exact SHA-locked dependency closures also pass pip dry-run.
  • Final assembled combined environment repeats all 46 LF and 24 native numerical tests with zero skips, plus exact tiny non-KT optimizer resume. The actual installed SGLang tokenizer factory preserves native Kimi encoding and accepts/rejects JSON-schema inputs correctly on CPU.
  • These are PR-source engineering candidates, not official-main release artifacts. New Kimi GPU training → optimizer resume → SGLang generation, quality and cross-model GPU regressions remain pending. Occupied GPUs are not interrupted.

Draft until these gates pass. This PR does not publish packages. SGLang's Kimi tokenizer compatibility fix is tracked in kvcache-ai/sglang#96.

Move the existing adapter converter into an installable tools package without changing conversion math. Add deterministic NekoQA preparation and candidate save/resume instructions; keep packing disabled.
Require all six KT CPU variants, distinguish CUDA SASS from host-only libraries, and reserve Accelerate payload capacity. Validate top-level distribution metadata and preserve PEFT/TRL runtime files when remapping approved dependency names. Keep pinned Torch-provided CUDA libraries external during auditwheel repair.
Archive CUDA objects under their original wheel-relative paths and include repaired shared libraries for cache extraction. Keep installed dependency copies for directly loaded extensions and require matching main loader support.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant