feat(k3): map prepared weights read-only - #965
Open
bherald wants to merge 3 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add an opt-in, CPU-only
K3_MMAP=1path for Kimi K3 containers whose dense,MLA, shared-expert, and head tensors are already stored in their final U8 data
aligned OS view and Windows mapping handle required for exact cleanup;
instead of copying them into private heap;
mapping failures instead of silently falling back; and
backend refusal contract, and Windows file/directory cleanup.
This is the focused
st_map+ K3 vertical slice discussed in #826. It does notimplement
TRUNK_RESIDENT_LAYERS, prefetching, page locking, or an OS working-set policy.
Field result
On a Ryzen 9 7845HX / 64 GB Windows host using the official Kimi K3 text model,
int4/MLA8/Head8, a 1-GiB expert cache, 1K context,
K3_VK=0, andK3_CUDA=0,the mapped path reported 1,906 prepared views / 33.8 GB file-backed. Mapping
reduced peak private memory from the prior heap-copy path's roughly 35.36 GB to
about 7.32 GB.
Mapping alone did not bound the active working set. The host therefore applied
its own verified hard working-set maximum to only the exact K3 engine; that
limit is external to this PR. Three isolated 26-token smoke repeats at 16 GiB
completed in 72.945-73.892 seconds with 2,126-2,578 MiB pagefile growth and
zero final K3 engines.
A follow-up exposed an important limit. This runtime uses
K3_CHUNK=32; promptsabove 32 total chat tokens require another full 93-layer prefill traversal. A
40-token synthetic prompt repeatedly crossed the unchanged 3,072-MiB pagefile
guard even with a 19-GiB host cap. Five concise 31-32-token synthetic cases did
complete safely at 19 GiB in 86.491-92.282 seconds with 1,759-1,881 MiB
pagefile growth and zero final K3 engines, but only four returned the expected
token. The NoFallback case returned an unexpected token. These results show
that prepared read-only mapping makes host-managed residency possible inside a
narrow single-prefill envelope; they are not a production-readiness,
throughput, broad-quality, or full-model-equivalence claim.
Validation
make -C c checkmake -C c test-asanWindows skips
hash and clean final process state
the expected token and the mismatch failed closed
or Vulkan backends
measurements above
Compatibility
included
K3_MMAPdefaults off and changes no existing loader behaviorCloses no issue; contributes the mapped-residency slice toward #826 while
leaving the broader
TRUNK_RESIDENT_LAYERSdesign open.