[feat](kt-kernel): --kt-mmap-experts-dir to keep CPU expert weights in a file, not anonymous RAM

[tim-odonnell]

What

--kt-mmap-experts-dir <dir>: keep the CPU-resident per-expert BufferB weight
blocks (the repacked/quantized gate/up/down matrices) in a MAP_SHARED file
instead of anonymous heap.

The CPU MoE kernels allocate one std::aligned_alloc(64, ~5 MiB) block per
CPU-resident expert matrix. For a large MoE that is tens of GiB of anonymous
memory per model — the OOM killer's target, reclaimable only via swap. Those
blocks are written once during weight load and read-only afterwards.

With the flag set, each block becomes a MAP_SHARED slice of one file per
(layer, NUMA part) under <dir>. File-backed pages are clean once written, so
the kernel reclaims a cold expert's pages without swap and refaults them from
the file (NVMe) on the rare cold route. A host whose CPU expert working set is
close to total RAM can then serve the model.

Unset (the default) → unchanged std::aligned_alloc path, zero overhead.

This is a lightweight complement to #1421 (SSD offload request) and MESH (#2003):
no io_uring, no residency manager — the kernel page cache does the tiering. Useful
as the "just don't OOM" floor.

How

• operators/kt_weight_arena.hpp (new): KtWeightArena — open() creates and
  sizes the backing file; alloc(n, &file_backed) returns a 2 MiB-rounded
  mmap(MAP_SHARED) slice, or falls back to std::aligned_alloc on any error
  (setting file_backed = false); madvise(MADV_RANDOM); numa_tonode_memory to
  the constructing thread's NUMA node so first-touch stays local on multi-socket;
  the destructor munmaps every slice and closes the fd.
• GeneralMOEConfig::mmap_weights_dir (operators/common.hpp) + pybind
  (ext_bindings.cpp), next to the existing path / save / load.
• AMX_MOE_BASE and AVX2_MOE_BASE each get a KtWeightArena bb_arena_
  member; init() opens it once (no-op when the dir is empty) and routes the 3
  per-expert BufferB allocs through bb_arena_.alloc(). AMX_MOE_BASE never
  frees BufferB itself (non-owning views), so the arena's destructor is the only
  cleanup; AVX2_MOE_BASE keeps freeing its owned_aligned_allocs_ and simply
  does not add file-backed pointers to that list.
• Python: KTMoEWrapper(..., mmap_experts_dir=...) →
  _create_inference_wrapper sets it as an attribute on the returned wrapper (so
  no backend __init__ signature changes) → each backend's load_weights()
  copies self.mmap_experts_dir into MOEConfig.mmap_weights_dir.
  BaseMoEWrapper defaults the attribute to "". SFT path untouched.

The matching sglang-side flag (--kt-mmap-experts-dir in server_args.py, carried
on KTConfig, passed to the inference KTMoEWrapper) is a companion PR against
kvcache-ai/sglang.

Testing

Measured on 1× CMP-170HX (GA100, sm_80) + Ryzen 9 9900X (single NUMA node) +
128 GB DDR5-5600, DeepSeek-V4-Flash-0731 native MXFP4/FP8 via SGLang + kt-kernel,
--kt-method MXFP4 --kt-num-gpu-experts 80, ctx 32k, fp8 KV, CUDA graphs on:

metric	anonymous (default)	--kt-mmap-experts-dir
decode, steady tok/s	13.1	13.0 (12.77 / 12.74 / 12.70 over 3×300 tok)
VRAM	57,634 MiB	57,632 MiB
CUDA-graph capture	4.2 s / 0.14 GB	4.2 s / 0.14 GB
scheduler process anon RSS	~105 GiB	2.4 GiB (rest Private_Clean, file-backed; Swap: 0)
outputs (greedy)	—	bit-identical
mmap fallbacks	—	0 across ~22.7k allocations; 43 arenas

On this box the model does not otherwise fit in 128 GB. The first decode pass
after a cold start is ~30 % slower while the routed experts' pages fault in from
the file; steady state is unchanged. A future MADV_WILLNEED / warm-on-load could
remove that.

The AVX2 path and multi-socket numa_tonode_memory behaviour are written to
mirror the AMX path but I don't have that hardware to exercise them — review welcome
there in particular.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KW36fR7syvcTaG7Yqqq8pP