Proposal: optional expert cache warmup — pre-warm page cache + hot experts before the first request

[outtodata]

Proposal: optional expert cache warmup — pre-warm page cache + hot experts before the first request

TL;DR. On disk-bound machines the first user request is doomed to be slow: the page cache is empty and every expert slice comes from disk. We measured ~12 ms per cold expert slice vs ~0.64 ms warm (OLMoE-1B-7B, Ryzen 7 5700U, Windows), and tok/s volatility across identical runs (3.5 → 2.0 → 1.4) is fully explained by page-cache state, not code. We propose an opt-in MOE_WARMUP=1 (or a separate olmoe-warmup utility) with three levels, all OFF by default.

Data behind the proposal

1. Expert popularity (OLMoE, 800 prompts, 5 domains): per-layer top-32/64 experts cover ~75% of activation slots; the global hot zone is weak (top-256/1024 = 52%), but per-layer coverage is enough for prewarm.
2. Temporal locality: Jaccard of adjacent tokens (union of top-8 × 16 layers) = 0.828.
3. Cold start is not a predictor problem (recall at pos<8 = 0.850) — the problem is the disk.
4. Honest cold numbers (unbuffered, 64 KB-aligned): 3 GB/s aggregate disk, i5 slice = 4.7 MB → full warmup of the hot set (top-32/layer × 16 layers) ≈ 2.4 GB ≈ seconds on a quiet disk, tens of seconds under load.

Proposed levels (all opt-in, OFF by default)

1. Touch pass: sequential read of the weight file into page cache after startup, in the background, not blocking the first request (streaming read, low priority).
2. Preload hot experts: per-layer top-K (K=32) from popularity statistics into the LFRU cache before the first request. Covers ~75% of first-session activations.
3. Idle decode: 1 warmup token at startup — warms allocators, SIMD paths, thread pool.

Style requirements (per the review gates in #906)

• OFF by default, loud when on: a [WARMUP] loaded N slices (X MB) in Ys log line.
• Bit-identical when OFF — the flag does not touch inference at all.
• Warmup volume cap (MB, default = sane fraction of RAM) — on 8–16 GB machines a full 5.4 GB touch would evict everything else from page cache.

Open questions for the maintainer

• Popularity warmup needs statistics — ship a top-K table in the repo/converter, or compute on-the-fly from the first N requests?
• I/O priority for the touch pass: Windows has FILE_FLAG/QoS, POSIX has IOPRIO_CLASS_IDLE. What level is acceptable?

Related

• Degradation policy for prefetch misses: zero-fill beats substitution (measured) #865 (DEGRADE_ZERO), feat(moe): add DEGRADE_ZERO miss-slot zero-fill policy (issue #865) #906 — same line of work: the cost of a late expert read.
• Our i5 numbers (HQQ5-g32, snapshot −25%): warmup gets even cheaper there — see the separate design proposal if the direction is interesting.