qwen4exp: prefetch requested lazy PLE pages (upstream reference)

[GenerelSchwerz]

Overview

Lazy Qwen PLE gathers can wait on serial faults for sparse rows. Advise the pages for the current batch's known row indices before graph execution, preserving the existing mmap, GET_ROWS, and arithmetic. The mmap helper checks lazy ranges, file bounds, page crossings, and retained fragments; it merges requested pages in fixed 256-range chunks. Unsupported platforms use a no-op fallback. No worker pool, direct-reader mode, or generic GET_ROWS scheduling change is included.

Owner-authorized fork reference for a possible later manual upstream submission. This draft is based entirely on pristine upstream history, with no MoE-cache or partial-pinning dependencies.

• Pristine upstream and fork PR base: df03399b885831b2a1603b3abb0d8c156808e363.
• Head: b5fd33bc9145ac7eefd056e8b7c6049c0b847acc.
• Intended later upstream target: ggml-org/llama.cpp:master, refreshed and reviewed by the owner before submission.
• Related alternative: qwen4exp: direct reads for the lazy PLE table (>2x prefill performance improvement on GB10) ggml-org/llama.cpp#28136 (explicit buffered reads; still open when checked).

Validation and limitations

CPU Release build passed. Existing IQ4_NL GET_ROWS tests passed 8/8. Standalone checks using the changed mmap implementation passed ASan/UBSan for bounds, page crossings, duplicate ranges, removed fragments, advice failure, and compile-time no-op fallback.

A bounded real-shard experiment using upstream IQ4_NL dequantization measured mean cold gather time of 4.13 -> 1.98 ms for 16 rows and 250.1 -> 22.7 ms for 1792 rows. Warm 16-row time increased from 4.9 to 11.8 us. F32 bytes matched on identical row indices in all 72 batches. Peak RSS was 167 MiB. Cold arms used separate nonresident row sets, selected with mincore without cache eviction; the experiment overlapped CPU compilation. These are exploratory gather measurements, not full-model throughput results. Advice itself can block and accounted for most remaining cold latency.

A CUDA full-model A/B used Qwen3.8 Flash Next UD-Q3_K_XL with --load-mode none --lazy-mode on, 4K context, 512/256 batches, Q8_0 KV, Flash Attention, 12 CPU threads, automatic offload, and a 1024 MiB fit margin. Each binary processed the same 800-token prompt twice and generated exactly 1024 tokens per request. All four generated token-ID arrays and response texts matched exactly.

First-request baseline -> patched results were 206.01 -> 258.55 prompt tok/s and 21.758 -> 23.129 decode tok/s. Immediate second-request results were 236.37 -> 296.74 prompt tok/s and 22.984 -> 22.746 decode tok/s. Across both requests, prompt time fell 20.33%, decode time fell 2.53%, and combined prompt plus decode time fell 3.84%. Peak GPU use was 14825 MiB. The 8 GiB available-memory guard did not fire, both servers stopped cleanly, and both ports closed.

Only the PLE byte range was selectively evicted before each server; no global cache flush was used. Minimum available host memory was 8.71-8.96 GiB, and PLE residency remained low, so the immediate second request is not proven fully page-warm. This is one baseline-then-patched server pair, not a statistical or sustained-pressure benchmark. Windows currently takes the no-op fallback. Advice bounds requested I/O and scratch, but does not cap accumulated OS page-cache residency.

Local reproducibility artifacts: /home/gencoolpc/llama-ple-prefetch-upstream/build-ple-validation/REPORT.md, with exact commands, baseline source archive, test harnesses, request/response JSON, token hashes, memory samples, residency snapshots, raw CSV, and logs. No new tests/* files.

Requirements

• Contributing guidelines reviewed under the owner's fork automation exception; the owner remains responsible for reviewing, understanding, and maintaining the change.
• AI usage disclosure: YES. Codex implemented the owner-approved design, ran local checks, and prepared this fork reference commit and PR. Any upstream submission will be handled personally by the owner.