Conversation
…ok-ahead design
…kahead Wire the consumer half of docs/moe-lookahead-design.md: a new --moe-lookahead N gate (default 0, env LLAMA_ARG_MOE_LOOKAHEAD) that asks the CUDA MoE cache to prefetch the next layer's expert weights during the current layer's compute. The prefetch path mirrors acquire with is_prefetch=true, wait_for_compute=false and pin=false: no stream synchronize, no compute-stream wait, pinned slots are skipped, expert ids outside [0, n_experts) are ignored, and prefetch telemetry is accounted separately from demand. Host-to-device copies are batched through cudaMemcpyBatchAsync when CUDART >= 12.80, with a per-copy fallback otherwise. Targets whose buffer is not MoE-cached are skipped with a one-shot warning. --moe-lookahead 0 leaves the decode step unchanged. A non-zero width without --moe-expert-cache-size > 0 fails loudly at model load. Adds test-moe-cache --lookahead-prefetch-only and the lookahead prefetch legacy layer case. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
The speculative prefetch path picked its victim with plain LRU, so a prediction could displace an expert that demand had already proven hot. The eviction guard the design requires (docs/moe-lookahead-design.md, "Invariants to preserve" #2 and the colibri safety invariant at c:1462-1478) was missing: only the pin guard - never evict a slot a running GEMM is reading - was implemented, on both the demand and speculative paths. Port colibri's PILOT_EVICT_GUARD. A speculative fill may displace a resident only when that resident is not genuinely warm; a resident is protected when it has at least 2 demand accesses AND is clearly hotter than the prediction, by the 25% + 4-frequency hysteresis in LFRU score units (score ported from colibri c/tier.h:40-43: frequency in the high bits, recency saturating in the low byte). A blocked prediction is dropped rather than forced in, so it can never thrash a demand-loaded expert. Heat is recorded on demand accesses only. Counting predictions would let a speculation inflate the very score that decides whether it may displace a resident, which is the failure colibri hit in ggml-org#490. Refactor the eviction core so the demand and speculative paths cannot drift: ggml_cuda_moe_cache_select_victim_locked picks the LRU unpinned slot and applies the guard for speculative fills, and ggml_cuda_moe_cache_install_fill_locked records the eviction telemetry and publishes the new entry. The copy-batching split in the prefetch path is unchanged, and the free-slot, pin and unknown-eid behaviour is identical. Add phase_prefetch_dropped so a guard that drops everything (colibri ggml-org#490) is visible rather than silent, and extend the test prefetch accessor with it. Add the lookahead prefetch eviction guard case: a 3-slot pool, a warm resident survives a prediction for a never-seen expert (dropped, no copy, no eviction, still a demand hit), while a once-demanded resident is evicted for it. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…kahead Predict layer il+1's top-k experts from layer il's post-attention state and page them while layer il still computes, so the H2D copy overlaps compute instead of stalling the next layer. Adds GGML_OP_MOE_PREFETCH (an explicit no-op outside CUDA), the graph-level producer, the per-context gate and a reserve-time pool preinstall, plus a debug-only recall instrument. Measured on the rig (Qwen3.8-Flash-Next-APEX-I-Mini, -b 1 -ub 1 so every ubatch is a decode row, --moe-expert-cache-size 84, look-ahead width 8): - decode 37.6 t/s against 43.2 t/s with the feature off, about -13%, because the ids readback synchronizes the stream and forces use_cuda_graph = false. - the MoE grouped-decode certificate rejects the graph unless GGML_OP_MOE_PREFETCH is excluded from the backend use counts, otherwise the run dies with "graph=unproven(14)" and a failed graph compute. - logits are not preserved: PPL 2.9434 against 3.0613, and byte-different dumped logits. Isolation shows the extra MUL_MAT that reads ffn_gate_inp is the trigger; a neutral extra node does not reproduce it, disabling CUDA graphs does not reproduce it, and the prefetch op itself contributes nothing. - every prediction is dropped in this configuration. A pool can only be installed while its group is under legacy cache authority with admission open, and the groups here are not, so preinstall_legacy_pools() fails for all 144 targets and the look-ahead has nothing to page into. Not landable as is. The review PR carries the full isolation matrix. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…roducer Adds a measured-outcome section to the design note: the three blockers with their numbers, the isolation matrix that pins the trigger to the extra MUL_MAT reading ffn_gate_inp, Q1 answered negatively (the authority regime refuses the cross-layer pool install), the throughput cost of the ids readback, the already-recorded byte-identical claim refuted, and the future direction (early-router copy worker plus device-to-host mailbox) if the track is ever re-approached. Also corrects ffn_gate_up_exps being null, the preinstall point, Q3 and Q4. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
Recall was the one unmeasured number in the park decision. Measured it properly: the horizon is one MoE layer (never a token), and the demand path cannot score it because during decode only blk.47 reaches the host-visible ids read - the rest run in the certified grouped path. Ground truth was therefore built in-graph from each layer's own router matmul on its real FFN input. Width 2/4/6/8/10 gives 97/95/91/86/79% recall of the predicted set against 19/38/55/69/79% coverage of the 10 experts the model actually uses; width 8 is about 22x chance. RTX 3060 reproduces width 8 within noise (85.85% / 68.68%). Also records the two traps that had kept this unmeasured: the committed instrument's GGML_LOG_INFO never reaches the server log at default verbosity, and cached-path scoring only ever sees the one layer holding a legacy lease. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…rence The design cites colibri's 71.6% PILOT recall on GLM-5.2 as the expectation for this mechanism. Records that the producer measures 91.50% at width 6 and 86.21% at width 8, with the explicit caveat that the models and expert counts differ so the comparison is not like-for-like. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…design invariant preinstall_legacy_pools returned -1 for all 144 targets and every prediction was dropped. Recording the ruling so it is not re-litigated: the acquire_legacy_cache new-record latch (authority == LEGACY && !admission_closed) binds pool creation to a certified execution, and an unauthenticated install would let a caller claim VRAM pools certification never proved - the overcommit class README.md:25 warns about. L+1 is never the authoritative group while L runs, so cross-layer install is unreachable by design, not broken. If the track reopens the seam is authority publication at graph reserve/certification time, which is a certification-contract change needing its own review. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…evice-side) Reviewer ruling on how blocker 3 would be fixed if the track reopens. Mailbox approved as the transport (it reuses the shipped early-router copy worker plus the async paging half); the FreeToken-style device-side gather is rejected because it needs device-resident slot management and would be a second cache implementation, reintroducing the demand/speculative drift that select_victim_locked and install_fill_locked eliminated. Deleting the use_cuda_graph=false rule alone does not suffice: the op is capture-incompatible as written (pageable D2H memcpy, stream sync, host-side slot booking), and a captured cuStreamWaitValue32 bakes an expected value, so it must be refreshed per step or replaced by a polling kernel. Minimal change set, required guards, and the unchanged sequencing (census exclusion #128 -> authority publication -> transport) are recorded. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
Two env-gated diagnostics, off by default, kept for troubleshooting the look-ahead track: - GGML_CUDA_MOE_PHASE_PROBE=1 attributes each step's GPU time to op classes (attention ops, the MoE expert path, the per-layer embedding read, dense compute, plumbing), separates decode from prefill by inter-step wall time, reports the dispatch mode that produced each step, and prints a per-layer table. One CUDA event pair per maximal run of same-(phase, layer) nodes, read back once per step after a single stream sync (~2.6% of decode throughput on the RTX 3060). - GGML_CUDA_MOE_LOOKAHEAD_DEBUG=1 scores the recorded prediction against the ids the router actually selected, for every MUL_MAT_ID node (the ids are in src[2]), i.e. off the legacy blk.47 lease, and reports recall and precision plus the used-not-predicted population. The other two populations are cache counters that were counted but never printed; they now appear on the moe-cache-phase line as prefetch_dropped and evicted_prefetched_unused. Both probes turn themselves off with a single stderr note if an event or a memcpy cannot be issued, which is what happens under CUDA graph capture, so a diagnostic cannot break a capture run; during replay the host never visits the dispatch loop either, so an attribution needs GGML_CUDA_DISABLE_GRAPHS=1. With the env vars unset the only added work is a cached getenv lookup behind a static bool. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…sults Instrument A on the RTX 3060 at ctx 81920 / cache 42: a decode step is 69% MoE execution, 25% dense compute, 1.5% attention ops and 0% per-layer embedding, with the phase rows summing to the step time; prefill is 88% MoE. In the same window SM sits at 95-99% and the DRAM controller at 19-26%, the PCIe link is gen4 x4 carrying 337.6 MiB of expert traffic per token at ~47% of its ceiling, and the grouped telemetry reports every staged copy already complete, so the limiter is MoE kernel execution rather than PCIe or VRAM bandwidth. Instrument B, scoring predictions in the dispatch loop instead of the legacy blk.47 lease: 64.5% of the used experts were predicted and 80.6% of the predictions were used, the same order as the earlier in-graph measurement. The other two populations are structurally zero in this configuration because the consumer is inert (decode phase line ops=0, legacy cache authority printed once), so every prediction is simply unused. Also records the independent finding that the rig's -ot per_layer_token_embd=CPU is redundant and inert. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
Owner
Author
|
Measured addendum from the same 3060 run, because it changes what the consumer is worth: During prefill the legacy lease path installs 122,327 speculative slabs, uses 4,720 of Two consequences:
Blocker status is unchanged: the producer is still not landable. |
Adds the staged look-ahead fill (PR-B) end to end. The producer's predicted ids are filtered on the device to predicted-and-not-resident, published from a device kernel into a cudaHostAllocMapped mailbox, and DMA'd into the target group's staging lane by the existing early-router copy worker. The gather consumes staging bytes when the expert's position is ready and falls back to the host slab otherwise, so a late or absent copy degrades the speculation instead of failing it. The completion handshake writes the flag with a 4-byte H2D copy of exactly 1. The previous cudaMemset(..., 1, 4) writes 0x01010101, which never matches the EQ 1 wait, so the first publish stalled the stream forever: GPU idle, two host threads spinning, /health alive, SIGTERM ignored. The set now fails fast when the worker is gone, so a dead worker cannot wedge the compute stream again. Also: - device-side lane counters (staged bytes and count, consumed, publishes, dropped, resident-skipped), drained and emitted every 16 dispatches; - mmid branch counters and per-node phase attribution in ggml-cuda.cu, env gated and inert when unset; - deterministic expert-id and stride modes for the mul_mat_id sweep in test-backend-ops, so the stride/ids matrix is reproducible rather than dependent on the random shuffle; - design-document updates and a new residency-model note. Gate-off cost on the staging path is one atomic load; no D2H copy and no host synchronization are added to the decode graph. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
The look-ahead predicted layer il+1's experts immediately after layer il's attention and issued the staging DMA there, on the stated intent that the H2D would overlap "this layer's MoE compute". The MoE bucket is not compute-bound: of its 81.04 ms, roughly 77 ms is the demand gather reading expert host memory over the same PCIe link the staging transfer needs. A transfer issued alongside the gather therefore does not overlap compute - it competes with the gather and displaces it byte for byte. Displacement coefficient measured at 1.00 over three runs (gather pattern 5.93 GB/s solo, 5.08 GB/s with copy-engine traffic concurrent). Issue the look-ahead after this layer's FFN instead, so the transfer rides the following layer's attention, where nothing is reading host memory. Measured on the real-prompt harness (767-token prompt, 200 output tokens, same binary, RTX 3060, --moe-expert-cache-size 28, --moe-lookahead 1): control 9.643 t/s per-token p50 103.73 ms lookahead 1, paced 9.357 t/s per-token p50 107.76 ms (-3.0%) lookahead 1, unpaced 9.040 t/s per-token p50 112.40 ms (-6.4%) Pacing recovers about half the deficit, so the look-ahead remains net-negative and the residual cost is not yet attributed. Control is unchanged by this commit (9.66 -> 9.64, within run-to-run spread), confirming the reorder is inert when --moe-lookahead is off. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…consumption ledger Three changes, all in service of one question: why does the look-ahead stage and consume, yet not pay? 1. Rolling readiness (moe-cache.cu). Readiness for the look-ahead staging was decided once per layer by moe_early_router_ready_positions, which sampled the copy worker's landing counter a single time and demoted EVERY not-yet-landed position to -1. Readiness was therefore all-or-nothing: a transfer that was 95% landed and one that was absent got the same outcome. Readiness is now evaluated per expert inside the gather against the live counter, re-sampled as the sweep advances, so a copy lands and is consumed progressively. A caller that passes no counter keeps the legacy meaning of position >= 0, so the early router path is untouched. 2. GGML_MOE_LOOKAHEAD_ISSUE_AFTER_FFN (qwen4exp.cpp, qwen35moe.cpp). Selects whether the look-ahead for layer il+1 is issued after this layer's FFN (default, as in 168a907) or before it (legacy). The two positions couple the prediction to different parts of the layer, so this A/Bs them in one binary without a rebuild. 3. Consumption ledger (moe-cache.cu). The look-ahead's own counters printed only every 16 dispatches and never showed the consumed side, so staged-vs-consumed could not be read; the available number was the plan's miss count, which the look-ahead does not change when it merely replaces a demand read. Now drained every dispatch and printing consumed_mib_total / consumed_total / consume_pct_total (lane-lifetime). Existing fields keep their names, order and meaning; the new ones are appended. Measured, real prompt, 200 output tokens, same binary, RTX 3060, probe off: control 9.699 t/s lookahead 1, after FFN 9.398 t/s (-3.1%) lookahead 1, before FFN 9.030 t/s (-6.9%) All three arms produce 202/202 identical completion tokens at temperature 0, so neither the reorder nor the rolling predicate changes model output. The consumption ledger is the finding: consume_pct_total is ~95% in BOTH placements. The staged bytes are consumed. An earlier reading of the plan-level miss counter as "the reuse collapses to zero under pacing" was wrong and is retracted; that counter does not measure consumption. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…-in plan-side admission) `copy_mib` was n_misses * resident_bytes_per_miss - a derived figure that charges every miss to the demand side of the ledger whether or not the gather actually fetched it. Once the look-ahead stages an expert, the gather serves that expert from staging and no host read happens, but the ledger still billed it. The look-ahead therefore appeared to ADD demand traffic (+979.6 MiB over 199 steps) when the instrument could not see the saving at all. GGML_MOE_LOOKAHEAD_ADMIT_IN_PLAN=1 makes the plan classify a would-be miss whose slab has already landed as an admission served from staging, and subtracts exactly those from the demand-side counters. It is classification only: the miss entry, its slot and its committed mapping are unchanged, the gather lands the staged payload in that same slot, and no slot table is written - the plan stays the single writer of slot_for_expert/expert_for_slot/ last_used/expert_frequency, and no new synchronization is introduced. The predicate is the gather's staged_ready test verbatim, sampled one launch earlier; the landing counter only grows, so every expert classified here is staged-served by the gather below. Default OFF is behaviourally today's: probe arm reproduces need=94570 resident=43125 copy_mib=92239.6 exactly. Gate ON changes the ledger and not the clock (9.3915 vs 9.3984 t/s). Measured, real prompt, 200 output tokens, probe on, steps=199: control need=94570 resident=43125 copy_mib=92239.6 lookahead 1, gate 0 need=95520 resident=43529 copy_mib=93219.2 lookahead 1, gate 1 need=95520 resident=43529 copy_mib=86625.0 93219.2 - 86625.0 = 6594.2 MiB, which equals the look-ahead's independently counted consumed_mib_total (6594.29 MiB) to four significant figures. Two instruments agreeing is what makes this a measurement rather than a relabel. Against the control the look-ahead removes 5614.6 MiB over 199 steps = 28.2 MiB/step = -6.1% of demand traffic. So the earlier reading of copy_mib as evidence that the look-ahead does not reduce demand traffic was wrong, and is retracted: the counter could not express the saving. The feature does remove demand traffic; what it costs is that it stages more than it saves. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…activation path Adds GGML_CUDA_MOE_PHASE_PROBE_PER_STEP: one `moe-step: step= need= resident= misses= miss_mib=` line per decode step, independent of the cumulative summary gate. Non-perturbing by construction: each dispatch snapshots the counter table asynchronously (event record on the step's streams, wait on a private non-blocking stream, async D2H into pinned memory, event record) and the host reads it at the NEXT drain with cudaEventQuery. No cudaStreamSynchronize and no cudaDeviceSynchronize anywhere in the decode loop, so the pipeline is never drained and the clock being measured is not disturbed. Measured cost: 9.6593 t/s instrumented vs 9.6375 uninstrumented on the same binary, i.e. free. Root cause of a defect found while validating it: a replayed CUDA graph never calls prepare_decode, so per-token notes taken there existed only for the first few CAPTURE tokens. The ledger emitted 3 lines for a 199-step run. The notes are now also taken in activate_graph_resources, which runs for CAPTURE and for REPLAY, so every token is accounted. Emitted lines now equal the decode step count and their sums partition the cumulative summary exactly. Real prompt, 200 output tokens, --moe-lookahead 0, 199 steps: need=95520 resident=43529 misses=51991 miss_mib=93219.28 (480 demands/step = 48x10) misses/token: min 107 mean 261.3 p50 262 p95 345 max 480 (step 0, cold cache) Per-token latency model, fitted on the 196 non-CAPTURE steps: observed_ms = 25.7 + 0.3003 * misses, r2 = 0.990 0.3003 ms/miss at 1.881 MB/miss = 6.27 GB/s, which is the measured achievable link bandwidth. Decode is bandwidth-bound on expert misses, and miss count explains 99% of the per-token latency variance. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…16, unchanged) The grouped replacement rule is LFU-with-decay: plan->frequency_epoch = step >> 4, and the effective frequency shifts right once per elapsed epoch, so the stored demand history halves every 16 grouped planning steps and is zeroed after 32 epochs (512 steps). A decode token is 48 planning steps (48 MoE layers, one launch each), so the policy's memory was about one to two tokens. GGML_CUDA_MOE_FREQUENCY_HALFLIFE retunes the epoch length without changing the decay law; unset, empty, non-numeric or non-positive falls back to 16, and the value is read once at grouped-context creation next to the existing FREQUENCY gate. This exists to settle a question with a measurement rather than an argument. Capacity is VRAM-capped at N~53, so if the remaining headroom were in WHICH experts are kept, a longer frequency memory would show it. It does not. All arms at N=53, look-ahead off, real prompt, 200 output tokens, per-step ledger on: half-life 16, LFU (default) r = 58.0% 11.702 t/s half-life 256, LFU r = 54.4% 11.026 t/s (-6.1%) half-life 2048, LFU r = 54.4% 11.033 t/s (-6.0%) pure LRU (GGML_CUDA_MOE_FREQUENCY=0) 11.236 t/s (-4.3%) Longer frequency memory makes it worse, and so does dropping frequency entirely: the recent set IS the recurrence set for this workload, and a long-horizon accumulator protects experts that are no longer needed and evicts the ones that are. The default is the best of the four, so the replacement policy is not a lever and was not one before this change. The default arm also reproduces the pre-change 11.738 t/s to 0.31% on the previous build, which is the regression check for the added kernel argument. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
The startup warning for --moe-expert-cache-size said the flag was set but never said to what. That makes a cache-size sweep unverifiable from logs: the only way to tell which N a run used was to infer it from the miss count, which is exactly the quantity the sweep is trying to change. Emit the resolved slot count in the same one-line warning. The value is params.moe_expert_cache_slots, the same field handed to the CUDA set_slots entry point a few lines earlier, so the logged value cannot drift from the installed one. Verified: loading with --moe-expert-cache-size 53 and 28 prints n_slots=53 and n_slots=28 respectively. Logging only, no behaviour change. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
The moe-lookahead-stage line could not answer the obvious question about the feature: of the experts it fetches ahead of time, how many does the next layer actually want? It printed staged BYTES and consumed BYTES, and it printed counters[3] twice under two names - prefetch_used (interval delta) and consumed_total (lifetime) - so there was no staged-expert count and no ratio. Add staged_experts_total (counters[2]) and used_pct_total (counters[3]/counters[2], expert counts, not bytes). Guarded like the existing consume_pct_total. All twelve existing fields keep their names and order, so existing parsers are unaffected. Measured at --moe-expert-cache-size 42 --moe-lookahead 8: 35222 staged, 672 consumed, 1.91%. The same ratio is 95.2-95.5% at width 1, which is the whole difference between the widths. Also worth recording since the field names mislead: counters[4], printed as "published", is a count of publish EVENTS, not experts. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
Adds GGML_CUDA_MOE_DEMAND_TRACE (+ ..._FILE) which drains, once per decode step, the expert ids the grouped plan actually commits: one line per (step, layer) as step=<u> layer=<u> ids=<comma separated>. The record is taken from the plan's own unique_experts staging array at the same statement that stages it, so it is the same data the plan acts on, and the step index comes from a device-side per-layer counter so that CUDA graph capture/replay advances it exactly like a direct launch. This is the input an offline Belady oracle needs to bound how many of the ~225 misses/token any per-layer policy could avoid. At N=42 it captures 199 steps x 48 layers and reproduces need=480/step exactly. Gate is presence-only and defaults OFF; nothing is allocated when it is off. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
added 6 commits
September 15, 2026 18:14
Stores the harness that produced every number in the MoE decode-throughput
work, so it travels with the code instead of living only on the measuring
rig.
benches/moe-cache/
stream.sh base driver: serve once, stream a real prompt, record
the arrival time of every token
arm_*.sh 19 arms: cache sweep, order control, policy sweep,
VRAM budget, pacing, recall, correctness (greedy token
identity and perplexity), demand tracing, eviction and
admission A/B
phase.sh etc. older attribution drivers, kept for provenance.
phase.sh is the retired synthetic harness - its numbers
must never be mixed with stream.sh real-prompt numbers
mixed_contention.cu the PCIe gather-contention microbench
expert-quant-*.txt per-tensor quantization maps (1224 tensors; the 144
expert tensors retargeted)
run_imatrix.sh importance matrix pass - required because a sub-q2
run_requant.sh reduction is refused without one
slotalloc/ the calibrated policy simulator. Reproduces the shipped
LFU-16 policy per-step on 8 of 8 recorded ledgers, which
is what makes its Belady and horizon results usable
docs/moe-lookahead-improvement-plan.md
The design and measurement record, revision by revision.
Outputs (logs, per-token TSVs, demand traces) are gitignored: they are rig-local
evidence and are not source.
Assisted-by: Oh My Pi (deepseek-v4.1-flash)
The policy engine reproduces the shipped LFU-16 policy per-step exactly on 8 of 8 recorded ledgers, so its results are usable. It finds that prediction-guided victim selection caps at +4.2% with a PERFECT same-layer next-step predictor, and at exactly zero with the predictor in the tree, whose horizon is one MoE layer rather than one step. The +32% Belady gap needs 8-16 steps of same-layer horizon, and that cannot exist for a non-speculative decoder: layer L's router at step T+1 consumes layer L-1's output at step T+1, a function of token T+1, which is sampled after the moment the prediction would be needed. The retention axis is closed in principle, not merely unprofitable. What remains reachable is the existing one-layer look-ahead, whose blocker is named and measured: a ~2.05 MiB staging window against 15.4 MiB needed gives 1.91% usefulness at width 8 versus 95.5% at width 1. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
ev0p isolates the one retention policy with a measured positive sign in the offline engine (223.57 vs 225.00 misses/step). The existing protect arm ran it composed with look-ahead, which confounds the measurement. Records what each arm is expected to show, so a null result is evidence rather than an unexplained reading: ev8p should be exactly null because the tree's predictor targets a different layer, and ev8a's win is timing, not bytes. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
Measured on the rig, one build, real prompt, 200 tokens, N=42:
width 0 225.00 misses/step 10.8320 t/s (default path, ledger byte-identical
to the recorded baseline via cmp)
width 1 225.00 misses/step 10.3841 t/s -4.14% 94.78% of staged data used
width 2 225.00 misses/step 10.4677 t/s -3.36%
width 4 225.00 misses/step 10.0130 t/s -7.6%
width 8 225.00 misses/step 8.0052 t/s -26.08% 2.33% of staged data used
Misses are identical at every width, so the look-ahead never changes the hit rate,
and it loses even at width 1 where the staging mechanism works as designed. With a
displacement coefficient of 1.00 the fabric is serial, so a staged fetch displaces
a demand fetch that still has to happen: staged bytes are additive, not
substitutive. The earlier 1.91%-at-width-8 reading was a lateness symptom, not the
cause.
Consequence: N=53 with the look-ahead off is 11.79 t/s re-derived from
perftok-n53.tsv (+8.9% over 10.832, both with lookahead=0 per arm_cache_sweep.sh:25),
so N=53 now dominates every look-ahead configuration and the recorded mutual
exclusion costs nothing.
Also: ev0p measured 223.57 misses/step against the offline engine's predicted
223.57, validating the simulator end to end; ev8p - ev8r = -0.08 confirms the
predicted-mask pin is a no-op.
Adds arm_width.sh. Documents the admission-path crash (abort on step 2, before any
admission was committed) as an open blocker.
Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…closed Admission (the owner's request) works and is coherent but loses. With the plan-residency trap fixed, ev8a completes 199 steps: 44.92 admissions/step, real traffic falling 225.0 -> 223.9, so predicted experts do become hits - but at ~1.1 fewer misses bought with 45 extra fetches, a 41:1 loss ratio, and 98.3% of admissions evicting a resident rather than filling an empty slot. Net -9.1% on top of the look-ahead's own -26.1%. Mechanism resolved: staged_mib is a per-dispatch delta and *_total fields are lifetime sums; conflating them produced a phantom 150x mismatch. Per-step staging is 29.2/62.9/139.4/319.5 MiB at widths 1/2/4/8 and the waste column fits ~1.3 ms fixed plus ~0.10 ms per MiB/step wasted, so width 8 is a traffic problem and the small widths are a fixed per-feature cost. Retention closed at the operating point: recent1 buys -1.43 misses/step at N=42 but only -0.21 (-0.10%) at N=53 - the headroom shrinks as capacity grows. Harness: anchored all kill patterns to bin/<binary>; a wrapper containing "--target llama-server" was killing its own parent shell via the hygiene pkill. Adds TAG_PREFIX and an arm selector. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…nged
Implements GGML_MOE_EVICT_POLICY (lfu default, lru, protect, recent1),
GGML_MOE_ADMIT_PREDICTED and GGML_MOE_ADMIT_MAX, plus the counters
admitted_experts / admit_evictions / admit_skipped and staged_mib_total.
Committed because the measurement record now cites the build that produced
it; the feature it adds measures as a loss and is kept as the tested path.
Default path verified byte-identical, not merely argued: with the env unset
the plan kernel writes no counter, pin mode is NONE, admit_cap is 0, both new
branches are dead, and the ledger is cmp-clean against the recorded baseline
(199 steps, 225.00 misses/step).
Measured, N=42, real prompt, 200 tokens, look-ahead off unless stated:
- shipped LFU-16 225.00 misses/step 10.8320 t/s
- recent1 223.57 10.8316 t/s (deterministic miss gain,
invisible in time)
- protect, width 8 223.49 8.0368 t/s (mask pin = -0.08 vs
recent1: a no-op)
- admit=1, width 8 268.80 misses/step, 44.92 admissions/step, real traffic
223.9, 98.3% of admissions evicting a resident, 7.2730 t/s.
Coherent - predicted experts do become hits - but ~1.1 fewer
misses bought with 45 extra fetches, a 41:1 loss ratio.
Fixes a plan-residency trap that aborted the server on step 2 whenever
admission was enabled: the candidate list is built from slot tables as of
publish time, so it compensated for the previous plan's misses but not for its
admissions, letting two slots claim one expert.
Assisted-by: Oh My Pi (deepseek-v4.1-flash)
Owner decision: stay at N=42 or N=36 rather than N=53, deliberately keeping VRAM for MTP, and drive the look-ahead from the MTP head instead of the current cross-layer heuristic. N=36 measured, same session and binary, look-ahead off: 239.11 misses/step at 10.3626 t/s, i.e. -4.37% from N=42, for ~618 MiB freed. Marginal -2.35 misses/slot, consistent with -2.25 measured from 42 to 48 and Belady's -2.0. Records that section 7.28's impossibility argument is scoped to a NON-SPECULATIVE decoder, which is now load-bearing: MTP is speculation, so running the router on draft hidden states yields same-layer demands for T+1..T+H before the real tokens exist. That is the 8-16 step horizon the retention axis needs, so the axis is closed for the pre-MTP fork rather than permanently. With the caveat that MTP fixes the prediction quality completely but the staging lane only partly: retention removes fetches while staging can only move them in time, and the lane lost 3.7% at width 1 where 94.85% of staged bytes were useful. Spend MTP predictions on victim choice, not on staging. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…e fork
Sizing (arithmetic on measured totals, no head run - none exists):
one appended block = 913.3 MiB experts + 78.3 MiB non-expert = 991.6 MiB of
weights; under --n-cpu-moe only the 78.3 MiB is VRAM, plus the head's own MoE
cache at 3 banks x N slots x 1.793 MiB = 226 MiB at N=42 / 194 MiB at N=36.
Total ~310-330 MiB (N=42) and ~280-300 MiB (N=36) against 1.55 / 2.2 GiB of
headroom. Either operating point covers it. The traffic term is the risk:
a full MoE block per draft step is +17.8% at H=4 unless cache-resident.
Source findings:
- src/models/qwen4exp.cpp has a working DECODER_MTP graph for this arch
(:52 nextn_predict_layers, :186 draft-only export, :313 graph_mtp).
- The NextN tensors are created inside the trunk layer loop, after the
`il < n_layer` continue, so a block is a full layer plus a head.
- embed_tokens / shared_head_head are TENSOR_NOT_REQUIRED and absent for
qwen4exp - the head reuses the trunk's, so no vocab-sized tensor.
- build_moe_lookahead() already takes nextn_state as its input, so the plan
is a change of input producer, not a new mechanism. Its only consumer today
is GGML_OP_MOE_PREFETCH - the staging lane measured as a net loss in 7.30.
Retargeting the consumer to victim choice is the actual work item.
- The trained block is absent: 0 of 1224 tensors match nextn|mtp|draft.
Also records the erratum on the per-token latency/miss regression, which was a
misreading of perftok-*.tsv rows (sub-token deltas, not one row per token).
Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…e-dispatch 7.31 mapped the offline horizon curve onto MTP draft length H. Checked against the driver this cycle: the premise does not hold. What is true - verification does batch every draft position: server-context.cpp:580 sampled token and draft tokens go in the same batch server-context.cpp:619 batch asserted large enough for both server-context.cpp:1273 n_max + 1 > n_ubatch is rejected: one ubatch server-context.cpp:4559 acceptance samples over all n_draft + 1 positions moe-cache.cu grouped cache, unique_experts per dispatch = union Why that is not a look-ahead: layer L's router output for T+1..T+H is computed in the same grouped dispatch that consumes it. The union is fetched once, in the operation that revealed it. Retention changes which residents survive into LATER dispatches, and those demands are unknown for the original reason - layer L's demand for a future token needs layers 0..L-1's output for that token, and MTP supplies draft TOKENS, not their trunk hidden states. 7.28's impossibility argument is restated, not escaped. Consequences recorded: the horizon curve is an offline bound in the same sense Belady is; the +8%/+13% acceptance-discounted figures are withdrawn; MTP's only ahead-of-time signal is for its own layer, whose consumer is the staging lane measured dead. One loophole left as a measurement, not an argument: --decode-overlap could split draft positions into separate dispatches, which the moe-step ledger and the demand trace would show directly. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
…t condition
server-context.cpp:1273 refuses to start unless n_max + 1 tokens fit in a
single ubatch ("capped MTP replay requires an ubatch of at least %d tokens").
The whole draft is replayed in one forward, so overlap cannot be splitting it -
that check would fail on every start. 7.31's H-curve mapping is permanently
withdrawn, not conditionally withheld, and the ledger run proposed to confirm it
would only restate the start condition.
Assisted-by: Oh My Pi (deepseek-v4.1-flash)
#124 7.32.5 item 1 said the trained block does not exist. That was wrong in the way that matters. It ships separately and is on the rig: /mnt/SSD/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf 2657.48 MiB /mnt/SSD/MTP/mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf 1818.80 MiB Enumerating its tensors shows a complete block 48 - attn q/k/v/output/norms, indexer, all 8 hc_attn_*/hc_ffn_*, the full MoE (ffn_gate_inp, ffn_*_exps, ffn_*_shexp) and the 6 nextn.* tensors - declaring nextn_predict_layers, nextn_shared_target_tensors and arch qwen4exp. So 7.32.2's "a full layer plus a head", derived from the loader's control flow, is confirmed against the artifact, and the head's cost is now measurable directly rather than by arithmetic. Only the trunk-file claim survives: 0 of 1224 tensors match nextn|mtp|draft. The more useful find is issue #124: the expert cache and MTP are mutually exclusive on this model today. With --moe-expert-cache-size > 0 plus --spec-type draft-mtp the grouped plan is unavailable (required_unsupported 63-95), the graph fails closed, draft-mtp never produces or verifies drafts, and the server falls back to legacy. Any MTP number taken with the cache on is measuring the fallback. That is the next item. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
The operating rule for this track is "the doc wins". Section 8's CURRENT OPERATING POINT block still presented N=53 as the live best configuration, and 7.29's "Recommended operating point: N=53" was unmarked, so the doc would have resurrected the configuration the owner declined. Both now state the owner's decision (N=42 or N=36, look-ahead off, VRAM kept free for MTP) and mark the N=53 text as a record of why the trade was believed to exist rather than a recommendation. Section 8 also gets the intra-session table (N=36 10.3626 / N=42 10.8362 / N=48 11.3891 / N=53 11.7824) so the older different-session N=53 figures are not mistaken for the comparable pair, and the stale HEAD pointer is refreshed. No new measurement; this is corrections to a document that is load-bearing for the next agent. Assisted-by: Oh My Pi (deepseek-v4.1-flash)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-P1: look-ahead prefetch producer (NOT LANDABLE)
Base is
feat/763-reconcile-qwen4exp-mtp(PR #120 head), the same deviation PR #126took for the llama.cpp-side consumer, as accepted by the reviewer.
This branch contains the complete producer implementation plus the measured blockers.
It must not merge in this state. The three blockers below are all reproducible,
deterministic and measured on the rig; the first one is a correctness blocker.
What is implemented
GGML_OP_MOE_PREFETCHend to end: enum, constructor, op name, symbol, CPU dispatchas a documented no-op,
ggml_get_n_tasks, RPC op list, backend-ops skip list.build_moe_lookahead()picks layeril+1's experts fromlayer
il's post-attention state (build_lora_mmonffn_gate_inp, thenggml_argsort_top_k) and ends inggml_moe_prefetch(), gated on--moe-lookahead > 0,ctx_type == DEFAULTand decode rows only.src/models/qwen4exp.cppandsrc/models/qwen35moe.cpp.preinstall_legacy_pools()), the-1"snapshot notpublished" contract, and a bounded self-healing retry in the consumer.
GGML_CUDA_MOE_LOOKAHEAD_DEBUG.ffn_up_exps, notffn_gate_up_exps:ffn_gate_up_expsisnull for every layer of this model. The consumer resolves the layer by name, so any
of the layer's cached expert tensors works.
Blocker 1: the producer changes the model output
Logits are not preserved. Same prompt, same flags, only
--moe-lookaheaddiffers:fc1da5b9e41d249e7da51791a934d5618e0a9d10e9095ce7806f645e7edbba0efc1da5b9e41d249e7da51791a934d5618e0a9d10e9095ce7806f645e7edbba0e4ba9f8e7a60ca9830b0dcbed073ffc3e9bbf44eb3b98b5503ed2d1f9b028dda3fc1da5b9e41d249e7da51791a934d561Readings, all deterministic across reruns:
MUL_MATthat readsffn_gate_inpis the trigger. Removing the argsort andthe prefetch op does not remove the divergence.
logits byte-identical, so the scheduler is not merely shape-sensitive; the platform is
shape-neutral for a node that does not touch the router.
capture is not the cause of the numeric change.
use_countsproof that rejectsthe graph for the prefetch node (
graph=unproven(14)) also covers the router tensors,and an extra reader of
ffn_gate_inpchanges those counts. This is unconfirmed, and itis the one open question left.
An earlier handoff recorded these logits as byte-identical. That reading came from a
build in which the producer never ran. It is refuted here.
Blocker 2: every prediction is dropped, so the feature is inert
acquire_legacy_cache()installs a new record only while the target group's authority isGGML_CUDA_MOE_GROUP_AUTHORITY_LEGACYwith admission open. On this rig the groups arenot under legacy authority, so the pool install fails for every target:
The producer runs and the consumer resolves the right layer, but no copy is ever
enqueued. This answers the design's open question Q1 negatively: the cross-layer pool
install is not reachable in this configuration.
Blocker 3: -13% decode throughput
The ids readback synchronizes the stream, which forces
use_cuda_graph = falsefor everygraph containing the op. The precedent is
ggml-cuda.cuggml_cuda_mul_mat_id_needs_syncat the same decision point, and the demand path's own ids readback has the same cost class.
The feature stays default-off.
Correctness work that is independent of the blockers
ggml.c: the per-node src loop must not countGGML_OP_MOE_PREFETCHsources. The opreads no tensor values, and counting them made the MoE grouped-decode certificate reject
the graph:
grouped decode certificate failed: ... graph=unproven(14)followed by a failed graphcompute. With the exclusion, zero certificate failures.
preinstall_legacy_pools()returned 0 when the candidate snapshot was not published,which is indistinguishable from success, so the caller latched "pools ready" forever.
It now returns -1 and the caller only latches on a real result.
Verification performed
test-moe-cacheon the final code: OK, exit 0, includingtest_lookahead_prefetch_eviction_guard OK.ninja -C build llama-server test-moe-cache llama-perplexity, CUDA arch86;120.
Recommended decision
perfect producer has nothing to page into while the target layers are not under legacy
cache authority (blocker 2), and the readback cost is structural (blocker 3).
ffn_gate_inpmust not be able to change the model's output. That is a property of thefork's router identification or of its count-based proofs, not of the prediction math,
and it needs to be fixed where it lives.
Diagnostics kept on the branch (env-gated, inert when unset)
Added after the reopen-transport ruling commit, at the owner's request that the
instruments stay in the tree for troubleshooting rather than be reverted:
9d24345de- phase attribution probe (GGML_CUDA_MOE_PHASE_PROBE=1) and off-leaserecall scoring (
GGML_CUDA_MOE_LOOKAHEAD_DEBUG=1), plus the two cache populations thatwere counted but never printed (
prefetch_dropped,evicted_prefetched_unused).570370183- measured results indocs/moe-lookahead-design.md.Both probes are inert with the env vars unset (one cached
getenvbehind a static bool)and both turn themselves off with a single stderr note if an event or a memcpy cannot be
issued, which is what happens under CUDA graph capture, so they cannot break a capture
run. An attribution needs
GGML_CUDA_DISABLE_GRAPHS=1, because replay steps never visitthe host dispatch loop where the events are recorded. This does not change the three
blockers below: the producer is still not landable.
Measured on the RTX 3060, ctx 81920, cache 42, single 13,946-token prompt, 256 decode
steps at 96.90 ms/step (10.43 t/s). Phase rows sum to 95.7 ms, so the split covers the
step:
MUL_MAT_ID,MOE_PREFETCH,ARGSORT,TOP_KGET_ROWSonper_layer_token_embdDispatch state
mode_legacy=3 mode_direct=253; prefill (28 ubatches, 4555 ms each) is87.9% MoE. Recall scored in the dispatch loop instead of the legacy blk.47 lease: 64.5%
of the used experts predicted, 80.6% of the predictions used; the other two populations
are structural zeros because nothing is offered to the installer (decode phase line
ops=0,legacy cache authority).The bytes/token this implies (GGUF tensor inventory, 10 of 512 experts per layer):
1031 MiB of expert weights, 3.34 GiB of dense/attention/head/shared, 4.35 GiB total, and
the MoE phase streams its 1.01 GiB at 16 GB/s, i.e. 5.4% of the card's bandwidth, while
the dense phase reaches 126 GB/s. The decode step is MoE-kernel-bound, not transport- or
capacity-bound: prefetching ahead layers cannot make the kernels faster, and the expert
H2D (337.6 MiB/token, 3.71 GB/s of the gen4 x4 link's 7.88 GB/s) is already fully hidden
(
calls=ready=12240,ready_min=255).