Conversation
Reframes look-ahead as expert prioritisation across three execution sites (VRAM-resident / PCIe-streamed / CPU) sized by their bandwidths. Core rule: split the per-dispatch miss set in proportion to link and host-DDR bandwidth, serving it at B_link+B_cpu = 38.6 GB/s instead of 32 GB/s CPU-only. No predictor, no lead time, self-tuning. Records what is already closed by measurement so it is not re-litigated: prefetch at every width, MTP-as-predictor, static offline tiers, per-workload tiers, and non-uniform slot allocation. Design only; T-1 (#124) blocks the rest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t 65-70 Section 4 claimed the hybrid frees 4.5-5 GiB of VRAM for slots, reaching N~65-70 and h~0.62-0.64. Wrong. Plan 7.15 measured the ceiling directly: 9293 MiB used / 2619 MiB free at N=28, 103 MiB per slot, and N=64 is a hard cudaMalloc OOM. The hybrid does not move it - backing slots out of the N=28 figure leaves a ~6.4 GiB base of non-expert weights + KV + compute buffers that the hybrid does not reduce. Consequences: the N=53 control is already at the ceiling, T-4 is demoted to a last-or-never confirmation worth ~1pp h, and section 5's top row falls from ~47 to ~44.5 t/s. T-3 remains the real increment. Also closes the 64K-VRAM question raised in review: every capacity arm ran -c 81920, so 80K KV is already inside those figures and a 64K run adds no new tenant. And pins 2.148 MiB as the VRAM-sizing byte figure - it predicted the N=64 OOM, so it is better-evidenced than the ledger's 1.793 MiB for that purpose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… refutation T-2 measured 10.40/10.08 vs a 21.36 control, but the split never engaged: H1 is unengaged under CUDA graphs (defer_completion) and H2 is skip-heavy off-graphs, equal to cache-32 no-split. Both legs measured the cache-only path, which is a known point (sweep N=36 10.36, N=42 10.84) and which the model predicts at ~2x slower than CPU-MoE (measured 2.05x). Adds the direct evidence for tier 1: decomposing both exp1 arms with a shared fixed term yields F 29.14 (278 hits/token) and F 28.34 (zero hits), so a resident hit costs 0.0029 ms against 0.2842 ms for a miss - 99x cheaper. The premise holds; routing misses to tier 3 was never tested. Applies a 1.14x haircut from the measured cache-only point, moving the hybrid projection from ~40 to ~36 t/s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rrection T-2b stopped at cpu_fraction 0.0074. T-2c found the cause: MTP verify-combine packs 3 pos x topK10 = 30 routes/plan while stage-3 validates a 10-row reader against it (moe-cache.cu:12601). Width mismatch - an implementation gap, not a design result. Recalibrates on arm 000: queued=reused=72 gives 2.78 tok/dispatch (not 3.43), and t_cpu refits to 0.0677 vs the nominal 0.0704 - the model holds to 4 percent on a config it was not fitted to. Prices the 6.5 percent machinery tax measured with the cache inert (covered=1, split=0). Records an OPEN conflict: T-2c reads 54 percent miss at N=42 while cache-32's bank ledger implies ~18 percent at N=32. More slots cannot yield more misses. Gates therefore assert cpu_fraction against the arm's own ledger, never a target imported from another config. Corrects the N ceiling a second time: fresh footprint (425 MiB free at decode) retires the 9293 ledger, box max is N=42, and T-4 is dead rather than demoted. Adds width-scope as the highest open risk and makes all throughput gates relative to a same-binary control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ually found T-2g removed the stage-3 veto entirely and engagement still read cpu_fraction 0.0048. The layout audit settled negatively on its own terms: the reader-row to route-offset mapping does not survive to finish time (no per-reader index retained, ~98.6% of groups read back garbage), and residency is consulted after unconditional admission so any post-hoc splice duplicates work. Either is beyond one arm; together they are a plan-structure plus kernel-ordering change. The premise was not falsified - a resident hit still costs 0.0029 ms against 0.2842 for a miss. The durable finding is hardware: on an x4 link the cache is strictly worse than CPU-MoE at every tested N, because the link moves an expert in 0.342 ms where host DDR computes one in 0.0677. The best config on this rig uses none of the machinery built for it. Records two things closure does not mean: the 15-20 t/s goal was at 64K and 21.23 is a 767-token number that extrapolates to ~13 t/s at 64K; and the best config is currently unbuildable because base 0fc51e0 has a pre-existing 47 GB allocation bug. Both follow-ups are non-tiering. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A design for prioritising MoE experts into tiers, per the owner's framing:
the value of "look-ahead" is not foreseeing experts, it is deciding where each
expert runs. Design only — no code. Someone else picks this up and implements it.
The idea
An expert costs what its execution site costs, and this rig has three sites whose
bandwidths differ by 50x:
They run in parallel, so layer time is
max(T1,T2,T3), not the sum. Note T3 is5.2x cheaper than T2 — host DDR beats the x4 link, which is why the CPU is the
right home for the cold tail.
Core rule: split each dispatch's miss set in proportion to the two pipes'
bandwidths —
f_link = B_link/(B_link+B_cpu) = 0.171. The miss set is then servedat 38.6 GB/s instead of 32, a 17% cut in the dominant term. Self-tuning: move
the card to the x16 slot and
f_linkrebalances to 0.45 with no code change.No predictor, no speculation, no lead time. Routing a miss over the link needs
no foreknowledge — only a budget.
Why prediction is gone
Every prediction-flavoured mechanism has now been measured on this rig and lost:
frac>0.8 = 0.26%§2 of the doc tabulates these so they are not re-litigated.
Projected
Only the first row is measured. The rest are projections from a model that has
reproduced two MTP arms at different depths and validated out-of-sample across
N=28/40/53 to 1.2%. Also: 64K has never been measured — we have 767 and 13,946
tokens only, so every 64K figure is extrapolation (T-6 fixes this).
Sequencing
T-1 #124 (blocking) → T-2 land/measure hybrid → T-3 T2 budget → T-4 raise N →
T-5 seeded priors → T-6 verify at 64K. Each has a numeric gate in the doc.
fails closed into legacy fallback while still serving — so an arm with the
cache on measures the wrong thing and merely looks slow. Any such arm must assert
the
draft acceptanceline before its number is used.Notes for the reviewer
baseline-flash-next; single new file, no code touched.plan Nsections live indocs/moe-lookahead-improvement-plan.mdonfeat/moe-lookahead-p1, not on this branch.1.79–1.88 MiB by the per-step ledger. That ~20% gap moves every absolute
number in §5; ratios are unaffected. Worth reconciling before T-3.
🤖 Generated with Claude Code