feat: add staged page-level prefix reuse - #138

Draft

Andrewxu313 wants to merge 230 commits into

mainfrom

feature/add-staged-page-level-prefix-reuse

Commits on Apr 26, 2026

start: Add staged page-level prefix reuse
TairanXU
committed
fcdabd5
Implement page-level prefix cache reuse
lausannel
committed
49e5130
Fix prefix reuse exact duplicate prefill drift
lausannel
committed
dabe7b2
Merge staged page-level prefix reuse branch
lausannel
committed
968f4fa

Commits on Apr 27, 2026

Add safe prefix reuse scheduling guards
lausannel
committed
b2b6965
Restore batch-level prefill offload drain
lausannel
committed
de62491
Backport deterministic batch sampling sync
lausannel
committed
28ac33d
Merge remote-tracking branch 'origin/main' into tmp-prefix-main-merge-9408
lausannel
committed
a17cc3f
Use legacy host KV release without shared prefix
lausannel
committed
0a51c0e
Keep prefix-only sync out of no-prefix runs
lausannel
committed
e9bf52c
Keep GPT-OSS no-prefix prefill offload async
lausannel
committed
8797ecb
Avoid pool-mode completion detokenize on worker ranks
lausannel
committed
ef8d0e4
Keep plain host KV offload on legacy path
lausannel
committed
a105db0
Use backend page lookup for non-prefix host KV loads
lausannel
committed
a60d566
Use legacy batch release without shared prefix
lausannel
committed
5f9ed96
Clear completed pool batch before next admission
lausannel
committed
990427a
Keep GPT-OSS no-prefix attention call unchanged
lausannel
committed
e8fabe2
Bypass prefix decode isolation when disabled
lausannel
committed
42b0db3
Reset pool local slot allocator between groups
lausannel
committed
f19957b
Reset pool buffer state between batch groups
lausannel
committed
d9e4dd9
Keep pool sequence ids monotonic across batches
lausannel
committed
96271e4
Keep no-prefix pool admission on main path
lausannel
committed
65e80e8

Commits on Apr 28, 2026

Use exact pointer pairs for host KV copy dedupe
lausannel
committed
ea5bcc6
Remove full KV reuse implementation plan doc
lausannel
committed
3735dd1
Merge remote-tracking branch 'origin/main' into tmp-prefix-main-merge-9408
lausannel
committed
c7400c3
Keep prefix reuse transparent to decode scheduling
lausannel
committed
b874f9d
Implement prefix cache eviction
lausannel
committed
486c120
Remove fragile prefix rank cache unit test
lausannel
committed
3f7a48a
Add prefix eviction rank cache tests
lausannel
committed
761fdf4
Fix active prefix eviction KV assertion
lausannel
committed
e2f5098
Evict prefix cache before prefill admission stalls
lausannel
committed
f5aa7dc
Add prefix cache diagnostics and safety tests
lausannel
committed
1a4220d

Commits on Apr 29, 2026

Log prefix cache lookup stats
lausannel
committed
1dea7d1
Allow overriding host KV shm names
lausannel
committed
c853161

Commits on Apr 30, 2026

Bound host KV async offload concurrency
lausannel
committed
2ed5be2
Chunk greedy argmax sampling
lausannel
committed
f945911
Bound prefix prefill offload backlog
lausannel
committed
0b5c210
Avoid full decode logits fp32 cast
lausannel
committed
6c19f23
Reduce greedy sampling fp32 chunk size
lausannel
committed
e0b3ae9
Reuse grouped MoE output buffers
lausannel
committed
8c858cc
Reset GPU KV between prefix pool groups
lausannel
committed
21be1ed
Reserve decode scratch in GPU KV budget
lausannel
committed
b2f422a
Fix cumulative decoded length after host KV reentry
lausannel
committed
8e21a52
Fix host KV reentry decode limit clamp
lausannel
committed
39e17a2
Synchronize pool prefill admission across ranks
lausannel
committed
70e7b42

Commits on May 1, 2026

Optimize prefix reuse rank affinity lookup
lausannel
committed
2487e74
Avoid tiny back-to-back prefill batches
lausannel
committed
d6bddf6

Commits on May 3, 2026

Make prefix admission page estimates reuse-aware
lausannel
committed
6b9a9e6

Commits on May 4, 2026

Add cached token usage reporting
lausannel
committed
604f452
Refactor prefix reuse worker logic
lausannel
committed
75013bc

Commits on May 5, 2026

Refactor prefix cache model adapters
lausannel
committed
e6e17de
Milestone 1: add dual prefix cache coordinators
lausannel
committed
ee313d7
Milestone 2: route prefix reuse through host coordinator
lausannel
committed
c9a88ef
Milestone 3: add GLM5 prefix-aware offload plumbing
lausannel
committed
393b850
Milestone 4: add GLM5 prefix-aware suffix prefill
lausannel
committed
dd194a9
Milestone 5: add GLM5 exact full-hit prefix replay
lausannel
committed
95df342
Milestone 6: enable GLM5 DSA prefix reuse gate
lausannel
committed
7616e54
Document local import failure policy
lausannel
committed
7889749
Extract shared MLA prefix replay helpers
lausannel
committed
017da9a
Add DeepSeek MLA prefix replay support
lausannel
committed
829a514
Add Kimi MLA prefix replay support
lausannel
committed
c230f60
Add MiniMax GQA prefix replay support
lausannel
committed
3c374c4
Enable prefix reuse gating for supported model backends
lausannel
committed
8765d2a
Restore GPT-OSS logits casting behavior
lausannel
committed
5a61153

Commits on May 6, 2026

Stop tracking local agent instructions
lausannel
committed
601367e
Merge remote-tracking branch 'origin/main' into feature/add-staged-page-level-prefix-reuse
lausannel
committed
59db642
Centralize MLA prefix reuse adapters
lausannel
committed
f32322e
Stop tracking prefix cache eviction plan
lausannel
committed
b852c8e
Route MLA prefix reuse through prefill backend
lausannel
committed
09667f9
Add first-class attention forward metadata
lausannel
committed
02bbdb6
Add forward metadata binding context
lausannel
committed
f8973f1

Commits on May 7, 2026

Build prefill forward metadata explicitly
lausannel
committed
638f98f
Prefer explicit metadata in prefix cache wrappers
lausannel
committed
7b1018b
Introduce prefix-aware attention backends
lausannel
committed
bc4de6b
Consolidate MLA prefix metadata adapters
lausannel
committed
e7daa7d
Add GPU paged-KV prefix extend path
lausannel
committed
524c330
Deduplicate prefix extend replay helpers
lausannel
committed
182c029
Fix decode GPU KV residency before metadata sync
lausannel
committed
5bd4cbb
Restore synchronous host KV load wait semantics
lausannel
committed
2bd2786
Align GPT-OSS GQA decode call signature
lausannel
committed
d0534af
Fix prefix prefill microbatch sampling metadata
lausannel
committed
4e327d9

Commits on May 10, 2026

Remove experimental GPU prefix extend path
lausannel
committed
e2ab089
Extract shared MLA prefix absorb helpers
lausannel
committed
79d9d68
Materialize prefix reuse pages into GPU paged KV
lausannel
committed
7f28412
Fix prefix GPU materialization cleanup flag
lausannel
committed
f5eed99
Use scoped GPU KV managers for prefix materialization
lausannel
committed
99e0610
Reduce redundant prefix materialization validation
lausannel
committed
96e0b99
Remove decode scratch memory reservation
lausannel
committed
2c826d2
Remove prefix metadata runtime validation from forward path
lausannel
committed
7a918b6
Allow dual prefix caches to use independent host page IDs
lausannel
committed
bf6a94c
Limit prefix GPU materialization to GQA replay
lausannel
committed
625f692

Commits on May 11, 2026

Use GPU paged materialization for MLA prefix prefill
lausannel
committed
37f0eeb
Pass resolved host KV budget to workers
lausannel
committed
ae72ce0
Allocate host KV after resolving server budget
lausannel
committed
bb5a33d
Revert "Allocate host KV after resolving server budget"
lausannel
committed
c2e993e
Revert "Pass resolved host KV budget to workers"
lausannel
committed
fef641d
Support dual host prefix GPU materialization
lausannel
committed
6f25264
Pass prefix materialization through GQA replay
lausannel
committed
8b09d75
Fix MiniMax MoE decode buffer growth
lausannel
committed
4dd12a1
Merge remote-tracking branch 'origin/main' into feature/add-staged-page-level-prefix-reuse
lausannel
committed
06356a5
Use standard prefill batching for prefix reuse
lausannel
committed
5456e15

Commits on May 12, 2026

Commits on May 13, 2026

Revert MLA extend refactor
lausannel
committed
0092425
Use FlashInfer for MLA prefix extend prefill
lausannel
committed
08ac13c
Remove legacy MLA prefix replay fallback
lausannel
committed
fe1f6fb
Route MLA full-hit prefill through FlashInfer
lausannel
committed
0f54d88
Stream prefix page materialization by layer
lausannel
committed
b1191f6
Use batched FlashAttention for GQA prefix extend prefill
lausannel
committed
6108559
Rename FlashInfer MLA extend prefill module
lausannel
committed
dbe609d
Shorten FlashInfer MLA extend module name
lausannel
committed
a4b2fa4

Commits on May 23, 2026

Merge main and remove obsolete prefix cache coordinator state
lausannel
committed
cfd8e55

Commits on May 24, 2026

feat: support full-history SWA host KV (#159)
lausannel
authored
7493c12
Add host prefix cache coordinator
lausannel
committed
07ea675
Add non-mutating host prefix lookup estimate
lausannel
committed
f02b501
Add host prefix cache metadata eviction
lausannel
committed
617b540
Fix host prefix eviction protection test
lausannel
committed
8661520
Add host prefix cache clear API
lausannel
committed
7e85dd7
Track host prefix pending loads
lausannel
committed
271ad5d
Expose host prefix cache lifecycle stats
lausannel
committed
48739a0
Add namespace-scoped host prefix clearing
lausannel
committed
2b4e185
Factor shared memory utilities
lausannel
committed
87a861e
Add host prefix page materialization load API
lausannel
committed
1b594e1
Add prefix reuse GPU materialization helper
lausannel
committed
d2fbe1b
Share host page materialization copy logic
lausannel
committed
24a1722
Batch GQA full-hit prefix decode
lausannel
committed
9e0543a
Guard prefix materialization attachments during load
lausannel
committed
4ba65e1
Materialize single KV group lookup results
lausannel
committed
b7c668f
Fix prefix materialization load guard ordering
lausannel
committed
c6232b5

Commits on May 25, 2026

Format added prefix cache files
lausannel
committed
6a77d21
Unify full-hit prefix prefill planning
lausannel
committed
bfb7031
Clamp prefix materialization for full hits
lausannel
committed
da5eb39
Route full-hit prefix reuse through extend prefill
lausannel
committed
d5bd9c7
Format added prefix-cache files
lausannel
committed
1ccda00
Update full-hit legacy metadata expectation
lausannel
committed
21776b4
Cover one-token full-hit prefix planning
lausannel
committed
235743b
Remove unused host prefix region identifiers
lausannel
committed
7849309
Simplify prefix reuse metadata flow
lausannel
committed
420c97d
Simplify FlashInfer MLA wrapper import
lausannel
committed
790f6a9
Remove redundant GLM host KV view check
lausannel
committed
f0df1db
Prune redundant MLA prefix adapter arguments
lausannel
committed
505edfa
Remove redundant prefix reuse validation helpers
lausannel
committed
93bb77c
Derive host prefix cache runtime configuration
lausannel
committed
5810ff6
Initialize host prefix cache coordinator lifecycle
lausannel
committed
63a8d64
Add prefix cache prefill lookup helpers
lausannel
committed
6a2a31a
Propagate prefix cache usage from workers
lausannel
committed
df4f387
Add aligned prefix cache commit helpers
lausannel
committed
e60db60
Add grouped prefix materialization bundle
lausannel
committed
6ab424c
Collect Host KV pages for prefix commits
lausannel
committed
a5b86dc
Estimate prefix cache hits during prefill
lausannel
committed
d10ed59

Commits on May 27, 2026

Fix prefix prefill range offload metadata
lausannel
committed
e900865
Align prefix reuse prefill planning to page boundaries
lausannel
committed
fb76049
Clarify prefix reuse planner boundaries
lausannel
committed
36b912a
Attach shared prefix pages to Host KV views
lausannel
committed
028f207
Wire prefix cache lookup into prefill admission
lausannel
committed
3375319
Publish aligned Host prefix cache pages
lausannel
committed
d1d9dac
Factor prefix cache commit publishing helpers
lausannel
committed
4d1cd5e
Use explicit attributes in prefix reuse paths
lausannel
committed
85246d0
Release GPU KV cache before prefill reload
lausannel
committed
9836455
Avoid reinitializing oversized decode KV cache for prefill
lausannel
committed
01da6ed
Report prefix cached tokens from owner ranks
lausannel
committed
d701851
Overlap prefix page materialization with layer compute
lausannel
committed
43de419

Commits on May 28, 2026

Integrate host prefix eviction with multi-rate KV groups
lausannel
committed
e2a33fc
Cover prefix eviction and multi-rate commit helpers
lausannel
committed
e119073

Commits on May 30, 2026

Unify prefix cache allocation and full-hit semantics
lausannel
committed
09d4c62
Remove prefix prepack metadata adapter
lausannel
committed
b4fee8f
Fix prefix eviction helper tests
lausannel
committed
a051282
Avoid prefill scheduler stall under prefix cache pressure
lausannel
committed
d5c310b
Use allocation eviction for prefix cache host pressure
lausannel
committed
0aa4d9b
Coordinate prefix cache eviction across host ranks
lausannel
committed
460bc7c

Commits on May 31, 2026

Simplify prefix cache runtime coordination
lausannel
committed
9254c4e

Commits on Jun 4, 2026

Commits on Jun 5, 2026

Store prefix cache attachments per coordinator
lausannel
committed
17ce0a6
Fix prefix coordinator cross-process eviction test
lausannel
committed
b25fe55

Commits on Jun 7, 2026

Commits on Jun 8, 2026

Add prefill reentry diagnostics
lausannel
committed
f98991f
Avoid per-node prefix eviction filtering
lausannel
committed
7200ecd
Release completed decode KV before prefill admission
lausannel
committed
af4b16e
Unify decode completion KV release
lausannel
committed
0d58ecf
Retain inserted prefix page ranges
lausannel
committed
6b11a92
Retain committed prefix pages by id
lausannel
committed
e817e87
Scope prefix materialization to prefill microbatches
lausannel
committed
961d545
Fix prefix reuse microbatch planner import
lausannel
committed
be35967
Cap prefix reuse prefill microbatch size
lausannel
committed
2f26170
Release prefix prefill GPU memory between microbatches
lausannel
committed
cdd1516
Release prefix materialization GPU cache before reuse
lausannel
committed
9c21b5e
Avoid clearing non-null core GPU manager binding
lausannel
committed
e723ab0
Cap prefix prefill microbatches by GPU page footprint
lausannel
committed
1cd6e15
Align prefix prefill microbatching with full-context token cap
lausannel
committed
f14bdff
Release prefix materialization at prefill microbatch boundary
lausannel
committed
4750e6d
Fix prefix materialization cleanup invariant test
lausannel
committed
cb21896
Release CUDA cache after prefix materialization reset
lausannel
committed
39aa355
Cap prefix-hit prefill materialization microbatches
lausannel
committed
44b6ae6
Stream prefix-hit prefill KV materialization by layer
lausannel
committed
a5f9bc0
Align prefix commit capacity tests with delta entries
lausannel
committed
52f22d1
Retire prefill offload tensors per microbatch
lausannel
committed
7c47073
Fix paged GQA extend KV length metadata
lausannel
committed
e4657dd
Align GQA prefix materialization with FA paged KV pages
lausannel
committed
cd87729
Delay prefix materialization reuse until prefill offload retires
lausannel
committed
c8b5651

Commits on Jun 9, 2026

Optimize host prefix cache commit path
lausannel
committed
b7a1348
Pass FA3 new-KV boundaries for GQA extend prefill
lausannel
committed
49bd39b
Avoid FA3 new-KV boundaries without append tensors
lausannel
committed
c00e7fe
Add prefix extend debug synchronization
lausannel
committed
c6caaa5
Add GPT-OSS MoE debug synchronization
lausannel
committed
47a4181
Add GPT-OSS prefill fused gate fallback
lausannel
committed
e7fea21

Commits on Jun 10, 2026

Defer tiny prefix-cache prefill waves
lausannel
committed
1384756
Add prefix cache pin management endpoints
lausannel
committed
8999481
Optimize prefix cache page release tracking
lausannel
committed
e93f483
Stop committing decode tokens to prefix cache
lausannel
committed
4dd78cb
Make prefix prefill admission reuse-aware
lausannel
committed
d8477cd
Fix prefix admission scheduler test expectations
lausannel
committed
18fda97
Reserve resident prefix pages during prefill admission
lausannel
committed
4c4b3dd
Charge selected shared prefix pages in prefill admission
lausannel
committed
2b216e2

Commits on Jun 11, 2026