sched, ggml-meta : pipeline the host KV delivery under split mode tensor - #92
Closed
Piggidragon wants to merge 53 commits into
Closed
Piggidragon wants to merge 53 commits into
Piggidragon wants to merge 53 commits into
Conversation
With --no-kv-offload the attention history lives in host RAM and reaches the accelerator on every decode token. The scheduler issued that transfer on the consumer's own stream immediately before the kernels that read it, so a token cost copy + compute in series. The bytes and the attention operations are unchanged; only the point at which the transfer is issued moves. Greedy output is byte-identical to the ordered path -- verified against a build without these changes, at every look-ahead tested, single GPU and layer-split across two. Three pieces, each load-bearing: - ggml_tensor::stable_prefix records how many leading bytes of a tensor's storage the graph about to run will not write. The KV window is not stable for a whole graph -- a CPU split writes this ubatch's rows into it between one layer's attention and the next -- but everything below the lowest written row is, and at decode depth that is essentially all of it. llama_kv_cache sets it from apply_ubatch(), before the graph is built and allocated, so the plan and the deliveries are decided against the same write position even when the graph is reused; build_graph_shift() clears it. - A staging ring the graph allocator cannot reach. ggml-alloc may recycle a graph-owned input copy after its last graph-level consumer while a look-ahead transfer is still in flight. The scheduler allocates the ring itself and points the staged copies at it before allocation; a ready/release event pair per slot carries the handover in each direction. Every eligible accelerator gets its own ring, cursor and budget, so a layer-split model pipelines on each device and a device with no room falls back alone. - A look-ahead that stays clear of the ring's tail. A delivery L splits ahead recycles the slot of the split L - n_slots back, so n_slots == L + 1 recycles the split just enqueued and still running. The ring keeps two slots of margin, deliveries are issued after a split is enqueued rather than before, and slot recycling is ordered stream to stream rather than through the host. Each of those three alone costs the entire gain while still producing correct output. --kv-pipeline-depth N, default 1, 0 restores the ordered path exactly. It only engages where a host-resident cache produces the deliveries. Because a host-resident cache exists to keep device memory free, the staging is capped outright by --kv-pipeline-budget (default 128 MiB per device) rather than by a fraction of what happens to be free. A ring is (N + 2) slots of one attention layer's K and V over the whole context, so it grows with the context: 27 MiB at 4k, 213 MiB at 32k, 1.7 GiB at 256k. Past the cap the scheduler declines and keeps the ordered path, and declining costs nothing -- the check runs before anything is allocated, the decision is latched because a context only grows, and the transfer backend is created lazily and released with the ring. Single GPU (RTX 4070, Qwen3.8-27B-UD-IQ2_M, -nkvo --kv-cpu-pinned, q8_0 K/V), A/B/A/B with reversed arm order: depth ordered pipelined gain 4,096 31.7324, 31.7363 37.0889, 37.0741 +16.9% 16,384 19.6765, 19.6854 31.5352, 31.5807 +60.4% 32,768 13.0264, 13.0254 15.5325, 15.5329 +19.3% (needs a raised budget) Server decode behind an 18,422-token prompt: 18.468 -> 30.685 t/s, +66.2%. Two GPUs (RTX 4070 + RTX 3060, Qwen3.8-27B-UD-Q5_K_M, -sm layer), both rings engaged: 13.06 -> 18.05 t/s at 4,096 and 6.86 -> 9.75 t/s at 16,384. The gain narrows with depth because compute is a shrinking share of the token, so there is less to hide the copy behind. That is arithmetic, not an implementation limit, and more look-ahead makes it worse rather than better. Tensor parallelism keeps the ordered path: the scheduler sees one meta backend there and the ring is a byte arena, while a meta buffer places tensors as per-device slices rather than at offsets. It declines rather than staging into something it cannot address. docs/kv-transport-pipelining.md carries the design, the numbers and the limits; docs/repro/ carries the scripts that produced them. Assisted-by: Claude Opus 5
…cripts llama-bench does not expose --kv-cpu-pinned or --recurrent-state-offload the way llama-server does, so two of the reproduction scripts passed flags the binary rejects. They now probe --help and pass only what it takes. The feature doc gains what is actually left: why -sm tensor keeps the ordered path (no events in the meta layer, and a ring that is a byte arena while a meta buffer places tensors as per-device slices), the correctness problem underneath it that is not this feature's, and the rest of the open list -- the transient device-memory peak, the --kv-gpu-layers comparison, and the exactness harness's dependence on baseline determinism it does not have. The decline for a backend that cannot record events now says so by name rather than falling into the generic "no backend supports" line, because the backend it catches is the meta backend and the next person to look will want to know that. Assisted-by: Claude Opus 5
The host-time breakdown the feature doc describes had no code behind it: the counters existed but nothing accumulated or printed them. GGML_SCHED_TRANSPORT_DEBUG=2 now reports the split loop as a mean over each 128 graphs, with the bytes the ordered path still moves and why the look-ahead stopped; =3 names the tensors that are still on it. That is what found the rest: 40 blocking copies a token moving 0.4 MiB, 32 of them the device-to-host KV store. The budget warning now reports what the ring costs at the full context next to what it costs now, so --kv-pipeline-budget can be sized against the number that matters. It is still applied per graph: enforcing the projection would refuse the ring for every large -c even when the window never gets near it. llama-bench gains -kvcp and -rso. Without them a host-resident run measures something else entirely -- 9.02 against 19.43 t/s ordered at 16,384 -- and the repro scripts had been silently dropping both since llama-bench lost them. The exactness harness gives every task a nonce derived from its own name and length, so no two share a prefix the server can restore, and fails a task whose prompt_n says one was reused anyway. Assisted-by: Claude Opus 5
…t it does GGML_SCHED_TRANSPORT_DEBUG=3 now reports what each remaining blocking copy cost, not just its name. It turns out one of them is almost all of it: attn_inp_k_rot, 256 KiB, 18 us on the ordered path and 3.4 ms behind one split of look-ahead. That is the copy engine, not latency. A blocking copy waits for the deliveries already queued on it, and two staged splits at 22.0 GB/s is 3.6 ms. Issuing the delivery in pieces does not help, the engine is FIFO across streams. Putting the copy on the consumer's stream so the host never blocks moves the time into the consumer wait and leaves throughput alone. The doc records both, so the next person does not spend the afternoon on it again. Assisted-by: Claude Opus 5
records@18432 was giving different answers across otherwise identical N = 0 runs, which made it useless as a gate and looked like the pipeline breaking exactness. It is not the task: asked on its own with the prompt cache off it returns the same hash three times running, at -c 32768 and at -c 65536. It is the harness. All eight tasks share one server with prompt caching on, and records@18432 is about 29.6k tokens with a task of about the same size ahead of it, so the two do not both fit in a 32,768 cache and placement depended on what was still resident. The nonce stops a prefix being restored, it does not stop the pressure. cache_prompt=false does. Two independent N = 0 passes now agree on all eight tasks, and N = 0, N = 1 and N = 4 agree on all eight. Assisted-by: Claude Opus 5
Restrict staging to annotated CUDA inputs, make backend decline complete, re-evaluate budgets, freeze scheduler configuration, and preserve tensor layout. Add regressions for prefix changes and fallback behavior. Assisted-by: OpenAI Codex
Assisted-by: OpenAI Codex
The ring laid its entries out with ggml_nbytes() and bound them by writing data and buffer directly. A buffer type may ask for more than ggml_nbytes() for a tensor -- CUDA does for a quantized one, and MMQ clears that padding -- so an entry could reach into the next one. Entries are now sized with ggml_backend_buft_get_alloc_size() and bound with ggml_backend_tensor_alloc(), which also gives them the buffer's own initialization and its bounds check. test-alloc gets a dummy buffer type whose get_alloc_size exceeds ggml_nbytes, and a two-entry ring test that checks the entries stay inside the ring and out of each other, that every byte of an entry is delivered once from the matching source offset, and that nothing waits on an event before it is recorded. llama-bench takes -kvpb/--kv-pipeline-budget and reports it. The repro scripts pass 512 and now fail closed: they refuse a build without -kvcp, -rso or -kvpb instead of dropping the option, and every arm propagates its status. The llama-bench table in the doc was measured before the budget existed, so it says so, and the 32,768 row is marked as needing a re-measurement. Assisted-by: Claude Opus 5
The rings are laid out and allocated before the graph is, so a device that can hold the graph alone but not the graph next to a ring turned into GGML_STATUS_ALLOC_FAILED. The configuration is locked by then, so the caller could not turn the ring off and retry either. When graph reservation fails the rings are now released and the reservation is retried once on the ordered path, and that scheduler keeps the ordered path from then on. A plan over a split list with no inputs left input_staged unallocated and passed it to memset, which UBSan reports even at size 0. Such a plan stages nothing, so it now returns after putting every split back on the ordered path. test-alloc gets a device capacity on the dummy backend and a test that sizes it to hold the graph or the ring but not both. The prose and public comments this branch added were hard-wrapped to a fixed column, against the repository rule. They are unwrapped, one sentence per line. Assisted-by: Claude Opus 5
The llama-bench table predated the budget and said so, and the context sweep and the exactness gate were last run before the ring allocation and binding changed. All three are re-run on an RTX 4070 with a CUDA build of this head, at --kv-pipeline-budget 512. Greedy server output is identical at depth 0, 1 and 4 across all eight tasks. Throughput is +16.1% at 4,096, +56.6% at 16,384, +20.1% at 32,768 and +13.4% at 65,536, for +28, +104, +206 and +410 MiB of device memory. The 131,072 and 262,144 arms are not re-measured and say so. The server table is replaced with the four 18,432-prefill tasks of the exactness gate, which is what this head was actually run on; the copy/compute breakdown keeps its earlier numbers and says which head they came from. Assisted-by: Claude Opus 5
… MiB The context sweep is measured with the budget raised, so its deep rows read as default behaviour when they are not: past 20,556 rows of window the default declines and those depths stay ordered. Say so, and add a finer sweep that puts the peak at 16,384 rows and shows the gain per MiB falling off as 1/rows^2 above it. The peak is where copy and compute are equal, and the ring size there works out to (n_slots / n_attn) * compute * BW - the bytes per row cancel, so the budget is quant-invariant. That predicts 101 MiB against the 102 MiB measured, which is what the 128 MiB default is sized against. Assisted-by: Claude Opus 5
A window over several streams is one view of a tensor whose streams sit end to end, and both the prefix and the delivery treated it as one flat byte range. That made the lowest-writing stream cap the stable prefix for every stream above it, and it copied the cells between one stream's window and the next, which the graph never reads. The prefix is now counted within a stream, and a staged input whose last dimension indexes streams is delivered as one range per stream. A window over one stream keeps the single flat range it had, so single-sequence timing and bytes are unchanged. Behind 8 slots of a non-unified cache this takes decode from 72.18 to 112.70 t/s, against 70.61 ordered. At one slot it measures 35.31 against 35.32 before. test_transport_multi_stream_ranges pins the delivery: every stream's window covered once from its own source offset, the unread cells between them never moved, and the early and late bytes split as the prefix says. Concurrent slots cannot be gated on output the way one sequence can - their batching varies between runs, so the same depth gives different greedy output - which is why this is a unit test. Assisted-by: Claude Opus 5
The q4_0 arm of the context sweep moves the peak from 16,384 rows to 32,768 and leaves the ring at it at 108 MiB against 102, which is what "the bytes per row cancel" claims. It was derived before and is measured now. Add the parallel numbers and say what the 8-slot row depends on: run on its own the unified ring fits, and only after a sweep has allocated for 1, 2 and 4 slots in the same process does the headroom guard refuse it. Assisted-by: Claude Opus 5
ne[2]*nb[2] is one KV cell, not the window: attention permutes the window before reading it, so its rows sit on dimension 1. A ubatch over several streams delivered one cell per stream and attention read whatever the ring slot held before. The test built the window in the pre-permute shape, so it passed. Assisted-by: Claude Opus 5
A staged delivery reads its host source after the call that issued it returns. The stable prefix keeps the host off that source within a graph, but the next graph writes wherever its own ubatch lands, so a recycled cell below the previous window could be rewritten under an in-flight copy. The ordered path gets this from its blocking copy, once per split. Assisted-by: Claude Opus 5
Freeing the ring already goes through the backend that allocated it, so that backend is alive here and its kernels may still be reading the slots. Teardown skipped the wait and released the memory under them. Assisted-by: Claude Opus 5
Assisted-by: Claude Opus 5
The range check left only the inner loop and inserted the values anyway. Assisted-by: Claude Opus 5
The scheduler does not own its backends, and llama_context declares its scheduler before them, so member destruction frees the backends first and sched->backends[] dangles by the time the ring is freed. A slot's release event is recorded past every kernel that reads it and dispatches through the device, so waiting on it orders the free after the consumer without touching the backend. Assisted-by: Claude Opus 5
The per-stream loop in both delivery paths is what that helper does, and it lets a backend with a 2d set issue one copy instead of one per stream. Assisted-by: Claude Opus 5
The count is per stream, so the contract has to name the stride it goes with rather than leave it as "the first nbytes". Assisted-by: Claude Opus 5
The server's batching varies between runs, so it cannot gate concurrent sequences. llama-parallel seeds its client schedule, so it can, and its clients ask different questions: with one shared prompt every stream holds the same bytes and a cross-stream read stays invisible. Fails at depth 1 on the commit before the multi-stream span fix. Assisted-by: Claude Opus 5
Assisted-by: Claude Opus 5
The parallel table was taken before the multi-stream span fix, so it reported a delivery that moved a fraction of the window. Gates 1, 2 and 5 are re-run here, and the numbers that are still from an earlier head now say so. Assisted-by: Claude Opus 5
A graph that stages nothing kept the ring and the second device context for the life of the scheduler. Give them back, from every path that ends with nothing staged. A window wider than the ring holds frees the ring and allocates it again, which a prefill did on nearly every ubatch. Allocate a slot in powers of two, capped by the full context, the budget and the headroom check, so a 16k prefill reallocates 6 times rather than 32. The decline decision still goes by what the graph needs. Wait at a graph boundary only for a ring that delivered, not for every ring that has a transfer backend. Assisted-by: Claude Opus 5
The platform check it had can never fail on a 64-bit size_t, so any value was accepted and a large one was silently the same as 0. Cap it at 65536 MiB in all three places that parse it. Assisted-by: Claude Opus 5
A staged delivery reads the host cache long after the decode that issued it returned, so a memset of those buffers races it. llama_memory_clear holds no context and cannot wait, so say so on the public function, and wait where a context is at hand. Assisted-by: Claude Opus 5
clear(data=true) memsets the buffers while a decode can still be reading them: a staged delivery for a host-resident cache, the graph itself for a device-resident one. The second one is not new and is easy to hit - llama_decode followed by llama_memory_clear(mem, true) changed the logits of that decode on every trial. Every memory type already passes the context down through init_update, so the caches keep it from there and wait on it before the memset. Assisted-by: Claude Opus 5
Keep a staged input's producer on the CPU or on the consumer itself, keep the transfer context over a graph that stages nothing, and stop asking a device that cannot give one. Say in the header and the docs that the destination is CUDA-only, and that a delivering graph costs the pipelining of n_copies > 1. Assisted-by: Claude Opus 5
- Fix kv_pipeline_budget_mib handling to preserve negative sentinel value (-1 = not set, 0 = no cap) - Add null check for transport backend before synchronizing - Disable pipelined transport when n_copies > 1 to avoid conflicts with pipeline parallelism - Expose set_lctx() method in kv_cache for hybrid index context management Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PbyQFddbGJQ6MpRwUH2Rtj
The ordered copy of a multi-stream window landed in llama/dev on its own. It brings its own copy of the range geometry, which this branch already had in a form that also carries the copy's stride and the stable prefix, so drop the duplicate and keep the one the ring uses. The ordered copy now steps by the copy's own stride. A copy is laid out for the ring only while it is staged, so the two agree today, but reading it from the copy is what keeps them agreeing. The dummy backend's set_tensor_async records a delivery and now performs it as well when the backend is asked for real memory, so the test that checks the bytes of the ordered copy still sees them. ggml_new_tensor_impl needs a brace for the array inside the union. Assisted-by: Claude Opus 5
The ordered copy of a multi-stream window landed in llama/dev, so the baseline the parallel table compares against moves. Only the ordered column of a cache split into streams changes; the rest is within noise. The pipeline is worth +8.5%, +13.8%, +19.8% and +26.0% at 1, 2, 4 and 8 slots over a non-unified cache, against +72% and +73% read off the old baseline, and packing the ring is worth +25.7% at the default budget rather than +49.6%. The single-sequence table does not move at all: a unified window is one range and has no gaps to have been wasting. Gate 5 is re-stated against a build of llama/dev rather than an earlier head of this branch, which is the stronger comparison. Assisted-by: Claude Opus 5
Also gives the stable prefix one value per stream, so a slot that was just reset no longer caps the early region of every other stream. Assisted-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JycLdWs6KRgizbdnNAfZqM
Assisted-by: Claude Opus 5
github-actions
Bot
added
documentation
A copy of an input is named "<backend>#<source>#<copy>" in a name field of fixed size. With many devices the backend label of the meta backend lists all of them and the source name is what gets cut, so a consumer can no longer tell which tensor the copy was made from. Cut the label instead. Assisted-by: Claude Opus 5
A KV cache in host memory reaches attention as a scheduler copy, which is a leaf in the compute buffer. Such a leaf never reached the device split-state callback and fell through to MIRRORED, while the queries stayed split by head: each device then attended heads whose keys live on the other device. With more than one KV head that aborts in the FlashAttention kernel, or returns wrong output where the query split happens to stay a multiple of the KV head count. Offer a copied-in leaf to the callback under the name the graph gave it, and let the callback recognise the cache there. The cache folds its heads into one flat axis but the copy arrives permuted, with the heads on an axis of their own, so its segments and granularity are rescaled to whole heads. Assisted-by: Claude Opus 5
The KV granularity was derived from the query granularity through n_gqa, which assumes the V side has the head size of the K side. Count whole KV heads and scale each side by its own head size. Assisted-by: Claude Opus 5
A fused QKV puts Kcur and Vcur in a strided view, which a host-resident cache reads back through the meta buffer. The rows split, so the chunk splice could not express the read. Assisted-by: Claude Opus 5
A linear-attention op packs the state it writes back together with its output, so that split does not line up with the one a host-resident state expects. On device the two orders agree; in host memory they disagree, and the split state of the fused op no longer resolves. The state is small next to the attention cache that -nkvo exists to move, so keep it device-resident. Assisted-by: Claude Opus 5
Run the tensor-split architecture matrix a second time with the cache in host memory, and add an 8-device CI run, where the scheduler copy name is long enough to be truncated. Assisted-by: Claude Opus 5
The strided head split path ran only for a single stream, so a host-resident cache built with --parallel N fell through to the chunk splice, which cannot express that write. Loop over ne[3] and offset each side by its own stride. Assisted-by: Claude Opus 5
A split that ends with a view of a host tensor left the subgraph bookkeeping short of the node count and aborted. That happens with more than one cache stream, which the test matrix now covers. Also make the scheduler copy name a stated contract instead of a grammar that two files reconstruct on their own, note the 2d transfer fallback at both strided cache paths, say out loud that split mode tensor overrides the recurrent state placement, and record the Gemma 4 host cache accuracy gap where it is skipped. Assisted-by: Claude Opus 5
The scheduler flags a copy as an input when it keeps more than one, which hid a host-resident cache from the split state callback. A cut source name now asserts instead of naming another tensor, and memset writes a head split like set_tensor does. Assisted-by: Claude Opus 5
The ranged copy of a host-resident window calls set_tensor_async once per stream. A meta backend only takes a whole contiguous tensor, so with two or more streams it aborts. Keep the whole-span copy for a meta destination. Assisted-by: Claude Opus 5
Piggidragon
commented
Sep 14, 2026
Piggidragon
left a comment
Piggidragon
left a comment
Author
There was a problem hiding this comment.
Merged llama/dev (after #94) into this branch and ran the arch-coverage suite on CPU + RTX 4070 + RTX 3060 (CUDA, Meta tensor split). Merged instead of rebased because the branch already has merge commits carrying resolutions; the only conflict was the <filesystem> / <map> includes in test-llama-archs.cpp.
- ctest
workspace|generate-models|rollback|save-load|sched-copy: 6/6 pass test-llama-archs -s 1: 4 FAIL, all in the #94 Parallel column. llama/dev alone has 0 FAIL.
| gpt2 | Meta -nkvo -np 2 | Dense | OK (4.68e-08) | SKIP | FAIL |
| gpt2 | Meta -nkvo -np 2 -kvpd 1 | Dense | OK (4.68e-08) | SKIP | FAIL |
| starcoder | Meta -nkvo -np 2 | Dense | OK (4.68e-08) | SKIP | FAIL |
| starcoder | Meta -nkvo -np 2 -kvpd 1 | Dense | OK (4.68e-08) | SKIP | FAIL |
Same cause as in #66: on n_seq_max > 1 rows logits_dev is a 2-sequence decode (second half at pos 0), but test_parallel_seqs() uses it as the reference for one sequence at pos 0..n-1. NMSE vs CPU is OK, so the pipelined transport output itself is fine; only learned-position archs expose the mismatch. Fixing it in #66 (skip Parallel when dc.kvc.n_seq_max > 1) fixes both rows here.
| // the same copy, delivered ahead of the split that reads it | ||
| kv_config kvc_host_pipelined = kvc_host_streams; | ||
| kvc_host_pipelined.kv_pipeline_depth = 1; | ||
| dev_configs.emplace_back(devices_meta, "Meta -nkvo -np 2 -kvpd 1", LLAMA_SPLIT_MODE_TENSOR, kvc_host_pipelined); |
Author
There was a problem hiding this comment.
Parallel column FAILs on this row for gpt2 and starcoder, see the review body. Not a transport bug: test_parallel_seqs() gets 2-sequence logits as a single-sequence reference.
A multi-stream row decodes its reference as several sequences, so it cannot serve as a one-sequence reference - skip the check there. Give the check the row's offload_kqv, so it covers a host-resident cache with two streams. Assisted-by: Claude Opus 5
…port (GenerelSchwerz#39) # Conflicts: # ggml/src/ggml-backend.cpp
A meta event is one event per simple device, recorded and waited for on each device's own stream. set_tensor_async and set_tensor_2d_async take part of the window of a host cache split by head: whole cells from an offset, once per stream, as one 2d copy per stream per device. A mirrored tensor passes the offset through. Add ggml_backend_meta_init_transfer, a meta backend that does not start a communicator, for streams that only move data. Rotate the compute containers of buffers that are only sources of a graph, so tensors bound into a buffer the graph allocator does not own do not pile up. Assisted-by: Claude Opus 5
A meta backend is eligible when each of its simple backends is. Its transfer backend has no communicator, and the headroom check uses the simple device with the least free memory, since a meta buffer allocates the whole ring on every device. A device of the meta type that is not the ggml meta backend stays ordered. test-llama-archs runs the host-resident tensor split with two streams a second time at pipeline depth 1. Assisted-by: Claude Opus 5
The A/B and server gate scripts take LLAMA_KV_SM, as the parallel gate already did. Assisted-by: Claude Opus 5
Piggidragon
force-pushed
the
kv/pipelined-transport-tensor-split
branch
from
September 17, 2026 21:19
7b8fb71 to
0f0f11e
Compare
Author
|
Moved to Piggidragon#8, stacked on its parent branch there. The diff is now +298/-45 instead of +4104/-272, because the parent's commits no longer show up in it. Same commits, same result. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Pipeline the host-to-device delivery of a host-resident KV cache under
-sm tensor, the one case #39 left on the ordered path.Depends on #39 (pipelined transport) and #66 (host cache split by head). The branch merges both, so the diff against
llama/devincludes them. The change itself is the last three commits:ggml-meta : add events and a ranged write for a head-split host cacheset_tensor_async/set_tensor_2d_asyncaccept part of the window of the head-split copy (whole cells from an offset, per stream), as one 2d copy per stream per device.ggml_backend_meta_init_transferbuilds a meta backend without a communicator for the transfer streams.sched : pipeline the host KV delivery under split mode tensortest-alloccase).test-llama-archsaddsMeta -nkvo -np 2 -kvpd 1.docs : measure the pipelined transport under split mode tensordocs/kv-transport-pipelining.md#tensor-parallelism.LLAMA_KV_SMworks for the A/B and server gate scripts.Testing
RTX 4070 (gen4 x16) + RTX 3060 (gen3 x4), CUDA + NCCL, Qwen3.8-27B-UD-IQ2_M,
-sm tensor -nkvo --kv-cpu-pinned -ctk q8_0 -ctv q8_0.llama-benchtg128, A/B/A/B,-kvpb 512:At 16,384 the ordered copy is 104 ms against 31 ms of compute. That is the 3060's x4 link, so the gain is bounded by the slower link, not by the scheduling.
Output is identical at depth 0, 1 and 4 in every gate:
LLAMA_KV_SM=tensor): all 8 tasks125cb9c2082d36cf64e86551f7ef1638, the same hash as a device-resident cache; gemma-45525e3f5ac7337d7(-ts 50,50) and4f8986fb3655a567(-ts 55,45)Existing paths are unchanged: the parallel gate still gives
17f946c340db110bwith-sm noneanddb661b7a08686b97with-sm layerat depth 0 and 1.test-llama-archs -s 1passes on 2, 3 and 4 CUDA devices, andtest-allocpasses.Not done here
Requirements
Assisted-by:commit trailers.🤖 Generated with Claude Code
https://claude.ai/code/session_0145V8xvxxUQMUVbg6ZiCZMg