kv-cache : spend the partial residency budget on the slowest link first

[Piggidragon]

Overview

Depends on #67. The base repository has no branch to target, so this branch includes the #67 commits. Merge #67 first.

Builds on the residency PR. There, --kv-gpu-layers spreads round-robin over the owning devices, so each device gives up the same number of layers. That is the right default only when the devices are alike.

A device-resident layer saves the host-to-device transfer it would otherwise cost every token, and that transfer costs most where the link is slowest. On an asymmetric pair the two links differ by a lot:

llama_pick_gpu_resident_layers: CUDA0: host-to-device 24.5 GB/s   (RTX 4070, gen4 x16)
llama_pick_gpu_resident_layers: CUDA1: host-to-device  3.3 GB/s   (RTX 3060, gen3 x4)

This PR measures the host-to-device bandwidth once at startup, using the storage the cache is delivered from (pinned or pageable), and spends the budget on the slowest device first. Devices within 15% of each other stay in one group and keep the round-robin, so a symmetric machine sees no change.

Memory budget

The residency set is bounded by the free memory of each device. The first version of the bound kept only 1/8 of free memory back. The compute buffers and recurrent state allocated after the cache did not fit in that, so a request that reached the bound failed to start (failed to allocate compute pp buffers). It also sized layers with its own formula, which charged SWA layers the full context (about 85x too much for Gemma 4 at 131k).

Now:

1. With --kv-gpu-layers set, the context first builds the caches without KV storage and runs the existing size-only reserve (the path the fit dry run uses), which gives the compute buffer size of each device.
2. In that pass each cache records the K/V tensors of the layers it owns, so the picker sizes the real tensors. SWA cell counts, DSV4, MLA, shared layers and the tensor split come from the caches, not from per-arch code.
3. Budget per device = free - compute buffers - non-KV device state (recurrent) - free/8 margin for runtime pools.
4. The fit dry run keeps attention KV on the host. Fit plans the weights, and residency uses the free memory that fit leaves.

Not reserved: memory that other processes or a later context (draft/MTP) allocate.

Testing

Ordering, Qwen3.8-27B-UD-Q5_K_M, 15216-token prompt, -c 20480 -n 128 -ngl 99, f16 cache, -sm layer -nkvo --kv-cpu-pinned --recurrent-state-offload. 16 owned attention layers, 8 per GPU. "round-robin" is #67, "slowest first" is this PR.

--kv-gpu-layers	pp round-robin	pp slowest first	tg round-robin	tg slowest first
0	611.67	611.23	4.34	4.34
2	620.45	627.17	4.79	5.20
4	629.41	644.61	5.35	6.50
8	649.38	979.28	6.97	12.97
16 (all)	979.95	977.19	17.77	17.77

At 8 layers, where the 3060 holds none of the host-resident cache, both orders use the same memory, but generation is +86% faster and prefill +51% faster.

Memory bound, RTX 4070 + RTX 3060, 17k-token prompt:

Model, context, request	Before	After
Qwen3.8-27B Q5_K_M, 131k, kvgl 16, -fit off	compute buffer out of memory on CUDA1	6 layers, runs, 652/701 MiB free
Qwen3.8-27B Q4_K_XL, 131k, kvgl 999, default fit	-	10 layers, runs, 794/605 MiB free
Gemma 4 26B (iSWA), 131k, kvgl 999	-	all 30 layers, runs
Nemotron-H 30B, 262k, kvgl 999	-	all 6 attention layers, runs

test-llama-archs --test-kv-residency passes, with an added capacity case for memory the context reserves. These also pass: test-phase-aware-workspace, test-live-context-workspace, test-recurrent-state-rollback*, test-save-load-state and test-llama-archs -s 1.

Not done here

• Deriving --attn-split or --tensor-split from the same measurement. It would change the reduction order for every multi-GPU user, so it waits until it has run on more than one machine.
• Retrying with fewer layers when an allocation still fails.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - implemented by an agent on my instruction, see the Assisted-by: commit trailers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TrrQUGmn1qZsxKF3p9TvYh

[GenerelSchwerz]

Automated preliminary review by Codex; the repository owner plans a separate manual review.

Verdict: FAIL.

Blocking

• The current live-base synthetic merge does not compile at src/llama-model.cpp:2532: specialized_placement is undeclared. Compiler output from Linux x64, Linux ARM64, Windows ARM64, and WASM all reaches this diagnostic.
• llama_dev_h2d_bandwidth() always allocates its source with ggml_backend_dev_host_buffer_type() (src/llama-model.cpp:2245), while the real KV cache uses pageable CPU storage unless kv_cpu_pinned is enabled (src/llama-kv-cache.cpp:21; the default is false at src/llama-context.cpp:4077). The ranking therefore measures a different transfer path for the default configuration.
• The slow-link-first loop at src/llama-model.cpp:2338 spends every eligible layer on the slowest group before moving on, but considers neither device free memory nor each layer's actual KV size/layout, including SWA versus full-cache and MLA K-only cases. Allocation is attempted only later and fails outright at src/llama-kv-cache.cpp:382; there is no retry or redistribution, so this policy can turn an otherwise working configuration into a startup allocation failure.

Will slow review

• The automatic multi-device placement policy at src/llama-model.cpp:2316 has no linked issue or design agreement. The added test at tests/test-llama-archs.cpp:1047 only proves one MTP layer leaves host storage; it does not cover asymmetric link ranking, default pageable storage, memory pressure, SWA/MLA, or fallback behavior. This PR is also stacked on kv-cache : resolve the partial KV residency set once for the model #67, which has a FAIL review verdict.

Developmental progress

• Actual residency is now reconciled back into cparams.kv_gpu_layers at src/llama-model.cpp:2833, and MTP coverage was added at tests/test-llama-archs.cpp:1047. Those are useful fixes, but integration and policy correctness remain unacceptable.

Reviewed head: fe9617e