fix(scheduler): let decodes preempt in a prefill turn that scheduled nothing (restore-admission stall)

[voipmonitor]

Report (mrweiner, Discord). GLM-5.3-Flash Spark TP2/DCP2 with LMCache on-evict, MAX_NUM_SEQS=4, KV_CACHE_MEMORY_BYTES=3.8e9 (319 blocks), image karmic-kraken-beta-20260929-7476365edc3d8ee5 (vLLM 99cbe78, LMCache 75f2b59d). Under 12–13 agents, generation stopped from 17:29:01 to 17:38:44 UTC: Running 4, Waiting 6 (3 capacity, 3 deferred), KV 98.1%, 0 tok/s, GPUs idle, EngineCore stepping in a loop with both workers idle, num_preemptions_total 1 for the whole run, and no log lines. Aborting the waiting requests (17:38:26–17:38:39) changed nothing. The engine resumed at 17:38:45, right after the clients aborted the four running requests (all four at 17:38:44.97, 600 s after about 17:28:45, which fits a 600 s client read timeout counted from their last streamed token).

Root cause (line numbers on 99cbe78). Compute sharing is on (prefill-compute-share 0.4 from the GLM profile; his metrics show vllm:scheduler_prefill_compute_share 0.4). The loop:

1. schedule() counts any waiting request as prefill work (has_prefill_candidate), even when no running slot is free, so decode and prefill are always contended.
2. PrefillComputeShareController.record() lets prefill's virtual runtime trail decode's by at most one charge (compute_fairness.py:323). Because prefill never runs, it stays behind, and select() returns "prefill" for every step (compute_fairness.py:241).
3. In that prefill turn the waiting pass stops at once (num_running >= self.max_num_running_reqs, scheduler.py:1360), and decodes run only in the leftover pass schedule_running_requests("decode", allow_preemption=False, ...) (scheduler.py:2037-2042).
4. When a decode needs a KV block from a full pool, that pass reaches if not allow_preemption: break (scheduler.py:1187). No request is scheduled or preempted. An empty step records no compute time, so the controller state does not change and the next step is the same prefill turn. This repeats indefinitely and logs nothing.
5. Because the waiting pass never runs, the waiting restores are never polled. That silences LMCache's 30 s "has waited" log, and it also skips vLLM's VLLM_CHECKPOINT_RESTORE_MAX_WAIT_S check, which lives in reserve_external_boundary_checkpoint and is reached only from that poll.

The leftover pass that cannot preempt predates restore admission (dc9f7fc, compute sharing). Restore admission (#929) makes the trigger common, because restores that wait for capacity stay queued instead of being admitted to recompute. The same loop also happens with a free slot when the waiting requests cannot be admitted, for example a restore still waiting for GPU capacity. The new test covers both cases.

Fix. A prefill turn that scheduled nothing has no admitted work to protect, so its decode pass may preempt as in a decode turn: allow_preemption=not num_scheduled_tokens. This is the rule the saved-logits fallback already uses. Once a slot frees, the waiting restores are considered again, and a restore still refused past its bound recomputes its prompt with the existing warning.

Not changed: the restore wait bound. It is still checked where a restore competes for GPU blocks. A request waiting for a running slot is queued like any other request. Expiring its restore on a timer would not admit it, since a recompute needs a slot too. It would also make the request recompute its whole prompt after a long slot wait, even when its restore fits by then.

Tests (CPU, beta image with the tree's Python overlaid, CUDA hidden).

• New test_decodes_on_a_full_pool_preempt_while_restores_wait[full|free] (tests/v1/core/test_boundary_admission.py). It uses a real scheduler with compute share 0.4, two decodes that fill the pool, and two restores waiting either behind full slots or for capacity. On the base both cases fail: from the fifth step on every step is empty and nothing is preempted. With the fix both pass: the decode preempts the other one and keeps going, and the restores are polled again as capacity waiters.
• tests/v1/core in full (1468 tests; CPU platform via VLLM_TARGET_DEVICE=cpu, facebook/opt-125m from the local HF cache, no network): base 1363 passed, 103 failed, 2 errors; fix 1365 passed, 101 failed, 2 errors. Compared test by test from the JUnit output, the only differences are the two new tests. Every other failure is identical on both and comes from models that this offline container does not have (ModelConfig validation errors or LocalEntryNotFoundError).
• tests/v1/engine/test_prefill_fairness_runtime.py and test_structured_output_draft_handoff.py: 9 passed on both.
• LMCache 75f2b59d bridge suite (tests/v1/test_vllm_semantic_checkpoint_transfer.py) against this tree: 45 passed, plus 2 CUDA-only skips, the same as on the base.

GPU. A base-vs-fix kit that recreates his setup is prepared on .4 (/root/restore-stall, GPUs 5,6, 12 agents at 40–100K tokens, stall monitor). It has not been run yet, because GPUs 5–8 are in use.

Fragment vllm-restore-admission-stall.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• dev/*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: e2afc896-c4b9-4ab4-bbce-046dbe030a8f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[voipmonitor]

GPU A/B on RTX PRO 6000 (.4), mrweiner's setup: GLM-5.3-Flash Spark TP2, LMCache on-evict, MAX_NUM_SEQS=4, KV_CACHE_MEMORY_BYTES=3.8e9, 12 agents at 40–100K tokens, 600 s client timeout, stall monitor. Base is the beta scheduler; fix is this PR. Only scheduler.py differs.

	base	fix
stalls	1 (240 s, run stopped by the monitor)	0 in 30 min
turns / errors / timeouts	267 / 0 / 0	285 / 0 / 0
max KV	99.4 %	99.7 %
preemptions	3	4
restores recomputed after the 60 s max wait	2	18
tracebacks	0	0

In the base run, waiting restores are never polled while the engine loops, so they neither restore nor hit the max wait. With the fix they are polled again, and the ones still refused after 60 s recompute with the existing warning. The averages are not like for like: base stopped after its stall at 23 min, while fix ran the full 30 min with growing contexts (313 vs 421 tok/s decode, 86 vs 95 % cached).

Merging. The same leftover-decode pass exists on dev/karmic-kraken (scheduler.py:1881); a canonical twin follows.