Conversation
router_kv_hit_rate is a histogram of per-request overlap/isl, so sum/count is a mean of ratios. The engine's vllm:prefix_cache_hits/queries is a ratio of sums. On a skewed ISL distribution these diverge badly -- on DeepSeek-V4-Flash-0731 (44% of requests under 1800 tokens, mean 14211, tail past 130k) the router read 0.320 against the engine's 0.767 and looked half blind. Add router_kv_overlap_blocks_total / router_kv_isl_blocks_total so the router's predicted overlap can be compared with the engine directly. Measured on the same fleet right after deploy: block-weighted router 0.790 vs engine 0.767, i.e. the router's view was accurate all along. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pernekhan
temporarily deployed
to
external_collaborator
August 31, 2026 21:23 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
The metric help and the code comment pointed at vllm:prefix_cache_hits/queries. Every supported backend reports prefix-cache hits and queries as running totals, so the ratio-of-sums vs mean-of-ratios distinction these counters exist to close is not vLLM-specific. State it in terms of the backend engine instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pernekhan
had a problem deploying
to
external_collaborator
September 15, 2026 23:53 — with
GitHub Actions
Failure
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Single commit of substance on top of
main, +36 lines across 3 files, no behavior change. Supersedes #34, which carried the same commit on top of 11 unrelated ones.The trap this closes
dynamo_component_router_kv_hit_rateis a histogram of per-requestoverlap_blocks / isl_blocks, sosum/countis a mean of ratios (request-weighted).Backend engines report prefix-cache hits and queries as running totals, so a hit rate derived from them is a ratio of sums (block-weighted). That holds for every supported backend, not just one of them.
The two aggregations are not comparable, and on a skewed ISL distribution they diverge badly. On a production model — 44% of requests under 1800 tokens (≤7 blocks), mean ISL 14,211, tail past 130k tokens (~508 blocks) — the router read 0.320 against the engine's 0.767 and looked half blind.
It isn't. That mis-comparison consumed most of a debugging session and nearly triggered an unnecessary indexer rewrite on a subsystem that was working correctly.
The fix
Add
router_kv_overlap_blocks_total/router_kv_isl_blocks_total, incremented at routing time fromselection.effective_overlap_blocksandisl_blocks. Dividing them yields a block-weighted router hit rate directly comparable to whatever the backend reports, with no per-backend arithmetic on the dashboard side.Verified live, immediately after deploy
Within 3% of the engine and slightly ahead, which is expected — some blocks are evicted between the routing decision and execution.
Independently corroborated: 10/10 paired requests sharing an 8k-token prefix showed 97.7% block-weighted reuse (range 96.0–98.5%), i.e. the router picked the prefix-holding worker every time.
Risk
None to serving. Two
IntCounters and their increments; no routing logic touched. The code comment states the aggregation difference explicitly so the next person doesn't walk into it.🤖 Generated with Claude Code