Add blog post: KV caching on Valkey - #642
cnuthalapati wants to merge 3 commits into
Conversation
Adds a Technical Deep Dive on using Valkey as a shared L2 KV cache tier for LLM inference, behind vLLM and LMCache. The post covers why the GPU-memory (L0) and host-memory (L1) tiers evict under real traffic, what a shared external tier changes, a five-step setup on the open-source stack, benchmark results on Llama-3.1-70B across two GPU nodes, and sizing and tuning guidance. Scheduled with draft = true and date = 2026-09-03. Signed-off-by: Chaitanya Nuthalapati <cnu@amazon.com>
Signed-off-by: Chaitanya Nuthalapati <cnu@amazon.com>
Signed-off-by: Chaitanya Nuthalapati <cnu@amazon.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
pnbrown
left a comment
pnbrown
left a comment
There was a problem hiding this comment.
Read through it quickly.
Also curious if there's a place to tie in other Valkey content like this blog: https://valkey.io/blog/modern-ai-workloads-mapping-to-valkey/
| You can self-host open-weight LLMs on GPUs for better cost economics, accuracy, control, or data privacy. | ||
| Unlike traditional applications that benefit from scaling economies, LLM inference costs scale directly with your application, and the biggest line item is the GPU bill. | ||
|
|
||
| A surprising share of that bill is spent on work the model has already done, reprocessing prompt instructions, tool definitions, and conversation history on every call. |
There was a problem hiding this comment.
This could be a colon after done
| That takes sub-millisecond operations and read bandwidth that grows as you add shards, which is exactly what a Valkey cluster provides. | ||
| Object and file storage have their place, but not on the first-token path. | ||
|
|
||
| We benchmarked Valkey to quantify the impact. |
There was a problem hiding this comment.
We should probably have a caveat for those controlling where they're running their apps? Maybe the have only one server?
Could we also talk about the differences in speed between HBM and running on Valkey? Might be an important distinction.
Maybe also worth mentioning the size of models and how much is being kept in the relative caches
|
|
||
| **Where it will not help.** | ||
|
|
||
| - If your whole working set fits in GPU and CPU memory, you do not need an L2 yet. Little is evicted, so there is little to reload. Come back when traffic grows. |
There was a problem hiding this comment.
That actually may address one of the earlier points.
3 participants
Closes #639.
Adds a Technical Deep Dive on using Valkey as a shared L2 KV cache tier for large language model (LLM) inference, sitting behind vLLM and LMCache.
What the post covers
Scheduling
Set with
draft = trueanddate = 2026-09-03, so the daily publish job picks it up on that date. Opening this as a draft PR while we settle the items below.Open items
Validation
Builds clean with Zola 0.22.1, matching the version pinned in
.github/workflows/zola-deploy.yml. Confirmed thatdraft = trueholds the post out of a normal build, that all three figures resolve, that both tables and all seven code fences render, and that the three author pages link correctly.All commits are signed off per the DCO.