fix(kv-cache): guard never-hit entries from immediate self-eviction

[audreyt]

Summary

Fresh cold-saves at the start of a new conversation are smaller and less
token-dense per byte than mature multi-thousand-token snapshots. The
existing (hits + 1) * tokens / file_size eviction score correctly
captures the long-run "value per byte stored" ranking but has one
pathology under budget pressure: on a saturated cache the new cold-save
is the lowest-scoring entry the moment it is written, so the eviction
loop reaps it on the same wall-clock second — before its session's next
turn can issue its first lookup. The cold-save infrastructure
(cold-prefix boundary alignment, BPE boundary trim via
--kv-cache-boundary-trim-tokens, SHA-keyed text lookup on subsequent
requests) is then effectively dead under any real workload that uses
the default 4 GB budget.

This PR adds a narrow guard: a never-hit entry whose last_used is
within KV_EVICTION_FRESH_GUARD_SECONDS (60s) gets a +INF eviction
score, so the eviction loop cannot pick it until either it ages out
without being hit, or it is hit at least once (after which the existing
(hits + 1) factor protects it).

Reproduction

Three-turn chat workload, ~2500-token initial prompt, default 8 GB
--kv-disk-space-mb saturated by old snapshots:

	turn 1 cold	turn 2	turn 3
before	7.1 s	6.5 s	6.5 s
after	7.1 s	2.1 s	2.0 s

Server log before, every turn:

kv cache stored tokens=2048 trimmed=… reason=cold size=49.78 MiB
kv cache evicted reason=disk-cache-full tokens=2048 hits=0 size=49.78 MiB

i.e. each cold-save is created and reaped in the same second.

Server log after:

kv cache hit text tokens=2048 text=… load=5-6 ms file=…

Turn 2+ now load the just-written 2048-token cold prefix from disk in
~6 ms and only prefill the tokens beyond it.

Why not a different fix

Three earlier candidates were considered and rejected on this code path:

• (hits + K) * tokens / file_size with K=4: doesn't help. A
  2048-token / 50 MB cold-save has ~39 t/B; a mature 90000-token / 1.3 GB
  snapshot has ~71 t/B. The size-density gap is intrinsic (fixed-cost
  header overhead amortizes better at larger sizes), and no bias on the
  (hits + …) factor flips the ranking.

• Capping hits at a small constant: same problem. With hits capped
  at 4 and bias 4, the cold-save scores 156 while the mature entry scores
  285. The dominant signal is tokens / file_size, not hits.

• Time-decayed score everywhere: introduces exp() calls per entry
  per eviction pass and a tuning constant for the decay timescale.
  Strictly more code than the narrow guard, and changes the long-run
  ranking among aged entries, which is the part the current formula gets
  right.

Eviction loop change

When every entry is guard-protected (i.e. all victim_score == DBL_MAX),
the loop short-circuits with a break rather than evicting a guarded
entry. The cache then overruns budget transiently until guard windows
expire. That excursion is bounded by KV_EVICTION_FRESH_GUARD_SECONDS
and made explicit, rather than silently looping forever or evicting
through the guard.

Tests

• New: test_kv_cache_eviction_guards_just_written_never_hit_entry —
  reproduces the failure mode. An old high-density entry and a fresh
  low-density entry compete for a one-entry budget. Under the old score
  the fresh entry would be evicted; under the guard the fresh entry
  survives and the old entry is evicted.
• Existing: test_kv_cache_eviction_values_fresh_snapshots and
  test_kv_cache_eviction_keeps_aligned_continued_frontiers continue to
  pass — they use ancient last_used values (100, 200, 300) so the
  guard window never applies and scoring reverts to the original formula.

Test plan

• make ds4_test && ./ds4_test passes locally (the three eviction
  tests in particular).
• A multi-turn chat-completions workload against a budget-saturated
  cache shows turn-2 disk-text hits in the server log instead of
  kv cache evicted reason=disk-cache-full of the just-written
  cold-save.
• Default --kv-disk-space-mb 4096 workloads benefit too (the
  pathology is worst at smaller budgets).

🤖 Generated with Claude Code

[antirez]

Already fixed when I saw the issue. Sorry!