Continuous batching perf: fused tick NIF, cross-slot prefix caching, server-routed chat completions, cancellation

[nyo16]

Summary

End-to-end performance and robustness work on the batching Server and the stateless
generation paths, implemented from the perf-batching-prefix-cache plan (22 tasks,
4 phases + benchmarks). Baseline and final numbers are committed under bench/results/.

Headline (Llama-3.2-1B, 8 interleaved conversations × 4 turns on 4 slots, shared
system prompt — bench/prefix_cache_concurrent.exs):

Mode	TTFT median	Hit ratio
cache off	115.0 ms	0%
per-slot cache	39.0 ms	75.6%
cross-slot + session affinity	35.5 ms (3.2×)	74.8%

Also: multi-turn chat_completion through the Server is 1.6× faster than the
stateless path (346 → 211 ms / 4 turns); abandoned streams now cost ~4 tokens instead
of decoding to max_tokens; server_concurrent throughput +11–21% vs baseline.

Phase 1 — Server hot loop & scheduler hygiene

• Fused tick NIF batch_eval_sample: one dirty-CPU call per tick does decode,
  per-slot sampling (per-entry sampler resources — no shared-sampler resets),
  detokenization, and EOG checks. Replaces batch_eval + 2 normal-scheduler NIFs per
  generating slot + per-token token_to_piece/vocab_is_eog. Streaming sends moved
  out of strategy side effects to post-NIF handling and fire one tick earlier.
• KV-pressure policy (llama-server parity): on llama_decode == 1, purge idle
  slots' cached KV, then recursively halve the batch; a sequence that can't fit one
  token fails alone with {:error, :context_full} — other slots keep generating.
• Persistent llama_batch reused across decode NIFs; sampler_sample,
  sampler_sample_at, chat_apply_template_jinja, json_schema_to_grammar,
  speculative_init dirty-flagged; Server n_batch default → min(n_ctx, 2048).

Phase 2 — Prefix caching that hits under concurrency

• Per-request cache_prompt, server default flipped to true (llama-server parity).
• Cross-slot prefix sharing via memory_seq_cp under unified KV (kv_unified: true
  new Server default; plumbed through Context.create): a shared system prompt
  prefills once, ever — any slot adopts it with a metadata-only copy, even from a
  still-generating donor. Gated on kv_unified + partial-seq_rm support: split-mode
  partial cross-stream seq_cp hard-aborts (verified in vendored source) and hybrid-GDN
  recurrent state can't be partially copied, so those configs keep per-slot caching.
• Slot pick: best cached match only when LCP/prompt_len > 0.1, else LRU — a tiny
  request no longer evicts a long cache. Session affinity: session: request
  option pins conversations to their slot (queue served session-first).
• Level-2 RAM prompt cache (prompt_cache_ram_mb, default 0 = off): evicted slot
  states are serialized via new llama_state_seq_* NIFs and restored later instead of
  re-prefilling; byte budget checked before copying (degrades to disabled, never OOMs).
• Prefix-instability telemetry: requests matching only 10–50% of a slot's cached
  history (template rewrote history, e.g. thinking-strip) emit
  [:llama_cpp_ex, :server, :prefix_instability] — previously a silent full re-prefill.

Phase 3 — Chat completions through the Server; load protection

• Per-request sampling params (:temp, :seed, :grammar, …): fresh sampler per
  request from request opts merged over server defaults.
• chat_completion/stream_chat_completion accept a running Server (also via
  ModelManager): templating + tokenization in the caller, generation on the batching
  server with prefix caching. New Server.complete_tokens/3 returns
  {text, completion_tokens, finish_reason}.
• Caller-side tokenization: the Server's %Model{} is cached in :persistent_term;
  get_model/1 no longer round-trips the server mailbox.
• :max_queue backpressure (finally enforced): bounded queue rejects with
  {:error, :queue_full} in ~1 ms; streams emit a single {:error, reason} element
  instead of silence-then-timeout.
• Cancellation: consumers are monitored per slot; dead or halted stream consumers
  free their slot immediately (Server.cancel/2, auto-called by stream cleanup).
  Stateless loops (generate, generate_tokens, MTP) poll a new CancelFlag NIF
  resource and install it as llama_set_abort_callback, so even a long prefill aborts
  mid-decode.

Phase 4 — Copies, allocation, streaming correctness

• Embeddings as f32 binaries: one 16 KB refc binary instead of ~100 KB of boxed
  floats per 4096-dim call; list API preserved, format: :binary for Nx.from_binary.
• Embedding decodes clear only their own sequences (no more whole-context clears).
• UTF-8-safe streaming: a shared pending-buffer (LlamaCppEx.UTF8Stream) at every
  emission point — consumers never see a codepoint split across token pieces.
• decode ported off llama_batch_get_one (no more logits for the last token of every
  chunk). The i32-token-binary spike measured 0.012% and was skipped per the plan's
  own ≥-a-few-% criterion.

Notable behavior changes

• cache_prompt defaults to true; kv_unified defaults to true (slots share the
  full n_ctx budget instead of n_ctx/n_parallel each); n_batch defaults to
  min(n_ctx, 2048).
• Error replies normalized to atoms/tagged tuples: :context_full, :queue_full,
  :empty_prompt, {:inference_failed, reason}.
• Stream elements are String.t() | {:error, term()}; a hard error kills only the
  affected request, not all active slots.

Verification

• 252 tests pass, 0 failures (mix test --include smoke with real models),
  --warnings-as-errors, mix format, credo --strict clean.
• New tests: UTF8Stream unit tests, pure slot-pick/donor/RAM-cache unit tests
  (test/server_test.exs), and 6 :smoke integration tests (cache semantics, session
  affinity, backpressure, cancellation, overflow isolation).
• Parallel specialist review: PASS WITH WARNINGS, requirements 22/22 MET; all
  findings fixed in the final commit. Review doc:
  .claude/plans/perf-batching-prefix-cache/reviews/perf-batching-prefix-cache-review.md.
• Benchmarks: baseline (bench/results/v0.8.32-e6e1ef1-baseline-perf-m1max.md) and
  final (bench/results/v0.8.32-perf-branch-final-m1max.md), plus two new bench
  scripts (prefix_cache_concurrent.exs, chat_completion_server.exs).

Deferred (per plan non-goals)

Multi-seq-token prefill, cache_reuse chunk splicing, context shift on overflow,
on-disk prompt cache, GPU backend sampling, stateless-path context pooling.