feat(server): serve BitNet --keep-quant containers over HTTP (/v1/infer + OpenAI SSE)

[gburd]

The ternary engine landed here in #159 / b5b892e, but the HTTP wiring that reaches it did not. larql_inference::ternary exports predict_bitnet, infer_bitnet_walk and generate_streaming_bitnet, and today nothing in larql-server calls any of them:

$ git grep -ln "ternary" -- crates/larql-server/src
(no output)

Only larql-cli (convert_cmd.rs, run_cmd.rs) does. So a --keep-quant container can be built and run from the CLI, but larql-server cannot serve it — the engine is there and unreachable over HTTP.

This adds that wiring.

What it does

state/loaded_model.rs — bitnet_model: OnceLock<RwLock<BitnetModel>> plus a bitnet_init guard, mirroring the existing weights / weights_init single-flight pattern exactly. is_bitnet(), is_dense_only(), get_or_load_bitnet(), force_load_bitnet_model().

bootstrap/mod.rs — eager-loads the ternary path instead of the dense one for BitNet containers (~5 GB of dense allocation saved on a 2B BitNet), and excludes them from the startup memcheck: estimate_resident_bytes() models the dense path and would massively over-count a container that allocates no dense BitLinear tensors.

routes/infer.rs — native-ternary /v1/infer, checked before the has_model_weights gate. That ordering is load-bearing: a --keep-quant container carries ternary artifacts rather than the dense weight manifest that gate looks for, so it would otherwise be refused as weightless. Walk-mode goes through residual capture + KNN override.

routes/openai/{completions,chat/stream}.rs — SSE streaming for both OpenAI surfaces via the ternary path, which also skips the dense lock_weights_for_gen() write lock that serialises all generation. Chat refuses tools and constrained generation rather than ignoring them: both need masked logits over the dense path, and answering a tool request with prose looks like a model that declined to call the tool.

is_dense_only()

A --dense-only container has the dense weights and I2_S artifacts but no gate vectors, so walk-mode runs against an empty KNN store and returns nothing useful. /v1/infer defaults mode to walk when a client omits it — which any client posting {prompt, top} does — so this silently produced garbage on a correct model. Walk and compare now coerce to dense on such containers.

Adapted to current main, not replayed

This was originally written against a June tree; state.rs and the openai routes have both since become modules, so it is ported rather than merged. Four things changed in the process, each of which would have been a defect if copied verbatim:

• chat streaming uses the current TokenTap for stop-string handling instead of the original hand-rolled buffering, so both paths share one implementation rather than two that must agree;
• pick_template now takes &ModelWeights, which this path deliberately never loads — the template comes from ChatTemplate::for_family(&config.family), the same string weights.arch.family() would have produced;
• both branches record a GenerationTally (add_v3), else /v1/stats reports BitNet traffic as zero throughput;
• no inline SSE_DONE — the response stream already chains it, so the original would have emitted it twice.

Verification

Pinned 1.98.0, on a 32-vCPU machine rather than my laptop (which produces spurious SIGSEGVs in this crate's integration binaries — on both this branch and unmodified main, so not from these changes):

• cargo clippy -p larql-server --all-targets -- -D warnings: clean. --all-targets matters here — seven LoadedModel literals in tests/ need the two new fields, and cargo build alone does not see them.
• cargo test -p larql-server --no-fail-fast: 1279 passed, 0 failed, 0 crashes, against 1277 passed, 0 failed, 0 crashes on unmodified 23a56db1. The +2 are the two tests added here (is_dense_only_detects_empty_gate_layers, bitnet_model_not_loaded_by_default).

I have not tested this against the real microsoft/bitnet-b1.58-2B-4T checkpoint on this rebase — the original work was verified end-to-end against it (Paris 94.5%, and a no-mode request returning the same as explicit mode:dense), but that was on the June tree. Worth a run against real weights before merging if you have one handy.

[gburd]

Heads-up on the two red checks (cargo-audit, cargo-deny · advisories) — they are not from this PR.

RUSTSEC-2026-0285 is a rustls advisory published 2026-09-14, a few hours after main's last quality.yml run (23a56db1, 2026-09-14T11:45Z), which is why main still shows green. main's own lockfile pins rustls 0.23.39; the advisory wants >= 0.23.45.

This PR's Cargo.lock is byte-identical to main's:

$ git diff --quiet upstream/main..<this branch> -- Cargo.lock && echo IDENTICAL
IDENTICAL

So any PR opened against main right now inherits the same two failures, and a rustls bump is needed on main independently of this change. Happy to add that bump here if you'd like it bundled, but it seemed wrong to fold an unrelated dependency change into a Nix fix.

[gburd]

Re-verified against the real microsoft/bitnet-b1.58-2B-4T

I flagged above that this rebase hadn't been checked against real weights. It has now, and it found a bug — fixed in 964166b1.

The bug

The three non-streaming generation paths (/v1/completions batch loop, chat/handler.rs, responses/engine.rs) all take &mut ModelWeights and so call lock_weights_for_gen(). A --keep-quant container has no dense weights, so that reached for a manifest which doesn't exist:

503  "failed to load model weights: IO error: No such file or directory (os error 2)"

Useless to an operator: the model is loaded and working, just not reachable through that route.

Guarded in lock_weights_for_gen() rather than at the three call sites — every non-streaming path funnels through that one method, and the streaming paths are unaffected because they check is_bitnet() and return before reaching the lock. The message now names the routes that do work.

Only real weights could have found this. synthetic_vindex builds a dense V2 container, which has the files whose absence this is about.

Verification

ggml-model-i2_s.gguf (1.2 GB) → --keep-quant --dense-only --f16 --level inference → 210 I2_S tensors, 30 layers, hidden 2560, bitnet_layout stamped, 0 gate layers.

request	result
/v1/infer, no mode field	Paris 0.9494, mode=bitnet
/v1/infer, mode=dense	Paris 0.9494, mode=bitnet
/v1/infer, mode=walk	Paris 0.9494, mode=bitnet (coerced)

All three agree to 4dp — which is the point of is_dense_only(): walk coerces to dense instead of answering from an empty KNN store. 0.9494 matches the 94.5% measured on the June tree.

Also confirmed on the real model:

• eager ternary pre-load: Pre-loaded BitNet model for 'bitnet2b' in 3.3s (ternary path, not dense)
• both SSE surfaces stream coherent text (" Paris. Paris is a city that"), exactly one [DONE], finish_reason: length
• chat refuses tools with the intended message rather than answering with prose
• /v1/runtime reports decode_tokens_per_second: 0.98 — the GenerationTally added during the port reaches the stats surface instead of reporting zero

Throughput (32 vCPU x86_64, A/B alternated, 3 reps, medians)

case	median	spread	effective
infer_short	4.757s	0.008s	—
infer_long	24.785s	0.100s	—
gen_8tok	11.629s	2.690s	0.69 tok/s
gen_32tok	32.661s	0.773s	0.98 tok/s

~1 tok/s is expected here, not a regression: ternary_matvec has a NEON path under cfg(target_arch = "aarch64") and no x86 SIMD equivalent, so x86_64 runs the scalar kernel. An AVX2/AVX-512 ternary kernel is the obvious follow-up — happy to open a separate issue for it.

Suite

clippy -p larql-server --all-targets -- -D warnings clean; cargo test -p larql-server --no-fail-fast 1283 passed, 0 failed, 0 crashes.

[gburd]

Follow-up on the two red checks: I've opened #481, which bumps rustls 0.23.39 → 0.23.45 (lockfile only) and clears RUSTSEC-2026-0285.

On that branch cargo-audit and cargo-deny · advisories both pass. Merging it first (or rebasing this branch onto it) turns these two green here as well — they are inherited from main, not introduced by this PR.