Benchmark's default --model is silently ignored by llama.cpp, so results can name a model that never ran

[MrJev]

Hi — reviewing choosekit for a write-up on mrjev.com, at 5f1099e (v0.4.2). I ran it against llama.cpp b11042 serving LFM2.5-1.2B-Q8_0 in Docker on a 4-core CPU-only box. No Jev calls.

This is the tightest-engineered package in the batch I'm working through, and specifically:

• No dependencies field at all in package.json, typescript pinned exactly as the only devDependency, and tests/package.test.mjs asserting both zero runtime dependencies and that importing the public entry points calls no fetch and logs nothing. 94 + 12 tests, all green.
• src/llama-cpp.ts:141 refusing to proceed when the server applied post-sampling settings, after explicitly sending post_sampling_probs: false, backend_sampling: false. A tool whose entire output is a probability distribution declining to build one from post-sampling numbers is exactly right, and I'd rather have the rigidity than a silent downgrade.
• cache_prompt: true earning its keep — 64% of prompt tokens came back cached in my runs, and it gave the steadiest latency I measured across five local projects (p75/p25 = 1.10).
• benchmarks/data/semif-authored144.jsonl vendored byte-for-byte with SEMIF-LICENSE.txt and the upstream commit hash. That's how to do it.
• types.ts:27's "Shannon entropy in nats, not a probability of correctness" — worth having in the type.

One finding: the benchmark's default --model is silently ignored, so results can be attributed to a model that never ran.

benchmarks/run-semif.mjs:100:

const model = option("--model", process.env.LLAMA_CPP_MODEL ?? "qwen3.8-27b-text-64k");

llama.cpp doesn't reject an unknown model name. I started a server holding only LFM2.5-1.2B-Instruct-Q8_0.gguf and posted a request naming qwen3.8-27b-text-64k:

model echoed: /gguf/LFM2.5-1.2B-Instruct-Q8_0.gguf
content: ': The'
HTTP: 200

HTTP 200, answered by whatever is actually loaded. So anyone running node benchmarks/run-semif.mjs without --model gets rows labelled qwen3.8-27b-text-64k regardless of what their server holds, with no warning.

Two suggestions, either of which would close it:

• Query /v1/models at startup and fail (or warn loudly) if the requested model isn't among them.
• Drop the default entirely and require --model or LLAMA_CPP_MODEL, recording the server's echoed model name in the results rather than the requested one.

The second also helps with something adjacent: benchmarks/README.md:41 says result files are ignored "because they can contain environment-specific timing and provider metadata". I understand the reasoning, and it's a defensible call — but combined with the above it means the comparison table on the README front page has no committed raw data behind it and no record of which model produced it. Reproducing it needs an RTX 4090 and a paid API key, so for a reader it isn't checkable. A committed summary with the environment fields stripped, or even just the per-row verdicts without timings, would make it auditable without leaking anything.

For calibration, here's what I measured myself, so you can see where I'm coming from: median 233 ms (p25 222 / p75 245, p95 316) for one question with three options against LFM2.5-1.2B on a 4-core i3-9100T, machine idle, n=20 after 3 warm-ups. And running your own SemIf harness over the first 40 rows with that same small model gave 17/40 — which is a statement about a 1.2B model, not about choosekit. The harness worked out of the box, which is more than I can say for most benchmark scripts I run.

Two notes, not requests:

• labels mode caps at 26 options (internal-chooser.ts:26) and substitutes A/B/C for the keys in the prompt, so option descriptions have to be self-sufficient. Worth a line in the README next to the mode description — it's the kind of thing you discover from a bad result rather than an error.
• The core is zero-dependency, but choosekit-mcp pulls zod ^4.6.4 with a floating caret. Given how hard you've held the line elsewhere, pinning it exactly would be consistent.

Happy to send a PR for the model check.

[NotXf1le]

Thanks, good catch. The labels documentation was already clarified in 8035fd2. A PR for the model check would be welcome