[Feat] RAG eval harness: dataset format, runner, metrics #75

Context

The README can say "Semantic search with sqlite-vec", which is an implementation detail. There is no way to answer "did this change make retrieval better?" That question has to be answerable before any hybrid/reranker work (v1.5), otherwise tuning is folklore.

Proposal

Add an eval harness (suggested eval/):

eval/
  corpus/               # ≤ 20 public-domain or first-party documents
  questions.jsonl       # one question per line
  run.ts

Question schema:

{ "id": "q001",
  "question": "...",
  "relevant": [
    { "document": "papers/attention-is-all-you-need.pdf", "page": 5, "block": 12, "quote": "..." }
  ],
  "goldAnswer": "..."   // optional
}

Ground truth uses corpus identity, not database identity:

• document is the corpus-relative path / slug (papers/foo.pdf), never the random runtime documentId. The runner builds corpus path → runtime documentId after ingestion, so production id generation is not changed for the sake of a test.
• block is the block ordinal within the document (the document's document_blocks.order), not blk_<documentId>_<order> (which embeds the random id).
• quote is an optional short excerpt used to verify the ordinal did not drift when a parser changes.

This matters for #78: ground truth must not be bound to chunkId or any runtime id, or changing the chunking strategy would invalidate the ground truth instead of being measured by it.

Runner:

1. create a throwaway database,
2. index the corpus through the normal ingestion path (so the eval exercises blocks/chunking, not a shortcut),
3. resolve each document slug to its runtime documentId, then each block ordinal to a blockId,
4. run the Retriever from the Retrieval-interface issue,
5. compute Recall@1/5/10, MRR, nDCG@10, Citation Recall (resolved answer citations ⊆ ground truth), and latency p50/p95,
6. emit machine-readable JSON plus a markdown summary.

Constraints

• Offline and deterministic. Reuse the built-in local embedding backend from feat(embedding): built-in local backend with on-demand model download #54 (multilingual-e5-small). No API keys, no network, same numbers across runs.

• Bootstrap the pinned model explicitly:

  • npm run eval:prepare performs the one-time, networked bootstrap: download the pinned multilingual-e5-small revision, verify revision/checksum, write it into the existing model cache.
  • npm run eval performs no network access. If the pinned model is not in the cache it exits with a clear message (Eval model is not available locally. Run: npm run eval:prepare) instead of silently downloading.

  A clean checkout therefore means npm run eval:prepare && npm run eval — no user model/provider configuration, but not "no download".

• CI caches the model with a key that includes the pinned revision, so baselines across PRs stay comparable.

• Add npm run eval:prepare and npm run eval scripts.

• Ground truth is committed together with the corpus.

Acceptance

• npm run eval:prepare && npm run eval works on a clean checkout without user model/provider configuration
• After preparation, npm run eval performs no network access, and fails with the documented message when the model is absent
• Two consecutive runs produce identical metrics
• Corpus ≤ 20 documents, all redistributable
• Ground truth is expressed in corpus identity (relative path + page + block ordinal + optional quote), never runtime ids
• Metrics JSON is stable enough to diff in a PR

Verification

Run npm run eval twice and diff; document the commands in the eval README.

Frozen baseline (absorbed from #76)

The harness is only useful with a fixed reference point. Without one, every future "the reranker helps" claim is unverifiable — so the baseline is part of this deliverable, not a follow-up.

Run the harness on the frozen v1.4 configuration and commit docs/eval/baseline-v1.4.md:

Embedding:  multilingual-e5-small q8 (384d, local backend)
Chunking:   <current chunkSize / chunkOverlap / separators>
Retrieval:  dense only
TopK:       <n>, threshold <x>
Corpus:     eval/corpus @ <commit>

Include the raw metrics JSON and the exact command used to produce it. State explicitly that v1.5 experiments (#77, #78) must be reported as delta vs this file, with an adopted-change threshold (e.g. "adopt only if Recall@5 improves and nDCG does not regress").

• Report is reproducible from the committed corpus and harness
• Raw JSON is committed alongside the summary
• The report states the comparison protocol for future experiments
• Numbers are not hand-copied — they are the harness output
• Re-running npm run eval reproduces the committed numbers

Part of the v1.4 — Trusted Research Loop epic: #82

Absorbed #76 — "freeze the baseline" is this issue's Definition of Done, not a separate work item.