Add retrieval-accuracy eval (cached y/n judgments → precision@k / MRR across models)

[ssrajadh]

Problem

We have no objective measure of retrieval accuracy. The only accuracy-adjacent signal anywhere is the raw cosine "top score" — but that's uncalibrated (0.64 on qwen8b ≠ 0.64 on qwen2b) and a high score doesn't mean the right clip came back. So questions like "is qwen2b actually good enough?" or "did this refactor regress search quality?" are answered by vibes.

Unlike the hardware benchmark (#68), accuracy is hardware-independent — the same model produces the same embeddings on any machine (float precision aside). So the maintainer can run this locally; it doesn't need crowdsourcing.

Goal

A repeatable retrieval-accuracy eval that scores each backend/model on the fixed benchmark clip and emits a comparison table. Drives model recommendations (which local model to ship/recommend) and catches search-quality regressions.

Why not a pre-labeled ground-truth set

Broad queries like "car driving on road" have an effectively unbounded set of correct answers. Pre-enumerating every valid window is tedious and unfair: if a model returns a correct clip we didn't happen to label, fixed recall@k scores it wrong. So we judge what's actually returned, not a guessed answer key.

Design: cached y/n judgments (pooling)

1. Run the eval → for each query, get the model's top-5 clips (spans from search_footage, sentrysearch/search.py).

2. For each unseen (query, time-window), prompt the human y/n on the clip. Store the verdict.

3. Cache all verdicts in-repo, keyed by (query, rounded start/end time):

   # tests/data/accuracy_judgments.yaml
   "white toyota pickup truck":
     "124.0-151.0": true
     "300.0-327.0": false
   "amazon prime van":
     "233.0-260.0": true

4. Re-runs reuse cached verdicts — only genuinely new clips need a fresh y/n. Re-scoring an already-judged model is zero-effort.

Metrics

Recall is out (no known denominator once we stop enumerating). Report per query and aggregate:

• precision@1 / precision@3 / precision@5 — of the top-k returned, how many are correct.
• MRR — rank of the first correct clip.
• Aggregate floor queries (car driving on road, highway driving, black car) separately from fine-grained ones. Floor near 1.0 confirms the pipeline works; fine-grained is the real model-quality signal.

Deliverable

A dev-facing script (not a user CLI command — maintainer eval), e.g. scripts/accuracy_eval.py:

1. Index the benchmark clip into a throwaway _accuracy collection (delete on exit).

2. Load the embedder once, loop all queries, collect top-5 with spans.

3. Consult the judgment cache; prompt y/n only for unseen clips; write new verdicts back.

4. --backend / --model flags to run qwen8b, qwen2b, qwen-cloud, gemini; print a side-by-side table to stdout:

   | Model   | P@1 (fine) | P@3 (fine) | MRR (fine) | P@1 (floor) |
   |---------|------------|------------|------------|-------------|
   | qwen8b  | 0.70       | 0.63       | 0.79       | 1.00        |
   | qwen2b  | 0.30       | 0.28       | 0.41       | 1.00        |
   

Reuse #68's fixed workload

Same benchmark clip (benchmark-clip-v1) and same query list. The only new asset is the judgment cache.

Synergy with #68

Once the cache has verdicts, #68 can replace its meaningless "Top score" column with "top-1 correct?" at near-zero contributor effort: same model on different hardware returns the same clips → already judged → looked up from cache. Only a never-before-seen window (e.g. a quantized model retrieving garbage) shows as unjudged — which is itself a useful signal.

Non-goals

• Not crowdsourced — maintainer-run, hardware-independent.
• No pre-enumerated answer key.
• Not a user-facing CLI command.