Exact tag-scoped recall via getByIds (stop losing tagged results to the top-50 cap)

[rahilp]

Problem

When recall is called with a tag, recallEntries (src/index.ts:899-1030) runs an unconstrained Vectorize query capped at topK = min(topK * 3, 50) (src/index.ts:930-934), then drops non-matching tags in memory afterwards (src/index.ts:957-964).

Because the tag filter is applied after the semantic search, any tagged memory whose global semantic rank falls outside the top 50 is never seen. You can have 200 work memories and get back only a handful — or zero — for a tag=work query, purely because the best ones ranked #60+ globally. Tag filtering silently loses recall.

We can't push the filter into Vectorize: metadata indexes only support string/number/boolean (max 10, fixed property names), so free-form array tags aren't indexable. But D1 already stores each entry's vector_ids (db/schema.sql:9, written by storeEntry src/index.ts:567-572), so we can score the tag's own vectors directly — no schema change, no backfill.

Proposed solution

In recallEntries, branch on whether tag is set:

tag set (new path):

1. SELECT id, vector_ids FROM entries WHERE tags LIKE ? (replaces the id-only lookup at :920-926). Empty → return { matches: [], insight: "" }.
2. Flatten + dedupe the vector IDs (JSON-parse each row; skip "[]").
3. VECTORIZE.getByIds(ids) (batched ~500/call) → vectors with values + metadata; missing/stale ids are simply omitted.
4. Cosine-score each candidate against the query embedding via a new cosineSim(a, b) helper (BGE vectors aren't normalized → normalize).
5. Feed the resulting VectorizeMatch-shaped objects into the existing rerankWithTimeDecay → dedupe-by-parentId → D1-hydration pipeline (:954-1016). Drop the now-redundant tagFilterIds post-filter (:961).

tag not set: unchanged — keeps the current VECTORIZE.query path including the low-score retry (:931-941).

Ranking stays purely semantic (cosine vs. the query). Only the candidate pool changes — from "whole brain, top 50" to "this tag's vectors" — so tag-filtered results become exact instead of being knocked out before the tag is even considered.

Files

• src/index.ts — recallEntries tag branch; new cosineSim helper.

Tests / Verification

• test/integration/recall.test.ts — mock VECTORIZE.getByIds to return vectors with values + metadata; assert a tagged entry that would rank outside a top-50 global query still surfaces and ranks correctly. The getByIds mock already exists in test/helpers/make-env.ts.
• npm test green.

Trade-off

Large tags (thousands of entries) now load + cosine-score all their vectors in-memory instead of a top-50 slice. Fine at personal scale (and batched); worth revisiting only if a single tag ever holds tens of thousands of entries.

Out of scope

No wrangler.toml change and no schema migration (reuses existing vector_ids). The unused tag_${t} boolean metadata at :553-555 stays as-is.