index: make the BM25 MaxScore bound authority-aware (fixes k-dependent ranking)

docs/ENV.md

@@ -153,13 +153,13 @@ not re-enable them without the confidentiality decision in that section.
 | `COSIFT_BM25_K1` | float | `1.2` (PebbleBM25 default) | BM25 term-frequency saturation `k1`; applied only if parseable and `> 0`. | `serve_search.go:1350` |
 | `COSIFT_BM25_B` | float | `0.75` (PebbleBM25 default) | BM25 length-normalization `b`; applied only if parseable and `> 0`. | `serve_search.go:1358` |
 | `COSIFT_BM25_MIN_IDF` | float | `0.5` | IDF floor below which a query term is dropped as a stopword. `0` disables pruning. Must be `>= 0`. | `internal/index/pebble_bm25.go:38` |
-| `COSIFT_BM25_DISABLE_MAXSCORE` | bool (non-empty disables) | unset → MaxScore optimization **enabled** | Any non-empty value disables the WAND/MaxScore early-termination optimization for benchmark-grade lossless ranking. | `internal/index/pebble_bm25.go:223` |
+| `COSIFT_BM25_DISABLE_MAXSCORE` | bool (non-empty disables) | unset → MaxScore optimization **enabled** | Any non-empty value disables the WAND/MaxScore early-termination optimization for benchmark-grade lossless ranking. Phrase queries skip the optimization unconditionally. | `internal/index/pebble_bm25.go:230` |
 | `COSIFT_BM25_TOPK_POOL_FACTOR` | int | `50` | Sizes the metadata-resolution pool at `factor*k` candidates (PebbleBM25 top-k pool). Must be `>= 1`. Raise if the pool-cap log line fires on ranking-sensitive traffic. | `internal/index/pebble_bm25.go:50` |
-| `COSIFT_BM25_DISABLE_TOPK_POOL` | bool (non-empty disables) | unset → top-k pool **enabled** | Any non-empty value restores the resolve-all metadata path (every scored candidate gets a `GetDocMeta`) — the pre-pool behavior, for lossless A/B comparison. | `internal/index/pebble_bm25.go:295` |
+| `COSIFT_BM25_DISABLE_TOPK_POOL` | bool (non-empty disables) | unset → top-k pool **enabled** | Any non-empty value restores the resolve-all metadata path (every scored candidate gets a `GetDocMeta`) — the pre-pool behavior, for lossless A/B comparison. | `internal/index/pebble_bm25.go:294` |
 | `COSIFT_DEFAULT_DECAY_DAYS` | float | `180` (6-month half-life) | Default recency half-life (days) applied when a request has no explicit `?decay=`. `0` disables decay globally. Must be `>= 0`; explicit `?decay=N` still wins. | `serve_search.go:1592` |
-| `COSIFT_AUTHORITY_ALPHA` | float | scorer's built-in default (`authority.New()`) | Authority-score blend weight; applied only if parseable and `>= 0`. | `serve_setup.go:188` |
-| `COSIFT_TRANCO_CSV` | string (path) | unset → embedded whitelist + TLD heuristics only | Path to a Tranco rankings CSV to enrich authority scoring. | `serve_setup.go:193` |
-| `COSIFT_MAJESTIC_CSV` | string (path) | unset → embedded whitelist + TLD heuristics only | Path to a Majestic Million CSV to enrich authority scoring. | `serve_setup.go:202` |
+| `COSIFT_AUTHORITY_ALPHA` | float | scorer's built-in default (`authority.New()`) | Authority-score blend weight; applied only if parseable and `>= 0`. **Also a BM25 scan-cost knob**: it sets the MaxScore pruning ceiling `maxMult = 1+alpha` (`internal/index/pebble_bm25.go:242`), so raising it prunes less (more posting scans, larger per-query score map) and lowering it prunes more. Lower alpha before reaching for `COSIFT_BM25_DISABLE_MAXSCORE`. | `serve_setup.go:135` |
+| `COSIFT_TRANCO_CSV` | string (path) | unset → embedded whitelist + TLD heuristics only | Path to a Tranco rankings CSV to enrich authority scoring. | `serve_setup.go:140` |
+| `COSIFT_MAJESTIC_CSV` | string (path) | unset → embedded whitelist + TLD heuristics only | Path to a Majestic Million CSV to enrich authority scoring. | `serve_setup.go:149` |
 | `COSIFT_DISABLE_ENTITY_EXPAND` | bool (non-empty disables) | unset → entity expansion **enabled** | When unset, query is expanded with canonical-attribute rewrites (`qexpand.RewriteEntity`). Any non-empty value disables it. | `serve_search.go:2064` |
 | `COSIFT_HYDE_CACHE_SIZE` | int | `256` | Capacity of the HyDE hypothetical-document cache; must be `> 0` (warns on bad value). | `serve_setup.go:161` |
 | `COSIFT_PARA_CACHE_SIZE` | int | `256` | Capacity of the paraphrase cache; must be `> 0` (warns on bad value). | `serve_setup.go:173` |

docs/K-DEPENDENCE.md

@@ -0,0 +1,124 @@
+# Why the same query returns different results at different k
+
+Measured on production 2026-09-15: 15,718,478 docs / 78,652,652 vectors, `POOL_FACTOR=200`,
+authority alpha 2.0. Five binary arms, the 60 golden queries at k=10 and k=50.
+
+The short version: **there are two independent defects, and the one everybody was chasing is not the
+one that matters most.** This document exists so the next person does not repeat the three wrong
+turns below.
+
+## The symptom
+
+`GET /search?q=X&k=10` and `GET /search?q=X&k=50` disagree about the top 10. On the goldens, 51 of 60
+queries disagreed. `quantum gate teleportation logical qubits` was the reported example: wrong at
+k=10, correct from k=15.
+
+## Defect 1 — time decay re-ranks a k-sized window (the bigger one)
+
+`serve_search.go` applies time decay *after* retrieval, over the candidate list it fetched. For a
+plain query `fetchK == k`, so k=10 decays a 10-item list and k=50 decays a 50-item list. A deeper
+list gives decay more to choose from, and different documents surface into the top 10.
+
+Isolating it, on one binary at one moment:
+
+| retriever | queries whose k=10 top-10 ≠ k=50 top-10 |
+|---|---|
+| `bm25+decay:180d` (the default) | 50/59 |
+| `bm25` (`?decay=0`) | **0/59** |
+
+On stock v0.2.5, turning decay off takes 51/60 → 19/60. So decay accounts for roughly two thirds of
+the reported symptom, and it is **not reachable from `internal/index`** — no BM25 change affects it.
+
+Magnitude, stock v0.2.5, k=10 against the k=50 reference:
+
+| | decay on | decay off |
+|---|---|---|
+| mean overlap@10 | 0.7133 | 0.9450 |
+| queries with an identical top-10 *set* | 12/60 | 46/60 |
+
+**Not fixed.** The counterpart fix is to decay over a fixed-depth candidate window rather than a
+k-sized one, so the input to decay stops depending on the caller's k. Not attempted.
+
+## Defect 2 — BM25 makes two k-shaped decisions (the real engine bug)
+
+With decay off, 19 of 60 goldens still disagree. Two places in `pebble_bm25.go` branch on the
+caller's k:
+
+1. **The MaxScore break.** `theta := kthLargest(scores, k)` — a smaller k gives a *higher* theta, so
+   the scan stops earlier and produces a different score map before anything is ranked.
+2. **The resolution pool.** `poolCap = factor * k` — a smaller k resolves fewer candidates. At
+   factor 200 and k=10 the pool is 2,000, while the cap-bind log shows a median bound query with
+   **68,497 in-band candidates** and a worst case of 403,943. Under 3% of eligible candidates get
+   resolved.
+
+Running both at `kEff = max(k, COSIFT_BM25_RANK_DEPTH)` and truncating to k afterwards makes the
+top-k of any k ≤ depth a prefix of one ranking: **19/60 → 0/59, exactly zero.**
+
+### Choosing the depth
+
+k-sweep on stock v0.2.5 with decay off, each k's top-10 against the k=50 reference:
+
+| k | queries differing | mean overlap@10 | p50 | p90 |
+|---|---|---|---|---|
+| 10 | 19/60 | 0.9450 | 42 ms | 162 ms |
+| **15** | 15/60 | **0.9817** | 55 ms | 176 ms |
+| 20 | 16/60 | 0.9800 | 66 ms | 184 ms |
+| 30 | 14/60 | 0.9850 | 94 ms | 213 ms |
+| 50 | 0/60 | 1.0000 | 147 ms | 324 ms |
+
+The knee is at **15**: it captures most of the accuracy for +31% p50 and +9% p90, and it is where
+`quantum gate teleportation logical qubits` goes from 0.20 to 1.00. Beyond 15 the curve is flat until
+50, which is the reference and therefore trivially perfect.
+
+`COSIFT_BM25_RANK_DEPTH` defaults to 0 (off, current behaviour). Depth 15 has **not** been measured
+as its own production arm.
+
+## Three wrong turns, recorded so they are not repeated
+
+**1. "MaxScore × authority dominates."** The engine tracker concluded this on 2026-09-12 after
+raising the pool factor 50 → 200 and seeing k-dependence barely move (48/60 → 46/60). The correct
+reading is *raising the factor does not help*, not *MaxScore is the cause*. Cap-bind rate was
+essentially identical across every arm here (15 vs 13 over comparable windows).
+
+**2. Making the MaxScore bound authority-aware does not fix it, and is expensive.** The bound
+`remainingMax < theta` ignores the authority multiplier applied after the scan, so with alpha 2.0 it
+is 3× too loose and drops genuine top-k members. Correcting it to `remainingMax*maxMult < theta` is
+provably more exact in isolation — against a lossless oracle on a small corpus where the pool never
+binds, agreement with exact ranking goes 0.8362 → 0.9793. On production it measured:
+
+| | k-dependence | k=10 p50 | k=10 p90 |
+|---|---|---|---|
+| control | 51/60 | 56 ms | 168 ms |
+| authority-aware bound | 53/59 | 184 ms | 943 ms |
+
+No improvement to the symptom, 3.3× p50 and 5.6× p90. It leaves theta k-shaped, so it changes *where*
+the scan stops without changing *that it stops somewhere k-dependent*. The correctness gain is real
+but is not separately measurable at production scale, and the cost is not.
+
+**3. Flooring the pool alone does not fix it either.** An absolute floor on `poolCap` makes the
+candidate universe k-independent but leaves theta k-shaped, so the score map still differs before the
+pool is built: 48/59, and 210 ms p50. Both floors are needed, which is why the knob is a single rank
+depth rather than a pool minimum.
+
+## Things worth knowing that came out of this
+
+- **`/search` does not pass the caller's k to the index.** It computes `fetchK` from `keepCap`, which
+  widens 5× for include/exclude filters, 10× for a date filter, a flat 300 for `site=`, and 2× for
+  rerank, capped at 500. For a plain query `fetchK == k`. Anything reasoning about "the k the index
+  sees" has to start there.
+- **`COSIFT_BM25_DISABLE_MAXSCORE=1` is a usable oracle on small corpora only.** It is exact by
+  construction and far too slow at 15.7M documents.
+- **Latency numbers here are not comparable across sessions.** The later arms ran alongside the 16:00
+  UTC scheduled snapshot; measured contention on an unchanged binary was +7% p50 / +8% p90. The
+  k-sweep ran on a quiet box, which is why its k=10 p50 reads 42 ms against 56 ms elsewhere.
+
+## Reproducing
+
+```sh
+# On the box. decay=0 is the flag that separates the two defects.
+python3 cosift-golden-capture.py -q cosift-golden-queries.txt -o out.jsonl \
+  -b http://127.0.0.1:7777 -k 10,50 -t 75 -x decay=0
+```
+
+Then compare each query's k=10 top-10 against its k=50 top-10. Raw captures for all five arms are in
+the monorepo at `tools/golden/2026-09-15-t0-prod/`.

docs/TUNING.md

@@ -111,6 +111,26 @@ set (same cost as before the pool, plus a small selection overhead).
 `COSIFT_BM25_DISABLE_TOPK_POOL=1` restores the resolve-all path for lossless
 A/B comparison; both vars are read per query, no restart needed.
 
+### MaxScore pruning depth: `COSIFT_AUTHORITY_ALPHA`
+
+The MaxScore early-break compares the k-th *raw* score against
+`remainingMax * (1 + alpha)`, because a doc's final score is
+`raw × authority multiplier` and that multiplier tops out at `1 + alpha`. So
+alpha is a latency knob as well as a ranking knob: **higher alpha prunes less**
+(queries whose `theta / remainingMax` lands in `[1, 1+alpha)` now scan the
+lower-IDF term's full posting list and grow the per-query score map
+accordingly); **lower alpha prunes more**.
+
+If BM25 p50 regresses, lower alpha first — it narrows the non-pruning band
+proportionally and keeps ranking sound. `COSIFT_BM25_DISABLE_MAXSCORE=1` is the
+opposite of a mitigation: it removes pruning entirely and makes common-term
+queries strictly slower. Alpha is read once at server start, so changing it
+needs a restart (unlike the two pool vars above).
+
+Phrase queries (`"like this"`) skip MaxScore unconditionally and pay the full
+scan regardless of alpha: the threshold is a k-th over all scored docs, which
+says nothing about the k-th of the phrase-filtered subset.
+
 ## Latency budget
 
 ### Don't enrich what you don't need: `?enrich=false`

internal/index/pebble_bm25.go

@@ -55,6 +55,46 @@ func bm25TopKPoolFactor() int {
 	return 50
 }
 
+// bm25RankDepth floors the depth at which ranking decisions are made, so they
+// stop depending on the caller's k.
+//
+// Two decisions are k-shaped. The MaxScore break compares against
+// kthLargest(scores, k): a small k gives a higher theta, so the scan stops
+// earlier and produces a different score map. The resolution pool is factor*k:
+// a small k resolves fewer candidates. Measured on prod 2026-09-15 at factor
+// 200, 51 of 60 goldens returned a different top-10 at k=10 than at k=50, and a
+// median cap-bound query resolved 2,000 of ~68,500 in-band candidates.
+//
+// Running both at kEff = max(k, depth) and truncating afterwards makes the
+// top-k of any k <= depth a prefix of the same ranking. The cost is that every
+// query pays the depth-k price. 0 keeps the k-shaped behaviour.
+func bm25RankDepth() int {
+	if v := os.Getenv("COSIFT_BM25_RANK_DEPTH"); v != "" {
+		if n, err := strconv.Atoi(v); err == nil && n >= 0 {
+			return n
+		}
+	}
+	return 0
+}
+
+// bm25EffectiveK is the internal ranking depth for a caller asking for k.
+func bm25EffectiveK(k int) int {
+	if k <= 0 {
+		return k
+	}
+	if d := bm25RankDepth(); d > k {
+		return d
+	}
+	return k
+}
+
+// bm25MaxScoreAuthorityBound reports whether the MaxScore early-termination
+// bound accounts for the authority multiplier applied after the scan. 0 selects
+// the pre-2026-09 behaviour, which prunes harder and is measurably less exact.
+func bm25MaxScoreAuthorityBound() bool {
+	return os.Getenv("COSIFT_BM25_MAXSCORE_AUTHORITY") != "0"
+}
+
 // PebbleBM25 mirrors BM25 but reads from a PebbleStore.
 type PebbleBM25 struct {
 	store *store.PebbleStore
@@ -94,6 +134,14 @@ func (b *PebbleBM25) WithAuthority(a *authority.Scorer) *PebbleBM25 {
 	return b
 }
 
+// maxAuthorityMult upper-bounds Scorer.Multiplier (= 1 + alpha*score, score in [0,1]).
+func (b *PebbleBM25) maxAuthorityMult() float64 {
+	if b.authority == nil {
+		return 1.0
+	}
+	return 1.0 + b.authority.Alpha()
+}
+
 // WithBoost returns a shallow copy of b with the given docID→multiplier map
 // applied post-scoring. Use this for site= queries: enumerate the site's
 // docIDs, pass a 50× multiplier, and the site's docs will always appear in
@@ -214,32 +262,28 @@ func (b *PebbleBM25) Search(ctx context.Context, q string, k int) ([]Hit, error)
 	// The BM25 contribution of one term to any doc is bounded above by
 	// idf*(k1+1) — saturates as tf grows relative to docLen. Sum of these
 	// upper bounds across remaining terms = the maximum score a doc not
-	// yet seen can ever accumulate. If that sum drops below the current
-	// K-th best partial score, no future doc can enter top-K. Top-K
-	// membership stays lossless; in-top-K ranking can shift (the rerank
-	// pipeline re-orders anyway, and the score-decay/MMR stages tolerate
-	// approximate partial scores). COSIFT_BM25_DISABLE_MAXSCORE=1 disables
-	// the optimization for benchmark-grade lossless ranking.
-	maxScoreEnabled := os.Getenv("COSIFT_BM25_DISABLE_MAXSCORE") == ""
+	// yet seen can ever accumulate. Docs already in `scores` stop
+	// accumulating at the break too, so top-K membership is approximate,
+	// not lossless. COSIFT_BM25_DISABLE_MAXSCORE=1 disables the
+	// optimization for benchmark-grade lossless ranking. Phrase queries opt
+	// out: theta cannot threshold the phrase-filtered subset (empty results).
+	maxScoreEnabled := os.Getenv("COSIFT_BM25_DISABLE_MAXSCORE") == "" && len(phrases) == 0
+	kEff := bm25EffectiveK(k)
+	maxMult := 1.0
+	if bm25MaxScoreAuthorityBound() {
+		maxMult = b.maxAuthorityMult()
+	}
 	remainingMax := 0.0
 	if maxScoreEnabled {
 		for _, c := range active {
 			remainingMax += c.idf * (b.k1 + 1.0)
 		}
 	}
 	for i, c := range active {
-		// Pre-scan MaxScore check: decide whether term i (and everything
-		// after it, since active is sorted descending by IDF) can still push
-		// an unseen doc into top-K BEFORE paying to scan term i's postings.
-		// remainingMax here is the sum of max contributions of terms i..end
-		// (idf*(k1+1) each). A doc not yet in `scores` can gain at most
-		// remainingMax from the un-scanned terms; if that's below the current
-		// K-th best score, scanning i..end can only reorder within top-K —
-		// which the reranker fixes — so stop. This catches the common-term
-		// last-term full scan the old post-scan i==len-1 guard always paid.
-		if maxScoreEnabled && i > 0 && len(scores) >= k {
-			theta := kthLargest(scores, k)
-			if remainingMax < theta {
+		// theta is raw; an unseen doc's final ceiling is remainingMax*maxMult.
+		if maxScoreEnabled && i > 0 && len(scores) >= kEff {
+			theta := kthLargest(scores, kEff)
+			if remainingMax*maxMult < theta {
 				break
 			}
 		}
@@ -268,13 +312,12 @@ func (b *PebbleBM25) Search(ctx context.Context, q string, k int) ([]Hit, error)
 
 	// Apply per-doc boosts (e.g. site= queries) before sorting. Boosted docs
 	// already present in `scores` (i.e. they matched ≥1 query term) get the
-	// full multiplier. Boosted docs with zero term overlap are absent from
-	// `scores`; for small boost sets we seed them with a tiny base score
-	// (boostSeedBase) before multiplying so they still enter the candidate
-	// pool — landing at boostSeedBase*mult (e.g. 0.05), below any genuine
-	// single-term BM25 match but enough for the reranker to judge them. This
-	// also closes the MaxScore early-termination gap: a boosted doc whose only
-	// matching posting list was skipped still gets seeded and surfaced.
+	// full multiplier on their (possibly MaxScore-truncated) partial. Boosted
+	// docs with zero term overlap are absent from `scores`; for small boost
+	// sets we seed them with a tiny base score (boostSeedBase) before
+	// multiplying so they still enter the candidate pool — landing at
+	// boostSeedBase*mult (e.g. 0.05), below any genuine single-term BM25
+	// match but enough for the reranker to judge them.
 	//
 	// Seeding is gated by boostSeedMaxIDs: large boost sets (big sites) would
 	// inject a GetDocMeta per zero-overlap doc and flood the pool, so above
@@ -293,7 +336,14 @@ func (b *PebbleBM25) Search(ctx context.Context, q string, k int) ([]Hit, error)
 	// resolved; COSIFT_BM25_DISABLE_TOPK_POOL restores the resolve-all
 	// path (also taken for k<=0 = "return everything").
 	if k > 0 && os.Getenv("COSIFT_BM25_DISABLE_TOPK_POOL") == "" {
-		return b.resolveTopKPool(ctx, scores, phrases, k)
+		hits, err := b.resolveTopKPool(ctx, scores, phrases, kEff)
+		if err != nil {
+			return nil, err
+		}
+		if len(hits) > k {
+			hits = hits[:k]
+		}
+		return hits, nil
 	}
 
 	hits := make([]Hit, 0, len(scores))
@@ -342,10 +392,7 @@ const topKResolveSlack = 16
 // the cap does bind (more than factor*k candidates inside that band) the
 // truncation is logged — the operator signal to raise the factor.
 func (b *PebbleBM25) resolveTopKPool(ctx context.Context, scores map[int64]float64, phrases []string, k int) ([]Hit, error) {
-	maxMult := 1.0
-	if b.authority != nil {
-		maxMult += b.authority.Alpha()
-	}
+	maxMult := b.maxAuthorityMult()
 
 	poolCap := bm25TopKPoolFactor() * k
 	pool := topCandidates(scores, poolCap)