perf(fts): cut per-window and per-candidate overhead in bulk AND search

[BubbleCal]

Performance issue

A production FTS workload that is mostly 1–3-token MatchQuery AND searches regressed between the 2026-08-14 and 2026-09-14 releases. Bisecting lance with production build flags on long posting lists showed two sources:

• fix(fts): preserve exact wand score bounds #8666 (exact WAND score bounds) made every bulk-AND window fill its 64-bucket frequency prune table with 64 score_sum_cannot_compete evaluations and re-ran the same exact bound for every surviving candidate. That is +18–22% CPU per 2/3-token query; the AVX2 merge kernels and bitpack decoding were unchanged and index_comparisons did not move.
• perf(fts): defer decoding for underfilled conjunctions #9031 (lazy frequency decode) left doc() re-materializing the frequency for every candidate, which single-clause queries pay on every candidate of every block they decode (+6% CPU on 1-token queries).

How this PR improves it

• exclusive_partial_score_limit computes, once per (floor, followers' block max) pair, the largest f32 partial score that score_sum_cannot_compete rejects under the exclusive floor. Both the frequency prune table and the per-candidate prune are monotone in that score, so each becomes a single f32 comparison. The decisions are bit-identical to the exact bound; a property test sweeps ULP neighbourhoods of the boundary across floors, remaining bounds and clause counts.
• next_doc_id carries the candidate's frequency when the block's frequency stream is already decoded, so doc() takes its fast path instead of re-materializing per candidate. Frequencies stay lazy for blocks that are never scored.
• Bulk conjunction windows jump to the lead clause's next document after an empty window, resume each clause's slice from the previous slice end instead of a fresh binary search, rebuild the prune table only when the floor or the followers' block maxes change, run the generic (4+ clause) cursor merge as an AVX2 kernel, and stop preparing score-first windows the bulk path never reads.

No query API, scoring, or index format changes.

Correctness

• Result dumps (row ids and f32 scores, k=10) for 14 query sets on two corpora are bit-identical between main and this PR, and between this PR and the Sept-14 release wheel (score_bit_diff=0, row_set_diff=0; the synthetic corpus only differs in tie order, exactly as two builds of the same commit do).
• cargo test -p lance-index --lib scalar::inverted: 788 passed. cargo test -p lance --lib fts: 249 passed. New tests: exclusive_partial_score_limit_matches_predicate, exclusive_partial_score_limit_handles_non_finite_inputs, bulk_and_windows_jump_to_next_lead_document.
• cargo fmt --all clean; cargo clippy -p lance-index --tests -- -D warnings clean.

Benchmark

Environment: AWS c7i.16xlarge (Sapphire Rapids, 64 vCPU, 123 GiB), Ubuntu 24.04, rustc 1.97.0. Each build is a pylance wheel built with the production plan-executor flags (--release, -C target-cpu=skylake -C target-feature=+avx2,+fma,+f16c -C force-frame-pointers=yes -C debuginfo=1). Baseline is main at 2602724; this PR is that commit plus these four commits; the Aug-14 release column (v11.0.0-beta.10) is the pre-regression reference. Only the wheel changes between runs.

Datasets: mmlb_10m (10M synthetic ~900-token documents, ~10K vocabulary, long posting lists, V3 index, block size 256, 6 partitions) and MS MARCO passages (8.8M real passages, short posting lists, 1 partition). Queries: 300 per set, k=10, MatchQuery with operator=AND, index prewarmed; "mid-df" = tokens with document frequency 1–6%, "sequential" = consecutive tokens from real documents (any frequency), "rare" = df 0.02–0.3%, "rare+common" = at least one rare and one common token.

Methodology: closed loop from one Python process, 3 s warm-up then 12 s measured per cell, three passes in forward/reverse/forward order, values are means over the three passes. CPU ms per query is process CPU time divided by completed queries (the metric that maps to plan-executor saturation); p50 is the client-observed latency. Lower is better for both. Concurrency 1 isolates per-query cost; concurrency 8 shows the same under moderate parallelism.

Concurrency 1, CPU ms per query (lower is better; mean of N passes)

Scenario / metric	Aug-14 release (v11.0.0-beta.10)	Baseline main 2602724	This PR	Benefit vs main
mmlb mid-df 1-token, CPU ms/query	1.83 ms	1.76 ms	1.66 ms	1.06x speedup
mmlb mid-df 2-token AND, CPU ms/query	6.62 ms	8.16 ms	6.65 ms	1.23x speedup
mmlb mid-df 3-token AND, CPU ms/query	12.71 ms	16.37 ms	12.89 ms	1.27x speedup
mmlb sequential 2-token AND, CPU ms/query	9.84 ms	12.12 ms	10.05 ms	1.21x speedup
mmlb sequential 3-token AND, CPU ms/query	25.47 ms	32.54 ms	26.54 ms	1.23x speedup
mmlb sequential 5-token AND, CPU ms/query	54.81 ms	52.94 ms	43.42 ms	1.22x speedup
msmarco rare 2-token AND, CPU ms/query	1.17 ms	1.03 ms	1.00 ms	1.03x speedup
msmarco rare 3-token AND, CPU ms/query	1.15 ms	0.98 ms	0.96 ms	1.02x speedup
msmarco rare+common 2-token AND, CPU ms/query	1.91 ms	1.89 ms	1.71 ms	1.10x speedup
msmarco rare+common 3-token AND, CPU ms/query	1.88 ms	1.79 ms	1.58 ms	1.13x speedup
msmarco rare+common 5-token AND, CPU ms/query	1.34 ms	1.78 ms	1.55 ms	1.14x speedup
msmarco sequential 3-token AND, CPU ms/query	1.29 ms	1.14 ms	1.08 ms	1.06x speedup
msmarco sequential 5-token AND, CPU ms/query	1.28 ms	1.54 ms	1.39 ms	1.11x speedup

Concurrency 1, p50 latency ms (lower is better)

Scenario / metric	Aug-14 release	Baseline main	This PR	Benefit vs main
mmlb mid-df 1-token, p50	2.93 ms	2.06 ms	1.96 ms	1.05x speedup
mmlb mid-df 2-token AND, p50	6.73 ms	7.22 ms	6.10 ms	1.18x speedup
mmlb mid-df 3-token AND, p50	10.54 ms	11.68 ms	9.76 ms	1.20x speedup
mmlb sequential 2-token AND, p50	10.02 ms	11.14 ms	9.38 ms	1.19x speedup
mmlb sequential 3-token AND, p50	24.14 ms	28.31 ms	24.10 ms	1.17x speedup
mmlb sequential 5-token AND, p50	40.07 ms	46.02 ms	37.55 ms	1.23x speedup
msmarco rare 2-token AND, p50	2.23 ms	1.29 ms	1.25 ms	1.04x speedup
msmarco rare 3-token AND, p50	2.21 ms	1.25 ms	1.23 ms	1.02x speedup
msmarco rare+common 2-token AND, p50	2.83 ms	2.02 ms	1.87 ms	1.08x speedup
msmarco rare+common 3-token AND, p50	2.88 ms	1.95 ms	1.77 ms	1.10x speedup
msmarco rare+common 5-token AND, p50	2.31 ms	2.01 ms	1.81 ms	1.11x speedup
msmarco sequential 3-token AND, p50	2.27 ms	1.35 ms	1.28 ms	1.05x speedup
msmarco sequential 5-token AND, p50	2.27 ms	1.74 ms	1.59 ms	1.09x speedup

Concurrency 8, CPU ms per query (lower is better; mean of N passes)

Scenario / metric	Aug-14 release (v11.0.0-beta.10)	Baseline main 2602724	This PR	Benefit vs main
mmlb mid-df 1-token, CPU ms/query	1.79 ms	1.87 ms	1.78 ms	1.05x speedup
mmlb mid-df 2-token AND, CPU ms/query	6.46 ms	8.06 ms	6.65 ms	1.21x speedup
mmlb mid-df 3-token AND, CPU ms/query	11.95 ms	15.09 ms	12.30 ms	1.23x speedup
mmlb sequential 2-token AND, CPU ms/query	9.33 ms	11.86 ms	9.72 ms	1.22x speedup
mmlb sequential 3-token AND, CPU ms/query	26.03 ms	34.00 ms	27.33 ms	1.24x speedup
mmlb sequential 5-token AND, CPU ms/query	49.42 ms	49.15 ms	40.85 ms	1.20x speedup
msmarco rare 2-token AND, CPU ms/query	1.12 ms	1.14 ms	1.11 ms	1.02x speedup
msmarco rare 3-token AND, CPU ms/query	1.11 ms	1.09 ms	1.07 ms	1.02x speedup
msmarco rare+common 2-token AND, CPU ms/query	1.81 ms	1.91 ms	1.76 ms	1.09x speedup
msmarco rare+common 3-token AND, CPU ms/query	1.84 ms	1.89 ms	1.70 ms	1.11x speedup
msmarco rare+common 5-token AND, CPU ms/query	1.28 ms	1.85 ms	1.64 ms	1.13x speedup
msmarco sequential 3-token AND, CPU ms/query	1.24 ms	1.25 ms	1.18 ms	1.06x speedup
msmarco sequential 5-token AND, CPU ms/query	1.23 ms	1.64 ms	1.48 ms	1.11x speedup

Concurrency 8, p50 latency ms (lower is better)

Scenario / metric	Aug-14 release	Baseline main	This PR	Benefit vs main
mmlb mid-df 1-token, p50	3.85 ms	3.21 ms	3.34 ms	0.96x speedup
mmlb mid-df 2-token AND, p50	7.03 ms	7.29 ms	6.22 ms	1.17x speedup
mmlb mid-df 3-token AND, p50	9.86 ms	10.72 ms	9.04 ms	1.18x speedup
mmlb sequential 2-token AND, p50	9.90 ms	11.02 ms	9.18 ms	1.20x speedup
mmlb sequential 3-token AND, p50	23.19 ms	28.42 ms	23.07 ms	1.23x speedup
mmlb sequential 5-token AND, p50	37.18 ms	44.68 ms	36.61 ms	1.22x speedup
msmarco rare 2-token AND, p50	3.59 ms	3.69 ms	3.76 ms	0.98x speedup
msmarco rare 3-token AND, p50	3.63 ms	3.72 ms	3.73 ms	1.00x speedup
msmarco rare+common 2-token AND, p50	3.82 ms	3.24 ms	3.49 ms	0.93x speedup
msmarco rare+common 3-token AND, p50	3.90 ms	3.35 ms	3.53 ms	0.95x speedup
msmarco rare+common 5-token AND, p50	3.55 ms	3.35 ms	3.43 ms	0.98x speedup
msmarco sequential 3-token AND, p50	3.63 ms	3.66 ms	3.75 ms	0.97x speedup
msmarco sequential 5-token AND, p50	3.65 ms	3.63 ms	3.55 ms	1.02x speedup

Limitations

• 5-token AND over short posting lists is still ~8–13% above the Aug-14 release in CPU: perf(fts): accelerate wide and queries with bulk intersection #9030 routes 4+ clause conjunctions into the pairwise bulk intersection unconditionally, which does not pay for itself on short lists. Left for a follow-up with a posting-length heuristic; this PR already recovers 1.1–1.2x on those queries.
• The mid-df 1-token latency at concurrency 8 is within noise of main (0.96x) while its CPU improves 1.05x.

🤖 Generated with Claude Code

[LuQQiu]

From codex: Could this skip initializing freq_cannot_beat after a zero-to-positive floor transition?

If a window starts with threshold == 0, the per-candidate path below can later set first_score_limit_key once the heap fills, while freq_cannot_beat is still the all-false default. If the next window has the same threshold and others_block_max, this key comparison would skip the body, so the frequency LUT would remain disabled until the key changes.

I may be missing an invariant that forces the key to change between those windows. Could we add a focused two-window test where the floor becomes positive mid-window and the follower bounds stay unchanged, and verify that the second window actually applies frequency pruning?

[BubbleCal]

Good catch, this was real. When a window starts with a zero floor, the heap can fill in pass B and the per-candidate prune caches the limit under (threshold, others_block_max), but the table stays all-false. If the next window has the same floor and the same follower block max, the shared key matched and the table was never built, so kernel-level frequency pruning stayed off until the key changed. Results were not affected (the per-candidate prune uses the same limit), only the pruning was lost.

Fixed in 9834e75 by keying the table separately from the limit, so each is rebuilt only when it is stale. I added the two-window test you described (bulk_and_builds_frequency_prune_table_after_mid_window_floor): the floor turns positive on the first candidate of window 1, the dense clause has the same block max in both blocks so the key is unchanged, and the test asserts that window 2 applies the prune table. It fails on the previous commit (0 pruned windows) and passes now; results and the final floor still match the classic loop.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The current revision fixes the mid-window floor-transition cache gap identified in the review discussion. The cache is now keyed after the transition, and the focused regression plus the WAND suite on the refreshed base pass; scoring and result membership remain unchanged.

The remaining non-blocking risk is rollout performance: the benchmark driver and raw runs are not in the tree, several concurrency-8 latency cells are flat or slightly slower, and 5-token short-list CPU still inherits #9030 routing overhead. Monitor production query-shape buckets; LANCE_FTS_BULK_AND=off remains the bounded rollback for conjunction regressions. No code change is requested.