perf(knn): share candidate reads for batched refinement

[TheR1sing3un]

Batched ANN queries with refine_factor currently execute a separate search and vector take for each query. This shares the IVF search, deduplicates candidate vector reads, and computes a separate top-k over each query's original candidates. The existing query API and file format are unchanged.

The fast path uses fixed positive nprobes with supported flat-style IVF indices. Adaptive probes, HNSW, incomplete index coverage, stale vector overlays, and external row masks retain per-query execution. Vector reads and scoring gathers target 32 MiB chunks; candidate metadata and top-k heaps use additional memory. Metrics expose vectors read, read batches, and peak vector buffer bytes.

Performance

Apple M5 Pro, local disk, release-with-debug; 32,768 synthetic 1,024-dimensional Float32 vectors across eight fragments, IVF_SQ/L2, 64 partitions, nprobes=16, k=10, refine_factor=10. Warm index cache and OS cache, 21 randomized/interleaved repetitions. Median complete-batch latency:

Queries	Batch size	Serial singles	Eight concurrent singles	Shared batch
Independent	8	16.69 ms	7.71 ms	9.11 ms
Independent	32	72.04 ms	30.27 ms	30.31 ms
Independent	64	148.24 ms	58.19 ms	54.45 ms
Nearby	8	18.31 ms	8.26 ms	5.41 ms
Nearby	32	74.15 ms	30.86 ms	15.59 ms
Nearby	64	148.72 ms	57.61 ms	28.52 ms

At B=64, nearby queries read 136 distinct vectors for 6,400 candidate references; independent queries read 5,813. The nearby workload is 2.02x faster than eight concurrent singles; independent queries show little benefit and B=8 is about 18% slower. An initial 11-repeat run reproduced the same pattern (1.89x nearby, approximately equal independent at B=64). These compare query execution modes on the same build, not before/after native batch implementations. They are not remote-storage or cold-device measurements.

Validation

• 85 targeted Rust tests and 16 Python batch/refinement tests passed, covering result equivalence, recall, filters, deletions, NULL/NaN vectors, candidate isolation, repeated candidates, ties across chunks, and fallbacks. Python tests also passed on the final optimized extension.
• All 20,160 timed query executions across both benchmark runs matched single-query result IDs and distances within floating-point tolerance. Runs reused the same index and fixed query sets. Exact recall@10 over eight queries per distribution: 0.6875 independent, 0.9875 nearby.
• Workspace Clippy, Rust formatting, Python full lint, benchmark Ruff/Pyright and commit hooks passed.
• Reproduce with uv run --no-sync python python/benchmarks/batch_refinement.py --directory /tmp/native-batch-refinement --output /tmp/native-batch-refinement/results.json --repeats 21 after building with release-with-debug.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The dedicated refinement stage keeps each query's candidate set isolated while sharing original-vector reads, and it preserves the established batch-IVF fallbacks for adaptive probes, HNSW, incomplete coverage, stale overlays, and external masks. Keeping refinement above the index layer avoids coupling dataset reads into the IVF implementation while retaining repeated-single-query semantics.