Skip to content

perf(index): overlap IVF query partition load and score - #9372

Draft
BubbleCal wants to merge 3 commits into
mainfrom
yang/ivf-query-cold-latency-d704
Draft

BubbleCal wants to merge 3 commits into
mainfrom
yang/ivf-query-cold-latency-d704

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Main’s IVF query path already loads partitions one-by-one. The remaining cold-query cost is scheduling: the global-top-k path waits for 64 MiB / 128 partitions before the first score, and prepare concurrency follows CPU count (ncpu - 2), not I/O parallelism.

This PR keeps per-partition loads and adds four independently gated optimizations. Unset LANCE_IVF_QUERY_OPTS enables all of them; none is main’s behavior; a comma-separated list enables one opt at a time.

  • first_wave: score after the first prepare wave. Later chunks keep the 64 MiB / 128 bound so a high-nprobes warm query does not pay a dispatch per wave.
  • io_prepare: prepare/load concurrency is max(io_parallelism, cpu) (local default 8, not 2 on a 4-core box).
  • parallel_files: try_join index.idx and auxiliary.idx in load_partition_entry. Neutral for IVF_RQ (empty index rows); helps IVF_PQ/HNSW.
  • overlap_prefilter: start partition I/O without waiting for the prefilter to become ready.

Isolated cold-query results

Open-excluded protocol on local disk: untimed get_ivf_model / centroids, posix_fadvise(DONTNEED) on _indices, then one search. Each row is one opt vs none (main). IOPS, bytes, and recall were identical across every opt.

Mean cold latency (ms), 4-core host, local io_parallelism=8:

Dataset Index nprobes none first_wave io_prepare parallel_files overlap_prefilter all
vibe-msmarco-qwen-1024 (8.8M) IVF_RQ1 20 82 67 (−18%) 61 (−26%) 69 68 58 (−29%)
vibe-msmarco-qwen-1024 IVF_RQ1 80 192 153 (−20%) 134 (−30%) 151 151 141 (−27%)
vibe-msmarco-qwen-1024 IVF_RQ5 20 114 109 (−5%) 108 (−5%) 109 112 104 (−9%)
vibe-msmarco-qwen-1024 IVF_RQ5 80 316 309 (−2%) 268 (−15%) 319 313 271 (−14%)
coyo-ve-qwen3vl-2048 (15.4M) IVF_RQ1 20 168 160 (−5%) 154 (−8%) 163 162 146 (−13%)
coyo-ve-qwen3vl-2048 IVF_RQ5 20 201 199 (−1%) 206 (+2%) 201 201 194 (−3%)

Takeaways:

  • io_prepare is the largest isolated win, especially at nprobes=80 (more in-flight partition reads).
  • first_wave helps when the whole probe set is under 64 MiB (RQ1). It does little once a single wave already fills the byte budget (RQ5).
  • parallel_files and overlap_prefilter are ~neutral on these IVF_RQ fixtures (empty index.idx rows, no deletion prefilter). They stay in all because they are the right shape for IVF_PQ/HNSW and filtered queries.
Open in Web Open in Cursor 

Main's global-top-k path waits for 64 MiB / 128 partitions before the
first score, and prepares only as many partitions as CPU cores. On a
cold IVF_RQ query that serializes I/O and scoring.

Gate four independent opts behind LANCE_IVF_QUERY_OPTS (default all):
- first_wave: score after the first prepare wave
- io_prepare: prepare concurrency follows I/O parallelism
- parallel_files: load index.idx and auxiliary.idx together
- overlap_prefilter: start partition I/O without waiting for prefilter

Co-authored-by: Yang Cen <BubbleCal@users.noreply.github.com>
Load every auxiliary column except RaBitQ codes when the index is opened
(`LANCE_IVF_QUERY_OPTS=preload_aux`). Query-time partition loads then
read only the code columns and merge the resident side buffers.

The preloaded batch is stashed on IvfIndexState so in-process
reconstruction does not re-read the side columns. Not enabled by `all`.

Co-authored-by: Yang Cen <BubbleCal@users.noreply.github.com>
@github-actions github-actions Bot added the A-index Vector index, linalg, tokenizer label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants