Conversation
Main's global-top-k path waits for 64 MiB / 128 partitions before the first score, and prepares only as many partitions as CPU cores. On a cold IVF_RQ query that serializes I/O and scoring. Gate four independent opts behind LANCE_IVF_QUERY_OPTS (default all): - first_wave: score after the first prepare wave - io_prepare: prepare concurrency follows I/O parallelism - parallel_files: load index.idx and auxiliary.idx together - overlap_prefilter: start partition I/O without waiting for prefilter Co-authored-by: Yang Cen <BubbleCal@users.noreply.github.com>
Load every auxiliary column except RaBitQ codes when the index is opened (`LANCE_IVF_QUERY_OPTS=preload_aux`). Query-time partition loads then read only the code columns and merge the resident side buffers. The preloaded batch is stashed on IvfIndexState so in-process reconstruction does not re-read the side columns. Not enabled by `all`. Co-authored-by: Yang Cen <BubbleCal@users.noreply.github.com>
This reverts commit e3c266e.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main’s IVF query path already loads partitions one-by-one. The remaining cold-query cost is scheduling: the global-top-k path waits for 64 MiB / 128 partitions before the first score, and
prepareconcurrency follows CPU count (ncpu - 2), not I/O parallelism.This PR keeps per-partition loads and adds four independently gated optimizations. Unset
LANCE_IVF_QUERY_OPTSenables all of them;noneis main’s behavior; a comma-separated list enables one opt at a time.first_wave: score after the first prepare wave. Later chunks keep the 64 MiB / 128 bound so a high-nprobeswarm query does not pay a dispatch per wave.io_prepare: prepare/load concurrency ismax(io_parallelism, cpu)(local default 8, not 2 on a 4-core box).parallel_files:try_joinindex.idxandauxiliary.idxinload_partition_entry. Neutral for IVF_RQ (empty index rows); helps IVF_PQ/HNSW.overlap_prefilter: start partition I/O without waiting for the prefilter to become ready.Isolated cold-query results
Open-excluded protocol on local disk: untimed
get_ivf_model/centroids,posix_fadvise(DONTNEED)on_indices, then one search. Each row is one opt vsnone(main). IOPS, bytes, and recall were identical across every opt.Mean cold latency (ms), 4-core host, local
io_parallelism=8:Takeaways:
io_prepareis the largest isolated win, especially at nprobes=80 (more in-flight partition reads).first_wavehelps when the whole probe set is under 64 MiB (RQ1). It does little once a single wave already fills the byte budget (RQ5).parallel_filesandoverlap_prefilterare ~neutral on these IVF_RQ fixtures (emptyindex.idxrows, no deletion prefilter). They stay inallbecause they are the right shape for IVF_PQ/HNSW and filtered queries.