Skip to content

fix(index): enforce the precomputed-input contracts in the current IVF build path - #9376

Open
LuciferYang wants to merge 3 commits into
lance-format:mainfrom
LuciferYang:fix/ivf-rq-precomputed-buffers-contract
Open

LuciferYang wants to merge 3 commits into
lance-format:mainfrom
LuciferYang:fix/ivf-rq-precomputed-buffers-contract

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Closes #9375

IvfBuildParams documents that precomputed_shuffle_buffers and precomputed_partitions_file are mutually exclusive and that the buffers require centroids. sanity_check_ivf_params enforces that, but only the legacy IVF_PQ and IVF_HNSW_PQ writers reach it, so IvfIndexBuilder accepted all three broken combinations. Buffers without centroids produced an index whose stored centroids came from a fresh sample while its rows kept the __ivf_part_id the buffers were written with, so queries probed partitions chosen by centroids that say nothing about where the rows went. Both inputs at once panicked in a worker task on a duplicate __ivf_part_id column.

The contract now lives as IvfBuildParams::validate(), next to the field comments that state it, and IvfIndexBuilder::new calls it, so the build fails at construction with an invalid_input error naming which pairing is wrong. The legacy writer keeps its own copy of the checks: it is a frozen write path, and routing it through the new method would change the error variant it has always returned.

Two combinations that used to be accepted are now rejected, and both were already broken. accelerator=... together with precomputed_partition_dataset=... sets both fields from Python and panicked inside tokio::spawn; that is now a clean error. Passing precomputed_partitions_file or precomputed_shuffle_buffers as a raw **kwargs entry without ivf_centroids bypasses the guard in dataset.py and silently built a mismatched index. Every first-party flow supplies centroids alongside a precomputed input, since the accelerator path overwrites ivf_centroids with what it just trained, the precomputed_partition_dataset path raises without them, and the distributed path requires precomputed centroids. So nothing that worked stops working.

One rule from the legacy check is deliberately not carried over: the buffers also require a PQ codebook. Quantization and QuantizerBuildParams expose no way to ask whether a model was supplied, so that check needs a trait change or per-quantizer handling at the dispatch sites rather than a line in IvfBuildParams.

How was this patch tested?

test_validate_rejects_precomputed_inputs covers the three rejected shapes and asserts the error variant and the message, test_validate_accepts_supported_combinations pins the shapes that must stay accepted, and test_new_rejects_precomputed_buffers_without_centroids pins that the V3 builder calls the check itself rather than leaving it to a caller. Dropping the call from new fails that last test, and removing either rejection arm fails its case.

The field docs promise that precomputed_shuffle_buffers is mutually
exclusive with precomputed_partitions_file and requires centroids, but
neither was validated. Combining both silently used two different
assignment sources, and building without centroids trained fresh ones
while the buffers were produced against different centroids — silently
wrong search results. Enforce both contracts at the start of build().

Assisted-by: GLM-5.3
@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Sep 18, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

This correctly enforces the centroid and mutual-exclusion contracts at the current V3 builder boundary while leaving the frozen legacy writer unchanged. The remaining risk is the existing V3 gap called out in the PR: precomputed PQ buffers can still be accepted without the originating PQ codebook, allowing raw callers to pair stored codes with a newly trained model and receive incorrect distances. The quantizer abstraction does not currently expose enough model provenance to close that gap in this focused patch.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: the current IVF build path does not enforce the precomputed-input contracts

1 participant