fix(index): reject incompatible vector segments at segment commit

[u70b3]

Problem

Dataset::commit_existing_index_segments validated only index_details.type_url across the segment set, so vector segments built with different distance metrics (or dimensions, sub-index types, or quantizer kinds) could be committed as one logical index. Scan planning derives the metric from the first segment and applies it to the whole logical index, so the remaining segments would be searched and ranked under the wrong metric — silently returning incorrect nearest-neighbor results.

Reproduced via the distributed build loop (the path lance-c's lance_dataset_commit_index_segments binds): build fragment 0 as IVF_FLAT/L2 and fragment 1 as IVF_FLAT/Cosine, then commit both metadata blobs — the commit succeeds at ab6b5bbe.

Surfaced during review of lance-format/lance-c#84.

Fix

After replacement selection, validate the coexisting segment set — incoming segments plus retained existing segments — with the same validate_vector_query_compatibility check the query/optimize paths already require (metric, dimension, sub-index type, quantizer kind; independently trained IVF centroids and PQ codebooks may still differ). Incompatible sets are rejected with InvalidInput before any manifest change, so the dataset version and existing index are untouched.

Because incoming segments are not in the manifest yet, the validation opens each segment through a new Dataset::open_vector_index_from_metadata helper — DatasetIndexInternalExt::open_vector_index minus the manifest load_index lookup, extracted mechanically with no behavior change for committed segments.

A complete replacement may still change the metric: replaced segments are no longer part of the coexisting set, so they impose no constraint.

Tests

Three new tests in rust/lance/src/index/vector/ivf/v2.rs:

• two incoming segments with L2 + Cosine metrics over disjoint fragments are rejected (error names the metric; version unchanged, no index created);
• an incoming Cosine segment conflicting with a retained L2 segment is rejected (retained index untouched);
• a full-coverage replacement switching L2 → Cosine is accepted.

test_commit_existing_index_segments_commits_multiple_segments previously committed metadata-only fake segments (payload b"seg0"); since compatibility validation now opens coexisting segments, it builds real IVF segments via execute_uncommitted instead — same assertions otherwise.

• cargo test -p lance --lib: 3069 passed, 0 failed
• cargo clippy -p lance --lib --tests: clean

Targeting release/v11.0 so lance-c can pin a v11-line revision containing the fix (its current pin ab6b5bbe = v11.0.0). Happy to forward-port to main if you'd like it there first.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals	none (0/3)
Latest commit approved by	none — one PMC member must approve the latest commit
Vetoes	none
Voting period	ends Tue 2026-09-22 15:42 UTC (08:42 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[lance-gatekeeper]

✅ Gate recommendation: approve.

The index compatibility fix and synchronized rustls update remain sound. The added compatibility-harness tier is the same change already merged in #8903: it keeps pylance 12.0.0b5 on lance-namespace 0.8 and selects the compatible 0.11 range for later releases.

[u70b3]

Superseded by #9432 (identical commits). A stale format-change label — applied while this PR briefly targeted main, where the branch-vs-main diff included the release branch's proto changes — keeps the format-spec-vote gate blocking merge, and the label can't be removed from a fork.