feat(format): record the requested IVF partition count on a vector index

[xuanyu-z]

Proposes recording the IVF partition count a vector index was asked to train, as target_num_partitions on VectorIndexDetails.

A vector index segment records its build parameters in index_details so an engine can rebuild the index without reading the index files. The partition count is not among them: target_partition_size records a target size per partition, and nothing records a requested count. An index rebuilt from its details is therefore auto-sized from the data, so an index created as IVF-8 comes back with whatever count the data suggests.

This matters most for a segment that covers no fragments. Such a segment carries its details with no index files at all, so the recorded parameters are the only statement of what the index is to become once its column holds enough vectors to train — and without the count it cannot reliably preserve the requested shape however far the table grows. That is the case behind #4034, where creating a vector index on a table too small to train its quantizer records the definition and fills it in later.

The field is a request, not a description of the index that was built. A build trains fewer partitions when the data cannot support the count asked for, so the partitions an index holds are read from the index itself. The name pairs with target_partition_size, the other way of asking for a partitioning, and the spec states that an explicit count wins when both are set. Absent means no count was requested, and a rebuild derives the partitioning from the data again.

runtime_hints is not the place for this. It is specified as optional build preferences that do not affect index structure, and requires that unrecognized keys be silently ignored. A partition count is exactly index structure — it is the number that defines the index's shape — so recording it there would contradict the field's stated contract and make an ignorable key load-bearing for readers that do recognize it.

A recorded count is at least 1. Zero partitions already fails badly on the build
path — KMeans indexes into an empty cluster list and panics — and a count is
easy to compute as zero, since rows / target is zero on a small table. What a
recorded count would add is distance: a definition writes no index, so nothing
rejects the count before the details record it, and the failure would surface on
a later rebuild instead of at the call that set it. The spec therefore requires a
reader to treat zero as no request, and a writer to record nothing rather than a
truncated value.

Field number 10 rather than 9's successor 7: the existing compression oneof skips 7 and the message carries no reserved statement, so I left the gap alone rather than assume it is free.

The new field is set nowhere in this PR; the five exhaustive struct literals updated are the minimum needed to compile. Reading and writing it lands in a follow-up alongside #9134.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals (this commit)	none (0/3)
Vetoes	none
Voting period	elapsed — ended Wed 2026-09-16 14:22 UTC (07:22 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[xuanyu-z]

@jackye1995 @hamersaw @wjones127

[westonpace]

What's the goal here? I worry that target_num_partitions can become a bad target as a table grows. For example, let's say I have 100Ki rows and so I want 25 partitions and I set target_num_partitions=25. Now my table grows to 200Ki rows. Should I still respect target_num_partitions=25?

Maybe it makes more sense to just think of num_partitions as an alternate way of specifying target_partition_size. For example, if I create an index with num_partitions=25 and there are 100Ki rows then I store target_partition_size=4096. If there are 200Ki rows then I store target_partition_size=8192.

[westonpace]

It might be nice to give some examples here (there are examples in the protobuf message comment we can copy over)

[westonpace]

It's odd that we use target_num_partitions here and num_partitions in all other contexts.

[wjones127]

+1 to Weston's comment. We excluded num_partitions intentionally for that reason. It's not a good parameter, unless you assume the dataset is a fixed size, in which case it's okay but no better than target_partition_size.

[lance-gatekeeper]

🟡 Gate recommendation: maintainer decision required.

The latest maintainer feedback confirms that num_partitions was intentionally excluded because a fixed count becomes a poor target as the dataset grows. The deferred-training case still needs a durable statement of creation intent, but the format contract now has two materially different choices: persist target_num_partitions as a fixed rebuild target that preserves the exact requested shape, or convert the creation-time count into target_partition_size so later rebuilds adapt as the table grows. Maintainers need to choose which invariant should survive rebuilds before this field is made part of the format.