feat(format): record the requested IVF partition count on a vector index

docs/src/format/index/vector/index.md

@@ -53,6 +53,42 @@ The Lance vector index format has gone through 3 versions so far.
 This document currently only records version 3 which is the latest version.
 The specific version of the vector index is recorded in the `index_version` field of the generic [index metadata](../index.md#loading-an-index).
 
+### Index Details
+
+An index segment records its build parameters in the `index_details` field of the
+generic [index metadata](../index.md#loading-an-index), as a `VectorIndexDetails`
+message. These are the parameters an engine needs to rebuild the index without
+reading the index files: the distance metric, the quantization scheme, and the
+partitioning the index was asked for.
+
+`target_num_partitions` records the IVF partition count requested for building or
+rebuilding the index. The count an index actually holds may differ from it and is
+read from the index files. Where a segment covers no fragments and carries no
+index files, this field preserves an explicit partition-count request that has no
+built index to read it from.
+
+`target_partition_size` records the target number of vectors per partition, and is
+0 when unset. A positive `target_num_partitions` takes precedence over it. Absent
+or zero means no explicit count is available: a rebuild uses
+`target_partition_size` when that is positive, and the engine's own sizing
+otherwise. Absence does not imply the original build used automatic sizing, since
+a writer that predates this field records no count either way.
+
+Writers record only positive, representable counts, and omit a count they cannot
+represent rather than truncating it.
+
+`runtime_hints` carries optional build preferences that do not affect index
+structure, keyed by reverse-DNS name. Unrecognized keys must be silently ignored.
+
+<details>
+  <summary>Full protobuf definition</summary>
+
+```protobuf
+%%% proto.message.VectorIndexDetails %%%
+```
+
+</details>
+
 ## Storage Layout (V3)
 
 Each vector index is stored as 2 regular Lance files - index file and auxiliary file.

protos/index.proto

@@ -229,6 +229,22 @@ message VectorIndexDetails {
    * Unrecognized keys must be silently ignored by all runtimes.
    */
   map<string, string> runtime_hints = 9;
+
+  /* Requested IVF partition count for building or rebuilding this index.
+   *
+   * The actual partition count may differ from this target and is read from the
+   * index files.
+   *
+   * A positive value takes precedence over `target_partition_size`. Absent or
+   * zero means no explicit count is available: a rebuild uses
+   * `target_partition_size` when positive, and the engine's own sizing
+   * otherwise. Absence does not imply the original build used automatic sizing,
+   * since a writer that predates this field records no count either way.
+   *
+   * Writers record only positive, representable counts, and omit a count they
+   * cannot represent rather than truncating it.
+   */
+  optional uint32 target_num_partitions = 10;
 }
 
 // Hierarchical Navigable Small World (HNSW) parameters, used as an optional configuration for IVF indexes.

rust/lance/src/index/vector/details.rs

@@ -162,6 +162,7 @@ pub fn vector_index_details(params: &VectorIndexParams) -> prost_types::Any {
         hnsw_index_config,
         compression,
         runtime_hints,
+        target_num_partitions: None,
     };
     prost_types::Any::from_msg(&details).unwrap()
 }
@@ -564,6 +565,7 @@ fn convert_legacy_proto_to_details(proto: &pb::Index) -> Result<prost_types::Any
         hnsw_index_config: None,
         compression,
         runtime_hints: Default::default(),
+        target_num_partitions: None,
     };
     Ok(prost_types::Any::from_msg(&details).unwrap())
 }
@@ -703,6 +705,7 @@ async fn convert_v3_metadata_to_details(
         hnsw_index_config,
         compression,
         runtime_hints: Default::default(),
+        target_num_partitions: None,
     };
     Ok(prost_types::Any::from_msg(&details).unwrap())
 }
@@ -752,6 +755,7 @@ mod tests {
             hnsw_index_config: hnsw,
             compression,
             runtime_hints: Default::default(),
+            target_num_partitions: None,
         };
         prost_types::Any::from_msg(&details).unwrap()
     }
@@ -901,6 +905,7 @@ mod tests {
                 hnsw_index_config: None,
                 compression: None,
                 runtime_hints: Default::default(),
+                target_num_partitions: None,
             };
             prost_types::Any::from_msg(&d).unwrap()
         };