diff --git a/docs/src/format/index/vector/index.md b/docs/src/format/index/vector/index.md index 91d73a23a16..3a1b77b6cbd 100644 --- a/docs/src/format/index/vector/index.md +++ b/docs/src/format/index/vector/index.md @@ -53,6 +53,42 @@ The Lance vector index format has gone through 3 versions so far. This document currently only records version 3 which is the latest version. The specific version of the vector index is recorded in the `index_version` field of the generic [index metadata](../index.md#loading-an-index). +### Index Details + +An index segment records its build parameters in the `index_details` field of the +generic [index metadata](../index.md#loading-an-index), as a `VectorIndexDetails` +message. These are the parameters an engine needs to rebuild the index without +reading the index files: the distance metric, the quantization scheme, and the +partitioning the index was asked for. + +`target_num_partitions` records the IVF partition count requested for building or +rebuilding the index. The count an index actually holds may differ from it and is +read from the index files. Where a segment covers no fragments and carries no +index files, this field preserves an explicit partition-count request that has no +built index to read it from. + +`target_partition_size` records the target number of vectors per partition, and is +0 when unset. A positive `target_num_partitions` takes precedence over it. Absent +or zero means no explicit count is available: a rebuild uses +`target_partition_size` when that is positive, and the engine's own sizing +otherwise. Absence does not imply the original build used automatic sizing, since +a writer that predates this field records no count either way. + +Writers record only positive, representable counts, and omit a count they cannot +represent rather than truncating it. + +`runtime_hints` carries optional build preferences that do not affect index +structure, keyed by reverse-DNS name. Unrecognized keys must be silently ignored. + +
+ Full protobuf definition + +```protobuf +%%% proto.message.VectorIndexDetails %%% +``` + +
+ ## Storage Layout (V3) Each vector index is stored as 2 regular Lance files - index file and auxiliary file. diff --git a/protos/index.proto b/protos/index.proto index 8284b986bb5..e13ca934340 100644 --- a/protos/index.proto +++ b/protos/index.proto @@ -229,6 +229,22 @@ message VectorIndexDetails { * Unrecognized keys must be silently ignored by all runtimes. */ map runtime_hints = 9; + + /* Requested IVF partition count for building or rebuilding this index. + * + * The actual partition count may differ from this target and is read from the + * index files. + * + * A positive value takes precedence over `target_partition_size`. Absent or + * zero means no explicit count is available: a rebuild uses + * `target_partition_size` when positive, and the engine's own sizing + * otherwise. Absence does not imply the original build used automatic sizing, + * since a writer that predates this field records no count either way. + * + * Writers record only positive, representable counts, and omit a count they + * cannot represent rather than truncating it. + */ + optional uint32 target_num_partitions = 10; } // Hierarchical Navigable Small World (HNSW) parameters, used as an optional configuration for IVF indexes. diff --git a/rust/lance/src/index/vector/details.rs b/rust/lance/src/index/vector/details.rs index 836da47adfa..6cab24fd286 100644 --- a/rust/lance/src/index/vector/details.rs +++ b/rust/lance/src/index/vector/details.rs @@ -162,6 +162,7 @@ pub fn vector_index_details(params: &VectorIndexParams) -> prost_types::Any { hnsw_index_config, compression, runtime_hints, + target_num_partitions: None, }; prost_types::Any::from_msg(&details).unwrap() } @@ -564,6 +565,7 @@ fn convert_legacy_proto_to_details(proto: &pb::Index) -> Result