From db9f8489b7cf4fb99f58d24bd0922d5c2180d69d Mon Sep 17 00:00:00 2001 From: Vivek Date: Fri, 28 Aug 2026 11:30:38 -0700 Subject: [PATCH] feat(format): specify carried-column storage and allow carrying a keyed column #8535 declared `IndexMetadata.covering_fields` but left the physical side unspecified; this specifies it -- carried values are extra columns in `auxiliary.idx`, discovered by exclusion against the quantizer's internal columns and bound to their dataset fields by a new `covering_field_ids` metadata key, with no `index_version` bump. It also permits an index to carry a column it is also keyed on. Such a column is listed once in `fields` and named in `covering_fields`, so `covering_fields` is a subset rather than a suffix, and readers must take the carried set from `covering_fields` rather than subtract the keyed prefix. Adds `VectorQueryProto.covering_projection` (field 15) to reserve the tag, with the one initializer the new field forces on `query_to_proto`; no writer emits carried values yet, so the implementation follows separately. --- docs/src/format/index/index.md | 57 ++++++++++++++++----------- docs/src/format/index/vector/index.md | 24 +++++++++++ protos/ann.proto | 22 +++++++++++ protos/table.proto | 10 +++-- rust/lance/src/io/exec/ann_proto.rs | 3 ++ 5 files changed, 88 insertions(+), 28 deletions(-) diff --git a/docs/src/format/index/index.md b/docs/src/format/index/index.md index 01970133208..ba8026334ec 100644 --- a/docs/src/format/index/index.md +++ b/docs/src/format/index/index.md @@ -98,12 +98,15 @@ Index segments are created and updated through a transactional process: 2. **Prepare the metadata**: Create an `IndexMetadata` message with: - `uuid`: The newly generated UUID - `name`: The index name (must match existing segments if adding to an existing index) - - `fields`: The columns the index depends on: the keyed column(s) it is searched on, followed - by any merely-carried columns named in `covering_fields`. `fields[0]` is always a keyed column. - - `covering_fields`: The trailing subset of `fields` whose values the index carries but is not - keyed on, letting a query that only projects those columns be answered without a fragment take. - Empty for an index that carries no extra columns. Declaring a column here does not by itself - make it servable -- see [Serving carried columns](#serving-carried-columns). + - `fields`: The columns the index depends on: the column(s) it is keyed on, plus any it + merely carries, as named in `covering_fields`. No id is repeated, and `fields[0]` is always + a column the index is keyed on. + - `covering_fields`: The subset of `fields` whose values the index carries, in the order it + emits them, letting a query that only projects those columns be answered without a fragment + take. A column is carried if and only if it is named here, including a column the index is + also keyed on. Empty for an index that carries no extra columns. Declaring a column here + does not by itself make it servable -- + see [Serving carried columns](#serving-carried-columns). - `fragment_bitmap`: The set of fragment IDs covered by this segment - `index_details`: Index-specific configuration and parameters - `version`: The format version of this index type @@ -141,18 +144,21 @@ fragments that would have been covered by that segment. carries. It does not establish that the segment's storage holds their values. **The segment's storage schema is authoritative.** Before answering a query from a -carried column, an engine must confirm that column is present in the storage it opened, -and fall back to a take against the base table when it is not. A segment whose -declaration names a column its storage does not hold is a legal state, not corruption: -a maintenance operation that cannot carry the payload through a rebuild is permitted to -withdraw it and leave the declaration standing. - -!!! note "Current state" - - No index builder writes carried values yet, so today every declaration is ahead of - its storage. Engines that read `covering_fields` must therefore treat it purely as a - declaration and serve every column from the base table until they have verified the - storage themselves. This is transitional; the rule above is not. +carried column, an engine must confirm that column is present and bound to the declared +logical field in the storage it opened, and fall back to a take against the base table +when it cannot. A segment whose metadata identifies a column its storage does not hold is +a legal state, not corruption: a maintenance operation that cannot carry the payload +through a rebuild is permitted to withdraw it and leave the `covering_fields` listed in +the metadata. + +!!! note "Capability varies by segment" + + Whether a segment's storage holds a declared column depends on the index type, on the + writer that produced the segment, and on what later maintenance did to it, so one + logical index may hold values for some of its segments and not others. An engine + therefore verifies each selected segment rather than inferring capability from the + index type, the writer version, or the `covering_fields` metadata alone, and serves + from the base table every column it cannot verify. ## Loading an index @@ -172,12 +178,15 @@ When loading an index: The `IndexMetadata` message contains important information about the index segment: - `uuid`: the unique identifier of the index segment. -- `fields`: the columns the index depends on: the keyed column(s) the index is searched on, followed - by any columns it merely carries, as named in `covering_fields`. `fields[0]` is always a keyed column. -- `covering_fields`: the trailing subset of `fields` whose values the index carries alongside its own - data but is not keyed on. Empty for an index that carries no extra columns. This declaration is - not authoritative for what the segment can serve -- see - [Serving carried columns](#serving-carried-columns). +- `fields`: the columns the index depends on: the column(s) the index is keyed on, plus any it + merely carries, as named in `covering_fields`. No id is repeated, and `fields[0]` is always a + column the index is keyed on. +- `covering_fields`: the subset of `fields` whose values the index carries alongside its own data, + in the order it emits them. A column is carried if and only if it is named here, including a + column the index is also keyed on. Empty for an index that carries no extra columns. Every id in + `covering_fields` names a top-level field. Covering a struct column carries the whole struct, + its children included, as one column. This metadata is not authoritative for what the segment + can serve -- see [Serving carried columns](#serving-carried-columns). - `fragment_bitmap`: the set of fragment IDs covered by this index segment. - `index_details`: a protobuf `Any` message that contains index-specific details, such as index type, parameters, and storage format. This allows different index types to store their own metadata. diff --git a/docs/src/format/index/vector/index.md b/docs/src/format/index/vector/index.md index 91d73a23a16..c7f385accf4 100644 --- a/docs/src/format/index/vector/index.md +++ b/docs/src/format/index/vector/index.md @@ -161,6 +161,19 @@ the Arrow schema of the Lance file varies depending on the quantization method u !!! note All partitions are stored in the same file, and partitions must be written in order. +Every quantization format below lists only its internal columns. When a V3 IVF +writer materializes carried values, it appends one trailing column per carried +field after them, named and typed exactly as in the dataset schema. This physical +payload may be a subset of the manifest's `covering_fields` declaration (see +[Index Metadata](../index.md)). A reader returns only columns whose physical +schema and dataset field ids it verifies across every selected segment; all other +projected columns come from a base-table take. + +A reader discovers carried columns by exclusion, not by position: any column in the +auxiliary file's schema that is not one of the quantizer's internal columns is a +carried column. Writers append them in trailing order, but a reader must not depend +on that ordering to identify them. + ##### FLAT No quantization applied - stores original vectors in their full precision: @@ -229,6 +242,17 @@ Contains RabitQ-specific metadata in JSON format (only present for RQ quantizati This includes the rotation matrix position, number of bits, and packing information. See the RQ metadata specification in the "storage_metadata" section below. +##### "covering_field_ids" + +The dataset field ids of the storage file's physical carried columns, +comma separated in physical schema order (only present when the storage carries +values). Arrow fields carry no Lance field id, so names and types alone cannot +prove which logical column a payload came from. Readers use these ids to bind +physical values to the segment's `covering_fields` declaration, and treat missing, +malformed, ambiguous, or mismatched metadata as no servable carried capability. +A merge must not combine shards whose carried columns disagree on these ids, even +when those columns match by name and type. + ##### "storage_metadata" Contains quantizer-specific metadata as a list of JSON strings. diff --git a/protos/ann.proto b/protos/ann.proto index ff848c67fe9..c8f0449fcc0 100644 --- a/protos/ann.proto +++ b/protos/ann.proto @@ -25,6 +25,21 @@ enum VectorApproxMode { Accurate = 2; } +/* The covering ("included") index columns a query needs materialized, as dataset + * field ids. + * + * Exists as a message rather than a bare `repeated int32` because proto3 has no + * presence tracking for repeated fields, and the empty list is a distinct, meaningful + * state here: + * + * * absent: no narrowing computed; materialize every covering column declared. + * * present and empty: materialize nothing, though the index does declare covering. + * * present and non-empty: materialize exactly these. + */ +message CoveringProjection { + repeated int32 field_ids = 1; +} + // Serialized vector query parameters. message VectorQueryProto { // Query vector as Arrow IPC bytes (supports Float16, Float32, Float64, UInt8, etc.) @@ -46,6 +61,13 @@ message VectorQueryProto { * indexes, such as IVF_RQ. Other index types ignore this setting. */ VectorApproxMode approx_mode = 14; + /* Which covering columns the index must materialize for this query. Absent means + * "not computed" -- see CoveringProjection. Carried across the wire so a remote + * executor declares the same search output schema the planner did; without it the + * executor's node is wider than the plan it came from, and the surrounding nodes + * were built against the planner's narrower schema. + */ + CoveringProjection covering_projection = 15; } /* Serializable form of ANNIvfSubIndexExec — the IVF sub-index search node. diff --git a/protos/table.proto b/protos/table.proto index 2781b2383cb..f6b1c768726 100644 --- a/protos/table.proto +++ b/protos/table.proto @@ -334,10 +334,12 @@ message IndexMetadata { * data, so a query projecting only those columns can be answered from the * index without a take against the base table. * - * Must be a suffix of `fields`: the columns the index is keyed on come first, - * the columns it merely carries come last, and at least one keyed column - * always remains. Empty for an index that carries no extra columns, which is - * every index written before this field existed. + * Must be a subset of `fields`, in the order the index emits them. A column is + * carried if and only if it is named here, so a column the index is both keyed + * on and carries is listed once in `fields` and named here; no id repeats in + * `fields`, and `fields[0]` remains a column the index is keyed on. Empty for + * an index that carries no extra columns, which is every index written before + * this field existed. * * Carried columns are listed in `fields` as well. That is deliberate: every * consumer that reads `fields` as the index's dependency set -- staleness, diff --git a/rust/lance/src/io/exec/ann_proto.rs b/rust/lance/src/io/exec/ann_proto.rs index c57ad4ca1b7..7ecb14211e6 100644 --- a/rust/lance/src/io/exec/ann_proto.rs +++ b/rust/lance/src/io/exec/ann_proto.rs @@ -118,6 +118,9 @@ pub fn query_to_proto(query: &Query) -> Result { dist_q_c: Some(query.dist_q_c), query_parallelism: Some(query.query_parallelism), approx_mode: approx_mode_to_proto(query.approx_mode) as i32, + // No planner narrows the covering projection yet, so this is always absent: + // "materialize every covering column declared". See `CoveringProjection`. + covering_projection: None, }) }