feat(format): specify carried-column storage and allow carrying a keyed column

docs/src/format/index/index.md

@@ -98,12 +98,15 @@ Index segments are created and updated through a transactional process:
 2. **Prepare the metadata**: Create an `IndexMetadata` message with:
    - `uuid`: The newly generated UUID
    - `name`: The index name (must match existing segments if adding to an existing index)
-   - `fields`: The columns the index depends on: the keyed column(s) it is searched on, followed
-     by any merely-carried columns named in `covering_fields`. `fields[0]` is always a keyed column.
-   - `covering_fields`: The trailing subset of `fields` whose values the index carries but is not
-     keyed on, letting a query that only projects those columns be answered without a fragment take.
-     Empty for an index that carries no extra columns. Declaring a column here does not by itself
-     make it servable -- see [Serving carried columns](#serving-carried-columns).
+   - `fields`: The columns the index depends on: the column(s) it is keyed on, plus any it
+     merely carries, as named in `covering_fields`. No id is repeated, and `fields[0]` is always
+     a column the index is keyed on.
+   - `covering_fields`: The subset of `fields` whose values the index carries, in the order it
+     emits them, letting a query that only projects those columns be answered without a fragment
+     take. A column is carried if and only if it is named here, including a column the index is
+     also keyed on. Empty for an index that carries no extra columns. Declaring a column here
+     does not by itself make it servable --
+     see [Serving carried columns](#serving-carried-columns).
    - `fragment_bitmap`: The set of fragment IDs covered by this segment
    - `index_details`: Index-specific configuration and parameters
    - `version`: The format version of this index type
@@ -141,18 +144,21 @@ fragments that would have been covered by that segment.
 carries. It does not establish that the segment's storage holds their values.
 
 **The segment's storage schema is authoritative.** Before answering a query from a
-carried column, an engine must confirm that column is present in the storage it opened,
-and fall back to a take against the base table when it is not. A segment whose
-declaration names a column its storage does not hold is a legal state, not corruption:
-a maintenance operation that cannot carry the payload through a rebuild is permitted to
-withdraw it and leave the declaration standing.
-
-!!! note "Current state"
-
-    No index builder writes carried values yet, so today every declaration is ahead of
-    its storage. Engines that read `covering_fields` must therefore treat it purely as a
-    declaration and serve every column from the base table until they have verified the
-    storage themselves. This is transitional; the rule above is not.
+carried column, an engine must confirm that column is present and bound to the declared
+logical field in the storage it opened, and fall back to a take against the base table
+when it cannot. A segment whose metadata identifies a column its storage does not hold is
+a legal state, not corruption: a maintenance operation that cannot carry the payload
+through a rebuild is permitted to withdraw it and leave the `covering_fields` listed in
+the metadata.
+
+!!! note "Capability varies by segment"
+
+    Whether a segment's storage holds a declared column depends on the index type, on the
+    writer that produced the segment, and on what later maintenance did to it, so one
+    logical index may hold values for some of its segments and not others. An engine
+    therefore verifies each selected segment rather than inferring capability from the
+    index type, the writer version, or the `covering_fields` metadata alone, and serves
+    from the base table every column it cannot verify.
 
 ## Loading an index
 
@@ -172,12 +178,15 @@ When loading an index:
 The `IndexMetadata` message contains important information about the index segment:
 
 - `uuid`: the unique identifier of the index segment.
-- `fields`: the columns the index depends on: the keyed column(s) the index is searched on, followed
-  by any columns it merely carries, as named in `covering_fields`. `fields[0]` is always a keyed column.
-- `covering_fields`: the trailing subset of `fields` whose values the index carries alongside its own
-  data but is not keyed on. Empty for an index that carries no extra columns. This declaration is
-  not authoritative for what the segment can serve -- see
-  [Serving carried columns](#serving-carried-columns).
+- `fields`: the columns the index depends on: the column(s) the index is keyed on, plus any it
+  merely carries, as named in `covering_fields`. No id is repeated, and `fields[0]` is always a
+  column the index is keyed on.
+- `covering_fields`: the subset of `fields` whose values the index carries alongside its own data,
+  in the order it emits them. A column is carried if and only if it is named here, including a
+  column the index is also keyed on. Empty for an index that carries no extra columns. Every id in
+  `covering_fields` names a top-level field. Covering a struct column carries the whole struct,
+  its children included, as one column. This metadata is not authoritative for what the segment
+  can serve -- see [Serving carried columns](#serving-carried-columns).
 - `fragment_bitmap`: the set of fragment IDs covered by this index segment.
 - `index_details`: a protobuf `Any` message that contains index-specific details, such as index type,
   parameters, and storage format. This allows different index types to store their own metadata.

docs/src/format/index/vector/index.md

@@ -161,6 +161,19 @@ the Arrow schema of the Lance file varies depending on the quantization method u
 !!! note
     All partitions are stored in the same file, and partitions must be written in order.
 
+Every quantization format below lists only its internal columns. When a V3 IVF
+writer materializes carried values, it appends one trailing column per carried
+field after them, named and typed exactly as in the dataset schema. This physical
+payload may be a subset of the manifest's `covering_fields` declaration (see
+[Index Metadata](../index.md)). A reader returns only columns whose physical
+schema and dataset field ids it verifies across every selected segment; all other
+projected columns come from a base-table take.
+
+A reader discovers carried columns by exclusion, not by position: any column in the
+auxiliary file's schema that is not one of the quantizer's internal columns is a
+carried column. Writers append them in trailing order, but a reader must not depend
+on that ordering to identify them.
+
 ##### FLAT
 
 No quantization applied - stores original vectors in their full precision:
@@ -229,6 +242,17 @@ Contains RabitQ-specific metadata in JSON format (only present for RQ quantizati
 This includes the rotation matrix position, number of bits, and packing information.
 See the RQ metadata specification in the "storage_metadata" section below.
 
+##### "covering_field_ids"
+
+The dataset field ids of the storage file's physical carried columns,
+comma separated in physical schema order (only present when the storage carries
+values). Arrow fields carry no Lance field id, so names and types alone cannot
+prove which logical column a payload came from. Readers use these ids to bind
+physical values to the segment's `covering_fields` declaration, and treat missing,
+malformed, ambiguous, or mismatched metadata as no servable carried capability.
+A merge must not combine shards whose carried columns disagree on these ids, even
+when those columns match by name and type.
+
 ##### "storage_metadata"
 
 Contains quantizer-specific metadata as a list of JSON strings.

protos/ann.proto

@@ -25,6 +25,21 @@ enum VectorApproxMode {
   Accurate = 2;
 }
 
+/* The covering ("included") index columns a query needs materialized, as dataset
+ * field ids.
+ *
+ * Exists as a message rather than a bare `repeated int32` because proto3 has no
+ * presence tracking for repeated fields, and the empty list is a distinct, meaningful
+ * state here:
+ *
+ * * absent: no narrowing computed; materialize every covering column declared.
+ * * present and empty: materialize nothing, though the index does declare covering.
+ * * present and non-empty: materialize exactly these.
+ */
+message CoveringProjection {
+  repeated int32 field_ids = 1;
+}
+
 // Serialized vector query parameters.
 message VectorQueryProto {
   // Query vector as Arrow IPC bytes (supports Float16, Float32, Float64, UInt8, etc.)
@@ -46,6 +61,13 @@ message VectorQueryProto {
    * indexes, such as IVF_RQ. Other index types ignore this setting.
    */
   VectorApproxMode approx_mode = 14;
+  /* Which covering columns the index must materialize for this query. Absent means
+   * "not computed" -- see CoveringProjection. Carried across the wire so a remote
+   * executor declares the same search output schema the planner did; without it the
+   * executor's node is wider than the plan it came from, and the surrounding nodes
+   * were built against the planner's narrower schema.
+   */
+  CoveringProjection covering_projection = 15;
 }
 
 /* Serializable form of ANNIvfSubIndexExec — the IVF sub-index search node.

protos/table.proto

@@ -334,10 +334,12 @@ message IndexMetadata {
    * data, so a query projecting only those columns can be answered from the
    * index without a take against the base table.
    *
-   * Must be a suffix of `fields`: the columns the index is keyed on come first,
-   * the columns it merely carries come last, and at least one keyed column
-   * always remains. Empty for an index that carries no extra columns, which is
-   * every index written before this field existed.
+   * Must be a subset of `fields`, in the order the index emits them. A column is
+   * carried if and only if it is named here, so a column the index is both keyed
+   * on and carries is listed once in `fields` and named here; no id repeats in
+   * `fields`, and `fields[0]` remains a column the index is keyed on. Empty for
+   * an index that carries no extra columns, which is every index written before
+   * this field existed.
    *
    * Carried columns are listed in `fields` as well. That is deliberate: every
    * consumer that reads `fields` as the index's dependency set -- staleness,

rust/lance/src/io/exec/ann_proto.rs

@@ -118,6 +118,9 @@ pub fn query_to_proto(query: &Query) -> Result<pb::VectorQueryProto> {
         dist_q_c: Some(query.dist_q_c),
         query_parallelism: Some(query.query_parallelism),
         approx_mode: approx_mode_to_proto(query.approx_mode) as i32,
+        // No planner narrows the covering projection yet, so this is always absent:
+        // "materialize every covering column declared". See `CoveringProjection`.
+        covering_projection: None,
     })
 }