feat(bindings): build and inspect covered indexes from Java and Python

docs/src/format/index/index.md

@@ -100,10 +100,11 @@ Index segments are created and updated through a transactional process:
    - `name`: The index name (must match existing segments if adding to an existing index)
    - `fields`: The columns the index depends on: the keyed column(s) it is searched on, followed
      by any merely-carried columns named in `covering_fields`. `fields[0]` is always a keyed column.
-   - `covering_fields`: The trailing subset of `fields` whose values the index carries but is not
-     keyed on, letting a query that only projects those columns be answered without a fragment take.
-     Empty for an index that carries no extra columns. Declaring a column here does not by itself
-     make it servable -- see [Serving carried columns](#serving-carried-columns).
+   - `covering_fields`: The trailing subset of `fields` whose values the index carries, letting a
+     query that only projects those columns be answered without a fragment take. Usually these are
+     columns the index is not keyed on, but a keyed column may also be carried. Empty for an index
+     that carries no extra columns. Declaring a column here does not by itself make it servable --
+     see [Serving carried columns](#serving-carried-columns).
    - `fragment_bitmap`: The set of fragment IDs covered by this segment
    - `index_details`: Index-specific configuration and parameters
    - `version`: The format version of this index type
@@ -141,18 +142,36 @@ fragments that would have been covered by that segment.
 carries. It does not establish that the segment's storage holds their values.
 
 **The segment's storage schema is authoritative.** Before answering a query from a
-carried column, an engine must confirm that column is present in the storage it opened,
-and fall back to a take against the base table when it is not. A segment whose
-declaration names a column its storage does not hold is a legal state, not corruption:
-a maintenance operation that cannot carry the payload through a rebuild is permitted to
-withdraw it and leave the declaration standing.
-
-!!! note "Current state"
-
-    No index builder writes carried values yet, so today every declaration is ahead of
-    its storage. Engines that read `covering_fields` must therefore treat it purely as a
-    declaration and serve every column from the base table until they have verified the
-    storage themselves. This is transitional; the rule above is not.
+carried column, an engine must confirm that column is present and bound to the declared
+logical field in the storage it opened, and fall back to a take against the base table
+when it cannot. A segment whose declaration names a column its storage does not hold is
+a legal state, not corruption: a maintenance operation that cannot carry the payload
+through a rebuild is permitted to withdraw it and leave the declaration standing.
+
+!!! note "Capability varies by segment"
+
+    Whether a segment's storage holds a declared column depends on the index type, on the
+    writer that produced the segment, and on what later maintenance did to it, so one
+    logical index may hold values for some of its segments and not others. An engine
+    therefore verifies each selected segment rather than inferring capability from the
+    index type, the writer version, or the declaration alone, and serves from the base
+    table every column it cannot verify.
+
+!!! note "A keyed column may also be carried"
+
+    `covering_fields` usually names columns the index is *not* keyed on, but an index is
+    permitted to carry a column it is also keyed on -- for instance a vector index that
+    keeps full-precision vectors so a refine pass can re-rank without a base-table take.
+    The id then appears twice in `fields`, once as `fields[0]` and again as the trailing
+    carried entry, and once in `covering_fields`. This is the only case in which an id
+    repeats in `fields`.
+
+    A reader must therefore take the carried set from `covering_fields` directly, and
+    never derive it by subtracting the keyed prefix from `fields`: that set difference
+    silently drops a column that is both. The trailing-subset rule is stated over
+    `covering_fields` and is unaffected: `fields[0]` remains the column the index is
+    searched on, and an engine serves the repeated id from storage like any other
+    carried column.
 
 ## Loading an index
 
@@ -175,9 +194,9 @@ The `IndexMetadata` message contains important information about the index segme
 - `fields`: the columns the index depends on: the keyed column(s) the index is searched on, followed
   by any columns it merely carries, as named in `covering_fields`. `fields[0]` is always a keyed column.
 - `covering_fields`: the trailing subset of `fields` whose values the index carries alongside its own
-  data but is not keyed on. Empty for an index that carries no extra columns. This declaration is
-  not authoritative for what the segment can serve -- see
-  [Serving carried columns](#serving-carried-columns).
+  data -- usually columns it is not keyed on, though a keyed column may also be carried. Empty for an
+  index that carries no extra columns. This declaration is not authoritative for what the segment can
+  serve -- see [Serving carried columns](#serving-carried-columns).
 - `fragment_bitmap`: the set of fragment IDs covered by this index segment.
 - `index_details`: a protobuf `Any` message that contains index-specific details, such as index type,
   parameters, and storage format. This allows different index types to store their own metadata.

docs/src/format/index/vector/index.md

@@ -161,6 +161,19 @@ the Arrow schema of the Lance file varies depending on the quantization method u
 !!! note
     All partitions are stored in the same file, and partitions must be written in order.
 
+Every quantization format below lists only its internal columns. When a V3 IVF
+writer materializes carried values, it appends one trailing column per carried
+field after them, named and typed exactly as in the dataset schema. This physical
+payload may be a subset of the manifest's `covering_fields` declaration (see
+[Index Metadata](../index.md)). A reader returns only columns whose physical
+schema and source field ids it verifies across every selected segment; all other
+projected columns come from a base-table take.
+
+A reader discovers carried columns by exclusion, not by position: any column in the
+auxiliary file's schema that is not one of the quantizer's internal columns is a
+carried column. Writers append them in trailing order, but a reader must not depend
+on that ordering to identify them.
+
 ##### FLAT
 
 No quantization applied - stores original vectors in their full precision:
@@ -229,6 +242,17 @@ Contains RabitQ-specific metadata in JSON format (only present for RQ quantizati
 This includes the rotation matrix position, number of bits, and packing information.
 See the RQ metadata specification in the "storage_metadata" section below.
 
+##### "covering_field_ids"
+
+The *source dataset* field ids of the storage file's physical carried columns,
+comma separated in physical schema order (only present when the storage carries
+values). Arrow fields carry no Lance field id, so names and types alone cannot
+prove which logical column a payload came from. Readers use these ids to bind
+physical values to the segment's `covering_fields` declaration, and treat missing,
+malformed, ambiguous, or mismatched metadata as no servable carried capability.
+Distributed merges use the same identity to reject shards whose columns match by
+name and type but come from different fields.
+
 ##### "storage_metadata"
 
 Contains quantizer-specific metadata as a list of JSON strings.

docs/src/guide/performance.md

@@ -529,3 +529,17 @@ Set `LANCE_DISABLE_AMX=1` to take the AMX paths out of service without rebuildin
 A/B measurement, or to get the previous behaviour back. Because it also moves partition
 assignment back to the approximate path, an index built with it set is not equivalent to one
 built without it; compare recall, not just build time.
+
+#### Covering Columns
+
+A vector index can carry the values of extra columns beside its vectors, so a query whose
+projection they satisfy is answered from the index without a take against the base table.
+That trade only pays off where the search settles into a single global top-k heap, because
+the covering read is then bounded by the query's survivors. Every HNSW index, and any query
+with `query_parallelism` above one, emits results per partition instead, so covering reads
+scale with the number of partitions probed rather than with `k`, and are issued serially.
+
+On those shapes covering is *slower* than the base-table take it exists to avoid — roughly
+2.8x a plain index's latency warm at nprobe 32, and 1.18x cold. Prefer IVF_PQ or IVF_FLAT
+when declaring covering columns; index creation logs a warning when covering is combined
+with HNSW.

java/lance-jni/src/utils.rs

@@ -24,7 +24,7 @@ use lance_linalg::distance::DistanceType;
 use crate::error::{Error, Result};
 use crate::ffi::JNIEnvExt;
 
-use crate::traits::FromJObjectWithEnv;
+use crate::traits::{FromJObjectWithEnv, import_vec_from_method};
 use lance_index::vector::{ApproxMode, Query};
 use std::collections::HashMap;
 use std::str::FromStr;
@@ -261,6 +261,11 @@ pub fn build_compaction_options(
 }
 
 // Convert from Java Optional<Query> to Rust Option<Query>
+//
+// This builds a `Query` directly rather than through `Scanner::nearest`, and is used
+// only by `JniTestHelper.parseQuery` to check the Java-side field marshalling. Java's
+// real search path is `blocking_scanner.rs`, which configures a `Scanner`, so it picks
+// up every plan-derived setting -- including the covering projection below.
 pub fn get_query(env: &mut JNIEnv, query_obj: JObject) -> Result<Option<Query>> {
     let query = env.get_optional(&query_obj, |env, java_obj| {
         let column = env.get_string_from_method(&java_obj, "getColumn")?;
@@ -305,6 +310,11 @@ pub fn get_query(env: &mut JNIEnv, query_obj: JObject) -> Result<Option<Query>>
             dist_q_c: 0.0,
             query_parallelism,
             approx_mode,
+            // Not a user-settable search parameter: the covering projection is derived
+            // per plan from what the scan reads, so it stays `None` (materialize
+            // whatever the index declares) at the binding boundary. Java's real search
+            // path resolves it in `Scanner`; see the note on this function.
+            covering_projection: None,
         })
     })?;
 
@@ -509,12 +519,22 @@ pub fn get_vector_index_params(
                 stages.push(StageParams::RQ(rq_params));
             }
 
+            // Covering ("included") columns: names of extra dataset columns stored inline in
+            // the index. Empty when absent. The core validates them when the index is built.
+            let covering_columns: Vec<String> = import_vec_from_method(
+                env,
+                &vector_index_params_obj,
+                "getCoveringColumns",
+                |env, elem| Ok(env.get_string(&JString::from(elem))?.into()),
+            )?;
+
             Ok(VectorIndexParams {
                 metric_type: distance_type,
                 stages,
                 version: IndexFileVersion::V3,
                 skip_transpose: false,
                 runtime_hints: Default::default(),
+                covering_columns,
             })
         },
     )?;

java/src/main/java/org/lance/index/Index.java

@@ -16,6 +16,7 @@
 import com.google.common.base.MoreObjects;
 
 import java.time.Instant;
+import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
@@ -55,8 +56,19 @@ private Index(
       Long sizeBytes,
       IndexType indexType) {
     this.uuid = uuid;
-    this.fields = fields;
-    this.coveringFields = coveringFields;
+    // Both lists are defensively copied and unmodifiable so a caller cannot mutate them
+    // after construction: equals/hashCode would silently drift from the index files that
+    // actually carry the payload, and `coveringFields` is documented as the trailing slice
+    // of `fields`, so mutating either alone breaks that relationship. Null-coalesced so an
+    // explicit null from the builder reads as "none" rather than blowing up at first use.
+    this.fields =
+        fields == null
+            ? Collections.emptyList()
+            : Collections.unmodifiableList(new ArrayList<>(fields));
+    this.coveringFields =
+        coveringFields == null
+            ? Collections.emptyList()
+            : Collections.unmodifiableList(new ArrayList<>(coveringFields));
     this.name = name;
     this.datasetVersion = datasetVersion;
     this.fragments = fragments;
@@ -89,7 +101,7 @@ public List<Integer> fields() {
    * <p>These ids also appear in {@link #fields()} — that is deliberate, so that every consumer
    * reading {@code fields()} as the index's dependency set also covers them with no change.
    *
-   * @return the covering field IDs
+   * @return the covering field IDs, as an unmodifiable list
    */
   public List<Integer> coveringFields() {
     return coveringFields;

java/src/main/java/org/lance/index/IndexDescription.java

@@ -69,7 +69,14 @@ public String getName() {
     return name;
   }
 
-  /** Field ids that this index is built on. */
+  /**
+   * Field ids that this index is built on -- the columns it can answer queries for.
+   *
+   * <p>This is the index's <em>keyed</em> prefix only. An index may additionally carry values for
+   * columns it is not keyed on; those are deliberately absent here, because the index cannot be
+   * searched on them. They stay reachable per segment via {@link Index#coveringFields()} on the
+   * entries of {@link #getMetadata()}.
+   */
   public List<Integer> getFieldIds() {
     return fieldIds;
   }

java/src/main/java/org/lance/index/vector/VectorIndexParams.java

@@ -17,6 +17,9 @@
 
 import com.google.common.base.MoreObjects;
 
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
 import java.util.Optional;
 
 /** Parameters for creating a vector index. */
@@ -27,6 +30,7 @@ public class VectorIndexParams {
   private final Optional<HnswBuildParams> hnswParams;
   private final Optional<SQBuildParams> sqParams;
   private final Optional<RQBuildParams> rqParams;
+  private final List<String> coveringColumns;
 
   private VectorIndexParams(Builder builder) {
     this.distanceType = builder.distanceType;
@@ -35,6 +39,10 @@ private VectorIndexParams(Builder builder) {
     this.hnswParams = builder.hnswParams;
     this.sqParams = builder.sqParams;
     this.rqParams = builder.rqParams;
+    this.coveringColumns =
+        builder.coveringColumns == null
+            ? Collections.emptyList()
+            : Collections.unmodifiableList(new ArrayList<>(builder.coveringColumns));
     validate();
   }
 
@@ -179,6 +187,7 @@ public static class Builder {
     private Optional<HnswBuildParams> hnswParams = Optional.empty();
     private Optional<SQBuildParams> sqParams = Optional.empty();
     private Optional<RQBuildParams> rqParams = Optional.empty();
+    private List<String> coveringColumns = Collections.emptyList();
 
     /**
      * Create a new builder to create a vector index.
@@ -235,6 +244,20 @@ public Builder setRqParams(RQBuildParams rqParams) {
       return this;
     }
 
+    /**
+     * Set the columns to cover ("include") in the index. Their values are stored inline in the
+     * index so a query projecting only covered columns is answered from the index without a take
+     * from the base table. Each must name a top-level, non-key column of the dataset; the columns
+     * are validated by the core when the index is built. Empty by default (no covering).
+     *
+     * @param coveringColumns the covering column names
+     * @return Builder
+     */
+    public Builder setCoveringColumns(List<String> coveringColumns) {
+      this.coveringColumns = coveringColumns;
+      return this;
+    }
+
     public VectorIndexParams build() {
       return new VectorIndexParams(this);
     }
@@ -268,6 +291,16 @@ public Optional<RQBuildParams> getRqParams() {
     return rqParams;
   }
 
+  /**
+   * Get the covering ("included") columns whose values are stored inline in the index. Empty when
+   * the index has no covering columns.
+   *
+   * @return the covering column names, as an unmodifiable list
+   */
+  public List<String> getCoveringColumns() {
+    return coveringColumns;
+  }
+
   @Override
   public String toString() {
     return MoreObjects.toStringHelper(this)
@@ -277,6 +310,7 @@ public String toString() {
         .add("hnswParams", hnswParams.orElse(null))
         .add("sqParams", sqParams.orElse(null))
         .add("rqParams", rqParams.orElse(null))
+        .add("coveringColumns", coveringColumns)
         .toString();
   }
 }