feat(index): covering columns for IVF vector indexes

[vivek-bharathan]

Lets an IVF vector index store the values of chosen extra columns alongside its quantization
codes, so a search returns them directly. A query whose projection those columns satisfy skips
the base-table take.

Covers every IVF index type (PQ, SQ, RQ, FLAT and the HNSW variants) and the index lifecycle:
schema evolution, overlays, remap, compaction, optimize_indices, merge_insert and concurrent
commits. Builds on #8535 and #8856.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The revised test now exercises covered take elision without conflating that contract with HNSW’s default search beam. Production behavior is unchanged, and the covering correctness and lifecycle assessment remains supported.

Covered searches can still be materially slower than a plain index when results stream per partition—HNSW, query_parallelism > 1, or a pending deferred-remap/fragment-reuse state—because payload reads scale with probed or contributing partitions instead of final k. For latency-sensitive deployments, prefer a global-top-k IVF path and avoid or finish deferred remaps, or let planning fall back to the base-table take when survivor positions are unavailable.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

This revision makes covering_fields a serving guarantee even though the accepted format contract defines it only as a declaration. The safe dependency order is to land #8856 first, then validate each selected segment’s physical field-ID bindings and retain the base-table take for anything not proven present.

[lance-gatekeeper]

covering_fields cannot define the emitted schema until the selected segment’s physical capability is verified. The current format contract explicitly makes declaration-without-payload legal and requires a base-table fallback. Here the declaration widens ANNIvfSubIndexExec, so Scanner::take skips that fallback before storage is checked.

Reproducer

I added this focused regression in an isolated worktree:

#[tokio::test]
async fn test_declared_covering_without_storage_falls_back_to_take() {
    let uri = TempStrDir::default();
    let mut dataset = covering::write_vector_payload_dataset(&uri).await;
    covering::create_ivf_pq_index(&mut dataset, "vec").await;
    covering::declare_covering(&mut dataset, "vec", "payload").await;

    let query = generate_random_array(covering::DIMENSION as usize);
    let mut scan = dataset.scan();
    scan.nearest("vec", &query, 10).unwrap();
    scan.project(&["payload"]).unwrap();
    assert!(scan.try_into_batch().await.is_ok());
}

Run with:

CARGO_TARGET_DIR=/home/agent/tmp/pr8811-target-impl cargo test -p lance --lib dataset::tests::dataset_index::test_declared_covering_without_storage_falls_back_to_take --locked -- --exact --nocapture

It fails in 0.37s with PhysicalExpr Column references column 'payload' at index 2 ... input schema only has 2 columns: ["_distance", "_rowid"].

#8856, this PR’s stated prerequisite, defines the missing field-ID binding but is still open and absent from this head. Land/rebase it first, persist and validate those bindings for every selected segment, derive effective coverage only from columns every segment proves, and leave unverifiable fields to the base-table take.