feat(format): specify carried-column storage and allow carrying a keyed column

[vivek-bharathan]

#8535 declared IndexMetadata.covering_fields but left the physical side unspecified; this specifies it -- carried values are extra columns in auxiliary.idx, discovered by exclusion against the quantizer's internal columns and bound to their source fields by a new covering_field_ids metadata key, with no index_version bump.

It also permits an index to carry a column it is also keyed on, the only case where an id repeats in fields, so readers must take the carried set from covering_fields rather than subtract the keyed prefix.

Adds VectorQueryProto.covering_projection (field 15) to reserve the tag; no writer emits carried values yet, so the implementation follows separately.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — vetoed by @westonpace

Approvals (this commit)	@wjones127 (1/3) — stale, re-approve needed: @jackye1995
Vetoes	@westonpace
Voting period	elapsed — ended Wed 2026-09-02 20:59 UTC (13:59 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[lance-gatekeeper]

✅ Gate recommendation: approve.

The new optional per-segment covering projection is represented explicitly while preserving absent-state behavior, and the distributed query serializer remains compatible with the generated schema.

[jackye1995]

looks good, please raise a corresponding voting thread

[vivek-bharathan]

looks good, please raise a corresponding voting thread

Happy to do that - but it looks like the process was recently changed

The pull request is the proposal. Open a PR with the specification change, and the PMC votes on it there — there is no separate design document or discussion thread to write first, and the requirement is enforced structurally in CI rather than by convention

[wjones127]

-1 until the blocking questions is decided. Have you considered using field ids instead of string names / paths? If so, what made you reject that?

[westonpace]

This looks good but I have a few questions. Thanks for working on this!

[westonpace]

Is this true? Wouldn't the default be to materialize nothing?

[vivek-bharathan]

Absent means nobody computed a projection -- either an encoder older than this
field, or a planner that skipped the narrowing step. Intended to reproduce the behavior from before this field existed, when a covered index always materialized everything it declared

[westonpace]

I understand the need for the two sides to agree on the schema before-hand but can we do this?

If the planner has to specify the covering projection, and the available fields are determined from index storage, does this mean the planner will have to load the index into memory? Isn't that something we probably want to avoid?

[vivek-bharathan]

That is correct. Today it reads both index files' footers and schema metadata, plus their IVF models. But it probably makes sense to build a version that excludes the IVF models. I can do that as a follow up - I do not think that changes this PR though

[vivek-bharathan]

@westonpace @wjones127 - I believe all concerns are addressed now. ptal

[wjones127]

This looks good now. Also note the change proposed in #9159 - I think that should be released alongside this.

[westonpace]

My concerns have been addressed, thanks!