feat(format): separate index keys from covering fields

[Ali2Arslan]

Summary

• define the new IndexMetadata contract so fields contains key fields only and covering_fields is an independent declaration that may overlap it
• define the index dependency set as the union of fields and covering_fields
• reserve FLAG_INDEPENDENT_COVERING_FIELDS (bit 11 / 2048) as a paired reader/writer fence and keep it unsupported until the implementation lands
• specify activation: setting the flag atomically normalizes every legacy-form entry of the index section in the same manifest (or requires that none declares covering fields), a retry normalizes against the index section it is retrying against, and derived manifests retain the bit
• update index lifecycle, overlay-staleness, and feature-flag specifications to use the independent declarations

Compatibility

The subset contract for covering fields (added in #8535, refined by the now-merged #8856) shipped in Lance 11.0.0, so the same protobuf bytes cannot be reinterpreted unconditionally. A manifest using the new contract must set both FLAG_COVERED_INDEX_METADATA and FLAG_INDEPENDENT_COVERING_FIELDS. A manifest with only FLAG_COVERED_INDEX_METADATA retains the stable legacy interpretation, where covering_fields is a subset of fields with no id repeated.

The new bit makes older releases reject independent declarations instead of treating covering fields as keys. No protobuf field number or wire type changes. Bit 11 avoids the reserved bit 9 and the bit 10 FLAG_FRAGMENT_REUSE_INDEX from #9136.

This PR intentionally reserves and specifies the contract without implementing readers or writers, following the format-change process. Bit 11 sits above FLAG_UNKNOWN (1 << 9) and supported_flags_when starts from FLAG_UNKNOWN - 1, so no build can open or write a manifest that sets it; FLAG_FRAGMENT_REUSE_INDEX is in the same reserved state on main today. Consequently the existing keyed_fields derivation, which subtracts the carried count from fields, can never observe an independent declaration and is left for the implementation PR. This PR layers on the carried-column storage contract from #8856 so physical storage and query behavior can follow the approved metadata contract.

Testing

• cargo fmt --all
• cargo test -p lance-table feature_flags --lib
• cargo clippy --all --tests --benches -- -D warnings
• RUSTDOCFLAGS='-D warnings' cargo doc -p lance-table --no-deps
• uv run --project docs python ci/check_proto_comments.py
• cd docs && uv run mkdocs build

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — vetoed by @wjones127

Approvals	none (0/3)
Latest commit approved by	none — one PMC member must approve the latest commit
Vetoes	@wjones127
Voting period	elapsed — ended Wed 2026-09-16 19:27 UTC (12:27 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[wjones127]

Need to validate my assumption, but if we can I'm fine with making a breaking change to the existing spec so we avoid having these two separate flags.

[wjones127]

suggestion: I don't think there's any existing covering indexes implements. Right @vivek-bharathan? If that's the case, then can we just change the meaning of 1 << 7 now rather than introducing more possible states?

[Ali2Arslan]

It's probably easier to reuse the existing flag, at the cost of not knowing if anyone was using the public Rust/Python transaction APIs?

[Ali2Arslan]

Update: I rolled that revision back for now while we work through the compatibility-safe representation. The PR is restored to 0888867 with the distinct bit-11 proposal.

[Ali2Arslan]

Ultimately it seems like we could make either approach work correct?

• Sometimes we will want to know just which fields are key fields.

  • For example, to test if a filter can be applied

• Sometimes we will want to know just which fields are covering fields.

  • For example, to test which columns we don't need to take

• Sometimes we will want to know all fields, covered or not.

  • For example, to test if a column change invalidates an index

The rationale for the current approach was that much of the commit transaction logic was the third category. By just extending fields we were able to avoid changes to that code.

However, I think it ultimately doesn't matter, as long as we can distinguish between the three cases (and I think we can in both cases).

I will admit that this approach is slightly more intuitive for those new to the spec. I'll leave it up to you @Ali2Arslan and @vivek-bharathan which approach we want. As long as both of you agree we can definitely merge this PR.

Or am I missing something and is there something we could not identify in the previous implementation?

The main issues with the current approach, from my perspective, are:

• It allows for invalid states to be represented, and passes the responsibility to the code to enforce them. It allows you to interleave ordering b/t key and covering fields (assuming we will have multi-field keys in the future).
• It does not specify the behavior when we want to include a field as both a key and covering field. This might be important in the future if the key encoding cannot preserve the bits of the value losslessly, for stuff like floats, collations, etc.

I don't necessairly think that we can't work around these in the implementation, but it definitely increases the complexity imho.

[vivek-bharathan]

• It allows for invalid states to be represented, and passes the responsibility to the code to enforce them. It allows you to interleave ordering b/t key and covering fields (assuming we will have multi-field keys in the future).

Yes - multi-field keys would require a reworking of the contract. Maybe enforce that they are the leading prefix

• It does not specify the behavior when we want to include a field as both a key and covering field. This might be important in the future if the key encoding cannot preserve the bits of the value losslessly, for stuff like floats, collations, etc.

This is being addressed in #8856 - particularly to support performing refine without an additional take.

But that said - I agree that this is the cleaner contract. The primary reason we went with the other approach was to simplify state tracking and the commit transaction logic. Either way we pay the cost elsewhere. I'm fine with taking this approach

[westonpace]

I've merged #8856 which added (a different) concept of "this is how you specify a key column is also covered" since it already had 3 votes. Let's go ahead and rebase this PR on top of the new wording (since it seems we have alignment on this approach over the other approach). Then we can merge this PR too.

@Ali2Arslan let me know if you need any help. Thanks!

[lance-gatekeeper]

✅ Gate recommendation: approve.

The activation contract now closes the remaining compatibility gap: transition is atomic, retries normalize the current index section, and bit 11 remains sticky until any reverse conversion is atomic. The distinct bit preserves released bit-7 manifests and remains unsupported until a follow-up implementation, while the rebase retains #8856ʼs carried-column storage contract.

[Ali2Arslan]

I've merged #8856 which added (a different) concept of "this is how you specify a key column is also covered" since it already had 3 votes. Let's go ahead and rebase this PR on top of the new wording (since it seems we have alignment on this approach over the other approach). Then we can merge this PR too.

@Ali2Arslan let me know if you need any help. Thanks!

Thanks @westonpace , I updated the wordage if you guys want to take another look.