feat(format): separate index keys from covering fields

docs/src/format/index/index.md

@@ -98,14 +98,11 @@ Index segments are created and updated through a transactional process:
 2. **Prepare the metadata**: Create an `IndexMetadata` message with:
    - `uuid`: The newly generated UUID
    - `name`: The index name (must match existing segments if adding to an existing index)
-   - `fields`: The columns the index depends on: the column(s) it is keyed on, plus any it
-     merely carries, as named in `covering_fields`. No id is repeated, and `fields[0]` is always
-     a column the index is keyed on.
-   - `covering_fields`: The subset of `fields` whose values the index carries, in the order it
-     emits them, letting a query that only projects those columns be answered without a fragment
-     take. A column is carried if and only if it is named here, including a column the index is
-     also keyed on. Empty for an index that carries no extra columns. Declaring a column here
-     does not by itself make it servable --
+   - `fields`: The columns the index is keyed on, in key order.
+   - `covering_fields`: An independent declaration of the columns whose values the index carries,
+     letting a query that only projects those columns be answered without a fragment take. A field
+     may appear in both lists when the index carries a value it is also keyed on. Empty for an index
+     that carries no extra columns. Declaring a column here does not by itself make it servable --
      see [Serving carried columns](#serving-carried-columns).
    - `fragment_bitmap`: The set of fragment IDs covered by this segment
    - `index_details`: Index-specific configuration and parameters
@@ -118,9 +115,9 @@ Index segments are created and updated through a transactional process:
 
 When updating a column in place (without deleting the row), the engine must remove the
 affected fragment IDs from the `fragment_bitmap` field of any index segment whose `fields`
-include that column — whether the index is keyed on it or merely carries it. This marks
-those fragments as needing re-indexing without invalidating the entire segment and prevents
-invalid data from being read from the index.
+or `covering_fields` include that column. This marks those fragments as needing re-indexing
+without invalidating the entire segment and prevents invalid data from being read from the
+index.
 
 ## Index Compatibility
 
@@ -138,6 +135,44 @@ Before using an index segment, engines must verify they support it:
 When an engine cannot use an index segment, it should fall back to scanning the
 fragments that would have been covered by that segment.
 
+### Covering-field contracts
+
+A manifest that uses independent key and covering declarations must set
+`FLAG_COVERED_INDEX_METADATA` and `FLAG_INDEPENDENT_COVERING_FIELDS` in both its reader
+and writer feature flags. In this contract, `fields` contains only the key fields and
+`covering_fields` separately contains the carried fields. The lists have no positional
+relationship and may overlap. The index dependency set is their union.
+
+For backwards compatibility, a manifest with `FLAG_COVERED_INDEX_METADATA` but without
+`FLAG_INDEPENDENT_COVERING_FIELDS` uses the legacy contract: `covering_fields` is a
+subset of `fields` in emission order, no id is repeated in `fields`, and a column the
+index is both keyed on and carries is listed once in `fields` and named in
+`covering_fields`. Readers must select the contract from the manifest flags before
+interpreting any `IndexMetadata`.
+
+### Activating independent declarations
+
+Setting `FLAG_INDEPENDENT_COVERING_FIELDS` reinterprets every `IndexMetadata` in the
+manifest's index section, so a writer must not set it while any entry remains in legacy
+form. A legacy `fields = [vector, payload]` with `covering_fields = [payload]` reads
+under the new contract as keyed on both columns, and a query on `payload` would then be
+planned against an index that is really keyed on `vector`.
+
+Activation is therefore a single atomic manifest change. A writer that sets the flag must
+normalize every entry of the index section in that same manifest, dropping from `fields`
+the ids that appear only in `covering_fields` and leaving the key fields in key order. A
+writer that cannot normalize an entry must not set the flag; it may instead set the flag
+only on a manifest in which no entry declares covering fields at all. No committed
+manifest may set the flag while holding a legacy-form entry.
+
+A commit that loses a conflict and retries must normalize the index section of the
+manifest it is retrying against rather than the one it first read, so an entry another
+writer committed in between is never carried forward unnormalized.
+
+Derived manifests retain the flag. Every manifest built from one that sets it keeps the
+bit set, unless a writer converts every entry back to the legacy contract in one atomic
+change of its own.
+
 ### Serving carried columns
 
 `IndexMetadata.covering_fields` records the columns an index segment *declares* it
@@ -178,15 +213,14 @@ When loading an index:
 The `IndexMetadata` message contains important information about the index segment:
 
 - `uuid`: the unique identifier of the index segment.
-- `fields`: the columns the index depends on: the column(s) the index is keyed on, plus any it
-  merely carries, as named in `covering_fields`. No id is repeated, and `fields[0]` is always a
-  column the index is keyed on.
-- `covering_fields`: the subset of `fields` whose values the index carries alongside its own data,
-  in the order it emits them. A column is carried if and only if it is named here, including a
-  column the index is also keyed on. Empty for an index that carries no extra columns. Every id in
-  `covering_fields` names a top-level field. Covering a struct column carries the whole struct,
-  its children included, as one column. This metadata is not authoritative for what the segment
-  can serve -- see [Serving carried columns](#serving-carried-columns).
+- `fields`: the columns the index is keyed on, in key order.
+- `covering_fields`: the independently declared columns whose values the index carries alongside
+  its own data. A field may also occur in `fields`. Empty for an index that carries no extra
+  columns. Every id in `covering_fields` names a top-level field. Covering a struct column
+  carries the whole struct, its children included, as one column. This declaration is not
+  authoritative for what the segment can serve -- see
+  [Serving carried columns](#serving-carried-columns) and the legacy interpretation under
+  [Covering-field contracts](#covering-field-contracts).
 - `fragment_bitmap`: the set of fragment IDs covered by this index segment.
 - `index_details`: a protobuf `Any` message that contains index-specific details, such as index type,
   parameters, and storage format. This allows different index types to store their own metadata.
@@ -227,20 +261,20 @@ There are four situations to consider:
 3. **A fragment has had one of the index's columns updated in place.** This cannot be detected
    just by examining metadata. To prevent reading invalid data, the engine should filter out any
    row addresses that are not in the index's current `fragment_bitmap`.
-   The column need not be one the index is keyed on: every column in `fields` counts, including
-   the merely-carried ones named in `covering_fields`. A carried column can be updated while the
-   keyed column is untouched, and a segment left covering that fragment would answer from an
-   obsolete carried value.
+   The column need not be one the index is keyed on: every column in the union of `fields` and
+   `covering_fields` counts. A carried column can be updated while the keyed column is untouched,
+   and a segment left covering that fragment would answer from an obsolete carried value.
 4. **A fragment has an updated value in an [overlay file](../table/data_overlay_file.md).**
    This can be detected by checking if any of the fragments in the index's `fragment_bitmap`
    have overlay files. For each overlay whose `committed_version` is greater than the index
    segment's `dataset_version`, the overlay carries updated values not reflected in the index,
    so its covered rows must be excluded from index results. Excluded rows are re-evaluated
    against their current (overlaid) values on the flat path — dropping them without
    re-evaluation would silently lose rows that match under the new value. Exclusion is
-   field-aware: only overlays covering a column in the index's `fields` matter — keyed or
-   merely carried. Restricting this to the keyed column would leave a fragment covered after
-   an overlay updated a carried one, and the index would then serve a stale carried value.
+   field-aware: only overlays covering a column in the union of the index's `fields` and
+   `covering_fields` matter. Restricting this to the keyed columns would leave a fragment
+   covered after an overlay updated a carried one, and the index would then serve a stale
+   carried value.
    You may exclude just the affected rows or the whole fragment; the latter is simpler and
    safer but re-evaluates more rows than necessary.
    See [Data Overlay Files](../table/data_overlay_file.md#index-integration)
@@ -277,8 +311,8 @@ logical identifier that remains constant even when rows are moved during compact
 **Benefits:**
 
 - No remapping needed after compaction
-- Updates only invalidate the index if data in one of its `fields` changes — the keyed
-  column(s) or any column named in `covering_fields`
+- Updates only invalidate the index if data in the union of its `fields` and
+  `covering_fields` changes
 
 **Tradeoffs:**
 

docs/src/format/table/data_overlay_file.md

@@ -147,12 +147,10 @@ restricted to field `F`, of every overlay whose `committed_version >
 index.dataset_version`. The exclusion is **field-aware**: an overlay that touches
 only unrelated columns does not exclude anything from the index on `F`.
 
-`F` here ranges over every field in the index's `fields`, not only the ones it is
-keyed on. An index that carries columns it is not keyed on (see
-[`covering_fields`](../index/index.md#serving-carried-columns)) depends on those
-columns too: an overlay updating a merely-carried column leaves the keyed value
-correct while making the carried value stale, so it must exclude those rows just
-the same.
+`F` here ranges over the union of the index's `fields` and `covering_fields`
+(see [Serving carried columns](../index/index.md#serving-carried-columns)).
+An overlay updating a merely-carried column leaves the keyed value correct while
+making the carried value stale, so it must exclude those rows just the same.
 
 The query then proceeds as:
 

docs/src/format/table/versioning.md

@@ -30,10 +30,11 @@ they should return an "unsupported" error on any read or write operation.
 | 16       | `FLAG_BASE_PATHS`               | Yes             | Yes             | Dataset uses multiple base paths (for shallow clones or multi-base datasets).                               |
 | 32       | `FLAG_DISABLE_TRANSACTION_FILE` | No              | Yes             | Transactions are recorded in the manifest rather than in a separate transaction file.                       |
 | 64       | `FLAG_UNSTABLE_DATA_OVERLAY_FILES` | Yes          | Yes             | Fragments may carry data overlay files. Unstable: release builds reject it unless explicitly opted in.      |
-| 128      | `FLAG_COVERED_INDEX_METADATA`   | Yes             | Yes             | Some index declares covering columns (`IndexMetadata.covering_fields`), so `fields` means keyed columns followed by carried ones. An implementation without this flag selects an index by membership of `fields` and would answer a query on a merely-carried column with an index keyed on a different one. |
+| 128      | `FLAG_COVERED_INDEX_METADATA`   | Yes             | Yes             | Some index declares covering columns (`IndexMetadata.covering_fields`). Without `FLAG_INDEPENDENT_COVERING_FIELDS`, `covering_fields` is a subset of `fields` with no id repeated. An implementation without this flag may select or maintain an index using the wrong fields. |
 | 256      | `FLAG_MIXED_DATA_FILE_VERSIONS` | Yes             | Yes             | The snapshot may reference recognized V2 data files with different exact versions. Both bits must be set and remain set on later versions. |
 | 1024     | `FLAG_FRAGMENT_REUSE_INDEX`     | Yes             | Yes             | The fragment reuse index records tagged transitions (`IndexMetadata.index_version >= 1`). Readers must translate row addresses through them; writers must preserve them. An implementation without this flag would decode the details as the legacy format and silently drop the transitions when it next rewrites the fragment reuse index. See [FRI index versions](../index/system/frag_reuse.md#fri-index-versions). |
+| 2048     | `FLAG_INDEPENDENT_COVERING_FIELDS` | Yes          | Yes             | Requires `FLAG_COVERED_INDEX_METADATA`. `IndexMetadata.fields` contains only key fields, while `covering_fields` independently declares carried fields and may overlap `fields`. Implementations that only support the legacy subset contract must reject the dataset. |
 
 </div>
 
-Flag bit 512 is reserved. Flags with bit values 2048 and above are unknown; unknown flags cause implementations to reject the dataset with an "unsupported" error. The paired mixed-version reader and writer bits must either both be set or both be clear; a half-set manifest is invalid.
+Flag bit 512 is reserved. Unlisted flags are unknown and cause implementations to reject the dataset with an "unsupported" error. The paired mixed-version reader and writer bits must either both be set or both be clear; a half-set manifest is invalid.

protos/table.proto

@@ -123,17 +123,28 @@ message Manifest {
    * * 1 << 6: data overlay files are present (see DataOverlayFile). Readers that do
    *   not understand overlays must refuse the dataset, since ignoring an overlay
    *   would silently return stale base values.
-   * * 1 << 7: some index declares covering columns, so IndexMetadata.fields means
-   *   the keyed columns followed by the carried ones named in covering_fields (see
-   *   IndexMetadata). Readers that do not understand it must refuse the dataset,
-   *   since selecting an index by membership of fields would answer a query on a
-   *   merely-carried column with an index keyed on a different column. Writers must
-   *   refuse it too: one that treats every entry of fields as keyed would maintain
-   *   the index against the wrong dependency set.
+   * * 1 << 7: some index declares covering columns. Without 1 << 11,
+   *   covering_fields is a subset of IndexMetadata.fields: carried columns are
+   *   also listed in fields, with no id repeated (see IndexMetadata). Readers
+   *   that do not understand covering columns must refuse the dataset, since
+   *   selecting an index by membership of fields would answer a query on a
+   *   merely-carried column with an index keyed on a different column. Writers
+   *   must refuse it too: one that treats every entry of fields as keyed would
+   *   maintain the index against the wrong dependency set.
    * * 1 << 8: the dataset may reference recognized V2 data files with different
    *   exact versions. Readers and writers must use each DataFile's version
    *   instead of treating data_format.version as a snapshot-wide identity.
    *   This bit is paired in the reader and writer feature words and is one-way.
+   * * 1 << 11: IndexMetadata.fields and covering_fields are independent
+   *   declarations. This bit requires 1 << 7 in both the reader and writer
+   *   flags. Fields contains only the columns the index is keyed on, while
+   *   covering_fields contains the columns whose values it carries. A field may
+   *   occur in both lists. Implementations that only support the legacy subset
+   *   contract must refuse the dataset. Setting this bit reinterprets every
+   *   IndexMetadata in the index section, so a writer must normalize every
+   *   legacy-form entry in the same manifest -- or set the bit only when no
+   *   entry declares covering fields -- and manifests derived from it must keep
+   *   the bit set until a reverse conversion is likewise atomic.
    */
   uint64 reader_feature_flags = 9;
 
@@ -273,10 +284,14 @@ message IndexMetadata {
   // Unique ID of an index. It is unique across all the dataset versions.
   UUID uuid = 1;
 
-  /* The columns to build the index. These refer to file.Field.id.
+  /* The columns the index is keyed on, in key order. These refer to file.Field.id.
    *
-   * fields[0] is always a column the index is keyed on. Trailing entries may
-   * instead be merely carried, not keyed on -- see `covering_fields` below.
+   * When the manifest has feature flag 1 << 11, this list contains only key
+   * fields. `covering_fields` independently declares the values the index
+   * carries.
+   *
+   * Without feature flag 1 << 11, the legacy contract applies: carried fields
+   * are also present in this list, which holds no repeated id.
    */
   repeated int32 fields = 2;
 
@@ -336,21 +351,23 @@ message IndexMetadata {
    */
   repeated IndexFile files = 10;
 
-  /* The subset of `fields` whose values this index co-locates alongside its own
-   * data, so a query projecting only those columns can be answered from the
-   * index without a take against the base table.
-   *
-   * Must be a subset of `fields`, in the order the index emits them. A column is
-   * carried if and only if it is named here, so a column the index is both keyed
-   * on and carries is listed once in `fields` and named here; no id repeats in
-   * `fields`, and `fields[0]` remains a column the index is keyed on. Empty for
-   * an index that carries no extra columns, which is every index written before
-   * this field existed.
-   *
-   * Carried columns are listed in `fields` as well. That is deliberate: every
-   * consumer that reads `fields` as the index's dependency set -- staleness,
-   * commit conflict detection, schema evolution guards -- then covers them with
-   * no change and no way to forget one.
+  /* The fields whose values this index co-locates alongside its own data, so a
+   * query projecting only those columns can be answered from the index without
+   * a take against the base table.
+   *
+   * When the manifest has feature flag 1 << 11, this declaration is independent
+   * of `fields` and has no positional relationship to it. A field may occur in
+   * both lists when the index carries a value it is also keyed on. Consumers
+   * must use the union of `fields` and `covering_fields` as the index dependency
+   * set.
+   *
+   * Without feature flag 1 << 11, the legacy contract applies: this list must
+   * be a subset of `fields`, in the order the index emits them. A column the
+   * index is both keyed on and carries is listed once in `fields` and named
+   * here; no id repeats in `fields`.
+   *
+   * Empty for an index that carries no extra columns, which is every index
+   * written before this field existed.
    *
    * This declaration is not authoritative for what the segment can actually
    * serve. The segment's own storage schema is: a reader must confirm the