Skip to content

Iceberg v3 _row_id is shifted after equality deletes #2493

Open

Description

@DimensionWieldr

Describe the bug

On Antalya, an Iceberg v3 data file that inherits _row_id from first_row_id returns the wrong ids after an equality delete removes a row that is not a suffix of the chunk. Each surviving row is numbered as if the deleted rows had never occupied a file position, so later rows take earlier rows' ids.

Upstream does not have this. Equality deletes there are applied with FilterTransform(..., update_row_numbers_info = true), which rewrites ChunkInfoRowNumbers when rows are removed. _row_id is first_row_id plus that file position. This branch's FilterTransform has no such parameter, and the equality-delete path still uses the 4-argument constructor:

return std::make_shared<FilterTransform>(header, std::make_shared<ExpressionActions>(std::move(dag)), "notInResult", true);

addRequestedFileLikeStorageVirtualsToChunk then rebuilds positions from chunk.getNumRows() whenever applied_filter is unset. Position deletes and deletion vectors update applied_filter and are not affected. Equality deletes go through FilterTransform, which shrinks the chunk and leaves the row-number info stale.

The virtual column itself was ported by #2308. The wrong result appears only because that port landed on a branch whose equality-delete filter does not keep file positions.

To Reproduce

  1. Iceberg format-version 3 Parquet data file with first_row_id = 0, four rows, and no materialized _row_id column. Inherited ids are 0, 1, 2, 3.
  2. An equality delete that removes the row at file position 1 (not the last row).
  3. SELECT _row_id FROM <table> ORDER BY _row_id

Expected behavior

The three surviving rows keep _row_id 0, 2, and 3.

Actual behavior

They come back as 0, 1, and 2. No exception is thrown.

Key information

Additional context

Caused by: #2308

Fix direction: when the equality-delete filter removes rows, update ChunkInfoRowNumbers::applied_filter the same way DeletionVectorTransform does, before _row_id is built. Upstream already does this inside FilterTransform via update_row_numbers_info.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions