You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Iceberg v3 _row_id is shifted after equality deletes #2493
On Antalya, an Iceberg v3 data file that inherits _row_id from first_row_id returns the wrong ids after an equality delete removes a row that is not a suffix of the chunk. Each surviving row is numbered as if the deleted rows had never occupied a file position, so later rows take earlier rows' ids.
Upstream does not have this. Equality deletes there are applied with FilterTransform(..., update_row_numbers_info = true), which rewrites ChunkInfoRowNumbers when rows are removed. _row_id is first_row_id plus that file position. This branch's FilterTransform has no such parameter, and the equality-delete path still uses the 4-argument constructor:
addRequestedFileLikeStorageVirtualsToChunk then rebuilds positions from chunk.getNumRows() whenever applied_filter is unset. Position deletes and deletion vectors update applied_filter and are not affected. Equality deletes go through FilterTransform, which shrinks the chunk and leaves the row-number info stale.
The virtual column itself was ported by #2308. The wrong result appears only because that port landed on a branch whose equality-delete filter does not keep file positions.
To Reproduce
Iceberg format-version 3 Parquet data file with first_row_id = 0, four rows, and no materialized _row_id column. Inherited ids are 0, 1, 2, 3.
An equality delete that removes the row at file position 1 (not the last row).
SELECT _row_id FROM <table> ORDER BY _row_id
Expected behavior
The three surviving rows keep _row_id 0, 2, and 3.
Actual behavior
They come back as 0, 1, and 2. No exception is thrown.
Fix direction: when the equality-delete filter removes rows, update ChunkInfoRowNumbers::applied_filter the same way DeletionVectorTransform does, before _row_id is built. Upstream already does this inside FilterTransform via update_row_numbers_info.
Activity