fix(lance-table): treat trailing empty segments as no excess in rechunk_sequences - #9302
Open
LuciferYang wants to merge 2 commits into
Open
LuciferYang wants to merge 2 commits into
LuciferYang wants to merge 2 commits into
Conversation
The fill loop drains empty segments only while a chunk still needs rows, so once the last chunk is satisfied a trailing empty segment stays in the iterator and the final too-many-segments check rejected the call even though the total ids matched the chunk sizes exactly. A trailing empty segment carries no ids — skip them before the excess check, matching how the fill loop already treats interior and boundary empties.
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The terminal check now matches rechunk_sequences' row-count contract: leftover segments are rejected only when they still contain row IDs. The added regression cases cover consecutive trailing empties while confirming non-empty and partially consumed excess remains invalid.
Contributor
Author
|
The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9301
rechunk_sequenceserrors when segments are left over after the last chunk is filled. Its fill loop drains empty segments only while a chunk still needs rows, so a trailing empty segment survives to that check and a call whose chunk sizes sum to exactly the number of row ids fails withInvalidInput. The check now asks whether a leftover segment still carries ids.Empty segments are already tolerated everywhere else in the function, with a test case for each position: inside a chunk, at a chunk boundary, and several in a row. Trailing was the one gap. The outcome changes only for inputs where every leftover segment is empty, including the degenerate case of an all-empty input against empty
chunk_sizes; the check is not gated onallow_incomplete, so this applies in both modes. A segment the last chunk only partially consumed is not empty, so leftover ids are still reported.No caller reaches this today. Production row removal goes through
RowIdSequence::mask, which drops empty segments, and the sequences the four callers pass in come either fromrechunk_sequencesitself or from a capture stream whose scan nodes drop zero-row batches. The capture path has no zero-row guard of its own:extract_row_idscaptures every batch it polls, andRowIdSequence::from(&[])yields one empty segment, so the invariant lives in the upstream nodes rather than at the capture site.How was this patch tested?
test_row_id_sequence_rechunk_with_empty_segmentsgains three cases: two trailing empty segments with the row counts matching exactly must produce one chunk of[0, 1]; a segment with ids behind a trailing empty must still be reported as excess; and so must a segment the last chunk only partially consumed. All were mutation-checked. With the old check the first case fails with "Got too many segments for the provided chunk lengths. Processed 1 chunks out of 1 expected"; with a check that never reports excess, the other two fail along with the pre-existing excess case intest_row_id_sequence_rechunk.