Conversation
`CompactFragReuseIndex::try_new` builds the index every dataset open loads, and its lookups decide whether an index row is found, moved or gone after a compaction. The existing tests cover a one-to-many split, a two-link chain, corrupt bytes and three layout validations. Add unit tests for the behaviours between those: moved, deleted and untouched rows in one group, including that an output address is not remapped again and an offset past a fragment's recorded row count passes through; positional pairing in the order the old fragments are listed rather than by address; offsets in run and bitmap containers on a 200k-row fragment; a task rolling into five new fragments; an emptied fragment mid-group not shifting later positions; a 32-link chain; a deletion mid-chain being terminal; independent positions for groups sharing a version; empty payloads; and both directions of the rewritten-rows versus new-rows count check. The corrupt-bytes test gains a truncated-body case. In the `lance` crate, check the compact index against `transpose_row_ids_from_digest`, the per-row transposition compaction uses under `IndexRemapMode::Direct`, on every address of an ascending-fragment payload: that is the case compaction produces, and the two must agree on it.
amunra
force-pushed
the
upstream/frag-reuse-open-path-tests
branch
from
September 17, 2026 08:23
48f4a8b to
0d97be1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
We had independently implemented the equivalent of #8887 on our own branch: opening the fragment-reuse index into a compact per-fragment representation instead of expanding every reuse version into a per-row map. Same reason, the expansion dominates memory on a long reuse chain, and much the same shape, positional mapping of rewritten rows onto new fragments with rank lookups over roaring bitmaps.
By the time we came to propose it, #8887 was already merged, so none of that implementation is needed here. What our branch also carried was the tests written while getting the mapping right, and those cover behaviours the current suite does not. This PR is only those tests, adapted to the code on
main.What it adds
The existing tests in
rust/lance-table/src/system_index/frag_reuse.rscover a one-to-many split, a two-link chain, corrupt bytes and three layout validations. These fill in the gaps:RankedOffsetsvariants at the sizes real payloads reach, where a rank error returns a wrong row address rather than failing.transpose_row_ids_from_digestis still the compaction-side path underIndexRemapMode::Direct. One test in thelancecrate checks the compact index against it on every address of an ascending-fragment payload, the ordering compaction actually produces. Nothing in the tree asserted that the two agree.Notes
Test-only; no production code changes. The tests build
FragReuseIndexDetailsdirectly with the module's existing helpers plusgroupanddetailsconstructors, and run synchronously.Tracking: Ported in part from rerun-io#59.