feat: enable the fragment reuse index on stable row id datasets - #9120
Conversation
github-actions
Bot
added
A-python
ec31dd7 to
1b17fdc
Compare
404d41e to
6222bf0
Compare
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes. Status: ❌ Blocked — 0 of 3 required approvals
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
K-changes
6222bf0 to
a70d498
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
a70d498 to
09c42ce
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
lance-gatekeeper
Bot
added
K-approved
09c42ce to
ad7e62a
Compare
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
ad7e62a to
7513ebf
Compare
lance-gatekeeper
Bot
removed
the
K-changes
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
7513ebf to
662da11
Compare
On a stable-row-id dataset most indices store stable row ids, which a fragment reuse index must not remap: stable ids start at 0 and collide with fragment 0's row addresses. Index loading, merging and building now hand the fragment reuse index only to indices that store row addresses. Cleanup and remap_column_index treat stable-id indices as needing only their fragment coverage repaired, and the manifest build derives the FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDS compatibility flag, which this build now supports. A JSON index answers in its target's domain; both domain predicates now follow the target, and malformed JSON details are an error rather than a guess. Under stable row ids the rewrite branch no longer recalculates the fragment reuse index's own coverage, which the eager path never touched.
Compaction with defer_index_remap no longer rejects stable row ids. The row addresses the fragment reuse index records are derived from the fragment metadata (every physical row minus the deletion vector), which is what the scan yields without stable row ids, while the row id sequences are rechunked as before. Closes lance-format#8977
lance-gatekeeper
Bot
removed
the
K-approved
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
662da11 to
670460d
Compare
lance-gatekeeper
Bot
removed
the
K-approved
There was a problem hiding this comment.
✅ Gate recommendation: approve.
This revision rebases the same change series onto the updated base. The feature-flag integration preserves the newly reserved tagged-FRI bit as unsupported while enabling stable-row-ID FRI; the focused feature-flag suite passes, and the prior FRI correctness evidence remains valid.
lance-gatekeeper
Bot
added
the
K-approved
2 participants
Part of the stable row ID GA work in §5.1. Enables defer_index_remap on datasets with stable row IDs.
Compaction, index loading, and maintenance now decide whether to apply FRI mappings based on the identifiers each index stores. Indexes that store stable row IDs keep their entries and update fragment coverage as needed. Indexes that store physical row addresses use the FRI. JSON indexes follow the identifier type of their underlying index.
Sets the reader and writer feature flag from #9119 for stable-row-ID datasets with an FRI. Shallow clones preserve the flag, and metadata-only index repairs preserve the source location of index files.
FRI creation remains opt-in through defer_index_remap and depends on indexed data or an existing FRI chain. Enabling it by default and the remaining cleanup integration are follow-up work.
Depends on #9119.