Skip to content

fix: preserve projected files across rebased writes - #9219

Open
lance-gatefixer[bot] wants to merge 6 commits into
mainfrom
gatekeeper/fix-9217-1
Open

lance-gatefixer[bot] wants to merge 6 commits into
mainfrom
gatekeeper/fix-9217-1

Conversation

@lance-gatefixer

@lance-gatefixer lance-gatefixer Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • apply a rebased Delete by updating only the deletion file on the current fragment, including when Project may alter nullability
  • prune fully projected-out files from updated and newly written fragments in every Update mode
  • preserve files that still contain any live field and keep the field-id watermark correct

Root cause

Delete and Update manifest application reused fragment post-images staged against the transaction read version. When a concurrent Project pruned files whose fields were all removed, those stale post-images could restore the files during rebase. RewriteColumns could also restore a newly written file for a column the Project had removed.

Delete now carries only its deletion-file delta onto the current fragment. The same rule covers a Project with preserves_nullability set to false; both Update rewrite modes are rejected earlier by the nullability conflict pre-check. Update filters both updated and new fragment files against the live projected schema while retaining mixed files that still contain a live field.

Validation

  • cargo fmt --all
  • cargo test -p lance-table
  • cargo clippy --all --tests --benches -- -D warnings

Fixes #9217

@github-actions github-actions Bot added the bug Something isn't working label Sep 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 14, 2026
@lance-gatefixer lance-gatefixer Bot changed the title fix: preserve projected files when rebasing deletes fix: preserve projected files across rebased writes Sep 14, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 14, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 17, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

With #9218 now in the base, this revision cleanly supplies the remaining Update half: it applies Project’s live-field rule to both staged updated fragments and new fragments, preserving mixed files while preventing fully projected-out files from returning. The strengthened Delete regression confirms the two fixes compose.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: a legacy Delete reinstates a data file a concurrent Project pruned

0 participants