fix(fast-path): use manifest max_field_id for new fields#46
Merged
universalmind303 merged 1 commit intoJul 5, 2026
Merged
Conversation
9233c54 to
a160516
Compare
0c5852b to
ecbe266
Compare
universalmind303
approved these changes
Jul 4, 2026
lance_schema.fields() returns only top-level fields. Nested types such as structs have child fields with their own IDs, so max(f.id() for top-level f) can choose a next_fid that collides with an existing child field ID. Use Lance's manifest-level max_field_id as the high-water mark for new column field IDs. This also accounts for nested child fields and field IDs from prior schema evolution. Fixes daft-engine#41
ecbe266 to
b164df9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #41
Problem
FastPathFragmentWritercomputed the field ID for new columns from top-level schema fields:lance_schema.fields()does not provide a complete high-water mark for field IDs. Nested fields, such as struct children, have their own Lance field IDs. Schema evolution can also leave previously used field IDs behind. Ifnext_fidis based only on visible top-level fields, the new data file can be committed with a field ID that collides with an existing nested or historical field ID.Example for a table with
id(fid=0) and a structmeta(fid=1) with childrenmeta.a(fid=2) andmeta.b(fid=3):meta.amax_field_idWith the wrong
next_fid, fragment metadata can associate the new file with the wrong schema field. Reads of the intended new column can then return null or wrong values.Fix
Use Lance's manifest-level field ID high-water mark:
max_field_idreflects nested child fields and field IDs reserved by prior schema evolution, so it is the correct source for assigning new field IDs.Test
Adds
TestRegressions::test_next_fid_uses_manifest_max_field_id, which creates a dataset with a struct column, merges a new scalar column through the fast path, and verifies that the new column reads back with the expected values.