fix(index): match fresh-encode output schema in RQTransformer early return - #9396
Open
LuciferYang wants to merge 3 commits into
Open
LuciferYang wants to merge 3 commits into
LuciferYang wants to merge 3 commits into
Conversation
…eturn The already-encoded early return kept the vector and centroid-dist columns that the normal path drops, so re-processed batches carried a different schema than freshly-encoded ones. Drop both columns when present in the early return. Assisted-by: GLM-5.3
github-actions
Bot
added
A-index
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The already-encoded exit now applies the same schema normalization as fresh encoding, so callers receive a stable transformer contract without needing caller-specific projections. The change reuses the existing no-op-safe column removal and is covered by a regression that exercises the pre-encoded path.
lance-gatekeeper
Bot
added
the
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9395
The fresh-encode path drops the vector and centroid-distance columns before returning; the early return for an already-encoded batch returned it untouched, so the transformer's output schema depended on whether the codes were already present. It now drops the same two columns on that path, and
drop_columnis a no-op when they are absent, so a pre-encoded batch that has already lost them is unaffected.No in-tree caller changes behaviour: the pre-encoded batches Lance produces have already dropped the vector column by the time they reach here.
How was this patch tested?
test_rq_transformer_early_return_matches_fresh_encode_schemaencodes a batch, re-attaches the vector and centroid-distance columns to the encoded result, feeds that back through the transformer, and compares the two output schemas field by field. RestoringOk(batch.clone())fails it.