refactor: decouple index build utilities for shared use - #821
Merged
Merged
Conversation
This was referenced Sep 13, 2026
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
ScalarSegmentIndexJob now takes a SparkContext and primitive parameters instead of the AddIndexExec instance, so REFRESH INDEX can share the same distributed build path without coupling to CREATE INDEX. Move extractNamespaceInfo and isSystemIndex to IndexUtils, extract parseNumSegments, and add methodForIndexType (reverse IndexType lookup), resolveRetainedSegments (commit-time index re-resolution), requireUniformSegmentDetails (FTS segment compatibility guard), and improve balancedRunLengths to split the heaviest run first via a priority queue instead of left-to-right.
ivscheianu
force-pushed
the
feat/refresh-index-rebased
branch
from
September 14, 2026 15:40
4de5461 to
11e9659
Compare
Trim ~40 lines of prose comments down to constraint-only notes, add a test for retainedSegments returning empty when all segments cover only retired fragments, and remove the empty ShowIndexesExec companion object.
lance-gatekeeper
Bot
removed
the
K-approved
There was a problem hiding this comment.
✅ Gate recommendation: approve.
This remains a coherent prerequisite split for PR #822: the distributed segment job takes explicit build inputs, and shared resolution, validation, and batching stay centralized without changing CREATE INDEX behavior. The rebase preserves that implementation; this revision only trims comments, removes an empty companion object, and adds coverage for the all-retired-segments case. The remaining atomic drop/commit window is still isolated to lance#6806.
lance-gatekeeper
Bot
added
the
K-approved
Contributor
Author
|
hey @geruh, can we continue with this one? thanks a lot for all the support in reviewing all these PRs! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #789 (1/2). Prepares the index build infrastructure for
REFRESH INDEX(#784).Summary
ScalarSegmentIndexJobtakes aSparkContextand primitive parameters instead of theAddIndexExecinstance, soREFRESH INDEXcan share the same distributed build path without coupling toCREATE INDEX.extractNamespaceInfoandisSystemIndexmove toIndexUtils;parseNumSegmentsis extracted there as well.IndexUtilsmethods for the refresh path:methodForIndexType(reverseIndexType→ SQL method lookup),resolveRetainedSegments(commit-time index re-resolution to catch concurrentDROP INDEX),requireUniformSegmentDetails(FTS segment compatibility guard).balancedRunLengthsimproved to split the heaviest splittable run first via a priority queue, so the extra batches land where they help most.No new SQL commands or grammar changes; existing behavior is preserved. 19 new unit tests for the extracted and added methods (61 total in
IndexUtilsTest, all pure JVM, no native deps).