Background
PR #755 adds ALTER TABLE ... OPTIMIZE INDEX for incremental scalar-index maintenance. It intentionally does not expose the retrain option, because in lance-core retrain applies only to v3 vector indexes and is a no-op for the scalar indexes this command targets — exposing it via SQL would promise behavior it cannot deliver for scalar targets. retrain remains available through the SDK (Dataset.optimizeIndices(OptimizeOptions.builder().retrain(true).build())).
Proposal
Expose retrain through OPTIMIZE INDEX ... WITH (retrain = true) for vector index targets, rejecting it for scalar targets with a clear error.
Two parts, ideally sequenced:
- lance-core: have
optimizeIndices return an explicit error when retrain=true targets an index type that does not support it, instead of silently no-op'ing. This makes the contract enforceable at one authoritative place.
- lance-spark: once core validates, accept
retrain in OptimizeIndexExec, pass it through, and surface core's error. (If we want to fail fast in Spark before the core call, gate on Index.indexType() being a vector type — detectable via the IndexType enum — and handle the all-index case, but the core-side error is the more robust source of truth.)
Why not do it in #755
Doing this correctly wants the lance-core change first (so the Spark layer doesn't have to re-encode the "v3-vector-only" contract, which can drift across core releases). Keeping it out of #755 avoids expanding that PR's validation surface.
References
Background
PR #755 adds
ALTER TABLE ... OPTIMIZE INDEXfor incremental scalar-index maintenance. It intentionally does not expose theretrainoption, because in lance-coreretrainapplies only to v3 vector indexes and is a no-op for the scalar indexes this command targets — exposing it via SQL would promise behavior it cannot deliver for scalar targets.retrainremains available through the SDK (Dataset.optimizeIndices(OptimizeOptions.builder().retrain(true).build())).Proposal
Expose
retrainthroughOPTIMIZE INDEX ... WITH (retrain = true)for vector index targets, rejecting it for scalar targets with a clear error.Two parts, ideally sequenced:
optimizeIndicesreturn an explicit error whenretrain=truetargets an index type that does not support it, instead of silently no-op'ing. This makes the contract enforceable at one authoritative place.retraininOptimizeIndexExec, pass it through, and surface core's error. (If we want to fail fast in Spark before the core call, gate onIndex.indexType()being a vector type — detectable via theIndexTypeenum — and handle the all-index case, but the core-side error is the more robust source of truth.)Why not do it in #755
Doing this correctly wants the lance-core change first (so the Spark layer doesn't have to re-encode the "v3-vector-only" contract, which can drift across core releases). Keeping it out of #755 avoids expanding that PR's validation surface.
References
OPTIMIZE INDEX; documents this omission under "Notes / follow-ups")OptimizeOptions.retraindoc: "only supported for v3 vector indices"