feat: add distributed REFRESH INDEX for incremental index maintenance

[ivscheianu]

Part of #789 (2/2). Depends on #821 — review that first; the diff here includes it until it merges.

Design: #788. Supersedes #784 (rebased and split).

Summary

CREATE INDEX rebuilds every fragment, so keeping an index current on an append-heavy table means re-indexing the whole table. This adds an incremental counterpart:

ALTER TABLE lance.db.users REFRESH INDEX user_id_idx [WITH (num_segments = 8)];

It indexes only the fragments the index does not cover, distributed the same way CREATE INDEX is: the driver diffs coverage against the table and balances the remainder by row count, executors build one uncommitted segment per batch, and the driver commits them as one logical index. Lance core keeps existing segments whose fragments are disjoint from the incoming ones, so prior coverage survives the commit.

A deferred index (train = false) covers nothing, so refreshing one builds the whole table through the same distributed path — replacing the guidance to call Dataset.optimizeIndices, which runs on a single node.

Guards

• The index is re-resolved on a fresh handle before the commit, so a DROP INDEX during the build is caught rather than silently undone.
• FTS segments are compared against the ones they would join and rejected on configuration mismatch before anything is committed, because an inverted index loads one configuration for all of its segments.

Known gap

A concurrent DROP INDEX between re-resolution and the commit can be silently undone. This needs lance-format/lance#6806 on the core side. Pinned by a disabled regression test.

Not in scope

• Distributed segment merge — each refresh adds segments; consolidation still requires CREATE INDEX.
• Vector indexes — not creatable from Spark SQL.
• Inheriting build parameters — no binding reads a built index's parameters back; options come from the WITH clause with type defaults as fallback.

Changes (this PR only, excluding #821)

Component	Files
Grammar	LanceSqlExtensions.g4 (+2 lines: refreshIndex rule, REFRESH keyword)
Parser visitors	5 files, visitRefreshIndex added
Logical plan	RefreshIndex.scala (new)
Strategy	LanceDataSourceV2Strategy.scala (+3 lines)
Physical exec	RefreshIndexExec.scala (227 lines, new)
Tests	BaseRefreshIndexTest.java (766 lines, 26 tests), parser tests, per-version stubs
Docs	refresh-index.md (214 lines)

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The distributed refresh path matches the incremental-maintenance contract: executor work is pinned to the planned snapshot, current coverage and full-text configuration are checked before the logical segment commit, existing coverage is preserved, and the operational limits are documented.

Until lance#6806 is released and pinned, avoid running DROP INDEX concurrently with a refresh. A drop in the narrow check-to-commit window can be undone and leave a partially covered index; SHOW INDEXES makes that state detectable and another drop recovers it, so this remains a non-blocking operational risk.

[ivscheianu]

hello, @geruh! Here's the final one. Looks big, but a significant part of it are tests and also a bit of documentation, hopefully it's manageable. Thanks a lot!