Concurrent index optimization can resurrect a dropped index

[majin1102]

Description

A stale index-maintenance commit can undo a concurrently committed DROP INDEX and make the dropped index visible again.

Index creation, index optimization, and index removal are represented through index transaction operations. When optimize_indices prepares an operation from version N while another writer drops the same index and commits version N+1, the stale optimize operation may still be accepted/merged. Its new index segments can then reintroduce the index metadata removed by the drop.

This is a Core transaction-conflict issue and affects callers in any binding. It was identified while exposing the existing Core index-maintenance API through Spark SQL in lance-format/lance-spark#745.

Steps to reproduce

# Transaction-level regression-test outline (the race needs a barrier before
# the optimize commit to make it deterministic):
#
# 1. Create a dataset and an index named "idx".
# 2. Open two handles, optimizer and dropper, at the same dataset version N.
# 3. Start optimizer.optimize_indices(index_names=["idx"]), and pause it after
#    its CreateIndex operation has been prepared but before it commits.
# 4. Run dropper.drop_index("idx") and let it commit version N+1.
# 5. Resume the stale optimize commit.
# 6. Reopen the latest dataset version and list/describe indexes.
#
# Actual: "idx" can be present again because the stale optimize operation is
# accepted or merged after the drop.

The most reliable regression test is likely at the Rust transaction/conflict-resolver layer, using an explicit barrier or manually constructed operations rather than timing public API calls.

Expected behavior

An optimize operation based on stale metadata must conflict when the target index was concurrently dropped or replaced. Core should reject or retry that commit; after retry, optimization should observe that the target index no longer exists. A successfully committed drop must not be undone by an older maintenance operation.

Conflict handling should distinguish safe maintenance merges from target-index identity changes. Comparing only the index name is insufficient if an index can be dropped and recreated under the same name.

Lance version

Observed against the transaction behavior used by Java lance-core 11.0.0-beta.21; the relevant conflict behavior is also present in current Core transaction code.

Language binding

Rust, Java

Environment

Storage-backend independent; local filesystem is sufficient for a deterministic transaction-level test.