feat: report segment index build progress

[hfutatzhanghb]

Summary

Report distributed CREATE INDEX progress using the independent segment architecture.

This is a clean replacement for #747. PR #701 replaced the old shared UUID plus mergeIndexMetadata FTS flow with one uncommitted Lance segment per Spark task followed by an atomic commitExistingIndexSegments call. Lance Core does not expose a progress callback on createIndex or segment commit, so this implementation observes progress at the Spark driver boundary as successful segment partitions return.

Changes

• Add Spark SQL metrics for eager distributed segment builds:
  • index build completed segments
  • index build total segments
• Execute segment tasks with the SparkContext.runJob driver result handler so progress is visible before the full job result is available.
• Capture the SQL execution ID on the command thread and publish every driver-side snapshot with SQLMetrics.postDriverMetricUpdates so the SQL listener, status store, and Spark UI receive live updates.
• Count each successful Spark partition once, preserving independent segment retry and speculation isolation.
• Validate that parallelize(tasks, tasks.size) produces exactly one segment task per partition, with explicit errors for empty or multi-task partitions.
• Keep metric mutation, publication, logging, and warning failures informational so they cannot fail index creation or suppress the other observation channels.
• Log segment build start and completion progress on the driver.
• Document both the Spark UI descriptions and executed-plan metric keys.
• Add pure JVM tests for accounting, out-of-order completion, duplicate partitions, publication ordering, observation failure isolation, metric selection, and the empty/exact/multi-task partition invariant.
• Extend the existing real Lance zonemap segment integration test to verify final metric values and listener snapshots for 0/total, intermediate, and total/total states, including bounds and monotonicity. Listener timeouts include received snapshots, and the listener is removed on every test exit path.

Scope

Progress is reported for eager segment builds: zonemap, bitmap, label list, ngram, bloomfilter, rtree, FTS or inverted, and BTree fragment mode. Range-mode BTree keeps its dedicated path. Deferred and empty-table builds do not launch segment tasks.

This does not restore the removed mergeIndexMetadata path and does not change atomic segment commit behavior. When no SQL execution ID exists, Spark skips SQL-listener publication and driver logging remains the informational fallback.

Validation

• ./mvnw spotless:check
• ./mvnw test -pl lance-spark-3.4_2.12 -Dtest=IndexUtilsTest — 34 tests passed
• ./mvnw test -pl lance-spark-3.5_2.12 -Dtest=IndexUtilsTest — 34 tests passed
• ./mvnw test -pl lance-spark-3.5_2.13 -Dtest=IndexUtilsTest — 34 tests passed
• ./mvnw test -pl lance-spark-4.2_2.13 -Dtest=IndexUtilsTest with JDK 17 — 34 tests passed

The local native listener integration run cannot start because the installed Lance artifact lacks the darwin-x86-64 JNI library; GitHub Actions is the native validation gate.

[hfutatzhanghb]

Hi, @fangbo . Have opened this new PR. Could you please review this when have free time? Thanks very much !!!

[fangbo]

@hfutatzhanghb This is great work for improving the observability of index building.

That said, I’m wondering whether adding these metrics is really necessary. At the moment, each segment build corresponds to a Spark task, and Spark jobs already expose task-level metrics such as total tasks, completed tasks, and running tasks. These seem to map directly to total segments and completed segments. What do you think?

[hfutatzhanghb]

@hfutatzhanghb This is great work for improving the observability of index building.

That said, I’m wondering whether adding these metrics is really necessary. At the moment, each segment build corresponds to a Spark task, and Spark jobs already expose task-level metrics such as total tasks, completed tasks, and running tasks. These seem to map directly to total segments and completed segments. What do you think?

That is a fair point. Our original motivation was that, when index building is slow, a Spark task may remain in the Running state for a long time, and we would like to know whether it is stuck or still making progress, as well as how far the internal index build has progressed. However, the current implementation only reports progress when a segment task completes, so it does largely overlap with Spark’s task-level metrics.

[hfutatzhanghb]

@fangbo ， Thanks very much for your valuable suggestion.
I’m implementing this in two stages. First, I opened lance-format/lance#8823 to expose a new Dataset.createIndex(IndexOptions, IndexBuildProgress) Java/JNI overload. It forwards the existing Rust IndexBuildProgress callbacks from CreateIndexBuilder, including stage start, progress, and completion events. After that API is released, lance-spark will create an IndexBuildProgress adapter inside each distributed FTS segment task and use the callbacks to update executor-side Spark SQL activity metrics and structured logs while the task is still running. This preserves the current independent-segment architecture and atomic commitExistingIndexSegments publication path. Since different stages report different units and may not have a known total, the Spark metric will represent observable forward-progress activity rather than an aggregated percentage.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The callback-driven design addresses the prior mid-task observability gap, but this revision cannot build against its declared Lance dependency. Update to a released lance-core artifact containing lance#8823 and verify that dependency across the supported modules before this can be accepted.

[lance-gatekeeper]

pom.xml still pins lance-core 11.0.0-beta.21, which predates lance#8823 and exposes only createIndex(IndexOptions). This new two-argument call therefore makes every module compiling the shared base fail before tests run. Update the pinned dependency to a published artifact containing that overload, then run the focused callback and heartbeat tests against it.

Reproducer

./mvnw -q -pl lance-spark-3.5_2.13 -am -DskipTests compile

AddIndexExec.scala:848: overloaded method createIndex ... cannot be applied to (IndexOptions, SparkIndexBuildProgress)