feat: add index build progress callbacks to segment builders

[u70b3]

E3 of the RFC #55 distributed-build track: an FFI bridge over lance core's IndexBuildProgress trait for the uncommitted segment builders. Build stages (train_ivf, train_quantizer, shuffle, merge_partitions, ...) and per-stage progress are now observable from C/C++.

API

typedef enum {
    LANCE_INDEX_BUILD_PROGRESS_STAGE_START = 0,
    LANCE_INDEX_BUILD_PROGRESS_STAGE_PROGRESS = 1,
    LANCE_INDEX_BUILD_PROGRESS_STAGE_COMPLETE = 2,
} LanceIndexBuildProgressEvent;

typedef void (*LanceIndexBuildProgressCallback)(
    void* callback_ctx, int32_t event, const char* stage,
    uint64_t total, const char* unit, uint64_t completed);

int32_t lance_index_segment_builder_set_progress_callback(
    LanceIndexSegmentBuilder* builder,
    LanceIndexBuildProgressCallback callback,
    void* callback_ctx);

• total/unit are meaningful only for STAGE_START (0/"" otherwise); completed only for STAGE_PROGRESS. stage is always non-NULL, borrowed, valid for the call duration. Stage names are diagnostic-only, not a stable cross-version contract.
• callback_ctx may be NULL; setting a callback replaces any previously set callback; it must be set before execute (the builder is single-use), otherwise the setter returns -1.

Contract (documented on the typedef and setter)

• Invoked from lance-c's internal tokio runtime worker threads; certain stages report concurrently from parallel worker tasks (e.g. tokenized docs), so the callback must be thread-safe and reentrant, non-blocking, and must not call back into any lance_* function.
• Invocations occur only while execute_uncommitted runs, and this is enforced rather than contractual: a shared retirement gate (ProgressCallbackGate) disables new callback entries and drains in-flight invocations before execute_uncommitted returns — on success, error, and panic paths via a drop guard — so a core worker task detached on an error path (e.g. the inverted index's tokenize_docs workers, whose JoinHandles can be dropped early) degrades to a no-op instead of touching freed context. callback/callback_ctx must remain valid until execute_uncommitted returns.
• Invoked without a panic guard (house position for user callbacks): it must return normally; unwinding/throwing across this boundary can abort the host process.
• Advisory only: the callback cannot abort the build (cancellation remains the separate E4 track, blocked on a core hook).

Deviations from the RFC sketch

• Naming: RFC §7 sketched lance_index_segment_builder_set_progress / user_data; this PR uses set_progress_callback / callback_ctx to match the newer in-tree lance_scanner_set_statistics_callback convention. No semantic change.
• NULL callback: RFC §12's test plan sketched "NULL callback = no-op"; the setter instead rejects NULL with -1, matching the lance_scanner_set_statistics_callback boundary convention (reject invalid values at the API boundary). Consequence: a set callback can be replaced but not unset — install a no-op callback to disable reporting.

Implementation notes

• The retirement gate was added in response to review: core's inverted builder clones the progress handle into spawned tokenize_docs workers whose JoinHandles can be dropped early on error paths, which could otherwise invoke the raw C context after the build returned. The gate (SeqCst disable + drain, Arc-shared across every clone core hands to worker tasks) closes that window and makes the documented boundary enforceable.
• The bridge always returns Ok(()); the only error path is a defensive interior-NUL invariant violation in stage/unit strings (never panics).
• Direct dependency async-trait added for the #[async_trait] impl of lance_index::progress::IndexBuildProgress (already in the dependency graph transitively via lance-index; Cargo.lock gains only the lance-c edge).

Tests

Gate-level unit tests (src/index_segment.rs):

• detached_clone_cannot_enter_callback_after_retire — the review reproducer, inverted: a task-owned clone calling after gate retirement is a no-op and never touches the context.
• retire_drains_in_flight_invocation — retirement disables new entries, then blocks until an in-flight invocation exits.

Rust (tests/c_api_test.rs):

• test_vector_index_segment_progress_callback — IVF_PQ 256x16 through the segment builder; event codes, per-stage START/COMPLETE pairing, numeric mapping invariants (PROGRESS ⇒ total==0, START ⇒ completed==0), shuffle and merge_partitions observed, shuffle PROGRESS completed <= total with unit == "rows", ctx round-trip.
• test_scalar_index_segment_progress_callback_sees_load_data — BTree build sees load_data START/COMPLETE.
• test_vector_index_segment_progress_callback_multi_fragment_subset — 2-fragment dataset, fragment-scoped build with callback active.
• test_index_segment_builder_progress_callback_edge_cases — NULL builder / NULL callback / NULL callback_ctx (arrives verbatim) / sentinel ctx round-trip / set after execute / set twice (only the replacement receives events).
• test_index_segment_builder_progress_callback_success_clears_error.

C (tests/cpp/test_c_api.c) and C++ (tests/cpp/test_cpp_api.cpp): test_index_segment_builder_progress each — real compile-and-run coverage, including the fluent progress_callback setter on the C++ side.

Full suite green: cargo fmt, cargo check --all-targets, cargo clippy --all-targets -- -D warnings, cargo test (358 c_api tests incl. the 5 new), cargo test --test compile_and_run_test -- --ignored (C + C++ + static OSS transport).

Refs #55.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The callback-lifetime blocker is fixed in 4a5f53b: a gate shared by all progress clones disables late entries and drains in-flight callbacks before execute_uncommitted returns on success, error, or panic. The C and C++ lifetime contract now matches that enforced boundary.