perf: prune CDF fragments using row-version metadata

[Abhisheklearn12]

Fixes #8853.

Inserted, updated, and upserted CDF queries currently scan every fragment. This adds pruning in delta.rs using available inline RLE version metadata, while keeping the existing row filters for exact results.

Missing, unreadable, or non-inline metadata is treated as unknown rather than used to exclude matching rows. Decode failures emit a contextual warning. When every fragment survives, the scanner reuses the manifest's shared fragment list without copying it.

Public API signatures and the file format are unchanged. Scanner-level pruning, the old external-sequence loader, and the storage redesign tracked in #8931 are outside this PR.

Performance

Added a Criterion benchmark comparing the CDF APIs with equivalent full scans.

Criterion point estimates from a warm local-file run using release-with-debug, with 20,000 initial fragments and 16 rows per fragment:

Query	Full scan	Pruned scan	Result
Inserted	577.59 ms	2.60 ms	About 222x faster
Updated	607.56 ms	2.64 ms	About 230x faster
Upserted	654.25 ms	3.93 ms	About 166x faster
All-row control	638.73 ms	641.42 ms	0.4% slower

A control-only confirmation run measured 639.01 ms versus 640.72 ms. Both differences are inside overlapping confidence intervals; decoding the inline version metadata of all 20,000 fragments takes about 1 ms. An earlier revision reported the control as 5.5–13.2% slower. That was a benchmark artifact: dev-dependencies enable lance-io/test-util, whose I/O request log grew with every scan and slowed later iterations, and Criterion ran the pruned samples last. the benchmark now drains that log after each scan.

These measurements isolate small-fragment overhead and are not production speedup guarantees.

Run the benchmark with:

cargo bench -p lance --bench cdf --profile release-with-debug

Validation

• All 70 CDF tests passed.
• Tests verify equivalence with full scans, actual file pruning, compaction, range boundaries, empty results, and metadata fallbacks.
• Existing integration tests include 12 additional comparisons with one version column's metadata missing or unreadable.
• Required workspace Clippy and CI additional-feature Clippy passed.
• Workspace documentation passed with warnings denied.
• Formatting and git diff --check passed.
• The optimized benchmark completed earlier with all 24 row-count checks passing.

The full workspace test suite was not run locally.

[Abhisheklearn12]

I checked the code and tracked down the all-row control regression. It turns out the issue was in the benchmark, not the pruning logic.

The benches are built with dev-dependencies, which enable lance-io/test-util. this makes the IO tracker keep every read request in memory until io_stats_incremental() is called. The benchmark wasn't calling it after each scan.

As a result, every 20k-file scan added roughly 20k more records to the tracker, making each iteration a little slower than the previous one. For example, a Full-only loop drifted from 616 ms → 705 ms over 14 rounds, even though pruning wasn't involved.

Since Criterion runs all the Full samples before the Pruned samples, the Pruned path was always measured after this accumulated drift. That made it look like there was a regression in the all-row control.

The pruning itself only adds about 1 ms at 20k fragments, compared with a roughly 640 ms scan.

Fix

The fix was just 5 lines in the benchmark to drain the tracker after each scan. no library code was changed (i fiixed that).

After rerunning:

Query	Full scan	Pruned scan	Result
Inserted	577.59 ms	2.60 ms	~222× faster
Updated	607.56 ms	2.64 ms	~230× faster
Upserted	654.25 ms	3.93 ms	~166× faster
All-row control	638.73 ms	641.42 ms	0.4% slower (within the CIs)

I also ran the control by itself as a sanity check: 639.01 ms vs 640.72 ms.

So the control path is effectively unchanged. imo, the Full-scan numbers came down as well because they were affected by the same accumulated drift. The speedup ratios are therefore slightly smaller, but these are the more accurate numbers.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The benchmark correction changes the basis of the prior review: the apparent all-row regression came from the dev-only I/O request log growing between Criterion samples, not from the pruning implementation. The benchmark now drains that tracker after every scan; the reset mechanism is confirmed in the implementation, and a focused 20,000-fragment rerun produced overlapping Full and Pruned intervals with no slowdown.

The delta-specific RLE prefilter remains a sound, focused solution to #8853. It excludes fragments only when inline metadata proves no row can match, retains unknown metadata, and keeps the row-level predicate authoritative, preserving results across boundaries and compaction without changing the file format.