perf: prune V2 LIMIT/OFFSET scans by live row counts

[LeoReeYang]

Summary

• Unify V2 LIMIT/OFFSET planning across stable and non-stable row IDs, with and without deletions.
• Prune complete leading and trailing fragments from manifest live-row counts (physical_rows - num_deleted_rows) while preserving the requested fragment order.
• Reuse the loaded physical row count and deletion vector to compute visible rows in FilteredReadExec; keep stable and address-based row-ID mapping as separate branches.
• Preserve V1 behavior and the existing non-pushdown fallback when writer or row-count metadata is missing, inconsistent, or otherwise untrusted.
• Keep filters, ordering, external masks, and include_deleted_rows on their existing fallback paths.

The correctness invariant is that the optimized scan over an ordered fragment sequence equals a full scan over the same sequence followed by slice(offset, limit). Tests cover explicit/reordered fragment subsets as well as all four row-ID/deletion combinations.

Benchmark

cargo bench -p lance --bench stable_row_id_limit --profile release-with-debug -- --noplot

Release-profile Criterion benchmark on the same WSL host, using the same benchmark source on the parent commit and the final branch:

• 128 V2 stable-row-ID fragments
• 1,024 physical rows and one deletion per fragment
• OFFSET targets the eighth fragment from the end; LIMIT is 16
• 2 second warmup, 5 second measurement, 20 samples

Revision	Time
Parent 88b455fbd (deletions disable pushdown)	9.4555-9.5994 ms; center 9.5333 ms
This branch at 1029a8f2a	427.99-449.22 us; center 438.55 us

The final implementation is approximately 21.7x faster, reducing center latency by 95.4% for this workload. The pruning regression independently verifies that 32 candidate fragments are reduced to one planned fragment in every row-ID/deletion combination.

Validation

• cargo fmt --all -- --check
• cargo check -p lance --tests --benches
• cargo clippy --all --tests --benches -- -D warnings
• cargo test -p lance limit_pushdown -- --nocapture (15 passed)
• Parameterized correctness and fragment-pruning tests: stable/non-stable row IDs x with/without deletions
• Stable-row-ID update/compaction and metadata-fallback coverage
• Existing V2 filtered-read scan-range/deletion coverage

Addresses #8850.

[BubbleCal]

Please unify the V2 LIMIT/OFFSET planning path across stable and non-stable row IDs, with and without deletions. Fragment pruning only needs scan order and live-row counts; it does not depend on row-ID encoding. The current stable && has_deletions branch also leaves append-only scans loading metadata for all fragments before trimming.

Please move live-row counting out of the row-ID branch as well, reusing the loaded physical count and deletion vector for ordinary scans. Keep the branches for ID mapping, preserve existing fallback behavior when metadata is unavailable, and leave V1 unchanged. Parameterize the correctness and fragment-pruning tests across both row-ID modes and deletion states.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The generalized V2 planner now uses one validated live-row prefix for stable and address row IDs, with or without deletion vectors, while keeping row-ID mapping separate and retaining the prior V1 or incomplete-metadata fallbacks. Parameterized correctness and structural-pruning coverage exercises all four V2 combinations, making this shared mechanism preferable to the narrower stable-only branch without introducing a new format contract.

[LeoReeYang]

Addressed in 1029a8f2a. V2 LIMIT/OFFSET planning now uses the same metadata live-row pruning path for stable and non-stable row IDs, with and without deletions. FilteredReadExec computes visible rows once from the loaded physical count and deletion vector, while the two row-ID mapping branches remain separate. V1 behavior and the metadata-unavailable fallback are preserved. I also parameterized both correctness and structural pruning tests over all four row-ID/deletion combinations; the targeted suite passes 15/15, and full clippy passes with warnings denied.

[BubbleCal]

LGTM, @wjones127 could you take a look?