Skip to content

perf(scan): resolve deletions from row offsets instead of row addresses - #9373

Open
taozhiyi13 wants to merge 1 commit into
lance-format:mainfrom
taozhiyi13:perf/scan-fragment-deletion-overhead
Open

taozhiyi13 wants to merge 1 commit into
lance-format:mainfrom
taozhiyi13:perf/scan-fragment-deletion-overhead

Conversation

@taozhiyi13

@taozhiyi13 taozhiyi13 commented Sep 18, 2026

Copy link
Copy Markdown

Problem

Filtering a scan batch against a deletion vector does not need row addresses.

A deletion vector stores fragment-local offsets. The batch already knows which offsets it covers. The keep bitmap for filter_record_batch is just those two sets of offsets. The old path still materialized a u64 row address per row and probed the vector once per row, only to throw away the fragment id and recover the offset.

Where this sits

load deletion vector
  → plan live row ranges
  → read ranges into batches
  → apply_row_id_and_deletes   ← this change
       old: each row → row address → probe DV → keep bitmap → filter
       new: DV offsets ∩ batch offsets → keep bitmap → filter

The first three steps are unchanged. This PR only changes how apply_row_id_and_deletes builds that bitmap.

Change

The batch is still filtered with a keep bitmap. This PR only changes how that bitmap is produced.

apply_row_id_and_deletes already knows the batch's fragment-local offset ranges. DeletionVector::build_keep_mask intersects those ranges with the DV's offsets and writes hits onto a bitmap in batch-row order. That bitmap goes to filter_record_batch, same as before.

_rowid and _rowaddr are still attached in this same function when the caller asks for them. They are just no longer a prerequisite for deletion filtering. If this batch hits no deleted offset, skipping the bitmap is the same as filteringwith an all-true mask: every row is kept.

Test

cargo test -p lance-core --lib utils::deletion
cargo test -p lance-table --lib test_deletes

build_keep_mask is checked bit-for-bit against the old build_predicate, including a batch whose physical ranges are not one run (0..50, 60..110). test_deletes still covers _rowid and make_deletions_null.

Effect

Local to_table() on one 1M-row fragment. 00d87ac5 vs 9f463f63, pylance 13.0.0-beta.4, best of 10 after 3 warmups. Multipliers are vs the unpatched no-deletion scan (3.64 ms):

case deletes 00d87ac5 9f463f63
no_deletions 0 3.64 ms (1.0×) 3.95 ms (1.1×)
contiguous_deletions 10 in one run 19.45 ms (5.3×) 4.00 ms (1.1×)
scattered_deletions 10 spread out 19.24 ms (5.3×) 4.30 ms (1.2×)

Contiguous and scattered cost the same on the old path, so the extra ~16 ms was not IO. After this change both sit on the clean-fragment scan (about 4.5–4.9× faster than the old deletion path). Row counts and sum(id) matched across builds.

base/bin/python -m pytest python/python/benchmarks/test_deletion_scan.py
patched/bin/python -m pytest python/python/benchmarks/test_deletion_scan.py

A scan over a fragment with deletions probed the vector once per row and
materialized a u64 address for every row just to recover the offset.
Cost followed rows scanned, not rows deleted: 10 deletes in a 1M-row
fragment made a full scan ~3x slower.

Build the keep mask from the batch's offset ranges. Stop fetching row
addresses only because the fragment has deletions.
@github-actions github-actions Bot added A-python Python bindings performance labels Sep 18, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The range-driven keep mask addresses the actual hot path: its work follows deletions intersecting a batch instead of all scanned rows, without materializing row addresses unless requested. It preserves batch order for disjoint selections and retains the existing filter/null and system-column semantics. No blocking issues found.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings K-approved Latest Gatekeeper recommendation permits acceptance. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant