feat: expose scanner blob handling - #81
Merged
jja725 merged 2 commits intoSep 17, 2026
Merged
Conversation
MisterRaindrop
force-pushed
the
feat/scanner-blob-handling
branch
from
September 10, 2026 10:22
7450446 to
c1ec76d
Compare
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
✅ Gate recommendation: approve.
This implements the scan-mode follow-up accepted in #76, preserves descriptor-by-default behavior, and keeps validation in Rust behind thin C/C++ wrappers. For bulk scans, the opt-in scanner policy is a smaller fit than a per-blob handle API, and its materialized-byte path covers inline, packed, dedicated, empty, null, and multi-fragment Blob v2 data.
lance-gatekeeper
Bot
added
the
K-approved
jja725
approved these changes
Sep 16, 2026
jja725
left a comment
jja725
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, can you resolve the conflict
MisterRaindrop
force-pushed
the
feat/scanner-blob-handling
branch
from
September 17, 2026 04:01
c1ec76d to
4a1dc17
Compare
lance-gatekeeper
Bot
removed
the
K-approved
Adds LanceBlobHandling and lance_scanner_set_blob_handling(), applied in build_scanner() right after project(). Values outside the enum and calls after the scan has started are rejected.
Two-fragment Blob v2 fixture (inline, packed, dedicated, empty and null rows) and six tests. The C and C++ smoke programs get a blob dataset as a third argument and check the setter too.
MisterRaindrop
force-pushed
the
feat/scanner-blob-handling
branch
from
September 17, 2026 04:13
4a1dc17 to
be21568
Compare
Contributor
Author
I’ve updated it |
jja725
pushed a commit
that referenced
this pull request
Sep 18, 2026
Builds on #81; the first two commits are that PR. Adds the row-addressed blob API from #76: `lance_dataset_take_blobs` (by `_rowid`) and `lance_dataset_take_blobs_by_indices` (by row offset) fill a caller array of `LanceBlobFile*`, NULL for a null value; the handle has `size`, `read`, `read_up_to`, `read_range`, `seek`, `tell` and `close`. Reads copy into caller buffers and return 0 / -1 with the thread-local error set. The handle wraps upstream `BlobFile` only, so it outlives the dataset. Take is all-or-nothing (`out` untouched on error), `read` rejects a buffer smaller than the remaining bytes instead of truncating, and an unknown column is an invalid argument rather than `LANCE_ERR_INTERNAL`. `lance.hpp` gets an RAII `BlobFile` and `Dataset::take_blobs*()` returning `std::vector<std::optional<BlobFile>>`. Tests: sixteen `test_blob_*` cases (both entry points over every storage layout, with and without stable row ids, cursor semantics, bounds and overflow, handles outliving the dataset, NULL and invalid arguments with `out` untouched) plus the C and C++ smoke programs. Open: `read` goes from the cursor to the end like the Rust and Java `read()`, and `seek` is not bounds-checked, like upstream. `take_blobs_by_addresses` and `kind` / `uri` accessors are not included.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
LanceBlobHandlingandlance_scanner_set_blob_handling()so a scan can return Blob v2 columns as bytes (ALL_BINARY) instead of the descriptor struct (#76). The value is applied inbuild_scanner()right afterproject(); values outside the enum and calls after the scan has started are rejected.lance.hppgetsScanner::blob_handling().Note: on lance v11.0.0
ALL_DESCRIPTIONSonly affects columns that carry blob metadata, so a plainBinarycolumn keeps its bytes and values 0 and 2 currently give the same schema. Documented in the header and pinned by a test.Tests: two-fragment v2.2 fixture (inline / packed / dedicated / empty / null blobs), six Rust tests, and the C and C++ smoke programs.