feat(io): accept block_size as a storage option - #9380
Merged
Merged
Conversation
github-actions
Bot
added
A-encoding
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-approved
block_size is the gap below which the I/O scheduler merges two reads of one file into a single request, and it is the knob for random access to rows of several KB spread across a file. It could only be set through the Rust ObjectStoreParams field or the Python dataset argument. It is now also read from the block_size storage option, resolved once when the registry builds a store so every provider honours it; the explicit parameter still wins and invalid values are rejected.
dshepelev15
force-pushed
the
perf/io-coalesce-gap
branch
from
September 18, 2026 11:13
d8ef101 to
eac27df
Compare
lance-gatekeeper
Bot
removed
the
K-approved
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The revision keeps resolution centralized at store construction and preserves the existing defaults and explicit-parameter precedence. Narrowing validation to malformed numeric strings aligns the storage option with the existing dedicated parameter while retaining consistent behavior across local and cloud providers.
lance-gatekeeper
Bot
added
the
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
block_sizeis the gap below which the I/O scheduler merges two reads of one file into a single request (4 KiB for local files, 64 KiB for object stores). It is the knob for random access to rows of several KB spread across a file: the same take of 2048 rows over 8 fragments of 2.0 files costs 1,352 requests / 143 MiB at 64 KiB, 589 requests / 243 MiB at 256 KiB and 220 requests / 420 MiB at 1 MiB (counted withio_stats_incremental).Today it can only be set through a dedicated parameter:
ObjectStoreParams::block_sizein Rust,lance.dataset(block_size=...)in Python,ReadOptions.blockSizein Java. Integrations that only forwardstorage_options(LanceDB, the Spark and Ray connectors, namespace catalogs, config-driven deployments) cannot set it, although every other store setting goes through that map.Change
StorageOptions::block_size()parses ablock_sizeoption given in bytes. Non-numeric values are rejected with anInvalidInputerror instead of being ignored.ObjectStoreParams::resolved_block_size(): the explicit parameter wins, then the storage option, then the store's default.ObjectStoreRegistry::build_store, so every provider honours it the same way, and in the customobject_storepath.The defaults do not change. The bindings pass
storage_optionsthrough untouched, so no binding changes are needed.Tests
test_block_size_used_cloud/test_block_size_used_fileextended: the option sets the block size when no parameter is given, and the explicit parameter wins over the option.test_block_size_option_rejects_invalid_values: a non-numeric value is rejected.