Conversation
Dataset::versions() already knows each manifest's on-disk size from
ManifestLocation but drops it. Keep it on the Version struct (and expose
it in the Python versions() dicts) so metadata growth per version is
queryable:
ds.versions()[-1]['manifest_size']
This is pure bookkeeping - no format change. The field is Option since
manifest listing may not always provide a size (it is #[serde(default)]
for backward-compatible deserialization).
Motivation: on wide tables the full manifest is rewritten on every
commit; being able to chart manifest_size across versions is the
cheapest way to quantify metadata amplification.
github-actions
Bot
added
A-python
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
Addresses the two review requests on lance-format#9102: 1. Add `manifest_size: int | None` to the public `Version` TypedDict in `python/python/lance/dataset.py`. The binding already returns the key, but typed callers could not use it: Pyright reported "manifest_size" is not a defined key in "Version". The field is optional because `ManifestLocation::size` is `Option<u64>` and is documented as possibly unknown. Also documents the returned fields on `versions()`. 2. Extend `test_versions` to assert the returned `manifest_size` against the corresponding `_versions/*.manifest` file size, so both the Rust propagation and the binding key are pinned. Resolving the file needs to handle V1 (`{version}.manifest`), V2 (`{u64::MAX - version:020}.manifest`) and detached (`d{version}.manifest`) naming.
hiltonhe
changed the title
lance-gatekeeper
Bot
added
K-changes
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The public Python type contract and regression coverage now match the implementation. The change remains small and reuses manifest metadata already returned by the commit handler, preserving an unknown size as None.
lance-gatekeeper
Bot
added
the
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dataset::versions() already knows each manifest's on-disk size from ManifestLocation but drops it. Keep it on the Version struct (and expose it in the Python versions() dicts) so metadata growth per version is queryable:
This is pure bookkeeping - no format change. The field is Option since manifest listing may not always provide a size (it is #[serde(default)] for backward-compatible deserialization).
Motivation: on wide tables the full manifest is rewritten on every commit; being able to chart manifest_size across versions is the cheapest way to quantify metadata amplification.