Reject overlapping safetensors data offsets - #48
Open
glasses666 wants to merge 1 commit into
Open
Conversation
The safetensors format relies on tensor data offsets fully describing the byte buffer: offsets are relative to the payload, must be ordered without gaps or overlaps after sorting by offset, and the final end must match the payload size. h3 already checked each tensor shape against its own byte range, but it accepted layouts where two tensor ranges overlapped or left bytes unindexed. This validates the global offset layout once the header has been parsed and adds a regression with an out-of-order valid header plus overlapping tensor ranges. Constraint: Safetensors README requires the byte buffer to be entirely indexed and hole-free to prevent polyglot payloads. Rejected: Pairwise overlap-only scan | would still accept trailing unindexed bytes. Confidence: high Scope-risk: narrow Directive: Keep this check at header parse time so all safetensors consumers share the same trust-boundary validation. Tested: make test Tested: ASan/UBSan h3_tests Not-tested: Full model-weight parity tests requiring released fixtures not present locally
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
The safetensors format defines data_offsets relative to the byte buffer and requires the byte buffer to be entirely indexed, with no holes, to prevent polyglot files: https://github.com/huggingface/safetensors#format
h3 already checked each tensor shape against its own byte range, but a header could still make two tensors overlap or leave payload bytes unindexed. This keeps that validation at the header parser boundary so all safetensors consumers share it.
Tests