Skip to content

fix: return actionable errors for binary/string offset overflow - #9403

Open
wjones127 with Copilot wants to merge 15 commits into
mainfrom
copilot/fix-offset-overflow-errors
Open

wjones127 with Copilot wants to merge 15 commits into
mainfrom
copilot/fix-offset-overflow-errors

Conversation

Copilot AI commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Reading or compacting string / binary data could still hit Arrow offset-overflow panics when a single batch exceeded 2GiB, leaving users with an opaque failure and no clear mitigation. This change converts those cases into normal Lance errors with guidance to reduce batch size or move to large_string / large_binary.

  • Decode overflow handling

    • Add explicit size checks before narrowing offsets in legacy binary/string decode paths.
    • Return NotSupported errors instead of allowing Arrow internals to panic on oversized batches.
    • Preserve existing Arrow errors for non-overflow failures (for example, invalid UTF-8 or malformed offsets).
  • User-facing diagnostics

    • Replace the generic “offset overflow” / “too many lists” style failure with actionable messaging for binary-like columns.
    • Point users to the two real mitigations:
      • lower batch_size / LANCE_DEFAULT_BATCH_SIZE
      • convert the column to large_string / large_binary
    • For list decoding, clarify which prefix of rows fits and which requested range overflows.
  • Compaction-adjacent behavior

    • Ensure the same decode-time protection applies when compaction reads string/binary data through these paths, so the failure is surfaced as a normal Lance error instead of a panic.
  • Representative error shape

    Could not create array with more than 2GiB of string/binary data in a single batch.
    Please reduce the batch_size, set LANCE_DEFAULT_BATCH_SIZE to a smaller value,
    or convert the column to large_string/large_binary.
    

Copilot AI and others added 13 commits September 18, 2026 16:36
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix offset overflow errors for binary and string columns fix: return actionable errors for binary/string offset overflow Sep 18, 2026
Copilot AI requested a review from wjones127 September 18, 2026 17:11
@github-actions github-actions Bot added the A-encoding Encoding, IO, file reader/writer label Sep 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

Comment thread rust/lance-encoding/src/array_encoding/logical/binary.rs Outdated
Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Copilot AI requested a review from wjones127 September 18, 2026 18:39
@github-actions github-actions Bot added the bug Something isn't working label Sep 18, 2026
@wjones127
wjones127 marked this pull request as ready for review September 18, 2026 20:16

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

Both decode paths now reject oversized normalized spans before narrowing offsets, while fallible Arrow construction preserves unrelated failures. This keeps normal decoding and the storage format unchanged, and avoids the fixed-length UTF-8 cost reported by the always-64-bit alternative.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Offset overflow errors can be confusing for users

2 participants