Skip to content

fix(table/dv): read deletion vectors by content_offset when the file is not a Puffin container - #2071

Open
sevbanbayrak wants to merge 1 commit into
apache:mainfrom
sevbanbayrak:fix/dv-read-by-offset
Open

sevbanbayrak wants to merge 1 commit into
apache:mainfrom
sevbanbayrak:fix/dv-read-by-offset

Conversation

@sevbanbayrak

@sevbanbayrak sevbanbayrak commented Sep 28, 2026 •

Copy link
Copy Markdown

Fixes #2070

Problem

Databricks writes deletion vectors for IcebergCompatV3 (UniForm) tables as a Delta deletion_vector_<uuid>.bin file: a one-byte version prefix followed by deletion-vector-v1 blobs, with no Puffin header or footer. The manifest entry is well-formed (file_format=PUFFIN, referenced_data_file, content_offset, content_size_in_bytes point at a valid blob; length, magic D1D33964 and CRC-32 check out). ReadDV/ReadDVs opened the file with puffin.NewReader and failed with puffin: invalid header magic.

The Java reference reader (BaseDeleteLoader.readDV) reads these tables because it never consults the Puffin footer: it reads content_size_in_bytes bytes at content_offset and deserializes the blob.

Change

  • puffin: NewReader wraps the header-magic and too-small errors in a new sentinel ErrNotPuffinFile so callers can detect "no Puffin container" with errors.Is. Error messages keep their existing text.
  • table/dv: when the DV file is not a Puffin container, ReadDV/ReadDVs fall back to reading the blobs directly at content_offset (readBareDVs), matching Java. DeserializeDV still verifies the blob's length, magic and CRC-32, and the decoded cardinality is validated against the manifest record_count. The footer-only checks (blob type, referenced-data-file property, cardinality property) cannot be performed and are skipped with a slog warning. Real Puffin files keep the existing strict path unchanged.
  • Tests: TestReadDVBareBlobWithoutPuffinContainer (single blob, two blobs in one bare file, cardinality mismatch, range beyond file); TestReadDVInvalidPuffin now asserts the direct-read error.

Verification

  • go test ./puffin/ ./table/dv/, go vet, make lint (0 issues).
  • Against a real Databricks Unity Catalog IcebergCompatV3 table (Iceberg format-version 3) read through the UC Iceberg REST catalog with vended credentials, after a 1,020-row DELETE and a 1,020-row UPDATE produced one DV (total-delete-files=1, total-position-deletes=2040): Scan().ToArrowRecords returns 968,980 rows with 119,960 / 1,020 rows carrying the updated markers, identical to the SQL result. Copy-on-write and non-DV tables on the same catalog are unaffected.

…is not a Puffin container

Databricks writes deletion vectors for IcebergCompatV3 (UniForm) tables as a
Delta deletion_vector_*.bin file: a one-byte version prefix followed by
deletion-vector-v1 blobs, with no Puffin header or footer. The manifest entry
is well-formed (file_format=PUFFIN, referenced_data_file, content_offset,
content_size_in_bytes point at a valid blob), and the Java reference reader
(BaseDeleteLoader.readDV) reads such files because it never consults the
Puffin footer: it reads content_size_in_bytes bytes at content_offset.

ReadDV/ReadDVs opened the file with puffin.NewReader first and failed with
"puffin: invalid header magic" on these tables.

- puffin: NewReader wraps the header-magic and too-small errors in a new
  sentinel, ErrNotPuffinFile, so callers can detect "no Puffin container".
- table/dv: when the DV file is not a Puffin container, fall back to reading
  the blobs directly at content_offset (Java parity). The blob's length,
  magic and CRC-32 are still verified by DeserializeDV and the decoded
  cardinality is validated against the manifest record_count; footer-only
  checks (blob type, referenced-data-file property, cardinality property)
  are skipped with a slog warning. Real Puffin files keep the existing
  strict path unchanged.

Verified against a Databricks Unity Catalog IcebergCompatV3 table read through
the UC Iceberg REST catalog with vended credentials: 968,980 rows after
1,020 deletes + 1,020 updates, matching the SQL result; copy-on-write and
non-DV tables unaffected.

Fixes apache#2070

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MqCR5xLuYcb7Baw6Xbj4be

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DV read fails on Databricks UniForm v3 tables: DV file has no Puffin container, only the deletion-vector-v1 blob at content_offset

1 participant