[Epic] PyIceberg table encryption

[kevinjqliu]

Goal

Add support for reading and writing tables using Iceberg's encryption model, interoperable with other engines.

Deliver this incrementally in PyIceberg. Reuse Rust implementations through pyiceberg-core when the necessary bindings are available, without making complete Rust-backed readers and writers a prerequisite.

This tracks Iceberg table encryption, not object-store server-side encryption such as S3 SSE-KMS.

Approach

• Keep Python's public APIs, catalog commits, and provider integrations.
• Define narrow interfaces so individual Python implementations can later delegate to Rust.
• Keep key management and encrypted I/O reusable across manifests, manifest lists, and data files.
• Deliver encrypted reads before requiring the full encrypted-write path.

1. Models, configuration, and key-metadata codec

• Add encryption fields to table and snapshot metadata: apache/iceberg-python#1972.
• Add StandardKeyMetadata: apache/iceberg-python#3948.
• Complete configuration validation and supported-version checks.
• Delegate the key-metadata codec to Rust while retaining the Python class.

Rust prerequisite: merge and release apache/iceberg-rust#3206, then establish the required pyiceberg-core version or capability check. This step does not need file I/O, KMS, or manifest bindings.

2. Key management

• Define a KMS provider interface for wrapping, unwrapping, and optional server-side key generation.
• Add provider integration and a deterministic test provider.
• Add an encryption manager for per-file keys/AAD, key resolution, wrapping, reuse, and caching.

Depends on: step 1.

Rust option: bind EncryptionManager and add a Python KMS provider adapter. Rust can own Iceberg's key-management mechanics while Python keeps cloud-provider integration. Define callback, concurrency, and error behavior before switching implementations.

3. Encrypted streams and file access

• Add encrypted input/output compatible with PyIceberg's file interfaces.
• Support the required sequential and range reads, length handling, buffering, and write finalization.
• Define ownership, close, cancellation, and authentication-failure behavior.

Depends on: steps 1 and 2.

Rust option: expose Rust's encrypted input/output through a storage adapter. Python's seekable streams must be adapted to Rust's asynchronous range-read/write interfaces. Complete native manifest or data-file bindings are not required.

4. Data files and delete files

• Configure PyArrow's Parquet encryption/decryption using the selected per-file key and AAD.
• Carry key metadata between file operations and DataFile records, including newly rolled output files.
• Cover the delete-file formats PyIceberg supports, including deletion-vector/Puffin handling where applicable. Track unsupported write formats separately.

Depends on: steps 1 and 2; encrypted-stream consumers also need step 3. Parquet integration can proceed alongside step 3 because Parquet uses its own encryption mechanism.

Rust option: reuse the codec and encryption manager while retaining PyArrow. Replacing entire data-file readers/writers is separate work requiring native bindings, Arrow batch transfer, and equivalent schema, writer-option, and output-metadata behavior.

5. Manifests and manifest lists

• Read encrypted manifests through the existing Python manifest APIs.
• Read encrypted manifest lists by resolving the snapshot's key reference.
• Add encrypted writes, carrying manifest key metadata in manifest-list entries and wrapped manifest-list key metadata in table key entries.

Depends on: steps 1-3. Read support can land independently; encrypted V3 writes also require the writer/version support tracked in step 6.

Rust option: Python readers/writers can use Rust-backed encrypted streams first. Moving complete operations into Rust additionally requires complete manifest and manifest-list bindings, version support, metadata conversion, and stable Python errors. Today's byte-parsing helpers are not that full integration.

6. Table writes, snapshots, and commits

• Complete the V3 write support needed for encrypted tables: apache/iceberg-python#1551.
• Wire encrypted file operations into table reads and writes.
• Commit new key entries together with their snapshot and file references.
• Handle retries, failed commits, and orphaned-file cleanup without losing key references.

Depends on: the relevant file integrations from steps 4 and 5. Start V3 write prerequisites in parallel with the earlier steps.

Boundary: keep catalog transactions in Python. Rust-backed operations must return the file information and key entries Python needs to commit; completing a file write is not a table commit.

Completion criteria

• PyIceberg reads encrypted tables produced by another Iceberg implementation.
• Another implementation reads encrypted tables written by PyIceberg.
• Supported data/delete files, manifests, and manifest lists work together end to end.
• Existing unencrypted behavior remains unchanged.
• Missing keys, wrong keys/AAD, unsupported capabilities, and interrupted operations fail clearly without silently falling back to plaintext.

Add focused coverage with each step, rather than leaving all integration work until the end.

Optional follow-up: complete native readers and writers

Full Rust-backed manifest and data-file I/O can follow independently. Where Rust performs the next operation, keep decoded keys and encryption state native rather than round-tripping through Python.

This is not a prerequisite for shipping Python encryption support.

Related work

• Existing encryption tracking: apache/iceberg-python#3222.
• Python V3 tracking: apache/iceberg-python#1818.
• Rust encryption tracking: apache/iceberg-rust#2034.