You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4ecaf15
Browse filesBrowse the repository at this point in the historyBrowse files
Paul Mathew
committed
feat(2152): support pa.RecordBatchReader in Table.append/overwrite
Currently `Table.append(df)` and `Table.overwrite(df)` only accept a
materialised `pa.Table`, which forces callers to load the entire dataset into
memory before writing. This makes pyiceberg unusable for large or unbounded
inputs and has been a recurring complaint (#1004, #2152, dlt-hub#3753).
Allow `pa.RecordBatchReader` as an alternative input. When a reader is
provided, batches are streamed and microbatched into target-sized Parquet
files via the new `bin_pack_record_batches` helper, then committed in a
single snapshot via the existing fast_append path. Memory is bounded by
`write.target-file-size-bytes` (default 512 MiB) per worker rather than the
full input size.
Scope of this PR — unpartitioned tables only. Streaming into partitioned
tables raises NotImplementedError pointing back to #2152; partitioned support
needs additional design (high-cardinality partition handling, per-partition
rolling writers) and is tracked as a follow-up. Mirrors iceberg-go#369's
staging — that project shipped unpartitioned streaming first.
Internal note: the implementation buffers up to `target_file_size` of in-
memory RecordBatches before flushing to a Parquet file. A more memory-
efficient rolling-ParquetWriter approach is a planned follow-up that will
benefit from the `OutputStream.tell()` API added in #2998.
Copy file name to clipboardExpand all lines: mkdocs/docs/api.md
+11Lines changed: 11 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -365,6 +365,17 @@ for buf in tbl.scan().to_arrow_batch_reader():
365
365
print(f"Buffer contains {len(buf)} rows")
366
366
```
367
367
368
+
### Streaming writes from a `RecordBatchReader`
369
+
370
+
`tbl.append()`and `tbl.overwrite()` also accept a `pyarrow.RecordBatchReader` directly, which lets you write datasets that don't fit in memory without materialising them as a `pa.Table` first. PyIceberg consumes the reader once and microbatches it into Parquet files of approximately `write.target-file-size-bytes` (default 512 MiB), keeping memory usage bounded by the target size. All files are committed in a single snapshot.
Streaming writes are currently only supported on **unpartitioned** tables. For a partitioned table, materialise the reader as a `pa.Table` first, or follow [#2152](https://github.com/apache/iceberg-python/issues/2152) for the partitioned support tracked as a follow-up.
378
+
368
379
To avoid any type inconsistencies during writing, you can convert the Iceberg table schema to Arrow:
0 commit comments