Add Arrow IPC stream output (--format ipc) - #45
Merged
Merged
Conversation
Add `--format ipc` on the row-producing commands (cat/head/tail/take/ sample), writing the Arrow IPC streaming format to stdout via `arrow::ipc::writer::StreamWriter`. The path bypasses value rendering entirely — batches go straight from the scan to the writer — so output is lossless and fully streaming at constant memory. IPC slots into the existing `RowWriter` seam (start(schema) → write_batch → finish), which is already batch-level, so command code is unchanged; the new `IpcRowWriter` defers `StreamWriter` construction to `start`. Central validation in `dispatch` (before any output): reject IPC on metadata/summary commands, reject the inapplicable value-rendering flags rather than silently ignore them, and refuse to write the binary stream to a terminal via a pure, unit-tested guard predicate. Round-trip, empty-stream, TTY-guard, flag-rejection, and command-rejection tests; README + CHANGELOG. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reviewer-predicted E0004 unions: the generic two-dataset diff (#13) rejects --format ipc like its other non-jsonl formats, and command_rejecting_ipc gains Blob (#22) plus a defensive Completions (#14) arm. tests/ipc.rs initializers gain no_progress (#14), Diff::other (#13), and Cat row_ids (#21). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
--format ipc: lossless Arrow IPC streaming to stdout for the row-producing commands (cat,head,tail,take,sample) viaStreamWriterat constant memory, composing arrs with DuckDB (read_arrow), pyarrow, Polars, and ADBC tools with zero fidelity loss. Refuses a terminal stdout, rejects the inapplicable value-rendering flags, and is rejected on metadata/summary commands. pyarrow + DuckDB interop verified by the reviewer.Rebased onto current main over #13/#22/#14/#21 exactly as the reviewer pre-identified: two textual unions (dispatch IPC validation +
show_progresscall; README examples) and the two compiler-enforced E0004 unions —Some(Format::Ipc)rejection in the generic two-dataset diff (#13) andBlob/defensiveCompletionsarms incommand_rejecting_ipc(#22/#14) — plus post-cut initializer fields in tests/ipc.rs.Post-rebase interaction check with #21:
head --with-row-id --with-row-addr --format ipcdecodes via arrow's StreamReader with_rowid/_rowaddras trailing UInt64 columns — pseudo-columns flow through the IPC stream.Gates: fmt/clippy clean, 337 tests / 0 failures.
Closes #12
🤖 Generated with Claude Code