read a frame where it lies

[tamnd]

register() no longer copies. It reads the pointers out of the Arrow arrays the caller handed over, describes them to the engine, and keeps the batch alive for as long as the engine holds the name. A statement that matches the name builds vectors that point straight at the caller's buffers, so nothing is moved and nothing is stale.

This is the client half of the frames work. The engine half landed in tamnd/zu#357, which added the C ABI entry point, and the rev in Cargo.toml moves up to it.

What it costs now

Registering ten rows costs 2.6 us. Registering ten million costs 2.3, because the work is describing columns and not moving bytes. A string column is walked once at registration so that a later read cannot fail on a bad offset, and that walk runs at 362 us per million rows.

Scanning a frame is faster than scanning the stored table, since the vectors point at memory that is already warm: 662 us against 833 summing a column of a million rows, and 1.8 ms against 3.1 grouping a million rows by a string.

What is still copied

Two things, and the README says both out loud. A stream that arrives in more than one batch is concatenated, because a frame is one set of buffers. A dictionary of Python lists is read into buffers of this client's own, because a list holds objects rather than numbers and there is nothing to point at.

What changed in the semantics

A frame is a view and not a snapshot. Write into the array behind it and the next statement answers what is there now. There is a test that sums a column, writes into the bytearray the column was built on, and sums again to a different answer.

The rule that a name kept the columns it was first used with is gone, because there is no longer anything cached that could disagree with a new shape. Registering a name again replaces what it stands for, whatever shape the new frame has.

A frame with no rows registers and matches nothing, which is what an empty table does. It used to be refused. The engine's checks all loop over the rows, so a frame with none of them dereferences nothing, and there was no reason left to refuse it.

Writing to a registered frame is refused by the engine with the reason, and unregister hands the bytes back when the last statement reading them has finished with them. There is a test that unregisters and then resizes the bytearray, which only succeeds once the export is gone.

The unsafe part

Column and Layout hold raw pointers and are not Send, so the description travels into the detached call inside a type that says who keeps those pointers alive. Frame::new runs in there with the GIL down. The held batch is dropped with the GIL taken back, because releasing imported Arrow buffers can decref Python objects, and that must not happen on a thread that does not hold it.

One thing worth knowing about arrow-rs turned up here. The FFI import can fold a sliced string array's row offset onto the buffers, leaving offset() at zero with a nonzero first offset, so the skew check looks at both. Undoing the skew needs concat with an empty slice in front, because concat of a single array hands that array straight back, which is right for a concatenation and wrong when the copy is the point.

Tests

32 in tests/test_register.py, all passing. New ones cover the sliced column, the write-through, the bytes going back, the frame belonging to its connection, the different shape under the same name, writes being refused, and registering costing the same whatever the frame holds.

Local gate is green: ruff check, ruff format, cargo fmt, cargo clippy -D warnings, and the full pytest suite in a release build.