an appender: rows in columns, a batch to a commit

[tamnd]

A statement writes one row at a time and commits each one, which is the wrong shape for loading data into a database that already exists. load is the right shape for one that does not exist yet and this is the right shape for one that does: rows go into per-column buffers in memory, and a flush turns the whole buffer into one commit.

with conn.appender("person") as rows:
    for uid, name in enumerate(names):
        rows.append_row([uid, name])

The columns come from the table and not from the first row. An appender that read its shape off the row it was handed first would believe a wrong row and then refuse every right one after it, and its messages would name a column by number because that is all it would know. Reading the property directory at open costs one catalog read and buys a message that names the column, its type and the table.

The engine's appender borrows the connection for as long as it lives, which is a promise a Python object cannot make, so this one buffers on this side and opens an engine appender for the length of a flush. That is a catalog read per flush against a commit and a fold that cost time proportional to the table, so it is not where a load spends its time. What it buys is an appender that can be held in a variable, passed to a function and closed by a with block.

The GIL is released for the flush, and the buffer's own lock is taken after it is released rather than before: a lock waited for with the GIL held stops every other thread in the process for the length of the wait. That is the DX2 line about the GIL around an appender flush, and there is a test that counts the turns the main thread gets while 50,000 rows go in.

Closing flushes, including on the way out of a block that raised. The Rust appender flushes when it is dropped for the same reason: a load that stopped partway is better served by its rows arriving than by them vanishing, and a caller who wants the other answer writes discard() and gets exactly it.

An edge to a row that is not there is refused by the flush rather than left to the fold. The fold's refusal arrives once the write is durable, and the frame it leaves behind is refused again by every writer that opens the database afterwards, so one bad edge makes a database nobody can write to. That is an engine bug and it wants fixing there too, at the ingest that writes the frame; this refuses the batch before anything is written, so the file is untouched either way. The row counts are read at the flush and not at the open, so an edge to a row another appender wrote a moment ago is a good edge.

The Python to column conversion moves to buffer.rs, since the loader and the appender both do it and only differ in what settles the type: the table when there is one, the first value when there is not. The loader keeps its one widening, a column of integers that meets a float, because nothing there has said what the column is. The appender has been told and refuses it. A load of a column of bytes is refused where it starts rather than written, because the store takes one and no statement can read one back yet.

Numbers on this machine: 200,000 rows of an integer and a string take 11 ms to buffer through append_rows and 93 ms to flush, which is 1.9 million rows a second including the commit. Against INSERT, 2,000 rows take 32 seconds a row at a time and 28 ms through an appender, and the gap widens with the table because every INSERT is a commit and a fold.

Thirty-three tests: the rows arriving and reading back as themselves for every type a column holds, the refusals landing on the row that caused them, a refused row leaving the buffer a rectangle, a refused batch leaving a database that still opens, two threads appending to one appender, the main thread's turns during a flush, and the comparison against INSERT with a gate at twenty times. The whole suite is green locally, and so are cargo clippy, ruff check and ruff format.