Bulk loading and appending

[tamnd]

The Java client could read anything and write nothing. This adds both of the C ABI's write paths, which between them are the only way values get into a database at all while the engine has no DDL.

The loader

Loader builds a database out of whole columns. create refuses a path that already exists, table names the one table and how many rows it has, and columns go in one at a time until finish writes the file.

try (Loader loader = Loader.create(Path.of("social.zu1"))) {
    loader.table("Person", "Follows", 3);
    loader.column("id", 1L, 2L, 3L);
    loader.column("name", "ada", "grace", "alan");
    loader.edges(new int[] {0, 1}, new int[] {1, 2});
    loader.finish();
}

The row count is declared rather than counted, so a column with a value missing is an error and not a shorter table. An edge table name is wanted even for a load that adds no edges, because the engine wants one, and naming it here rather than guessing a name later is the difference between a schema you chose and one that happened.

Zero copy, and where it stops

Columns go in as arrays or as java.nio buffers, and which one you pass is the difference between a copy and no copy. A direct buffer reaches the engine through MemorySegment.ofBuffer with nothing crossing the boundary but a pointer. An array is copied into a confined arena first, because native code cannot address a Java array without either a copy or a pause. Linker.Option.critical(true) would let the array through without either, at the price of blocking the collector for the length of the copy, and on a column of a hundred million values that is not a trade worth making.

A string column has no zero-copy shape at all and is not offered one. Every string is encoded and checked for UTF-8 on the way in, which is the price of never reading back a value no query could have returned.

The appender

Connection.appender(String) opens one on a table that already exists. Values go in a value at a time in declared column order and a row is a row once endRow has ended it.

try (Appender rows = conn.appender("Person")) {
    rows.append(4L).append("hedy").endRow();
    rows.append(5L).append("katherine").endRow();
    rows.finish();
}

There is no way to append no value, because the C ABI has none and inventing one here would only move the surprise. A value the column will not take ends its row there and rolls back the values already written into it, so a refused append never leaves half a row behind.

Closing an appender that was never finished writes what it has. A loop that threw halfway keeps the rows it managed, because throwing away work that succeeded is not a decision a close should make on its own. discard() is there for when it should be thrown away, and finish() is there for the caller who needs the count and needs to know the last write worked, which a close has nowhere to report.

Numbers

Per row, over a hundred thousand rows, M-series laptop on JDK 25.

How	Per row
loader.column(name, direct LongBuffer)	0.44 ns
loader.column(name, long[])	1.1 ns
loader.column(name, List<String>)	108 ns
a whole two-column load, write included	630 ns
appender.append(...).endRow()	87 ns
the same row as an INSERT statement	4.0 ms

The first two lines are the copy: 0.68 ns a row is 68 microseconds to move 800 KB, about what a memcpy costs and about what a direct buffer saves. The last line is the reason the appender exists, at four and a half orders of magnitude.

Tests

Twenty seven new ones, 138 in all and green, also green under -ea -esa. They cover every column kind through both the loader and the appender, buffer slices rather than whole buffers, edges appended across calls, a refused value leaving no half of a row behind, an appender and a query interleaved on one connection, and every misuse the client catches before the call rather than after it.

Two things the tests found and worked around rather than fixed, both in the engine: zu_loader_table refuses an empty edge name, which the header does not say, and MATCH (e:Event) WHERE e.id = 2 matches by row offset rather than by property, which is a known loose end.