DX4: Go, Java, and the ecosystem

[tamnd]

Milestone DX4 of the developer-experience program (Spec/2064g/dx/00-overview.md). Specs: dx/07, dx/08, dx/09 §2, dx/13 §1, §3. Runs with M4 (#5).

Go and the JVM, which between them are most of the server-side install base, plus the Arrow path that makes every other tool able to talk to us without a driver.

The JVM baseline moved as a result of the 2026 audit (AUDIT-2026-08.md §2.4). FFM was finalised in JDK 22, which is what the spec originally keyed on, but 22 has been EOL since September 2024; the current LTS is 25. More importantly, from JDK 24 a downcall from a module without granted native access warns and is on a path to becoming an error, and JNI is restricted on the same schedule, so both artifacts need the manifest attribute and the documented flag. A JVM warning printed three frames from any of our code is not a diagnosis a user can act on, so the binding detects the ungranted state at Database.open and throws a message containing the exact flag.

Go SDK v1 (dx/07):

• github.com/tamnd/zu-go, go 1.26.6 floor
• context.Context first on every blocking call, error last, never a panic
• rows.All() iter.Seq[Row], zu.Collect[T], zu.Iter[T]
• errors.Is/errors.As with *zu.Error carrying Code, Position, DocURL, Retryable
• database/sql driver as the zusql subpackage, with its limits stated
• vendored static libs per platform under build tags, cross-compilation recipes including the zig cc one
• zulint analyzer: columnar view after Close, missing rows.Err(), *zu.Conn shared across goroutines

Java SDK v1 (dx/08):

• Panama provider targeting JDK 25 LTS, JNI provider for 17-21
• Enable-Native-Access manifest attribute, the documented --enable-native-access flag, and a clear throw when it is missing
• sealed value types and records, Stream results, AutoCloseable everywhere
• JPMS module, Maven Central publishing, GraalVM reachability metadata
• CI on 17, 21, 25, 26
• bulk loader and appender, columns handed over without a copy when they come as direct buffers

C++ wrapper (dx/09 §2):

• header-only, RAII, ranges, an std::expected mirror of the error model

Arrow (dx/13 §3):

• Arrow export in all five SDKs: the C Data Interface in Rust, Python, Go and the JVM, IPC bytes in Node because JavaScript has no importer for the C one
• ADBC driver v0 over ArrowArrayStream

language server (dx/13 §1):

• zu lsp --stdio shipped in the CLI binary: diagnostics, completion, hover, formatting, semantic tokens
• VS Code extension bundling it

corpus:

• complete at ~1230 cases: graph values, concurrency, Arrow

done when:

• five tier-1 SDKs, each with a scorecard and a named maintainer
• the differential suite is green across all five
• the LSP gives catalog-aware completion in VS Code

[tamnd]

The ADBC driver v0 line is done, in #427.

crates/zu-adbc is an ADBC 1.1.0 driver over zudb::Database and zudb::Connection. It builds a cdylib a driver manager opens by name, ZuDriverInit is its entrypoint, and an Arrow stream is the only thing it hands back. A program in Python, R, Java, Go or C++ that already has a driver manager now has a zu client, with no package of ours in its dependency list and with nobody writing one. That is what this milestone meant by the Arrow path that makes every other tool able to talk to us without a driver, and it arrived before the Go and Java SDKs rather than after them because it costs one crate.

It costs one crate because the two halves already meet. ADBC is the one database interface whose result is Arrow rather than rows, and this engine's result has been columnar since the sink stopped flattening, so the batches it returns are the buffers zu-arrow builds for the Python and Node clients, cut where the caller asked, and no row is built anywhere on the path. The forty function pointers of AdbcDriver are laid out by Apache's adbc_ffi rather than by us, since a field written into the wrong slot of that struct is a segfault in the caller's process and not a failed test here.

The condition model survives the crossing, which is the part a binding usually drops. GQLSTATUS is five characters and ADBC's sqlstate is five characters, so a condition goes across as itself, the ADBC status comes off its class, and what ADBC has no field for goes into the detail keys 1.1.0 added: the code again, the doc URL, whether retrying is sensible, the line and column, and the excerpt. From Python a syntax error is a ProgrammingError whose sqlstate reads 42002 and whose details carry https://zu.dev/docs/errors/42002.

The Python program in docs/clients/adbc.md was run against the driver rather than written for the page, with adbc_driver_manager 1.12.0 and pyarrow 25.0.1 on Python 3.14.6, and fetchall, cursor.description, fetch_arrow_table and adbc_get_table_types all answer. Twenty-eight unit tests cover the Rust traits and eight in tests/c_abi.rs go the whole way through a driver manager, which is the half that matters, because every caller this crate is for arrives that way and none of them link it. Those eight pass on Linux, macOS and Windows in this repository's own matrix.

Two things it refuses that are worth recording here rather than in a file nobody opens. get_objects and get_table_schema both have to name a table's property columns, the columns are in the catalog with a LogicalType each, and nothing in this tree maps a LogicalType to an Arrow DataType yet, so v0 says so instead of guessing in the one place those calls are read, which is a schema browser. That mapping belongs in zu-arrow and is the first thing v1 owes. Partitions refuse and will keep refusing, since reading a result in partitions is for a database spread over several machines and an embedded one is a file in this process.

The other line in the Arrow section, the C Data Interface export in all five SDKs, is where this goes next along with the SDKs themselves: Python has it, Node has the buffers and owes the interface, and Go, Java and C++ owe both because they do not exist yet.

[tamnd]

Go client core is in: tamnd/zu-go#1, merged.

Four of the seven Go lines are done. github.com/tamnd/zu-go with the 1.25 floor and the 1.26.6 toolchain, context first and error last on every call with no panic anywhere, rows.All() as an iter.Seq[Row] beside Collect[T] and Iter[T], and an error model where a Status is itself an error so errors.Is(err, zu.Conflict) needs no unwrapping and an *Error carries the condition, the standard's own words, the doc URL, the severity, the retryable flag, the position and the source line the position points into.

Two decisions worth writing down, since the other C hosts will face both.

Cancellation is one goroutine for the length of one statement, started only for a context that can actually be cancelled, and waited for before the call returns so that an interrupt meant for this statement cannot land on the next one. The failure answers errors.Is against both context.DeadlineExceeded and zu.Interrupted, because the caller who wrote the deadline and the caller handling the failure are asking different questions of the same error.

Both handles hold a read-write lock, read-locked by every call that touches the handle and write-locked by Close, so a close waits for the calls already inside the engine rather than freeing under them. It deliberately does not serialise those calls: two goroutines on one connection is a mistake the engine has to be the one to report, and ZU_MISUSE_CONCURRENT with nothing done is a better answer than a silent queue.

On performance, one finding that applies to every cgo binding anybody writes over this ABI. A local variable whose address is handed to a cgo call escapes to the heap, so every accessor that writes through a pointer costs an allocation per call, and reading one integer cell costs two. Ten thousand rows scanned a row at a time allocated twenty thousand times for values that died in the loop body. Moving the out-parameters into fields of the result, and taking the struct field addresses once per result rather than once per row, took that to one allocation for the whole loop:

Read, ten thousand rows on an M4	Before	After
Scan into *int64	1.13 ms, 20001 allocs	0.72 ms, 1 alloc
Collect[int64]	1.01 ms, 30001 allocs	0.84 ms, 2 allocs
Int64s, the whole column borrowed	68 us, 1 alloc	28 us, 1 alloc

The columnar path is about twenty five times faster than reading a row at a time, and what is left in the row path is the cgo call floor at three calls per cell.

60 tests, green under the race detector, plus seven runnable doc examples and eight benchmarks. CI builds libzu from this repository at head, stages a pkg-config prefix with xtask package, and runs vet, the suite, the race detector and the benchmarks on Go 1.25 and 1.26 across Linux and macOS.

Three things this turned up here rather than there.

There is no way to name a node's table over the C ABI, so a node value can only carry the numeric table id. That is a gap for every C host, not just Go: a user who asks for a node and gets node(0:17) has been handed an internal row number and the name of nothing. Worth a zu_conn_table_name before the C++ wrapper and the Panama binding both work around it separately.

A statement that fails at the end of its input carries no position, so RETURN 1 + reports 42001 with nowhere to point. RETRUN 1 reports 42001 at 1:1 with the line to underline, which is the shape the rest should have.

A nested UNWIND notices the interrupt near the end of its scan rather than at the next chunk, so a 20 ms deadline on a 100 ms statement returns at about 120 ms. Same family as ORDER BY never checking the interrupt at all.

Next on this milestone, in the order the checklist lists them: the zusql subpackage with its limits stated, then the vendored static libraries per platform, then the zulint analyzer.

[tamnd]

tamnd/zu-go#2 is merged, and the database/sql line of the Go section is done.

zusql registers under the name zu. The connection string is the path, or :memory:, followed by create, read_only, threads and memory_limit, and anything else in it is refused at sql.Open with a message naming the four that are real. sql.Open opens nothing, which is what database/sql promises, so the database is opened on the first connection and the failure is kept: a bad path fails the same way on the tenth connection as on the first rather than being retried by every caller. Every connection from one sql.DB shares one zu.DB, which is what makes :memory: a database the pool can hand out rather than a new empty one per connection.

The three limits are in the package doc's first screen rather than left to be discovered. A statement that writes reports nothing, because there is no rows-affected count and no last-insert-id anywhere in the C ABI, so sql.Result answers both with an error naming which one is missing rather than a zero somebody reads as a count. Parameters are named and never positional, so every argument is a sql.Named and a positional one is refused with a message that says what to write instead; NumInput is -1 for the same reason, since a statement may mention one parameter three times and a count would be a number database/sql checks the argument list against and rejects correct calls over. A value that is not a number, a string, a bool or an instant comes back as itself, and database/sql passes it through to Scan unchanged.

What is not lost: the pool, the context handling, the transactions, and every error, which arrives as the same *zu.Error the client raises, so one errors.As covers the driver and the engine together. CheckNamedValue takes every value through untouched, which is the point of implementing it, because without it a time.Time would arrive having lost its zone and an int32 would arrive as an int64, and the bind in the client is the one place that should know what the engine takes. BeginTx refuses an isolation level the engine does not have rather than quietly giving the one it does. ResetSession rolls back a transaction the caller lost so the next caller out of the pool does not inherit it. Underlying, through sql.Conn.Raw, reaches the connection for Interrupt, RowsRead and the columnar reads, which database/sql has no word for.

ColumnTypeScanType needed a cell read without moving the cursor, since sql.Rows.ColumnTypes is asked before the loop starts. That is Rows.RowAt in the client now: the result is an array and not a cursor, so reading it out of order or looking at one row twice is something the shape allows. The row index is checked in Row.cell, where the column is known and the message can name it, and Row.Type goes through the same check. A column here is not declared, since this language types values and not columns, so the type is read off the first row and a result with no rows says any.

The extra interface costs 0.6 microseconds and 8 allocations on a single-row query, and about three quarters of an allocation per row on a thousand-row one, on an M4. The per-row part is the driver.Value boxing, which is any in the interface and which no driver can avoid.

31 tests and 5 runnable examples for the driver, 2 more in the client for the index, gofmt, go vet, go test and go test -race green on both packages.

Next on this milestone, in the order the checklist lists them: the vendored static libraries per platform under build tags with the cross-compilation recipes, then the zulint analyzer for the columnar view used after Close, the loop that never reads rows.Err(), and the *zu.Conn shared across goroutines.

[tamnd]

The Go client links its own library now, which was the sixth line of the Go SDK section. go get github.com/tamnd/zu-go and go build need no Rust toolchain, no pkg-config and nothing installed on the machine. tamnd/zu-go#4.

The shape is duckdb-go-bindings', read end to end at v0.10505.0 rather than from memory, with three deliberate departures. lib/<goos>-<goarch> is five modules inside the client's own repository, each holding its libzu.a, a REVISION naming the commit of the engine it came from, and a NATIVE_STATIC_LIBS naming what rustc said that build needs beside it at link time. The root imports the one that matches for its cgo directives and for nothing else, which is how a linker flag written in one module reaches a link started in another.

The three departures. The build tags are named for what they mean, zu_system for a libzu you installed and zu_static for an archive of your own in CGO_LDFLAGS, rather than duckdb's duckdb_use_lib and duckdb_use_static_lib, which name the mechanism and not the choice. There is one copy of zu.h in the root module instead of a copy in each of the five, so a header can no longer be stale in one platform and current in another. And a platform with no shipped archive is told so by name at compile time instead of by an undefined symbol at link time.

That last one took two files rather than one. With CGO_ENABLED=0 every declaration touching C is excluded, so the honest failure otherwise reads as a hundred undefined types and not as a build that turned cgo off. The file that says which it is is named so that it sorts before the first file that would produce that noise, because the compiler prints errors in file order and only the first few.

Worth writing down: the archive is 19 MB and not 60 MB. cargo build -p zu-capi builds all three crate types the crate declares and leaves an archive three times the size with the same 132 exported symbols in it. cargo rustc --release -p zu-capi --crate-type staticlib is what the packaging should have been using all along. Five of those in git history is 95 MB rather than 300 MB, which is a different conversation about whether they belong in the repository at all.

The libraries are built by a workflow on a runner whose own platform is the target, never by hand, and the two Linux ones inside a manylinux_2_28 container, because a static archive carries the glibc symbol versions it was compiled against and one built on the runner's own Ubuntu would refuse to start on half the distributions still in service. Windows is the gnu ABI and not msvc, since cgo drives a gcc-family linker there and cannot read an MSVC archive. The zig cc recipes the line asks for are in the README under Linking, along with the clang one for the other Mac and the note that cross-compiling to darwin from anything that is not darwin needs Apple's SDK and is the one direction none of it covers.

CI runs the shipped mode first and finishes in about a minute with no Rust at all. The engine build stays, for the other two modes and for one diff that holds the vendored header to the engine's, so a red job still means the client is wrong about the engine rather than that somebody had an old copy installed.

Also corrected the first line of this section, which said the floor was 1.25 with a pinned toolchain. It is a 1.26.6 floor now, in the client and in the conformance workflow, #430 and tamnd/zu-go#3.

Next is the last Go line, the zulint analyzer for the three mistakes that actually happen: a columnar view used after Close, a loop that never reads rows.Err(), and a *zu.Conn shared across goroutines.

[tamnd]

The Go SDK is done. tamnd/zu-go#5 landed zulint, and with it every Go line on this milestone is ticked.

The reason it exists is that the client's documentation states three rules a compiler cannot enforce. A columnar view is valid until Close and not one statement longer, which is the price of handing back the engine's own memory instead of a copy. A loop over a result ends the same way on the last row and on a failure, so rows.Err() is where the answer went. A connection is the one piece of state that cannot be shared, which is why a program that queries from four goroutines connects four times. A program that breaks any of the three compiles, passes review, and is a use-after-free, a swallowed error or a query refused under load. Documentation does not catch that, so a vet tool does.

viewafterclose asks the control flow graph rather than the order of the lines, so a use written above a Close and reachable from it counts, and a view returned out of the function that closed the result is reported as the same bug moved into the caller. rowserr asks only about a result the function itself made, because one that arrived as a parameter or is handed back to the caller belongs to whoever made it. connshare reports a connection two goroutines can reach and not one handed to a single goroutine, and it does not count Interrupt or RowsRead, which are the calls meant to be made from another goroutine while a statement runs and both take the read side of the lock, nor Close, which is lifecycle rather than work. A //zulint:ignore comment turns one off, because a test that proves a connection refuses a second goroutine has to give it one.

It is a module of its own so that golang.org/x/tools never lands in the graph of a program that only imports the client, and it carries a workspace of one, because a go.work shares a single module graph and the root workspace would drag in five library modules nothing has tagged yet. Its tests declare the part of the client's surface the checks read, with the real import path and the real method names, so the suite runs with no cgo, no library and no database at all.

CI builds it and runs it over the client on every push, which is the only honest way to ship a linter. Doing that found three loops in the client's own tests that never read rows.Err(), now fixed, and one test that provokes the refusal on purpose, which now says so.

Next on this milestone is the Java SDK v1, starting with the Panama provider on JDK 25 and the JNI provider for 17 through 21.

[tamnd]

The Java SDK now exists and reads and writes. Two pull requests in tamnd/zu-java, both merged.

zu-java#1 is the client itself. Three artifacts: dev.zudb:zudb is the API at Java 17 with no native code and no FFM type anywhere in its public surface, dev.zudb:zudb-ffm is the Panama provider at Java 25, and a ServiceLoader picks between them so application code never names one. The FFM artifact targets 25 rather than the 22 that finalised the API, because 22 has been out of support since September 2024 and shipping against an unsupported release only moves the problem somewhere harder to see.

The downcall handles are written by hand against zu.h rather than generated with jextract. Seventy functions with a stable shape do not need a generator, and the interesting decisions are all in the layer a generator would have written for us: which calls take Linker.Option.critical because they are short pure accessors, where the out-parameter scratch comes from so a query in a loop allocates nothing, and how a zu_error becomes a typed Java exception exactly once.

The columnar surface is the reason to embed a database rather than talk to one. Every column of a result is readable as a borrowed LongBuffer, DoubleBuffer or ByteBuffer over the engine's own memory, read-only and in native byte order, valid until the result closes. java.nio rather than MemorySegment on purpose: a Java 17 caller can name a LongBuffer and both providers can hand one back without copying. Summing one integer column of a hundred thousand rows costs 0.45 ns a row that way, 4.1 ns a chunk at a time, and 45 ns through the row iterator. A row at a time is a boundary crossing a cell, and a hundred crossings cost about what one borrowed buffer costs.

zu-java#2 is the two write paths, which between them are the only way values get into a database at all while the engine has no DDL. Loader builds one out of whole columns, Connection.appender(String) adds rows to a table that already exists.

Columns go in as arrays or as java.nio buffers, and which one you pass is the difference between a copy and no copy. A direct buffer reaches the engine through MemorySegment.ofBuffer with nothing crossing the boundary but a pointer. An array is copied into a confined arena first, because native code cannot address a Java array without either a copy or a pause. Linker.Option.critical(true) would let the array through without either, at the price of blocking the collector for the length of the copy, and on a column of a hundred million values that is not a trade worth making. Over a hundred thousand rows the direct buffer hands a column over in 0.44 ns a row against 1.1 ns for the array, which works out at 68 microseconds of memcpy for 800 KB.

An appended row costs 87 ns. The same row written as an INSERT statement costs 4.0 ms, because a statement per row parses, plans, runs and commits per row and none of that work says anything the row before it did not already say. Four and a half orders of magnitude is a large enough gap that the appender is not an optimisation, it is the only usable answer.

138 tests, green, and green again under -ea -esa. CI runs the API on 17, 21, 25 and 26-ea, and runs the whole suite plus the benchmarks against a freshly built libzu on both Ubuntu and macOS. One CI step reads ZU_ABI_VERSION out of the engine's zu.h and checks it against the constant in Zu.ABI_VERSION, because the ABI version is a header macro and a binding with no C compile step has nowhere else to read it from.

Three things found on the way, none of them fixed here.

zu_version() is hard-coded c"0.0.1" at crates/zu-capi/src/lib.rs:851 rather than env!("CARGO_PKG_VERSION"), so it will drift from the crate version and already reads oddly beside an ABI of 0.11. Worth either wiring it up or adding a zu_abi_version() runtime symbol, which would also let a binding stop hard-coding the macro.

zu_loader_table refuses an empty edge name, which the header does not say. The client now asks for both names and documents why, but the header comment should say it too.

MATCH (e:Event) WHERE e.id = 2 matches by row offset rather than by the id property, which is the known loose end and cost a test half an hour before it was recognised.

Still open on the Java line: the JNI provider for 17 to 21, the dev.zudb:zudb-native-{platform} artifacts, GraalVM reachability metadata, and Maven Central publishing. Frames and the progress callback are next in the client, both of which need an FFM upcall stub, which is the first thing in this binding that hands the engine a pointer to Java code rather than the other way round.

[tamnd]

zu-java #4 is merged, which is the second of the two upcalls the last comment said were next, and with it the JVM binding hands the engine a pointer to Java code in both directions it ever needs to.

A connection can now be asked to call back every so often while a statement runs, with how far it has got and how long it has been going, and the answer it gives decides whether the statement carries on. That is the interrupt and the rows read counter folded into one arrangement with no thread of the caller's doing the polling, so a timeout is a comparison against a deadline and a progress bar is the same comparison with a repaint in front of it. The arrangement belongs to the connection rather than to a statement, so it is set once when the connection is opened, and a statement already running keeps the one it started with.

The interesting part is the stub lifetime rather than the call. A stub the engine holds outlives the call that handed it over, and a stub cannot close the arena it is standing in, so the arena machinery moved out of the frame release callback into an Upcall type that both callbacks now use: a shared arena and a function pointer, spent onto a queue that the next stub to be made drains. The provider keeps one progress stub per connection and spends it when the arrangement is replaced, taken back, or the connection closes, and all three of those are calls of ours on the connection, which is exactly when the engine is not running a statement on it.

An exception crossing an upcall would take the JVM down, so a watcher that throws is logged and answered as a stop. A callback that threw is a program that has stopped wanting the answer, and running the statement to the end would only mean throwing the answer away later.

Ten tests over a nested loop of nine million pairs. The pair count is not for weight, it is because a scan of ten million rows in a registered frame finishes in ten milliseconds and there is nothing in that for a progress callback to report. Worth writing down for whoever benchmarks this next: MATCH (a:Person), (b:Person) RETURN count(*) is folded by the planner and answers four hundred million pairs in eleven milliseconds, while the same pattern with WHERE a.id < b.id over it is a genuine nested loop and is the only reliable way we have found to make the engine slow on purpose.

Next in this binding are the three things still unbound, which are zu_memory, zu_config_set and the _z convenience variants, and then the JNI provider for 17 to 21, which is the checklist line above that the Panama half of is done. Frames and the progress callback are owed to zu-python, zu-node and zu-go on the same terms, since the JVM is only the first client to bind them.

[tamnd]

The JVM line about the module, Central and GraalVM is done, in three pull requests on zu-java.

The JPMS half was already there, so the work was the other two. The one that mattered most for a user is that adding a dependency now gets you an engine. There is a zudb-native artifact, Library.find looks for one on the class path between ZU_LIBRARY and the platform's own search, and a library inside a jar is copied out to a file once per process because no loader anywhere can map a resource. Eight jars come out of one staged directory so they cannot disagree: one holding all seven platforms for a program built on a laptop and deployed to a cluster, and seven of about three megabytes each for a container image that knows what it runs on. Alpine is one of the seven rather than a smaller Linux, since a shared object built against glibc does not load on musl and fails talking about an interpreter rather than about a database, and which of the two a JVM gets is decided by looking for musl's own loader, the one path the ABI fixes rather than a distribution.

Two things a user actually meets came out of building it. On the module path nothing requires that artifact, because there is no code in it to require, so a jar sitting on the path is never resolved and its library is invisible: the search notices it is on a module path and names --add-modules dev.zudb.natives instead of leaving somebody to work out why the same run on a class path worked. And the module name derived from the file name would have been zudb.native, which is not a legal module name because native is a keyword, so every jar carries Automatic-Module-Name in its manifest.

Publishing is one upload for the whole reactor rather than one per module, since Central validates a deployment as a unit and a half-published version is one nobody can use and nobody can replace. Auto-publish is off, so a tag produces a deployment waiting in the portal and dropping it there is the only undo that exists. The version lives in the tag rather than in the pom, because a repository whose version has to be bumped before every release is one where a checkout between the bump and the tag claims a version that was never built. The engine and this client move on one number, so the tag also names the engine release the libraries come from, and the suite then runs against the very library that is about to go out.

The reachability metadata is the part that is easy to get quietly wrong. An image has no linker in it: every downcall stub is machine code written while the image is built, and the builder can only write the ones it was told about, because a function descriptor is assembled out of ordinary objects at run time and there is nothing in the bytecode to read it off. Forty-odd entries maintained by hand with one of them stale is exactly the failure that produces an image that builds clean and dies on the first query. So the list is not maintained, it is derived: every binding registers its shape as it is made, a test writes out what that comes to, and the build fails if the file in the repository says something else, with the file that would have been right sitting in target. Adding a function to the C ABI either changes nothing, because its shape is one of the forty already listed, or fails loudly with the answer attached.

That test can only say the file matches the binding. Whether it is the file the builder wanted is a thing only an image can answer, so CI builds one on Linux and macOS and makes it run a query with no property, no environment variable and the library coming out of the image. The first one told me the spelling was wrong, which was the point of building it: the file names canonical layouts, so a pointer is void* rather than pointer, and a 64-bit integer is long long rather than long, because C's long is four bytes on Windows and eight everywhere else and this file is written once and read on all of them.

What is left on the Java line above is the JNI provider for 17 to 21, which is the half of that line the Panama side of is done.