DX0: foundations, the repository split, the API model

[tamnd]

Milestone DX0 of the developer-experience program (Spec/2064g/dx/00-overview.md). Specs: dx/17 §DX0, dx/18, dx/02, dx/03, dx/14 §10, dx/15 §2. Runs with M0 (#1).

Nothing user-facing. This is the machinery every later SDK depends on, and the part of it that is genuinely irreversible is the repository boundary, so it is settled before any binding exists. The rule dx/18 §1 lands on, after auditing what DuckDB actually does: code that compiles against the engine's internals stays in this repo, code that compiles against the frozen C ABI leaves. That puts the Rust SDK, zu.h, the CLI, and the conformance corpus here, and puts Python, Node, Go, Java, .NET, the C/C++ kit, and the site in their own repositories. tamnd/zu-c deliberately does not contain zu.h, because the header is generated from the API model and is therefore an output of this build.

The second irreversible thing is the pinned toolchain table (dx/14 §10), audited against the registries on 2026-08-15 and recorded in Spec/2064g/AUDIT-2026-08.md. It is one table, referenced everywhere, so "which version do we build against" never has two answers.

repository split (dx/18 §2, §5):

• create and configure zu-web, zu-c, zu-python, zu-node, zu-go, zu-java, zu-dotnet, zu-kit identically
• release-artifact contract: libzu-*, zu.h, model.json, gql.json, errors.json, cli.json, conformance-<version>.tar.zst
• conductor workflow skeleton, every dispatch a no-op (dx/14 §6)
• issue routing: a binding repo's template asks whether the bug reproduces through the CLI, and sends it here if it does
• clients/ index in the README and a repository map in the docs

API model and map (dx/03 §2, docs/04 §2, dx/18 §4):

• model.json generator from the Rust API surface
• api-map.toml schema, one per binding repository
• two-way completeness check: a new public symbol with no map entry fails here, an unmapped tier-1 entity fails the release

conformance corpus skeleton (dx/15 §2):

• YAML case schema and the {type, value} value encoding, strings for int64/decimal/timestamp
• the Rust runner
• release packaging as conformance-<version>.tar.zst
• ~150 type cases

foundations:

• terminology table zu-web/style/zu/terms.yml, shared by docs, SDKs, CLI, and error messages (docs/03 §5)
• tier-1 platform matrix building libzu in CI: linux gnu/musl x64+arm64, macOS x64+arm64, windows x64 (dx/14 §2)
• pinned toolchain table adopted and enforced in CI (dx/14 §10)
• ADRs: C ABI over Rust-only bindings; PyO3 and napi-rs as the two exceptions; TypeScript bigint for INT64; Panama over JNI; the repository split

done when:

• a PR adding a public Rust API symbol without a map entry fails CI
• the corpus runs in Rust
• a tagged build produces libzu for all seven tier-1 platforms and publishes every artifact in the contract
• the conductor dispatches to all eight repositories and collects their (empty) results

[tamnd]

The eight client repositories exist and are configured, and the decisions behind them are written down. #182 is merged.

tamnd/zu-web, zu-c, zu-python, zu-node, zu-go, zu-java, zu-dotnet, and zu-kit are public and Apache-2.0, with main protected, the same label set, wiki and projects off, and a security policy pointing here. Each carries a README written for the developer who arrives from that language rather than from this one, a SUPPORT.md, and issue templates whose first field is a required checkbox asking whether the problem reproduces through the zu CLI. That checkbox is the whole of the issue routing item: an engine bug filed in a client tracker is the standard way a multi-repository project loses one, so the template asks before it asks anything else. zu-c's README opens by saying the repository does not contain zu.h, since a repository with that name owes the reader the sentence immediately. Descriptions and topics are set on all nine, each client leading with its registry coordinate.

The map a reader needs is now in two places that stay in step. The README gained a Clients section listing all eight with what each one is and its tier, and docs/12-implementation-plan.md §3 gained the boundary rule and a pointer to the full topology.

Five ADRs are in docs/adr/, indexed by docs/adr/README.md: the C ABI over Rust-only bindings, PyO3 and napi-rs as the two exceptions to it, bigint for INT64 in TypeScript, Panama over JNI with a JNI fallback provider, and the repository split itself. Each carries its context, the decision, what it costs, and what was rejected and why. The fifth is the one with a rule rather than a preference behind it, and the rule is what puts the Rust SDK, zu.h, and the conformance corpus in this tree while putting the seven clients and the site outside it.

crates/zu-cli/tests/adr.rs holds the index to the records, checking that every record is listed and every listing resolves to a file, that numbers run from 0001 with no gaps and no reuse, that the index is in the same order so the log reads chronologically, that every status is one of the three and a superseded record names its replacement, and that every record opens with its own number and carries all four sections. The same test holds the README's table to the eight repositories the split creates and to the sentence saying where a bug found through a client belongs. A hand-maintained index that drifts from what it indexes is worse than no index, because a reader trusts it.

Next is the API model: the model.json generator over the Rust public surface, then the api-map.toml schema and the two-way completeness check that fails an engine PR adding an unmapped public symbol.

[tamnd]

model.json generator landed in #184.

cargo xtask model runs rustdoc for zu and the four crates zu re-exports from, normalizes what comes back, and writes docs/api/model.json: 809 entities, committed to the tree. Committed because an API change should be visible in the diff of the pull request that makes it, which is the only place anyone will look at it, and because every consumer downstream can then read a file instead of needing a nightly toolchain.

rustdoc's own output cannot play this part. It is nightly-only, its format is versioned and changes, it is over half a megabyte for a crate this size, and it describes one crate at a time while zu re-exports part of its surface from four others. So the normalizer follows pub use into the crate the item really lives in and files the result under the path a user types, flattens inherent impls into methods hanging off their type, leaves out trait and blanket and synthetic impls, rewrites crate:: to the crate it means, and carries the declared type of every field, constant, alias and variant so that a header generator has something to emit. A type tag the renderer does not know becomes ?tag rather than disappearing, and a test asserts the committed model holds none, so a format change shows up as a diff instead of as a quietly wrong signature.

CI runs cargo xtask model --check on nightly-2026-08-14, pinned to the date rather than floating, so a rustdoc format bump is a deliberate change to model.rs and not a red main on somebody else's release schedule. The check compares byte for byte, which is only a real check because the generator is deterministic, and on drift it names the identifiers that were added and removed rather than saying the file differs. It has already earned its keep: it caught the three entities #181 added while this was in flight.

Writing the benches found a real defect, in the JSON reader this pull request moved into crates/zu-json of its own. It copied string contents one scalar at a time and validated UTF-8 by calling from_utf8 on the whole remaining input per character, which is quadratic per document. A hundred byte shell frame was short enough to hide it; a half megabyte rustdoc document was not. Copying unescaped runs whole took the frame from 701 ns to 381 ns and the document from 1.24 MB/s to 434 MB/s. The bench now measures three document sizes and fails if the rate halves between them, because one measurement cannot see the shape of that defect.

21 tests across two suites: fixtures small enough to read for the normalizer, and properties of the committed file for the artifact. docs/api/README.md documents the schema.

Next in this milestone is the api-map.toml schema, which joins against this file on the entity id, and then the two-way completeness check that fails a pull request adding a public symbol nothing maps.

[tamnd]

api-map.toml, and the check that runs both ways (#186)

model.json landed last week and says what the Rust surface is. It says nothing about what any of it is for, which left nine repositories free to guess. That fails quietly: somebody adds a public type, nobody notices, and six months later two SDKs expose it under different names and a third does not have it.

docs/api/api-map.toml is where the decision gets written down. One schema does two jobs, told apart by target. The ledger in this repository classifies the surface into three tiers. A binding map lives in its own repository and gives each classified entity its name in that language. Tier 1 is what every tier-1 SDK exposes, and a release where one of them does not fails. Tier 2 is what a binding may expose, with nothing gated on it. Tier 3 is what no binding exposes, and the entry has to say why, because a false nobody thought about reads exactly like a decision.

An [[entity]] beats every [[group]] over it and the narrower group wins, so carving one symbol out of a module does not mean rewriting a prefix. Both lists stay sorted and unique, because nobody reviews a diff they cannot follow.

Three directions, and the obvious one is the least useful. Forward, every public symbol is classified or the check fails here, which is what a new type trips. Backward, every group prefix and every entry still names live code, which is what a deleted type trips, and that is the one worth having: a stale classification sits in the file looking like coverage and nothing ever joins the two by hand. Release, every tier-1 entity is named by every binding map, run once per repository over what the conductor collects.

The forward check is a test as well as a CI step. A test runs on stable, in the three-OS matrix that already exists, and on the machine of whoever added the symbol, which is the only place the answer is cheap. The CI step stays so a failure is attributed to the API job and so the command itself is exercised.

The reader is a strict subset of TOML that refuses what it does not understand, with a line number. A general parser would silently ignore an unknown key, and an ignored key is a mapping somebody wrote and nobody applied, which is exactly the drift this file exists to stop.

Classifying the surface put 73 entities at tier 1, 111 at tier 2 and 596 at tier 3, in 25 groups and 6 exceptions. Tier 1 is zu::session, zu::query::Value, zu::query::QueryResult, the error types and zu::NodeId. Tier 3 is the 522 zu::zu1 format internals, the storage engine, the SQLite path and the one-shot query helpers. Modules are left out rather than given 29 no-op entries, because a namespace is not something a binding can be asked to name.

The bench found the check quadratic and the fix went in the same commit. Resolving a tier scanned every group looking for the longest covering prefix, and the dead-group direction scanned every identifier per group. Twenty-five groups over eight hundred entities hid that completely; three thousand over thirty-two thousand did not, at 373 ms. Each identifier now walks its own path prefixes against the sorted groups, which is a binary search per segment and independent of how long the file gets, and both directions come out of one walk. 7.5 ms at that size, and 0.083 ms on the surface that exists.

31 new tests: 9 on the reader, 17 on the map and the two checks against fixtures small enough to read in full, 5 against the committed files. Both drift directions were also proved by hand against a doctored model, which reported exactly the added symbol and the removed one and nothing else.

Two checklist items and one acceptance criterion, "a PR adding a public Rust API symbol without a map entry fails CI".

One thing this leaves open. The binding repositories still hold only README.md, LICENSE and SUPPORT.md, so no real binding map exists yet and the release direction is exercised by fixtures. How a binding repository gets the ledger it checks against is a question the release-artifact contract has to answer, and that contract does not currently list api-map.toml among its artifacts. Worth settling when that item comes up.

Next is the conformance corpus skeleton: the YAML case schema, the {type, value} encoding, and the Rust runner.

[tamnd]

The conformance corpus skeleton is in, as #189: the case schema, the {type, value} encoding, the Rust runner, thirty cases, and a way to run them against an unpacked artifact. That ticks the schema line, the runner line, and the acceptance criterion that the corpus runs in Rust.

The map that landed last week says what a binding owes every entity in the model. This is the other half of the same problem and the half no surface check can reach. Nine clients, seven languages, two of them reaching the engine by a different mechanism than the rest, and every one of them decoding values with code no other client shares. Nothing until now said whether a value put in through one and taken out through another means the same thing.

The rule the whole encoding exists for turned out to be which types are written in quotes. INT64, UINT64, the floats and every temporal type are, because a bare 9007199254740993 is a number and most YAML readers hand a number to a double and give it back as 9007199254740992. A case that writes one of them bare is refused rather than read leniently and rounded, which is the difference between a corpus and a set of examples. The narrow integers and the booleans go the other way, unquoted, so a reader cannot take one for a string.

Thirty cases to start, in two suites. Every one is a value a client has got wrong somewhere: the largest INT64, the smallest integer a double cannot hold, 0.1 + 0.2, negative zero, an overflow to infinity, a leap day, a nanosecond time, an offset that is part of the value rather than a rendering of it, a duration in months that is not a number of days.

Two things worth writing down here rather than leaving in the PR.

The first is a gap in the Rust runner. QueryResult carries no column type metadata, so this runner checks the value a case declares and cannot check the declared type: a case saying INT8 and one saying INT64 are the same assertion here when the value matches. That is a gap in this runner and not in the corpus, and it is the part the eight typed-language runners will check most usefully, because a client in a language with an i8 has somewhere for INT8 to go and something to get wrong. It is in conformance/README.md so that nobody writes a case believing it proves more than it does.

The second is an engine gap the cases found. RETURN 9223372036854775807 + 1 fails with no GQLSTATUS at all, "invalid argument: integer overflow", where 22003 is what the standard asks for. There is a divide-by-zero case raising 22012 and no overflow case beside it for that reason. Worth its own issue against the query layer rather than a case that asserts the wrong thing today.

Writing the cases against the engine rather than guessing at them was worth the hour it cost. int64-min is a subtraction because -9223372036854775808 as a literal raises 22003, the temporal literals are spelled LOCAL TIME and ZONED DATETIME rather than LOCALTIME and ZONEDTIME, and date() and duration() do not exist. Every one of those would have shipped as a broken case.

The reader stays linear to 6400 cases at about 2.9 us each, and a case costs about 3.9 ms to run, which is the database-per-case rule and not the reader. Three orders of magnitude apart, so the rule that keeps a leaked table out of the next case is affordable well past the size the corpus is heading for, and the reader is not worth optimizing.

Next in the corpus: the roughly 150 type cases, then release packaging as conformance-<version>.tar.zst.

Still open from the last comment: api-map.toml is not listed among the artifacts in the release-artifact contract line above, and it should be, since every binding repository needs its own copy of the schema to check against.

[tamnd]

~150 type cases is done, in #190, merged as a5375c2.

The skeleton shipped thirty cases in two suites, which was enough to prove the reader, the runner and the encoding but proved nothing about the engine. There are now 223 cases in nine suites:

Suite	Cases	What it pins
scalar	19	The values that cross a language boundary badly.
temporal	11	Dates, times and durations.
numeric	28	Integer division, the sign of a remainder, the edge of a type.
string	23	What a character is, how many of them a string has, whether two that look identical are the same.
null	24	Three-valued logic, including the two places the answer is known and a mechanical null propagation says it is not.
list	31	The one composite type, and therefore the one place a decoder recurses.
cast	37	Truncation towards zero, a range check rather than a wrap, a refusal rather than a zero.
order	20	The total order across types, and where nulls go in both directions.
aggregate	30	Where a type changes on the way through, and the three different empty answers.

Every answer came from the engine rather than from what the standard ought to say, which is the only way to write a corpus that gates releases: a case asserting what somebody believes is a case that fails on the day somebody was wrong, and there is then no way to tell which of the two is at fault.

Every case is at expression level. That is deliberate where it can be, because an expression is the shortest statement that isolates the thing under test, and it is forced everywhere else: CREATE is not implemented, so no case can store a value and read it back. Property round-trip is the obvious next batch and it waits on the write path. It is worth saying that the corpus therefore does not yet touch storage at all, and the question it exists to answer, whether a value put in through one client and taken out through another means the same thing, is currently answered only for values the engine computes.

One reader fix came out of it. A quote that opens after whitespace and never closes was an error, which refused RETURN cast(' 42 ' AS INT64) AS n, because the closing quote has a space before it and so looks like an opening one. It is now an ordinary character, which is what real YAML makes of a quote inside a plain scalar. A value that really does open a quoted scalar and never closes it is still refused, where the scalar is read, which is the only place the value's own start is known.

Reading the whole corpus costs 0.76 ms and running it costs 838 ms, a thousand to one, so the database-per-case rule is still the only thing in the runner that would ever need revisiting and does not need it yet. The reader's per-case cost is flat out to 6400 cases.

Four engine gaps turned up while probing, none of them fixed there:

• 9223372036854775807 + 1 fails with no GQLSTATUS and invalid argument: integer overflow, where 22003 is what the standard asks for. Raised in the previous comment and still open.
• -DURATION 'PT1H' fails with no GQLSTATUS and cannot negate Temporal(...).
• RETURN sum(n) * 2 fails with no GQLSTATUS and aggregate item 'c' must be a bare call for now. An expression over a set function being unimplemented is fine at this stage; failing without a code is not.
• --format json renders a NaN and an infinity as null, so a float and an absent value are indistinguishable in JSON output. The values are correct and --format table shows them, so this is the formatter and not the engine.

The first three are #192, one issue against the query layer, that an unimplemented or refused operation must still carry a GQLSTATUS. The fourth is #193, against the CLI.

Also worth recording: the corpus cannot express a warning. A statement can succeed and raise a notice, as a set function that dropped a null does, and no case can say so. The cases that provoke one say so in prose. A notices: key is the obvious addition and I have not added it, because a key with one user is a key whose shape is guessed, and the second user will say what shape it wants.

Next in the corpus is release packaging as conformance-<version>.tar.zst, which is the last item under the corpus heading and is also the line item under the release-artifact contract above. The open question from the previous two comments still stands: api-map.toml is not listed among the release artifacts and should be, because a binding repository checking its own map against this one needs the map at the version it pinned.

[tamnd]

Release packaging is in, #195, which closes out the corpus heading.

cargo xtask corpus-pack builds conformance-<version>.tar.zst: the case files as committed, their README beside them so an unpacked artifact explains itself, and a manifest.json with the artifact schema, the engine version, the case schema, the total count, and per suite its path, its case count, its size and a CRC32C. 9 suites, 223 cases, 19 KiB packed from 98.

It is a plain tar of text under zstd because every language in the set has a reader for both, and nothing in the artifact needs a zu client to open. The manifest is JSON rather than TOML to match model.json, gql.json, errors.json and cli.json, and because it is readable in all seven client languages without a parser dependency. The checksum catches a truncated download and a corrupted unpack, and both the code and the README say plainly that it is not a signature: what a release is signed with is the release's business, and a checksum sitting inside the archive could only ever agree with an archive that was rewritten wholesale.

Two things it does that a tar czf in a release script would not. The archive is reproducible, because every header field that could carry the packing machine's clock, user id or umask is fixed instead, so two mirrors of one release are comparable rather than merely trusted. And the packer parses every case through the runner's own reader before it ships one, which is why --check is worth running on every pull request on a branch that will never cut a release: an artifact eight repositories cannot load is not a thing to find out about on release day.

The ustar writer is hand-rolled, which is the same call the TOML, JSON and YAML readers here already made. It is a header, a body and two zero blocks, and it is smaller than the dependency review would have been.

Verified outside the Rust toolchain too, since that is how it will actually be consumed: system tar --zstd -xf extracts it, diff -r says the cases are identical to the committed ones, and zu corpus on the unpacked directory reports 223 of 223 passing under --strict. The bench holds the per-case packing cost linear as the corpus grows, because a packer that is quadratic is a release step that stops finishing somewhere between the corpus of today and the corpus of next year.

Standing question, and this is the fourth time I have raised it, so I will make it concrete rather than ask again. The release-artifact contract above lists libzu-*, zu.h, model.json, gql.json, errors.json, cli.json and now this. It does not list api-map.toml, which every binding repository needs a copy of the shape of and which the completeness check runs against. Unless somebody objects I will add it to the contract line when I take that item, and note the reason there.

Next is the terminology table, zu-web/style/zu/terms.yml, which the docs, the SDKs, the CLI and the error messages all share.

[tamnd]

Terminology table done, in two repositories, since it has two halves that belong in different places.

The table is style/zu/terms.yml in tamnd/zu-web (tamnd/zu-web#1, tamnd/zu-web#2): 46 terms in seven groups, each with the group it belongs to, a one sentence definition, and the forms that must give way to it. It is in the site's repository because the site is where this program's prose is published from and the glossary the site renders is the table itself. style/README.md there has the schema and the rule for adding a term.

The reader is cargo xtask terms here (#198), because the prose with the most readers is here and because a table checked by nine repositories against nine readers would be nine tables. CI clones zu-web and tracks its main rather than a pin: the table is prose about what words mean, and a word that changed meaning has changed for this repository the moment it changed for the site.

Every entry was taken from what the repositories already say rather than invented. rel table because CREATE REL TABLE is what a user types. optimizer because optimizer.rs exists, planner.rs does not, and docs/02's crate table already said optimizer. node group, column segment and chunk as three different sizes, because they are three different sizes and the storage chapter turns on the difference. client, SDK and binding all kept with definitions and no preference between them, because all three are in live use here in three different senses. tag was in the first draft and came out again: it produced twenty five false positives, every one of them a union discriminant or an XML tag, and the distinction it was there for is one sentence in label's definition instead.

What is checked is markdown and the doc comments that become reference pages, and nothing else. Not identifiers, which answer to the language they are written in; not code spans or fenced blocks, because a check that fires on Vertex in a signature teaches people to ignore it; not link targets, which are addresses. A form matches whole words only, the last word of a form may pick up a plural s, the words of a multi-word form have to be one phrase, and case is ignored except where a form differs from its own term only in case, which is what lets zu refuse Zu at the start of a sentence without refusing every other capitalised word.

A page that quotes somebody else exempts the form and says which, with <!-- terms: allow row group, vertex -->. It has to name the forms so it can never become a blanket switch, a named form the page does not contain is reported stale, and a named form the table never had is reported as exempting nothing. That is the rule the API map already follows for a classification of code that is gone. Four places use it: docs/01 describes other people's systems in their words, docs/04 contrasts with the row group design it borrowed from, the graphalytics bench names things the way the published dataset it validates against names them, and the GQLSTATUS table test means XML's attribute rather than a graph element's.

Making the tree obey it cost seventeen planner, two edge table, and a handful of smaller things. The check now reports nothing over 216 files.

Two consumers this line names are not covered yet, and are not covered quietly. The CLI's help text and the text of an error are prose a user reads, and neither is extractable today: the help lives in usage strings among a hundred other literals, and an error's message is built where it is raised. Both become extractable when cli.json and errors.json exist, which are two items of the release artifact contract three lines above this one, and the right time to check them is when they are generated rather than by grepping for string literals and guessing which ones a user sees. The module doc says so, so it is a documented gap and not a silent one. Evidence that the gap is real: exec.rs raised an error that said "the planner missed a flatten", which nothing in this check would have caught and which was fixed by hand.

Forms are indexed by their first word, so a line costs one hash lookup per word no matter how large the table gets. The whole tree is 3.7 MiB of source and the check is about 14 ms. The bench asserts the cost stays linear in the prose and flat in the size of the table, from 13 forms to 1013, which is what makes adding a term a decision about the word and not about the budget.

Next in this milestone: the tier-1 platform matrix, the pinned toolchain table, the release-artifact contract, and the conductor workflow skeleton.

[tamnd]

The pinned toolchain table is in, #201. toolchains.toml at the root: 23 components, each with the release CI runs, the oldest release still promised, the repositories that build against it, and a sentence saying why the numbers are what they are, plus the date the versions were audited against the registries.

A component this repository builds against says where its version is written, in a [[site]] naming the file, the key, whether the site holds the pin or the floor, and whether it has to say the version exactly or name the series it is in. Exact is what rust-toolchain.toml and a workflow say, since those name a release to install. A series is what a cargo requirement of 59 says against 59.2.0, since writing the patch level in a manifest is a lockfile in the wrong file.

cargo xtask pins checks both directions, which is the shape the API map already has. Forward: every site exists, holds the key it claims, and agrees with the version. Backward: every toolchain: a workflow pins is one the table names, so a job added next month cannot quietly introduce a tenth answer, and a component this repository builds against with no site is reported, since a row nothing holds is a row nobody maintains. It runs as a test on all three platforms of the matrix and as a command in CI, the second because a command nothing exercises is a command that rots.

Adopting it meant fixing the two sites that did not agree with the audited numbers. rust-toolchain.toml pinned the 1.97 series rather than 1.97.1, which is a range where a release is wanted, and the arrow and parquet requirements were still on 56 against an audited 59.2.0. The whole suite passes on arrow 59, so what is recorded is what the tree actually builds against rather than what it was supposed to.

The bench asserts the shape rather than the number, the cost per site staying flat as the table grows from eight to five hundred, since the way a check like this dies is the accidental square that costs nothing at six sites and is unaffordable at two hundred. The committed table against this tree is 0.18 ms.

Bumping a version is now a pull request that touches the table and every site of it together, which is the entire point of writing it down. Next in this milestone is the tier-1 platform matrix.