From e0e9a404d9c056d72a7355d463311d2c909ffa09 Mon Sep 17 00:00:00 2001 From: Tam Nguyen Duc <1218621+tamnd@users.noreply.github.com> Date: Sun, 16 Aug 2026 02:57:46 +0700 Subject: [PATCH] Publish a release against one artifact contract A release is a tag here and a run that drives the eight other repositories, and every one of them builds against what this one published. That makes the list of what gets published a contract rather than a step in a workflow: a binding that fetches model.json for the version it pins and finds nothing cannot tell a release that dropped the artifact from a version that never had it, and the failure surfaces in somebody else's CI a day later. artifacts.toml is the list, and release.yml has none of its own. It assembles from the table and reads the directory back against it, so an artifact that stopped being produced fails the release that dropped it rather than the eight that wanted it. Four kinds of row, because there are four ways an artifact comes to exist: a file the tree already holds, the packed corpus, one artifact per tier 1 target expanded against platforms.toml, and a row the contract names that nothing makes yet with the milestone that will make it. Naming those three early is the point rather than an oversight, since a consumer needs to know what a release will eventually carry. The train itself is the skeleton dx/14 section 6 asks for. The build is the same matrix every pull request runs, the assemble and the verify are real, and every publish step prints what it would do in the order it will do it, crates.io before the repositories that build against it and the Go tag last of the registries because pushing a tag cannot be taken back. A rehearsal runs on workflow_dispatch with a version rather than a tag, since a release train is exactly the machinery that must not run for the first time on the day of a release. A platform's build now stages the library, the CLI, the import library where there is one and the header into a directory named for the target, and that directory is the artifact, so libzu-.tar.zst unpacks to four files rather than four levels of somebody's build path. Both archives are written by one tar writer, moved out of the corpus packer to tarball.rs, because a second implementation of a format this dull would be a second set of headers to get subtly wrong in somebody else's language. Twenty tests, a bench holding the per-artifact cost of verifying flat as the contract grows, and the check runs as a test and as a command in CI like the two tables beside it. --- .github/workflows/ci.yml | 11 + .github/workflows/libzu.yml | 28 +- .github/workflows/release.yml | 113 ++++ artifacts.toml | 72 +++ crates/xtask/Cargo.toml | 4 + crates/xtask/benches/artifacts.rs | 139 +++++ crates/xtask/src/artifacts.rs | 905 ++++++++++++++++++++++++++++++ crates/xtask/src/corpus.rs | 113 +--- crates/xtask/src/lib.rs | 2 + crates/xtask/src/main.rs | 146 ++++- crates/xtask/src/tarball.rs | 147 +++++ docs/10-api-and-tooling.md | 14 +- toolchains.toml | 7 + 13 files changed, 1596 insertions(+), 105 deletions(-) create mode 100644 .github/workflows/release.yml create mode 100644 artifacts.toml create mode 100644 crates/xtask/benches/artifacts.rs create mode 100644 crates/xtask/src/artifacts.rs create mode 100644 crates/xtask/src/tarball.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0485d009..ed6e7d27 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,6 +79,17 @@ jobs: # targets, and the matrix in libzu.yml held to them. This runs as # a test too, and here for the same reason as the two above. - run: cargo run -p xtask -- platforms + # And the table on the end of the release: what a tag publishes, + # which eight repositories fetch by version. release.yml assembles + # from it and verifies the directory back against it, so an + # artifact that stopped being produced fails the release that + # dropped it rather than the eight that wanted it. + - run: cargo run -p xtask -- artifacts + # A release is compression and copying and costs what those cost. + # What this measures is the bookkeeping either side of it, whose + # per-artifact cost has to stay flat as the contract grows past + # the seven platforms it has today. + - run: cargo bench -p xtask --bench artifacts deny: runs-on: ubuntu-latest diff --git a/.github/workflows/libzu.yml b/.github/workflows/libzu.yml index 387632d8..376d2941 100644 --- a/.github/workflows/libzu.yml +++ b/.github/workflows/libzu.yml @@ -127,16 +127,30 @@ jobs: --default-toolchain $toolchain export PATH=/w/.cargo/bin:\$PATH scripts/libzu-build.sh ${{ matrix.target }} ${{ matrix.smoke }}" + # What a consumer of the C ABI needs, flat in one directory named + # for the target, because this directory is the release artifact: + # `cargo xtask artifacts --assemble` packs it as + # libzu-.tar.zst and an unpacked release should be four + # files rather than four levels of somebody's build path. The + # import library goes beside the DLL because a C caller on Windows + # links against that and not against the DLL, and it exists + # nowhere else, which is why the copy is conditional. + - name: Stage the artifact + shell: bash + env: + out: dist/libzu-${{ matrix.target }} + from: target/${{ matrix.target }}/release + run: | + set -eux + mkdir -p "$out" + cp "$from/${{ matrix.lib }}" "$from/${{ matrix.exe }}" "$out/" + if [ -f "$from/zu.dll.lib" ]; then cp "$from/zu.dll.lib" "$out/"; fi + cp crates/zu-capi/include/zu.h "$out/" # One artifact per platform, named for the target, so that a # release assembles what CI built rather than building it again - # somewhere else. The import library goes with the DLL because a C - # caller on Windows links against that and not against the DLL. + # somewhere else. - uses: actions/upload-artifact@v4 with: name: libzu-${{ matrix.target }} if-no-files-found: error - path: | - target/${{ matrix.target }}/release/${{ matrix.lib }} - target/${{ matrix.target }}/release/${{ matrix.exe }} - target/${{ matrix.target }}/release/zu.dll.lib - crates/zu-capi/include/zu.h + path: dist/libzu-${{ matrix.target }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 00000000..237854b0 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,113 @@ +name: release + +# The release train of dx/14 section 6: one version number, one day, +# one orchestrated run across nine repositories. This is the skeleton of +# it, and what is real here is deliberate. The build is the same matrix +# every pull request runs, the assemble step gathers exactly the rows of +# artifacts.toml, and the verify step reads the directory back against +# the same table. Every publish step is a no-op that says what it would +# do. +# +# The ordering is the part worth having this early, because it is the +# part that is expensive to discover late: crates.io lands before the +# repositories that build against it, and the Go tag is last of the +# registries because pushing a tag is the one publish that cannot be +# taken back. A skeleton that runs the real order on every rehearsal is +# how the order stops being a paragraph in a specification. + +on: + push: + tags: ["v*"] + # A rehearsal, on a branch, with no tag. The train is the thing that + # must not be run for the first time on the day of a release. + workflow_dispatch: + inputs: + version: + description: The version to rehearse as + required: false + default: 0.0.0 + +concurrency: + # Two releases at once is two versions publishing to nine registries + # in an order neither of them chose, and cancelling the one already + # part way through is worse than queueing behind it. + group: release + cancel-in-progress: false + +env: + CARGO_TERM_COLOR: always + +jobs: + # The seven tier 1 platforms, called rather than repeated, so a + # release ships the artifacts CI has been building all along. + libzu: + uses: ./.github/workflows/libzu.yml + + assemble: + needs: libzu + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: dtolnay/rust-toolchain@master + with: + toolchain: 1.97.1 + - uses: Swatinem/rust-cache@v2 + # One directory per platform, under the name that platform's job + # uploaded it as, which is the name the release publishes it as. + - uses: actions/download-artifact@v4 + with: + pattern: libzu-* + path: built + # The tag is the version, with the `v` off it, because a git tag + # is written `v0.5.0` and a package is not. A rehearsal says which + # version it is rehearsing instead. + - name: Assemble the release + shell: bash + run: | + set -eux + tag="${{ inputs.version || github.ref_name }}" + version="${tag#v}" + cargo run -p xtask -- artifacts --assemble dist --built built --version "$version" + cargo run -p xtask -- artifacts --verify dist --version "$version" + - uses: actions/upload-artifact@v4 + with: + name: release + if-no-files-found: error + path: dist + + publish: + needs: assemble + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + - uses: actions/download-artifact@v4 + with: + name: release + path: dist + - name: What is being published + shell: bash + run: ls -l dist + # Each step below is the step that will exist, in the order it + # will run, doing nothing. They are separate steps rather than one + # loop because the run's own step list is then the ordering, which + # is what a person reads when a release stops half way through. + - name: GitHub release + run: echo "no-op: upload dist/ to the release for ${{ github.ref_name }}, signed and attested (dx/14 section 8, DX5)" + - name: crates.io + run: echo "no-op: publish zudb, zudb-async and zu-cli, first because everything else builds against them" + - name: PyPI + run: echo "no-op: publish the wheels zu-python built against these artifacts" + - name: npm + run: echo "no-op: publish zudb and its platform packages, then the JSR re-export" + - name: Maven Central + run: echo "no-op: publish dev.zudb, which is a staging repository that has to be closed and released" + - name: NuGet + run: echo "no-op: publish ZuDb" + - name: Go module tag + run: echo "no-op: push the tag on zu-go, last of the registries because a Go module cannot be unpublished" + - name: Docs and release notes + run: echo "no-op: deploy zu-web against this version, publish the notes, bump Homebrew, Scoop and AUR" + - name: What this run did not do + run: | + echo "The dispatches to the eight repositories are the conductor's, which is the next item of DX0." + echo "Every publish above is idempotent when it is real, so a partial release is resumed and not restarted." diff --git a/artifacts.toml b/artifacts.toml new file mode 100644 index 00000000..619cb47d --- /dev/null +++ b/artifacts.toml @@ -0,0 +1,72 @@ +schema = 1 +doc = "What a release of zu publishes, one table for nine repositories." +audited = "2026-08-16" + +# A release is a tag on this repository and a run that drives the eight +# others (dx/14 section 6). Each of them builds against what this one +# published, so the list of what gets published is a contract rather +# than a step in a workflow: a binding that fetches `model.json` by +# version and finds nothing has no way to tell a release that dropped +# the artifact from a version that never had it. +# +# So the list lives here, once, and the release workflow has none of its +# own. `cargo xtask artifacts --assemble` gathers exactly these rows and +# `--verify` reads the directory back, which means an artifact that +# stopped being produced fails the release that dropped it rather than +# the eight repositories that wanted it. +# +# `made` says where a row comes from and there are four answers. `file` +# is a path this tree already holds and the release copies. `corpus` is +# packed by the corpus packer. `platform` is one row per tier 1 target +# of platforms.toml, so the seven move with that table and not with this +# one. `later` is an artifact the contract names and nothing makes yet, +# with the milestone that makes it: writing it down early is the point, +# since a consumer needs to know what a release will eventually carry +# and the alternative is eight repositories each guessing. + +[[artifact]] +name = "libzu-.tar.zst" +made = "platform" +consumers = ["zu-c", "zu-go", "zu-java", "zu-dotnet", "zu-kit"] +doc = "The shared library, the CLI and the header for one tier 1 target, as the platform's job built them. The five repositories here reach the engine through the C ABI rather than compiling against it, so this archive is the whole of what they link." + +[[artifact]] +name = "zu.h" +made = "file" +from = "crates/zu-capi/include/zu.h" +consumers = ["zu-c", "zu-go", "zu-java", "zu-dotnet", "zu-kit", "zu-web"] +doc = "The C ABI, published beside the libraries as well as inside each of them, because a consumer generating bindings needs the header without downloading a platform it does not build for. tamnd/zu-c deliberately does not hold a copy (dx/18 section 2)." + +[[artifact]] +name = "model.json" +made = "file" +from = "docs/api/model.json" +consumers = ["zu-c", "zu-python", "zu-node", "zu-go", "zu-java", "zu-dotnet", "zu-kit", "zu-web"] +doc = "The public Rust surface as data. Every binding checks its api-map.toml against the model of the version it builds against, and the site renders the reference pages from it, so it is fetched by version rather than read from this repository's main branch." + +[[artifact]] +name = "conformance-.tar.zst" +made = "corpus" +consumers = ["zu-c", "zu-python", "zu-node", "zu-go", "zu-java", "zu-dotnet", "zu-kit"] +doc = "The cross-client conformance corpus for this exact version. A client pins an engine version and needs the cases that shipped with it, not the cases on this branch, which are the cases for a version it has not adopted (dx/15 section 2)." + +[[artifact]] +name = "cli.json" +made = "later" +milestone = "D1" +consumers = ["zu-web"] +doc = "Every command, flag and exit code of the CLI, which D1 renders twelve reference pages from. It arrives with the `--format json` surface of DX1, since the generator reads the CLI's own description of itself rather than its help text." + +[[artifact]] +name = "gql.json" +made = "later" +milestone = "D2" +consumers = ["zu-web"] +doc = "Statements, functions, types and signatures, which the language reference is generated from and which an agent reads directly (dx/16 section 2). D2 is where it can first exist, because it is the shipped grammar and registry rather than a hand-written list." + +[[artifact]] +name = "errors.json" +made = "later" +milestone = "D2" +consumers = ["zu-python", "zu-node", "zu-web"] +doc = "Every condition with its GQLSTATUS, its meaning, its fix and whether retrying it is sensible. The bindings are consumers as well as the site: an SDK that maps a status to an exception class is the same list read a second way, and reading it from the artifact is what keeps the two spellings the same." diff --git a/crates/xtask/Cargo.toml b/crates/xtask/Cargo.toml index 25d92e5e..5f17432e 100644 --- a/crates/xtask/Cargo.toml +++ b/crates/xtask/Cargo.toml @@ -36,6 +36,10 @@ harness = false name = "pins" harness = false +[[bench]] +name = "artifacts" +harness = false + [dependencies] crc32c.workspace = true zstd.workspace = true diff --git a/crates/xtask/benches/artifacts.rs b/crates/xtask/benches/artifacts.rs new file mode 100644 index 00000000..5c123886 --- /dev/null +++ b/crates/xtask/benches/artifacts.rs @@ -0,0 +1,139 @@ +//! What holding a release to the artifact contract costs. +//! +//! Two numbers, and neither of them is the release itself: assembling +//! is compression and copying, and it costs what those cost. What is +//! measured here is the bookkeeping around them, because that is the +//! part that runs on every pull request as well as on release day, and +//! because it is the part with a shape that can go wrong. +//! +//! The second column is the one that matters. Verifying reads the +//! release directory once and then looks each expected name up, so the +//! cost per artifact is flat. The failure worth catching is the +//! accidental square, a scan of the directory per row, which costs +//! nothing at ten artifacts and is the reason a check gets skipped at +//! several hundred, and several hundred is where this goes when tier 2 +//! platforms and per-binding artifacts arrive. +//! +//! Run: cargo bench -p xtask --bench artifacts + +use std::hint::black_box; +use std::path::{Path, PathBuf}; +use std::time::Instant; + +use xtask::artifacts::{PATH, Table, tier1}; + +fn main() { + println!("{:>9} {:>9} {:>9}", "rows", "parse ms", "us/row"); + let mut per_row = None; + for rows in [8usize, 32, 128, 512] { + let text = table(rows); + let ms = best(|| { + black_box(Table::parse(black_box(&text)).expect("the generated table parses")); + }); + let us = ms * 1e3 / rows as f64; + println!("{rows:9} {ms:9.3} {us:9.2}"); + + // Parsing is a pass over the lines and a lookup per row. Four + // times the cost per row is a table whose own validation went + // quadratic, which is what a name checked against every name + // before it looks like. + if let Some((before, was)) = per_row { + assert!( + us < was * 4.0, + "parsing went from {was:.2} us/row at {before} rows to {us:.2} us/row at {rows}, \ + which is not linear" + ); + } + per_row = Some((rows, us)); + } + + println!("\n{:>9} {:>9} {:>9}", "files", "verify ms", "us/file"); + let mut per_file = None; + for rows in [8usize, 32, 128, 512] { + let text = table(rows); + let table = Table::parse(&text).expect("the generated table parses"); + let dir = release(&table, rows); + let ms = best(|| { + let (shipped, faults) = table + .verify(black_box(&dir), "0.5.0", &[]) + .expect("the release is readable"); + assert!(faults.is_empty(), "{faults:?}"); + black_box(shipped); + }); + let us = ms * 1e3 / rows as f64; + println!("{rows:9} {ms:9.3} {us:9.2}"); + + // One read of the directory and one lookup per name. Four times + // the cost per file is a lookup that walks the directory again. + if let Some((before, was)) = per_file { + assert!( + us < was * 4.0, + "verifying went from {was:.2} us/file at {before} files to {us:.2} us/file at \ + {rows}, which is not linear" + ); + } + per_file = Some((rows, us)); + let _ = std::fs::remove_dir_all(&dir); + } + + // Cargo runs a bench from the package directory, so the tree is two + // levels up. + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); + let committed = Table::load(&root.join(PATH)).expect("the committed table loads"); + let targets = tier1(&root).expect("the platform table loads"); + let ms = best(|| { + black_box( + committed + .check(black_box(&root)) + .expect("the tree is readable"), + ); + }); + println!( + "\ncommitted contract: {} rows, {} names for one release, {ms:.2} ms", + committed.artifacts.len(), + committed.names("0.5.0", &targets).len(), + ); +} + +/// A contract of `rows` artifacts, all of them published, which is the +/// shape that costs the most: a row nothing makes yet is a row neither +/// the assemble nor the verify step looks at. +fn table(rows: usize) -> String { + let mut text = + String::from("schema = 1\ndoc = \"What the bench publishes.\"\naudited = \"2026-08-16\"\n"); + for n in 0..rows { + text.push_str(&format!( + "\n[[artifact]]\nname = \"thing{n}-.json\"\nmade = \"file\"\nfrom = \ + \"docs/thing{n}.json\"\nconsumers = [\"zu-python\", \"zu-web\"]\ndoc = \"An artifact \ + the bench generated, which a release publishes like any other.\"\n" + )); + } + text +} + +/// A release directory holding exactly what that table publishes, which +/// is the case with no faults in it and therefore the one that does the +/// most work before answering. +fn release(table: &Table, rows: usize) -> PathBuf { + let dir = + std::env::temp_dir().join(format!("zu-artifacts-bench-{}-{rows}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("the scratch dir is writable"); + for name in table.names("0.5.0", &[]) { + std::fs::write(dir.join(name), b"{}\n").expect("writes"); + } + dir +} + +/// The best of seven, in milliseconds. The best rather than the mean +/// because the thing being measured is the work, and every sample above +/// the floor is the machine doing something else. +fn best(mut body: impl FnMut()) -> f64 { + let mut best = f64::MAX; + for _ in 0..7 { + let start = Instant::now(); + body(); + best = best.min(start.elapsed().as_secs_f64() * 1e3); + } + best +} diff --git a/crates/xtask/src/artifacts.rs b/crates/xtask/src/artifacts.rs new file mode 100644 index 00000000..7a6b1939 --- /dev/null +++ b/crates/xtask/src/artifacts.rs @@ -0,0 +1,905 @@ +//! The release-artifact contract, and the release that has to keep it. +//! +//! A release is a tag on this repository and a run that drives the eight +//! others (dx/14 section 6). Every one of them builds against what this +//! one published, which makes the list of what gets published a +//! contract rather than a step in a workflow. A binding that fetches +//! `model.json` for the version it pins and finds nothing cannot tell a +//! release that dropped the artifact from a version that never had it, +//! and the failure surfaces in somebody else's CI a day later. +//! +//! So `artifacts.toml` is the list, once, and the release workflow has +//! none of its own: it assembles from this table and verifies the +//! directory back against it. An artifact that stopped being produced +//! then fails the release that dropped it, which is the run that can +//! still do something about it. +//! +//! Four kinds of row, because there are four ways an artifact comes to +//! exist here. A `file` is a path the tree already holds. A `corpus` is +//! packed by the packer beside this. A `platform` row is one artifact +//! per tier 1 target, expanded against `platforms.toml` so the seven +//! move with that table rather than with this one. And a `later` row is +//! an artifact the contract names that nothing makes yet, carrying the +//! milestone that will make it: naming it early is the point, since a +//! consumer needs to know what a release will eventually carry, and the +//! alternative is eight repositories each guessing. + +use std::collections::BTreeMap; +use std::path::Path; + +use crate::toml::Doc; +use crate::{corpus, platforms, tarball}; + +/// The table's schema version, which moves when the shape of the file +/// changes and not when the artifacts do. +pub const SCHEMA: i64 = 1; + +/// Where the table is. +pub const PATH: &str = "artifacts.toml"; + +/// The workflow that has to assemble from it. +pub const WORKFLOW: &str = ".github/workflows/release.yml"; + +/// The nine repositories of dx/18 section 2. +pub const REPOS: [&str; 9] = [ + "zu", + "zu-c", + "zu-python", + "zu-node", + "zu-go", + "zu-java", + "zu-dotnet", + "zu-kit", + "zu-web", +]; + +/// The one that makes the artifacts, and therefore the one that cannot +/// be listed as consuming them. +pub const MAKER: &str = "zu"; + +/// The cases the corpus row is packed from, and the README that ships +/// beside them so an unpacked artifact explains itself. +pub const CASES: &str = "conformance/cases"; +pub const README: &str = "conformance/README.md"; + +/// Where an artifact comes from. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Made { + /// A path this tree holds, copied into the release as it is. + File { from: String }, + /// The conformance corpus, packed for the version being released. + Corpus, + /// One artifact per tier 1 target of `platforms.toml`. + Platform, + /// Named by the contract, made by nobody yet, and this is the + /// milestone that changes that. + Later { milestone: String }, +} + +impl Made { + /// What the assemble step prints, and what `--list` says a row is. + pub fn kind(&self) -> &'static str { + match self { + Made::File { .. } => "file", + Made::Corpus => "corpus", + Made::Platform => "platform", + Made::Later { .. } => "later", + } + } +} + +/// One row of the contract. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Artifact { + /// The name it is published under, which may hold `` or + /// ``. + pub name: String, + pub made: Made, + /// The repositories that fetch it. A row nobody fetches is a row + /// the release carries for its own amusement. + pub consumers: Vec, + pub doc: String, + pub line: usize, +} + +impl Artifact { + /// Whether a release publishes this today. + pub fn published(&self) -> bool { + !matches!(self.made, Made::Later { .. }) + } + + /// The name with its placeholders filled in. + pub fn expand(&self, version: &str, target: &str) -> String { + self.name + .replace("", version) + .replace("", target) + } +} + +/// One artifact the assemble step produced. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Shipped { + pub name: String, + pub bytes: u64, + pub kind: &'static str, +} + +impl std::fmt::Display for Shipped { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + f, + "{:<42} {:>7} KiB {}", + self.name, + self.bytes.div_ceil(1024), + self.kind + ) + } +} + +/// What the table says about the tree it is in. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Note { + /// A `file` row whose file is not there. The release would publish + /// the name and nothing under it. + Absent { + name: String, + from: String, + line: usize, + }, + /// A repository the split created that fetches nothing. Either the + /// contract forgot it or it should not have been split out. + Idle { repo: String }, + /// The release workflow does not run the table, which makes the + /// table a document rather than a contract. + Unheld { command: String }, +} + +impl std::fmt::Display for Note { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Note::Absent { name, from, line } => write!( + f, + "{PATH}:{line}: {name} is published from {from}, which is not in this tree" + ), + Note::Idle { repo } => write!( + f, + "{PATH}: {repo} is one of the nine repositories and consumes nothing a release publishes" + ), + Note::Unheld { command } => write!( + f, + "{WORKFLOW}: nothing runs `{command}`, so {PATH} is a document and not a contract" + ), + } + } +} + +/// A release directory read back against the contract. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Fault { + /// A name the contract publishes and the directory does not hold. + Missing { name: String }, + /// A name the directory holds and nothing wrote anything into. + Empty { name: String }, + /// A file in the release that no row accounts for. + Stranger { file: String }, +} + +impl std::fmt::Display for Fault { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Fault::Missing { name } => write!(f, "{name} is in {PATH} and not in the release"), + Fault::Empty { name } => write!(f, "{name} is in the release and is empty"), + Fault::Stranger { file } => { + write!(f, "{file} is in the release and not in {PATH}") + } + } + } +} + +/// The contract. +#[derive(Debug, Clone)] +pub struct Table { + pub doc: String, + pub audited: String, + pub artifacts: Vec, +} + +impl Table { + /// Reads the table at `path`. + pub fn load(path: &Path) -> Result { + let text = std::fs::read_to_string(path) + .map_err(|e| format!("reading {}: {e}", path.display()))?; + Table::parse(&text).map_err(|e| format!("{}: {e}", path.display())) + } + + /// Reads and validates the table. + pub fn parse(text: &str) -> Result { + let doc = Doc::parse(text)?; + if let Some(key) = doc.root.unknown(&["schema", "doc", "audited"]).first() { + return Err(format!("a table has no key {key:?}")); + } + if let Some(name) = doc.unknown_arrays(&["artifact"]).first() { + return Err(format!("a table has no [[{name}]]")); + } + match doc.root.int("schema") { + Some(SCHEMA) => {} + found => { + return Err(format!( + "this reader reads schema {SCHEMA} and the file says {found:?}" + )); + } + } + let doc_text = doc + .root + .str("doc") + .ok_or("a table with no doc")? + .to_string(); + let audited = doc + .root + .str("audited") + .ok_or("a table that does not say when it was audited")? + .to_string(); + + let mut artifacts = Vec::new(); + let mut seen: BTreeMap = BTreeMap::new(); + for table in doc.array("artifact") { + let line = table.line; + if let Some(key) = table + .unknown(&["name", "made", "from", "milestone", "consumers", "doc"]) + .first() + { + return Err(format!("line {line}: an artifact has no key {key:?}")); + } + let name = table + .str("name") + .ok_or_else(|| format!("line {line}: an artifact with no name"))? + .to_string(); + if name.contains('/') { + return Err(format!( + "line {line}: {name} is published as a file in one directory, so it has no /" + )); + } + let from = table.str("from").map(str::to_string); + let milestone = table.str("milestone").map(str::to_string); + let made = match table.str("made") { + Some("file") => Made::File { + from: from + .clone() + .ok_or_else(|| format!("line {line}: {name} is a file with no from"))?, + }, + Some("corpus") => Made::Corpus, + Some("platform") => { + if !name.contains("") { + return Err(format!( + "line {line}: {name} is one artifact per platform and its name is the \ + same for all of them" + )); + } + Made::Platform + } + Some("later") => Made::Later { + milestone: milestone.clone().ok_or_else(|| { + format!("line {line}: {name} is made later and by no milestone") + })?, + }, + found => { + return Err(format!( + "line {line}: {name} is made {found:?}, and the ways are file, corpus, \ + platform and later" + )); + } + }; + // The three keys that only one kind of row has. A `from` on + // a corpus row is somebody expecting a copy that will not + // happen, and it is worth more as an error than as a + // comment nobody reads. + if from.is_some() && !matches!(made, Made::File { .. }) { + return Err(format!( + "line {line}: {name} is made {} and has a from, which only a file has", + made.kind() + )); + } + if milestone.is_some() && !matches!(made, Made::Later { .. }) { + return Err(format!( + "line {line}: {name} is made {} and names a milestone, which is what a row \ + nothing makes yet does", + made.kind() + )); + } + if let Made::Later { milestone } = &made + && !is_milestone(milestone) + { + return Err(format!( + "line {line}: {name} waits for {milestone:?}, and the milestones are DX0 to \ + DX6 and D0 to D6" + )); + } + let consumers: Vec = table + .list("consumers") + .ok_or_else(|| format!("line {line}: {name} is fetched by nobody"))? + .iter() + .map(|c| c.to_string()) + .collect(); + if consumers.is_empty() { + return Err(format!("line {line}: {name} is fetched by nobody")); + } + for consumer in &consumers { + if consumer == MAKER { + return Err(format!( + "line {line}: {MAKER} makes {name} and is not a consumer of it" + )); + } + if !REPOS.contains(&consumer.as_str()) { + return Err(format!( + "line {line}: {name} is fetched by {consumer:?}, which is not one of the \ + nine repositories" + )); + } + } + let doc = table + .str("doc") + .ok_or_else(|| format!("line {line}: {name} has no doc"))? + .to_string(); + if let Some(before) = seen.insert(name.clone(), line) { + return Err(format!( + "line {line}: {name} is written twice, and first on line {before}" + )); + } + artifacts.push(Artifact { + name, + made, + consumers, + doc, + line, + }); + } + if artifacts.is_empty() { + return Err("a release that publishes nothing".to_string()); + } + + Ok(Table { + doc: doc_text, + audited, + artifacts, + }) + } + + /// The row of this name, as written rather than expanded. + pub fn artifact(&self, name: &str) -> Option<&Artifact> { + self.artifacts.iter().find(|a| a.name == name) + } + + /// Every name a release of `version` publishes, in table order, + /// with the platform row expanded across `targets`. + pub fn names(&self, version: &str, targets: &[String]) -> Vec { + let mut out = Vec::new(); + for artifact in self.artifacts.iter().filter(|a| a.published()) { + match artifact.made { + Made::Platform => out.extend(targets.iter().map(|t| artifact.expand(version, t))), + _ => out.push(artifact.expand(version, "")), + } + } + out + } + + /// Checks the tree under `root` against the table. + pub fn check(&self, root: &Path) -> Result, String> { + let mut notes = Vec::new(); + for artifact in &self.artifacts { + if let Made::File { from } = &artifact.made + && !root.join(from).exists() + { + notes.push(Note::Absent { + name: artifact.name.clone(), + from: from.clone(), + line: artifact.line, + }); + } + } + for repo in REPOS.iter().filter(|r| **r != MAKER) { + if !self + .artifacts + .iter() + .any(|a| a.consumers.iter().any(|c| c == repo)) + { + notes.push(Note::Idle { + repo: (*repo).to_string(), + }); + } + } + // A workflow that assembles from somewhere else is the second + // list this table exists to prevent, so the check is that the + // release runs these two commands and not that it mentions the + // artifacts. + let text = std::fs::read_to_string(root.join(WORKFLOW)).unwrap_or_default(); + for command in ["artifacts --assemble", "artifacts --verify"] { + if !text.contains(command) { + notes.push(Note::Unheld { + command: command.to_string(), + }); + } + } + Ok(notes) + } + + /// Gathers a release of `version` into `out`, from the tree at + /// `root` and the platform builds under `built`. + /// + /// What is assembled is the table and nothing else, which is what + /// makes the table the list. A row nothing makes yet is skipped and + /// reported rather than silently absent, since "the release has six + /// files" should be a sentence somebody can check against seven + /// rows. + pub fn assemble( + &self, + root: &Path, + built: &Path, + out: &Path, + version: &str, + targets: &[String], + ) -> Result, String> { + if version.is_empty() || version.contains(['/', '\\', ' ']) { + return Err(format!("{version:?} is not a version")); + } + std::fs::create_dir_all(out).map_err(|e| format!("creating {}: {e}", out.display()))?; + let mut made = Vec::new(); + for artifact in self.artifacts.iter().filter(|a| a.published()) { + match &artifact.made { + Made::File { from } => { + let name = artifact.expand(version, ""); + let source = root.join(from); + let bytes = std::fs::read(&source) + .map_err(|e| format!("reading {}: {e}", source.display()))?; + made.push(write(out, &name, &bytes, "file")?); + } + Made::Corpus => { + let readme = root.join(README); + let packed = corpus::pack( + &root.join(CASES), + readme.exists().then_some(readme.as_path()), + version, + )?; + let name = artifact.expand(version, ""); + made.push(write(out, &name, &packed.archive, "corpus")?); + } + Made::Platform => { + for target in targets { + let name = artifact.expand(version, target); + // The build uploaded one directory per target + // under the name the release publishes it as, + // so the two agree by construction rather than + // by a second convention written down here. + let prefix = format!("libzu-{target}"); + let archive = pack(&built.join(&prefix), &prefix)?; + made.push(write(out, &name, &archive, "platform")?); + } + } + Made::Later { .. } => unreachable!("published rows only"), + } + } + Ok(made) + } + + /// Reads a release directory back against the table. + /// + /// In both directions, for the same reason every other table in + /// this repository is: a missing artifact is a consumer's failure + /// tomorrow, and an unexpected one is a file somebody will fetch + /// and nothing promises. + pub fn verify( + &self, + dir: &Path, + version: &str, + targets: &[String], + ) -> Result<(Vec, Vec), String> { + let mut found: BTreeMap = BTreeMap::new(); + let entries = + std::fs::read_dir(dir).map_err(|e| format!("reading {}: {e}", dir.display()))?; + for entry in entries { + let entry = entry.map_err(|e| format!("reading {}: {e}", dir.display()))?; + let name = entry.file_name().to_string_lossy().to_string(); + let meta = entry + .metadata() + .map_err(|e| format!("reading {name}: {e}"))?; + if meta.is_dir() { + return Err(format!( + "{}/{name} is a directory, and a release is files", + dir.display() + )); + } + found.insert(name, meta.len()); + } + + let mut shipped = Vec::new(); + let mut faults = Vec::new(); + for name in self.names(version, targets) { + match found.remove(&name) { + None => faults.push(Fault::Missing { name }), + Some(0) => faults.push(Fault::Empty { name }), + Some(bytes) => shipped.push(Shipped { + name, + bytes, + kind: "published", + }), + } + } + faults.extend(found.into_keys().map(|file| Fault::Stranger { file })); + Ok((shipped, faults)) + } +} + +/// The tier 1 targets, which are the platform table's business and not +/// this one's. +pub fn tier1(root: &Path) -> Result, String> { + let table = platforms::Table::load(&root.join(platforms::PATH))?; + Ok(table + .platforms + .iter() + .filter(|p| p.tier == platforms::TIER1) + .map(|p| p.target.clone()) + .collect()) +} + +/// Whether this is a milestone of the two programs, which is what a row +/// nothing makes yet has to wait for. +fn is_milestone(name: &str) -> bool { + let digit = |rest: &str| matches!(rest.parse::(), Ok(0..=6)); + match name.strip_prefix("DX") { + Some(rest) => digit(rest), + None => name.strip_prefix('D').is_some_and(digit), + } +} + +/// One file of the release, written and weighed. +fn write(out: &Path, name: &str, bytes: &[u8], kind: &'static str) -> Result { + let path = out.join(name); + std::fs::write(&path, bytes).map_err(|e| format!("writing {}: {e}", path.display()))?; + Ok(Shipped { + name: name.to_string(), + bytes: bytes.len() as u64, + kind, + }) +} + +/// A directory packed under `prefix`, which is how a platform's build +/// becomes one file of the release. +/// +/// Everything the build uploaded goes in, sorted, so that adding a file +/// to the workflow's upload is the whole of adding it to the artifact. +/// An empty directory is an error, because a platform that built +/// nothing is otherwise the cheapest way to publish a release without +/// it. +fn pack(dir: &Path, prefix: &str) -> Result, String> { + let mut files = Vec::new(); + walk(dir, dir, &mut files)?; + if files.is_empty() { + return Err(format!("{} built nothing", dir.display())); + } + files.sort(); + let mut under = Vec::with_capacity(files.len()); + for name in files { + let bytes = std::fs::read(dir.join(&name)) + .map_err(|e| format!("reading {}: {e}", dir.join(&name).display()))?; + under.push((format!("{prefix}/{name}"), bytes)); + } + tarball::compress(&tarball::tar(&under)?) +} + +/// Every file under `dir`, named relative to `root`, with `/` on every +/// platform because that is what a tar holds. +fn walk(root: &Path, dir: &Path, out: &mut Vec) -> Result<(), String> { + let entries = std::fs::read_dir(dir).map_err(|e| format!("reading {}: {e}", dir.display()))?; + for entry in entries { + let path = entry + .map_err(|e| format!("reading {}: {e}", dir.display()))? + .path(); + if path.is_dir() { + walk(root, &path, out)?; + continue; + } + let relative = path + .strip_prefix(root) + .map_err(|e| format!("{}: {e}", path.display()))?; + let name: Vec = relative + .components() + .map(|c| c.as_os_str().to_string_lossy().to_string()) + .collect(); + out.push(name.join("/")); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + use std::path::PathBuf; + + const TABLE: &str = concat!( + "schema = 1\n", + "doc = \"What a release publishes.\"\n", + "audited = \"2026-08-16\"\n", + "\n", + "[[artifact]]\n", + "name = \"libzu-.tar.zst\"\n", + "made = \"platform\"\n", + "consumers = [\"zu-c\"]\n", + "doc = \"One per tier 1 target.\"\n", + "\n", + "[[artifact]]\n", + "name = \"zu.h\"\n", + "made = \"file\"\n", + "from = \"crates/zu-capi/include/zu.h\"\n", + "consumers = [\"zu-c\", \"zu-go\"]\n", + "doc = \"The C ABI.\"\n", + "\n", + "[[artifact]]\n", + "name = \"gql.json\"\n", + "made = \"later\"\n", + "milestone = \"D2\"\n", + "consumers = [\"zu-web\"]\n", + "doc = \"The language as data.\"\n", + ); + + fn table() -> Table { + Table::parse(TABLE).expect("the table parses") + } + + /// A scratch tree, so a test can assemble into something. + fn scratch(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("zu-artifacts-{}-{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("the scratch directory is writable"); + dir + } + + #[test] + fn a_row_is_read_as_written() { + let table = table(); + let header = table.artifact("zu.h").expect("the header is a row"); + assert_eq!( + header.made, + Made::File { + from: "crates/zu-capi/include/zu.h".to_string() + } + ); + assert!(header.published()); + assert_eq!(header.consumers, ["zu-c", "zu-go"]); + let gql = table.artifact("gql.json").expect("gql.json is a row"); + assert!(!gql.published(), "nothing makes it yet"); + assert_eq!(gql.made.kind(), "later"); + } + + #[test] + fn a_name_is_expanded_per_version_and_per_target() { + let targets = ["x86_64-apple-darwin".to_string()]; + let names = table().names("0.5.0", &targets); + assert_eq!(names, ["libzu-x86_64-apple-darwin.tar.zst", "zu.h"]); + } + + #[test] + fn a_row_nothing_makes_yet_needs_the_milestone_that_will() { + let text = TABLE.replace("milestone = \"D2\"\n", ""); + let error = Table::parse(&text).expect_err("later and no milestone"); + assert!(error.contains("by no milestone"), "{error}"); + let text = TABLE.replace("milestone = \"D2\"", "milestone = \"D9\""); + let error = Table::parse(&text).expect_err("a milestone that does not exist"); + assert!(error.contains("DX0 to DX6"), "{error}"); + } + + #[test] + fn a_key_that_belongs_to_another_kind_of_row_is_refused() { + let text = TABLE.replace( + "made = \"later\"\nmilestone = \"D2\"", + "made = \"later\"\nmilestone = \"D2\"\nfrom = \"docs/gql.json\"", + ); + let error = Table::parse(&text).expect_err("a later row with a from"); + assert!(error.contains("which only a file has"), "{error}"); + + let text = TABLE.replace( + "made = \"file\"\nfrom = \"crates/zu-capi/include/zu.h\"", + "made = \"file\"\nfrom = \"crates/zu-capi/include/zu.h\"\nmilestone = \"DX1\"", + ); + let error = Table::parse(&text).expect_err("a file row with a milestone"); + assert!(error.contains("nothing makes yet does"), "{error}"); + } + + #[test] + fn a_file_row_with_nothing_to_copy_is_refused() { + let text = TABLE.replace("from = \"crates/zu-capi/include/zu.h\"\n", ""); + let error = Table::parse(&text).expect_err("a file with no from"); + assert!(error.contains("a file with no from"), "{error}"); + } + + #[test] + fn a_platform_row_that_names_one_artifact_is_refused() { + let text = TABLE.replace("libzu-.tar.zst", "libzu.tar.zst"); + let error = Table::parse(&text).expect_err("one name for seven platforms"); + assert!(error.contains("same for all of them"), "{error}"); + } + + #[test] + fn an_artifact_the_maker_consumes_is_refused() { + let text = TABLE.replace("consumers = [\"zu-web\"]", "consumers = [\"zu\"]"); + let error = Table::parse(&text).expect_err("zu consuming its own release"); + assert!(error.contains("is not a consumer"), "{error}"); + } + + #[test] + fn a_consumer_that_is_not_a_repository_is_refused() { + let text = TABLE.replace("consumers = [\"zu-web\"]", "consumers = [\"zu-perl\"]"); + let error = Table::parse(&text).expect_err("a tenth repository"); + assert!(error.contains("nine repositories"), "{error}"); + } + + #[test] + fn an_artifact_written_twice_is_refused() { + let at = TABLE.find("[[artifact]]").expect("a row"); + let text = format!("{TABLE}\n{}", &TABLE[at..]); + let error = Table::parse(&text).expect_err("two rows with one name"); + assert!(error.contains("is written twice"), "{error}"); + } + + #[test] + fn a_schema_this_reader_does_not_read_is_refused() { + let error = Table::parse(&TABLE.replace("schema = 1", "schema = 2")).expect_err("schema 2"); + assert!(error.contains("reads schema 1"), "{error}"); + } + + #[test] + fn a_file_the_tree_does_not_hold_is_reported() { + let root = scratch("absent"); + let notes = table().check(&root).expect("the check runs"); + assert!( + notes + .iter() + .any(|n| matches!(n, Note::Absent { name, .. } if name == "zu.h")), + "{notes:?}" + ); + assert!( + notes.iter().any(|n| matches!(n, Note::Unheld { .. })), + "a tree with no release workflow holds the table to nothing: {notes:?}" + ); + let _ = std::fs::remove_dir_all(&root); + } + + #[test] + fn a_release_assembles_and_verifies_against_the_table() { + let dir = scratch("assemble"); + let root = dir.join("tree"); + let built = dir.join("built"); + let out = dir.join("dist"); + std::fs::create_dir_all(root.join("crates/zu-capi/include")).expect("writes"); + std::fs::write(root.join("crates/zu-capi/include/zu.h"), "#define ZU 1\n").expect("writes"); + std::fs::create_dir_all(built.join("libzu-x86_64-apple-darwin")).expect("writes"); + std::fs::write( + built.join("libzu-x86_64-apple-darwin/libzu.dylib"), + vec![9u8; 4096], + ) + .expect("writes"); + + let targets = ["x86_64-apple-darwin".to_string()]; + let made = table() + .assemble(&root, &built, &out, "0.5.0", &targets) + .expect("assembles"); + assert_eq!(made.len(), 2, "{made:?}"); + assert_eq!(made[0].name, "libzu-x86_64-apple-darwin.tar.zst"); + assert_eq!(made[1].kind, "file"); + + let (shipped, faults) = table().verify(&out, "0.5.0", &targets).expect("verifies"); + assert_eq!(faults, [] as [Fault; 0]); + assert_eq!(shipped.len(), 2); + assert!(shipped.iter().all(|s| s.bytes > 0)); + + // The platform archive holds what the build uploaded, under the + // name the release publishes it as. + let archive = std::fs::read(out.join("libzu-x86_64-apple-darwin.tar.zst")).expect("reads"); + let tar = zstd::bulk::decompress(&archive, 1 << 20).expect("unpacks"); + let names: Vec = tarball::entries(&tar).into_iter().map(|(n, _)| n).collect(); + assert_eq!(names, ["libzu-x86_64-apple-darwin/libzu.dylib"]); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_release_missing_an_artifact_says_which_one() { + let dir = scratch("missing"); + std::fs::write(dir.join("zu.h"), "#define ZU 1\n").expect("writes"); + let targets = ["x86_64-apple-darwin".to_string()]; + let (_, faults) = table().verify(&dir, "0.5.0", &targets).expect("verifies"); + assert_eq!( + faults, + [Fault::Missing { + name: "libzu-x86_64-apple-darwin.tar.zst".to_string() + }] + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_file_nothing_promised_is_reported_and_so_is_an_empty_one() { + let dir = scratch("stranger"); + std::fs::write(dir.join("zu.h"), "").expect("writes"); + std::fs::write(dir.join("secrets.env"), "x=1\n").expect("writes"); + let (_, faults) = table().verify(&dir, "0.5.0", &[]).expect("verifies"); + assert!( + faults.contains(&Fault::Empty { + name: "zu.h".to_string() + }), + "{faults:?}" + ); + assert!( + faults.contains(&Fault::Stranger { + file: "secrets.env".to_string() + }), + "{faults:?}" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_platform_that_built_nothing_stops_the_release() { + let dir = scratch("nothing"); + let built = dir.join("built"); + std::fs::create_dir_all(built.join("libzu-x86_64-apple-darwin")).expect("writes"); + std::fs::create_dir_all(dir.join("tree/crates/zu-capi/include")).expect("writes"); + std::fs::write(dir.join("tree/crates/zu-capi/include/zu.h"), "x").expect("writes"); + let error = table() + .assemble( + &dir.join("tree"), + &built, + &dir.join("dist"), + "0.5.0", + &["x86_64-apple-darwin".to_string()], + ) + .expect_err("an empty build directory"); + assert!(error.contains("built nothing"), "{error}"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_version_that_would_become_a_path_is_refused() { + let dir = scratch("version"); + for version in ["", "../etc", "1.0 rc1"] { + assert!( + table() + .assemble(&dir, &dir, &dir.join("dist"), version, &[]) + .is_err(), + "{version:?}" + ); + } + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn the_committed_contract_is_what_this_tree_publishes() { + let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); + let table = Table::load(&root.join(PATH)).expect("the committed table loads"); + let notes = table.check(&root).expect("the check runs"); + assert!(notes.is_empty(), "{notes:#?}"); + + // dx/14 section 6 lists what a release publishes, and it is + // this list. A row that quietly appeared or went is the + // contract changing without anybody saying so. + let mut names: Vec<&str> = table.artifacts.iter().map(|a| a.name.as_str()).collect(); + names.sort_unstable(); + assert_eq!( + names, + [ + "cli.json", + "conformance-.tar.zst", + "errors.json", + "gql.json", + "libzu-.tar.zst", + "model.json", + "zu.h", + ] + ); + + // The seven platform artifacts are the platform table's seven, + // which is the whole reason this row is expanded rather than + // written out. + let targets = tier1(&root).expect("the platform table loads"); + assert_eq!(targets.len(), 7); + assert_eq!(table.names("0.5.0", &targets).len(), 7 + 3); + } +} diff --git a/crates/xtask/src/corpus.rs b/crates/xtask/src/corpus.rs index d27c50aa..85098ad4 100644 --- a/crates/xtask/src/corpus.rs +++ b/crates/xtask/src/corpus.rs @@ -17,28 +17,24 @@ //! open. //! //! Two properties are worth more than they cost. The archive is -//! reproducible, because every field a tar header can carry a timestamp -//! or a user id in is fixed, so the same cases produce the same bytes -//! on any machine on any day and a mirror can be compared against a -//! release rather than trusted. And the packer parses every case before -//! it ships one, so a corpus that does not load cannot become an -//! artifact that eight repositories fail on. +//! reproducible, which is the shared tar writer's doing and the reason +//! a mirror can be compared against a release rather than trusted. And +//! the packer parses every case before it ships one, so a corpus that +//! does not load cannot become an artifact that eight repositories fail +//! on. use std::path::Path; use zu_json::Json; +use crate::tarball; + /// The manifest's schema version, which moves when the shape of the /// archive changes and not when the cases do. A client that unpacks an /// artifact it does not understand should say so rather than run half /// of it. pub const SCHEMA: i64 = 1; -/// The compression level. The artifact is packed once per release and -/// unpacked on every CI run of nine repositories, so the trade is -/// entirely one way. -const LEVEL: i32 = 19; - /// One case file in the archive. #[derive(Debug, Clone, PartialEq, Eq)] pub struct Entry { @@ -121,17 +117,15 @@ pub fn pack(dir: &Path, readme: Option<&Path>, version: &str) -> Result)> = files + .into_iter() + .map(|(name, bytes)| (format!("{prefix}/{name}"), bytes)) + .collect(); + let tar = tarball::tar(&under)?; + let archive = tarball::compress(&tar)?; Ok(Packed { archive, tar, @@ -171,58 +165,6 @@ fn manifest(version: &str, entries: &[Entry]) -> String { text } -/// One ustar file header and its content, padded to the block size. -/// -/// Every field that could carry a timestamp, a user, or a permission -/// bit from the machine that ran the pack is fixed instead. That is -/// what makes two packs of the same cases the same bytes, and it costs -/// nothing here: an archive of text files nobody executes has no use -/// for the mode, and a modification time that is the moment of the -/// release is a difference between two mirrors of one release. -fn append(out: &mut Vec, name: &str, body: &[u8]) -> Result<(), String> { - if name.len() > 99 { - return Err(format!( - "{name:?} is {} bytes and a ustar name holds 99", - name.len() - )); - } - let start = out.len(); - let mut header = [0u8; 512]; - header[..name.len()].copy_from_slice(name.as_bytes()); - // mode, uid, gid, size, mtime: octal, NUL terminated, fixed. - write_octal(&mut header[100..108], 0o644); - write_octal(&mut header[108..116], 0); - write_octal(&mut header[116..124], 0); - write_octal(&mut header[124..136], body.len() as u64); - write_octal(&mut header[136..148], 0); - // The checksum is computed with its own field read as spaces, - // which is the one piece of tar that cannot be described without - // saying it out loud. - header[148..156].fill(b' '); - header[156] = b'0'; - header[257..263].copy_from_slice(b"ustar\0"); - header[263..265].copy_from_slice(b"00"); - let sum: u32 = header.iter().map(|&b| u32::from(b)).sum(); - write_octal(&mut header[148..155], u64::from(sum)); - header[155] = b' '; - - out.extend_from_slice(&header); - out.extend_from_slice(body); - let padding = (512 - body.len() % 512) % 512; - out.resize(out.len() + padding, 0); - debug_assert_eq!((out.len() - start) % 512, 0); - Ok(()) -} - -/// An octal number, right aligned in `field` with a trailing NUL, -/// which is how every numeric field in a tar header is written. -fn write_octal(field: &mut [u8], value: u64) { - let digits = field.len() - 1; - let text = format!("{value:0digits$o}"); - field[..digits].copy_from_slice(&text.as_bytes()[text.len() - digits..]); - field[digits] = 0; -} - #[cfg(test)] mod tests { use super::*; @@ -252,28 +194,7 @@ mod tests { dir } - /// The files in a tar, which is what the archive promised and the - /// only way to check it kept the promise. - fn entries(tar: &[u8]) -> Vec<(String, Vec)> { - let mut out = Vec::new(); - let mut i = 0; - while i + 512 <= tar.len() { - let header = &tar[i..i + 512]; - if header.iter().all(|&b| b == 0) { - break; - } - let end = header.iter().position(|&b| b == 0).unwrap_or(100); - let name = String::from_utf8(header[..end].to_vec()).expect("a name is text"); - let size = std::str::from_utf8(&header[124..135]) - .ok() - .and_then(|s| u64::from_str_radix(s.trim_end_matches([' ', '\0']), 8).ok()) - .expect("a size is octal") as usize; - i += 512; - out.push((name, tar[i..i + size].to_vec())); - i += size.div_ceil(512) * 512; - } - out - } + use tarball::entries; #[test] fn the_archive_holds_the_cases_byte_for_byte() { diff --git a/crates/xtask/src/lib.rs b/crates/xtask/src/lib.rs index 3145fa3b..4948770a 100644 --- a/crates/xtask/src/lib.rs +++ b/crates/xtask/src/lib.rs @@ -3,11 +3,13 @@ //! drive the normalizer directly, without nightly and without cargo. pub mod apimap; +pub mod artifacts; pub mod corpus; pub mod fixture; pub mod model; pub mod pins; pub mod platforms; pub mod rustdoc; +pub mod tarball; pub mod terms; pub mod toml; diff --git a/crates/xtask/src/main.rs b/crates/xtask/src/main.rs index 58660abf..6e78ff8d 100644 --- a/crates/xtask/src/main.rs +++ b/crates/xtask/src/main.rs @@ -17,7 +17,7 @@ //! is nightly-only, and `model --check` needs it too. The map check //! reads two committed files and needs nothing. -use xtask::{apimap, corpus, model, pins, platforms, rustdoc, terms}; +use xtask::{apimap, artifacts, corpus, model, pins, platforms, rustdoc, terms}; use std::path::{Path, PathBuf}; use std::process::ExitCode; @@ -56,6 +56,15 @@ cargo xtask platforms [--table PATH] [--list] [--measure DIR --target TARGET] --measure DIR weigh what a build put in DIR against the size budgets --target TARGET the target that build was for, which says what the files are called +cargo xtask artifacts [--table PATH] [--list] [--assemble DIR] [--verify DIR] [--built DIR] [--version V] + + --table PATH the artifact contract (default artifacts.toml) + --list print `madenameconsumers` for every row, and check nothing + --assemble DIR gather a release into DIR, from this tree and the platform builds + --verify DIR read a release directory back against the contract + --built DIR where the platform jobs' artifacts were downloaded (default built) + --version V the version being released (default this workspace's) + cargo xtask terms [--table PATH] [--list] [PATH ...] --table PATH the terminology table (default zu-web/style/zu/terms.yml, here or one level up) @@ -77,6 +86,7 @@ fn main() -> ExitCode { Some("corpus-pack") => run(corpus_pack_command(&args[1..])), Some("pins") => run(pins_command(&args[1..])), Some("platforms") => run(platforms_command(&args[1..])), + Some("artifacts") => run(artifacts_command(&args[1..])), Some("terms") => run(terms_command(&args[1..])), Some("--help" | "-h") | None => { print!("{USAGE}"); @@ -366,6 +376,140 @@ fn platforms_command(args: &[String]) -> Result { Ok(ExitCode::FAILURE) } +fn artifacts_command(args: &[String]) -> Result { + let mut path = PathBuf::from(artifacts::PATH); + let mut list = false; + let mut assemble: Option = None; + let mut verify: Option = None; + let mut built = PathBuf::from("built"); + // Same rule the corpus packer follows: the version comes from the + // build rather than from a habit, so an artifact cannot be named + // for a version it does not hold. + let mut version = env!("CARGO_PKG_VERSION").to_string(); + let mut i = 0; + while i < args.len() { + match args[i].as_str() { + "--list" => list = true, + "--table" => { + path = PathBuf::from(args.get(i + 1).ok_or("--table wants a path")?); + i += 1; + } + "--assemble" => { + assemble = Some(PathBuf::from( + args.get(i + 1).ok_or("--assemble wants a path")?, + )); + i += 1; + } + "--verify" => { + verify = Some(PathBuf::from( + args.get(i + 1).ok_or("--verify wants a path")?, + )); + i += 1; + } + "--built" => { + built = PathBuf::from(args.get(i + 1).ok_or("--built wants a path")?); + i += 1; + } + "--version" => { + version = args.get(i + 1).ok_or("--version wants a version")?.clone(); + i += 1; + } + other => return Err(format!("no option {other:?}\n\n{USAGE}")), + } + i += 1; + } + let table = artifacts::Table::load(&path)?; + + if list { + for artifact in &table.artifacts { + println!( + "{}\t{}\t{}", + artifact.made.kind(), + artifact.name, + artifact.consumers.join(" ") + ); + } + return Ok(ExitCode::SUCCESS); + } + + // The table sits at the root of the tree it describes, the same as + // the two tables beside it. + let root = path.parent().unwrap_or(Path::new(".")).to_path_buf(); + let targets = artifacts::tier1(&root)?; + + if let Some(out) = &assemble { + let made = table.assemble(&root, &built, out, &version, &targets)?; + for one in &made { + println!("{one}"); + } + let later: Vec<&str> = table + .artifacts + .iter() + .filter(|a| !a.published()) + .map(|a| a.name.as_str()) + .collect(); + // A release of six files against a contract of seven rows is a + // sentence worth printing, because the alternative is a reader + // counting and wondering. + println!( + "{}: {} files for {version}, and {} the contract names that nothing makes yet ({})", + out.display(), + made.len(), + later.len(), + later.join(", ") + ); + } + + if let Some(dir) = &verify { + let (shipped, faults) = table.verify(dir, &version, &targets)?; + for one in &shipped { + println!("{one}"); + } + if !faults.is_empty() { + for fault in &faults { + eprintln!("{fault}"); + } + eprintln!( + "\n{} to fix in {}. What a release publishes is {}, in both directions.", + faults.len(), + dir.display(), + path.display() + ); + return Ok(ExitCode::FAILURE); + } + println!( + "{}: {} artifacts for {version}, every one of them in {}", + dir.display(), + shipped.len(), + path.display() + ); + } + + if assemble.is_none() && verify.is_none() { + let notes = table.check(&root)?; + if !notes.is_empty() { + for note in ¬es { + eprintln!("{note}"); + } + eprintln!( + "\n{} to fix. What a release publishes is {}, and the release workflow assembles \ + from it rather than from a list of its own.", + notes.len(), + path.display() + ); + return Ok(ExitCode::FAILURE); + } + let published = table.artifacts.iter().filter(|a| a.published()).count(); + println!( + "{} artifacts, {published} of them published today across {} platforms, audited {}", + table.artifacts.len(), + targets.len(), + table.audited + ); + } + Ok(ExitCode::SUCCESS) +} + /// The prose this repository publishes: the documentation, and the doc /// comments that become reference pages. The engine's own source is in /// here because a doc comment is a reference page, not because a diff --git a/crates/xtask/src/tarball.rs b/crates/xtask/src/tarball.rs new file mode 100644 index 00000000..98cfc8c3 --- /dev/null +++ b/crates/xtask/src/tarball.rs @@ -0,0 +1,147 @@ +//! The one tar writer, shared by everything this repository ships. +//! +//! Two release artifacts are archives: the conformance corpus and each +//! platform's `libzu` build. A second implementation of a format this +//! dull would be a second set of headers to get subtly wrong, and the +//! wrongness would surface in somebody else's language on the day they +//! unpacked a release. +//! +//! The archives are reproducible. Every field a tar header can carry a +//! timestamp, a user id or a permission bit from the packing machine in +//! is fixed instead, so the same inputs are the same bytes on any +//! machine on any day and a mirror can be compared against a release +//! rather than trusted. + +/// The compression level. An artifact is packed once per release and +/// unpacked on every CI run of nine repositories, so the trade is +/// entirely one way. +pub const LEVEL: i32 = 19; + +/// A tar of these files, in this order, terminated. +pub fn tar(files: &[(String, Vec)]) -> Result, String> { + let mut out = Vec::new(); + for (name, bytes) in files { + append(&mut out, name, bytes)?; + } + // Two zero blocks end a tar, and readers that check for them are + // the reason a truncated archive is detected rather than silently + // short. + out.extend_from_slice(&[0u8; 1024]); + Ok(out) +} + +/// The zstd of it, which is what ships. +pub fn compress(tar: &[u8]) -> Result, String> { + zstd::bulk::compress(tar, LEVEL).map_err(|e| format!("compressing the archive: {e}")) +} + +/// One ustar file header and its content, padded to the block size. +fn append(out: &mut Vec, name: &str, body: &[u8]) -> Result<(), String> { + if name.len() > 99 { + return Err(format!( + "{name:?} is {} bytes and a ustar name holds 99", + name.len() + )); + } + let start = out.len(); + let mut header = [0u8; 512]; + header[..name.len()].copy_from_slice(name.as_bytes()); + // mode, uid, gid, size, mtime: octal, NUL terminated, fixed. + write_octal(&mut header[100..108], 0o644); + write_octal(&mut header[108..116], 0); + write_octal(&mut header[116..124], 0); + write_octal(&mut header[124..136], body.len() as u64); + write_octal(&mut header[136..148], 0); + // The checksum is computed with its own field read as spaces, + // which is the one piece of tar that cannot be described without + // saying it out loud. + header[148..156].fill(b' '); + header[156] = b'0'; + header[257..263].copy_from_slice(b"ustar\0"); + header[263..265].copy_from_slice(b"00"); + let sum: u32 = header.iter().map(|&b| u32::from(b)).sum(); + write_octal(&mut header[148..155], u64::from(sum)); + header[155] = b' '; + + out.extend_from_slice(&header); + out.extend_from_slice(body); + let padding = (512 - body.len() % 512) % 512; + out.resize(out.len() + padding, 0); + debug_assert_eq!((out.len() - start) % 512, 0); + Ok(()) +} + +/// An octal number, right aligned in `field` with a trailing NUL, +/// which is how every numeric field in a tar header is written. +fn write_octal(field: &mut [u8], value: u64) { + let digits = field.len() - 1; + let text = format!("{value:0digits$o}"); + field[..digits].copy_from_slice(&text.as_bytes()[text.len() - digits..]); + field[digits] = 0; +} + +/// The files in a tar, which is what a caller can check an archive +/// against without a tar reader of its own. +pub fn entries(tar: &[u8]) -> Vec<(String, Vec)> { + let mut out = Vec::new(); + let mut i = 0; + while i + 512 <= tar.len() { + let header = &tar[i..i + 512]; + if header.iter().all(|&b| b == 0) { + break; + } + let end = header[..100].iter().position(|&b| b == 0).unwrap_or(100); + let name = String::from_utf8_lossy(&header[..end]).to_string(); + let size = String::from_utf8_lossy(&header[124..135]); + let size = usize::from_str_radix(size.trim_end_matches('\0').trim(), 8).unwrap_or(0); + i += 512; + out.push((name, tar[i..i + size].to_vec())); + i += size.div_ceil(512) * 512; + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn a_file_comes_back_out_byte_for_byte() { + let files = vec![ + ("box/a.txt".to_string(), b"one".to_vec()), + ("box/b.txt".to_string(), vec![7u8; 1500]), + ]; + let read = entries(&tar(&files).expect("a tar of two small files")); + assert_eq!(read, files); + } + + #[test] + fn packing_twice_gives_the_same_bytes() { + let files = vec![("box/a.txt".to_string(), b"one".to_vec())]; + assert_eq!(tar(&files), tar(&files)); + } + + #[test] + fn a_name_longer_than_a_ustar_header_is_refused() { + let files = vec![(format!("box/{}", "n".repeat(120)), b"one".to_vec())]; + let err = tar(&files).expect_err("a name that does not fit"); + assert!(err.contains("ustar name holds 99"), "{err}"); + } + + #[test] + fn an_empty_file_is_a_header_and_no_content() { + let files = vec![("box/empty".to_string(), Vec::new())]; + let tar = tar(&files).expect("a tar of one empty file"); + assert_eq!(tar.len(), 512 + 1024); + assert_eq!(entries(&tar), files); + } + + #[test] + fn compression_is_a_round_trip() { + let files = vec![("box/a.txt".to_string(), b"one two three".repeat(50))]; + let tar = tar(&files).expect("a tar"); + let archive = compress(&tar).expect("compressed"); + let back = zstd::bulk::decompress(&archive, tar.len()).expect("decompressed"); + assert_eq!(back, tar); + } +} diff --git a/docs/10-api-and-tooling.md b/docs/10-api-and-tooling.md index b1ab7af3..56ccff78 100644 --- a/docs/10-api-and-tooling.md +++ b/docs/10-api-and-tooling.md @@ -131,6 +131,18 @@ Every row that can run what it built runs a C program whose only knowledge of zu The same table carries the size ceilings of dx/14 §4, and `cargo xtask platforms --measure` weighs what a build produced against them on every platform. Binary size is a real adoption factor for serverless and mobile targets and it only ever drifts upward, so it is a number with a limit rather than a graph somebody looks at once a quarter. Today `libzu` is 2.3 MiB against a ceiling of 14 and the CLI is 4.6 against 15. A file the build did not produce is an error rather than a zero, since a missing artifact is otherwise the cheapest way to pass a size gate. -## 11. Documentation deliverables (v1.0 gate) +## 11. The release-artifact contract + +A release is a tag on this repository and a run that drives the eight others (dx/14 §6). Every one of them builds against what this one published, which makes the list of what gets published a contract rather than a step in a workflow. A binding that fetches `model.json` for the version it pins and finds nothing cannot tell a release that dropped the artifact from a version that never had it, and the failure surfaces in somebody else's CI a day later, which is the worst place for it. + +`artifacts.toml` is that list, and the release workflow has none of its own: it assembles from the table and reads the directory back against it. A row is a name, where it comes from, the repositories that fetch it, and a sentence saying why they do. There are four ways a row comes to exist, because there are four ways an artifact does. A `file` is a path this tree already holds, `zu.h` and `model.json` being the two. A `corpus` is packed by the packer of §7. A `platform` row is one artifact per tier-1 target, expanded against `platforms.toml`, so the seven move with that table rather than with this one. And a `later` row is an artifact the contract names that nothing makes yet, carrying the milestone that will make it: `cli.json` with D1, `gql.json` and `errors.json` with D2. Naming those early is the point rather than an oversight, since a consumer needs to know what a release will eventually carry and the alternative is eight repositories each guessing. + +`cargo xtask artifacts --assemble` gathers a release into a directory and `--verify` reads it back, in both directions like every other table here: an artifact the contract publishes and the directory does not hold is a consumer's failure tomorrow, and a file in the release that no row accounts for is something somebody will fetch that nothing promises. The check that runs on every pull request is the third direction, holding the table to the tree and to the workflow: a `file` row whose file has moved, one of the nine repositories that fetches nothing, and a release workflow that does not assemble from the table, which is what would turn the table back into a document. Assembling a release of ten files takes about as long as compressing them, and the bookkeeping either side of it is a hundredth of a millisecond. + +A platform's build stages the library, the CLI, the import library where there is one, and the header into a directory named for the target, and that directory is the artifact: `libzu-.tar.zst` unpacks to four files rather than to four levels of somebody's build path. Both archives are written by the same tar writer, reproducibly, so two mirrors of one release can be compared rather than trusted. + +`.github/workflows/release.yml` is the train of dx/14 §6 with its publish steps as no-ops. What is real is the build, which is the same matrix every pull request runs, the assemble, and the verify. What prints instead of running is the publishing, in the order it will happen: crates.io before the repositories that build against it, and the Go tag last of the registries because pushing a tag is the one publish that cannot be taken back. A rehearsal runs on `workflow_dispatch` with a version rather than a tag, because a release train is exactly the machinery that must not run for the first time on the day of a release. + +## 12. Documentation deliverables (v1.0 gate) Format spec (`docs/format-zu1.md`, byte-accurate, enough to write an independent reader), grammar EBNF, GQL conformance declaration, ops guide for s3 engine (cost tuning worked examples), migration guides (Neo4j/Kùzu → zu: data model mapping + Cypher dialect diffs). diff --git a/toolchains.toml b/toolchains.toml index fd95f67e..d6640599 100644 --- a/toolchains.toml +++ b/toolchains.toml @@ -44,6 +44,13 @@ key = "toolchain" holds = "pinned" match = "exact" +[[site]] +component = "rust" +file = ".github/workflows/release.yml" +key = "toolchain" +holds = "pinned" +match = "exact" + [[component]] name = "rust-nightly" pinned = "nightly-2026-08-14"