Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,84 @@ jobs:

- run: java -jar zudb-bench/target/benchmarks.jar -f 1 -wi 1 -i 1 -r 1s -w 1s

# The claim the zudb-native artifact makes is that a user who added a
# dependency and installed nothing has an engine. Nothing in the test
# suite can check that, because the suite is told where the library is
# so that it tests the binding rather than the search. So it is
# checked here, once, the way a user meets it: a classpath, no
# property, no environment variable, and a statement.
natives:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v5

- uses: actions/checkout@v5
with:
repository: tamnd/zu
path: engine

- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "25"
cache: maven

- uses: Swatinem/rust-cache@v2
with:
workspaces: engine

- name: Build libzu
working-directory: engine
run: cargo build --release -p zu-capi

# One platform rather than seven, because this runner can only
# build the one it is, so the rule that every platform is staged
# is stood down here and holds where it matters, in the release.
- name: Stage the one platform this runner is
run: |
set -eu
case "$RUNNER_OS" in
Linux) flavour=linux-amd64; library=libzu.so ;;
macOS) flavour=darwin-arm64; library=libzu.dylib ;;
*) echo "no row for $RUNNER_OS"; exit 1 ;;
esac
mkdir -p "zudb-native/lib/$flavour"
cp "engine/target/release/$library" "zudb-native/lib/$flavour/$library"

- run: mvn $MAVEN_ARGS -Pnatives -DskipTests -Denforcer.skip=true package

- name: A classpath, and nothing else
run: |
set -eu
mkdir -p "$RUNNER_TEMP/user"
cat > "$RUNNER_TEMP/user/Main.java" <<'EOF'
import dev.zudb.Connection;
import dev.zudb.Result;
import dev.zudb.Zu;

public class Main {
public static void main(String[] args) {
System.out.println("found " + Zu.library() + " through " + Zu.source());
try (Connection conn = Connection.memory();
Result r = conn.query("RETURN 1 AS one")) {
if (r.row(0).getLong(0) != 1L) {
throw new AssertionError("the engine answered something else");
}
}
System.out.println("the engine came out of the jar and answered");
}
}
EOF
cp=$(ls zudb/target/zudb-*.jar zudb-ffm/target/zudb-ffm-*.jar \
zudb-native/target/zudb-native-*.jar | grep -v sources | tr '\n' ':')
javac -cp "$cp" -d "$RUNNER_TEMP/user" "$RUNNER_TEMP/user/Main.java"
env -u ZU_LIBRARY java --enable-native-access=ALL-UNNAMED \
-cp "$cp$RUNNER_TEMP/user" Main

# What Maven Central will run over the artifacts, run here instead so
# that a release is not the first time anyone sees it.
javadoc:
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
.DS_Store
target/

# What scripts/stage-natives.sh downloads. A build of the engine is a
# thing a release fetches, not a thing a git history carries.
zudb-native/lib/
44 changes: 42 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,37 @@ Config config = Config.of(Map.of("threads", "1", "memory_limit", "1073741824"));

The keys and the parsing belong to the engine rather than to this client, so a key added to the engine since this client was built works anyway, and a key that never existed is refused with the typo named. A suffix such as `MB` is deliberately not parsed anywhere: its two readings differ by 4.9%, and the place to decide which one a user meant is where the user typed it.

## The engine, without installing one

The client is Java and the engine is a shared library, so something has to put a `libzu` on the machine. Adding one more dependency is that something:

```xml
<dependency>
<groupId>dev.zudb</groupId>
<artifactId>zudb-native</artifactId>
<version>${zu.version}</version>
<scope>runtime</scope>
</dependency>
```

That artifact carries a build for every platform this client supports and is about twenty megabytes. It is the right one for a program developed on a laptop and deployed to a cluster, and the only one that survives being shaded into an uber-jar. A container image knows exactly what it runs on, so it can name a platform and take about three megabytes instead:

| Classifier | What it holds |
|---|---|
| `linux-amd64` | glibc, x86-64 |
| `linux-arm64` | glibc, aarch64 |
| `linux-amd64-musl` | musl, x86-64 |
| `linux-arm64-musl` | musl, aarch64 |
| `darwin-amd64` | macOS, Intel |
| `darwin-arm64` | macOS, Apple silicon |
| `windows-amd64` | Windows, x86-64 |

Alpine is a separate row rather than a smaller Linux, because a shared object built against glibc does not load on musl and the message it fails with talks about an interpreter rather than about a database. Which of the two a JVM gets is decided by looking for musl's own loader on disk, which is the one path the ABI fixes rather than a distribution.

The library inside the jar is a resource, and no loader on any platform can map one of those, so it is copied to a temp file the first time anything needs it and the copy is what gets loaded. That happens once per JVM.

On the module path the artifact needs `--add-modules dev.zudb.natives`. Nothing `requires` it, since there is no code in it to require, and a jar nothing requires is a jar that is never resolved and whose resources are therefore invisible. The search says so itself when it comes up empty on a module path, so the failure names the flag rather than leaving a user to work out why the same classpath run worked.

## How it binds

The Foreign Function and Memory API is the primary path. The downcall handles are written by hand against `zu.h` rather than generated with `jextract`, because the C ABI here is around seventy functions with a stable shape, and a hand-written layer is where the interesting decisions live: which calls are `Linker.Option.critical` because they are short pure accessors, where the out-parameter scratch space comes from so that a query does not allocate, and how a `zu_error` becomes a typed Java exception exactly once. There is no native code in this repository beyond `libzu` itself.
Expand All @@ -196,7 +227,7 @@ An SDK that requires a recent JDK in 2026 excludes a large part of the enterpris
| `dev.zudb:zudb` | Java 17 | the API, no native code, no FFM types in the public surface |
| `dev.zudb:zudb-ffm` | Java 25 | the FFM provider, selected automatically |
| `dev.zudb:zudb-jni` | Java 17 | the fallback provider |
| `dev.zudb:zudb-native-{platform}` | | the `libzu` binaries |
| `dev.zudb:zudb-native` | | the `libzu` binaries, all platforms or one by classifier |

A `ServiceLoader` picks the provider at run time and application code never names one. The FFM artifact targets Java 25 rather than the Java 22 that finalised the API, because 22 has been out of support since September 2024 and shipping against an unsupported release only moves the problem. CI runs 17, 21, 25, and 26.

Expand Down Expand Up @@ -226,7 +257,16 @@ The engine has no DDL yet, so there is no `CREATE NODE TABLE` and no statement i
mvn test -Dzu.library=/path/to/libzu.dylib
```

The provider looks at `-Dzu.library`, then `ZU_LIBRARY`, then the platform library path. The tests skip rather than fail when no `libzu` is reachable, so a checkout with no engine build beside it is still green.
The library is looked for in four places, in order: `-Dzu.library`, then `ZU_LIBRARY`, then a `zudb-native` artifact on the class path, then the platform's own search. A named path is first because a bisect and a bug report both start by pointing this at a build, and the platform's search is last because it is the one that can pick up a library nobody in the process chose. `Zu.library()` and `Zu.source()` say which file was loaded and which of the four it came from, and a failure to bind lists what was ruled out on the way. The tests skip rather than fail when no `libzu` is reachable, so a checkout with no engine build beside it is still green.

The `zudb-native` module is not built unless it is asked for, because what it packages is downloaded rather than compiled:

```sh
scripts/stage-natives.sh v0.11.0
mvn -Pnatives package -DskipTests
```

The argument is a release tag of the engine, which is fetched with `gh`, or a directory that already holds the archives.

The benchmarks are JMH and are not published:

Expand Down
12 changes: 12 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@
<maven.source.plugin.version>3.4.0</maven.source.plugin.version>
<maven.javadoc.plugin.version>3.12.0</maven.javadoc.plugin.version>
<maven.shade.plugin.version>3.6.2</maven.shade.plugin.version>
<maven.enforcer.plugin.version>3.6.2</maven.enforcer.plugin.version>
<maven.gpg.plugin.version>3.2.8</maven.gpg.plugin.version>
<central.publishing.plugin.version>0.11.0</central.publishing.plugin.version>
</properties>
Expand Down Expand Up @@ -155,6 +156,17 @@
</build>

<profiles>
<!-- The libraries, which are downloaded rather than compiled, so an
ordinary build has no business needing them and does not build
this module. scripts/stage-natives.sh fills zudb-native/lib
first; CI passes this after it has run. -->
<profile>
<id>natives</id>
<modules>
<module>zudb-native</module>
</modules>
</profile>

<!-- Sources, javadoc and signatures, which Maven Central requires
and a local build has no use for. -->
<profile>
Expand Down
82 changes: 82 additions & 0 deletions scripts/stage-natives.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
#!/usr/bin/env bash
# Lay the seven libzu builds out where zudb-native packages them from.
#
# Usage: scripts/stage-natives.sh <source>
#
# scripts/stage-natives.sh v0.11.0 a tag of tamnd/zu
# scripts/stage-natives.sh ../zu/dist a directory of built prefixes
#
# The engine names its targets the way Rust does and this client names
# its platforms the way Go does, because that is what every other client
# of this engine names its artifacts after. The table below is the only
# place the two spellings meet, so a target added to the engine is one
# row here and nothing else.
#
# What is copied is the shared library and only the shared library. The
# archive also carries the static library, the header, the CLI, the
# pkg-config file and the CMake package, and none of those is anything a
# JVM can use: a jar that held them would be a jar that is four times
# the size for a file nobody opens.
set -euo pipefail

source="${1:?usage: stage-natives.sh <tag or directory>}"
here="$(cd "$(dirname "$0")/.." && pwd)"
out="$here/zudb-native/lib"

# rust target, go platform, library file name
rows="
x86_64-unknown-linux-gnu linux-amd64 libzu.so
aarch64-unknown-linux-gnu linux-arm64 libzu.so
x86_64-unknown-linux-musl linux-amd64-musl libzu.so
aarch64-unknown-linux-musl linux-arm64-musl libzu.so
x86_64-apple-darwin darwin-amd64 libzu.dylib
aarch64-apple-darwin darwin-arm64 libzu.dylib
x86_64-pc-windows-msvc windows-amd64 zu.dll
"

work=""
if [ -d "$source" ]; then
prefixes="$source"
else
# A tag, which means the release archives. Downloaded once into a
# directory of this run's own, so that a second run of the script
# cannot half-unpack over the first.
work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT
prefixes="$work"
echo "downloading libzu $source from tamnd/zu"
for target in $(echo "$rows" | awk 'NF {print $1}'); do
archive="libzu-$target.tar.zst"
gh release download "$source" --repo tamnd/zu --pattern "$archive" --dir "$work"
# The documented fallback as well as the first choice, because
# tar learned --zstd in 1.31 and RHEL 8 ships 1.30.
if tar --zstd -tf "$work/$archive" >/dev/null 2>&1; then
tar --zstd -xf "$work/$archive" -C "$work"
else
zstd -dc "$work/$archive" | tar -xf - -C "$work"
fi
done
fi

rm -rf "$out"
echo "$rows" | while read -r target platform library; do
[ -n "$target" ] || continue
from="$prefixes/libzu-$target"
# The library lives in bin/ on Windows, where a DLL is a thing that
# runs, and in lib/ everywhere else.
if [ -f "$from/bin/$library" ]; then
from="$from/bin/$library"
else
from="$from/lib/$library"
fi
if [ ! -f "$from" ]; then
echo "no $library for $target under $prefixes" >&2
exit 1
fi
mkdir -p "$out/$platform"
cp "$from" "$out/$platform/$library"
# A library a loader has to be able to map, whatever the transport
# did to the mode on the way here.
chmod 0755 "$out/$platform/$library"
echo "$platform $(du -h "$out/$platform/$library" | cut -f1)"
done
Loading
Loading