Build

Last updated: 2026-10-12

Changelog checks use Python's standard library, with no package installation. Run make changelog-check to validate pending fragments and make changelog-preview to print them. Neither command edits CHANGELOG.md; only an authorized release-preparation change updates published history. See contribution guidance.

Stack maintenance uses Python 3, Git, authenticated gh, and a configured commit signer; it requires no product build. Follow Maintain a pull-request stack for scripts/restack-after-squash.py checks, signed ancestry updates, and post-push verification. Branch updates still require the affected CI gates.

The provider test runner isolates daemon-state and loaded-model snapshots in a temporary directory for each run. Unit-test providers must not overwrite the operator's live status or recovery evidence (scripts/run-provider-tests.sh).

The Autopilot E2E harness creates canonical inventory records and persists live desired mode only for its running, authenticated, suite-owned provider handles before starting control (e2e/testbed/autopilot_cohort.go, bindAutopilotFixtureMachines). It uses the same isolated test store as the coordinator, not an environment allowlist or arbitrary UUID insertion. These isolated trusted inputs replace Apple attestation only in the testbed; they do not disable the production cohort gate or the real provider's live-lease acknowledgement. See cohort validation.

SSD epoch lookup regressions use the current provider test product and its normal source-matched resources. Filesystem permission cases require a non-root test user; they change permissions only on each fixture's temporary model root. The connected retirement snapshot oracle has an ordinary Go test and does not require a model download or running provider.

CI and Integration Tests cancel an older run only when a newer revision of the same pull request starts in that workflow. Concurrency groups include the workflow and event names; non-PR runs use a unique run ID, so default-branch pushes remain independent. Provider unit, SDK, parity and integration jobs remain parallel, without a shared-build dependency between workflows.

Integration Tests reuse their own compatible Swift debug and Rust build caches through scripts/provider-ci-cache.py (keys --lane integration). The restore prefix binds the lane, toolchain, SDK, OS, checkout path, dependency pins and build recipe; the exact key also binds the source commit. Restored source timestamps are checked by content, cached runtime resources are discarded, and Swift and sidecar build commands still execute. Metal uses an exact source/toolchain key shared with compatible provider lanes, followed by the existing source-matched validation and staging. Cache misses build normally; caches never skip E2E tests or enter the separate release-build namespace. Compare cache transfer plus build time and whole-job runtime on real runners before claiming a saving. Homebrew and zstd are installed before any integration cache restore, including Go's cache. Restore and save must use the same compression format, which is part of the Actions cache version; installing it between those steps produces misses even when the visible cache key matches. For offline validation of integration cache wiring and E2E commands with or without coverage instrumentation, run the workflow regression tests. The test-bundle staging helper uses APFS-capable clone copies on Darwin and ordinary copies on other platforms, so its offline Linux tests exercise the same atomic copy, byte-comparison and rename path (scripts/stage-test-metallib.sh).

Pull-request CI selects expensive component jobs through scripts/ci-component-paths.py, called by .github/workflows/component-changes.yml. The detector checks out full history without persisted credentials and compares the PR head with its merge base against the event's base SHA. It uses a local, NUL-delimited Git diff, including both sides of renames and deleted paths, not the truncated GitHub changed-files API. Docs and AGENTS.md changes alone do not build Swift, Go, Rust or the console, or schedule E2E integration/benchmarks. Release Integrity and Docs Lint remain unconditional. Default-branch pushes retain full coverage; a new-branch push or manual invocation selects all lanes. Other pushes compare the event's before/after SHAs. Missing revisions fail detection rather than producing skip outputs.

Component source, pinned SDK submodules, build actions and explicitly selected tooling activate their consumers. Protocol fixtures, prompt-contract producers, Rust sidecar dependencies and Go module pins also activate provider parity. Each CI workflow change exercises that workflow's lanes; changes to the shared detector exercise all callers. The existing push-only Swift cache job is unchanged. The separate release preparation workflow retains its existing triggers.

Registry performance comparisons use the same pinned Go toolchain for both revisions. Build each coordinator/tests/registry test binary with go test -c before timing, then run the prebuilt binaries without concurrent compilation. The reservation benchmark procedure describes fixtures, interleaving and the distinction between local routing cost and production latency.

How to build every component of Darkbloom from a fresh clone: the Go coordinator, the Rust prompt-contract sidecar, the Swift provider CLI (with its source-matched mlx.metallib), and the console and marketing Next.js UIs. make build builds those components; the admin UI is built separately below.

The SDK test product copies its MiMoOpenRouter media fixture directory as a test-only resource (libs/mlx-swift-lm/Package.swift). Those files are not provider product resources. Build SDK tests separately from provider tests when validating the media working-set change; see test gates.

The macOS integration and benchmark workflows run scripts/setup-macos-homebrew.sh before they install postgresql@16. The script uses brew if it is on PATH, at /opt/homebrew/bin/brew or at /usr/local/bin/brew. If it finds no brew, it downloads the official Homebrew installer from a pinned commit, checks its SHA-256, and runs it with NONINTERACTIVE=1. In both cases it writes the brew shellenv values to $GITHUB_ENV and $GITHUB_PATH for later steps. The workflows do not use the Homebrew/actions/setup-homebrew action, because the organization Actions policy does not allow it. To move the installer pin, change the commit and the SHA-256 in the script together.

macOS jobs that download artifacts, verify source signatures through the GitHub API, or post benchmark results run scripts/install-macos-github-cli.sh first. It installs the pinned official Apple Silicon CLI archive into a fresh runner-temporary directory, verifies its SHA-256 before extraction, and exports its binary path. It does not depend on Homebrew or the runner's preinstalled CLI. The signing job also checks the system signing/archive tools and Xcode's notarytool/stapler before handling the artifact. A tooling-only recovery can reuse an already qualified unsigned build through the retained unsigned recovery procedure.

Coordinator CI builds the adversarial-number test once without instrumentation for its enforced performance budget, then builds the full suite with race detection and atomic coverage. See numeric parsing tests for the separate commands and their timing limits.

The coordinator executable still builds from coordinator/cmd/coordinator. Its command entrypoint validates configuration and delegates service assembly to coordinator/app. Memory and PostgreSQL constructors now live in coordinator/store/memory and coordinator/store/postgres; root store keeps contracts and the read-through decorator. The application selects and wraps the backend before binding the registry and HTTP domain owners. See the owner map before changing an import or moving a fixture.

Coordinator Go tests live in coordinator/tests/, mirroring the production owners. go build ./coordinator/... builds production code and ordinary go test ./coordinator/... discovers the mirrored suites. make coordinator-test adds checked shard discovery; coverage explicitly instruments the imported production packages, excluding all test helpers. See the test-boundary map for focused commands. The Autopilot reward worker and migration use the ordinary coordinator build; the saved-consent wire mirror requires rebuilding the Swift provider too. Use reward test selectors for both backends, HTTP, capture and wire coverage. Building does not enable payments, fund the pool, publish the provider or activate a live cohort. The account API contract suite uses the same testdb.Main database isolation as store tests for committed-erasure cleanup checks. Routing snapshot-age regressions run against the ordinary coordinator build; the test guide includes a race-enabled repetition command using the existing reservation-preparation fixture.

Registry-ID support changes Swift provider policy and Rust prompt normalization together. Build the paired coordinator/sidecar/provider candidate; the v6 prompt contract cannot reuse a v5 cache identity. The follow-up retains the merged native SDK pin and does not require new model weights. See prompt parity for the validation procedure.

Docs Lint needs Git history to validate moved source links in frozen records; its checkout uses fetch-depth: 0 (.github/workflows/ci.yml, docs job). Freshness stamps contain only dates. The checker recovers legacy verification commits from document history, or the particular link's introduction commit for new date-only records; it never selects a commit by date. A format-only migration uses scripts/docs-stamp.sh --from-git to preserve existing dates and evidence. The job also initializes the exact pinned libs/mlx-swift-lm submodule so model-support documentation links are checked against real SDK files.

The MiMo Rust parser/planner and provider prompt regressions require public metadata, not model weights. The sidecar, provider-unit and prompt-parity jobs run scripts/prepare-mimo-prompt-fixtures.py; follow the pinned fixture procedure for local runs. Missing inputs fail rather than silently skipping assertions.

The prompt-parity lane also runs scripts/verify-nemotron-prompt-parity.sh after the provider test product has been built. The existing Go artifact provisioner fetches only hash-verified prompt metadata from the committed Nemotron manifest; the gate uses no model weights or GPU generation. See the parity test procedure.

Provider CI also runs scripts/prepare-mimo-provider-fixtures.py offline. It writes a deterministic, bounded synthetic BF16 target/vision/audio-patch/three-head inventory, not the selected model or its audio codec. Routine metadata tests use the symmetric 128-context fixture; the separate complete-prefix process uses --asymmetric for K64/V128 and 1,024-context geometry. No Python MLX installation, native evaluation, model download, private corpus or verification receipt is needed to provision these files. See MiMo provider CI tests for environment bindings and the isolated native selections. The provider-unit job retains all three native selections after its general suite. Each selection requires the successful shared build/Metal preparation and synthetic fixtures, and still runs if an earlier test fails. The SDK qualification lane in the shared release-build action provisions the same routine fixtures before its watchdog-driven provider tests. Release-only builds do not provision test fixtures or enable native qualification.

scripts/prepare-mimo-audio-fixtures.py separately downloads hash-pinned public metadata, the genuine 1.87 GB MiMo input codec, and the public OpenRouter AAC video fixture. It feeds prepare-mimo-provider-fixtures.py --audio-source to build a small synthetic target with the selected tokenizer/audio IDs. The codec is never synthesized or relabelled, and normal native loading verifies it again. CI and SDK release qualification cache those immutable inputs and run an isolated authenticated PCM8 and AAC-video inference gate. This proves the route with a small target, not the full model's output quality or production peak memory.

Production prompt parity compares the generated corpus byte-for-byte with its checked-in fixture, including its EOF format with no extra newline.

Changes to native loading estimates and retirement require a rebuilt provider test product, not only a new CLI. Bind both products and the SDK/metallib to the same checkout before running the memory and lifecycle gates. See historical source references for local setup.

Native CI test isolation reuses these built test products and their staged metallib, including the MiMo memory-admission heartbeat and pending-request gates. The provider media gate and nested SDK native-media deadline suite also reuse these products to check guarded learning and target-only rate admission. Their shared test wrapper uses bounded temporary filenames independently of filter length; selecting more cases does not require rebuilding the products. Those gates use bounded synthetic weights and production memory reserves; their 2 GiB logical contiguous grants do not preallocate 2 GiB of KV storage. See the memory regression procedure for the scope of this evidence.

Other native CI test isolation also reuses the built test products and staged metallib; it does not rebuild or download a model. Follow the provider test procedure to run GPU-global assertions in separate processes with the exclusive opt-in scoped to the named test.

The exact-cache E2E fixture uses the built provider and pinned Gemma checkpoint with an explicit SSD-cache opt-in. Its ephemeral storage setting alone does not enable caching; see the E2E prerequisites.

The Bonsai performance qualification uses a separate optimized test build with -enable-testing and -DDEBUG for test-only ownership/scheduler seams. Do not add these switches to the ordinary production build or substitute its benchmark archive with a test binary.

Model publishing can pass HUGGING_FACE_ARTIFACT_JSON through scripts/publish-model.sh to registration. See the model publishing procedure.

The admin, smoke and fleet helpers use the tools pinned here. Their local fixture checks are covered by script validation; the dev operations runbook covers invocation.

Profiler wire changes require both coordinator and provider builds; the shared Go/Swift fixture and focused checks are described in test.md and prediction telemetry.

The ProviderAppAttest Swift target uses public DeviceCheck/Security APIs. Its shadow packaging and live-validation requirements are separate from a successful local compile.

The provider email operator command builds separately with go build -o /tmp/provider-emails ./coordinator/cmd/provider-emails. It is not part of the coordinator server process. See the provider email runbook.

Provider signing, R2 staging and publication run in separate jobs in .github/workflows/release-swift.yml. scripts/provider-release-publication.py retains the final signed bundle and metadata, stages an immutable digest path, then registers the exact official build through the existing scoped POST /v1/releases. Deploy the platform persistence change first. No per-release hardware qualification or manual approval is required. A staging or publication retry reuses the original signed artifact without rebuilding or re-signing. scripts/provider_release_github.py resumes draft/upload state, verifies asset hashes and never replaces completed mismatched bytes. See official-build registration.

Secret-bearing release jobs accept only verified signed annotated production tags in reviewed master ancestry, or trusted master for dev/manual/recovery runs. They use existing repository-scoped secrets without environment approval attachments; shared GitHub settings are unchanged. These checks assume trusted repository writers. CI establishes official identity, not runtime Apple success.

The revision publisher accepts optional per-version HF repo, commit and path-prefix flags. It runs the SwiftPM darkbloom-publish executable to hash artifacts. It also needs Python 3 and the AWS CLI; use the existing pinned tools. The revision runbook describes its invocation.

Bedrock review workflow

The conditional gate uses a separate organization-membership read token. python3 .github/scripts/test-threat-bedrock.py checks active member identity, outsiders, bots, pending membership, repeat lookups and the independent human review path. The budget suite checks that local validator reasons remain visible without exposing raw provider output. See review configuration.

The optional Bedrock reviewer installs hash-locked dependencies from .github/scripts/requirements-bedrock.txt in its trusted workflow. Review, preflight and smoke checkouts fetch only the trusted scanner scripts and threat definitions, avoiding unrelated repository blobs before verification. python3 .github/scripts/test-threat-bedrock.py covers explicit provider fallback and conditional merge clearance without cloud calls. Live validation and activation are separate: see the rollout runbook.

The threat-review preflight checks the configured OpenRouter budget mode and remaining normal-attempt capacity as well as writer access and funding. Offline budget regressions cover migration without resetting charges, daily caps, unknown reservations and exhausted-pilot detection; see review configuration.

Standalone SSD accounting check

bash scripts/test-ssd-write-budget.sh compiles only the production SSDWriteBudget.swift and SSDWriteRateLimiter.swift sources, then links their Swift Testing suites with Xcode's testing framework. It needs no MLX submodule build or GPU work and removes its temporary build output on exit. See the SSD endurance tests for scope and limitations.

Nightly Linear workflow

The nightly Linear package needs Git, Python 3, local Codex desktop, and the teammate's own Linear connection. The one-time setup prompt creates a dedicated managed clone and links its two skills into the user's skill directory. Personal configuration and recovery state stay outside that clone. Claude Code and Pi supply saved work histories; they need no plugin installation.

Each trigger runs automations/nightly-linear/refresh.py (refresh) to fetch one revision of the shared skills and playbook before any Linear updates. The test procedure covers the updater's failure and preservation guarantees. The package's CI workflow runs these offline tests; it does not schedule anyone's nightly task or require Linear credentials.

SDK 27 release builds and caches

All checked-in d-inference workflow jobs use Blacksmith runners. macOS build, unit/SDK/parity, integration, benchmark, cache, signing and validation jobs pin blacksmith-12vcpu-macos-27 (M4, 12 vCPU, 48 GB); the signed-artifact older-OS smoke pins blacksmith-12vcpu-macos-26. Coordinator Tests uses blacksmith-16vcpu-ubuntu-2404; other Linux jobs retain blacksmith-4vcpu-ubuntu-2404. The macOS 27 image is currently a public beta; see Blacksmith's runner catalog. This migration is limited to this repository; SDK repository workflows are separate.

prepare-provider-release-toolchain.sh resolves the image-selected Xcode via xcode-select/xcrun after checkout and refuses any SDK other than 27.0 or Apple Swift other than 6.4. Its repository wrapper forces native SwiftPM and the same SDK for build/test invocations. CI fingerprints the selected compiler, SDK and wrapper so it cannot reuse incompatible prior-image products. Pinned Python 3.12.10, checksum-verified Rustup 1.28.2/Rust 1.88.0 (where needed), and CMake 3.31.12 supply tools missing from the image. Metal caches include the CMake recipe/identity; changing generators cannot reuse a stale metallib.

GitHub Actions remains the orchestrator and artifact store. Protected signing and publication jobs retain their existing approvals, credential scopes and same-run artifact checks. Existing runs keep the workflow from their source commit: merging runner changes does not move an already-started release.

Serving performance work changes the pinned CBv2 library as well as the provider. Initialize the recorded submodules before building, and retain source-matched Metal libraries for benchmarks. The profile qualification procedure records the exact model/runtime/backend/hardware identity; a successful build alone does not qualify a wider serving limit. Use python3 scripts/build-serving-qualification.py --output /tmp/qualification-build after committing the candidate. This cleans prior products, selects the dedicated ServingQualificationTests target with release optimization and -enable-testing, stages its source-matched Metal library, and runs the actual executable-identity test. The ordinary package graph still includes all unit tests. Pass the generated build-receipt.json to scripts/run-serving-qualification.py --build-receipt; the runner verifies the source, binary and metallib binding before collecting model/runtime evidence. Neither command installs a provider. Release Integrity CI runs the offline scripts/serving_performance/ tests without building Swift or downloading weights; hardware qualification still requires the managed build. Archived raw-corpus replay is opt-in; see the local evidence checks.

The release pipeline runs optimized products and SDK qualification on separate blacksmith-12vcpu-macos-27 runners. Both call .github/actions/provider-release-build/action.yml; only the optimized lane transfers an unsigned app and its file inventory to signing. All binaries, SwiftPM resource bundles and the source-matched Metal library travel together. Signing verifies the same-run artifact's source commit, version, inventory and entitlements before importing its certificate.

.github/workflows/provider-release-cache.yml runs the same two lanes after relevant master changes. Release tags can restore those default-branch caches; they cannot reuse another tag's cache. Release-plumbing PRs run these lanes with PR-scoped caches and no signing or publishing secrets. Changes to BoundedSingleConsumerPipeline.swift or its tests also run both lanes to catch shutdown lifetime regressions with the release compiler. Two concurrent SDK 27 runners are needed for the parallel wall-time benefit; a smaller runner quota queues the jobs without changing their gates.

scripts/provider-release-cache.py (keys) separates optimized and qualification Swift caches by selected compiler/SDK identity, machine architecture, absolute checkout/toolchain paths, dependencies and build recipe. A source commit names an immutable generation; restore prefixes stay inside that compatibility boundary. The qualification lane separately caches Rust 1.88.0 dependencies and target objects. It always cleans and recompiles the local promptsidecar package while retaining third-party objects: independently restored Swift and Rust caches must not combine source timestamps with a different generation of local Rust outputs. The Metal helper retains its exact source/toolchain contract. scripts/prepare-metal-toolchain.py requires the selected Xcode's Metal compiler to execute successfully. A successful component download alone is insufficient: it clears stale lookup state, waits a bounded interval for registration, and uses Apple's explicit component export/import path if registration remains incomplete. It never switches to an older Xcode or silently accepts an unavailable compiler.

The helper's snapshot-mtimes and restore-mtimes commands retain timestamps for tracked files whose contents are unchanged. Changed/new files retain their fresh timestamps and rebuild. Cached metadata never restores source contents or touches untracked files, links, Git metadata or paths outside the checkout. These commands make compilation incremental; every restored build still runs its build and qualification commands. Only unsigned build directories are cached, never signing keys or notarized bundles.

See the release cache procedure for first-run costs and rerun behavior.

Parallel Provider CI Builds

When provider dependencies change, the CI workflow runs provider tests, nested SDK correctness gates, and production prompt parity as three independent macOS jobs. Each job owns a separate checkout, build directory, GPU, and unified-memory allocator. No job waits for another job's test outcome. The nested SDK job still builds all of its test products; a provider test build does not compile a dependency's tests. The existing required Provider Tests check is a small aggregate gate with verified intentional-skip handling. Selected unit, SDK, and parity lanes must all succeed; unexpected skips or cancellation fail. It does not serialize their work or change branch-protection settings.

.github/actions/provider-ci-build/action.yml builds each lane using scripts/provider-ci-cache.py (keys). Provider debug tests, SDK debug tests, and Swift/Rust parity have separate cache prefixes bound to the actual compiler, SDK, architecture, checkout path, recursive dependency pins, and CI build recipe. Each source commit names a generation. Content-verified timestamp restoration reuses only unchanged tracked source files; changed files retain fresh timestamps. Every cache hit still runs the full build and runtime-resource staging commands.

The source-matched Metal cache is separate and shared only across compatible lanes. Its key binds the native MLX pin, Xcode/SDK identity, the independently downloaded Metal compiler version and bytes, helper contract, and deployment target. Restored runtime bundles and metallibs are discarded before building; scripts/stage-test-metallib.sh invokes the validating source builder and stages the library beside the actual test runner and inside its resource bundle. A cache hit is not permission to skip these checks.

Successful build jobs save their debug objects before assertions run, so an unrelated test failure does not force a complete rebuild on the next attempt. The parity job cleans local Rust products and saves its Cargo cache only after the real tokenizer/vector/load-proof script succeeds. The push-only release cache job no longer duplicates the SDK debug-test build. These caches contain unsigned build products, not release artifacts or signing material.

Parallel speedup requires capacity for three concurrent macOS runners. More jobs queued behind a provider quota do not shorten the critical path. See provider CI tests for execution gates and validation.

Prerequisites

Start commands from the repository root. Component examples that use (cd path && command) run in a subshell and preserve your current directory.

Toolchain via mise. Every version is pinned in mise.toml; mise install installs them all.

Tool	Pin	Used by
`go`	`1.26.8`	coordinator, e2e (matches the container builder and satisfies the `1.26.0` minimum in `go.mod`)
`rust`	`1.88.0`	`coordinator/promptsidecar` (matches `rust-version = "1.88"` in `coordinator/promptsidecar/Cargo.toml` and the `rust:1.88.0-alpine` builder in `coordinator/Dockerfile`)
`node`	`22`	`console-ui`, `admin-ui`
`swift`	`6.3`	`provider-swift` (the local `libs/mlx-swift` package declares `swift-tools-version: 6.3`; `provider-swift/Package.swift` itself is `6.1`)
`python`	`3.12`	`scripts/*.py`, benchmark wrapper tests
`jq`, `gh`, `awscli`, `gcloud`	`latest`	scripts, release and deploy tooling

macOS on Apple Silicon for anything under provider-swift/ (MLX + Metal). The coordinator, sidecar, e2e harness, and UIs build on macOS or Linux.
Xcode Command Line Tools + cmake (brew install cmake) — the metallib helper compiles MLX's Metal kernels with cmake.
Git submodules checked out: libs/mlx-swift, libs/mlx-swift-lm, libs/mlx (git clone --recurse-submodules … or git submodule update --init --recursive). provider-swift/Package.swift depends on ../libs/mlx-swift and ../libs/mlx-swift-lm by local path.
Docker only for the coordinator container image (step 9).

Pinned MLX dependencies

The provider consumes the local packages through immutable Git submodule pins:

Package	Merged revision	Included update
`libs/mlx`	`cb77239be31b1df7f5db895226af55c39fc4f093`	`gather_mm` / `gather_qmm` row-tile backport
`libs/mlx-swift/Source/Cmlx/mlx`	`cb77239be31b1df7f5db895226af55c39fc4f093`	Same merged MLX source used by Cmlx and the provider metallib
`libs/mlx-swift`	`6923a80f624f5c91fbf456efe4e00e9698a72961`	PR #34: merged nested MLX row-tile backport and regenerated kernel sources; retains PR #28 exact constant reuse for eligible Bonsai packed projections
`libs/mlx-swift-lm`	`3fd4944c3b3ee5cb45c5cbfac8805876332d3a29`	Includes PR #289 bounded demanded checkpoint capture, retention and continuation, PR #290 native cancellation retirement, and PR #165 typical MTP acceptance

Keep both local packages in the provider build. The SDK's standalone package manifest pins Swift 0f4fe403bef6899e8a72882bc6d4036a7a62ae31, not the provider's current Swift gitlink; the nested-test procedure in test.md binds it to the recorded local Swift gitlink. Keep libs/mlx and libs/mlx-swift/Source/Cmlx/mlx on the same merged MLX commit for the gather_mm / gather_qmm row-tile backport. The Swift pin includes kernel sources regenerated from that nested MLX revision, including the signed integer floor-division corrections from MLX PR #32 in the generated binary-operations and CPU preamble sources. Build mlx.metallib from the nested source with scripts/fetch-metallib.sh; changing only the top-level MLX gitlink does not change provider bytes. Rebuild the consumer after changing pins; earlier full-model measurements are evidence for their recorded dependency set, not a new benchmark of these pins. The SDK's acceptance default remains exact; the provider's eligible sampled target-prefix MTP requests default to typical acceptance, while greedy requests remain exact. See the serving policy. Neither that policy nor the newer native retirement and MLX kernels are qualified by the original demanded-prefix full-model results. The dependency validation procedure separates required tiny-fixture regressions from fresh full-model performance qualification. The earlier SDK PR #170 pin 4101d4c1bfa6b3175e7f34393e8c235a75a7c1be had production libraries and a package manifest matching review head b52335b839d80c8e6d4194ebbd8809d737cd8eb3. That historical comparison is not validation of the current SDK pin above.

Native Flash-Next candidate

The Qwen 3.8 Next integration originally landed in SDK PR #149, commit 729fa45c67a8b1cb26b1debeacf7f1d16ef3a21e. That commit's complete Git tree is identical to the approved review head ae3ecdc835a895091f8929749fdb3e14383295a9. The current SDK gitlink above retains this support. Use the recorded gitlink, not a floating branch or a private experiment.

Use the repository-owned conversion tools for the pinned official source. Metadata verification is distinct from full payload hashing; conversion validates each source shard and writes a new output/manifest while retaining the trained assistant and packed PLE table. Inspect the tool's storage requirements before full hashing or conversion and coordinate the model/GPU operator. Never create a missing mount path or reuse an existing output directory. These commands do not publish an artifact.

Before describing the candidate as reproducible:

Record the selected source trees/patch digests and approved immutable core, C, Swift, SDK and provider pins. Inspect the composed SDK's libs/mlx-swift-lm/docs/qwen4/composition.md for required source selection and excluded experiments.
Resolve CMake/package revisions and nested gitlinks in a fresh recursive private checkout. A machine-specific dependency symlink, local package override or unrecorded core patch does not close this gate. After each dependency merge, record the resulting approved commit, update its consumers and repeat affected checks; a review-head pin is not a final merged pin.
Build the provider with the source-matched metallib and required SwiftPM resources using the procedures below; record actual binary/library/resource identities. Source parsing alone is not a build or runtime test.
Complete the candidate test matrix on that final artifact. Keep private draft staging, signing/release, model publication, catalog activation and deployment as distinct outcomes.

Current source and validation limits are in the candidate reference.

SwiftPM may mark generated resource bundles hidden on macOS. The SDK's PagedAttentionResources.locate must still discover their readable Metal source inside its existing search roots. Do not clear filesystem flags or disable paged eligibility to hide a failed preflight; keep sealed-app lookup and conflicting-resource rejection intact. Stage and verify resources for both the test host and any separately invoked CLI child.

Private prefill experiments, including packed-read lookahead and ordered NAX, are excluded from this publication pin. Rebuild and rebind both the SDK tests and provider when its gitlink changes; an earlier executable cannot qualify the new source merely because the core metallib hash is unchanged. Record the actual compiled NAX capability and precision posture; a hardware product name does not prove which kernels or arithmetic were used.

The connected Go API matrix can reuse an independently hashed production provider via DARKBLOOM_PROVIDER_BINARY; it does not build or substitute a different native runtime. Record the Go coordinator/test source separately. The connected qualification instructions bind coordinator traffic and native metrics to one authenticated unified provider, with the Python runner and original oracles owned by this repository.

Repository layout for builders

Path	Toolchain	Notes
`go.mod` (repo root)	Go	Single module `github.com/eigeninference/d-inference`; contains `coordinator/...` and `e2e/...`. There is no `go.work` and no nested `go.mod`.
`coordinator/cmd/coordinator/`	Go	The coordinator binary (`main.go`).
`coordinator/promptsidecar/`	Rust	Crate `promptsidecar`, edition 2024, `Cargo.lock` committed; built with `--locked`.
`provider-swift/`	SwiftPM	Products: `darkbloom` (CLI), `darkbloom-enclave`, `darkbloom-fan-helper`, `darkbloom-publish`; libraries `ProviderCore`, `ProviderCoreFoundation`, `DarkbloomFan*`. Platform `macOS 14+`.
`console-ui/`	Next.js 16 / React 19	`npm`; tests with Vitest.
`admin-ui/`	Next.js 16 / React 19	`npm`; dev/start on port `4001`.
`landing/`	Next.js 16 / React 19	Standalone npm app; `make landing` installs, lints, builds and runs route tests; dev/start on port `3008`.
`Makefile`	—	Every target below; `make help` lists them.

Provider tests are grouped by subsystem inside their existing SwiftPM targets. See finding provider tests for the folder map; provider-swift/Package.swift (package) retains recursive source discovery. The inference source map locates engine, memory, caching and request-processing code within the same ProviderCore target; building these folders requires no separate products or commands.

Steps

1. Install the toolchain and hooks

mise install                          # installs every pin in mise.toml
git submodule update --init --recursive
git config core.hooksPath .githooks   # enables pre-commit + pre-push (see "Git hooks")

mise activates the pinned versions per shell; on macOS the system Xcode swift is also acceptable for provider-swift.

2. Build everything

make build      # coordinator-build prompt-sidecar-build provider-build ui-build
make all        # test + build (see test.md)

Continue with the per-component steps when you need one piece or want to understand what make runs.

3. Coordinator (Go)

The owned two-host Go fixture embeds e2e/testbed/provider_host.py; rebuild its test binary after helper or lifecycle changes. The CPU-only TestPrepareConnectedInputBindings check uses the actual fixture input/report types to prepare canonical catalog entries before a physical run. The helper waits within the existing five-minute prelaunch bound for GPU ≤42°C and load1 ≤4, under the same control lease used after launch. See the test procedure.

Rebuild the E2E test executable after shared sidecar lifecycle or release-default fixture changes. startExactCacheSidecar and the restart fixtures install the API preload controller before starting it, matching coordinator startup. The CPU-only API-readiness regression in the linked procedure uses a verified Rust sidecar and immutable prompt artifacts; compiling it does not execute the provider/API cache smoke or qualify model restoration.

CI checks formatting of tracked Go source while preserving frozen report evidence bytes; see the coordinator checks. Documentation-impact checks also cover the contact-export and legacy-cohort store owners; see the soft-delete reference.

The idle-provider routing regressions run without a Swift provider binary or downloaded model. Use those focused checks before make coordinator-test when changing exploration or rate ranking.

make coordinator-build            # cd coordinator && go build ./cmd/coordinator
make coordinator-build-linux      # GOOS=linux GOARCH=amd64 CGO_ENABLED=0 → coordinator/coordinator-linux

The host build writes ./coordinator/coordinator. Version identity is injected only by the container build (-ldflags -X …api.BuildVersion/BuildCommit/BuildDate in coordinator/Dockerfile); a local go build reports dev/unknown on GET /health (coordinator/api/inference/consumer.go, HandleHealth).

4. Prompt-contract sidecar (Rust)

make prompt-sidecar-format   # cargo fmt --all -- --check
make prompt-sidecar-check    # cargo check --locked --all-targets; cargo clippy --locked --all-targets -- -D warnings
make prompt-sidecar-build    # cargo build --locked --release --bin promptsidecar
make prompt-sidecar          # format + check + test + build

Output: ./coordinator/promptsidecar/target/release/promptsidecar. The coordinator container needs a statically linked Linux binary, built the way coordinator/Dockerfile does it (stage prompt-sidecar-builder):

cd coordinator/promptsidecar
rustup target add x86_64-unknown-linux-musl
cargo build --locked --release --target x86_64-unknown-linux-musl --bin promptsidecar
file target/x86_64-unknown-linux-musl/release/promptsidecar   # must say "statically linked" or "static-pie linked"

On macOS cross-compiling to musl needs a Linux linker; use the Docker build in step 9 instead. scripts/verify-prompt-sidecar-linux.sh <binary> runs the production prompt vectors against a Linux sidecar binary (CI job "Prompt Sidecar Tests").

5. Provider CLI (Swift) with source-matched metallib

Build the test product again after changing fixture helpers or assertions; --skip-build alone reuses the previous executable. The provider test procedure covers isolated CLI configuration, artifact integrity, SSD authentication, paged-preflight diagnostics, and stream ordering. Synthetic MLX fixtures need the matched metallib; enabled live-model fixtures also need their documented model inputs.

To compile all test targets without executing fixtures:

(cd provider-swift && swift build --build-tests)

The test products include the provider/standalone lifecycle, CLI/service/fan, SSD-cache and benchmark harness groups described in the provider test-group map. Most use scripted dependencies and isolated files; the generated tiny-model load tests execute actual MLX kernels without downloading a checkpoint. A successful compile alone does not run either group, and neither substitutes for full-checkpoint qualification. Stage the matched metallib before execution and use the runner's serial/fresh-process isolation rather than parallelizing tests that share MLX or model-cache state.

For coverage, the provider CI lane builds tests and the fan-helper product with --enable-code-coverage; its cache key distinguishes instrumented builds from the SDK and parity lanes (.github/actions/provider-ci-build/action.yml). Follow the isolated local coverage recipe for fresh profiles, state paths, all reported executable objects and retained test exit status. Product, CLI and benchmark rows are report-only, not a release or performance gate. Keep these instrumented debug products separate from the optimized binaries used for performance measurements.

make provider-build
# = cd provider-swift && swift build
#   ./scripts/fetch-metallib.sh "$(cd provider-swift && swift build --show-bin-path)"

swift build produces ./provider-swift/.build/debug/darkbloom (plus darkbloom-enclave, darkbloom-fan-helper, darkbloom-publish). MLX loads mlx.metallib from beside the running executable, and SwiftPM does not compile the Metal kernels, so scripts/fetch-metallib.sh builds them with cmake from libs/mlx-swift/Source/Cmlx/mlx (the exact source the host side links) and copies the result next to the binary. Despite its name it builds, it does not download.

MLX also embeds shader source in Cmlx for runtime compilation. After changing an MLX kernel header, regenerate the affected embedded sources with libs/mlx-swift/Tools/update-mlx.sh from a clean, isolated libs/mlx-swift checkout and review its generated diff. For quantized.h, keep both Source/Cmlx/mlx-generated/quantized.cpp and Source/Cmlx/mlx-generated/metal/quantized.h synchronized with the core header. Rebuild Cmlx and relink the provider as well as rebuilding mlx.metallib; replacing the Metal library alone leaves the embedded QMV implementation intact. Run the bias-accumulation regression against the resulting runtime.

./scripts/fetch-metallib.sh            # next to the latest debug build
./scripts/fetch-metallib.sh release    # next to the release build
./scripts/fetch-metallib.sh /some/dir  # → /some/dir/mlx.metallib

Knobs: METALLIB_CACHE_DIR (default /tmp/mlx-metallib-cache; cache key includes the MLX tree SHA, toolchain hash and deployment target) and MLX_METALLIB_DEPLOYMENT_TARGET (default 26.2; the _nax kernels are only compiled at SDK/deployment target ≥ 26.2). The script fails if required kernel symbols (_nax, gemv, the affine_qmv_wide_* variants) are missing from the produced library.

After changing doctor subprocess capture or status canonicalization, rebuild the Swift test targets with cd provider-swift && swift build --build-tests. Then run the focused capture and canonical-byte regressions.

Instrumented candidate benchmarks

Build radix-engine with its matching native dependency and RADIX_CANDIDATE define to emit schema-3 actual-forward-width evidence. The scalar observer and benchmark validator must come from the same reviewed source cut. See the prefix-cache benchmark checks for scope, completion and B2/B4 acceptance requirements. The executable's scripts/benchmarks/radix-engine/Package.resolved is tracked and pins the reviewed provider dependency set. Keep locked resolution enabled and verify the resulting source graph when applying local package overrides.

Restored SwiftPM runtime resources

The Provider Tests job removes restored metallibs and resource bundles from all macOS build configurations in both package caches before building its debug test product (.github/workflows/ci.yml). An inactive package's cached debug bundle can contain older source just as a release bundle can. Each subsequent package build recreates its own resources; compiled objects and dependency checkouts remain cached. Runtime lookup accepts byte-identical copies and continues to reject divergent copies.

The separate signing-validation workflow checks packaging and Apple signing without selecting a deployment environment.

Release configuration, as the release workflow builds it:

cd provider-swift && swift build -c release --product darkbloom
swift build -c release --product darkbloom-fan-helper
cd .. && ./scripts/fetch-metallib.sh release

darkbloom --version must print the value of ProviderCore.version (provider-swift/Sources/ProviderCore/ProviderCore.swift); scripts/check-release-version.sh enforces this against the coordinator's LatestProviderVersion (see ../operations/provider-release.md).

For a signed provider bundle needed by isolated tests, use the validation-only signing workflow. It retains an Actions artifact after the normal signing and final-bundle checks.

Standalone attention operator replay

scripts/benchmarks/attention-replay links only MLX and MLXLMCommon. It consumes validated packet bytes without loading a model or provider. Use an isolated build directory and the reviewed dependency pins/local MLX package binding from the replay validation manifest:

REPLAY_SOURCE_ROOT=/absolute/path/to/pinned-worktree
ATTENTION_REPLAY_SOURCE_ROOT="$REPLAY_SOURCE_ROOT" \
  swift build --package-path "$REPLAY_SOURCE_ROOT/scripts/benchmarks/attention-replay" \
    --scratch-path /absolute/path/to/replay-build \
    -c release --product attention-replay --jobs 4 --disable-automatic-resolution

Retain the executable SHA-256, source/dependency inventory, build graph, mlx.metallib and SwiftPM resource bundles. An executable hash alone does not bind external Metal resources. The Python driver never builds or downloads them. Use the offline NumPy environment for packet validation and the independent reference. See replay validation and the source/test milestone.

Segmented metadata profiler

Build the native BenchSegmentedDecode target in an isolated directory with the same pinned local MLX dependency used by the nested native tests:

swift build --package-path libs/mlx-swift-lm --scratch-path /absolute/path/to/segment-profiler-build \
  -c release --product BenchSegmentedDecode --jobs 4 --disable-automatic-resolution

Stage the matching mlx.metallib and SwiftPM resource bundles beside the binary, and retain their hashes with the build's source/dependency inventory. See profiler validation for the bounded run.

Prefix-cache benchmark executable

scripts/benchmarks/radix-engine links the real provider factory and MLX packages from an explicitly selected worktree. Keep baseline and candidate source worktrees separate, with recursive submodules pinned. From the repository root:

RADIX_BENCH_SOURCE=/absolute/path/to/pinned-worktree
cp "$RADIX_BENCH_SOURCE/provider-swift/Package.resolved" scripts/benchmarks/radix-engine/Package.resolved
RADIX_SOURCE_ROOT="$RADIX_BENCH_SOURCE" RADIX_CANDIDATE_BUILD=0 \
  swift build --package-path scripts/benchmarks/radix-engine \
    --scratch-path "$RADIX_BENCH_SOURCE/provider-swift/.build" \
    -c release --product radix-engine --jobs 4 --disable-automatic-resolution

Use RADIX_CANDIDATE_BUILD=1 for a source tree containing EngineV2Factory.makeBenchmarkSession and the current cache APIs. The same harness source compiles against the older baseline with 0. Candidate SSD mode uses the normal slot factory after a fresh pre/post-load weight-hash check; --cache-mode resident explicitly reproduces the earlier resident-cache arm. The baseline conditional and resident reproduction use the direct production engine factory (BenchmarkLoader.swift).

The candidate also accepts --mtp-acceptance exact|typical after its positional arguments; historical baseline builds reject an explicit flag. Omission keeps exact. Rebuild against the provider SPI that exposes EngineV2Factory.makeBenchmarkSession(mtpAcceptanceConfig:), a String argument defaulting to "exact", in provider-swift/Sources/ProviderCore/Inference/Engine/Factory/EngineV2Factory+BenchmarkSessionConstruction.swift. The candidate SSD route forwards the flag through that normal slot factory; the resident reproduction sets CBv2MTPConfig.acceptance directly. Neither route changes model/assistant verification or memory gates. The flag does not enable MTP and offers no delta override. See sampled acceptance validation.

The paired persistent-test namespace/access-group options require a candidate build containing SSDPersistentTestKeyNamespace; historical builds reject them. The same RADIX_CANDIDATE_BUILD=1 define also enables the namespace test target. Building this source does not authorize a Keychain group or establish persistent restart. See isolated persistent namespace validation.

Archive the source manifest, compile define, binary hash, matching mlx.metallib, and SwiftPM resource bundles before changing that source tree. Stage the Metal library beside radix-engine, as for the provider CLI. Before model loading, the candidate SSD harness calls bindRuntimeMetallibForMLX, the normal startup binder, and requires a valid immutable digest; placing the file beside the binary alone does not establish the identity used by the complete cache (provider-swift/Sources/ProviderCore/Security/BinaryHasher.swift). The benchmark session is exposed only through @_spi(Benchmarking); its raw events preserve token IDs. Segmented storage uses the shared native process owner; bridge dispatch, contiguous bridge admission and HTTP framing require the separate provider HTTP probe. See cache validation for replay, key-mode and process-restart requirements.

The current candidate accepts --concurrency 1|2|4 and either --production-kv-grant or --kv-budget-gib N after its positional arguments. Archive one binary for all compared arms. Use --production-kv-grant for the single-model production-capacity run: the existing slot session derives its logical grant from loaded target and assistant weights using the production grant policy. Archive the Python evaluator source with the binary evidence as well: the schema-2 cache comparator requires an off-to-on pair and validates idle/shutdown ownership. A binary whose terminal snapshots precede publication of retired engine gauges cannot satisfy that idle gate; preserve the raw refusal and use a harness with coherent observations for the final comparison (scripts/benchmarks/radix_engine_evidence.py, retirement_errors). The current candidate includes bounded observation of published idle snapshots. Its pure BenchmarkIdleObservationTests target can be run with swift test using the same package, pinned source environment and scratch path as the build. Keep test and release artifacts distinct, and use the final release binary for both compared arms. Preserve logs that name the executed Swift Testing cases; an XCTest runner may report zero tests before Swift Testing executes its suite. It retains the separate post-build live OS/activation headroom gate. The mode requires the candidate SSD serving path; it cannot be combined with resident reproduction, native-probe-only mode or an explicit grant (provider-swift/Sources/ProviderCore/Inference/Engine/Factory/EngineV2Factory+BenchmarkGrant.swift, benchmarkProductionGrant; BenchmarkOptions.swift).

For explicit envelope controls, use --kv-budget-gib N. Without either flag, the existing default remains one request and a 16 GiB explicit slot grant. Explicit mode retains its measured post-load allocator guard; that diagnostic does not size a production grant. Candidate paged_storage metrics separately report committed backing and the mutable logical grant. Follow the validation steps to retain policy inputs, live headroom and actual engine capacity with each result.

For a bounded target diagnostic, append --native-kv-probe-only with cache-off mtp-off and concurrency one. This candidate-only mode loads the verified model and runs the same native KV type probe used before paged backend construction: two prefill tokens followed by one decode token. It records each attention row's actual K/V types and shapes, with model and metallib hashes. It creates no serving engine or SSD store.

The standalone product also includes optional bounded target-logit capture through --logit-diagnostic-position and --logit-diagnostic-candidates. Build it with the matching native submodule; follow the diagnostic procedure to preserve the original request and compare observation against an uninstrumented control. These flags belong to the standalone benchmark, not the provider CLI.

The matching native submodule also supports --attention-packet-position and --attention-packet-layer for a bounded native-byte capture from one attention owner. Follow the capture procedure before passing the exported packet to the offline analyzer below.

Offline attention analysis environment

The optional attention packet analyzer uses a separate Python environment and the pinned NumPy requirement. It needs no Swift build, model weights or GPU.

python3 -m venv /tmp/darkbloom-attention-venv
/tmp/darkbloom-attention-venv/bin/python -m pip install -r scripts/benchmarks/attention_packet/requirements.txt

Use that interpreter for packet analysis and its tests.

6. Console UI (Next.js)

make ui-install   # cd console-ui && npm install
make ui-lint      # npx eslint src/
make ui-test      # npm test  (vitest run)
make ui-build     # npm run build  (next build)
make ui           # install + lint + test + build

Local dev server: cd console-ui && npm run dev. Bundle budget check: npm run bundle:check (console-ui/scripts/analyze-bundle.mjs). CI uses npm ci.

7. Admin UI (Next.js)

No make target. From admin-ui/:

npm install
npm run lint     # eslint src/
npm test         # vitest run
npm run build    # next build
npm run dev      # next dev -p 4001

8. Landing page

The imported Next.js site lives in landing/ and has its own npm lockfile and configuration. It does not depend on the eigen-homepages workspace. From the repository root:

make landing
cd landing && npm run dev

Copy landing/.env.example to .env.local inside that directory to configure runtime integrations. A production build needs no credentials. The hosting root is landing, with the Next.js preset; its API routes require a server runtime rather than a static export. See the marketing README for deployment handoff.

9. Coordinator container image

Run python3 scripts/check-go-toolchain.py before building. Release Integrity runs the same guard and its regression suite: mise.toml and the digest-pinned Go builder must declare the same exact patch version, at least the go.mod minimum. Do not downgrade dependencies or rely on automatic toolchain downloads to compensate for an older builder. The guard checks declarations, not registry contents; verify the image digest and platform when updating its tag.

The production image is built by coordinator/Dockerfile from the repo root (it copies both coordinator/ and the sidecar crate):

docker build \
  --build-arg BUILD_VERSION="$(awk -F'"' '/public static let version =/ {print $2}' provider-swift/Sources/ProviderCore/ProviderCore.swift)" \
  --build-arg BUILD_COMMIT="$(git rev-parse HEAD)" \
  --build-arg BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -f coordinator/Dockerfile -t coordinator:local .

Stages: prompt-sidecar-builder (rust:1.88.0-alpine, musl static build) → builder (golang:1.26.8-alpine, -ldflags version injection) → final image FROM eigengajesh/d-inference-base:v1-amd64 with /usr/local/bin/coordinator and /usr/local/bin/promptsidecar, OCI labels org.opencontainers.image.{version,revision,created}, EXPOSE 8080, entrypoint start.sh (coordinator/deploy/start.sh). Cloud Build wraps exactly this in deploy/gcp/cloudbuild.yaml (dev) and deploy/gcp/cloudbuild-prod.yaml (prod); see ../operations/coordinator-deploy.md.

10. Use the database-only coordinator command

The normal coordinator build also supports coordinator --migrate-only. It requires EIGENINFERENCE_DATABASE_URL, validates store configuration rather than full application/serving prerequisites, and exits without starting the server or seeding an admin key. EIGENINFERENCE_MIGRATION_TIMEOUT bounds the full command; EIGENINFERENCE_CONCURRENT_INDEX_LOCK_TIMEOUT independently bounds concurrent build lock waits (see configuration). Container execution must override the default MicroMDM entrypoint script; see the schema migration runbook.

The startup measurement tool requires Python 3.10+ and no third-party packages or build step. Its tests use local stub servers; its default observation mode sends only public GETs.

`make` targets

Target	What it runs
`help`	List targets (default goal)
`coordinator-test`	Runner self-tests, then `python3 scripts/run-coordinator-tests.py` (complete coordinator suite; process-isolated API shards, also registry under `--race`, see test)
`coordinator-build`	`go build ./cmd/coordinator` → `./coordinator/coordinator`
`coordinator-build-linux`	`GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o coordinator-linux ./cmd/coordinator`
`coordinator`	`coordinator-test` + `coordinator-build`
`sqlc-generate`	`go run github.com/sqlc-dev/sqlc/cmd/sqlc@v1.31.1 generate -f coordinator/store/postgres/sqlc.yaml` → `coordinator/store/postgres/storedb`
`sqlc-check`	Needs `DATABASE_URL`. `TestMigrationsBuildCheckedInSchema` (fails when `coordinator/store/postgres/schema/schema.sql` is stale), then `sqlc diff` (fails when `coordinator/store/postgres/storedb` is stale)
`prompt-sidecar-format`	`cargo fmt --all -- --check`
`prompt-sidecar-check`	`cargo check --locked --all-targets` + `cargo clippy --locked --all-targets -- -D warnings`
`prompt-sidecar-test`	`cargo test --locked --all-targets`
`prompt-sidecar-build`	`cargo build --locked --release --bin promptsidecar`
`prompt-sidecar`	format + check + test + build
`provider-build`	`swift build` + `scripts/fetch-metallib.sh <bin-path>`
`provider-test`	`swift build --build-tests`, stage `mlx.metallib` into the bin dir and every `*.xctest` test bundle, then `swift test --skip-build`
`provider`	`provider-build` + `provider-test`
`benchmark-wrapper-test`	Python unittest discovery for `gemma_contbatch/tests` and `serving_performance` from `scripts/`
`benchmark-gemma-contbatch`	`python3 scripts/benchmark-gemma-contbatch.py $(GEMMA_BENCHMARK_ARGS)` (needs GPU + weights)
`ui-install` / `ui-lint` / `ui-test` / `ui-build` / `ui`	`npm install` / `npx eslint src/` / `npm test` / `npm run build` in `console-ui/`
`e2e-integration`	`go test ./e2e/... -run TestIntegration -v`
`e2e-benchmark`	`go test ./e2e/... -run TestBenchmark -v`
`e2e`	`e2e-integration`
`docs-check`	`scripts/docs-check.sh` (stamps, links, cited paths, orphans)
`docs-stamp`	`scripts/docs-stamp.sh $(FILES)` — refresh freshness stamps
`test`	`coordinator-test prompt-sidecar-test provider-test ui-test benchmark-wrapper-test docs-check`
`build`	`coordinator-build prompt-sidecar-build provider-build ui-build`
`all`	`test build`
`clean`	remove `./coordinator/coordinator{,-linux}`, `./coordinator/promptsidecar/target`, `./provider-swift/.build`, `./console-ui/.next`, `./console-ui/node_modules`

Git hooks

Enable once with git config core.hooksPath .githooks. Both hooks only act on components that changed.

Hook	Trigger	Checks
`.githooks/pre-commit`	staged `coordinator/**.go`	`gofmt -l` on the staged files (fix: `gofmt -w <file>`)
	staged `console-ui/**.ts{,x}`	`cd console-ui && npx eslint src/` (fix: `npx eslint --fix src/`)
	Swift	skipped — no enforced formatter
`.githooks/pre-push`	any `coordinator/` change in the pushed range	`gofmt -l .` over `coordinator/`, then `go test ./coordinator/...` from the repository root: ordinary discovery includes every mirrored package and production package, with no package exclusion. Use `make coordinator-test` for runner guards, isolated shards and production coverage
	any `console-ui/` change	`npx eslint --quiet src/` and `npm run build`

CI runs the fuller set (gofmt, golangci-lint, -race tests, Swift, Rust, docs lint); see test.md.

Verify

make build
ls -l coordinator/coordinator coordinator/promptsidecar/target/release/promptsidecar
ls -l provider-swift/.build/debug/darkbloom provider-swift/.build/debug/mlx.metallib
./provider-swift/.build/debug/darkbloom --version    # prints ProviderCore.version, e.g. 0.8.16
ls console-ui/.next

Troubleshooting

Symptom	Cause	Fix
`swift build` cannot resolve `../libs/mlx-swift`	submodules not checked out	`git submodule update --init --recursive`
provider starts but MLX fails to load kernels / "metallib not found"	`mlx.metallib` missing beside the binary	`./scripts/fetch-metallib.sh` (or `make provider-build`)
`fetch-metallib.sh` fails on missing `_nax` symbols	deployment target/SDK below 26.2	update Xcode; or set `MLX_METALLIB_DEPLOYMENT_TARGET` only if you accept a kernel set that differs from release builds
`cargo build --locked` fails on lockfile	`Cargo.lock` out of date with `Cargo.toml`	run `cargo update -p <crate>` deliberately and commit the lock; never drop `--locked` in CI
`go build` picks a different Go	`mise` not activated in this shell	`eval "$(mise activate bash)"` (or zsh) then retry
`docker build` fails at `file … statically linked`	sidecar not statically linked (musl target missing)	the Dockerfile adds the target itself; check Docker platform is `linux/amd64`

App Attest release qualification

For local provider dispatch and Swift/Go recovery checks, build the test targets and stage the source-matched Metal library using make provider-test. The App Attest provider dispatch tests use loopback and software callbacks; they do not qualify the signed release.

Use the macOS 27 SDK for a candidate that needs Apple code-measurement extensions. The release workflow explicitly selects Command Line Tools 27.0 / Swift 6.4, then runs provider tests under that same SDK; ordinary development retains the Swift 6.3 minimum. Set SDKROOT to that SDK for both compilation and linking: a CLT 27 beta 6 Swift probe compiled with --sdk alone embedded the deployment target as its SDK; setting SDKROOT produced the correct linked SDK. Verify LC_BUILD_VERSION with xcrun vtool -show-build on the final executable. Confirm the final signed executable produces the current launch category and full CodeDirectory digest on physical macOS 27; SDK 26 builds can collect ordinary shadow proofs but cannot qualify replacement readiness. See the observed SDK and measurement contract.

Run go test ./tests/appattest/... ./tests/api/... ./tests/store/... -run 'TestAppAttest|TestAuthorization|TestApple|TestMacCodeMeasurement' from coordinator/, using a disposable local DATABASE_URL for the store contracts (the test harness truncates tables). Add -race for concurrency checks. Run swift test --filter ProviderAppAttestTests from provider-swift/. The private admin queries have PostgreSQL coverage in admin-ui/src/lib/queries/app-attest.test.ts and admin-ui/src/lib/queries/app-attest-diagnostics.test.ts.

After the optimized provider is packaged with its resources, run DARKBLOOM_NO_UPDATE_CHECK=1 DARKBLOOM_GEMMA4_PREFILL_CHUNK_EVAL=18 MLX_GEMMA4_FUSED_WEIGHTED_UNSORT=1 MLX_GATHER_QMM_EXPERT_SLICES=1 Darkbloom.app/Contents/MacOS/darkbloom runtime-smoke (the child validates retained latches that MLX reads at its first Metal touch, so the caller seeds them, exactly as SelfUpdater and install.sh do). Require all four markers: app-attest-callback-runtime-smoke: ok, gemma-optimizations-runtime-smoke: ok, paged-kernel-runtime-smoke: ok, and qwen4-metal-resources-runtime-smoke: ok. The first line is build-environment-runtime-smoke: <prod|dev> coordinator=<url> cdn=<url>. The release workflow requires <prod|dev> to match the release environment. Callback completion and expiry are exercised without Apple service calls or a Keychain item. This linked-binary check catches a release-only allocator failure that debug tests missed. Run bash scripts/test-install-atomic.sh for installer acceptance and rollback cases. The rollout runbook separates these checks from real Apple receipt renewal and final signed-artifact fleet qualification.

test.md — unit, e2e, CI.
../operations/provider-release.md — provider release runbook.
../operations/coordinator-deploy.md — container build and deploy on GCP.
../architecture/components/mlx-swift.md — why the metallib must match the MLX source.

Candidate native prefix-cache benchmarks must build ProviderCore and scripts/benchmarks/radix-engine from the same source revision: the benchmark prompt SPI carries production sampling parameters into each engine request. See native benchmark validation for sampling scope, regression filters and diagnostic restrictions.

Qwen packaged resource regression

python3 scripts/test-qwen4-packaged-resources.py compiles the actual Qwen Metal resource accessor into a small optimized app, then runs it from a relocated app and an installer-style executable symlink. It checks all three preamble hashes, rejects missing or empty files and resource links outside the app, and proves that developer/cwd copies cannot mask a broken packaged resource. It needs Swift on macOS, but no model weights or GPU. Both SDK 27 release lanes and Provider Tests run this check. The full provider runtime-smoke exercises the same accessor before publication, installation, and update.

Provider Tests also runs python3 scripts/test-profile-inventory-auth.py on macOS. It compiles the production ProfileInventoryAuthorization helper into a terminal fixture without invoking real sudo or changing profiles; the test procedure is in test.md.

Promotion payload helper

python3 scripts/model-token-promotion.py --help prepares a model-specific, calendar-day grant payload without making API calls. It requires Python with zoneinfo and timezone data. The promotion runbook covers review and approved application; the test guide covers calendar and settlement validation.

Source-matched test libraries

After swift build --build-tests, run scripts/stage-test-metallib.sh with the package's swift build --show-bin-path directory. The helper builds or verifies the matching MLX library. Then it copies the library into every *.xctest bundle in that directory: beside the test executable and in the nested resource bundle that native checkpoint identity tests use. The native build system makes one <Package>PackageTests.xctest. CI uses the native build system through scripts/provider-release-swift.sh. A local Swift 6.4 swift build uses Swift Build, which makes one <Target>.xctest for each test target. make provider-test and the provider/nested CI jobs invoke this helper. A missing test runner or failed source verification is an error; an existing library is always replaced. Use scripts/run-provider-tests.sh after staging rather than replacing it with one unfiltered test process. Its mandatory CPU checkpoint gates own fresh MLX compile-cache history; the test guide describes their exact byte/state checks and process-isolation controls. Live complete-checkpoint tests must also call bindRuntimeMetallibForMLX before their first MLX diagnostic or model load. Colocation alone does not establish the runtime digest required by the cache identity. The model-prefix fixture performs this binding explicitly; see model prefix qualification. Required demanded-checkpoint continuation CI uses the same staged SDK test image and scripts/run-nested-suite.sh as the existing native-retention and short-partition gates. Its seven scheduler/tiny-fixture methods must execute without optional model/GPU gates and with the wrapper's nonempty/no-skip checks. The provider's four adjacent-fork host methods are built with the ordinary ProviderCoreTests suite; eight independent pure collator controls remain distinct qualification evidence. A filtered suite's exit code alone does not certify execution. New same-donor adjacent native runs require a newly frozen source/binary/spec cut; prior four-role benchmark reports cannot qualify the added frontier. See the model-prefix test contract.

Staging prefers cp -c on Darwin to retain APFS cloning, falling back to ordinary cp if cloning fails or is unsupported. Other hosts, including Linux fixture runners, use ordinary cp directly. Copy or byte-verification failures leave the destination unchanged; verified copies replace it atomically. See the live-test setup for the pinned DiffusionGemma artifact and opt-in encrypted transport gate.

Advisory review tooling

The threat-model PR review uses Python 3.9+ standard-library HTTP/JSON modules and requires no package installation. CI runs its regression tests against local HTTP fixtures; the live workflow uses a repository Actions secret and the trusted base checkout. Full PR scans read immutable Git blobs as data and batch complete changed-file text; they never build or execute PR code. Set the activation variable before merge after verifying the state writer with the manual, zero-spend preflight option. Repositories that restrict branch updates can configure a dedicated App writer with short-lived tokens. Sonnet 5.5 handles the first pass; selected Opus 5.5 and GPT-6.1 Sol reviews use the same OpenRouter key. Atomic budget reservations, cached analysis and durable reports live on a dedicated state branch. Paid scanning defaults to disabled; follow the linked setup instructions to verify writer permissions and pilot caps.

Stripe migration maintenance

Build the audit/repair binary with go build -o /tmp/payout-audit ./coordinator/cmd/payout-audit. It uses the configured database without running migrations and defaults to read-only bounded output. Applying a refund requires an exact withdrawal ID, expected amount and an operator-verified Stripe request. See the cutover runbook.

Exercise the API, funding and settlement contracts with go test ./coordinator/tests/api/... ./coordinator/tests/billing/... ./coordinator/tests/store/... ./coordinator/tests/cmd/payout-audit. Set DATABASE_URL to a disposable local PostgreSQL database to run transaction, concurrency and rollback coverage. Never point tests at production. Console migration coverage runs with npm test in console-ui.

Telemetry archive worker

The independent Python worker uses scripts/telemetry_archive/Dockerfile and hash-pinned requirements.lock. Run uv sync --locked in that directory for local tooling; build the container for Linux/amd64. It does not rebuild or deploy the coordinator. The same worker supports isolated accounting archives and indexed ID batches; deploy each archive job with its own destination permissions. See telemetry history and accounting history.

telemetry-archive analytics-preview compiles catalog-pinned accounting analytics SQL locally; --execute runs capped SELECT-only BigQuery checks. query-submit, query-status and query-cancel provide bounded asynchronous custom queries; see query operations. These commands require no coordinator build or deploy. The operational/history design defines the broader storage boundary.

The opt-in analytics reader is built with the coordinator; its private sync-analytics-snapshot adapter is part of the Python worker. Neither command enables a producer or schedule. See analytics snapshots.

FilesExpand file tree

build.md

Latest commit

History