Two-Mac cluster serving: real JACCL worker, guided link setup, coordinator and provider fixes

[jonathan308]

Summary

Experimental two-Mac ("cluster") serving: one model split across two Macs as a two-stage layer pipeline over MLX's JACCL backend (RDMA over Thunderbolt 5). This PR carries the provider and runtime side of the foundation extracted from the research drafts (#1226, #1227), completes the parts that extraction left out, and adds what was needed to run it on real hardware. Ordinary single-Mac serving is unchanged; everything here is opt-in (darkbloom start --local --distributed, darkbloom cluster …).

Status: draft. As of 2026-10-16 (4d0a2f475, adds darkbloom cluster local; see the first update below) and 2026-10-14 evening (round 6f, 69ba75be7) two Macs pair and serve with darkbloom cluster alone: the planner shards each model (pipeline with speculative decoding by default; tensor parallel when predicted faster), a missing model is copied over the cable, failover is automatic, the cluster API matches the single-Mac endpoint for tool calling and the common request types, and with the production coordinator the pair serves locally. See the 2026-10-14 updates.

Repository ownership (2026-10-09). #1408 moved all coordinator and backend code to Layr-Labs/darkbloom-platform. The coordinator half of this work is now a draft pull request there (number 6, branch feat/cluster-pairs), and coordinator/ on this branch is restored to master's content, byte for byte (763dea5b4). This pull request is provider-side only: provider-swift/, libs/, the shared wire-contract pages and the handoff files. Coordinator commits made before I had read that rule remain in the branch history; the tree no longer carries them, and none will be added.

Update 2026-10-16 (overnight hardware run, M4 Max 36 GB + M5 Max 128 GB)

Head 4d0a2f475 (fast-forward; all commits signed and Verified).

• a0a06c49f, 4d0a2f475: a cable moved to a never-set-up Thunderbolt port is named on the darkbloom cluster screen and set up with one key (f); the local API (single Mac and cluster) puts every model's reasoning in its own field, think-tag models included, on chat completions and the Responses API, while a stream whose prompt opens no think block still streams at once; the room-making covers the cache the gate already counts. Hardware: Gemma 4 26B restarted from the M5 Max's screen and passed the full API set; the Responses API (plain, streamed, tools) passes on the pair. Not yet run on the pair: the think-tag reasoning change with a Qwen model.

• 997c1fb70: a 36 GB Mac in everyday use keeps serving. With the owner's apps open the pair answered every request with 503 (each reservation on the loaded part asked for the 2.25 GiB load floor and had 1.6 GiB), and a restart said "Quit apps" for every model. A reservation on a loaded stage now asks for its own needs plus headroom, and a gate decision short only because it does not count droppable file cache lets the kernel drop that cache first (stopped at any compression, swap or pressure). Confirmed with the apps open and no help: Gemma 4 26B planned, started and answered in 37 s; the full API set passes; the M4 Max's compressor did not move.

• 8159469a4: fixes from a review of the night's commits. Relay: descriptors are closed only by the thread that owns them, stop waits until the port is free, it survives running out of descriptors, at most 64 connections with a 600 s idle bound, it runs only when the API listens on every address, and it leaves another local server's discovery file alone. Re-plan gives up after 3 same-split results. Page drop opens non-blocking. Charge memo keyed by MTP depth, limits and schedule. Error line logs the case or type only. Confirmed on hardware: relay under 4 parallel requests, then a switch that made the M4 Max rank 0 (the relay gave port 8000 back to it). Still open from the review: the memory margins that scale below 96 GiB (0d9057c86, the owner's request) need measurements on a small Mac.

• CI is green at 8159469a4 (and was at 0b480e3cc) (CI and SDK 27 release preparation pass) now that the PR merges cleanly. The first run (0ddc13e73) failed Release Integrity and the SDK 27 release lane on two release-workflow gaps from the master merge, and the unit lanes on the early-cancel probe (1 of 4,963 tests, failing before tonight too). Fixed in f17d6b9fc (provider-release-cache.yml passes the cluster helpers; release-swift.yml path filter lists the helper scripts; a check that every staging workflow passes them) and 0b480e3cc (the probe's missing cancel joins its existing known issue).

• All eight catalog models a cluster serves were brought up on this pair through darkbloom cluster local (Gemma 4 26B QAT, Qwen 3.8 27B, GPT-OSS 20B, Bonsai 2 27B, Nemotron 3.5 Lightning, Qwen3.5 35B A3B, Qwen 3.6 35B A3B, Qwen 3.5 9B; the two 28 GB Gemma builds were not downloaded for lack of disk on the 36 GB Mac). Qwen 3.5 9B (offered again after the manifest fix): copied in 3.1 s, first token 69 ms warm, ~65 tok/s with MTP. One key on every path: the other Mac reached the API over the LAN through the M4 Max's relay (first token 231 ms), over the cable (96 ms) and directly (88 ms).

• 87993a1ce, 0ddc13e73: a model's own pages are dropped from the file cache after a download, after a received copy, before planning and before a part starts (shared mapping + msync(MS_INVALIDATE), no privileges). Right after a download those pages sat in the active cache, which the load gate does not count: the 36 GB M4 Max admitted 1.7 GB and every model said "Quit apps"; dropped, 14.7 GB. This caused most of the night's refusals on that Mac. Qwen 3.6 35B A3B now serves with no help: planned, started and answering in 43 s; first token 78 ms warm, ~62 tok/s, soak 15/15. Synced with master (4a324c841 merges ab423c1c0; no conflicts left, so CI runs again).

• Full provider swift-testing suite on the merged sources (run on the M5 Max, matched metallib staged): 4,156 tests in 569 suites; it found the cluster API no longer answering exactly as the single-Mac API (my f8de477 gave only the cluster a default reasoning parser). f4dc0dd8a moves that default into the shared local HTTP layer for both APIs, for the channel formats only (Gemma 4, GPT-OSS Harmony); think-tag models keep the raw local stream on both. Parity (7 families), clean-stop and standalone-server suites pass (113 tests). Remaining failures: numerical-tolerance tests that fail on the M5 Max's GPU and pass on an M4 Max (GPT-OSS prefill output, CBv2 logits, sliding-window chunking), MiMo scanner tests that need their fixtures, and the aCancelRightAfterTheLaunchProbe probe (marked "not a requirement").

• Master sync (4a324c841): 14 conflicts resolved. libs/mlx-swift-lm now pins 167ca6bd, a merge of master's pin (mlx-swift-lm#311) into darkbloom/tp-engine (mlx-swift-lm#308, pushed there, Verified). The cluster-helper release steps moved into master's reusable release-provider.yml (with $APP_NAME) and the new dev-branch lane stages and ad-hoc signs the helpers; signing-validation tests 20/20. Every model-free runner passes on the merge, plus targeted swift tests (30/30). Hardware on the merged build: Qwen3.5 35B A3B (soak 15/15) and Gemma 4 26B QAT (stream, tools, tool result, JSON) through cluster local.

• Qwen3.5 35B A3B and Nemotron 3.5 Lightning serve. Qwen3.5 35B: first token 71 ms warm, ~61 tok/s, 5.5K prompt 1.7 s cold / 0.25 s repeated. Nemotron: first token 48 ms warm, ~67 tok/s, soak 20/20.

• 963b09925: the pair plans again when a Mac's load gate keeps refusing its share (proven on hardware: Qwen3.5 35B was planned with 24 of 40 layers on the 36 GB Mac, refused at load twice, then re-planned by itself to 36/4). Submodule pins unchanged; the cluster worker changed (prefix-cache budget), so both Macs need this build.

• GPT-OSS 20B serves through darkbloom cluster local (pipeline, M5 Max rank 0): first token 124 ms p50, decode ~75 tok/s, tools/JSON/5K prompt pass; soak of 30 sequential + 20 at 4 parallel with 0 failures, all reached through the other Mac's port.

• PrismML Bonsai 2 27B serves (copied M5 Max → M4 Max over the cable in 4.1 s at 2.1 GB/s): first token 146 ms warm, ~30 tok/s, prefix reuse (5.5K prompt 10.2 s → 0.39 s), soak 20/20. 201b91f5b: the worker's idle context refresh recomputed the Prism stage constants at every step of its search and starved requests (each failed after 10 s); now kept per plan/rank, idle worker CPU ~0%.

• c67bd1ddf: one address and one key for cluster local (the Mac that is not rank 0 passes its port to rank 0's API over the cable; the pair shares one API key, synced inside the keyed link channel; darkbloom local shows it on either Mac); a model chosen on a member is fetched there; the manifest encoding is the one this Mac's worker accepts (GPT-OSS pins the coordinator's, Qwen 3.8 the CDN's); plans use the other Mac's served speeds; no index estimates shown as seconds; fixes the runner build broken by 1f1baf0fb.

• Qwen 3.8 27B serves through darkbloom cluster local (pipeline + MTP, M5 Max 60 layers / M4 Max 4, no coordinator): downloaded on the M4 Max from the catalog (16.3 GB, verified), copied to the M5 Max over the cable in 8.0 s (2.03 GB/s), then served. Chat, streamed reasoning, tools (+ streamed, + tool result), JSON mode, stop, 5,554-token prompt, 4 concurrent (serialized), seeded. Decode ~47 tok/s, MTP acceptance ~83%, warm first token 105 ms; a repeated 5,554-token prompt starts in 0.28 s (7.10 s cold).

• 1e7948d5f: the leader downloads a model the cluster serves whatever its chip (the 27B's single-Mac "M5 only" requirement stopped an M4 Max leader).

• 646e5be9c: models taken from this Mac's own store were "not supported on a cluster yet" and copies were refused: the CDN's exact manifest was rejected by an order-sensitive file comparison and the coordinator's re-encoding written instead.

• 1f1baf0fb: (1) a memory-bound Mac's context now leaves the prefix cache a budget (it had 0 on the 36 GB Mac, which turned reuse off for both ranks); (2) a thinking model's reasoning streamed from the first token on the cluster API (superseded by f4dc0dd8a: think-tag models now keep the raw stream, as on a single Mac); (3) each Mac keeps its worker's own log, ~/.darkbloom/cluster-setup/worker.log.

• Observed, not fixed yet: stop sequences also match inside a thinking model's reasoning (GPT-OSS stop: ["7"] ended in its analysis); concurrent requests are served one at a time; Gemma has no prefix reuse or MTP on the cluster; a truncated non-streamed thinking answer returns its reasoning as content (same rule as single-Mac non-streaming);

Update 2026-10-15: darkbloom cluster local (serve the pair with no coordinator)

Head f8de47733 (fast-forward from 69ba75be7; 8 commits, all signed and Verified). Submodule pins unchanged.

• Hardware: darkbloom cluster local serves Gemma 4 26B QAT across an M4 Max 36 GB + M5 Max 128 GB pair (TB5 RDMA, pipeline, M4 Max layers 0-4 / M5 Max 5-29, no coordinator): models list, 401 without the key, chat, streaming (first token 61-64 ms, decode ~71 tok/s), tool call + streamed tool call + tool result, JSON mode, stop sequences, a 5,515-token prompt (first token 1.47 s, 3,741 tok/s prefill), 4 concurrent streams, seeded sampling identical twice, reached from the other Mac over the cable and over the LAN. Images are refused (known cluster limit); Gemma has no prefix reuse or MTP on the cluster yet.

• 0d9057c86: a smaller Mac takes the share it can hold. The stage gate's 6 GiB floor and 4 GiB loading headroom (measured on 256 GiB Macs) now scale with the Mac below 96 GiB (36 GB: 2.25 / 1.5 GiB); the 36 GB Mac's one-layer part was refused before. Also: a choice made on the member survives a leader restart, "Restart" only when nothing else frees memory, the last catalog is kept when it does not answer.

• f8de47733: fixes found on that pair: another server on port 8000 (oMLX) made every start end after both Macs loaded (now: a free port, checked before loading); Gemma's thought-channel markers reached content on the local API (now parsed by model type); the leader re-sent cache settings every 15 s to a member that drove the onboarding; the ready line could name an earlier split.

• Not proven yet: a 36 GB Mac with ~13 GB of apps open admits only ~3 GB for a part by the gate's file-cache rule; the run freed droppable file cache first (test harness, no settings changed). More models next.

• e7c0860d0: a cluster set up from the plain darkbloom command never started its leading part. The setup recorded the installer's PATH link (~/.darkbloom/bin/darkbloom) as each Mac's owner, and the owner start refuses symlinks by design ("Installed local owner is unsafe or not executable"); found on the 2x M4 Max / M5 Max pair, never seen on the bench (it ran the app by path). The setup now names the file; the check is unchanged.

• 4f763ab12: plainer onboarding (owner feedback from the same run): one line per model, details once chosen, no public-traffic line when serving locally only, ids for same-named models, "(this Mac)" never on the other Mac's screen, the Mac that joins carries cluster local on to the choice, pairing waits for the cable address, errors as words, and a part that keeps ending the same way is said after three tries instead of repeating.

• Hardware status (2x Mac, M4 Max 36 GB + M5 Max 128 GB): pairing, link setup, model choice, setup sync and the local-only start all ran through darkbloom cluster local; the leading part then hit the owner-path bug above (fixed). Serving not yet proven: the M4 Max's memory gate admits 1.2 GB now (file cache 15 GB after the model copy) against the 6 GB per-part minimum.

• b34d13506 (21:45): the cable can move to another port. Each Mac's cluster service bound its two link listeners (setup exchange, control lease) to the cable port active at start and never again; moving the cable to another Thunderbolt port (the 2x M4 Max pair went en3 → en1 → en2) left both Macs listening on dead ports and darkbloom cluster waiting at "Reading both Macs over the cable" until the services were restarted by hand. Every 5 s each listener now re-reads the active port and is started again there when the port or its address changed (ClusterSetupLinkExchange.cableMoved); a reading with no cable keeps it. Correction: at b95bb9e65 four console wizard checks still expected the old "planner cannot read" wording (my console run had compiled just before the wording change); b34d13506 updates them, console now 74,690 expectations / 38 groups pass.

What changed

• darkbloom cluster local [model] [--bind <address>]: the same onboarding as darkbloom cluster (pairing, model choice, planner, copies, MTP, caches, failover, stop, v), but no Mac of the cluster contacts a coordinator. The leader part starts with the new darkbloom start --local --distributed --no-coordinator, which never builds the control-only member connection, so a coordinator can no longer refuse the start (with production today, a refusal failed it).
• Saved until changed (owner decision 2026-10-15): the scope lives in the leader's setup record (ClusterOnboardingLeadership.localOnly / localBind) and survives restarts and reboots; l / p in darkbloom cluster switch. The cluster service keys rank 0's part by the scope, so a switch restarts that part and the screen waits until it serves. The member records the leader's scope from the start command (re-sent by the 15 s sync), so it holds if control moves; older peers carry none.
• API on the LAN/tailnet by default (owner decision), always with the local API key; --bind takes only localhost or a numeric IPv4/IPv6 address, and a saved address the API does not take falls back to loopback. darkbloom local prints a network: URL. The local API has no TLS (documented).
• Shorter onboarding screens (owner feedback): blocked models grouped by reason; two Macs on different builds named plainly ("the two Macs run different Darkbloom builds; install the same Darkbloom on both") instead of "Planning failed (Peer native build differs from the capability)"; "not supported on a cluster yet"; the development-build service line no longer explains login agents (the reason goes to the service log).
• macOS 27.0 SDK build fix (31ea55820): that SDK adds a nonisolated(nonsending) withTaskCancellationHandler overload, which made DistributedHTTPEventStream fail to compile; the call now names isolation: #isolation (no behaviour change).

flowchart LR
  subgraph Before
    A1[darkbloom cluster] --> B1["leader: start --local --distributed"]
    B1 --> C1{coordinator answers the member role?}
    C1 -- registers --> D1[public serving]
    C1 -- unreachable / no role --> E1[local API fallback]
    C1 -- refuses --> F1[start fails, restarts]
  end
  subgraph After
    A2[darkbloom cluster local] --> S2[(scope saved in setup record)]
    S2 --> B2["leader: start --local --distributed --no-coordinator --bind addr"]
    B2 --> E2[local API only: no member loop, nothing to refuse]
    A3[darkbloom cluster: l / p] --> S2
  end

Loading

Validation (model-free, M4 Max, macOS 27.2): onboarding checks 319/0 (local from scratch, switch while serving, l/p, named model, grouped reasons, scope values; failing first on 69ba75be7, where they do not compile), console 74,686 expectations / 37 groups (scope in the record, old records read as public, bad address → loopback), installed-session checks (start arguments, part key, log lines, wire round-trips), control 219/0, diagnostics pass; changelog check 25 valid; docs-check 189 OK. The darkbloom product builds (debug, --build-system native) with Xcode 27.0 beta's SwiftPM and the 27.0 CLT compiler; every new refusal checked on the binary.

Not yet (said plainly): no hardware run of cluster local yet (a test build is installed on a 2x M4 Max pair; result to follow); full provider swift-testing suite and the remaining check runners not re-run on this head; the AGENTS.md refactor-pass subagent is still to run; this branch still conflicts with master (no Swift CI until the master sync, next).

Build from this PR (both Macs must run the identical build): gh pr checkout 1407 -R Layr-Labs/d-inference && git submodule update --init --recursive, then cd provider-swift && swift build --build-system native --product darkbloom. Debug builds keep the pairing key in DARKBLOOM_DEBUG_SE_KEY_FILE.

Update 2026-10-14 (evening): rounds 6e and 6f: tensor parallel in the product, MTP on the sharded default, endpoint parity

Head 69ba75be7 (fast-forward from 30c084f88 via 6e c0fc4857d). All commits signed. Submodules from published Layr fork branches, each pin the head of a draft PR: libs/mlx-swift 37304a4d = Layr-Labs/mlx-swift#56, nested mlx 535d4e4e = Layr-Labs/mlx#35 and mlx-c 2415568 = Layr-Labs/mlx-c#15 (darkbloom/sealed-all-sum: the sealed all-sum), libs/mlx 535d4e4e = Layr-Labs/mlx#35, libs/mlx-swift-lm ba67a67a = Layr-Labs/mlx-swift-lm#308 (darkbloom/tp-engine, stacked on Layr-Labs/mlx-swift-lm#307). The fork PRs are stacked on Layr-Labs/mlx#33, Layr-Labs/mlx-c#14, Layr-Labs/mlx-swift#54 and Layr-Labs/mlx-swift-lm#307; merge each base first.

What changed

• Tensor parallel in the product for the dense Qwen rows (9B, 27B): each Mac loads only its shard from the verified checkpoint; every all-sum is sealed (AES-GCM, keys from the pinned data-plane exchange, rekeyed; tamper refused, no plaintext fallback); weighted shares for unequal Macs; distributed argmax / sampled gather; prefix cache per rank (both-or-neither); TP qualification records. The planner chooses TP or pipeline from measured inputs (link probe on the first load, per-rank shard prior from detected hardware, served-speed feedback at the shares that ran); a debug-only DARKBLOOM_CLUSTER_FORCE_MODE exists for pilots (release builds refuse it). Fixed in 6e: a normal end of request was read as a client stop and ended the session.
• MTP on the sharded default: pipeline_mtp_v1 records for the 27B, 9B and MiMo; the planner plans it by default from the records' measured speed-up (27B 35.6 → 55.6 tok/s on the bench pair, tokens equal to the plain pipeline).
• Records never block owner serving: a model is refused for the owner only if no sharded layout fits or its family has no cluster adapter; records gate public traffic and steer away from shapes that diverged on that chip pair; time budgets derived from the plan (no per-model rows).
• Endpoint parity: the cluster serves through the single-Mac pipeline's stop set, sampling defaults, tool-call constraint contract, SSE headers and failure frame (7 differences fixed); remaining differences are refused with the single-Mac error shapes (media for VL models, outputs over 4,096 tokens / prompts over 32K, token-constrained decoding, coordinator logprobs, one request at a time).
• Memory growth: rank-0 and worker-owner growth under load fixed (20.6 → ~0.7 MiB/min; owner flat).

Hardware (M3 Ultra + M5 Ultra, darkbloom cluster from fresh homes, coordinator without the member role → local serving)

• 27B: "pipeline with speculative decoding, M5 Ultra 56 layers / M3 Ultra 8 layers"; parity vs the single-Mac endpoint on 27 request types (tool call, tool result, two tools, streams, thinking, JSON, stops, max tokens, seeded sampling, invalid request): 27/27 structurally equal, 24/27 text-equal (two thinking texts and one long story differ late; same answers); seeded sampling identical twice on both.
• 15-minute mixed soak: 2,594 requests (tools, streams, thinking, JSON, stops), 0 failures; descriptors, footprints and wired memory flat.
• Forced tensor parallel (6e): 27B 59/41 ~36.6 tok/s, 9B 70/30 ~106 tok/s, seeded identical, client stop keeps the pair serving.

Tests: focused provider suites 558 tests / 72 suites (incl. the API parity tests), library and worker tests, all 35 model-free check runners.

Not in this head yet (on local branches, next merge): the tensor-parallel hang fix (rank lockstep + GPU-fence poisoning; 37 requests, 0 hangs), ~4x cheaper sealed all-sums, tensor parallel + MTP (27B 62–63 tok/s vs 39.5 plain TP), generic TP sharding for every family; A3B / VL / Bonsai parity runs; the remaining endpoint gaps above; coordinator Layr-Labs/darkbloom-platform#54 must list the new plan modes for a cluster-aware coordinator.

Update 2026-10-14: integration round 6 (6 → 6d): master sync, one-command setup, sharded deployments, no manual model downloads, failover, MiMo, chip-agnostic approvals

Head 30c084f88, on 2a6b186ae (round 5) via 6 (33da00709), 6c (34d7711fe) and 6d. All commits signed. libs/mlx-swift-lm pins 65af9799 (draft Layr-Labs/mlx-swift-lm#307; merge it with a merge commit).

What changed

• Master merged in (App Attest resume, the event-loop split, changelog fragments, backend removal); provider suite at master's count with only the 6 known baseline failures.
• One command: darkbloom cluster → setup → pick a model → serving. One admin approval per Mac, y for the pairing code. No plan/configure/flags/SSH for users; the setup screen states the deployment in one line (e.g. "Serving Qwen 3.5 9B across both Macs: pipeline, … layers / … layers"). 33 one-command findings fixed (setup entry points, offered set, two cable ports, updates, wording, FileVault/auto-login restart line).
• Sharded deployments only, chosen by the planner from detected hardware: pipeline (incl. compact decode and pipeline_mtp_v1) and tensor parallel modes in the protocol; phase split, prefill/decode hand-off hybrids and the per-request "alone" route stay in the code but are OFF by default (one developer opt-in); setups saved with a hand-off mode are refused and re-planned.
• No manual model downloads: the leader lists its installed models; a Mac missing the chosen model receives it over the Thunderbolt cable (encrypted link channel, digest-verified incl. file names, uncached, resumable, disk-checked, the receiving Mac's own worker checks the staged files before the exclusive rename) or it is downloaded once from the catalog.
• Derived control leader + failover: any Mac accepts darkbloom cluster commands (forwarded to the leader over the paired link); the leader is rank 0 and fails over at lease expiry; a Mac serves alone only after 10 minutes of absence and only if it can run the model; the pair re-forms by itself.
• Serve locally when the coordinator lacks the cluster member role (production today): the pair serves on the leader's local API with one plain line, devices stay held; a cluster-capable coordinator still gets the member registration.
• Chip-agnostic approvals: records approve a model + artifact + mode family on any Apple silicon the planner can place it on; a diverged record withholds only its exact chip combination + shapes. Mode-bound grants (a record for one mode family never approves another). New Macs without provider.toml now list the catalog.
• MiMo V2.6 Flash on the pair: qualified and served through the product (pipeline), prefix + SSD cache on by default, lookahead prefill, derived prompt cap; MTP on pipeline built (pipeline_mtp_v1).
• Reliability: F11 (restart after a pair-ending fault re-registers), F17/F18 (directory-descriptor and autorelease leaks that kept a pair down after a model switch), F14 (no solo crash-loop on a Mac that cannot run the model).
• Cache efficiency: restores page-aligned and handed to MLX without a copy, parallel artifact hashing, speculative history stored once, plan switches keep other plans' SSD records within one budget, the daily write cap survives restarts.

Hardware (M3 Ultra + M5 Ultra over Thunderbolt 5 RDMA, through darkbloom cluster from fresh homes)

• 9B: copied over the cable in 3.5 s, pipeline, 70 tok/s; repeated 8K prompt 2.03 s cold → 0.40 s resident → 0.50 s from SSD after both services restarted.
• 27B: pipeline 8/56, 35 tok/s; 5.26 s → 0.44 s → 0.67 s. MiMo: pipeline, 43 tok/s; 6.23 s → 0.35 s → 1.13 s.
• Against a coordinator built from platform master (no cluster role): the 27B served locally with the plain line. Model switches under load re-formed each time; descriptors flat over 15 minutes.
• Emulated 2x M4 Max with no provider.toml: the catalog listed from production, the 9B copied and served across both Macs.

Tests: provider suite (only the 6 baseline names), every cluster check runner, library + worker package tests at 6b; focused suites + 11/11 check runners at 6c; placement/onboarding/console/control/configuration runners at 6d.

Still open: tensor parallel serving in the product (in progress), pipeline_mtp_v1 records so MTP is planned by default, rank-0 memory growth under load (~20 MiB/min), records never blocking owner-only serving (built on a branch), Developer ID signed build + firewall-on reboot test (owner), coordinator draft Layr-Labs/darkbloom-platform#54 deploy (owner).

Earlier rounds (condensed; full text in this description's edit history)

• 2026-10-13 (evening): integration round 5 (prefix-SSD key off the environment, login agent, helpers packaging, paged engine, per-request router)
• 2026-10-13: integration round 4 (sampling over the link fixed, no orphaning, full caching with the SSD tier, leader-first onboarding, grants from qualification records)
• 2026-10-12: integration round 3 (SSH-free guided setup, public serving, paged KV and MTP by default, supervision, observability)
• 2026-10-11: integration round 2 (no fixed lifetime, 32K context, MTP and prefix reuse on by default, SSH-free link channel, setup wizard, MLX guard v2)
• 2026-10-10: every model family across both Macs, placement, clean hang-up, link setup v2, no orphaned memory

Linked issue

None.

Before and after

Before — the extracted branch could not start a collective, could not finish a distributed start, and gave the operator no way to see why.

flowchart LR
    classDef bad fill:#fde2e1,stroke:#b42318,color:#3b0d0a
    classDef ok fill:#e3f1e5,stroke:#1e7a34,color:#0b2e14
    classDef neutral fill:#eef1f5,stroke:#5b6676,color:#1a2230

    OP["Operator"]:::neutral --> START["start --local --distributed"]:::neutral
    START --> REG["Register as cluster member"]:::ok
    REG --> ACK["Coordinator never acknowledges"]:::bad
    ACK --> KILL["Leader tears down after 10 s"]:::bad
    START --> WRK["Launch rank worker"]:::neutral
    WRK --> NOSRC["Worker has no source in the tree"]:::bad
    WRK --> STUB["macOS 14 target compiles the JACCL stub"]:::bad
    OP --> LINK["Thunderbolt port without its own IPv4 address"]:::neutral
    LINK --> SILENT["Nothing reports it; JACCL refuses at init"]:::bad

Loading

After — a 26.2 worker package builds real JACCL, the coordinator (its side is in darkbloom-platform) acknowledges and guards members, and the link is inspected before anything is launched.

flowchart LR
    classDef ok fill:#e3f1e5,stroke:#1e7a34,color:#0b2e14
    classDef neutral fill:#eef1f5,stroke:#5b6676,color:#1a2230
    classDef pending fill:#fff3d6,stroke:#9a6700,color:#3d2a00

    OP["Operator"]:::neutral --> CL["darkbloom cluster link"]:::ok
    CL --> VERDICT["Per-port verdict and one sentence of guidance"]:::ok
    OP --> STAGE["darkbloom-cluster-stage-check"]:::ok
    STAGE --> LOADED["Each Mac loads and releases its stage of the real model"]:::ok
    OP --> START["start --local --distributed"]:::neutral
    START --> REG["Register as cluster member"]:::ok
    REG --> ACK["cluster_member_accepted"]:::ok
    START --> WRK["darkbloom-cluster-worker, macOS 26.2 build"]:::ok
    WRK --> JACCL["Real JACCL group"]:::ok
    JACCL --> XPORT["Collectives over RDMA: exact on two Macs"]:::ok
    XPORT --> RUN["Two-rank generation of the real model"]:::ok
    RUN --> SERVE["Served by start --distributed and the coordinator"]:::pending
    OP --> SETUP["darkbloom cluster: detect, one approval, ready"]:::ok

Loading

Green is exercised by a test or a run recorded below; amber is built but not yet run on two Macs.

What changed

Native worker package (libs/darkbloom-cluster-worker, new). The provider and the shared cluster modules build for macOS 14, where the pinned mlx-swift compiles the JACCL stub. A sibling package with a macOS 26.2 minimum builds the real backend with no submodule pin change. build-native-worker.sh refuses a binary that holds the stub or whose Mach-O minimum is not exactly 26.2. Products:

• darkbloom-cluster-worker — the rank worker the installed plan launches (from Preserve shared distributed runtime and provider integration #1226).

• darkbloom-cluster-collective-check — both ranks run it; verifies reductions, point-to-point transfers and ordering byte for byte. No model.

• darkbloom-cluster-stage-check — one Mac, one rank, the real artifact, no collective: verified stage load, release, and memory before and after.

• darkbloom-cluster-reference, darkbloom-cluster-pair-check — a single-Mac staged reference, a two-Mac driver (rank 0 local, rank 1 over SSH, never signals a worker) and a comparator with a closed set of verdicts (exact, tokensEqualLogitsDiffer, divergedAtNearTie, diverged, incomparable).

A pair could not decode before this PR: rank 0 handed over its residual while Metal completion handlers still held the buffer, and the sender refuses storage it does not own alone. One GPU stream synchronization on rank 0's path fixes it (fix(cluster): drain the GPU stream before rank 0 hands over its residual). That change is on the serving path and wants a careful read.

A rank's last resort against a native call that never returns is now a deadline thread (ProcessDeadline), not only alarm: on real hardware a rank spinning in the RDMA completion poll ran more than nine minutes past its alarm.

Coordinator. Moved to Layr-Labs/darkbloom-platform (draft pull request 6 there): the member acknowledgement, the pair lifecycle and quarantine release, pair formation, routing an owner's requests to its pair's leader, and the owner views. Nothing under coordinator/ differs from master here.

Provider

• darkbloom cluster (no subcommand): guided link setup. It checks RDMA, waits for a Thunderbolt connection, and if the connected port has no IPv4 address of its own it gives it a link-local one through one macOS approval prompt and installs a small root launchd job (one file under /Library/LaunchDaemons, two system tools, no installed program) that re-adds the address whenever the port has none; then verifies. A one-time address proved not durable on a Mac whose port is a bridge member: it was gone about two hours later with no restart or replug. --temporary keeps the address-only path, --dry-run prints the exact commands and changes nothing, and cluster link --remove removes the job, its file and the address. cluster link --fix, --remove and --watch are the pieces. Darkbloom never calls sudo. The root job has not run behind a real approval yet.
• Restored from Preserve shared distributed runtime and provider integration #1226 (dropped by the extraction): the streamed-completion terminal, the first-token budget on distributed submits, refusal before response headers, member-role registration, member-mode guards on load, prefetch, inference and autopilot, and the --cluster-member start mode; 21 research test files ported.
• Member side of coordinator-formed pairs: cluster_membership in the member registration, the pair approval saved by cluster configure, the member control installed before connecting and attached at acceptance. This build declines every pair preparation on purpose: a coordinator-formed pair needs an owner mode, a worker bootstrap and a rank-1 request channel that are not in the tree, so a member answers 503 instead of half-serving.
• darkbloom cluster console (and bare darkbloom cluster on a terminal that can draw): an interactive screen over the operations above. Readiness comes from the link probe and doctor, pairing and trust from the saved setup with an explicit approval that counts only once the whole question has been on screen for 500 ms, the admitted models from the runtime's own list; it can fix the link, start and stop the local session, recover a stranded journal and write a redacted diagnostic export. handoff/TUI-wiring.md lists 53 screen items: 31 wired to a real operation, 10 whose operation exists but is not connected, 12 with no operation yet, and a check fails if the screen shows anything outside the wired set. --plain and --json print the same state without a terminal.
• darkbloom cluster link [--json], also run by cluster doctor: read-only inspection of RDMA state, port state, interface address, bridge membership and the IPv4-mapped GID, with a closed set of verdicts. No address, MAC or GID is kept from tool output.
• Write provider.toml owner-only and atomically; a file first created by an ordinary save was 0644 and the cluster readers refused it.
• Installed path (start --local --distributed):
  • It could not start a worker at all: the owner always passed the owner-authenticated bootstrap arguments and the worker refuses them. One place now selects the bootstrap; this build selects the runtime's own exchange on the configured link address, and cluster status and cluster doctor say that nothing vouches for the peer that answers.
  • A fence closes the worker's command stream and nothing else. Before, the owner sent SIGTERM at the fence and SIGKILL two seconds later, to a worker that still held its model. The owner now signals only a process that outlived its own hard deadline.
  • The owner clears its own device journal once it has seen its child exit. Before, any failed launch left a journal that refused every later start. darkbloom cluster recover clears a journal whose owner died, and refuses while any process still holds the device or carries the recorded epoch.
  • The worker takes an optional startup deadline and ends itself when it is not ready by then.
  • The plan sets a JACCL progress limit for every rank, and the leader, the owner and cluster doctor refuse a worker binary that does not carry the progress guard.
  • The owner refuses to start a worker while this Mac's ordinary provider holds the instance lock.
  • The plan selects the generation mode beside the prefill schedule: the pipeline by default, so a saved setup behaves as before and its bytes do not change; a mode that a worker's capability record does not advertise is refused before launch. Startup, first-token and shutdown budgets come from the selected registered model, with the 9B's unchanged. The 27B stays closed to a distributed start, and the refusal says it is a policy decision.

Runtime

• A second registered model. The resident adapter selects its model from a closed catalog (QwenResidentModelDefinition), and admission, byte ceilings, capability and the worker's model and cut validation are per registered model. The registered Qwen3.8 27B is admitted with every structural cut; the 9B's receipts, capability bytes and tests are unchanged.

• Phase split, a second generation mode beside the pipeline, for both registered models: the pair prefills with lookahead, rank 0 hands its committed request state to rank 1 after the first token (one checked transfer per state tensor, verified by rank 1 before it adopts, a 20 s limit bound into the agreement), and rank 1, which holds the whole model, decodes alone and relays tokens to rank 0 in batches of 1, 2, 4, 8 then 16: 41 transfers after the first token for 128 outputs against 1,397 in the pipeline. The worker takes --generation-mode and advertises the modes it supports; the pipeline stays the default and its agreement bytes are unchanged and pinned by a test. A correctness-only local-socket transport lets two ranks run on one Mac for tests; it and the fault-injection switch are refused unless a worker is started with --qualification-switches yes.

• Benchmark tooling under scripts/benchmarks/cluster/ (a streamed load generator with a self-test, an isolated serve harness, artifact verification against manifests) and the night's report, docs/reports/2026-10-09-two-mac-baseline-and-model-matrix.md: the single-Mac "before" through the product's serving path on both Macs, and the state of every catalog model on each.

• Six of twelve slices of delivering a follower's stage over the link (design and measurements in handoff/DESIGN-stage-transfer.md): a pinned per-tensor content inventory with a generator and an independent Python cross-check, the pin for the registered 9B, the transfer plan and state machines, a loader seam that separates stage metadata from where the bytes come from (all 14 existing load receipts unchanged), stage metadata derived from the pin with no weight file, and the MLX-side intake with a stage-check transfer mode. In one process on the M3 Ultra a follower's stage (3.98 GB) is received and verified in 1.65 s against 4.72 s for a local load of the same stage, and corrupted, swapped and truncated inputs are refused; the follower is ready about when a local load would be, so the transfer buys a follower with no weights on disk, not time. Nothing has crossed the link or run between two processes, and the load path does not use it yet.

• Host gate: swap left from earlier is admitted while memory pressure is normal and refused under warning pressure. The previous rule refused any Mac with swap in use. Free-page requirements are unchanged.

• Resident admission tests: a real positive path on both ranks from the registered metadata, and one property per refusal. The previous suite could not fail for the reason each case named.

Test plan

Run on an M3 Ultra (256 GB, macOS 27.2) and an M5 Max (128 GB, macOS 27.0), Xcode 27.0, one Thunderbolt 5 cable.

• The focused cluster checks (17 runners, including ClusterLinkChecks with 1,353 expectations)
• swift test --package-path libs/darkbloom-cluster — 22 tests in 5 suites; DarkbloomClusterWorkerTests built for 26.2 — 9 tests
• make provider-test after merging master — the same 75 tests fail as on unmodified master on this machine (same 148 issues, same 56 XCTest errors; mostly a missing pagedattention.metal test resource and MiMo fixtures). The branch adds 142 tests to those targets and none of them fail
• darkbloom-cluster-stage-check on the registered Qwen3.5 9B artifact: every supported cut and both ranks on the M3 Ultra, cut 4 and both ranks on the M5 Max. Each loads in about 3 seconds, releases to a few KB active and 0 cached, and both Macs report the same storage commitment
• darkbloom cluster on both Macs: one reported ready; on the other the approval prompt was accepted, the address added and verified
• darkbloom-cluster-collective-check across the two Macs: 5.36 GiB of reductions and transfers, 0 mismatches on both ranks, IP byte counters at 0.0003% of the payload
• Dead peer on the real link: with the pinned JACCL the surviving rank spins until the deadline thread ends it; with the fork's progress guard it reports the error and exits about 4 seconds after its peer. No process or wired memory left on either Mac
• The real model across both Macs (rank 0 M3 Ultra, rank 1 M5 Max, fork JACCL with the progress guard): nine requests, 30 to 8,192 prompt tokens, cuts 4 and 16, serial and lookahead prefill, all complete with clean shutdown
  • Against each Mac alone: tokens equal for short prompts; for long prompts the first difference, where there is one, is at a tie or within 4 ulp between the top two logits. Not bit-exact across chips, and never a plain divergence. The two Macs alone differ from each other the same way; one Mac is exact across cuts; the pair is exact run to run
  • Prefill at 8,192 tokens: 971 tok/s (M3 Ultra alone), 2,788 (M5 Max alone), 2,160 (pair), 3,069 (pair, lookahead). Decode: 60–65, 77–81, 59–63. Pair figures include control and transport; single-Mac figures are summed frame times
  • Rank 1 ended with SIGTERM mid-decode with both stages loaded: rank 0 reported the JACCL error at its 60 s limit and exited by itself; no process on either Mac; wired memory back at its earlier level; the next run exact
• At 63fd73706 (after the 27B and stage-transfer commits): cluster library 37 tests in 8 suites; worker package 48 XCTest cases; all 18 check runners (11 library, 3 worker package including StageTransferChecks, 4 provider; the link runner has 2,072 expectations in 32 groups); the provider builds
• Registered Qwen3.8 27B, each Mac alone: artifact verified (14 of 14 files) on both; stage load and release for cuts 12, 16 and 20 on both ranks and both Macs with equal storage commitments; single-Mac references at 30, 4,096 and 8,192 prompt tokens deterministic on each Mac and exact across cuts; between the two Macs every token equal, logits differ. Prefill about 300 tok/s (M3 Ultra) and 770–880 (M5 Max), decode 27–29 on both. No pair run: the second Mac's port had no address
• darkbloom cluster --dry-run on both Macs with the new build: the M3 Ultra needs no prompt; the M5 Max prints the fourteen commands the approval would run and changes nothing. The job definition, built by the same command chain, was loaded as a harmless stand-in in a user launchd domain: runs at load and every 10 s, survives six immediate reinstalls, unloads cleanly
• The full provider unit suite at 4e5f70793, built with --build-system native as the release tooling builds it: 4,539 tests in 621 suites, 5 failing, the same five that fail on unmodified master on this machine (three MiMo tests that need CI's fixtures, two that depend on the home directory). With the toolchain's default build system 75 fail on master too, because nested resource bundles are not found
• Library tests 49 in 9 suites, worker package 48 XCTest cases and all 18 check runners at the same head
• At 657a22d8f (phase split and console): library tests 72 in 10 suites; worker package tests (16 and 41); 19 self-contained check runners, the console's with 11,689 expectations in 23 groups on pseudo-terminals; the full provider suite, native build system, 4,544 tests in 622 suites, failing only the five environment tests that fail on master
• Phase split on each Mac separately, both models: 26 of 26 comparisons against that Mac's reference exact (9B at cut 8, 27B at cut 16; two real workers over the local test socket, and the in-process hand-off); a flipped bit, a stalled rank and a rank ended on either side each end both ranks by themselves with nothing left allocated; hand-off 26–71 ms for the 9B and about 125 ms for the 27B's 173 MB at 8,192 tokens
• The console on one Mac: a 23-screen dry-run tour in a pseudo-terminal; q, Ctrl-C, SIGTERM and SIGHUP restore the terminal; a second instance is refused; fifteen existing read-only commands print the same output as before
• At 895aece69 and the test-only 12250136a (memory gate second version, stage-transfer slices 3, 4 and 6, installed-path mode selection): library tests 102 in 14 suites; worker package tests; 20 check runners (the worker startup checks 15 of 15 against a built worker); the full provider suite, native build system, 4,548 tests in 623 suites, failing only the five environment tests that fail on master
• Phase split across the two Macs; a real start, client request and stop from the console
• One HTTP completion through darkbloom start --local --distributed on the two Macs, then the lifecycle faults
• The same through the coordinator with two real members (coordinator side: darkbloom-platform draft 6)
• The 27B across the two Macs; the durable link job behind a real approval

make docs-check reports one broken link, in docs/reports/2026-10-03-all-model-prefix-qualification.md; it is on master (from #1326) and unchanged here. make docs-impact-check passes.

Components touched

• coordinator (Go) — moved to Layr-Labs/darkbloom-platform
• provider (Rust, legacy)
• provider-swift (Swift CLI)
• console-ui (Next.js)
• enclave (Swift)
• infra / CI / release
• docs

Protocol / interface changes

• No protocol/interface changes

• Yes — described above and matching side updated

• The provider registers cluster_membership in the member role and expects cluster_member_accepted; the coordinator side of both is in darkbloom-platform draft 6. A coordinator without it ignores the fields and never sends the acknowledgement, and the provider's negotiation timer then ends a distributed start.

• New CLI: darkbloom cluster (guided setup), darkbloom cluster link, darkbloom cluster recover. cluster doctor now also fails when the local link is not ready or the worker binary has no progress guard. cluster status reports the bootstrap selection and the progress limit.

• New optional worker argument --startup-deadline-uptime-nanoseconds; exit status 123 when it expires. New optional worker arguments --generation-mode and --qualification-switches; supportedGenerationModes in the capability record, omitted for a pipeline-only runtime so existing records keep their bytes. An owner and a worker must come from one tree: an owner without this change refuses a new worker's capability record.

• New CLI: darkbloom cluster console [--json|--plain] [--dry-run] [--temporary].

• New decision-receipt policy identifier registered_dense_selected_stage_load_resources_v2.

• New runtime model ID registered_qwen38_27b; the worker validates --model-id and --stage-cut against the closed catalog. New optional field contentInventorySHA256 on a registered specification, in no existing fingerprint.

• darkbloom cluster gains --dry-run and --temporary; cluster link reports gain addressKept, and guidance can be non-null while the state is ready.

• No release artifact or version change.

Documentation impact

• Canonical documentation updated
• No documentation needed — reason:

docs/reference/cluster-control-protocol.md, protocol-messages.md, configuration.md, docs/provider/installation.md (uninstall removes the link job), and the two package READMEs. Coordinator-only pages are back at master's text; citations of coordinator files that moved are marked as paths in the platform repository. make docs-impact-check passes; make docs-check reports only the broken link that is on master.

Not shown by this PR

• The provider's own distributed start has not served a request on the two Macs. The two defects that stopped it (bootstrap flags the worker refuses, workers ended on two-second timers) are fixed and covered by checks, but the end-to-end run is still to do. The model run above used the qualification driver.
• Against the product's own serving path the pair is not faster end to end for this model. Qwen3.5 9B, 8,192-token prompt, 128 outputs: first token 8.59 s on the M3 Ultra, 2.97 s (rested) to 3.78 s (settled) on the M5 Max, 2.67 s on the pair in the earlier runs; decode 77.9, 101.1 / 97.8 and 61.6 tok/s; total 10.32 s, 4.33 / 5.15 s and 4.75 s. The pair reaches the first token sooner on long prompts, is slower at 1,024 tokens, and decodes at half to four fifths of either Mac's product path, because the cluster runtime's decode is slower than the serving engine's even on one Mac and has no MTP. A phase split (pair prefill, then one rank decodes alone) is in this pull request and proven on each Mac separately; it has not run across the two Macs. A design for handing the pair's prefill to the serving engine is in handoff/DESIGN-pair-prefill-engine-decode.md; by its arithmetic the gain for this model is a few percent at the one measured pair configuration. A layer pipeline helps prefill and models that do not fit one Mac; tensor parallel is not in this runtime.
• When one rank fails the other waits the whole progress limit before failing, because no cancel is sent to it.
• The pinned JACCL has no progress bound and sends stale bytes after a partial payload. The fixes are drafts on the forks: JACCL: zeroed send-frame tails, a progress guard, the advertised GID and the upstream JACCL fixes mlx#33 (with the upstream JACCL fixes) and the pin in build: pin mlx and mlx-c to the JACCL fixes and the owned bootstrap bridge mlx-swift#54. This PR does not move libs/mlx or libs/mlx-swift. Until it does, the guarded JACCL exists only in builds that check out the mlx branch by hand, and the installed path refuses to serve without it.
• The data plane between ranks is plaintext; the authenticated record transport has no caller yet.
• Two registered models are admitted by the runtime, the Qwen3.5 9B and the Qwen3.8 27B. Only the 9B has run across two Macs. The provider still gates the 27B model IDs on M5 plus NAX and a member's capability set never contains NAX, so a distributed start refuses the 27B on any Mac; serving it on a mixed pair needs a decision, not a workaround. A session is bounded to 16 requests and 300 seconds.
• The host memory gate changed; this is its second version, after an independent review of the first. The old gate admitted on truly free pages only and refused a Mac whose memory is mostly file cache, the normal state after a download or a hash pass. The rule here counts file cache above the kernel's file-cache reserve while pressure is normal and the cache is inactive, and stops a load or request at its next check when the Mac's compressed or swapped-out pages have grown since its first sample (handoff/DESIGN-resource-gate-v3.md). The review's concern about the first version was confirmed on hardware: with the cache held mapped and read by another process, a load on the M5 Max compressed 18.4 GiB while the pressure level still read normal; that is what the growth check is for. Run on both Macs for both models: the after-download state, cache made of the model being loaded, cache held by another process, two loads started together, and held loads that must still be refused. Not done: each of its commits built and tested on its own, and the comparison of the final binaries with the ones the hardware runs used. Small Macs are still refused from the after-download state (a 32 GB Mac never admits a 10 GiB stage). This is a resource-admission policy change and wants a decision before it is relied on anywhere else.
• A coordinator-routed request does not reach a real pair yet. The coordinator side (darkbloom-platform draft 6) forms, routes and hands off, tested with a fake member, and the provider member here declines every preparation: it still needs an owner mode for a coordinator-chosen epoch, the worker bootstrap attachment (which needs the mlx-c bridge of Preserve the owned JACCL bootstrap callback bridge mlx-c#14), a rank-1 request channel and the serving seam. Only the owning account's requests are routed; billing and usage rows for a pair are not designed.
• The durable link address is built but unproven: nothing behind the macOS approval has run, including the root job, its Login Items entry, how fast the address and its GID return after a strip, and whether a running session survives the gap. What removed the one-time address is narrowed (no sleep, restart or cable event; the port is a member of a bridge that Internet Sharing runs over) but not proven.

Notes for reviewers

• Refactor pass. The link work had its own refactor pass and a second read-only review. The coordinator half had both before it moved. The library changes for the 27B and the stage-transfer slices have had correctness reviews by their authors' reviewers but not yet the dedicated refactor pass this repository requires; it will run over the library once the branches still in flight (phase split, memory gate) have landed, and this pull request stays a draft until then.

• A wait that could hang. Two provider unit tests in the suite "Owned native member invocation" failed intermittently in CI (12 of 30 isolated runs locally). The cause was a product defect, not the tests: Process.waitUntilExit() for a process started from a dispatch thread returned late and was sometimes never reported, so a member never learned its owner had exited, stayed quarantined and never released its pair. ClusterProcessExit now takes the exit from the termination handler and latches it, at the remote endpoint, the worker process pump and the capability probe; no bound was loosened and no signal added or moved. A check runs 300 sessions from a dispatch thread and fails at about session 47 on the old files. After: 0 of 30 isolated suite runs. Worth a reviewer's eye: the one SIGKILL that can reach a worker is in the worker process pump, after the worker's own hard deadline, SIGTERM and a further margin (the documented exception in ClusterWorkerSignalPolicy); and the endpoint can SIGKILL an owner past its ceiling, which would leave a still-live worker with a closed command stream and nobody to clear the journal.

• AGENT-HANDOFF.md and handoff/ are working documents (status by gate, gap map, evidence ledger, runbook). They are here so the state is reviewable; drop them before merge if they do not belong in the tree.

• The swap-gate change is a resource-admission policy change. The previous rule came from the research measurements; I found no measurement behind it, but it wants a second opinion.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
d-inference	Ready	Preview	Oct 16, 2026 5:00pm UTC
d-inference-console-ui-dev	Ready	Preview	Oct 16, 2026 5:00pm UTC
d-inference-landing	Ready	Preview	Oct 16, 2026 5:00pm UTC
darkbloom-status	Ready	Preview	Oct 16, 2026 5:00pm UTC
eigen-homepages-darkbloom	Ready	Preview	Oct 16, 2026 5:00pm UTC