Part of #9.
Problem
The current Spark deployment path updates one host at a time. It does not provide a cluster-level transaction that proves every required node is staged on the same artifact, orders gateway restarts, runs a distributed canary, and rolls the cluster back coherently.
Gateway-only releases may also restart an otherwise healthy model runtime when no runtime contract changed.
Proposed solution
Add a coordinated multi-node release workflow that:
- Preflights required nodes, service persistence, credentials, config compatibility, and current workload.
- Stages and verifies one immutable artifact and manifest on every target node.
- Restarts node gateways in a safe order while preserving healthy inference where possible.
- Restarts a runtime only when its effective runtime contract changed.
- Runs cluster health checks and a real gateway inference canary.
- Promotes all nodes together or rolls affected nodes back to their recorded prior artifacts.
Acceptance criteria
- Artifact and effective configuration digests are recorded per node.
- Partial staging or restart failure leaves the previous serving path recoverable.
- Gateway-only deployment does not restart unchanged inference containers.
- Canary verifies the public LLooM gateway model ID and selected runtime attribution.
- Rollback is exercised in automated failure-injection coverage.
- Human-readable and
--json receipts identify staged, active, promoted, and rolled-back nodes.
Non-goals
- Multiple independent lifecycle leaders.
- Deploying unreviewed configuration or model weights.
Part of #9.
Problem
The current Spark deployment path updates one host at a time. It does not provide a cluster-level transaction that proves every required node is staged on the same artifact, orders gateway restarts, runs a distributed canary, and rolls the cluster back coherently.
Gateway-only releases may also restart an otherwise healthy model runtime when no runtime contract changed.
Proposed solution
Add a coordinated multi-node release workflow that:
Acceptance criteria
--jsonreceipts identify staged, active, promoted, and rolled-back nodes.Non-goals