Skip to content

[Feature] Add transactional multi-node deployment and rollback #11

Description

@data-angel

Part of #9.

Problem

The current Spark deployment path updates one host at a time. It does not provide a cluster-level transaction that proves every required node is staged on the same artifact, orders gateway restarts, runs a distributed canary, and rolls the cluster back coherently.

Gateway-only releases may also restart an otherwise healthy model runtime when no runtime contract changed.

Proposed solution

Add a coordinated multi-node release workflow that:

  1. Preflights required nodes, service persistence, credentials, config compatibility, and current workload.
  2. Stages and verifies one immutable artifact and manifest on every target node.
  3. Restarts node gateways in a safe order while preserving healthy inference where possible.
  4. Restarts a runtime only when its effective runtime contract changed.
  5. Runs cluster health checks and a real gateway inference canary.
  6. Promotes all nodes together or rolls affected nodes back to their recorded prior artifacts.

Acceptance criteria

  • Artifact and effective configuration digests are recorded per node.
  • Partial staging or restart failure leaves the previous serving path recoverable.
  • Gateway-only deployment does not restart unchanged inference containers.
  • Canary verifies the public LLooM gateway model ID and selected runtime attribution.
  • Rollback is exercised in automated failure-injection coverage.
  • Human-readable and --json receipts identify staged, active, promoted, and rolled-back nodes.

Non-goals

  • Multiple independent lifecycle leaders.
  • Deploying unreviewed configuration or model weights.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions