Skip to content

[Feature] Add inference-only gateway failover without multiple lifecycle owners #12

Description

@data-angel

Part of #9.

Problem

A multi-node distributed model can remain healthy while the single client-facing LLooM leader gateway is unavailable. Without a standby inference path, the gateway remains a single point of failure even after member control-plane degradation is handled correctly.

Naive active/active gateways are unsafe because independent runtime managers could issue conflicting admission, start, stop, or eviction decisions.

Proposed direction

  • Support an inference-only standby gateway that can proxy already-serving model endpoints.
  • Keep lifecycle ownership explicitly single-leader until LLooM has a durable distributed lease/lock design.
  • Expose failover readiness and current lifecycle authority in status output.
  • Define a client discovery or virtual endpoint contract that does not expose raw backend ports.
  • Ensure the standby cannot mutate runtime state without an explicit authority transition.

Acceptance criteria

  • Loss of the leader gateway does not interrupt requests to an already-serving distributed runtime when the standby is healthy.
  • The standby rejects lifecycle and configuration writes while it lacks authority.
  • No dueling admission, eviction, start, or stop operation is possible.
  • Recovery of the leader reconciles cleanly without duplicate work.
  • Failure-injection tests cover leader loss during idle and active inference.

Non-goals

  • General distributed consensus in the first implementation.
  • Public exposure of backend model servers.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions