Skip to content

[Bug] Keep distributed inference serving when a member control gateway is unavailable #10

Description

@data-angel

Part of #9.

Problem

For a distributed runtime, LLooM currently derives aggregate health from the management-reported health of every member. If a worker's LLooM control gateway is unavailable, the aggregate runtime becomes unhealthy even when the distributed inference endpoint remains healthy and the worker process is still participating in inference.

Request-time admission then rejects the placement or re-enters member lifecycle startup, turning a control-plane outage into an inference outage.

Reproduction

  1. Configure a two-node distributed runtime with a worker member and a head member.
  2. Start the runtime and verify the head serving health endpoint and a real inference request.
  3. Stop only the worker node's LLooM gateway; leave both inference members running.
  4. Verify the head serving endpoint remains healthy.
  5. Send the same inference request through the leader LLooM gateway.

Current behavior: the request is rejected because member management state is unreachable.

Desired contract

  • Distributed runtime status separately exposes serving health and control/lifecycle health.
  • If the configured distributed serving endpoint is healthy, request-time ensure returns an already-serving result without invoking remote member lifecycle operations.
  • Admission requires node reachability only when a new allocation or lifecycle mutation is actually needed.
  • Start, stop, and reconfiguration continue to fail closed when required member control gateways are unavailable.
  • Telemetry clearly reports a management-degraded state without calling the model unavailable.

Acceptance criteria

  • Unit coverage for an aggregate runtime whose serving endpoint is healthy while one member gateway is unreachable.
  • Request-path coverage proving inference continues in that state.
  • Lifecycle coverage proving start/stop/reconfigure remain blocked or fail clearly.
  • Recovery coverage proving member telemetry reconciles after the gateway returns.
  • A real two-node failure-injection canary passes without exposing raw backend ports.

Non-goals

  • Ignoring an unhealthy distributed serving endpoint.
  • Allowing lifecycle mutation without control authority.
  • Making backend ports public.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions