Part of #9.
Problem
For a distributed runtime, LLooM currently derives aggregate health from the management-reported health of every member. If a worker's LLooM control gateway is unavailable, the aggregate runtime becomes unhealthy even when the distributed inference endpoint remains healthy and the worker process is still participating in inference.
Request-time admission then rejects the placement or re-enters member lifecycle startup, turning a control-plane outage into an inference outage.
Reproduction
- Configure a two-node distributed runtime with a worker member and a head member.
- Start the runtime and verify the head serving health endpoint and a real inference request.
- Stop only the worker node's LLooM gateway; leave both inference members running.
- Verify the head serving endpoint remains healthy.
- Send the same inference request through the leader LLooM gateway.
Current behavior: the request is rejected because member management state is unreachable.
Desired contract
- Distributed runtime status separately exposes serving health and control/lifecycle health.
- If the configured distributed serving endpoint is healthy, request-time
ensure returns an already-serving result without invoking remote member lifecycle operations.
- Admission requires node reachability only when a new allocation or lifecycle mutation is actually needed.
- Start, stop, and reconfiguration continue to fail closed when required member control gateways are unavailable.
- Telemetry clearly reports a management-degraded state without calling the model unavailable.
Acceptance criteria
- Unit coverage for an aggregate runtime whose serving endpoint is healthy while one member gateway is unreachable.
- Request-path coverage proving inference continues in that state.
- Lifecycle coverage proving start/stop/reconfigure remain blocked or fail clearly.
- Recovery coverage proving member telemetry reconciles after the gateway returns.
- A real two-node failure-injection canary passes without exposing raw backend ports.
Non-goals
- Ignoring an unhealthy distributed serving endpoint.
- Allowing lifecycle mutation without control authority.
- Making backend ports public.
Part of #9.
Problem
For a distributed runtime, LLooM currently derives aggregate health from the management-reported health of every member. If a worker's LLooM control gateway is unavailable, the aggregate runtime becomes unhealthy even when the distributed inference endpoint remains healthy and the worker process is still participating in inference.
Request-time admission then rejects the placement or re-enters member lifecycle startup, turning a control-plane outage into an inference outage.
Reproduction
Current behavior: the request is rejected because member management state is unreachable.
Desired contract
ensurereturns an already-serving result without invoking remote member lifecycle operations.Acceptance criteria
Non-goals