Replace first-strike RpcError eviction with a health-aware per-worker circuit breaker

[conradbzura]

Description

Today the dispatch loop classifies every RpcError from WorkerConnection.dispatch into a binary contract (the worker-health contract on wool.runtime.worker.connection.RpcError): a TransientRpcError makes the balancer skip to the next candidate without eviction, and any non-transient RpcError evicts the worker on its first occurrence via WorkerProxy._delegate_dispatch → ctx.remove_worker(...). Eviction is immediate and permanent — an evicted worker is only re-admitted if discovery re-emits it.

Introduce a third classification tier and a per-worker circuit breaker (passive outlier detection) sitting between "skip forever" and "evict on first strike":

• Transient (unchanged) — UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED: skip to the next candidate, no strike, worker retained.
• Suspect (new middle tier) — retried-with-strikes: the current request is served by the next candidate (skip, like transient), but a strike is recorded against the worker; the worker is ejected only once its strikes cross a threshold, and ejection is time-based and recoverable (see Expected outcome). An INTERNAL that wool did not deliberately set is the natural member.
• Non-transient / caller-fault (unchanged) — malformed Nack, protocol/version faults, caller-side encode failures, and wool's own deliberate abort(INTERNAL) (see below): evict-now or propagate.

With an unmarked INTERNAL treated as a strike rather than a first-strike eviction, the stream-open starvation fault never needs to be fingerprinted at all — a transient blip is erased by a subsequent success, and a persistently-INTERNAL worker is ejected by accumulated strikes.

Motivation

The load balancer classifies every dispatch fault into a binary contract — transient (skip, retain) or non-transient (evict) — and eviction is first-strike and permanent: one qualifying fault removes a worker for good, re-admitted only if discovery happens to re-emit it. This model has repeatedly dropped healthy, still-serving workers on faults that were transient or ambiguous, draining pools to a spurious NoWorkersAvailable while the worker's process was alive the whole time.

Two structural roots:

1. Status codes are a lossy instantaneous health signal — INTERNAL most of all. gRPC surfaces INTERNAL for a mix of causes with no field that reliably separates them: application-set aborts (wool itself aborts INTERNAL deliberately at service.py:407,549 as an evict signal), deterministic marshalling violations (compression mismatch, message-cardinality, serialization — retrying never helps), and transient C-core / flow-control faults (load-induced, clears on retry). No status field distinguishes the transient case from the deterministic one, so a single-event classifier is lossy by construction — and the same holds, to a lesser degree, for other codes a caller might want to forgive briefly rather than act on immediately. The robust signal is behavior over time, not a single response.

2. First-strike, permanent eviction has no margin and no recovery. Even a correctly classified non-transient fault costs the worker forever on one occurrence — a momentary internal error, a brief dependency hiccup, or a transient blip misfiled as non-transient permanently drains capacity, with re-admission left to chance. There is no forgiveness window and no path back for a worker that recovers.

Point fixes — a bigger transport ceiling here, a special-cased status there — address each fault's specific trigger but leave the structural weakness in place, so the next ambiguous or transient fault reproduces the same drain. A health-aware circuit breaker fixes the class: forgive briefly, evict on sustained failure, readmit on recovery, so no single response can drop a live worker.

Expected outcome

A per-worker circuit breaker owned by the health authority (the WorkerProxy / LoadBalancerContext, not the balancer — RoundRobinLoadBalancer stays pure routing), with:

• Strike accounting. Suspect-tier faults increment a per-worker counter; a successful dispatch resets or decays it. The counter is windowed / decaying (sliding window or leaky bucket), not a lifetime tally, so unrelated blips over a long-lived worker never accumulate to eviction and a burst of identical failures from one request-deterministic fault cannot strike out the whole pool. A minimum-request-volume gate prevents ejection on a cold-start unlucky request.
• Ejection is time-based, escalating, and recoverable — never permanent. On crossing the threshold the worker is ejected for a base duration; the duration escalates with prior ejections (base × ejections, jittered, capped), and a max_ejection_percent-style guard prevents ejecting the whole pool. A worker that never recovers stays effectively out (its cooldown grows); a worker that recovers rejoins.
• Half-open recovery. After the cooldown the worker is admitted for a single trial probe with in-flight concurrency limited to one — the concurrency cap, not just spacing, is the thundering-herd defense. Probe succeeds → readmit and reset; probe fails → re-eject and escalate. An optional hard cap (max ejection cycles / cumulative ejected time) drops the worker entirely and relies on re-discovery.
• wool's deliberate INTERNAL stays evict-now. Distinguish wool's own abort(INTERNAL) (service.py:407,549) from grpc's ambiguous INTERNAL by marking it on the server via trailing metadata (e.g., wool-fault: worker-internal), read on the client via error.trailing_metadata(). Marked → non-transient (evict-now); unmarked → suspect (strike). This keeps the classification anchored to a signal wool controls rather than one reverse-engineered from grpc internals.
• Configurable knobs (on WorkerOptions or a dedicated policy object): strike threshold, window, base ejection time, escalation cap, max ejection percent, half-open probe count — with sensible defaults. This is the health-aware forgiveness the RpcError docstring already earmarks as a follow-up.

Prior art

• Circuit breaker — Michael Nygard, Release It! — the closed/open/half-open state machine this generalizes.
• Envoy outlier detection — passive ejection in a client-side load balancer: consecutive_5xx / consecutive_gateway_failure trip conditions, base_ejection_time × num_ejections escalation with automatic un-ejection, max_ejection_percent. The closest reference for wool's exact shape (client-side LB, passive health, eject-and-return).
• AWS "Exponential Backoff and Jitter" — why probe/retry pacing must be jittered so multiple proxies don't synchronize and re-herd a recovering worker.
• Hystrix / resilience4j — failure-rate-over-window trip conditions with a minimum-volume gate, the robust alternative to consecutive-N when partial degradation matters.