Implement Venice client circuit breaker with half-open state and automatic recovery

[devJaja]

Summary

The Venice client has a circuit breaker, but it needs full three-state implementation (closed → open → half-open → closed) with automatic recovery, configurable thresholds, and metrics. Currently a Venice outage may cause immediate failures rather than gracefully shedding load.

Priority

P1 — A single Venice outage currently degrades the whole network.

Files to work on

backend/src/services/venice/circuitBreaker.ts
backend/src/services/venice/client.ts
backend/src/services/venice/circuitBreaker.test.ts

Requirements

• Implement full circuit breaker state machine:
  • CLOSED: normal operation; failure count resets on success
  • OPEN: reject all calls immediately after failureThreshold consecutive failures; start cooldown timer
  • HALF_OPEN: after cooldown, allow a limited probe of probeCount requests; success closes, failure reopens
• Configurable: failureThreshold (default 3), cooldownMs (default 60_000), probeCount (default 1)
• Expose metrics: { state, failures, successes, lastFailureAt, lastSuccessAt }
• Emit events on state transitions (opened, closed, half_opened)
• Add unit tests for all state transitions including timing with mocked timers

Acceptance Criteria

• After failureThreshold consecutive failures, circuit opens (calls rejected immediately)
• Rejected calls throw CircuitOpenError (not a Venice 5xx)
• After cooldownMs, circuit enters HALF_OPEN
• In HALF_OPEN, a single successful probe closes the circuit (closed state)
• In HALF_OPEN, a single failed probe reopens the circuit
• Successful calls in CLOSED reset the failure count
• Circuit state is observable via getCircuitState()

[Iker2522]

Hi, I’d like to take this task. I can implement the three-state circuit breaker for the Venice client, including automatic recovery, configurable thresholds, metrics, state transitions, and the required unit tests.