feat(auth): add health-aware cooldown and CPA-style failover

[zendext]

Requirement

Replace stateless global round-robin failure handling with health-aware Codex auth selection, account-wide cooldowns, sticky-session failover, deterministic final errors, and the three-layer retry behavior proven by CLIProxyAPI (CPA).

Decisions

Health state

• Auth health and cooldown apply globally to the auth across all sessions.
• Request-scoped 4xx errors do not penalize an auth.
• 401 first uses the same-auth refresh-and-retry behavior from fix(auth): make OAuth refresh concurrent-safe and recoverable #4; continued unauthorized responses make that auth unavailable for selection.
• 429 honors Retry-After or an explicit quota recovery time.
• Network failures, 408, and retryable 5xx use short in-memory cooldowns.
• Persist only authoritative or longer-lived states in SQLite: explicit 429 recovery deadlines, invalid_grant, and continued unauthorized state after refresh.
• Clear credential-related persisted unavailability when auth material changes or a refresh/request proves the auth healthy; preserve unexpired quota/429 cooldowns across same-account token updates.
• Model-not-supported errors are model capability exclusions, not auth-global cooldowns.
• Session-bound requests fail over through feat(auth): persist tenant-scoped Codex session affinity #5 when their bound auth is disabled, cooling, unavailable, or unable to serve the requested model.
• If every auth is blocked, do not force-select one.
• Health, cooldown, affinity, and retry behavior are enabled by default with no legacy feature flags.

Retry layers

1. Same-auth repair: after 401, refresh once and retry the same auth once. This is not a credential-switch budget.
2. Credential failover: one execution round tries distinct currently eligible auths, limited by max-retry-credentials; 0 means all eligible auths.
3. Cooldown retry: after a round exhausts eligible auths, wait for the nearest cooldown only when it does not exceed max-retry-interval, then start another round up to request-retry times.

Use these defaults:

request-retry: 3
max-retry-credentials: 0
max-retry-interval: 30

• Waiting and retries stop immediately when the client context is canceled.
• Do not retry after any downstream response bytes have been committed.
• Do not retry after a WebSocket upgrade succeeds.
• Do not repeatedly select the same auth for the same failure within one round.

Replay eligibility

• Apply cross-auth retries only to an explicit replayable allowlist: /v1/chat/completions, HTTP Responses, Responses compact, alpha search, JSON image generation, trace summarization, read-only GET/HEAD, and WebSocket handshake before successful upgrade.
• Do not retry multipart image edits, file upload/finalization, realtime calls/connections, side-effecting wham actions, hosted MCP tool calls, unknown non-read-only routes, or bodies that cannot be recreated safely.
• Buffer at most 32 MiB for retry. A known body over the limit or an unknown/chunked body is still proxied once but is not eligible for automatic retry; do not spool replay bodies to disk and do not reject the request merely because replay is unavailable.
• For allowlisted buffered requests, transient network errors may retry without distinguishing whether the request body had already been fully written. This intentionally favors availability and accepts a rare duplicate generation/billing risk.
• Non-replayable requests still receive proactive token refresh and update auth health from their one upstream result.

Final error selection

Use deterministic final errors rather than whichever auth failed last:

1. Request-scoped invalid input/context errors stop immediately.
2. All known auths lacking model support returns 404 model_not_found.
3. All eligible auths quota/rate limited returns 429 with the earliest known recovery time in Retry-After.
4. All auths disabled, credential-invalid, or continued unauthorized returns 503 auth_unavailable.
5. Exhausted network/retryable upstream failures returns 502 upstream_unavailable.
6. Mixed quota and transient failures returns 429 when there is a clear near-term recovery deadline; otherwise 502.
7. Client cancellation does not synthesize a new proxy error.

Only safe aggregate errors are returned to clients. Per-auth details remain in safe logs and management status.

Acceptance Criteria

• Selection skips auths in active global cooldown and preserves healthy session affinity.
• 401, 429, transient transport errors, retryable 5xx, model capability errors, and request-scoped 4xx produce the documented distinct state transitions.
• Explicit upstream recovery deadlines survive proxy restarts; short transient cooldowns do not.
• Restarting the proxy cannot bypass an unexpired persisted Retry-After deadline.
• request-retry, max-retry-credentials, and max-retry-interval follow the three independent CPA-style meanings above.
• Every allowed and denied replay route has regression coverage, including the 32 MiB boundary and unknown-length bodies.
• Network retry tests document the accepted duplicate-execution tradeoff for eligible buffered requests.
• Streaming and WebSocket tests prove no replay occurs after client-visible commitment or successful upgrade.
• Final client status, safe error code, and Retry-After are deterministic under mixed multi-auth failures and independent of auth ordering.
• Logs identify account IDs, health transitions, retry rounds, and safe cooldown reasons without exposing tokens or raw sensitive bodies.
• English and Chinese configuration and architecture documentation remain synchronized.

Non-goals

• CPA's generic multi-provider scheduler, plugin system, or management UI.
• Distributed cooldown coordination between multiple COP processes.
• Persistent storage for second-scale transport cooldowns.
• Retrying request-invalid errors.
• Disk spooling for retry bodies.
• httptrace-based detection of ambiguous post-write transport failures.
• Legacy behavior flags.

Dependencies

• Depends on fix(auth): establish stable Codex credential identity #3 for stable identity and hot-reload reconciliation.
• Depends on fix(auth): make OAuth refresh concurrent-safe and recoverable #4 for concurrent-safe proactive and reactive OAuth refresh.
• Depends on feat(auth): persist tenant-scoped Codex session affinity #5 for persistent tenant-scoped session affinity and CAS failover.
• Depends on fix(storage): fail fast on SQLite infrastructure errors #10 for process-level SQLite failure behavior.

Verification

• gofmt -w .
• go test ./...
• go build -o test-output ./cmd/server && rm test-output