Skip to content

gateway: a run of OpenRouter 402s cools its key when OpenRouter leads the walk (D261) #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jaredLunde merged 5 commits into from
Oct 4, 2026
Merged

gateway: a run of OpenRouter 402s cools its key when OpenRouter leads the walk (D261) #122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0357214
gateway: a run of unread OpenRouter 402s cools the key (D261)
jaredLunde Oct 4, 2026
a76e547
verify: D261's fix is 3506e42
jaredLunde Oct 4, 2026
9bddc0b
verify: D261's fix after the rebase onto main
jaredLunde Oct 4, 2026
7eeab94
gateway tests: an OpenRouter 403 is never a 402 strike (D261)
jaredLunde Oct 4, 2026
d830e80
verify: D260 and D261 fix commits on main
jaredLunde Oct 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions crates/gateway/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2375,10 +2375,24 @@ Two deliberate non-cases, plus one same-provider retry:
provider but OpenRouter (`remedy::unfunded_402`), and the walk cools that key as `logging` would
(same metric, same warn line) before moving on. OpenRouter's `402` may be one request larger than
the balance ("This request requires more credits"), which must not cool a shared key (D180, D200),
so an OpenRouter `402` the walk fails over on cools nothing; one it relays (OpenRouter last in the
row, as it is unless the row leads with it or `x-beyond-order` puts it first) is read and cooled
as above. An abandoned `FullBody` attempt never cools from its body in `logging`, so a body that
so one OpenRouter `402` the walk fails over on cools nothing by itself; one it relays (OpenRouter
last in the row, as it is unless the row leads with it or `x-beyond-order` puts it first) is read
and cooled as above. An abandoned `FullBody` attempt never cools from its body in `logging`, so a body that
raced the parent's drop is not counted twice. No work on any other status.
- **A run of unread OpenRouter `402`s cools the key (D261).** Each OpenRouter `402` the walk fails
over on (either abandoning path) is a strike against that pool key
(`Provider::strike_unread_402`, a relaxed `AtomicU32` beside the key's cooldown); the
`OPENROUTER_402_STRIKES`th (3) in a row cools it through the same path as D258 (same metric
reason `unfunded`, same warn line). Any 2xx from the key resets its count
(`Provider::key_served`: one relaxed load on the 2xx path, and a store only when the count is
non-zero), and any cooling spends it. A key already cooling does not count the `402`s of requests
sent before it cooled, so a burst cools it once. A too-costly `402` is that request's, so a funded
account's next ordinary request resets the count; a near-empty account sent only large requests
may cool for `KEY_COOLDOWN`, which is the accepted cost. A relayed OpenRouter `402` is read
instead: out of credit cools (and so resets), too costly neither counts nor resets. Checking the
balance out of band was rejected: `GET /api/v1/key` reports only a per-key limit (null for a key
drawing on account credit) and `GET /api/v1/credits` needs a management key. Per replica, in
memory, nothing stored.
- **When every candidate 5xxes, the client gets the last provider's own status**, not a synthetic
error. Better diagnostics than an exhausted retry loop produces.

Expand Down Expand Up @@ -2902,7 +2916,7 @@ already gone gets nothing, and a response that has started cannot be replaced.
| Client stalls mid-upload (stops sending, connection open) | Ends at the first body read timeout: pingora's HTTP/1.1 one (60s), or `read_timeout_secs` on the upstream (the only clock for h2c). Either way a JSON 408 (`request body timed out`), the client's own timeout: no failover, no breaker failure, no key cooled, nothing billed (D260). | Client retry, which is a fresh request. `tests/cancellation.rs` pins it over HTTP/1.1, h2c and chunked. |
| Provider brownout (sustained 5xx) | After `circuit_breaker_threshold` 5xx/connect failures in the window that are also at least half its outcomes (so a 50% brownout counts), the breaker opens; requests fast-fail 503 (`circuit_open`) instead of stalling against the read timeout. | Auto: after `circuit_breaker_reset_secs` a half-open probe is admitted — success closes the breaker, failure reopens it. Per-provider, so other providers are unaffected. |
| Provider throttles (429 storm) | Walk the next unused pool key on the same provider when the body is replayable; the last 429 is relayed with `Retry-After` if the upstream sent one. Does **not** trip the breaker (provider is healthy). Does **not** fail over to another vendor. | Client `Retry-After` backoff after keys are exhausted; no gateway-side circuit action. |
| Pool key out of credit (a credit-balance 400, `insufficient_quota`) | Relayed with a neutral message (the head is relayed before the body says why), the key cooled off for 60s (`ai_key_auth_failures_total{reason="unfunded"}`); a catalog walk leaves the provider out while all its keys cool, so later requests go to the row's next candidate (D180). A `402` a catalog walk fails over on is served by the next candidate and cools the key at the head, except OpenRouter's (D258). Not a breaker failure. | Fund the account or replace the key; the warn line `pool key is out of credit` names provider and key index. |
| Pool key out of credit (a credit-balance 400, `insufficient_quota`) | Relayed with a neutral message (the head is relayed before the body says why), the key cooled off for 60s (`ai_key_auth_failures_total{reason="unfunded"}`); a catalog walk leaves the provider out while all its keys cool, so later requests go to the row's next candidate (D180). A `402` a catalog walk fails over on is served by the next candidate and cools the key at the head (D258); OpenRouter's cools it on the third in a row with no 2xx between (D261). Not a breaker failure. | Fund the account or replace the key; the warn line `pool key is out of credit` names provider and key index. |
| Pool key revoked (401) | Walk the next pool key on the same provider when the body is replayable, and cool the refused key off for 60s so later requests start on a good one (`ai_key_auth_failures_total{reason="revoked"}`; a 403 naming the key counts as `key_named_403`). The last key's 401 is relayed on a provider route, a candidate failure on a catalog walk. A 403 never walks or cools (a catalog walk fails over on it), except that one whose body names the key cools it. Not a breaker failure. | Replace the key in config; the metric says which provider (log line names the key index). |
| Response body > 128KB before usage chunk | Tail compaction fires: `drain(..half)` discards first half, keeps tail. Usage extracted from retained tail. | No action — SSE usage is always in the final `data:` line, which always lands in the tail. |
| Client cancels mid-request (ESC on a slow turn) | Relayed as a downstream abort. **Not** counted against the provider's breaker — pingora tags it `ErrorSource::Downstream`. A streaming 2xx cut short this way is billed an estimate (`usage_estimated=true`), not zero. The tenant slot is released. | None. `tests/cancellation.rs` pins the breaker halves; `tests/cut_short.rs` the billing. |
Expand Down
43 changes: 30 additions & 13 deletions crates/gateway/src/proxy.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1770,6 +1770,23 @@ impl AiProxy {
provider.mark_key_bad(key);
}

/// Judge a `402` a catalog walk abandons at its head, unread (`by_status`: `None` for any other
/// answer, else [`remedy::unfunded_402`]'s verdict). Out of credit by its status alone: cool
/// the key (D258). OpenRouter's, which may be one request too costly instead: a strike against
/// the key, and the [`route::OPENROUTER_402_STRIKES`]th in a row with no 2xx between cools it
/// (D261). A relayed 402 is read by `logging` instead: out of credit cools the key (which spends
/// its strikes), too costly neither counts nor resets one.
fn abandon_402(&self, rc: &RequestCtx, by_status: Option<bool>, status: u16) {
let cool = match by_status {
Some(true) => true,
Some(false) => rc.provider.strike_unread_402(rc.pool_key),
None => false,
};
if cool {
self.cool_unfunded_key(&rc.request_id, &rc.provider, rc.pool_key, status);
}
}

/// Resolve a 2xx's pending health verdict from the first response bytes: count an
/// error-in-200 against the candidate's breaker.
fn settle_health(&self, rc: &mut RequestCtx, chunk: &[u8], end_of_stream: bool) {
Expand Down Expand Up @@ -5010,6 +5027,10 @@ impl ProxyHttp for AiProxy {
return Ok(());
};
let status = upstream_response.status.as_u16();
// A 2xx says this key's account pays: forget its unread OpenRouter 402s (D261).
if rc.managed && (200..300).contains(&status) {
rc.provider.key_served(rc.pool_key);
}

// Managed 401: this pool key is revoked (D71). Never the caller's fault. Cool it off so
// later requests start on a good key, then walk like a 429. Not a 403 (D84, see
Expand Down Expand Up @@ -5101,14 +5122,12 @@ impl ProxyHttp for AiProxy {
// tells `logging` an account is out of credit (D180), and every later request would pay a
// round trip to it first. A 402 says so by its status alone except on OpenRouter, whose
// 402 may be one request too large for the balance (`remedy::unfunded_402`): cool that key
// here, as `logging` would have, on either abandoning path below (D258).
let unfunded = key_failure
&& status == 402
&& rc
.auto
.as_ref()
.and_then(|a| a.candidate_at(at))
.is_some_and(|c| remedy::unfunded_402(c.provider));
// here, as `logging` would have, on either abandoning path below (D258). OpenRouter's
// unread 402 is a strike against its key instead, and enough in a row cool it (D261).
let unfunded_402 = (key_failure && status == 402)
.then(|| rc.auto.as_ref().and_then(|a| a.candidate_at(at)))
.flatten()
.map(|c| remedy::unfunded_402(c.provider));
// Fail over only when the body is **provably** replayable: fully read, and small enough
// that pingora buffered all of it.
//
Expand Down Expand Up @@ -5143,8 +5162,8 @@ impl ProxyHttp for AiProxy {
);
rc.relay_abandoned = fb.record(RelayRetry::Candidate(orig));
// Not recorded (the final attempt): this answer is relayed and `logging` reads it.
if unfunded && rc.relay_abandoned {
self.cool_unfunded_key(&rc.request_id, &rc.provider, rc.pool_key, status);
if rc.relay_abandoned {
self.abandon_402(rc, unfunded_402, status);
}
return Ok(());
}
Expand Down Expand Up @@ -5178,9 +5197,7 @@ impl ProxyHttp for AiProxy {
{
b.record_success_for(permit);
}
if unfunded {
self.cool_unfunded_key(&rc.request_id, &rc.provider, rc.pool_key, status);
}
self.abandon_402(rc, unfunded_402, status);
rc.advance_candidate(at);
let mut e = pingora_core::Error::new(pingora_core::ErrorType::HTTPStatus(status));
e.set_retry(true);
Expand Down
Loading
Loading