gateway: a client that stalls mid-upload times out its own request, no failover (D260)

crates/gateway/ARCHITECTURE.md

@@ -1761,7 +1761,7 @@ Two more endings the provider bills and we used to write as zero get the same tr
   closes before its body's framing-defined end (bytes of a `Content-Length` still to come, a chunked
   body's last chunk missing) did not get it there, though pingora marks such a body done
   (`client_body_cut_short`, D259). A request that never reached a provider (every breaker open, a
-  connect failure, a reset mid-upload, the client closing mid-upload) stays
+  connect failure, a reset mid-upload, the client closing or stalling mid-upload) stays
   unbilled, and so does one the upstream **closed or reset** before any head (the gateway's 502): the
   peer declined to answer, and nothing says it processed the request. The real edges in front of
   Anthropic, OpenAI and OpenRouter answer a request they forwarded and then lost with an HTTP error
@@ -2125,7 +2125,10 @@ the rate guardrails (which protect against abusive _inbound_ load):
   provider was waiting on the client: a stalled or abandoned upload. Neither counts. Before this, any
   caller, BYO included, could open a provider's breaker for every tenant with a few oversized or
   stalled uploads. A connect failure feeds no body byte, so it still counts. The cost: a provider
-  that resets the connection mid-upload is not blamed for it either; one that answers 5xx is.
+  that resets the connection mid-upload is not blamed for it either; one that answers 5xx is. A
+  stall's read timeout is retagged as the client's in `error_while_proxy` (D260, below), so a
+  catalog walk does not fail over either, which would have charged the candidate it left from
+  `upstream_peer`.
 - **Nor is either of those a success.** An attempt that ends with no provider outcome (no response
   head, and the error is the client's: an abort, a stalled or abandoned upload) gives its permit back
   without one (`CircuitBreaker::release`). Recording a success there closed a half-open breaker on a
@@ -2430,6 +2433,24 @@ the one rule, and a small body is read straight through to the upstream (nothing
 connecting), so "the client's body was read" tracks what went out. A timeout is never resent: it is
 a liveness verdict on a peer that may have the request.
 
+**A client that stalls mid-upload times out its own request.** A client that stops sending its
+body with the connection still open leaves a provider waiting for bytes that never come, until the
+upstream read timeout fires. That is the client's stall, not a provider failure, so it must not
+fail over (the next candidate would wait on the same missing bytes), charge a breaker, or bill an
+estimate. Pingora's upstream half waits on the response and the next body chunk at once and
+re-arms the read timeout on each chunk, and a body write blocked on an upstream that stopped
+reading is a `WriteTimedout` instead, so a `ReadTimedout` while the client still owes body bytes
+(`client_stalled_upload`: some fed, `is_body_done` false) means nothing moved either way for
+`read_timeout_secs` with the provider waiting on the client. `error_while_proxy` retags it as a
+downstream error (D260): the walk ends with no failover, `logging` gives the breaker permit back,
+`body_delivered` is false so nothing is billed (`outcome: client_cancelled`), and `fail_to_proxy`
+answers 408 (`request body timed out`), the same answer as pingora's own HTTP/1.1 body read
+timeout (60s, which wins over the default 600s `read_timeout_secs` on HTTP/1.1). Pingora's HTTP/2
+server has no body read timeout, so for an h2c client the upstream read timeout is the only clock.
+An HTTP/2 upstream connection is still marked for shutdown on that timeout (pingora does so before
+the hook runs). A chunked body on a catalog walk is read whole before connecting, so its stall is
+pingora's downstream 408 and never reaches a provider.
+
 **A pooled connection reset with the request unread is not delivered.** A **reused** HTTP/1.1
 connection reset before any response byte means the provider's kernel answered with RST because the
 socket was closed with our request unread in it: the idle-close race on a pooled connection, which
@@ -2878,6 +2899,7 @@ already gone gets nothing, and a response that has started cannot be replaced.
 | Provider TCP connect fails                                                                 | `fail_to_connect` retries up to 2×, then returns a JSON 502. Counts as a circuit-breaker failure.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | Client SDK retries with backoff. No HTTP-status retries (Pingora-idiomatic).                                                                                                                                                                                                                                             |
 | Provider connection dies (host gone, partition, wedged edge)                               | Upstream HTTP/2 PING goes unacknowledged → the connection and its streams fail within `h2_ping_interval_secs` + 5s (≤ 20s); on HTTP/1.1, TCP keepalive drops the peer within 30s (and `TCP_USER_TIMEOUT` fails unacknowledged request bytes as fast). Before the head: a JSON 502; mid-stream: the client gets an error, never a clean end, and the row is `cut_short`.                                                                                                                                                                                                                                                                                                                                            | Client retry.                                                                                                                                                                                                                                                                                                            |
 | Provider alive but silent (no head yet, or a quiet gap: a model thinking without emitting) | Never cut before `read_timeout_secs` (600s), the clients' own default timeout: the gateway cannot tell thinking from stuck, so it does not guess. Then a JSON 504 (`upstream timed out after receiving the request`), not resent. No total deadline: a stream that keeps sending lives until it ends. A row whose vendor documents requests taking "several minutes" and publishes no maximum (OpenAI's `-pro` models, `catalog::long_running`) has no silence deadline at all (D252): it ends at the client's timeout (a cancel, billed per D130) or a dead transport.                                                                                                                                            | Client timeout or retry.                                                                                                                                                                                                                                                                                                 |
+| Client stalls mid-upload (stops sending, connection open)                                  | Ends at the first body read timeout: pingora's HTTP/1.1 one (60s), or `read_timeout_secs` on the upstream (the only clock for h2c). Either way a JSON 408 (`request body timed out`), the client's own timeout: no failover, no breaker failure, no key cooled, nothing billed (D260).                                                                                                                                                                                                                                                                                                                                                                                                                             | Client retry, which is a fresh request. `tests/cancellation.rs` pins it over HTTP/1.1, h2c and chunked.                                                                                                                                                                                                                  |
 | Provider brownout (sustained 5xx)                                                          | After `circuit_breaker_threshold` 5xx/connect failures in the window that are also at least half its outcomes (so a 50% brownout counts), the breaker opens; requests fast-fail 503 (`circuit_open`) instead of stalling against the read timeout.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | Auto: after `circuit_breaker_reset_secs` a half-open probe is admitted — success closes the breaker, failure reopens it. Per-provider, so other providers are unaffected.                                                                                                                                                |
 | Provider throttles (429 storm)                                                             | Walk the next unused pool key on the same provider when the body is replayable; the last 429 is relayed with `Retry-After` if the upstream sent one. Does **not** trip the breaker (provider is healthy). Does **not** fail over to another vendor.                                                                                                                                                                                                                                                                                                                                                                                                                                                                | Client `Retry-After` backoff after keys are exhausted; no gateway-side circuit action.                                                                                                                                                                                                                                   |
 | Pool key out of credit (a credit-balance 400, `insufficient_quota`)                        | Relayed with a neutral message (the head is relayed before the body says why), the key cooled off for 60s (`ai_key_auth_failures_total{reason="unfunded"}`); a catalog walk leaves the provider out while all its keys cool, so later requests go to the row's next candidate (D180). A `402` a catalog walk fails over on is served by the next candidate and cools the key at the head, except OpenRouter's (D258). Not a breaker failure.                                                                                                                                                                                                                                                                       | Fund the account or replace the key; the warn line `pool key is out of credit` names provider and key index.                                                                                                                                                                                                             |

crates/gateway/src/proxy.rs

@@ -2257,6 +2257,18 @@ fn client_still_uploading(session: &mut Session, rc: &RequestCtx) -> bool {
     rc.body_bytes_fed > 0 && !session.as_mut().is_body_done()
 }
 
+/// Whether the upstream's read timeout fired while the client still owed body bytes (D260): the
+/// client stalled mid-upload with its connection open. Pingora's upstream half waits on the
+/// response and on the next body chunk at once, re-arming the read timeout on each chunk, and a
+/// body write blocked on an upstream that stopped reading is a `WriteTimedout` instead; so this
+/// timeout means neither side moved a byte for `read_timeout_secs`, with the provider waiting on
+/// the client. Zero bytes fed is excluded as [`client_still_uploading`] excludes it: a client
+/// that sent `Expect: 100-continue` waits on the provider's answer before its first byte. Any
+/// source: pingora's own downstream body read timeout (HTTP/1.1, 60 s) is already the client's.
+fn client_stalled_upload(session: &mut Session, rc: &RequestCtx, e: &pingora_core::Error) -> bool {
+    e.etype() == &pingora_core::ErrorType::ReadTimedout && client_still_uploading(session, rc)
+}
+
 /// The lowest set bit in `usable` at or after index `from`, or `None` if there is none.
 ///
 /// The candidate walk's only cursor primitive. `from` strictly increases across a request, so the
@@ -6068,6 +6080,17 @@ impl ProxyHttp for AiProxy {
         if matches!(e.etype(), T::HTTPStatus(_) | T::CustomCode(..)) {
             return e;
         }
+        // A client that stalled mid-upload timed its own request out (D260): retag the timeout as
+        // the client's, so the walk below ends here rather than failing over with a body nobody
+        // will finish, `logging` charges no breaker and bills no estimate, and `fail_to_proxy`
+        // answers 408 rather than blaming the provider with a 504.
+        if ctx
+            .rc
+            .as_ref()
+            .is_some_and(|rc| client_stalled_upload(session, rc, &e))
+        {
+            e.as_down();
+        }
         // One rule for every body size (D09, D51): a connection failure is retried only when the
         // upstream cannot have the whole request (`body_delivered` is false: the connection never
         // came up, the body was not read to its end, or writing it failed), and never once the