Stream upstream responses through the proxy (fixes #9 — long-request idle-timeout aborts)

[MattMencel]

Fixes #9. Thanks for confirming the diagnosis, @mr-beaver.

What this does

Makes proxy_anthropic and proxy_openai_compat stream the upstream response to the client incrementally instead of buffering the whole body, so long requests no longer trip Claude Code's idle/per-read timeout. The full body is still captured for usage accounting, optimizer savings, cache-state tracking, and dedup.

Approach

• A shared, provider-agnostic helper stream_upstream(...) opens the upstream with client.send(stream=True) and returns a StreamingResponse that tees each chunk to the client while accumulating the full body. After the stream ends — cleanly, on client disconnect, on upstream mid-stream error, or on connect-time failure — it calls a handler-specific finalize(...) closure exactly once for accounting + dedup.
• Request side is untouched (docs/adr/0001 boundary preserved); parsers _parse_anthropic/_parse_openai already work on the full buffer, so accounting is identical to the old buffered parse.
• Per-read timeout=120 kept; no total-duration cap (a code comment guards against re-tidying it).
• Dedup cache now round-trips the response content-type (cached SSE replays as text/event-stream, not hardcoded JSON).
• Partial/aborted streams record stop_reason="incomplete" (no schema change) instead of masquerading as clean 200s; connect failures record 502 + a [stream] log line with the cause.

Verification

• 345 tests pass (13 new: incremental delivery via body_iterator, SSE usage accounting, dedup content-type round-trip, client-disconnect partials, connect failure, JSON-through-stream, OpenAI-compat streaming).
• Live for ~24h on a fork deployment: the 300–600s request class that previously aborted now completes cleanly — e.g. a 607s request finishing 200 / tool_use / ~37k output tokens. Zero streaming-path failures in the logs over ~3,000 requests/day.

One open question for you

The implementation uses uniform streaming — one code path for SSE and JSON, no content-type branching. Consequence: non-streaming responses (count_tokens, model lists, stream:false, errors) also move from a buffered Content-Length response to chunked. The alternative is to branch on the response head and keep non-streaming responses buffered (two code paths). I went uniform for simplicity and to avoid dual-path drift; both are reversible. Happy to switch to the branch variant if you'd prefer — your call.

🤖 Generated with Claude Code