feat(proxy): optional SSE keepalive comments during streaming silence

[henricook]

Relevant issues

Addresses #14953 (closed not-planned) — reopening the ask with a working implementation, tests, and e2e proof.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Real call through the proxy running at commit e9f4177: Azure gpt-5.6-sol, reasoning.effort=high, LITELLM_SSE_KEEPALIVE_INTERVAL_SECONDS=3. The model reasons silently for ~30s and the keepalives land exactly in that gap:

$ curl -sS -N http://127.0.0.1:20488/v1/responses -H "Authorization: Bearer sk-..." \
    -d '{"model":"gpt-5.6-sol","input":"Think very carefully: how many primes below 500 have a digit sum of exactly 10?","reasoning":{"effort":"high"},"stream":true,"max_output_tokens":4000}'

data: {"type":"response.created", ...}
data: {"type":"response.in_progress", ...}
: keepalive
: keepalive
: keepalive        <- 10 keepalives over ~30s of reasoning silence
...
data: {"type":"response.output_item.added", ...}
data: {"type":"response.output_text.done", ... correct answer ...}
data: {"type":"response.completed", ...}

Without this, the same silence exceeds e.g. Envoy's 300s default stream_idle_timeout on longer thinks and the connection is killed mid-turn. We run this in production (LiteLLM gateway fronting Azure GPT-5.6 for Codex CLI users behind a Cilium/Envoy ingress) — the stream disconnected before completion retry loops disappear.

Tests: pytest tests/test_litellm/proxy/test_common_request_processing.py — 169 passed (5 new in TestSSEKeepalive).

Type

🆕 New Feature

Changes

• Why: reasoning models stream nothing while thinking (raw CoT is never sent), so /v1/responses connections sit wire-silent for minutes; any idle-timeout middlebox between client and proxy (nginx 60s default, Envoy/Cilium 300s, cloud LBs) kills them mid-turn. Users often can't raise those timeouts (shared infra) — same failure class as zai/glm-4.7 via Gateway: 30s idle timeout on api.z.ai drops tool_call streaming vercel/ai#12949 and Codex CLI requests consistently cut off at ~150s (stream disconnected); please increase or make timeout configurable openai/codex#3478.
• Opt-in via LITELLM_SSE_KEEPALIVE_INTERVAL_SECONDS (default 0 = disabled, zero behaviour change). When set, SSE streaming responses emit a comment line (: keepalive) whenever the upstream generator is silent past the interval.
• SSE comments are spec-mandated ignorable (https://html.spec.whatwg.org/multipage/server-sent-events.html); verified tolerated by OpenAI Codex CLI 0.144 before, between, and after events.
• Implemented in create_response() after first-chunk buffering, so first-chunk error sniffing / error-to-JSON status mapping is untouched; disabled path is byte-identical (covered by test).

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 86.20690% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/common_request_processing.py	86.20%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[henricook]

Companion docs PR: BerriAI/litellm-docs#531 (the documentation / code-quality checks validate env keys against that repo's config_settings.md, so they stay red here until it lands). lint fixed in 1651bbf.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing henricook:feat/sse-keepalive-heartbeat (1651bbf) with litellm_internal_staging (bf02a4a)}

[henricook]

Discovered that I could force codex to request summaries with model_reasoning_summary = "auto" - and that Azure supports it.