perf(relay): reconcile early SSE header flush with pre-output failover

[LIghtJUNction]

Upstream source

• QuantumNous/new-api PR: fix(stream): 优化ttft，在首个数据帧前flush QuantumNous/new-api#7509
• Upstream head: c58cfc1b4577a9fc129b0fc6e5dffda82038cf8e
• Upstream base: 9c293e8c02371bda844af79e3500ff2d516d1dda

Upstream #7509 adds CommitEventStreamHeaders: after an upstream response has been accepted as a stream, it writes and flushes the downstream SSE headers before scanning the first data frame. The goal is to avoid making clients wait for the first model token before receiving HTTP 200 / text/event-stream. It also treats header-flush failures as writer errors and updates stream handlers that bypass the shared scanner.

Current LMM behavior

Current main (997a1b673dcf718d75cbd957c437828a86c1a319) has the same basic transport gap:

• apps/api-go/relay/helper/common.go has SetEventStreamHeaders, but it only sets headers and does not flush them.
• apps/api-go/relay/helper/stream_scanner.go calls copyCodexSSEHeaders then SetEventStreamHeaders before reading the stream, so clients may not observe the HTTP/SSE response until the first body write.

However, LMM deliberately diverged from upstream with merged PR #312 (480cfc329c881a7670148b5aa335fa60f0d0babf): OpenAI-compatible streams can use OPENAI_FIRST_OUTPUT_TIMEOUT to fail over a channel that returns upstream headers but never produces visible output.

Current OaiStreamHandler:

1. starts FirstResponseTimeout after upstream headers;
2. only marks the first response after visible content/reasoning/tool/function output is successfully handled;
3. on timeout before visible output, marks ContextKeyUpstreamChannelFailure and returns 504 upstream_timeout, allowing the existing channel retry/failover path;
4. avoids retry after visible output.

StreamScannerHandler also intentionally prevents locally generated heartbeat writes before the first business response.

Why not copy #7509 directly

Flushing the downstream writer normally commits the HTTP response. If LMM commits 200 text/event-stream immediately after upstream headers, the later first-visible-output timeout can no longer reliably return a real 504 before output, and the existing pre-output failover boundary may be weakened or become dependent on already-committed downstream state.

That is a product/transport-policy conflict, not a mechanical backport. The upstream latency improvement is useful, but LMM should not trade away its stalled-stream failover silently.

Suggested LMM adaptation

Treat downstream header commitment as part of the retry boundary rather than applying CommitEventStreamHeaders unconditionally:

1. Add a small explicit helper for committing/flushing SSE headers, but call it only when the current attempt is no longer eligible for pre-output failover.
2. For OpenAI-compatible streams with FirstResponseTimeout > 0, keep the response uncommitted while the attempt is still in the pre-visible-output retry window. Commit immediately before the first downstream-visible event, then permanently disable retry for that attempt.
3. When the first-output timeout is disabled, allow early header flush after the upstream response has been validated as a stream, matching the latency benefit of upstream #7509.
4. Audit dedicated streaming handlers that do not use StreamScannerHandler; only adopt early commitment where there is no equivalent retry/failover contract to preserve.
5. If committing/flushing fails, record a bounded writer/client transport failure and stop the stream. Do not classify a downstream writer failure as an upstream channel failure and do not retry it against another provider.
6. Preserve existing copyCodexSSEHeaders, Cache-Control: no-cache, no-transform, X-Accel-Buffering: no, write deadlines, bounded queue/backpressure and heartbeat ordering.
7. Keep this transport-only. No pricing, group, model-price-lock, /fast, OAuth, routing-weight, quota, or administrator-assistant behavior may change.

Acceptance criteria

• With OPENAI_FIRST_OUTPUT_TIMEOUT=0, a validated SSE upstream can flush HTTP 200 + SSE headers before the first model frame, and a regression test observes the flush before any data callback.
• With first-output timeout enabled, no downstream response is committed while the attempt is still eligible for failover; headers-only / role-only / usage-only upstream activity can still end in the existing retryable 504 upstream_timeout behavior.
• The first successfully forwarded visible content/reasoning/tool/function event commits the stream and retires the pre-output retry boundary.
• No request is retried after downstream headers/body have been committed in a way that could duplicate user-visible output.
• A downstream flush/write error is terminal for that client request and is not counted as an upstream provider/channel failure.
• Dedicated stream handlers are covered by an explicit audit: each either uses the safe shared policy or documents why immediate header commit is safe.
• Tests cover: timeout disabled early flush, timeout enabled headers-then-stall, role-only-then-stall, visible output before timeout, flush failure, and existing heartbeat ordering.
• Relevant Go relay/helper/channel tests and repository CI/release qualification pass.

Related LMM work

• feat(go): fail over stalled streams before first output #312 — merged Go first-visible-output timeout / pre-output failover
• [Rust][Parity] Add first-visible-output timeout and pre-output failover for OpenAI streams #313 — Rust parity for the same first-visible-output contract
• [Bug] Responses SSE 在 response.completed 前断流并出现 response body 解码错误 #271 — broader Responses/SSE interruption handling

This issue is intentionally separate from those: it tracks the new upstream TTFT/header-commit optimization and the exact boundary needed to adopt it without regressing LMM's failover semantics.

[LIghtJUNction]

上游 #7509 在 2026-09-22 05:59 UTC 更新了 CodeRabbit 审查结论；代码 head 仍是 c58cfc1b，没有新 commit，但新增了一个需要纳入 LMM 实现边界的资源清理风险：如果客户端侧 SSE header flush/commit 失败，部分直连 streaming handler 可能在返回错误后仍保留上游工作或 HTTP body/connection。

这不会改变本 issue 关于“不能无条件提前 commit，必须保留 OPENAI_FIRST_OUTPUT_TIMEOUT 预输出 failover”的判断，但实现时应额外要求：

• CommitEventStreamHeaders/首次可见输出前的 flush 一旦失败，立即终止该下游请求，并取消/停止对应上游读取；
• 对持有 resp.Body 的路径确保执行已有的 graceful close/cleanup，不留下仍在生成的 upstream request 或连接；
• writer/flush failure 继续分类为 downstream transport failure，不标记为 upstream channel failure，也不得触发 provider retry；
• 增加回归测试：模拟 Flush 返回错误，断言 scanner/handler 不继续读取后续 frame、body 被关闭/取消、不会重试其他渠道。

Upstream review: QuantumNous/new-api#7509 (comment)

当前 main@f7a3c7a4 仍未出现 CommitEventStreamHeaders，因此 #434 继续有效；本轮不创建重复 issue/PR。

[LIghtJUNction]

上游 #7509 刚新增 commit 43ebcb9c319891ea84657651cf7f23648c8ba7b7（2026-09-22 07:27 UTC），直接处理了上一次审查指出的资源清理问题：

• 专用 streaming handler 统一把 resp.Body 关闭放到 defer，确保 header commit/flush 失败和其他早退路径都能清理上游响应；
• Cohere / PaLM / Zhipu 等先成功 CommitEventStreamHeaders，再启动读取/生产 goroutine，避免客户端 writer 已失败时上游仍继续工作；
• 新增 stream_commit_cleanup_test.go，模拟 request context 已取消导致 commit 失败，断言 upstream body 被关闭且 producer 在 commit 前没有发生任何读取；
• shared StreamScannerHandler 仍把 flush 失败标记为 writer_error，不当作 provider/channel 故障。

Upstream commit: QuantumNous/new-api@43ebcb9
Upstream PR: QuantumNous/new-api#7509

这意味着本 issue 上一条评论新增的 cleanup 验收条件，已经有了可直接参考的上游实现和测试模式。但 LMM 仍不能直接 cherry-pick #7509：当前 main@83ea7cde 的 StreamScannerHandler 仍保留 FirstResponseTimeout / first-visible-output failover 机制，而上游仍是在扫描首帧前无条件 commit 200 text/event-stream。直接照搬会让 OPENAI_FIRST_OUTPUT_TIMEOUT 的 headers-only / role-only stall 无法再稳定返回真实 504 并进行预输出 failover。

因此本轮不另开重复 PR。后续实现继续按 #434 的 LMM-native 边界：FirstResponseTimeout == 0 时允许早 flush；仍处于可 failover 窗口时保持未 commit；首次成功转发可见输出前/时再提交；所有 flush/write failure 都作为 downstream terminal failure 且必须关闭 upstream body、禁止 provider retry。

上游新 head 的 CI 当前仍在运行。