Skip to content

fix(token-spy): preserve upstream streaming status - #2970

Open
tang-vu wants to merge 1 commit into
Osmantic:mainfrom
tang-vu:fix/token-spy-stream-status
Open

fix(token-spy): preserve upstream streaming status#2970
tang-vu wants to merge 1 commit into
Osmantic:mainfrom
tang-vu:fix/token-spy-stream-status

Conversation

@tang-vu

@tang-vu tang-vu commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Why this matters

Token Spy created its downstream StreamingResponse before opening the upstream stream. FastAPI therefore committed HTTP 200 even when Anthropic or an OpenAI-compatible provider returned 4xx/5xx. Clients could treat rate limits and provider failures as successful requests, defeating normal retry and error handling.

Root cause: the upstream status was only available inside the response-body iterator, after downstream headers had already been selected.

Behavioral invariant: both supported streaming APIs return the exact upstream HTTP status and still close the upstream response after passthrough.

Overlap check

Searched open and closed PRs for Token Spy streaming/status/error/passthrough terms and inspected PRs changing ods/extensions/services/token-spy/main.py. PR #1963 improves streaming error-body logging but does not propagate the upstream status; other current Token Spy PRs cover SSE cursors, pricing, writes, and exception handling. This change is independent and does not alter the logging payload.

What changed

  • Open each upstream response with httpx's public build_request/send(stream=True) API before creating the downstream response.
  • Set the downstream StreamingResponse status from the opened upstream response for Anthropic and OpenAI-compatible routes.
  • Close the upstream response in a shared async context even when streaming is cancelled or fails.
  • Map a connection failure before headers to the existing 502 proxy-error contract.
  • Add authenticated public-endpoint regressions for both protocol routes.

Validation

  • Red before fix: both /v1/messages and /v1/chat/completions returned HTTP 200 for an upstream 429.
  • pytest tests/test_streaming_status.py -q — 2 passed.
  • pytest tests -q — 22 passed, 1 skipped.
  • python -m py_compile main.py tests/test_streaming_status.py — passed.
  • git diff --check — passed.

Tradeoffs and rollback

The proxy now waits for upstream response headers before sending downstream headers; that is required to preserve status and does not buffer the response body. SSE body processing and usage accounting remain unchanged. Rollback is one commit with no state or schema migration.

Batch compatibility

Validated as an independent ten-PR batch from upstream main 6ff9b4fc5190099705043acaab7e9b6ad9c8b8f1. The final PR heads merged without conflicts in this order: #2964 -> #2965 -> #2967 -> #2969 -> #2970 -> #2971 -> #2972 -> #2973 -> #2974 -> #2975. The resulting local synthetic merge head is 4ae60eadad9696a9accb735aad71af87d2be802d.

Combined validation on that exact tree:

  • Dashboard API boundary suites: 323 passed, 4 skipped.
  • Token Spy suite: 36 passed, 1 skipped.
  • Privacy Shield suite: 55 passed.
  • APE suite: 47 passed.
  • Python compile checks and git diff --check: passed.

The scopes are behaviorally independent. The stated order is the tested rollback/merge sequence for shared-file changes; each PR remains individually useful and revertible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant