fix(openai-ws): never reuse a socket with an abandoned response - #200
Merged
Merged
Conversation
A request abandoned mid-stream (the caller drops the future, e.g. a task abort on user interrupt) leaves the server streaming a response nobody reads. The next request on the same socket consumed that response's tail as its own answer, shifting every later reply by one, until the server rejected the incremental chain with previous_response_not_found. - Mark the connection busy before a request goes out; only the response's terminal event clears it. ensure_connection retires a marked socket. - Output events before this request's response.created fail the request as stale instead of being consumed. - A rejected chain or a stale stream is retried once on a fresh connection with the full input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Responses WebSocket client carries at most one response per socket and has no per-event correlation to a request. When a caller abandons a request mid-stream by dropping the future (a caller that aborts the request task on a user interrupt, say), no error path runs in the client: the connection survives and the server keeps streaming the dead response into it.
The next
response.createon that socket then reads the previous response's remaining events as its own answer. From there on every reply is shifted by one request, and theprevious_response_idbookkeeping no longer matches the server, until the server rejects the chain withprevious_response_not_foundand the connection is finally dropped.Observed in a real session: three
previous_response_not_foundfailures over two hours, each preceded by an interrupt, with the assistant answering the previous message instead of the current one in between.Fix
completed,failed,incomplete) clears the mark.ensure_connectionretires a marked socket instead of reusing it. This covers the task-abort case, where no code of the client runs.response.createdfail the request as stale rather than being consumed. Lifecycle events are not affected.previous_response_not_foundand a stale stream are retried once on a fresh connection, which resets the chain so the full input is sent.process_ws_streamnow reads from the connection's receiver directly and reports what it established about the response, which makes the reader testable with a scripted event flow.Tests
response.createdis rejected as stale and retryableresponse.createdare not stalecargo test -p llm: 129 passed. Clippy clean.