Repository navigation
gateway: hold a translated response's heap in the body budget; look up the walk's candidate once - #128
Merged
Conversation
`request_body_filter` resolved `a.candidate_at(a.candidate)` six times in one block; the slot does not move inside it, so bind it once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
jaredLunde
enabled auto-merge (squash)
October 4, 2026 20:15
A non-streaming JSON response translated onto another wire builds the same `Value`s a request does, up to ~250 bytes of heap per byte for many tiny objects, but only its raw size was capped (32 MiB). It is now held in the body budget before it is mapped, as a request is; one that does not fit is aborted, since its status line is already downstream. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
jaredLunde
changed the title
jaredLunde
disabled auto-merge
October 4, 2026 20:56
jaredLunde
enabled auto-merge (squash)
October 4, 2026 20:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
serde_json::Value, just as it does for a translated request. For bodies made of many tiny objects, that costs up to ~250 bytes of heap per byte of JSON. Requests already reservetranslate::translation_heap's estimate in the body budget before translating. Non-streaming responses were only limited by raw size (MAX_TRANSLATE_BUFFER, 32 MiB), so in the worst case one response could take several GB. They now make the same reservation at end of stream. If it doesn't fit, the response is aborted (its status line has already gone downstream), astranslate_overflowalready does. A same-wire 2xx is relayed as bytes and reserves nothing.request_body_filterlooked upa.candidate_at(a.candidate)six times in one block. It's now looked up once.Realistic exposure is smaller than the 32 MiB cap implies. Catalog output limits keep a model's own answer to a few hundred MB of heap at worst. Reaching the multi-GB case needs a broken or hostile upstream.
Test plan
body_memory::a_response_too_costly_to_translate_is_aborted: a 4 MiB Claude tool input of tiny objects, translated for a Chat client, is aborted and peak RSS stays within a few copies of the body. Without the fix it fails, because the response is translated in full.cargo clippy -p beyond-ai --all-targetscargo test -p beyond-ai(all 71 test binaries pass)🤖 Generated with Claude Code
https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr