fix(runtime): don't count image bytes as raw chars in the context-budget verdict - #4291
fix(runtime): don't count image bytes as raw chars in the context-budget verdict#4291liuxiaocs7 wants to merge 1 commit into
Conversation
d89637b to
416e2d3
Compare
…get verdict
The mid-turn final-request capacity verdict sized the outgoing payload with
`JSON.stringify(messages).length`, which serializes an inline image at the
size of its transported bytes: a Uint8Array/Buffer expands to a digit map
(`{"0":..}` or `{"type":"Buffer","data":[...]}` once toJSON runs) and a base64
string is counted at full length. Divided by charsPerToken that reads as
hundreds of thousands of tokens, so a single attached image could trip
`context_budget_exhausted` before any request was sent -- most visibly on an
unknown-window model whose `policy_fallback` capacity runs the verdict on
step 0 (Steps 0, ~263ms, no provider round-trip). The reported Desktop upload
path reads a session-file as a Node Buffer.
Walk the message content and discount only genuine IMAGES to a bounded
per-image estimate (~1.6K tokens, the vision per-image ceiling) scaled by the
policy's charsPerToken. Image classification mirrors the AI SDK's own
normalization: an `image` part, an `image` top-level mediaType (bare `image`
or `image/*`, incl. a data: URL's own type), or inline bytes whose signature
is a known image (which the SDK uses to override the declared type). Each
image is charged once regardless of how it is carried (inline bytes, a data:
URL, or a remote reference); only inline bytes are stripped, and a remote URL
stays verbatim. Non-image files (PDF, text, audio) and look-alike content
(e.g. a `{type:'data'}` tool-call input) stay fully measured, so the budget
never silently under-counts them, and the image=0 under-count of apache#3372 is
avoided.
Covered by estimator unit tests (Buffer / Uint8Array / base64 / data: URL /
remote URL string / bare `image` type / generic-MIME + magic bytes / PDF /
tool-call input / charsPerToken) and an end-to-end backend test: an
unknown-window first turn with a 200 KB Buffer image now reaches the provider
instead of a pre-send context_budget_exhausted.
Fixes apache#4290
Generated-by: Claude Code
416e2d3 to
efbfcf2
Compare
|
Closing as superseded. Current Rebasing this branch onto The only coverage this PR adds beyond Thanks! |
Summary
A first-turn message carrying a single user-uploaded image can fail immediately with
context_budget_exhausted/no_safe_completed_span— the runtime record showsSteps (0), ~263 ms, no provider round-trip. The turn is killed before anything is sent.Root cause: the mid-turn final-request capacity verdict sized the outgoing payload with
JSON.stringify(messages).length. That serializes an inline image at the size of its transported bytes — aUint8Array/Bufferexpands to a digit map ({"0":..}, or{"type":"Buffer","data":[...]}onceJSON.stringifyappliesBuffer.toJSON), and a base64 image is counted at full length. Divided bycharsPerTokena single ~150 KB screenshot reads as hundreds of thousands of "tokens", exceeding the window on its own. It surfaces most on a model with no known context window:resolveContextBudgetCapacityreturnspolicy_fallback(48,384 tokens), andpolicy_fallbackis exactly the case where the verdict runs on step 0, before the first request. Being the first turn, the current user message is the pinned head anchor and nothing is foldable, so the verdict resolves tono_safe_completed_span. The reported Desktop path reads a session-file upload as a Node Buffer (packages/storage/src/artifact-attachments.ts), so the estimate must surviveBuffer.toJSON.The fix walks the
ModelMessagecontent and discounts only genuine images to a bounded per-image estimate (~1.6K tokens, the vision per-image ceiling) scaled by the policy'scharsPerToken. Image classification mirrors the AI SDK's own normalization (convert-to-language-model-prompt), so the estimate matches what the provider is billed: animagepart, animagetop-level mediaType (bareimageorimage/*, including adata:URL's own type), or inline bytes whose signature is a known image (the SDK overrides the declared mediaType from the byte signature). Each image is charged once regardless of how it is carried (inline bytes, adata:URL, or a remote reference); only inline bytes are stripped from the serialized size, and a remotehttp(s)URL stays verbatim. Everything else is left to serialize verbatim:{ type: 'data' }tool-call input) is untouched.This removes the over-count without reintroducing the opposite image-counts-as-zero under-count fixed in #3372. All four capacity call sites share the one function, so they stay consistent.
Fixes #4290
Verification
mid-turn-image-payload-chars.test.ts, 13/13): the Buffer / Uint8Array / base64 /data:URL / remote-URL-string / bare-image-type / generic-MIME-with-magic-bytes / PDF / tool-call-input shapes,charsPerTokenscaling, and no change for text-only messages.mid-turn-capacity-backend.test.ts): an unknown-window model, a first turn whose only message is a 200 KB Buffer image upload — the provider now receives the request (with the image materialized) instead of a pre-sendcontext_budget_exhausted. The fixture's attachment reader was made to return a NodeBuffer, matching production.context-budget-mid-turn-policy(8),mid-turn-capacity-backend(67),overflow-reactive-recovery(43),ai-sdk-backend(217),history-compaction(22).tsc -p packages/runtime/tsconfig.jsonclean;biome checkclean on the changed files.Review focus
The
policy_fallbackfail-open (letting the provider's real token count decide for a first, unshrinkable user message) is intentionally not in this PR — it would change the verdict for large first-turn text, files, and tool schemas, and belongs in its own issue/PR with reactive-recovery coverage.AI use
Select exactly one:
Tool(s) and scope: Claude Code (Anthropic) performed the root-cause diagnosis, wrote the runtime fix in
ai-sdk-compaction.ts, and authored the regression tests. The commit carries aGenerated-bytrailer.Checklist
Does this PR entail a change in behavior?