Context budget: an undeclared window fabricates a capacity that kills live turns

[Joob1n]

Summary

A conversation of roughly 8,700 estimated tokens was terminated with context_budget_exhausted against a model whose real window is 200,000 — because the model's window was not declared, and the runtime substituted a fabricated capacity for it. The turn was killed before any request went out, so the provider never got to say whether it actually fit.

Five independent defects line up to produce this. Each is survivable alone; a relay model plus an image read hits all of them at once.

I would like to align on the target design before writing code, and on one sequencing question: add the diagnostics first, because the single number the verdict turns on is currently never recorded.

Evidence

From the session that failed (glm-5.3-flash on an openai-compatible relay, one turn, 33 runtime events):

runtime_events              33 rows, 34,706 bytes  ≈ 8,700 tokens
session_messages            22 rows, 16,062 bytes
usage_llm_calls             3 rows — all auxiliary
                              session_title       inputTokens 69
                              memory_proposal     inputTokens 756
                              memory_canonicalize inputTokens 417
                            0 rows for the conversation's own model calls
terminal event              contextBudgetExhaustedDetail: head_anchor_exceeds_capacity
                            no estimate, no capacity, no capacity source recorded
declared window at the time none  →  capacity = 32k + 16k = 48k

The relay's /models returns { "id": "glm-5.3-flash" } — no context_length, no limits — and openai-compatible carries no bundled metadata, so resolveSelectedModelContextWindow returns undefined.

Current pipeline (one step inside a turn)

new content enters the context
        │   function_response (tool result) | text (user message)
        ▼
capacity  resolveContextBudgetCapacity()
        │   window known    → capacity = window        source=selected_model
        │   window unknown  → capacity = 32k + 16k     source=policy_fallback   ← (1)
        ▼
estimate  estimateNextRequestTokens()
        │   with anchor → last provider inputTokens + appendedChars / 4
        │   no anchor   → whole payload / 4
        │                                                                        ← (2)
        │   · conversation usage is never persisted → a restart loses the anchor
        │   · charsPerToken is fixed at 4; Chinese runs closer to 2
        │   · an image is stored as a reference (~0 chars) and billed as
        │     hundreds-to-thousands of vision tokens
        ▼
   estimate > capacity ?
        │            │
       no│            │yes
        │            ▼
        │      compact
        │        selectSafeCompactionPrefix
        │          · may not end on a partial
        │          · may not split a call/result pair
        │          · mid_turn keeps ≥ 1 verbatim tail event                       ← (3)
        │        summarize with a model call → validate → write checkpoint
        │            │
        │            ▼
        │      re-estimate; still over?
        │            │yes
        │            ▼
        │      head_anchor_exceeds_capacity
        │      TERMINATE — the request is never sent                       ← core defect
        │      the provider never gets a say
        ▼
      send ──→ provider
                 ├─ rejected → reactive recovery (already implemented)
                 └─ ok → usage.inputTokens (discarded)                            ← (2)

The escape hatch that should have saved this is activeToolResultPrune, which can swap an oversized result for a small projection while archiveToolResultAsTransition keeps the full body in an artifact. It measures serialized bytes against a 2,048-token threshold. The image reference is 1,104 bytes ≈ 276 tokens, so the prune sees nothing to do. ← (4)

The five defects

#	Defect	Effect
1	Capacity is fabricated when the window is unknown	The threshold has nothing to do with the model
2	The estimate is distorted in three common cases	Lost anchor after restart / non-English text / images
3	Compaction's floor is summary + 1 tail event	The new message never folds; its size is a hard floor
4	The only escape hatch is blind to images	It judges by bytes, so it closes itself
5	Nothing is visible to the user until the task dies	The first mention of "context" is the failure banner

Target design

new content enters the context
        ▼
capacity
        │   window known   → capacity = window
        │   window unknown → capacity = undefined                          fixes (1)
        │                    no local threshold; the provider decides
        │                    the UI says the window is undeclared
        ▼
estimate  (drives compaction only — never termination)
        │   anchor = last real inputTokens, persisted, survives restart     fixes (2)
        │   charsPerToken derived from this session's own history
        │   images: only the step they first enter is mis-estimated, and
        │           they sit inside the anchor from the next step on
        ▼
   estimate > capacity ? ──yes──→ compact once (mark "compacted this step")
        ▼
      send   ←── the request goes out regardless of what the estimate said
        │
   ┌────┴─────┐
   ▼          ▼
provider ok   provider rejects
│             (context_length_exceeded / model_context_window_exceeded /
│              prompt is too long / request_too_large / …)
├ persist usage        │
├ → next step's anchor │
└ → usage indicator    │            fixes (5)
                  ┌────┴────────────────┐
                  ▼                     ▼
          not compacted            already compacted
          this step                this step
          compact, retry once      at the compaction floor
          (bounded re-entry)             │
                                         ▼
                          shrink the tail                        fixes (4)
                            identify an oversized artifact
                            archiveToolResultAsTransition
                              full body stays in the artifact
                              model sees a placeholder projection
                            retry once
                                         │
                                         ▼
                                  still does not fit
                          ┌──────────────────────────────────┐
                          │ terminate, stating only what is  │
                          │ known:                           │
                          │  · the provider's own error code │
                          │  · bytes/tokens actually sent    │
                          │  · the window                    │
                          │  · capacity source               │
                          │    (declared or fallback)        │
                          │  · which item is largest         │
                          │ no inferred cause                │
                          └──────────────────────────────────┘

Context usage indicator (defect 5)

In the composer toolbar (packages/ui/src/composer.tsx, .maka-model-selection-controls), immediately right of the thinking-level control:

window known
   [ ＋ ] [ 🛡 ] [ ◎ glm-5.3-flash ] [ 最高 ] [ ▤ 42% ]
                                                 │
              numerator: the last request's real usage.inputTokens
                         (includes images, priced by the provider)
              denominator: the declared or discovered window

window unknown
   [ ＋ ] [ 🛡 ] [ ◎ glm-5.3-flash ] [ 最高 ] [ ⚠ ]
                                                 │
              "Context window not declared — usage cannot be shown and
               the runtime cannot compact ahead of time."   → Settings

Being unable to compute the percentage is the evidence that the configuration is missing, so the indicator's slot is the natural home for the warning. The user learns this when they start working, not when a turn is killed.

Design principles to align on

1. An estimate may drive a reversible action (compaction) and must never drive an irreversible one (termination).
2. Termination belongs to the provider's real rejection. Only the provider counts accurately — especially images and non-English text.
3. Reports state what is known and do not infer a cause. Today's "change models or start a new task" and a tempting "the new message is too large" are the same mistake: both name a cause nobody measured.

Sequencing — the part I would most like a second opinion on

The number the verdict turns on is never recorded. The terminal event carries head_anchor_exceeds_capacity and nothing else: no estimate, no capacity, no capacity source. Everything above about how the estimate went wrong is inference from surrounding data, not measurement.

So I would like to land diagnostics first:

1. Persist conversation usage alongside the auxiliary calls that already record it, and put estimate / capacity / capacity-source / largest-item on the terminal event.
2. Then defect 1 (stop fabricating capacity) and defect 5 (the indicator), which share that data.
3. Then defect 4 (make the prune see artifacts) — this is what turns "context is full" from a dead end into something recoverable.
4. Then defect 2 (anchor persistence and a self-calibrating chars-per-token).

Happy to be argued out of this order — in particular, whether an unknown window should mean "no threshold" (my preference: the provider's answer is correct, ours is not) or a conservative default (my objection: any constant is wrong for some model, and being wrong low kills live work while being wrong high only costs one round trip).

中文

问题

一个估算约 8,700 token 的会话，被 context_budget_exhausted 终止，而该模型的真实窗口是 20 万 —— 原因是模型窗口未声明，运行时用一个编造的容量替代了它。终止发生在请求发出之前，供应商从未有机会判断它到底装不装得下。

有五个互相独立的缺陷叠在一起。任何一条单独存在都不致命；而「中转站模型 + 读一张图」会同时踩中全部。

我希望在动手写代码之前先对齐目标设计，以及一个顺序问题：先加诊断 —— 因为判定所依据的那个数字，目前从不被记录。

证据

来自失败的那个会话（glm-5.3-flash，openai-compatible 中转站，一个 turn，33 条运行时事件）：

runtime_events              33 条，34,706 字节 ≈ 8,700 token
session_messages            22 条，16,062 字节
usage_llm_calls             3 条 —— 全是辅助调用
                              session_title       inputTokens 69
                              memory_proposal     inputTokens 756
                              memory_canonicalize inputTokens 417
                            对话本身的模型调用：0 条
终止事件                     contextBudgetExhaustedDetail: head_anchor_exceeds_capacity
                            没有估算值、没有容量值、没有容量来源
当时声明的窗口               无  →  容量 = 32k + 16k = 48k

中转站的 /models 只返回 { "id": "glm-5.3-flash" } —— 没有 context_length、没有任何上限 —— 而 openai-compatible 又没有内置元数据，所以 resolveSelectedModelContextWindow 返回 undefined。

五个缺陷

#	缺陷	后果
1	窗口未知时容量是编造的	门槛与模型无关
2	估算在三种常见情况下失真	重启丢锚点 / 非英文文本 / 图片
3	压缩下限是「摘要 + 1 条尾部」	新消息永不折叠，它的体量就是硬下限
4	唯一的退路对图片视而不见	按字节判断，退路把自己关上
5	用户全程看不到任何指标	第一次听说「上下文」就是失败横幅

本该救场的是 activeToolResultPrune：它能把超大结果换成小投影，同时由 archiveToolResultAsTransition 把完整内容留在 artifact 里。但它按序列化字节对照 2,048 token 的阈值判断，而图片引用只有 1,104 字节 ≈ 276 token，于是它认为无事可做。← (4)

目标设计的三条原则

1. 估算只能驱动可逆动作（压缩），永远不能驱动不可逆动作（终止）。
2. **终止权属于供应商的真实拒绝。**只有供应商数得准 —— 尤其是图片和非英文文本。
3. **报告只陈述已知，不臆断原因。**现行的「换模型或开启新任务」，和一个听起来很顺的「新消息过大」，犯的是同一个错：都在指认一个没人测量过的原因。

上下文使用率指示器（缺陷 5）

放在输入框工具条（packages/ui/src/composer.tsx 的 .maka-model-selection-controls）里，思考档位控件右侧：

窗口已知
   [ ＋ ] [ 🛡 ] [ ◎ glm-5.3-flash ] [ 最高 ] [ ▤ 42% ]
              分子：最近一次请求的真实 usage.inputTokens（含图片，按供应商计价）
              分母：声明或探测到的窗口

窗口未知
   [ ＋ ] [ 🛡 ] [ ◎ glm-5.3-flash ] [ 最高 ] [ ⚠ ]
              「未声明上下文窗口 —— 无法显示用量，也无法提前压缩」→ 去设置

算不出百分比这件事本身，就是配置缺失的证据，所以指示器的位置天然就是提示该出现的地方。用户在开始工作时就看到，而不是在任务被杀时才第一次得知。

顺序 —— 这一点我最想听听别人的意见

判定所依据的那个数字从不被记录。终止事件只有 head_anchor_exceeds_capacity，没有估算值、没有容量值、没有容量来源。上面关于「估算是怎么错的」的所有分析，都是从周边数据做的推理，不是测量。

所以我想先落诊断：

1. 把对话的 usage 落盘（辅助调用已经在这么做了），并在终止事件上带上 估算值 / 容量 / 容量来源 / 最大的那一条。
2. 然后是缺陷 1（不再编造容量）和缺陷 5（指示器），二者共用同一份数据。
3. 然后是缺陷 4（让裁剪器看得见 artifact）—— 这一条把「上下文满了」从死局变成可恢复。
4. 最后是缺陷 2（锚点落盘 + chars-per-token 自校准）。

这个顺序欢迎反驳。特别是：窗口未知时到底应该「不设门槛」（我倾向这个：供应商的答案是对的，我们的不是），还是给一个保守默认值（我的反对理由：任何常数对某些模型都是错的，而猜小了会杀掉正在进行的工作，猜大了只是多一次往返）。

[Joob1n]

and I wanna change the default context-windows size to 128k.

[likun666661]

I agree with the core direction here. From an Occam perspective, the smallest correctness fix is not to replace the 48K fallback with another constant such as 128K; it is to stop treating an unknown window as a known capacity.

The invariant I would use is: a local estimate may trigger reversible compaction, but it must never terminate a live turn. When the window is undeclared, send the request and let the provider decide. On a real provider context-length rejection, compact once and retry; if it is still rejected, surface the provider error.

One implementation constraint is worth calling out: today MidTurnCapacityCompactState is only created when resolveContextBudgetCapacity returns a value, while reactive overflow recovery also requires that state. Therefore, simply returning undefined for an undeclared window would accidentally disable provider-driven recovery. The recovery/turn state needs to exist independently of the optional proactive capacity threshold.

I would keep the first fix narrowly scoped to that correctness path and defer image-aware pruning, self-calibrating token estimates, anchor persistence, and the composer indicator. Those are useful follow-ups, but none is required to prevent this false termination.

[Astro-Han]

Agreed — an estimate may drive compaction and must never end a turn, and 128K would only move the threshold.

Findings on main (6668af3f55).

Why it happened

The capacity was synthesized. With a declared window capacity = window, and the reserve / history / highWater derivations round-trip straight back to it. With an undeclared window the same round trip is rebuilt from two policy constants: 32000 + 16384 = 48384. policy_fallback also makes the verdict run at step 0, which a declared window does not (ai-sdk-compaction.ts:1511).

The escape hatch and the gate that fired measured different objects. For a ~500 KB image read:

active-tool-result-prune.ts:292   durable projection     ≈ 276 tok    → nothing worth replacing
ai-sdk-compaction.ts:1798         materialized request   ≈ 167K tok   → over capacity, terminate
                                  provider's own count   ≈ 1.5K tok   → never asked

Your state constraint is wider than one call site: there are two gates. This one is not behind MidTurnCapacityCompactState:

ai-sdk-backend.ts:3429   estimateRuntimeEventsTokens(runtimeContext) > maxHistoryEstimatedTokens
ai-sdk-backend.ts:3509   → status: 'context_budget_exhausted'   (provider calls: 0)

It reads maxHistoryEstimatedTokens directly, and that is 32_000 exactly when the window is undeclared. Returning undefined from resolveContextBudgetCapacity alone would keep this gate and lose recovery. Capacity and history budget have to go absent together.

The problem

1. An undeclared window is a missing fact; the code turns it into a value, and every threshold downstream inherits authority nothing earned.
2. "What the model sees" and "how large the request is" are the same question only for text. The artifact reference makes them diverge, and it does not carry its materialized cost.

Proposed

In your scope, one commit:

1. capacity and history budget go absent together;
2. both gates no-op when absent;
3. turn/recovery state stops depending on the capacity threshold, so provider-driven compact-and-retry survives.

Structurally, on #4283: the artifact part declares its materialized cost.

Why that piece belongs to #4283

#4348 unified four consumers onto the durable projection; all four read RuntimeEvent, and that is right. #4283 also keeps materialization pure and forbids inline images in RuntimeEvent JSON — also right, and exactly why an image is a reference on the ledger side.

But the artifact part is in #4283's own schema, and it carries identity only:

{ kind: 'artifact'; mediaType: string; ref: DurableProjectionArtifactRef }

No cost field, so the projection ruler is complete for text and silently partial for media — which is how the prune concluded there was nothing to replace. Byte length is free at all three projection sites.

Adding it makes #4350's own line true rather than aspirational — "the projection is what costs context" — and lets the wire ruler be deleted rather than corrected. midTurnRequestPayloadChars is system prompt + JSON.stringify(messages) + tool schemas; only the middle term touches media, and with cost in the projection it becomes computable before materialization. One ruler instead of two.

It needs a projection schema bump through the codec #4283 already has, and an older projection with no cost field means cost unknown — which must not drive a termination either.

One scoping note, read off the code path rather than reproduced: attachment images enter the payload as a Uint8Array (ai-sdk-backend.ts:4348), and JSON.stringify renders that as {"0":137,...} — about 2 estimated tokens per byte. A ~100 KB screenshot then estimates at ~200K against a correctly declared 200K window. The narrow fix closes the relay case and leaves that one open.

Happy to open the artifact-cost piece on #4283 so this issue stays on the correctness path.