feat: thinking-aware synthesis budgets

[moritzthecat]

Problem or use case

Reasoning summary models silently degrade to the deterministic fallback: the per-batch output budget is thinking-unaware, so thinking tokens + answer can never fit and every synthesis batch stops at length

Problem

In v9.6.2 each synthesis call's output budget is:

// src (decoded from 9.6.2 dist)
MODE_POLICIES = { balanced: { batchOutput: { min: 1000, perChunk: 250, max: 4096 }, ... }, ... }
batchOutputLimit(mode, chunks, providerMax)
  = Math.min(Math.max(budget.min, chunks * budget.perChunk), budget.max, providerMax)

so for the default balanced mode the per-batch max_tokens is capped at 4,096 (clamped to model.maxTokens in trackedComplete). This budget was designed for answer-only generation.

A reasoning summary model with thinking enabled must emit thinking tokens + answer tokens within the same budget — this is documented by the local-server provider itself:

"Pi resolves max_tokens … for each turn — so this is a real generation cap, and reasoning and the answer come out of the same budget. … A reasoning model that thinks its way to the ceiling spends the entire budget reasoning and returns no answer at all — which then reads as a truncated turn."

Once the per-batch input grows (measured threshold: session context ≈ 55–60 % of the model window), the model's thinking alone exceeds 4,096 tokens → the server stops the stream with finish reason length → the batch response is rejected (Malformed batch summary response: non-terminal stop reason length) → the pipeline degrades to its deterministic template fallback.

The degradation is nearly invisible to the user: the surfaced warning is only

Synthesis batch stopped · deterministic evidence fallback preserved coverage

— no stop reason, no hint that thinking tokens were the cause. The length reason appears only in the smart-compact debug log. There is also no warning at run start even when the configured summary model is a reasoning model with thinking enabled and a budget that small.

Net effect: the extension's headline value (a verified LLM summary) is silently replaced by the deterministic template on every compaction, for any local reasoning model whose server keeps thinking on by default — Qwen3.8-27B, Gemma-4, Llama-108B all default to thinking-on when served through LM Studio (the provider cannot disable thinking per-request for models on unmeasured backends; the page/server default is the only control).

Measured evidence (one machine, one model, one budget — only the thinking state differs)

48 GB Apple-Silicon box, LM Studio, summary model google/gemma-4-12b-qat (262 k window, 65 k maxTokens registry, balanced mode → 4,096/batch), both runs verified via prepareRun debug line:

run	page thinking	context	synthesis result
2026-09-15 22:44	on (checkpoint default)	79 %	4/4 calls stopped at length → 2× "Synthesis batch stopped · deterministic fallback" warnings; lower-quality fallback summary; ~3 min
2026-09-16 01:5x	off	~67 %	0 warnings, 0 errors, ~1 min, clean verified LLM summary, context → ~24 %

Supporting data point (qwen3.8-27b, 09/15 11:04, reduced-power profile): one synthesis call emitted 5,999 output tokens of which only 1,976 were the summary — ≈4 k thinking tokens competing for the same budget; took 17.5 min.

Workaround (verified working in our setup, but user-side and undocumented): set smartCompact.summaryThinkingLevel: "off" and disable thinking on the summary model in the server UI.

Impact

• Users with local reasoning models (the fastest-growing local-model class: Qwen3.8, Gemma-4, Llama-108B thinking builds) get template-quality summaries and a vague "coverage preserved" warning on every compaction above ~55–60 % context — with no diagnosable signal in the normal output.
• The working LLM path (thinking off) is undiscoverable from the warning text or the config docs.

Proposed solution

Proposed fix (options, any or all)

1. Thinking-aware budget: when the summary model is registered as a reasoning model, multiply the per-batch output budget by a configurable thinking factor (e.g. batchThinkingFactor, default 2–3) or reserve a floor for the answer — in batchOutputLimit / the per-call maxTokens clamp.
2. Self-healing retry: on a batch stop reason length for a reasoning model, retry that batch with reasoning: "off". The plumbing already exists — trackedComplete resolves summaryThinkingLevel into the per-call reasoning option (configuredReasoning), so a per-batch downgrade is a small change.
3. Actionable warning: surface the stop reason in the user-visible warning: Synthesis batch stopped (stop reason: length) · deterministic fallback — thinking tokens may have exhausted the 4 k output budget; set summaryThinkingLevel to "off" for reasoning models (paraphrased).
4. Preflight notice: in prepareRun, warn when the summary model is reasoning + thinking enabled + budget below a safe threshold, so the user knows before the run degrades.

Compatibility / scope

• Non-reasoning summary models are unaffected (factor applies only when model.reasoning is true).
• The thinking-off path is verified in production (table above); no watchdog/latency interaction (those are the companion timeout issue).
• No API/config breakage: option 1 adds a config key with a default; options 2–4 are behavior-only.

Alternatives considered

No response

Area

extraction

Additional context

No response

[alpertarhan]

Verified against source — mostly confirmed, with two corrections:

Confirmed

• batchOutput cap for balanced is 4096 (src/app/mode-policy.ts), formula as described.
• length stop → BatchSummaryFormatError → deterministic fallback, no retry. Exactly as reported.

Corrections

1. The warning does include the failure kind and action hint (steps/synthesize.ts appends formatGenerationFailureForUi(err)). The real bug was one level deeper: the failure classifier requires the literal text length limit, but the actual message says non-terminal stop reason length — it fell through to the generic /malformed/ branch and was classified as validation, producing the misleading "Check model availability" hint. Fixed in fix: quick wins from issue review (#61-#64) #65: it now classifies as output-limit, and the existing action hint ("Review the model output limit and reasoning level") applies.
2. summaryThinkingLevel: "off" is not a valid value in this package — valid values are minimal|low|medium|high|xhigh|max|null (src/utils/config.ts). Your workaround worked because the pi-level thinkingLevelMap: {"off": "none"} and the LM Studio server-side thinking toggle were also in play; the package-level setting would have been silently discarded. Also note the shipped default is minimal, not null — your config had null, which is what exposed you to the server default. The "every local reasoning model, every compaction" framing overstates it accordingly.

On the proposals

• Retry-on-length (self-healing): agree, and the plumbing exists. One interaction to design for: a retry doubles worst-case batch time, which collides with the watchdog ceilings from feat: Slow local models can never finish a run: 300 s per-call watchdog and 10-min run ceiling are hardcoded schema maxima #61 — that fix should land first.
• Preflight warning: cheap, will fold in.
• batchThinkingFactor: holding off until there's evidence retry + explicit levels are insufficient.

[alpertarhan]

Both halves landed: the misclassification fix in #65 (stop reason length now classifies as output-limit with the correct action hint) and the self-healing retry + unset-level preflight notice in #68. batchThinkingFactor is deliberately deferred — reopen with evidence if retry + explicit levels prove insufficient.