Skip to content

feat: show context window usage beside the send button - #35

Merged
iuga merged 2 commits into
masterfrom
tokenStats
Jul 8, 2026
Merged

feat: show context window usage beside the send button#35
iuga merged 2 commits into
masterfrom
tokenStats

Conversation

@iuga

@iuga iuga commented Jul 1, 2026

Copy link
Copy Markdown
Owner

Motivation

When chatting with a local model, there's no visibility into how much of the model's context window a conversation has consumed. Once the window fills up, response quality degrades and older context gets dropped silently. This meter lets users see how close they are so they can start a fresh session at the right time.

Description

Adds a small meter to the left of the send/stop button showing x% - used / total (e.g. 24% - 12.3K / 300K). Token counts update after each turn; the window size is read from the model itself. A new Token usage stats toggle in Chat Experience controls it and defaults to on.

Changes

  • Provider: expose the model's effective context window from Ollama's /api/show: the modelfile's num_ctx when set (what Ollama actually truncates at), falling back to the architecture maximum (ModelDetails.contextLength).
  • Agent: read per-turn token usage from the agent's checkpointed state (one state read per interrupt cycle, shared with the ask-user check) and surface it via an onUsage handler.
  • Plugin: resolve the context window on load and on every settings save; if the host was unreachable at load, retry on the next successful health check; push updates to open views.
  • View + styles: render the meter, reset the counter on new/loaded sessions, turn it red at >=90%. Counts formatted with Intl.NumberFormat compact notation.
  • Settings: add the showTokenStats toggle (default true).

TradeOffs

  • Token usage is read from the agent's checkpointed state after each turn rather than from the live stream. LangGraph's messages stream mode drops the final usage-bearing chunk, so the stream never sees token counts. Reading from state is reliable but means the meter updates once per turn (on completion), not token-by-token — an acceptable tradeoff for accuracy.
  • The context window is re-resolved on every settings save instead of caching per host/model. It's one cheap local /api/show call per user-initiated save, which keeps the plumbing much simpler and self-heals stale values.

iuga added 2 commits July 1, 2026 14:19
Add an opt-out meter that displays how much of the chat model's context
window a conversation is using, formatted as "x% - used / total". This
gives users visibility into how close they are to filling the model's
context so they can start a new session before quality degrades.

Token counts come from the model's reported usage after each turn, and
the window size is read from the model's own metadata (resolved on load
and whenever the model or host changes). A new "Token usage stats"
setting in Chat Experience toggles the meter and defaults to on.
The meter's denominator used the architecture maximum from model
metadata, but Ollama truncates prompts at the modelfile's num_ctx,
so the meter could show plenty of headroom while the prompt was
already being cut. Prefer num_ctx when set, and retry resolving the
window on the next successful health check if the host was down at
plugin load.

Also simplify the plumbing after review: read token usage only from
the checkpointed state (one state read per cycle), drop the dead
in-stream capture path and the host/model signature cache, and format
token counts with Intl.NumberFormat.
@iuga
iuga merged commit 9940ce2 into master Jul 8, 2026
4 checks passed
@iuga
iuga deleted the tokenStats branch July 8, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant