feat: show context window usage beside the send button - #35
Merged
Conversation
Add an opt-out meter that displays how much of the chat model's context window a conversation is using, formatted as "x% - used / total". This gives users visibility into how close they are to filling the model's context so they can start a new session before quality degrades. Token counts come from the model's reported usage after each turn, and the window size is read from the model's own metadata (resolved on load and whenever the model or host changes). A new "Token usage stats" setting in Chat Experience toggles the meter and defaults to on.
The meter's denominator used the architecture maximum from model metadata, but Ollama truncates prompts at the modelfile's num_ctx, so the meter could show plenty of headroom while the prompt was already being cut. Prefer num_ctx when set, and retry resolving the window on the next successful health check if the host was down at plugin load. Also simplify the plumbing after review: read token usage only from the checkpointed state (one state read per cycle), drop the dead in-stream capture path and the host/model signature cache, and format token counts with Intl.NumberFormat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When chatting with a local model, there's no visibility into how much of the model's context window a conversation has consumed. Once the window fills up, response quality degrades and older context gets dropped silently. This meter lets users see how close they are so they can start a fresh session at the right time.
Description
Adds a small meter to the left of the send/stop button showing
x% - used / total(e.g.24% - 12.3K / 300K). Token counts update after each turn; the window size is read from the model itself. A new Token usage stats toggle in Chat Experience controls it and defaults to on.Changes
/api/show: the modelfile'snum_ctxwhen set (what Ollama actually truncates at), falling back to the architecture maximum (ModelDetails.contextLength).onUsagehandler.Intl.NumberFormatcompact notation.showTokenStatstoggle (defaulttrue).TradeOffs
messagesstream mode drops the final usage-bearing chunk, so the stream never sees token counts. Reading from state is reliable but means the meter updates once per turn (on completion), not token-by-token — an acceptable tradeoff for accuracy./api/showcall per user-initiated save, which keeps the plumbing much simpler and self-heals stale values.