fix: report full prompt length in OpenAI usage - #729
Open
Javinator9889 wants to merge 3 commits into
Open
Javinator9889 wants to merge 3 commits into
Javinator9889 wants to merge 3 commits into
Conversation
The prompt cache erases the matching prefix from `tokens` before prefill, so `meta_info.prompt_tokens = tokens.size()` recorded only the newly evaluated suffix. The first turn of a conversation has no cache and looked correct, but every later turn reused the whole history as a cached prefix and reported just the new message. OpenAI defines usage.prompt_tokens as the entire input, with cached tokens counted inside it and broken out separately. Reporting the delta instead made the context appear to reset on every request, so clients that size the context from usage never learn how full it is -- an agent front-end deciding when to compact from that figure never compacts. Add the erased prefix back into prompt_tokens, carry the cached count on chat_meta_info_t, and expose it as usage.prompt_tokens_details.cached_tokens on the OpenAI endpoints. prefill_speed_tps now divides by the tokens actually evaluated, so a cache hit no longer inflates it. The Ollama-compatible prompt_eval_count is left reporting evaluated tokens only, matching upstream Ollama.
Two-turn check, streaming and non-streaming, that prompt_tokens counts the cached prefix and cached_tokens reports it. Needs a live server.
Author
|
Hey @zaneni6 this PR should be ready to be reviewed :) |
Single-turn models reject conversation history, so the two-turn shape does not apply. Detect the server's rejection and check the pinned system prefix across two one-shot requests instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
In OpenCode and similar OpenAPI-compatible-endpoint applications, the current context usage is wrongly reported, as it just takes into account the newly evaluated suffix/prompt rather than the cumulative count of tokens in the context.
This causes these tools to fail at certain point, receiving an error message stating that "the context was cleared" with no time to react. Built-in tools such as context compression that specifically prevent this problem for happening never run, and the conversation is re-fed again by the third party tool to the FLM model, causing a subsequent context overflow, and that in a loop.
Technical Details
The prompt cache erases the matching prefix from
tokensbefore prefill, someta_info.prompt_tokens = tokens.size()recorded only the newly evaluated suffix. The first turn of a conversation has no cache and looked correct, but every later turn reused the whole history as a cached prefix and reported just the new message.OpenAI defines usage.prompt_tokens as the entire input, with cached tokens counted inside it and broken out separately. Reporting the delta instead made the context appear to reset on every request, so clients that size the context from usage never learn how full it is -- an agent front-end deciding when to compact from that figure never compacts.
Add the erased prefix back into prompt_tokens, carry the cached count on chat_meta_info_t, and expose it as usage.prompt_tokens_details.cached_tokens on the OpenAI endpoints. prefill_speed_tps now divides by the tokens actually evaluated, so a cache hit no longer inflates it.
The Ollama-compatible prompt_eval_count is left reporting evaluated tokens only, matching upstream Ollama.
Test Plan
The testing is hard to automate as it requires a real model running on a real NPU and a third party tool (such as OpenCode) to be running to see lively how the context increases with every conversation exchange.
Test Result
Verified locally running OpenCode against Qwen3.5:9b, in a multi-turn conversation where the context was growing as new messages were exchanged
Submission Checklist
Closes #730