Skip to content

fix: let an unchanged conversation hit the prompt cache - #734

Open
Javinator9889 wants to merge 1 commit into
ROCm:mainfrom
Javinator9889:fix/prompt-cache-identical-resend
Open

Javinator9889 wants to merge 1 commit into
ROCm:mainfrom
Javinator9889:fix/prompt-cache-identical-resend

Conversation

@Javinator9889

Copy link
Copy Markdown

Fixes the first defect in #733.

can_use_cache requires at least two new messages:

const size_t prefix_len = messages.size() - 2;
const bool can_use_message =
    message_checksums_.size() <= prefix_len && matched == message_checksums_.size();

But message_checksums_ is updated to the full incoming list after every
request, so a resend of an unchanged conversation has
message_checksums_.size() == messages.size() and always fails the bound.

That is exactly what a client sends after a timeout. The miss re-prefills the
whole conversation — slower than the request that just timed out — so the retry
times out as well and the loop sustains itself. Observed with OpenCode on a
146-message, ~98k-token conversation.

Bounding by the full incoming length lets the resend hit. The second hunk guards
the case that exposes: with nothing left to prefill, _chunked_insert computes
zero chunks and returns a default-constructed (empty) logits buffer straight into
sampler->sample(). Without it this change crashes the connection.

Verified on qwen3.5:9b: usage contract and generation output unchanged, and a
buried-value probe returns the same string 4/4 on both cold and cached paths.

Note this does not make the fully-cached case cheap — it still re-prefills.
Doing better needs engine-side support; see #733 for the measurements.

can_use_cache required at least two new messages (messages.size() - 2), but
message_checksums_ is updated to the full incoming list after every request.
A resend of an unchanged conversation therefore fails the bound and misses,
re-prefilling everything -- which is slower than the request that timed out,
so the client's retry times out too and the loop sustains itself.

Bound by the full incoming length instead. Also guard the case this exposes:
with nothing left to prefill, _chunked_insert computes zero chunks and hands
an empty logits buffer to sampler->sample().

Refs ROCm#733
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant