Fix cloud agent constraint loss in long conversations - #579
Open
QuanCheng-QC wants to merge 6 commits into
Open
Conversation
Cloud agents rebuilt context from only the latest 10 channel messages, so constraints confirmed early in a conversation fell out of the prompt after about 5 turns and the model would re-decide settled questions. - raise the CLOUD_AGENT_MAX_CONTEXT_MESSAGES default from 10 to 100 - over-fetch and filter thinking/status/todos rows in Python so noise no longer consumes window slots - exclude the triggering message by event id instead of blindly dropping the newest row, which could evict real history under concurrent posts - mark Anthropic system prompt and conversation tail with ephemeral cache breakpoints so the resent history bills at cache-read rates - add tests covering window size, noise filtering, id-based exclusion and role mapping
- bound context to events causally prior to the trigger via a before_timestamp SQL filter so messages committed after the trigger can no longer leak into the request and get answered early, with same-millisecond rows conservatively dropped since events carry no finer ordering, and the id-only and positional exclusions kept as fallbacks when no timestamp is available - add a character budget (CLOUD_AGENT_MAX_CONTEXT_CHARS, default 60000, env-overridable) loaded newest-first alongside the message count cap, since 100 unbounded message bodies could exceed the context of small or custom models where 10 previously fit, and the newest message is truncated rather than dropped when it alone exceeds the budget, with the image prompt composer using its own smaller 8000-char budget - replace the fixed 3x over-fetch and noise blacklist with paginated scanning under a scan cap and a message_type chat allowlist so any amount of status/thinking/error/queue_cancel rows can no longer evict real conversation from the window - drop the prompt caching change from this PR per review since the rolling window invalidates the tail prefix once full, keeping only a trim of leading assistant messages which the Anthropic API rejects as the first message - update the concurrent-message test to assert exclusion and add coverage for the same-millisecond boundary, beyond-batch noise, the allowlist and the char budget
Second round of review fixes. - the char budget now covers the assembled request rather than history alone. The chat path subtracts the system prompt and trigger message from CLOUD_AGENT_MAX_CONTEXT_CHARS before budgeting history, and an overlong trigger is truncated with a warning, so the final payload char total is hard-bounded. The image composer subtracts its instruction and fixed system prompt the same way. Exact per-model token budgeting is not portable here since custom providers expose neither window sizes nor tokenizers, so chars remain the proxy and the config comment documents the CJK variance and advises lowering the knob for small models - the scan cap is now documented as deliberate best-effort truncation instead of implying strict semantics, and hitting it logs a warning naming the agent and channel so the condition is observable - add tests for the final provider payload staying within the total budget, trigger truncation, and the scan-cap-hit path asserting the warning and the documented empty result
Third round of review fixes. - when the configured system prompt alone reaches the whole-request budget, raise a clear ValueError instead of truncating the trigger to an empty string and silently dropping the turn. The caller's existing error handler turns the exception into a visible failure message in the channel. Regression test added - soften the scan-cap warning to say older history, if any, is invisible, since reaching the cap does not prove more rows exist - compute the image composer's fixed overhead from the actual system prompt length instead of a hardcoded 600 so future prompt wording changes cannot break the budget bound
Fourth round review fix. The image composer budget subtracted only the raw instruction, but the sent user message wraps it in fixed text worth 46 more chars, so a full history could push the request to 8046 chars. Build the complete user_prompt first, budget with its real length, and reuse the same string when sending. Adds a mock provider payload test asserting the composer request stays within _IMAGE_CONTEXT_MAX_CHARS.
Review follow-up. A single 10000-char history message now forces the builder to truncate to precisely the remaining budget, and the assertion pins the payload total to exactly the cap — the pre-fix implementation that missed the 46-char wrapper lands on 8046 and fails, whereas the previous 3000-char chunks left enough slack for it to pass.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cloud agents are stateless per turn and rebuilt context from only the latest
CLOUD_AGENT_MAX_CONTEXT_MESSAGESchannel messages (default 10, i.e. ~5 turnsof user+agent messages). In long conversations, constraints the user confirmed
early (e.g. agreed interface fields) fell out of the prompt entirely and the
model re-decided them from scratch — observed as the agent repeatedly
modifying already-confirmed interface fields around turn 15 of a multi-part
task. Three further defects compounded it: noise rows (
thinking/status/todos) were filtered after the SQLLIMITand consumed window slots; thetriggering message was excluded by blindly dropping the newest row, which
could evict real history under concurrent posts; and messages committed after
the trigger could leak into the request and get answered early.
Changes (
workspace/backend/app/services/cloud_agent.py,config.py)timestamp < trigger(SQL-level), so a message committed after the triggercan never leak into this request. Events carry no sub-millisecond ordering,
so rows sharing the trigger's millisecond are conservatively dropped;
id-based and positional exclusion remain as fallbacks when no timestamp is
available.
message_type == "chat"allowlist(error/queue_cancel/etc. excluded, not just known noise) with paginated
scanning until the window holds
CLOUD_AGENT_MAX_CONTEXT_MESSAGESchatmessages (default raised 10 → 100, env-overridable). The scan is explicitly
best-effort under a cap (10 batches) so a busy channel can't degrade into an
unbounded table scan; hitting the cap logs a warning naming agent and
channel.
CLOUD_AGENT_MAX_CONTEXT_CHARS(default60000, env-overridable) covers system prompt + history + trigger message, so
the assembled payload has a hard char bound — message count alone is not a
safe context budget for small/custom models. An overlong trigger is
truncated with a warning; a system prompt that alone consumes the budget
raises a clear error which the caller surfaces to the channel instead of
silently dropping the turn. Chars are a rough token proxy (CJK caveat
documented in config); exact per-model token budgeting isn't portable since
custom providers expose neither window sizes nor tokenizers.
computed from the actual system prompt and the full wrapped user message.
message, which the API rejects — leading non-user messages are trimmed in
_anthropic_chat.Prompt caching was deliberately excluded from this PR (a rolling window
invalidates the tail prefix once full); it will come as a follow-up with cache
usage observability and rollover tests.
Tests
tests/test_cloud_agent_context.py, 20 tests: trigger-boundary exclusion(incl. same-millisecond and id-only fallback), early-constraint survival at
turn 30, noise beyond a full over-fetch batch, allowlist, window cap, char
budget (newest-first loading, truncation), final provider payload bounded for
both chat and composer paths (composer pinned to the exact 8000 boundary),
system-prompt-over-budget raising, and the scan-cap warning path.
Full backend suite: the 20 pre-existing failures are identical with and
without this change (verified via stash diff) — no regressions.
Review history: five rounds of code review addressed across commits
6b5eb4b, 0128820, 2af86d6, afeb525, 1b7e7a3.
🤖 Generated with Claude Code