Staging (Week of 26-07-13) - #250
Merged
Merged
Conversation
…ps-error-envelope ENG-747: read status-error codes from the shape the SDK actually stores — revives the dead model-403 card
chore: back-merge main hotfix into staging
…ixes [ENG-688] Scheduled task fixes
…-inflates-scratchpad Fix: Optimize token usage for scratchpad tool
gdrive connector with google picker api
Sync main back into staging (weekly release cycle workflows)
The completion-verifier loop classified every tool-using turn that ended
by asking the user a question as INCOMPLETE and force-continued it, so the
agent answered its own questions (or fabricated) instead of waiting. Root
cause: the verdict vocabulary had no "waiting for the user" state — proven
in prod (verifier verdict literally read "the assistant is waiting for user
feedback ... has not been completed") and reproduced on a clean instance.
- Add a WAITING verdict: a turn ending in a genuine question to the user (or
a reasoned refusal) is a valid stop, not unfinished work.
- Move the verdict to the cheap coding model via generate_object_code with a
typed _VerifierVerdict over a compact request+reply view, instead of a
full-transcript call on the planning model parsed from free-text "STATUS:"
lines. Cheaper, and structurally reliable.
- Add verify_min_tool_rounds (default 1 = current behavior) to allow skipping
trivial single-tool-round turns once verdict logs confirm they're rarely
INCOMPLETE.
- Add non-disclosure clauses to the internal SYSTEM continuation/diagnosis
injections so they aren't echoed to the user ("the verifier note is noted").
- Log every verdict for measurability.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial self-review of the WAITING-verdict commit surfaced two issues:
- The reduced verify view was tool-blind: it showed only the user request +
the assistant's final text, so a turn where a tool errored but the assistant
claimed success would be judged COMPLETE — defeating the STUCK backstop the
ENG-296 fabrication mitigation relies on. Track a compact per-tool outcome
log (name=ok/error) for the turn and feed it to the verifier with explicit
guidance ("a depended-on tool that errored while the assistant implied
success is INCOMPLETE/STUCK, not COMPLETE"). Keeps the call cheap — outcomes,
not full tool output.
- The per-iteration `import logging` + getLogger ran on every verdict; hoisted
to one logger built once per turn before the verification loop.
Kept (intentional): on a verifier exception we fail-safe to COMPLETE rather
than retry — failing to INCOMPLETE would risk re-introducing the very
over-continuation this fix removes, and a transient stop is user-recoverable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t (ENG-716) The completion verifier now runs via generate_object_code (forced tool_choice on _VerifierVerdict), so the e2e stub's free-text "STATUS:" verification responses no longer matched — test_continuation_limit_respected regressed. - Stub: queue_verification_ok/incomplete/stuck now emit a structured _VerifierVerdict tool call; add queue_verification_waiting; teach _send_json to serialize tool_calls in non-streaming responses. - New test_waiting_verdict_stops_without_continuation: a tool-using turn that ends with a question + a WAITING verdict must stop with no "Continue working" injection, and the verifier must receive the tool-outcome cross-check log. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses both Major findings from code review: the reduced verify view dropped evidence needed to judge completion reliably. - Referential context: the verifier saw only the latest raw user message, so follow-ups like "now do the same for the other file" lost their task context. Now render a compact text view of the recent conversation (prior turns included). - Tool evidence: `name=ok/error` couldn't cross-check a *successful* tool claim (e.g. a fetch that returns 200 but empty). Now include truncated tool-result *content*, so a claimed-successful tool can be verified against what it actually returned. New `_render_verify_transcript` renders history as plain text (context + tool results, each truncated; internal SYSTEM injections omitted) — evidence- complete but far smaller than the raw transcript, and free of tool_use/tool_result pairing constraints. Drops the now-redundant per-tool ok/error tracking. Adds tests/test_verify_transcript.py covering context retention, tool-result evidence, SYSTEM omission, and truncation; updates the e2e WAITING test to assert tool-result evidence reaches the verifier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…review) Reviewing the evidence-complete view surfaced an edge case: the transcript is the last 16 messages, and a long tool-heavy turn (~7+ tool rounds = 2 messages each) can push the turn's opening user request out of that window — leaving the verifier to judge completion without knowing what was asked. Prepend the current request explicitly so it is always present regardless of turn length. The e2e WAITING test now asserts the request header reaches the verifier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…64 (PR #255 review) Addresses the follow-up review findings on _render_verify_transcript: - Multimodal role labeling (Major): list-based *user* content (image/file inputs) had its text blocks rendered as ASSISTANT. Speaker is now derived from the message role, so multimodal user requests/context are labelled USER. - Referential antecedent after long turns (Major): a flat last-16 window let a long tool loop evict prior conversational turns (and SYSTEM messages consumed slots before filtering). Now budget the conversation thread and tool activity separately (max_convo / max_tool), filtering SYSTEM before budgeting, so tool volume can't crowd out the turns needed to resolve referential requests. - Multimodal tool_result base64 (Minor): json.dumps materialized ~5MB of base64 then kept 400 chars of noise. New _render_tool_result_content renders block-by-block — bounded text kept, images as [image] placeholder, never raw encoded data. Adds unit coverage: >8-round tool turn preserves request + antecedent, multimodal user text labelled USER, multimodal tool_result keeps the text summary and drops base64, and SYSTEM injections don't consume the budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion (PR #255 review) Follow-up review found the conversation budget could still be evicted: assistant text emitted alongside a tool call ("Processing step 4" + tool_use, a normal response shape) was counted as a conversational turn, so a long tool loop's preambles filled max_convo and dropped the referential request and its antecedent. Route assistant text from any message that also contains a tool_use into the tool budget, reserving the conversation budget for real user turns and standalone replies. The long-turn regression test now emits preamble text on every round (proven to evict request + antecedent without this fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n-verifier-waiting-verdict ENG-716: add WAITING verdict to the completion verifier
hamishfagg
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Weekly release: staging → main
No releasable changes yet — staging's content currently matches main. Opened ahead of the week so it accumulates changes as they merge to staging; the Files changed tab is always current.
Review checklist