HOTFIX ENG-1081: give the completion verifier room to answer (silent task death, 25% of verdicts)

[alecantu7]

Hotfix: ENG-1081 — the completion verifier's 256-token ceiling

This is #277 retargeted at main as a hotfix. #277 is approved (@pnewsam) and stays open until this lands — see the plan at the bottom.

Provenance (7 commits). Six are the reviewed #277 commits, cherry-picked onto origin/main with zero conflicts. The seventh, bee97f0, hardens _safe_error_detail so it cannot raise from inside the verifier's except handler — found while assessing hotfix risk, after #277 was approved. It was applied to #277's branch as well (9b7906b), so the two branches are byte-identical: verified per file with git hash-object across session.py, client.py, provider.py, structured.py, scratchpad_boot.py, test_verifier_truncation.py, stub_server.py and test_loop_safety.py. Reviewing either branch reviews the same code.

Why it can't wait for next Monday

v2.26.7.27.1 shipped today at 11:34 UTC without it. Measured in prod Langfuse over 2026-07-14 → 07-27:

• 171 of 687 completion-verifier calls (24.9%; 37.1% for external users) return no usable verdict, land on the fail-safe as a fabricated COMPLETE, and end the turn with no message at all.
• 145/147 on mindshub_air (98.6%) — for those users it is not intermittent, it is every turn. Whole sessions where it never once worked: 15/15, 12/12, 8/8.
• 33 users, 51 sessions in 10 days. Users experience it as "Anton dies on the first hurdle", and restarting doesn't help because their model setting is unchanged.

Root cause is the verdict call's max_tokens=256 versus models that narrate ~1,200–1,300 characters before the forced tool call — the preamble alone exceeds the whole budget, so the call could never fit.

Why the diff is safe to put straight on main

• Every file it touches is byte-identical between main and staging (session.py, client.py, provider.py, structured.py, scratchpad_boot.py, conftest.py, stub_server.py). The two branches differ in exactly one unrelated file (prompts.py, ENG-838).
• Cherry-pick applied with zero conflicts, and the resulting files are hash-identical to the approved ENG-1081: give the completion verifier room to answer (fixes the silent-stop root cause) #277 branch — same change, byte for byte, not a re-implementation.
• 1334 unit tests pass, 0 failures on this base (the staging base has 2 pre-existing scratchpad-exec env failures; they don't reproduce here).
• All 37 e2e scenarios pass, including the ENG-716 WAITING test and the new truncation-retry scenario.
• Validated live against all 19 advertised MindsHub models: 12/17 reachable produce a valid verdict today → 16/17 with this fix. No model regresses.
• Contained: no new endpoints, no auth, no deserialization, no schema/migration. It raises one token budget, adds one prompt sentence, and adds a retry on a classified failure.

Merge plan (team hotfix convention — Option B)

1. Merge this → release.yml auto-tags and publishes (expect v2.26.7.27.2).
2. Merge main back into staging with a sync PR. Note staging-unfreeze.yml is gated on head.ref == 'staging', so the usual automatic sync-back will not fire for this PR — it needs doing by hand.
3. Close ENG-1081: give the completion verifier room to answer (fixes the silent-stop root cause) #277 as superseded once staging has the content.
4. #276 (ENG-1079) then rebases onto staging-with-this-fix and ships on the normal weekly release. That preserves the required order — ceiling fix first, honest-diagnosis fail-safe second — which matters because on mindshub_air fix(session): completion verifier fails toward a real message, not silence (ENG-1079) #276 alone would show "an internal check failed, how would you like to proceed?" on ~99% of turns.

Delivery note: the PyPI release is not the delivery path for either surface. Desktop needs the cowork-server anton-agent bump; hub agents need a snapshot rebuild.

Full detail

Evidence, the measured token distribution behind 2048, the review trail (4 adversarial passes, @pnewsam's privacy finding), and the model matrix are all on #277 — that discussion is the review of record for this change.

Closes ENG-1081.

[pnewsam]

Code Review

Verdict: COMMENT — merge-safe with one non-blocking documentation correction

I reviewed the complete main...bee97f0 diff and compared it with the approved #277 implementation. The six cherry-picked changes retain the reviewed behavior; the seventh commit safely hardens _safe_error_detail so hostile exception properties cannot escape from inside the verifier's error handler. I found no blocking or major code defects.

Summary of findings

Severity	Area	Issue
Minor	PR description	The description says this is the same six commits and hash-identical to #277, but the current branch has a seventh hotfix-only commit (bee97f0) and therefore differs in anton/core/session.py and tests/test_verifier_truncation.py. Please update the provenance wording so the release audit trail matches the code being merged.

Risk assessment

Overall code risk is Medium and acceptable for a hotfix:

• The change touches shared structured-output code in the LLM client, session, and scratchpad paths, but preserves catcher compatibility through a ValueError subclass and has focused unit plus subprocess-level coverage.
• The main operational risk is output cost/latency: verifier capacity rises from 256 tokens to 2,048, with one 4,096-token retry only after classified truncation. Measured successful calls stay well below those ceilings, but the theoretical double-truncation path can consume 6,144 output tokens.
• There are no migrations, persistent-data changes, dependency changes, auth changes, new endpoints, or deployment-configuration changes. A code rollback is straightforward.
• All required CI, CodeQL, and CLA checks are green, and GitHub reports a clean merge state.

Release recommendations

• Assign owners for the cowork-server anton-agent bump and hub snapshot rebuild; publishing PyPI alone does not deliver this fix to the affected surfaces.
• Create the manual main → staging sync immediately after release, since this hotfix branch does not trigger the normal staging unfreeze/sync path.
• Monitor verifier TRUNCATED, retry, token-usage, and latency rates after rollout; revert if retry volume or cost materially exceeds the measured distribution.

What's working well

The retry is bounded and evidence-based, truncation is classified across provider dialects and mid-argument tool calls, sync/async paths share the classifier, sensitive exception content stays out of ordinary logs, and regression coverage exercises provider → client → session behavior end to end.

---

Reviewed by Codex · alejandrocantu/eng-1081-hotfix-verifier-budget → main · 2026-07-27

[alecantu7]

Thanks — the provenance wording was stale and is now corrected in the description.

Detail worth recording for the audit trail: the seventh commit bee97f0 was also applied to #277's branch (as 9b7906b), so the two branches have converged rather than diverged. Verified per file with git hash-object — session.py, client.py, provider.py, structured.py, scratchpad_boot.py, test_verifier_truncation.py, stub_server.py, test_loop_safety.py all identical. Both branches now carry 7 commits. Your inference was correct at the time you reviewed; the pushes crossed.

On the 6,144-token double-truncation path you flagged: that is a deliberate tradeoff, and it replaced something worse. An earlier revision latched the retry off per session after one double-truncation; I removed it because output length is stochastic — 16 identical calls spanned 245–1654 tokens, a 6.7x spread — so one truncation is a tail sample, not proof about the next turn, and latching on that evidence would have reintroduced silent stops for a model that usually succeeds. Measured: 0 of 12 calls truncate at 2048, and the worst case across all 19 advertised models is 1054 tokens, so the retry should be rare and the 6,144 path rarer still. Both are visible in the logs (verdict=TRUNCATED … retrying=true), which is exactly what your monitoring recommendation keys on.

Agreed on all three release recommendations — the delivery owners, the immediate manual main → staging sync (the unfreeze path will not fire for this branch), and post-rollout monitoring of TRUNCATED/retry/token/latency rates. Relaying those to @alecantu7 as the merge checklist.