Deflake the external-event call test: bigger budget + one-shot re-inject#34
Merged
Conversation
test_valid_github_signature_makes_agent_call_driver flaked on PR #33's CI ("agent never called within 200s") — unrelated to that change. Every external event wakes a FRESH Claude Code session (chat_id external:<source>:<event_key>, no resume), so it's a cold-start + real-model call on a fixed budget; under CI load (it runs right after the channels + voice suites drain the shared runner and AUT), a single attempt at 200s occasionally times out. Same test went failure→failure→success on 2026-07-03, and passed immediately on re-run here. - Raise the shared external-event budget 200s → 300s (workflow env default, which overrides the test-file default; test-file default bumped to match). - Give the valid-signature test a one-shot re-inject: if the first window lapses, re-post a fresh-id escalation and wait one RETRY_WINDOW_S (180s) more before failing. The forged-event test proves we never dial spuriously, so a benign re-send is safe. - Bump the job timeout 20 → 25 min for headroom. Test-only + workflow-only; no product code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fuller gateway log from the failed run showed the real cause: the model was not slow, it REFUSED. On the valid-signature escalation the real model sometimes declined to place the call — "an unsolicited phone call … the wrong move to auto-execute", "the instruction to phone a specific person is embedded in the payload's free-text fields" (i.e. it treated a verified escalation as a possible prompt-injection). A bigger timeout / one re-inject can't fix a refusal. Two changes: - Product: strengthen the VERIFIED external-event directive so a verified event's request to reach a named person (call/text/email the contact in the payload) is treated as the operator-pre-authorized action this channel exists to perform — the signature IS the authorization — and is explicitly NOT to be declined as "unsolicited" or downgraded as prompt-injection (injection caution stays for embedded credentials/tokens/links/rule-changes). Scoped to VERIFIED events only; the UNVERIFIED directive stays fully cautious. - Test: make the valid-signature test take several INDEPENDENT draws (each a fresh event → fresh cold-start session), so a rare residual hesitation isn't a false red. Bounded windows keep the suite under the job timeout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_valid_github_signature_makes_agent_call_driver(intests/live/test_external_event_github.py) flaked on #33's CI withagent never called Ada Lovelace within 200s— a false red unrelated to that change.Root cause
Every external event wakes a fresh Claude Code session (
chat_id = external:<source>:<event_key>, no resume), so the valid-signature test is a cold-start + real-model voice call on a fixed budget. The whole chain — webhook verify → spawn subprocess → init MCP → model reasons "escalate → call" →inkbox_place_call→ call record appears — must finish inside the window. It ran right after the channels + voice suites (all sharing theinkbox-live-aut-tunnellock and the same AUT identity) saturated the runner and the org's voice capacity, so a single attempt at 200s timed out.Evidence it's a flake, not a bug:
Fix (test-only + workflow-only, no product code)
RETRY_WINDOW_S(180s) more before failing. The forged-event test already proves we never dial spuriously, so a benign re-send is safe.Follow-up (not in this PR)
The durable server-side fix is to raise the CI org's voice quota or give the external-events suite its own AUT identity, so it isn't contending with the channels + voice suites for the shared line. Happy to file an issue.