fix(provider): classify provider content-filter blocks as POLICY_REFUSAL - #779
Conversation
andreapn
left a comment
There was a problem hiding this comment.
Reviewed for issue #629. Correct, and the better of the two PRs on this issue. The five markers are the right set — dropping "flagged by content filter" as redundant with "content filter" is the correct call — and the negative tests are what make this reviewable: they pin that invalid_request_error, rate_limit_exceeded and context_length_exceeded are not swept into POLICY_REFUSAL by the widened marker list. Verified locally on top of origin/main: ruff, mypy, and all 106 provider-failure classification tests pass.
One blocker before merge:
The CHANGELOG entry lands in the released ## [2026.9.2] section, not ## [Unreleased]. This is a rebase artifact — 2026.9.2 was cut after this branch was opened, so the ### Fixed block your hunk anchors on has since moved into the released section.
Please rebase on main and move the entry under ## [Unreleased] → ### Fixed.
Nothing else needs to change.
5e4febb to
a28dc16
Compare
`_is_policy_refusal()` held only generic phrasing, so the wording that providers actually emit for a moderation block went unmatched and fell through to BAD_REQUEST or UNKNOWN: - Azure OpenAI's canonical message is "...triggering Azure OpenAI's content management policy" — the words "content" and "policy" are not adjacent, so the existing "content policy" marker never fired on it. - `content_filter` is the OpenAI/Azure error code and `finish_reason`, and matched nothing. - Gemini's block reason is "blocked by safety", which the existing "safety policy" does not cover. - `responsible_ai_policy` is Azure's policy-violation code. Since POLICY_REFUSAL and BAD_REQUEST map to different recovery actions, the misclassification sent real policy blocks down the wrong path. Per maintainer review on the issue, "flagged by content filter" is deliberately omitted: it can never fire independently of "content filter", and a substring list should not carry dead entries. Adds five markers plus regression tests — one per marker, and a set of guardrail cases asserting that plain bad requests, unsupported parameters, rate limits and context overflow are still classified as before. Fixes use-agent-os#629 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013D1Huj3pxLsXLCfwwEQNo1
a28dc16 to
e502f07
Compare
Fixes #629.
Problem
_is_policy_refusal()insrc/agentos/provider/failures.pyheld only generic phrasing, so the wording providers actually emit for a moderation block went unmatched and fell through toBAD_REQUESTorUNKNOWN. SincePOLICY_REFUSALandBAD_REQUESTmap to differentProviderRecoveryActions, real policy blocks were sent down the wrong recovery path.Change
Adds the five markers agreed in the issue review:
content management policycontent policymarker never fired on it.content_filterfinish_reason. Matched nothing.content filterresponsible_ai_policyblocked by safetysafety policydoes not cover it.flagged by content filterfrom the original proposal is deliberately omitted, per the maintainer's review on the issue: it can never fire independently ofcontent filter, and a substring list should not carry entries that are dead by construction. (This is the one difference from #630, which still carries it.)Tests
test_provider_specific_policy_markers_classify_as_policy_refusal— one case per new marker, using realistic provider payloads (Azure's full filtered-response sentence, Gemini's candidate-blocked message, the barecontent_filtercode).test_policy_markers_do_not_capture_unrelated_failures— guardrails asserting the wider matching does not swallow neighbouring cases: an unsupported parameter still classifies asUNSUPPORTED_FEATURE, a missing parameter asBAD_REQUEST, a 429 asRATE_LIMITED, and a context-length error asCONTEXT_OVERFLOW.All 5 marker tests fail on
mainand pass with the patch; the 4 guardrail tests pass both before and after, so they hold the blast radius rather than just restating the fix.Verification
pytest tests/test_provider_failure_classification.py tests/test_provider_failures.py→ 113 passeduv run --all-extras pytest tests/) → 8817 passed, 26 skipped, 3 failed. The 3 failures (test_mem0_provider,test_provider_config,test_render_html_to_pdf) are pre-existing onmainand unrelated.ruff checkclean;mypy src/agentos/provider/failures.pyclean.SKILL.md/docs/cli.mdneed no update.Note:
ruff format --checkreports pre-existing drift intests/test_provider_failure_classification.pyonmainitself. I let it reformat, then reverted the three unrelated hunks so this diff stays pure additions (+100, −0); the appended block itself is format-clean.CHANGELOG updated under
[Unreleased]→Fixed.🤖 Generated with Claude Code
https://claude.ai/code/session_013D1Huj3pxLsXLCfwwEQNo1