Skip to content

fix(provider): classify provider content-filter blocks as POLICY_REFUSAL - #779

Merged
andreapn merged 1 commit into
use-agent-os:mainfrom
iamhaniofficial:fix/policy-refusal-markers-629
Sep 2, 2026
Merged

fix(provider): classify provider content-filter blocks as POLICY_REFUSAL#779
andreapn merged 1 commit into
use-agent-os:mainfrom
iamhaniofficial:fix/policy-refusal-markers-629

Conversation

@iamhaniofficial

Copy link
Copy Markdown
Contributor

Fixes #629.

Problem

_is_policy_refusal() in src/agentos/provider/failures.py held only generic phrasing, so the wording providers actually emit for a moderation block went unmatched and fell through to BAD_REQUEST or UNKNOWN. Since POLICY_REFUSAL and BAD_REQUEST map to different ProviderRecoveryActions, real policy blocks were sent down the wrong recovery path.

Change

Adds the five markers agreed in the issue review:

Marker Why the existing list missed it
content management policy Azure's canonical wording is "…triggering Azure OpenAI's content management policy". The words "content" and "policy" are not adjacent, so the existing content policy marker never fired on it.
content_filter The OpenAI/Azure error code and finish_reason. Matched nothing.
content filter The spaced form, e.g. "flagged by content filter".
responsible_ai_policy Azure's policy-violation code.
blocked by safety Gemini's block reason; the existing safety policy does not cover it.

flagged by content filter from the original proposal is deliberately omitted, per the maintainer's review on the issue: it can never fire independently of content filter, and a substring list should not carry entries that are dead by construction. (This is the one difference from #630, which still carries it.)

Tests

  • test_provider_specific_policy_markers_classify_as_policy_refusal — one case per new marker, using realistic provider payloads (Azure's full filtered-response sentence, Gemini's candidate-blocked message, the bare content_filter code).
  • test_policy_markers_do_not_capture_unrelated_failures — guardrails asserting the wider matching does not swallow neighbouring cases: an unsupported parameter still classifies as UNSUPPORTED_FEATURE, a missing parameter as BAD_REQUEST, a 429 as RATE_LIMITED, and a context-length error as CONTEXT_OVERFLOW.

All 5 marker tests fail on main and pass with the patch; the 4 guardrail tests pass both before and after, so they hold the blast radius rather than just restating the fix.

Verification

  • pytest tests/test_provider_failure_classification.py tests/test_provider_failures.py → 113 passed
  • Full suite (uv run --all-extras pytest tests/) → 8817 passed, 26 skipped, 3 failed. The 3 failures (test_mem0_provider, test_provider_config, test_render_html_to_pdf) are pre-existing on main and unrelated.
  • ruff check clean; mypy src/agentos/provider/failures.py clean.
  • No CLI surface, flags, config keys, ports or paths changed, so SKILL.md / docs/cli.md need no update.

Note: ruff format --check reports pre-existing drift in tests/test_provider_failure_classification.py on main itself. I let it reformat, then reverted the three unrelated hunks so this diff stays pure additions (+100, −0); the appended block itself is format-clean.

CHANGELOG updated under [Unreleased]Fixed.

🤖 Generated with Claude Code

https://claude.ai/code/session_013D1Huj3pxLsXLCfwwEQNo1

@andreapn andreapn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed for issue #629. Correct, and the better of the two PRs on this issue. The five markers are the right set — dropping "flagged by content filter" as redundant with "content filter" is the correct call — and the negative tests are what make this reviewable: they pin that invalid_request_error, rate_limit_exceeded and context_length_exceeded are not swept into POLICY_REFUSAL by the widened marker list. Verified locally on top of origin/main: ruff, mypy, and all 106 provider-failure classification tests pass.

One blocker before merge:

The CHANGELOG entry lands in the released ## [2026.9.2] section, not ## [Unreleased]. This is a rebase artifact — 2026.9.2 was cut after this branch was opened, so the ### Fixed block your hunk anchors on has since moved into the released section.

Please rebase on main and move the entry under ## [Unreleased]### Fixed.

Nothing else needs to change.

`_is_policy_refusal()` held only generic phrasing, so the wording that
providers actually emit for a moderation block went unmatched and fell
through to BAD_REQUEST or UNKNOWN:

- Azure OpenAI's canonical message is "...triggering Azure OpenAI's
  content management policy" — the words "content" and "policy" are not
  adjacent, so the existing "content policy" marker never fired on it.
- `content_filter` is the OpenAI/Azure error code and `finish_reason`,
  and matched nothing.
- Gemini's block reason is "blocked by safety", which the existing
  "safety policy" does not cover.
- `responsible_ai_policy` is Azure's policy-violation code.

Since POLICY_REFUSAL and BAD_REQUEST map to different recovery actions,
the misclassification sent real policy blocks down the wrong path.

Per maintainer review on the issue, "flagged by content filter" is
deliberately omitted: it can never fire independently of "content
filter", and a substring list should not carry dead entries.

Adds five markers plus regression tests — one per marker, and a set of
guardrail cases asserting that plain bad requests, unsupported
parameters, rate limits and context overflow are still classified as
before.

Fixes use-agent-os#629

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013D1Huj3pxLsXLCfwwEQNo1
@iamhaniofficial
iamhaniofficial force-pushed the fix/policy-refusal-markers-629 branch from a28dc16 to e502f07 Compare September 2, 2026 11:32

@andreapn andreapn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andreapn
andreapn merged commit 8c36800 into use-agent-os:main Sep 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Add common provider-specific safety and policy refusal markers to _is_policy_refusal

2 participants