Skip to content

feat(provider): add provider-specific policy refusal markers - #630

Closed
bukeeastrey wants to merge 1 commit into
use-agent-os:mainfrom
bukeeastrey:feat/policy-refusal-markers
Closed

feat(provider): add provider-specific policy refusal markers#630
bukeeastrey wants to merge 1 commit into
use-agent-os:mainfrom
bukeeastrey:feat/policy-refusal-markers

Conversation

@bukeeastrey

Copy link
Copy Markdown
Contributor

Summary

When upstream LLM providers (such as Azure OpenAI, OpenAI, Gemini, or Anthropic) filter or reject a prompt or response due to content safety or moderation policies, they return specific error codes or message phrases (e.g. content_filter, responsible_ai_policy, content management policy, flagged by content filter, blocked by safety).

Previously, _is_policy_refusal() only matched a small set of generic phrases, causing these upstream safety blocks to fall through and be misclassified as BAD_REQUEST or UNKNOWN.

This PR adds common provider safety and moderation markers to _is_policy_refusal() to ensure all such errors consistently classify as ProviderFailureKind.POLICY_REFUSAL.

Changes

  • src/agentos/provider/failures.py: Added the following markers to _is_policy_refusal():
    • "content_filter" (Azure OpenAI / OpenAI error code & finish reason)
    • "content filter" (e.g. "The prompt was flagged by content filter")
    • "responsible_ai_policy" (Azure / cloud policy violation code)
    • "content management policy" (Azure OpenAI canonical error phrasing)
    • "flagged by content filter"
    • "blocked by safety" (Google Gemini block reason)
  • tests/test_provider_failure_classification.py: Added unit tests covering policy refusal error shapes across Azure OpenAI, OpenAI, Gemini, and Anthropic.

Verification

  • uv run ruff check src/agentos/provider/failures.py tests/test_provider_failure_classification.py (Passed)
  • uv run mypy src/agentos/provider/failures.py --show-error-codes (Passed)
  • uv run pytest tests/test_provider_failure_classification.py -q (93/93 tests passed)

closes #629

Add common content filter and safety markers to _is_policy_refusal() including 'content_filter', 'content filter', 'responsible_ai_policy', 'content management policy', 'flagged by content filter', and 'blocked by safety'. This ensures content moderation and safety policy refusals from Azure OpenAI, OpenAI, Gemini, and Anthropic are classified as POLICY_REFUSAL instead of falling through to BAD_REQUEST or UNKNOWN.
@andreapn

andreapn commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Closing as superseded. Reviewed alongside #779 on issue #629; #779 was picked.

Same markers, same place — the difference is the test coverage. #779 adds negative cases pinning that the widened list does not capture invalid_request_error, rate_limit_exceeded or context_length_exceeded, which is the actual risk when you loosen a substring matcher. It also drops "flagged by content filter" as redundant with "content filter".

This PR currently also conflicts with main on tests/test_provider_failure_classification.py.

Thanks for the contribution.

@andreapn andreapn closed this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Add common provider-specific safety and policy refusal markers to _is_policy_refusal

2 participants