Skip to content

fix: clamp unsupported reasoning effort to model's highest supported tier - #71

Open
Michael J. Jabbour (michaeljabbour) wants to merge 1 commit into
mainfrom
fix/clamp-reasoning-effort
Open

fix: clamp unsupported reasoning effort to model's highest supported tier#71
Michael J. Jabbour (michaeljabbour) wants to merge 1 commit into
mainfrom
fix/clamp-reasoning-effort

Conversation

@michaeljabbour

Copy link
Copy Markdown

Addresses microsoft-amplifier/amplifier-support#289 — a follow-up to the
shipped config-level effort knob (#61/#64). The issue may stay open for
other aspects of the per-model-capability-lag family (#304, #299), so this
PR addresses the clamp specifically rather than closing the issue outright.

Root cause

config["effort"] / request.reasoning_effort is validated against a
global legal list (low/medium/high/xhigh/max) at config-resolution
time, but is applied against the active model's ModelCapabilities. supported_efforts at request-build time. Those two checks can legitimately
disagree: provider config is intentionally model-agnostic (one config block
can serve many models via a routing matrix or mid-session model switch), so
"max" can pass the global check yet not exist in, say, claude-sonnet-5's
capability tuple (("low", "medium", "high", "xhigh") — no "max" until
Opus territory).

Before this fix, that mismatch hit a hard membership test with only one
fallback: warn and omit output_config.effort entirely. That meant:

  • A WARNING log line on every single request routed to the
    under-capable model (the field-reported symptom in #289).
  • The API received no output_config.effort at all, so it applied its own
    server-side default effort — silently downgrading the user's requested
    "max" intent to whatever the API defaults to, with no signal in the
    response that this happened.

What this PR does

  • Introduces EFFORT_ORDER, a single canonical module-level ladder
    (low < medium < high < xhigh < max), and reuses it at the effort
    ConfigField's choices and the config-level effort validation —
    previously two more hand-maintained literal copies of the same five
    strings, which is exactly the kind of drift that let this gap open up in
    the first place.
  • Adds _clamp_effort_to_supported(): on a mismatch, walks the ladder down
    from the requested rank and returns the highest tier the model actually
    supports
    (e.g. "max""xhigh" on claude-sonnet-5), instead of
    omitting output_config.effort. A genuinely unrecognized value (not on the
    ladder at all — a typo like "ultra") still falls back to the original
    warn-and-omit behavior; clamping only applies to values the provider
    understands but the active model doesn't support yet.
  • Demotes the downgrade notice from a per-request WARNING to a one-time
    INFO log per (model, requested-effort) pair, tracked in a module-level
    seen-set that mirrors the existing _warned_deprecated_models pattern
    already used for deprecated-model warnings in this file.
  • Leaves the happy path completely unchanged: models that already support
    the requested effort pass it through untouched (claude-fable-5 /
    claude-opus-4-8 + "max""max").

Tests (TDD)

Written first (watched red), then the fix was implemented to turn them
green. Extended tests/test_reasoning_effort.py with:

  • TestEffortClampToSupportedTier (6 cases): the claude-sonnet-5
    "max""xhigh" headline clamp; claude-fable-5 / claude-opus-4-8
    "max" pass-through unchanged; a model without output_config support at
    all still never gains the key; a genuinely unknown effort string is
    omitted+warned, not clamped; kwargs["effort"] precedence is also
    clamped (not just request.reasoning_effort).
  • TestEffortDowngradeLoggedOnce (2 cases, via caplog): two identical
    requests log the downgrade notice exactly once; a different
    (model, effort) pair logs again.

Updated an existing test to the new contract (per the task's request to
check for this): tests/test_opus_47.py::test_opus_47_invalid_effort_omits_ output_config asserted the old warn-and-omit behavior for
reasoning_effort="max" on Opus 4.7 (which lacks "max" but has
"xhigh"). Replaced with test_opus_47_max_effort_clamps_to_xhigh
(asserts the new clamp-to-"xhigh" behavior) and added a new
test_opus_47_unknown_effort_string_omits_output_config to keep the
genuinely-unrecognized-value coverage that the old test's docstring implied
but didn't actually test (it used "max", a recognized value, not an
unrecognized one).

Full suite: 535 passed, plus 3 pre-existing failures in
tests/test_tool_repair.py
(streaming MockStreamManager.__aiter__
mismatch) confirmed present on a clean origin/main checkout before this
change and unaffected by it — unrelated to this fix, left untouched.

Also in this PR

  • pyproject.toml/uv.lock: added amplifier-core as a dev-only
    dependency. The test suite imports amplifier_core directly
    (ModuleCoordinator, ChatRequest, message models) but it wasn't
    declared anywhere in the dependency graph, so uv sync && uv run pytest
    could not even collect the test suite from a clean clone. No runtime
    dependency changed.
  • README.md: updated the "Reasoning Effort" notes, which described the
    old omit-with-warning behavior, to describe the new clamp behavior.

Verification (offline payload assertions, no live API)

Ran the actual mocked-SDK payload through the provider and inspected
params["output_config"] on the call captured by the mock:

HEADLINE (sonnet-5 max->xhigh clamp): model='claude-sonnet-5' requested_effort='max' -> output_config={'effort': 'xhigh'}
PASS-THROUGH (fable-5 max->max): model='claude-fable-5' requested_effort='max' -> output_config={'effort': 'max'}
PASS-THROUGH (opus-4.8 max->max): model='claude-opus-4-8-20260101' requested_effort='max' -> output_config={'effort': 'max'}
NO SUPPORT (sonnet-4.6, no output_config key): model='claude-sonnet-4-6' requested_effort='high' -> output_config=None
UNKNOWN EFFORT (not clamped, omitted+warned): model='claude-sonnet-5' requested_effort='ultra' -> output_config=None
  [PROVIDER] Effort level 'ultra' not supported by claude-sonnet-5 (supported: ('low', 'medium', 'high', 'xhigh')) — omitting output_config.effort

Clamp-notice log lines seen across 2 identical calls: 1 (expect 1)
  [PROVIDER] Clamping effort 'max' to 'xhigh' for claude-sonnet-5 (model maximum; supported: ('low', 'medium', 'high', 'xhigh'))

Relationship to the broader hardening set

This PR is part of the cross-provider resume hardening set tracked on
microsoft-amplifier/amplifier-support#208. Sibling PRs: thinking-block
sanitization for #207 in this same repo, and a producer fix for #206 in
amplifier-module-provider-chat-completions.

…tier

Addresses microsoft-amplifier/amplifier-support#289: with a provider-level
`effort` (or request.reasoning_effort) set to a value that passes GLOBAL
validation (e.g. "max") but isn't declared in the ACTIVE model's
ModelCapabilities.supported_efforts (e.g. claude-sonnet-5, which tops out at
"xhigh"), the provider warned on every single request and omitted
output_config.effort entirely -- letting the API apply its own server-side
default effort instead of the model's actual ceiling. User's "max effort"
intent was silently downgraded to "default" with no way to tell from the
response.

Root cause: two independent validation passes with no way to reconcile a
mismatch. Config/request effort is checked against a global legal list
(low/medium/high/xhigh/max); the per-model gate at request-build time only
ever did a hard membership test against that model's OWN supported_efforts,
with warn-and-omit as its only fallback for a legal-but-unsupported value.

Fix:
- Introduce EFFORT_ORDER, a canonical module-level ladder, and reuse it at
  the "effort" ConfigField's choices and the config-level `effort`
  validation (previously two more hand-maintained copies of the same five
  strings -- exactly the kind of drift that let this gap open up).
- Add _clamp_effort_to_supported(): walk the ladder down from the requested
  rank and return the highest tier the model actually supports (e.g.
  "max" -> "xhigh" on claude-sonnet-5).
- On a mismatch, clamp instead of omit. Genuinely unrecognized values (not
  on the ladder at all, e.g. a typo) still fall back to the original
  warn-and-omit behavior -- clamping only applies to values the provider
  understands but the active model doesn't support yet.
- Demote the downgrade notice from a per-request WARNING to a one-time
  (per model+requested-effort pair) INFO log, tracked in a module-level
  seen-set mirroring the existing _warned_deprecated_models pattern.
- Models that support the requested effort are unaffected: the happy-path
  pass-through branch is untouched (fable-5/opus-4.8 + "max" -> "max").

Tests (TDD -- written first, watched fail, then implemented the fix):
- tests/test_reasoning_effort.py: new TestEffortClampToSupportedTier (6
  cases: sonnet-5 max->xhigh headline clamp, fable-5/opus-4.8 max
  pass-through, no-output_config-support model still omits the key, unknown
  effort string omitted+warned not clamped, kwargs["effort"] precedence is
  also clamped) and TestEffortDowngradeLoggedOnce (2 cases: repeated
  identical requests log once; a different model/effort pair logs again).
- tests/test_opus_47.py: test_opus_47_invalid_effort_omits_output_config
  asserted the OLD warn-and-omit contract for reasoning_effort="max" on
  Opus 4.7 (which lacks "max" but has "xhigh"). Replaced with
  test_opus_47_max_effort_clamps_to_xhigh (asserts the new clamp-to-xhigh
  behavior) and added test_opus_47_unknown_effort_string_omits_output_config
  to keep the genuinely-unrecognized-value coverage that test used to imply.

Also:
- pyproject.toml/uv.lock: added amplifier-core as a dev dependency. The
  test suite imports amplifier_core directly (ModuleCoordinator,
  ChatRequest, message models) but it was not declared anywhere in the
  dependency graph, so `uv sync && uv run pytest` could not collect the
  test suite from a clean clone. Dev-only; no runtime dependency change.
- README.md: updated the "Reasoning Effort" notes to describe the clamp
  behavior instead of the old omit-with-warning behavior it documented.

Pre-existing, unrelated: tests/test_tool_repair.py has 3 failing streaming
tests (MockStreamManager __aiter__ mismatch) on a clean origin/main
checkout, confirmed unaffected by this change (same 3 failures before and
after).
@michaeljabbour

Copy link
Copy Markdown
Author

Cross-provider resume hardening set — complete PR index

This PR is the effort-clamp fix in the set tracked on microsoft-amplifier/amplifier-support#208. Full set for reviewers:

PR Repo Role
provider-anthropic#72 provider-anthropic Option B — consumer-side sanitization of invalid thinking blocks — Fixes support#207
provider-chat-completions#12 provider-chat-completions Producer fix — stop fabricating signature: null thinking blocks — Fixes support#206
provider-anthropic#71 (this PR) provider-anthropic Effort clamp (max→highest supported tier, e.g. xhigh on sonnet-5) — Addresses support#289
amplifier-app-cli#232 amplifier-app-cli Option A — resume-time provider/model mismatch warning + confirm, plus provider persisted in session metadata — Addresses support#208

Design rationale and deferred items: support#208 comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant