Skip to content

fix(models): parse Claude 3-generation version ids - #108

Open
Joi Ito (Joi) wants to merge 1 commit into
microsoft:mainfrom
Joi:fix/detect-version-legacy-ids
Open

fix(models): parse Claude 3-generation version ids#108
Joi Ito (Joi) wants to merge 1 commit into
microsoft:mainfrom
Joi:fix/detect-version-legacy-ids

Conversation

@Joi

Copy link
Copy Markdown

Claude 3-generation model ids put the version before the family — claude-3-5-haiku-20241022. Every id since puts the family first — claude-haiku-4-5-20251001. _detect_version only knows the second shape.

For the first, the family word therefore sits directly against the snapshot date, and the major-only fallback rf"{family}-(\d+)(?:-|$)" matches the date:

>>> AnthropicProvider._detect_version("claude-3-5-haiku-20241022", "haiku")
(20241022, 0)
>>> AnthropicProvider._detect_version("claude-3-5-sonnet-20241022", "sonnet")
(20241022, 0)
>>> AnthropicProvider._detect_version("claude-3-7-sonnet-20250219", "sonnet")
(20250219, 0)
>>> AnthropicProvider._detect_version("claude-3-haiku-20240307", "haiku")
(20240307, 0)
>>> AnthropicProvider._detect_version("claude-3-opus-20240229", "opus")
(20240229, 0)

A major of 20241022 clears every >= gate in _get_capabilities, so these models are handed the newest capability tier.

The change: recognise the legacy shape ahead of the major-only fallback, and cap that fallback's major at two digits so no future id can have its date read as a version.

What it corrects. I diffed the full ModelCapabilities for each affected id before and after:

  • Haiku 3 and 3.5 — supports_thinking and supports_native_computer_use TrueFalse, thinking budget 320000. The capability matrix's own comment already says both are unsupported below 4.5.
  • Sonnet 3.5 / 3.7 and Opus 3 — supports_1m, supports_adaptive_thinking, supports_output_config, supports_task_budget, supports_native_computer_use TrueFalse; supported_efforts back to (low, medium, high); Opus 3 also loses speed. max_output_tokens 12800064000.

What it does not correct. Two things surfaced while checking the above. I left both alone rather than widen a parser fix into a capability-matrix change, but they are worth a follow-up:

  1. The 64000 output ceiling these models now land on is still too high — Claude 3-era ceilings are 4096–8192. Closer than the 128000 they had, not right.
  2. The opus and sonnet branches set supports_thinking=True unconditionally, so Opus 3 and Sonnet 3.5 still advertise extended thinking. This commit does not change that in either direction.

Happy to do either as a separate PR if you want them, though the real ceilings for the retired ids may need a call from someone who can still probe them.

Why the existing tests did not catch it: the capability tests pin behavior against claude-haiku-3-5-20250929, which is not a served model id and happens to match the family-first pattern, so it parses to (3, 5) and looks correct. The new tests use the real ids.

Family-first ids are unchanged: claude-opus-4-20250514(4, 0), claude-sonnet-4-5-20250929(4, 5), claude-opus-4-8(4, 8), claude-fable-5(5, 0).

Tests: 13 added — eight parse cases across both id shapes, five capability assertions covering the corrected legacy tiers and pinning current models. Full suite 753 passed, 7 skipped.

Claude 3-generation ids put the version before the family
(claude-3-5-haiku-20241022); every later id puts the family first
(claude-haiku-4-5-20251001). `_detect_version` only knew the second
shape, so for the first the family word sat directly against the
snapshot date and the major-only fallback read the date as the version:

    claude-3-5-haiku-20241022  -> (20241022, 0)   want (3, 5)
    claude-3-5-sonnet-20241022 -> (20241022, 0)   want (3, 5)
    claude-3-7-sonnet-20250219 -> (20250219, 0)   want (3, 7)
    claude-3-haiku-20240307    -> (20240307, 0)   want (3, 0)
    claude-3-opus-20240229     -> (20240229, 0)   want (3, 0)

A major of 20241022 clears every `>=` gate in `_get_capabilities`, so
these models were handed the newest tier. Adding the legacy shape ahead
of the major-only fallback gives the true versions. The fallback major is
also capped at two digits so no future id can have its date read as one.

What this corrects, measured by diffing the full ModelCapabilities for
each id before and after:

  - Haiku 3 / 3.5: supports_thinking and supports_native_computer_use
    True -> False, thinking budget 32000 -> 0. The capability matrix's
    own comment says both are unsupported below 4.5.
  - Sonnet 3.5 / 3.7, Opus 3: supports_1m, supports_adaptive_thinking,
    supports_output_config, supports_task_budget and
    supports_native_computer_use True -> False; supported_efforts back to
    (low, medium, high); Opus 3 also loses `speed`. max_output_tokens
    128000 -> 64000.

What it does NOT correct, and I have left alone as pre-existing rather
than widen a parser fix into a capability-matrix change:

  - The 64000 output ceiling these models now land on is still too high
    (Claude 3-era ceilings are 4096-8192). It is closer than the 128000
    they got before, not right.
  - The opus and sonnet branches set supports_thinking=True
    unconditionally, so Opus 3 and Sonnet 3.5 still claim extended
    thinking. Unchanged by this commit in either direction.

Family-first ids are unchanged: claude-opus-4-20250514 -> (4, 0),
claude-sonnet-4-5-20250929 -> (4, 5), claude-opus-4-8 -> (4, 8),
claude-fable-5 -> (5, 0).

Existing capability tests did not catch this because they pin behavior
against "claude-haiku-3-5-20250929", which is not a served model id and
happens to match the family-first pattern. The new tests use the real
ids.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant