fix(models): parse Claude 3-generation version ids - #108
Open
Joi Ito (Joi) wants to merge 1 commit into
Open
Conversation
Claude 3-generation ids put the version before the family
(claude-3-5-haiku-20241022); every later id puts the family first
(claude-haiku-4-5-20251001). `_detect_version` only knew the second
shape, so for the first the family word sat directly against the
snapshot date and the major-only fallback read the date as the version:
claude-3-5-haiku-20241022 -> (20241022, 0) want (3, 5)
claude-3-5-sonnet-20241022 -> (20241022, 0) want (3, 5)
claude-3-7-sonnet-20250219 -> (20250219, 0) want (3, 7)
claude-3-haiku-20240307 -> (20240307, 0) want (3, 0)
claude-3-opus-20240229 -> (20240229, 0) want (3, 0)
A major of 20241022 clears every `>=` gate in `_get_capabilities`, so
these models were handed the newest tier. Adding the legacy shape ahead
of the major-only fallback gives the true versions. The fallback major is
also capped at two digits so no future id can have its date read as one.
What this corrects, measured by diffing the full ModelCapabilities for
each id before and after:
- Haiku 3 / 3.5: supports_thinking and supports_native_computer_use
True -> False, thinking budget 32000 -> 0. The capability matrix's
own comment says both are unsupported below 4.5.
- Sonnet 3.5 / 3.7, Opus 3: supports_1m, supports_adaptive_thinking,
supports_output_config, supports_task_budget and
supports_native_computer_use True -> False; supported_efforts back to
(low, medium, high); Opus 3 also loses `speed`. max_output_tokens
128000 -> 64000.
What it does NOT correct, and I have left alone as pre-existing rather
than widen a parser fix into a capability-matrix change:
- The 64000 output ceiling these models now land on is still too high
(Claude 3-era ceilings are 4096-8192). It is closer than the 128000
they got before, not right.
- The opus and sonnet branches set supports_thinking=True
unconditionally, so Opus 3 and Sonnet 3.5 still claim extended
thinking. Unchanged by this commit in either direction.
Family-first ids are unchanged: claude-opus-4-20250514 -> (4, 0),
claude-sonnet-4-5-20250929 -> (4, 5), claude-opus-4-8 -> (4, 8),
claude-fable-5 -> (5, 0).
Existing capability tests did not catch this because they pin behavior
against "claude-haiku-3-5-20250929", which is not a served model id and
happens to match the family-first pattern. The new tests use the real
ids.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude 3-generation model ids put the version before the family —
claude-3-5-haiku-20241022. Every id since puts the family first —claude-haiku-4-5-20251001._detect_versiononly knows the second shape.For the first, the family word therefore sits directly against the snapshot date, and the major-only fallback
rf"{family}-(\d+)(?:-|$)"matches the date:A major of 20241022 clears every
>=gate in_get_capabilities, so these models are handed the newest capability tier.The change: recognise the legacy shape ahead of the major-only fallback, and cap that fallback's major at two digits so no future id can have its date read as a version.
What it corrects. I diffed the full
ModelCapabilitiesfor each affected id before and after:supports_thinkingandsupports_native_computer_useTrue→False, thinking budget32000→0. The capability matrix's own comment already says both are unsupported below 4.5.supports_1m,supports_adaptive_thinking,supports_output_config,supports_task_budget,supports_native_computer_useTrue→False;supported_effortsback to(low, medium, high); Opus 3 also losesspeed.max_output_tokens128000→64000.What it does not correct. Two things surfaced while checking the above. I left both alone rather than widen a parser fix into a capability-matrix change, but they are worth a follow-up:
64000output ceiling these models now land on is still too high — Claude 3-era ceilings are 4096–8192. Closer than the128000they had, not right.supports_thinking=Trueunconditionally, so Opus 3 and Sonnet 3.5 still advertise extended thinking. This commit does not change that in either direction.Happy to do either as a separate PR if you want them, though the real ceilings for the retired ids may need a call from someone who can still probe them.
Why the existing tests did not catch it: the capability tests pin behavior against
claude-haiku-3-5-20250929, which is not a served model id and happens to match the family-first pattern, so it parses to(3, 5)and looks correct. The new tests use the real ids.Family-first ids are unchanged:
claude-opus-4-20250514→(4, 0),claude-sonnet-4-5-20250929→(4, 5),claude-opus-4-8→(4, 8),claude-fable-5→(5, 0).Tests: 13 added — eight parse cases across both id shapes, five capability assertions covering the corrected legacy tiers and pinning current models. Full suite 753 passed, 7 skipped.