fix(cost): key Haiku 3.5 by its served model id - #107
Open
Joi Ito (Joi) wants to merge 1 commit into
Open
Conversation
`_RATES` keyed Haiku 3.5 as "claude-haiku-3-5-20250929". No model was
ever served under that string: Claude 3-generation ids put the version
before the family (claude-3-5-haiku-...), and the date belongs to Sonnet
4.5. `compute_cost` does an exact `_RATES.get()`, so the served id
missed:
>>> compute_cost("claude-3-5-haiku-20241022", input_tokens=1_000_000)
None
>>> compute_cost("claude-haiku-3-5-20250929", input_tokens=1_000_000)
Decimal('0.80')
None is the unknown-model signal, so Haiku 3.5 spend was silently
untracked rather than misreported.
Rates are unchanged ($0.80 / $4.00 / $0.08 / $1.00); only the key moves.
The old string is not kept as an alias -- nothing can be reporting a
model id that was never served.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_RATESkeys Haiku 3.5 underclaude-haiku-3-5-20250929. That is not a model id the API serves — the word order is inverted (real ids in that generation are family-first,claude-3-5-haiku-...) and the date is Sonnet 4.5's.compute_costdoes an exact_RATES.get(), so the real id misses:Noneis the unknown-model signal, so anywherecost_usdis stamped fromcompute_cost, Haiku 3.5 spend is silently untracked rather than wrong — no error, no warning, just a gap in the cost data.The change: rename the key to
claude-3-5-haiku-20241022. Rates are untouched ($0.80 / $4.00 / $0.08 / $1.00).The old string is not kept as an alias. No model was ever served under it, so nothing upstream or downstream can be reporting it. I checked the rest of the tree:
claude-haiku-3-5-20250929appears in several test files, but only as a_get_capabilitiesfixture, which does not consult_RATES— those tests are unaffected. (They are pinning capability behavior against a model id that does not exist either, which is a separate thing and left alone here.)Tests: three cases in
tests/test_cost.py— the real id prices at all four rates, and the former key returnsNone.Full suite: 743 passed, 7 skipped.