Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/completion/token_usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ model = "gpt-3.5-turbo"
print(get_max_tokens(model)) # Output: 4097
```

Note: `get_max_tokens` raises an `Exception` if the model is not in the model cost map, so wrap it in try/except when the model string is user-supplied.

### 8. `model_cost`

* Output: Returns a dict object containing the max_tokens, input_cost_per_token, output_cost_per_token for all models on [community-maintained list](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
Expand All @@ -155,6 +157,19 @@ from litellm import model_cost
print(model_cost) # {'gpt-3.5-turbo': {'max_tokens': 4000, 'input_cost_per_token': 1.5e-06, 'output_cost_per_token': 2e-06}, ...}
```

:::warning Pricing catalog, not an availability signal

A model appearing in `model_cost` does not mean your API key can call it. The map exists for cost tracking: it holds pricing data for every model LiteLLM can calculate costs for, including entries that are retired by the provider, not yet generally available, or gated to specific accounts; live keys can get 404s back for mapped models. To find out what your keys can actually call, for example to seed a model picker in your app, query the provider endpoints with [`get_valid_models(check_provider_endpoint=True)`](https://docs.litellm.ai/docs/set_keys#get_valid_modelscheck_provider_endpoint-true):

```python
from litellm import get_valid_models

# only models your configured keys can call right now
models = get_valid_models(check_provider_endpoint=True)
```

:::

### 9. `register_model`

* Input: Provide EITHER a model cost dictionary or a url to a hosted json blob
Expand Down
12 changes: 12 additions & 0 deletions docs/set_keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,8 @@ os.environ = old_environ

This helper will check the provider's endpoint for valid models.

Without `check_provider_endpoint=True`, the returned list comes from LiteLLM's static model map, which holds pricing data for every model LiteLLM knows about; it can include models that are retired, not yet generally available, or gated to specific accounts, so it is not an availability signal. With `check_provider_endpoint=True`, each provider's live model endpoint is queried instead, and the result is the set of models your keys can call right now. Use it when availability matters, for example to seed a model picker or validate a user-supplied model name.

Currently implemented for:
- OpenAI (if OPENAI_API_KEY is set)
- Fireworks AI (if FIREWORKS_AI_API_KEY is set)
Expand All @@ -210,6 +212,16 @@ valid_models = get_valid_models(check_provider_endpoint=True, custom_llm_provide
print(valid_models)
```

**Seed a model picker**:
```python
from litellm import get_valid_models

# only models the configured keys can call right now,
# unlike litellm.model_cost, which also holds pricing for retired / gated models
available = get_valid_models(check_provider_endpoint=True)
picker_options = sorted(available)
```

### `validate_environment(model: str)`

This helper tells you if you have all the required environment variables for a model, and if not - what's missing.
Expand Down