diff --git a/docs/completion/token_usage.md b/docs/completion/token_usage.md index d99564765..2eccecfe6 100644 --- a/docs/completion/token_usage.md +++ b/docs/completion/token_usage.md @@ -145,6 +145,8 @@ model = "gpt-3.5-turbo" print(get_max_tokens(model)) # Output: 4097 ``` +Note: `get_max_tokens` raises an `Exception` if the model is not in the model cost map, so wrap it in try/except when the model string is user-supplied. + ### 8. `model_cost` * Output: Returns a dict object containing the max_tokens, input_cost_per_token, output_cost_per_token for all models on [community-maintained list](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json) @@ -155,6 +157,19 @@ from litellm import model_cost print(model_cost) # {'gpt-3.5-turbo': {'max_tokens': 4000, 'input_cost_per_token': 1.5e-06, 'output_cost_per_token': 2e-06}, ...} ``` +:::warning Pricing catalog, not an availability signal + +A model appearing in `model_cost` does not mean your API key can call it. The map exists for cost tracking: it holds pricing data for every model LiteLLM can calculate costs for, including entries that are retired by the provider, not yet generally available, or gated to specific accounts; live keys can get 404s back for mapped models. To find out what your keys can actually call, for example to seed a model picker in your app, query the provider endpoints with [`get_valid_models(check_provider_endpoint=True)`](https://docs.litellm.ai/docs/set_keys#get_valid_modelscheck_provider_endpoint-true): + +```python +from litellm import get_valid_models + +# only models your configured keys can call right now +models = get_valid_models(check_provider_endpoint=True) +``` + +::: + ### 9. `register_model` * Input: Provide EITHER a model cost dictionary or a url to a hosted json blob diff --git a/docs/set_keys.md b/docs/set_keys.md index 295d9ec55..bbfb4b1e2 100644 --- a/docs/set_keys.md +++ b/docs/set_keys.md @@ -184,6 +184,8 @@ os.environ = old_environ This helper will check the provider's endpoint for valid models. +Without `check_provider_endpoint=True`, the returned list comes from LiteLLM's static model map, which holds pricing data for every model LiteLLM knows about; it can include models that are retired, not yet generally available, or gated to specific accounts, so it is not an availability signal. With `check_provider_endpoint=True`, each provider's live model endpoint is queried instead, and the result is the set of models your keys can call right now. Use it when availability matters, for example to seed a model picker or validate a user-supplied model name. + Currently implemented for: - OpenAI (if OPENAI_API_KEY is set) - Fireworks AI (if FIREWORKS_AI_API_KEY is set) @@ -210,6 +212,16 @@ valid_models = get_valid_models(check_provider_endpoint=True, custom_llm_provide print(valid_models) ``` +**Seed a model picker**: +```python +from litellm import get_valid_models + +# only models the configured keys can call right now, +# unlike litellm.model_cost, which also holds pricing for retired / gated models +available = get_valid_models(check_provider_endpoint=True) +picker_options = sorted(available) +``` + ### `validate_environment(model: str)` This helper tells you if you have all the required environment variables for a model, and if not - what's missing.