Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .prettierignore
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
available-hardware.mdx
available-hardware.mdx

# Vendored byte-for-byte from skills/cerebrium/SKILL.md in CerebriumAI/cerebrium-skills.
# A checker there fails that repo's CI on any drift, and Prettier re-wraps this file
# if allowed to touch it. Edit the skill upstream, never here.
/skill.md
6 changes: 6 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@
- title: Clear, descriptive page title
- description: Concise summary for SEO/navigation

## Vendored skill.md

- Root `skill.md` is vendored byte-for-byte from `skills/cerebrium/SKILL.md` in CerebriumAI/cerebrium-skills
- A checker in that repository fails when the two copies drift, so edit the skill upstream, never here
- The Voice and tone and Writing standards rules below do not apply to `skill.md`'s body. Never rewrite, reword, reflow, or re-wrap it in this repository

## Voice and tone

- Direct, matter-of-fact tone — write reference material, not a tutorial blog post
Expand Down
160 changes: 160 additions & 0 deletions skill.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
---
name: Cerebrium
description: >-
Use for any Cerebrium task: deploying Python code to serverless GPU or CPU, writing or fixing
cerebrium.toml, choosing hardware and regions, calling deployed endpoints (REST, streaming,
WebSocket, async), autoscaling and concurrency, cold starts, secrets, CI/CD, and debugging a
build or a running app from the terminal. Covers the cerebrium CLI, configuration defaults the
API actually applies, accepted GPU identifiers with per-plan limits, and troubleshooting.
license: MIT
metadata:
mintlify-proj: cerebrium
version: "1.0"
---

# Cerebrium

Cerebrium runs Python workloads on serverless GPU and CPU: REST endpoints, SSE streaming,
WebSockets, and async jobs, with scale-to-zero and per-second billing. One `cerebrium.toml`
describes hardware, scaling, dependencies and runtime; one CLI (`cerebrium`) drives everything.

Reach for it when the workload is an inference API for an LLM, an embedding model or a vision
model; a real-time voice, video or streaming app; bursty traffic that should scale from zero
without holding idle GPUs; a deployment that has to run in several regions for latency or data
residency; or a migration off Replicate, Hugging Face or Mystic, each of which has a guide under
`https://cerebrium.ai/docs/migrations`.

This file carries the workflow and the rules. Load the reference that matches the task:

| Read | When |
| --- | --- |
| [`references/cli.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/cli.md) | Running any `cerebrium` command: the full surface, flags, non-interactive auth, CI/CD, which commands cost money. |
| [`references/config.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/config.md) | Writing or fixing `cerebrium.toml`: every key, the default the API applies when it is omitted, accepted ranges, rebuild triggers. |
| [`references/hardware.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/hardware.md) | Choosing `compute`, `gpu_count`, `region`, `provider`, `compute_tier`: accepted GPU identifiers, per-GPU and per-plan limits, regional availability, storage. |
| [`references/troubleshooting.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/troubleshooting.md) | A build that failed, an app that 5xxs or queues, slow cold starts, settings that reverted. |

## Rules for agents

1. **Deploys cost money.** `cerebrium deploy`, `cerebrium run` and `cerebrium apps scale` start
billable compute, and `cerebrium apps delete` is destructive. State what will run on what
hardware and get the user's confirmation before the first one in a session.
2. **`cerebrium run` is not local.** It packages the working directory, uploads it, and executes
in the cloud on the hardware in `cerebrium.toml`. There is no local emulator.
3. **A `cerebrium.toml` key you leave out is reset to its default on deploy**, not left alone,
and a misspelled key does nothing in the CLI while still reaching the backend. Keep every
value that matters in the file, spelled as in [references/config.md](references/config.md).
4. **Never invent config keys or GPU identifiers.** Both are validated server-side and a wrong
value fails the deploy. The accepted sets are in the references.
5. **Adapt an example before writing from scratch.** `https://github.com/CerebriumAI/examples`
holds runnable references (vLLM, SDXL, Pipecat voice agents, ASGI apps), each with a working
`cerebrium.toml`.
6. **Check the live docs when this skill does not cover it**, rather than guessing: the
`cerebrium-docs` MCP server (search plus docs filesystem), any docs page as markdown by
appending `.md` to its URL, or the index at `https://cerebrium.ai/docs/llms.txt`.

## First run: check state before acting

```bash
cerebrium version # installed? if not: pip install cerebrium
cerebrium projects current # authenticated, and pointed at the intended project?
```

`cerebrium login` opens a browser and fails without a TTY. In CI or headless environments set
`CEREBRIUM_SERVICE_ACCOUNT_TOKEN` (or pass `--service-account-token`) instead: see
[`references/cli.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/cli.md).

## Zero to a deployed endpoint

Starting with no account: create one at `https://dashboard.cerebrium.ai`. The dashboard is also
where API keys and authentication tokens are created. Compute is billed per second; current rates
and any starting credit are at `https://www.cerebrium.ai/pricing`.

```bash
pip install cerebrium # thin wrapper that fetches the Go binary on first use
cerebrium login # interactive only, needs an account
cerebrium init my-app && cd my-app
cerebrium deploy
```

The full loop:

1. Create an account at `https://dashboard.cerebrium.ai`, then `cerebrium login`.
2. `cerebrium init my-app` writes `main.py` and `cerebrium.toml`.
3. Write a function in `main.py` that takes and returns JSON-serialisable values. Everything at
module scope runs once per replica at startup: load models there, not inside the function.
4. Set the config
([`references/config.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/config.md)).
Do not skip `disable_auth` (the scaffold ships `true`, which makes the endpoint public) or
`max_replicas` (default 1, the most common cause of queueing).
5. `cerebrium run main.py::run --prompt "test"` executes remotely on the configured hardware.
6. `cerebrium deploy` builds, uploads, starts the app, and prints the endpoint. Build output
streams from this command and nowhere else.
7. Once running, `cerebrium logs APP_NAME` shows runtime logs.

## Choosing the runtime

Cortex is the default and needs no configuration. A custom runtime means the container starts
your own web server, and you own routing, middleware and auth. Opt in by adding
`[cerebrium.runtime.custom]` with an `entrypoint` and a matching `port`.

| The app | Runtime | Why |
| --- | --- | --- |
| A Python function you want reachable as an endpoint | Cortex | Cerebrium builds the route, parses the request, applies auth. |
| Streaming output token by token | Cortex | `yield` from the function and the response is SSE. A custom runtime buys nothing here. |
| FastAPI, ASGI, Gradio, anything already serving its own HTTP | Custom | Two servers cannot both own the port. |
| WebSockets, or anything bidirectional | Custom | Cortex serves HTTP only. Clients connect over `wss://`. |
| A self-contained server such as vLLM or Triton, custom batching, custom auth | Custom | The process is already the server. |

## Choosing a scaling metric

`scaling_metric` picks what the autoscaler watches, `scaling_target` is the level it holds.

| `scaling_metric` | Reach for it when | `scaling_target` means |
| --- | --- | --- |
| `concurrency_utilization` (default) | GPU inference, and anything whose request times vary | Percent of `replica_concurrency` held per replica. At `replica_concurrency = 200`, target 80 holds 160 in flight. |
| `requests_per_second` | You have measured a rate one replica sustains | Requests per second per replica. Target 5 holds 5 req/s. |
| `cpu_utilization` | CPU-bound work | Percent of `cpu`. At `cpu = 2`, target 80 holds 1.6 cores. |
| `memory_utilization` | Memory-bound work | Percent of `memory`. At `memory = 10`, target 80 holds 8 GB. |

`cpu_utilization` and `memory_utilization` need a live replica to measure, so the API rejects
both with `min_replicas = 0`, and rejects `scaling_buffer` alongside either. Ranges, the rest of
`[cerebrium.scaling]`, and how to pick `load_balancing_algorithm` are in
[`references/config.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/config.md).

## Calling the endpoint

```
POST https://api.cerebrium.ai/v4/{PROJECT_ID}/{APP_NAME}/{FUNCTION_NAME}
```

`PROJECT_ID` already includes its `p-` prefix (`p-abcd1234`), so the path reads
`/v4/p-abcd1234/my-app/run`. Do not add a second `p-`.

```bash
curl -X POST 'https://api.cerebrium.ai/v4/p-abcd1234/my-app/run' \
-H 'Authorization: Bearer <JWT>' \
-H 'Content-Type: application/json' \
-d '{"prompt": "hello"}'
```

Response: `{ "run_id": "...", "run_time_ms": 326.34, "result": { ... } }`

- The token comes from the API Keys page of the dashboard, or from a service account.
- With `disable_auth = true` the endpoint takes unauthenticated requests from anyone.
- A function whose name starts with `_` is not exposed. Use that for helpers.
- **Streaming**: `yield` from the function; the response is `text/event-stream` (SSE).
- **Async**: append `?async=true` for fire-and-forget, bounded by `response_grace_period`
(default 900 seconds, ceiling 12 hours).
- **WebSockets**: require a custom runtime (`[cerebrium.runtime.custom]`) and a `wss://` client.
- Regional hostnames such as `api.aws.us-east-1.cerebrium.ai` still resolve but proxy through
the global router and add latency. Prefer `api.cerebrium.ai`.

## Secrets and automatic environment variables

```bash
cerebrium secrets add KEY=VALUE OTHER=VALUE # project-wide; --app APP_ID scopes to one app
```

Secrets arrive as environment variables, read at container start, so an existing replica needs a
restart or redeploy to see a new one. Set automatically for every app: `APP_NAME`, `PROJECT_ID`
(`p-` prefixed), `BUILD_ID`, and `HF_HOME` (`/persistent-storage/.cache/huggingface`).
Loading