From d84d4b452eb67019416b3f3f09e9e9aaa9a4e413 Mon Sep 17 00:00:00 2001 From: Mathew Joseph Date: Mon, 24 Aug 2026 15:34:07 +0530 Subject: [PATCH 1/7] Add skill.md to override the auto-generated agent skill Mintlify generates skill.md from the docs pages and serves it at /skill.md and the /.well-known/skills endpoints. The generated file links to no example code and carries no licence. Adding more links to the docs pages would not change that. The examples repository URL already appears 26 times across 19 pages and the generated skill still contains no reference to it, so an override is the only way to surface it. A skill.md at the repository root overrides the generated file. This copies the currently served content verbatim and adds exactly two things: a link to the examples repository under Resources, and license: MIT in the frontmatter. Kept as a single root file rather than a .mintlify/skills/ directory, because multiple skill files turn /skill.md into a redirect to a JSON index, which would break the `npx skills add https://cerebrium.ai/docs` instruction in docs.json. --- skill.md | 224 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 224 insertions(+) create mode 100644 skill.md diff --git a/skill.md b/skill.md new file mode 100644 index 00000000..2144e2d1 --- /dev/null +++ b/skill.md @@ -0,0 +1,224 @@ +--- +name: Cerebrium +description: Use when deploying serverless AI/ML workloads, building real-time inference APIs, configuring auto-scaling for GPU or CPU apps, managing containerized Python applications, or optimizing cold-start performance for production inference endpoints. +license: MIT +metadata: + mintlify-proj: cerebrium + version: "1.0" +--- + +# Cerebrium Skill + +## Product Summary + +Cerebrium is a serverless GPU/CPU platform for deploying real-time AI workloads with automatic scaling, low cold starts, and pay-per-second billing. Deploy Python apps as REST APIs, streaming endpoints, WebSockets, or async tasks using a single `cerebrium.toml` configuration file. The CLI (`cerebrium init`, `cerebrium deploy`, `cerebrium run`) handles containerization, dependency management, and infrastructure orchestration automatically. Key files: `cerebrium.toml` (configuration), `main.py` (app code), `requirements.txt` (optional dependencies). Primary docs: https://cerebrium.ai/docs + +## When to Use + +Reach for Cerebrium when: +- Deploying inference APIs for LLMs, embeddings, or vision models +- Building real-time voice, video, or streaming applications +- Needing automatic scaling from zero to thousands of concurrent requests +- Optimizing cold-start latency for production workloads +- Running GPU-intensive workloads with pay-per-use pricing +- Managing multi-region deployments for global latency +- Replacing Hugging Face Spaces, Replicate, or Mystic deployments +- Testing code snippets in the cloud with `cerebrium run` + +## Quick Reference + +### CLI Commands + +| Command | Purpose | +|---------|---------| +| `cerebrium login` | Authenticate CLI session | +| `cerebrium init ` | Create new project with `cerebrium.toml` and `main.py` | +| `cerebrium run main.py::function --arg value` | Execute function in cloud (testing/iteration) | +| `cerebrium deploy` | Build and deploy app as persistent endpoint | +| `cerebrium deploy -y` | Deploy without confirmation | + +### Core Configuration Sections + +| Section | Purpose | Key Fields | +|---------|---------|-----------| +| `[cerebrium.deployment]` | App metadata, Python version, dependencies | `name`, `python_version`, `disable_auth`, `use_uv` | +| `[cerebrium.hardware]` | CPU, memory, GPU specs | `cpu`, `memory`, `compute`, `gpu_count`, `region` | +| `[cerebrium.scaling]` | Auto-scaling behavior | `min_replicas`, `max_replicas`, `replica_concurrency`, `scaling_metric`, `scaling_target` | +| `[cerebrium.runtime.custom]` | Custom web server (FastAPI, etc.) | `entrypoint`, `port`, `healthcheck_endpoint` | +| `[cerebrium.dependencies.pip]` | Python packages | `torch = "latest"`, `transformers = "==4.30.0"` | + +### Endpoint URL Format + +``` +https://api.cerebrium.ai/v4/p-{PROJECT_ID}/{APP_NAME}/{FUNCTION_NAME} +``` + +### Authentication + +- Default: `disable_auth = true` (endpoints public) +- Secure: `disable_auth = false` (requires JWT token from API Keys dashboard) +- Token passed as: `Authorization: Bearer ` + +### Automatic Environment Variables + +- `APP_NAME` — app name +- `PROJECT_ID` — project ID +- `BUILD_ID` — current build ID +- `HF_HOME` — `/persistent-storage/.cache/huggingface` (HuggingFace model cache) + +## Decision Guidance + +### When to Use Cortex vs Custom Runtime + +| Scenario | Use | Reason | +|----------|-----|--------| +| Simple function-to-endpoint | Cortex (default) | Automatic REST API, minimal config | +| FastAPI/Flask app | Custom runtime | Full control over routing, auth, middleware | +| WebSocket or streaming | Custom runtime | Cortex doesn't support bidirectional comms | +| Gradio/Streamlit dashboard | Custom runtime | Requires ASGI server | +| LLM with vLLM/TensorRT | Custom runtime | Self-contained server, no Python wrapper | + +### Scaling Metric Selection + +| Metric | Best For | Example | +|--------|----------|---------| +| `concurrency_utilization` (default) | GPU inference, variable request times | `replica_concurrency=1`, `scaling_target=100` | +| `requests_per_second` | Benchmarked throughput targets | `scaling_target=5` maintains 5 req/s | +| `cpu_utilization` | CPU-bound workloads | `cpu=2`, `scaling_target=80` maintains 1.6 CPUs | +| `memory_utilization` | Memory-constrained apps | `memory=10`, `scaling_target=80` maintains 8GB | + +### Compute Tier Trade-off + +| Tier | Cost | Interruption Risk | Use Case | +|------|------|-------------------|----------| +| `interruptible` (default) | Base rate | May be interrupted/relocated | Dev, batch, cost-sensitive | +| `protected` | 2x base rate | No interruptions | Production, long-running requests, SLA-critical | + +### Load Balancing Algorithm + +| Algorithm | Best For | Tradeoff | +|-----------|----------|----------| +| `first-available` (default for `replica_concurrency <= 3`) | GPU inference | Maximizes warm replica utilization, uneven distribution | +| `round-robin` | Uniform request times | Even distribution over time, consistent p50 | +| `min-connections` | Variable request times (LLMs) | Best p90/p99 tail latency, higher selection overhead | +| `random-choice-2` | High-throughput, many replicas | O(1) selection, near-optimal distribution | + +## Workflow + +### 1. Initialize and Configure + +```bash +cerebrium init my-app +cd my-app +``` + +Edit `cerebrium.toml`: +- Set `name`, `python_version`, hardware (`cpu`, `memory`, `compute`) +- Add dependencies under `[cerebrium.dependencies.pip]` +- Configure scaling: `min_replicas`, `max_replicas`, `replica_concurrency` +- For custom servers, add `[cerebrium.runtime.custom]` with `entrypoint` and `port` + +### 2. Write App Code + +Create `main.py` with a function: + +```python +def predict(prompt: str): + # Your inference logic + return {"result": "output"} +``` + +For custom runtime (FastAPI): + +```python +from fastapi import FastAPI +app = FastAPI() + +@app.post("/predict") +def predict(prompt: str): + return {"result": "output"} +``` + +### 3. Test Locally (Optional) + +```bash +cerebrium run main.py::predict --prompt "test" +``` + +This executes in the cloud without deploying. + +### 4. Deploy + +```bash +cerebrium deploy +``` + +The CLI: +- Uploads code and dependencies +- Builds container image +- Creates persistent endpoint +- Returns endpoint URL in dashboard + +### 5. Call Endpoint + +```bash +curl -X POST https://api.cerebrium.ai/v4/p-xxx/my-app/predict \ + -H "Content-Type: application/json" \ + -d '{"prompt": "hello"}' +``` + +### 6. Monitor and Iterate + +- Check dashboard for logs, metrics, cold-start times +- Update `cerebrium.toml` and redeploy +- Use `cerebrium run` for quick testing before full deploy + +## Common Gotchas + +- **Auth disabled by default**: `disable_auth = true` makes endpoints public. Set to `false` for production. +- **Port mismatch**: Custom runtime `entrypoint` port must match `port` in config. +- **Secrets require restart**: Update secrets in dashboard, then restart container or redeploy. +- **GPU concurrency = 1**: GPU apps default to `replica_concurrency = 1`. Increase only if batching is handled in code. +- **Cold starts with `min_replicas = 0`**: Set `min_replicas > 0` to keep warm instances, but costs increase. +- **Python version changes trigger full rebuild**: Changing `python_version` rebuilds entire image; batch changes together. +- **APT/Conda changes trigger full rebuild**: System package changes are slower than pip-only updates. +- **Private Docker images need auth**: Public images (no namespace) work; namespaced images (e.g., `bob/image`) require `docker login -u username`. +- **Async functions run max 12 hours**: Bounded by `response_grace_period` (default 900s). Increase if needed. +- **Streaming requires custom runtime**: Cortex runtime doesn't support SSE or WebSocket streaming. +- **HuggingFace token in secrets**: Store as `HF_TOKEN` or `HF_AUTH_TOKEN` in dashboard secrets, access via `os.environ.get()`. +- **Persistent storage path**: `/persistent-storage/` is available across deployments; use for model weights. +- **Region selection**: Omit `region` to let platform choose; set `region = "global"` for any region with capacity. + +## Verification Checklist + +Before deploying to production: + +- [ ] `cerebrium.toml` has correct `name` (3-30 lowercase alphanumeric + dashes) +- [ ] `disable_auth = false` if endpoint should be protected +- [ ] `replica_concurrency` matches app's actual concurrency capability (1 for GPU inference) +- [ ] `max_replicas` set to prevent runaway costs +- [ ] `min_replicas` appropriate for cold-start tolerance vs. cost +- [ ] Custom runtime: `port` in entrypoint matches `port` in config +- [ ] All secrets added to dashboard and referenced in code via `os.environ.get()` +- [ ] Dependencies listed in `[cerebrium.dependencies.pip]` or `requirements.txt` +- [ ] `cerebrium run` test passes before full deploy +- [ ] Endpoint URL format verified: `https://api.cerebrium.ai/v4/p-{PROJECT_ID}/{APP_NAME}/{FUNCTION}` +- [ ] Response format includes `run_id`, `run_time_ms`, `result` fields +- [ ] Health/readiness endpoints configured if using custom runtime +- [ ] Scaling metrics and targets match workload (e.g., `concurrency_utilization` for GPU) +- [ ] Gradual rollout disabled in dev (`roll_out_duration_seconds = 0`) + +## Resources + +**Comprehensive navigation**: https://cerebrium.ai/docs/llms.txt + +**Example apps**: https://github.com/CerebriumAI/examples — runnable end-to-end projects covering LLMs, voice agents, image and video, batching, and embeddings. Adapt an example before writing an app from scratch. + +**Critical docs**: +1. [TOML Reference](https://cerebrium.ai/docs/toml-reference/toml-reference) — all configuration options +2. [Scaling Apps](https://cerebrium.ai/docs/scaling/scaling-apps) — auto-scaling, replicas, metrics +3. [Defining Container Images](https://cerebrium.ai/docs/container-images/defining-container-images) — dependencies, custom runtimes, base images + +--- + +> For additional documentation and navigation, see: https://cerebrium.ai/docs/llms.txt \ No newline at end of file From 546c1818316f6dfc1d0f59ea20c8bd82ecb127d1 Mon Sep 17 00:00:00 2001 From: mathew-builds Date: Mon, 24 Aug 2026 10:04:37 +0000 Subject: [PATCH 2/7] Prettified Code! --- skill.md | 84 +++++++++++++++++++++++++++++--------------------------- 1 file changed, 44 insertions(+), 40 deletions(-) diff --git a/skill.md b/skill.md index 2144e2d1..a8b3a8a0 100644 --- a/skill.md +++ b/skill.md @@ -3,8 +3,8 @@ name: Cerebrium description: Use when deploying serverless AI/ML workloads, building real-time inference APIs, configuring auto-scaling for GPU or CPU apps, managing containerized Python applications, or optimizing cold-start performance for production inference endpoints. license: MIT metadata: - mintlify-proj: cerebrium - version: "1.0" + mintlify-proj: cerebrium + version: "1.0" --- # Cerebrium Skill @@ -16,6 +16,7 @@ Cerebrium is a serverless GPU/CPU platform for deploying real-time AI workloads ## When to Use Reach for Cerebrium when: + - Deploying inference APIs for LLMs, embeddings, or vision models - Building real-time voice, video, or streaming applications - Needing automatic scaling from zero to thousands of concurrent requests @@ -29,23 +30,23 @@ Reach for Cerebrium when: ### CLI Commands -| Command | Purpose | -|---------|---------| -| `cerebrium login` | Authenticate CLI session | -| `cerebrium init ` | Create new project with `cerebrium.toml` and `main.py` | -| `cerebrium run main.py::function --arg value` | Execute function in cloud (testing/iteration) | -| `cerebrium deploy` | Build and deploy app as persistent endpoint | -| `cerebrium deploy -y` | Deploy without confirmation | +| Command | Purpose | +| --------------------------------------------- | ------------------------------------------------------ | +| `cerebrium login` | Authenticate CLI session | +| `cerebrium init ` | Create new project with `cerebrium.toml` and `main.py` | +| `cerebrium run main.py::function --arg value` | Execute function in cloud (testing/iteration) | +| `cerebrium deploy` | Build and deploy app as persistent endpoint | +| `cerebrium deploy -y` | Deploy without confirmation | ### Core Configuration Sections -| Section | Purpose | Key Fields | -|---------|---------|-----------| -| `[cerebrium.deployment]` | App metadata, Python version, dependencies | `name`, `python_version`, `disable_auth`, `use_uv` | -| `[cerebrium.hardware]` | CPU, memory, GPU specs | `cpu`, `memory`, `compute`, `gpu_count`, `region` | -| `[cerebrium.scaling]` | Auto-scaling behavior | `min_replicas`, `max_replicas`, `replica_concurrency`, `scaling_metric`, `scaling_target` | -| `[cerebrium.runtime.custom]` | Custom web server (FastAPI, etc.) | `entrypoint`, `port`, `healthcheck_endpoint` | -| `[cerebrium.dependencies.pip]` | Python packages | `torch = "latest"`, `transformers = "==4.30.0"` | +| Section | Purpose | Key Fields | +| ------------------------------ | ------------------------------------------ | ----------------------------------------------------------------------------------------- | +| `[cerebrium.deployment]` | App metadata, Python version, dependencies | `name`, `python_version`, `disable_auth`, `use_uv` | +| `[cerebrium.hardware]` | CPU, memory, GPU specs | `cpu`, `memory`, `compute`, `gpu_count`, `region` | +| `[cerebrium.scaling]` | Auto-scaling behavior | `min_replicas`, `max_replicas`, `replica_concurrency`, `scaling_metric`, `scaling_target` | +| `[cerebrium.runtime.custom]` | Custom web server (FastAPI, etc.) | `entrypoint`, `port`, `healthcheck_endpoint` | +| `[cerebrium.dependencies.pip]` | Python packages | `torch = "latest"`, `transformers = "==4.30.0"` | ### Endpoint URL Format @@ -70,38 +71,38 @@ https://api.cerebrium.ai/v4/p-{PROJECT_ID}/{APP_NAME}/{FUNCTION_NAME} ### When to Use Cortex vs Custom Runtime -| Scenario | Use | Reason | -|----------|-----|--------| -| Simple function-to-endpoint | Cortex (default) | Automatic REST API, minimal config | -| FastAPI/Flask app | Custom runtime | Full control over routing, auth, middleware | -| WebSocket or streaming | Custom runtime | Cortex doesn't support bidirectional comms | -| Gradio/Streamlit dashboard | Custom runtime | Requires ASGI server | -| LLM with vLLM/TensorRT | Custom runtime | Self-contained server, no Python wrapper | +| Scenario | Use | Reason | +| --------------------------- | ---------------- | ------------------------------------------- | +| Simple function-to-endpoint | Cortex (default) | Automatic REST API, minimal config | +| FastAPI/Flask app | Custom runtime | Full control over routing, auth, middleware | +| WebSocket or streaming | Custom runtime | Cortex doesn't support bidirectional comms | +| Gradio/Streamlit dashboard | Custom runtime | Requires ASGI server | +| LLM with vLLM/TensorRT | Custom runtime | Self-contained server, no Python wrapper | ### Scaling Metric Selection -| Metric | Best For | Example | -|--------|----------|---------| -| `concurrency_utilization` (default) | GPU inference, variable request times | `replica_concurrency=1`, `scaling_target=100` | -| `requests_per_second` | Benchmarked throughput targets | `scaling_target=5` maintains 5 req/s | -| `cpu_utilization` | CPU-bound workloads | `cpu=2`, `scaling_target=80` maintains 1.6 CPUs | -| `memory_utilization` | Memory-constrained apps | `memory=10`, `scaling_target=80` maintains 8GB | +| Metric | Best For | Example | +| ----------------------------------- | ------------------------------------- | ----------------------------------------------- | +| `concurrency_utilization` (default) | GPU inference, variable request times | `replica_concurrency=1`, `scaling_target=100` | +| `requests_per_second` | Benchmarked throughput targets | `scaling_target=5` maintains 5 req/s | +| `cpu_utilization` | CPU-bound workloads | `cpu=2`, `scaling_target=80` maintains 1.6 CPUs | +| `memory_utilization` | Memory-constrained apps | `memory=10`, `scaling_target=80` maintains 8GB | ### Compute Tier Trade-off -| Tier | Cost | Interruption Risk | Use Case | -|------|------|-------------------|----------| -| `interruptible` (default) | Base rate | May be interrupted/relocated | Dev, batch, cost-sensitive | -| `protected` | 2x base rate | No interruptions | Production, long-running requests, SLA-critical | +| Tier | Cost | Interruption Risk | Use Case | +| ------------------------- | ------------ | ---------------------------- | ----------------------------------------------- | +| `interruptible` (default) | Base rate | May be interrupted/relocated | Dev, batch, cost-sensitive | +| `protected` | 2x base rate | No interruptions | Production, long-running requests, SLA-critical | ### Load Balancing Algorithm -| Algorithm | Best For | Tradeoff | -|-----------|----------|----------| -| `first-available` (default for `replica_concurrency <= 3`) | GPU inference | Maximizes warm replica utilization, uneven distribution | -| `round-robin` | Uniform request times | Even distribution over time, consistent p50 | -| `min-connections` | Variable request times (LLMs) | Best p90/p99 tail latency, higher selection overhead | -| `random-choice-2` | High-throughput, many replicas | O(1) selection, near-optimal distribution | +| Algorithm | Best For | Tradeoff | +| ---------------------------------------------------------- | ------------------------------ | ------------------------------------------------------- | +| `first-available` (default for `replica_concurrency <= 3`) | GPU inference | Maximizes warm replica utilization, uneven distribution | +| `round-robin` | Uniform request times | Even distribution over time, consistent p50 | +| `min-connections` | Variable request times (LLMs) | Best p90/p99 tail latency, higher selection overhead | +| `random-choice-2` | High-throughput, many replicas | O(1) selection, near-optimal distribution | ## Workflow @@ -113,6 +114,7 @@ cd my-app ``` Edit `cerebrium.toml`: + - Set `name`, `python_version`, hardware (`cpu`, `memory`, `compute`) - Add dependencies under `[cerebrium.dependencies.pip]` - Configure scaling: `min_replicas`, `max_replicas`, `replica_concurrency` @@ -154,6 +156,7 @@ cerebrium deploy ``` The CLI: + - Uploads code and dependencies - Builds container image - Creates persistent endpoint @@ -215,10 +218,11 @@ Before deploying to production: **Example apps**: https://github.com/CerebriumAI/examples — runnable end-to-end projects covering LLMs, voice agents, image and video, batching, and embeddings. Adapt an example before writing an app from scratch. **Critical docs**: + 1. [TOML Reference](https://cerebrium.ai/docs/toml-reference/toml-reference) — all configuration options 2. [Scaling Apps](https://cerebrium.ai/docs/scaling/scaling-apps) — auto-scaling, replicas, metrics 3. [Defining Container Images](https://cerebrium.ai/docs/container-images/defining-container-images) — dependencies, custom runtimes, base images --- -> For additional documentation and navigation, see: https://cerebrium.ai/docs/llms.txt \ No newline at end of file +> For additional documentation and navigation, see: https://cerebrium.ai/docs/llms.txt From 043e2de3af6c404b21596425a38a05adb0913be2 Mon Sep 17 00:00:00 2001 From: Mathew Joseph Date: Tue, 25 Aug 2026 10:26:26 +0530 Subject: [PATCH 3/7] Serve the Cerebrium skill exactly as cerebrium-skills defines it Replace the root skill.md body with a byte-identical copy of skills/cerebrium/SKILL.md from CerebriumAI/cerebrium-skills, and copy the frontmatter description and license verbatim. Only name and metadata differ, which the sameness check in that repository permits so Mintlify keeps its own keys. --- skill.md | 326 ++++++++++++++++++++++--------------------------------- 1 file changed, 129 insertions(+), 197 deletions(-) diff --git a/skill.md b/skill.md index a8b3a8a0..8d7bc9e3 100644 --- a/skill.md +++ b/skill.md @@ -1,228 +1,160 @@ --- name: Cerebrium -description: Use when deploying serverless AI/ML workloads, building real-time inference APIs, configuring auto-scaling for GPU or CPU apps, managing containerized Python applications, or optimizing cold-start performance for production inference endpoints. +description: >- + Use for any Cerebrium task: deploying Python code to serverless GPU or CPU, writing or fixing + cerebrium.toml, choosing hardware and regions, calling deployed endpoints (REST, streaming, + WebSocket, async), autoscaling and concurrency, cold starts, secrets, CI/CD, and debugging a + build or a running app from the terminal. Covers the cerebrium CLI, configuration defaults the + API actually applies, accepted GPU identifiers with per-plan limits, and troubleshooting. license: MIT metadata: mintlify-proj: cerebrium version: "1.0" --- -# Cerebrium Skill - -## Product Summary - -Cerebrium is a serverless GPU/CPU platform for deploying real-time AI workloads with automatic scaling, low cold starts, and pay-per-second billing. Deploy Python apps as REST APIs, streaming endpoints, WebSockets, or async tasks using a single `cerebrium.toml` configuration file. The CLI (`cerebrium init`, `cerebrium deploy`, `cerebrium run`) handles containerization, dependency management, and infrastructure orchestration automatically. Key files: `cerebrium.toml` (configuration), `main.py` (app code), `requirements.txt` (optional dependencies). Primary docs: https://cerebrium.ai/docs - -## When to Use - -Reach for Cerebrium when: - -- Deploying inference APIs for LLMs, embeddings, or vision models -- Building real-time voice, video, or streaming applications -- Needing automatic scaling from zero to thousands of concurrent requests -- Optimizing cold-start latency for production workloads -- Running GPU-intensive workloads with pay-per-use pricing -- Managing multi-region deployments for global latency -- Replacing Hugging Face Spaces, Replicate, or Mystic deployments -- Testing code snippets in the cloud with `cerebrium run` - -## Quick Reference - -### CLI Commands - -| Command | Purpose | -| --------------------------------------------- | ------------------------------------------------------ | -| `cerebrium login` | Authenticate CLI session | -| `cerebrium init ` | Create new project with `cerebrium.toml` and `main.py` | -| `cerebrium run main.py::function --arg value` | Execute function in cloud (testing/iteration) | -| `cerebrium deploy` | Build and deploy app as persistent endpoint | -| `cerebrium deploy -y` | Deploy without confirmation | - -### Core Configuration Sections - -| Section | Purpose | Key Fields | -| ------------------------------ | ------------------------------------------ | ----------------------------------------------------------------------------------------- | -| `[cerebrium.deployment]` | App metadata, Python version, dependencies | `name`, `python_version`, `disable_auth`, `use_uv` | -| `[cerebrium.hardware]` | CPU, memory, GPU specs | `cpu`, `memory`, `compute`, `gpu_count`, `region` | -| `[cerebrium.scaling]` | Auto-scaling behavior | `min_replicas`, `max_replicas`, `replica_concurrency`, `scaling_metric`, `scaling_target` | -| `[cerebrium.runtime.custom]` | Custom web server (FastAPI, etc.) | `entrypoint`, `port`, `healthcheck_endpoint` | -| `[cerebrium.dependencies.pip]` | Python packages | `torch = "latest"`, `transformers = "==4.30.0"` | - -### Endpoint URL Format - -``` -https://api.cerebrium.ai/v4/p-{PROJECT_ID}/{APP_NAME}/{FUNCTION_NAME} -``` - -### Authentication - -- Default: `disable_auth = true` (endpoints public) -- Secure: `disable_auth = false` (requires JWT token from API Keys dashboard) -- Token passed as: `Authorization: Bearer ` - -### Automatic Environment Variables - -- `APP_NAME` — app name -- `PROJECT_ID` — project ID -- `BUILD_ID` — current build ID -- `HF_HOME` — `/persistent-storage/.cache/huggingface` (HuggingFace model cache) - -## Decision Guidance - -### When to Use Cortex vs Custom Runtime - -| Scenario | Use | Reason | -| --------------------------- | ---------------- | ------------------------------------------- | -| Simple function-to-endpoint | Cortex (default) | Automatic REST API, minimal config | -| FastAPI/Flask app | Custom runtime | Full control over routing, auth, middleware | -| WebSocket or streaming | Custom runtime | Cortex doesn't support bidirectional comms | -| Gradio/Streamlit dashboard | Custom runtime | Requires ASGI server | -| LLM with vLLM/TensorRT | Custom runtime | Self-contained server, no Python wrapper | - -### Scaling Metric Selection - -| Metric | Best For | Example | -| ----------------------------------- | ------------------------------------- | ----------------------------------------------- | -| `concurrency_utilization` (default) | GPU inference, variable request times | `replica_concurrency=1`, `scaling_target=100` | -| `requests_per_second` | Benchmarked throughput targets | `scaling_target=5` maintains 5 req/s | -| `cpu_utilization` | CPU-bound workloads | `cpu=2`, `scaling_target=80` maintains 1.6 CPUs | -| `memory_utilization` | Memory-constrained apps | `memory=10`, `scaling_target=80` maintains 8GB | - -### Compute Tier Trade-off - -| Tier | Cost | Interruption Risk | Use Case | -| ------------------------- | ------------ | ---------------------------- | ----------------------------------------------- | -| `interruptible` (default) | Base rate | May be interrupted/relocated | Dev, batch, cost-sensitive | -| `protected` | 2x base rate | No interruptions | Production, long-running requests, SLA-critical | - -### Load Balancing Algorithm - -| Algorithm | Best For | Tradeoff | -| ---------------------------------------------------------- | ------------------------------ | ------------------------------------------------------- | -| `first-available` (default for `replica_concurrency <= 3`) | GPU inference | Maximizes warm replica utilization, uneven distribution | -| `round-robin` | Uniform request times | Even distribution over time, consistent p50 | -| `min-connections` | Variable request times (LLMs) | Best p90/p99 tail latency, higher selection overhead | -| `random-choice-2` | High-throughput, many replicas | O(1) selection, near-optimal distribution | - -## Workflow - -### 1. Initialize and Configure +# Cerebrium + +Cerebrium runs Python workloads on serverless GPU and CPU: REST endpoints, SSE streaming, +WebSockets, and async jobs, with scale-to-zero and per-second billing. One `cerebrium.toml` +describes hardware, scaling, dependencies and runtime; one CLI (`cerebrium`) drives everything. + +Reach for it when the workload is an inference API for an LLM, an embedding model or a vision +model; a real-time voice, video or streaming app; bursty traffic that should scale from zero +without holding idle GPUs; a deployment that has to run in several regions for latency or data +residency; or a migration off Replicate, Hugging Face or Mystic, each of which has a guide under +`https://cerebrium.ai/docs/migrations`. + +This file carries the workflow and the rules. Load the reference that matches the task: + +| Read | When | +| --- | --- | +| [`references/cli.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/cli.md) | Running any `cerebrium` command: the full surface, flags, non-interactive auth, CI/CD, which commands cost money. | +| [`references/config.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/config.md) | Writing or fixing `cerebrium.toml`: every key, the default the API applies when it is omitted, accepted ranges, rebuild triggers. | +| [`references/hardware.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/hardware.md) | Choosing `compute`, `gpu_count`, `region`, `provider`, `compute_tier`: accepted GPU identifiers, per-GPU and per-plan limits, regional availability, storage. | +| [`references/troubleshooting.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/troubleshooting.md) | A build that failed, an app that 5xxs or queues, slow cold starts, settings that reverted. | + +## Rules for agents + +1. **Deploys cost money.** `cerebrium deploy`, `cerebrium run` and `cerebrium apps scale` start + billable compute, and `cerebrium apps delete` is destructive. State what will run on what + hardware and get the user's confirmation before the first one in a session. +2. **`cerebrium run` is not local.** It packages the working directory, uploads it, and executes + in the cloud on the hardware in `cerebrium.toml`. There is no local emulator. +3. **A `cerebrium.toml` key you leave out is reset to its default on deploy**, not left alone, + and a misspelled key is ignored in silence. Keep every value that matters in the file, spelled + as in [references/config.md](references/config.md). +4. **Never invent config keys or GPU identifiers.** Both are validated server-side and a wrong + value fails the deploy. The accepted sets are in the references. +5. **Adapt an example before writing from scratch.** `https://github.com/CerebriumAI/examples` + holds runnable references (vLLM, SDXL, Pipecat voice agents, ASGI apps), each with a working + `cerebrium.toml`. +6. **Check the live docs when this skill does not cover it**, rather than guessing: the + `cerebrium-docs` MCP server (search plus docs filesystem), any docs page as markdown by + appending `.md` to its URL, or the index at `https://cerebrium.ai/docs/llms.txt`. + +## First run: check state before acting ```bash -cerebrium init my-app -cd my-app +cerebrium version # installed? if not: pip install cerebrium +cerebrium projects current # authenticated, and pointed at the intended project? ``` -Edit `cerebrium.toml`: +`cerebrium login` opens a browser and fails without a TTY. In CI or headless environments set +`CEREBRIUM_SERVICE_ACCOUNT_TOKEN` (or pass `--service-account-token`) instead: see +[`references/cli.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/cli.md). -- Set `name`, `python_version`, hardware (`cpu`, `memory`, `compute`) -- Add dependencies under `[cerebrium.dependencies.pip]` -- Configure scaling: `min_replicas`, `max_replicas`, `replica_concurrency` -- For custom servers, add `[cerebrium.runtime.custom]` with `entrypoint` and `port` +## Zero to a deployed endpoint -### 2. Write App Code +Starting with no account: create one at `https://dashboard.cerebrium.ai`. The dashboard is also +where API keys and authentication tokens are created. Compute is billed per second; current rates +and any starting credit are at `https://www.cerebrium.ai/pricing`. -Create `main.py` with a function: - -```python -def predict(prompt: str): - # Your inference logic - return {"result": "output"} +```bash +pip install cerebrium # thin wrapper that fetches the Go binary on first use +cerebrium login # interactive only, needs an account +cerebrium init my-app && cd my-app +cerebrium deploy ``` -For custom runtime (FastAPI): - -```python -from fastapi import FastAPI -app = FastAPI() +The full loop: + +1. Create an account at `https://dashboard.cerebrium.ai`, then `cerebrium login`. +2. `cerebrium init my-app` writes `main.py` and `cerebrium.toml`. +3. Write a function in `main.py` that takes and returns JSON-serialisable values. Everything at + module scope runs once per replica at startup: load models there, not inside the function. +4. Set the config + ([`references/config.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/config.md)). + Do not skip `disable_auth` (the scaffold ships `true`, which makes the endpoint public) or + `max_replicas` (default 1, the most common cause of queueing). +5. `cerebrium run main.py::run --prompt "test"` executes remotely on the configured hardware. +6. `cerebrium deploy` builds, uploads, starts the app, and prints the endpoint. Build output + streams from this command and nowhere else. +7. Once running, `cerebrium logs APP_NAME` shows runtime logs. + +## Choosing the runtime + +Cortex is the default and needs no configuration. A custom runtime means the container starts +your own web server, and you own routing, middleware and auth. Opt in by adding +`[cerebrium.runtime.custom]` with an `entrypoint` and a matching `port`. + +| The app | Runtime | Why | +| --- | --- | --- | +| A Python function you want reachable as an endpoint | Cortex | Cerebrium builds the route, parses the request, applies auth. | +| Streaming output token by token | Cortex | `yield` from the function and the response is SSE. A custom runtime buys nothing here. | +| FastAPI, ASGI, Gradio, anything already serving its own HTTP | Custom | Two servers cannot both own the port. | +| WebSockets, or anything bidirectional | Custom | Cortex serves HTTP only. Clients connect over `wss://`. | +| A self-contained server such as vLLM or Triton, custom batching, custom auth | Custom | The process is already the server. | + +## Choosing a scaling metric + +`scaling_metric` picks what the autoscaler watches, `scaling_target` is the level it holds. + +| `scaling_metric` | Reach for it when | `scaling_target` means | +| --- | --- | --- | +| `concurrency_utilization` (default) | GPU inference, and anything whose request times vary | Percent of `replica_concurrency` held per replica. At `replica_concurrency = 200`, target 80 holds 160 in flight. | +| `requests_per_second` | You have measured a rate one replica sustains | Requests per second per replica. Target 5 holds 5 req/s. | +| `cpu_utilization` | CPU-bound work | Percent of `cpu`. At `cpu = 2`, target 80 holds 1.6 cores. | +| `memory_utilization` | Memory-bound work | Percent of `memory`. At `memory = 10`, target 80 holds 8 GB. | + +`cpu_utilization` and `memory_utilization` need a live replica to measure, so the API rejects +both with `min_replicas = 0`, and rejects `scaling_buffer` alongside either. Ranges, the rest of +`[cerebrium.scaling]`, and how to pick `load_balancing_algorithm` are in +[`references/config.md`](https://github.com/CerebriumAI/cerebrium-skills/blob/master/skills/cerebrium/references/config.md). + +## Calling the endpoint -@app.post("/predict") -def predict(prompt: str): - return {"result": "output"} ``` - -### 3. Test Locally (Optional) - -```bash -cerebrium run main.py::predict --prompt "test" +POST https://api.cerebrium.ai/v4/{PROJECT_ID}/{APP_NAME}/{FUNCTION_NAME} ``` -This executes in the cloud without deploying. - -### 4. Deploy +`PROJECT_ID` already includes its `p-` prefix (`p-abcd1234`), so the path reads +`/v4/p-abcd1234/my-app/run`. Do not add a second `p-`. ```bash -cerebrium deploy +curl -X POST 'https://api.cerebrium.ai/v4/p-abcd1234/my-app/run' \ + -H 'Authorization: Bearer ' \ + -H 'Content-Type: application/json' \ + -d '{"prompt": "hello"}' ``` -The CLI: +Response: `{ "run_id": "...", "run_time_ms": 326.34, "result": { ... } }` -- Uploads code and dependencies -- Builds container image -- Creates persistent endpoint -- Returns endpoint URL in dashboard +- The token comes from the API Keys page of the dashboard, or from a service account. +- With `disable_auth = true` the endpoint takes unauthenticated requests from anyone. +- A function whose name starts with `_` is not exposed. Use that for helpers. +- **Streaming**: `yield` from the function; the response is `text/event-stream` (SSE). +- **Async**: append `?async=true` for fire-and-forget, bounded by `response_grace_period` + (default 900 seconds, ceiling 12 hours). +- **WebSockets**: require a custom runtime (`[cerebrium.runtime.custom]`) and a `wss://` client. +- Regional hostnames such as `api.aws.us-east-1.cerebrium.ai` still resolve but proxy through + the global router and add latency. Prefer `api.cerebrium.ai`. -### 5. Call Endpoint +## Secrets and automatic environment variables ```bash -curl -X POST https://api.cerebrium.ai/v4/p-xxx/my-app/predict \ - -H "Content-Type: application/json" \ - -d '{"prompt": "hello"}' +cerebrium secrets add KEY=VALUE OTHER=VALUE # project-wide; --app APP_ID scopes to one app ``` -### 6. Monitor and Iterate - -- Check dashboard for logs, metrics, cold-start times -- Update `cerebrium.toml` and redeploy -- Use `cerebrium run` for quick testing before full deploy - -## Common Gotchas - -- **Auth disabled by default**: `disable_auth = true` makes endpoints public. Set to `false` for production. -- **Port mismatch**: Custom runtime `entrypoint` port must match `port` in config. -- **Secrets require restart**: Update secrets in dashboard, then restart container or redeploy. -- **GPU concurrency = 1**: GPU apps default to `replica_concurrency = 1`. Increase only if batching is handled in code. -- **Cold starts with `min_replicas = 0`**: Set `min_replicas > 0` to keep warm instances, but costs increase. -- **Python version changes trigger full rebuild**: Changing `python_version` rebuilds entire image; batch changes together. -- **APT/Conda changes trigger full rebuild**: System package changes are slower than pip-only updates. -- **Private Docker images need auth**: Public images (no namespace) work; namespaced images (e.g., `bob/image`) require `docker login -u username`. -- **Async functions run max 12 hours**: Bounded by `response_grace_period` (default 900s). Increase if needed. -- **Streaming requires custom runtime**: Cortex runtime doesn't support SSE or WebSocket streaming. -- **HuggingFace token in secrets**: Store as `HF_TOKEN` or `HF_AUTH_TOKEN` in dashboard secrets, access via `os.environ.get()`. -- **Persistent storage path**: `/persistent-storage/` is available across deployments; use for model weights. -- **Region selection**: Omit `region` to let platform choose; set `region = "global"` for any region with capacity. - -## Verification Checklist - -Before deploying to production: - -- [ ] `cerebrium.toml` has correct `name` (3-30 lowercase alphanumeric + dashes) -- [ ] `disable_auth = false` if endpoint should be protected -- [ ] `replica_concurrency` matches app's actual concurrency capability (1 for GPU inference) -- [ ] `max_replicas` set to prevent runaway costs -- [ ] `min_replicas` appropriate for cold-start tolerance vs. cost -- [ ] Custom runtime: `port` in entrypoint matches `port` in config -- [ ] All secrets added to dashboard and referenced in code via `os.environ.get()` -- [ ] Dependencies listed in `[cerebrium.dependencies.pip]` or `requirements.txt` -- [ ] `cerebrium run` test passes before full deploy -- [ ] Endpoint URL format verified: `https://api.cerebrium.ai/v4/p-{PROJECT_ID}/{APP_NAME}/{FUNCTION}` -- [ ] Response format includes `run_id`, `run_time_ms`, `result` fields -- [ ] Health/readiness endpoints configured if using custom runtime -- [ ] Scaling metrics and targets match workload (e.g., `concurrency_utilization` for GPU) -- [ ] Gradual rollout disabled in dev (`roll_out_duration_seconds = 0`) - -## Resources - -**Comprehensive navigation**: https://cerebrium.ai/docs/llms.txt - -**Example apps**: https://github.com/CerebriumAI/examples — runnable end-to-end projects covering LLMs, voice agents, image and video, batching, and embeddings. Adapt an example before writing an app from scratch. - -**Critical docs**: - -1. [TOML Reference](https://cerebrium.ai/docs/toml-reference/toml-reference) — all configuration options -2. [Scaling Apps](https://cerebrium.ai/docs/scaling/scaling-apps) — auto-scaling, replicas, metrics -3. [Defining Container Images](https://cerebrium.ai/docs/container-images/defining-container-images) — dependencies, custom runtimes, base images - ---- - -> For additional documentation and navigation, see: https://cerebrium.ai/docs/llms.txt +Secrets arrive as environment variables, read at container start, so an existing replica needs a +restart or redeploy to see a new one. Set automatically for every app: `APP_NAME`, `PROJECT_ID` +(`p-` prefixed), `BUILD_ID`, and `HF_HOME` (`/persistent-storage/.cache/huggingface`). From afbbfaa160fd8c1bae377df39647dfb0bb701e76 Mon Sep 17 00:00:00 2001 From: Mathew Joseph Date: Tue, 25 Aug 2026 13:01:49 +0530 Subject: [PATCH 4/7] no-mistakes(review): Exempt root skill.md from prettier in .prettierignore --- .prettierignore | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.prettierignore b/.prettierignore index 5ba8e276..e064d2c1 100644 --- a/.prettierignore +++ b/.prettierignore @@ -1 +1,2 @@ -available-hardware.mdx \ No newline at end of file +available-hardware.mdx +/skill.md From 6f13a63f44489669f948f1cc8b775ea708a8d180 Mon Sep 17 00:00:00 2001 From: Mathew Joseph Date: Tue, 25 Aug 2026 13:16:30 +0530 Subject: [PATCH 5/7] no-mistakes(document): Document why skill.md is prettier-exempt --- .prettierignore | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.prettierignore b/.prettierignore index e064d2c1..25325ea4 100644 --- a/.prettierignore +++ b/.prettierignore @@ -1,2 +1,6 @@ available-hardware.mdx + +# Vendored byte-for-byte from skills/cerebrium/SKILL.md in CerebriumAI/cerebrium-skills. +# A checker there fails that repo's CI on any drift, and Prettier re-wraps this file +# if allowed to touch it. Edit the skill upstream, never here. /skill.md From 2972a17d3ddb45b712a54c989db4e12d67215e85 Mon Sep 17 00:00:00 2001 From: Mathew Joseph Date: Tue, 25 Aug 2026 13:28:28 +0530 Subject: [PATCH 6/7] no-mistakes(document): Note vendored skill.md exemption in CLAUDE.md --- CLAUDE.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 13526cb4..6604697b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,6 +30,12 @@ - title: Clear, descriptive page title - description: Concise summary for SEO/navigation +## Vendored skill.md + +- Root `skill.md` is vendored byte-for-byte from `skills/cerebrium/SKILL.md` in CerebriumAI/cerebrium-skills +- A checker in that repository fails when the two copies drift, so edit the skill upstream, never here +- The Voice and tone and Writing standards rules below do not apply to `skill.md`'s body. Never rewrite, reword, reflow, or re-wrap it in this repository + ## Voice and tone - Direct, matter-of-fact tone — write reference material, not a tutorial blog post From e5e0f398ee797052a094805e1adc2d130e7b2a49 Mon Sep 17 00:00:00 2001 From: Mathew Joseph Date: Tue, 25 Aug 2026 15:58:54 +0530 Subject: [PATCH 7/7] Align skill.md rule 3 with the reworded upstream rule Rule 3 is reworded in CerebriumAI/cerebrium-skills PR #3, which PR #4 deliberately left alone. Once both land, the copy published here would no longer match that repository's master and its sameness check would fail. Rule 3 is taken verbatim from PR #3, relative reference link included. Body is 9002 bytes, sha256 88a723f688f4617766174e56ed42e68c36836c115c3f0a258c2ebc2262a36fe6. --- skill.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/skill.md b/skill.md index 8d7bc9e3..ff7dfdab 100644 --- a/skill.md +++ b/skill.md @@ -41,8 +41,8 @@ This file carries the workflow and the rules. Load the reference that matches th 2. **`cerebrium run` is not local.** It packages the working directory, uploads it, and executes in the cloud on the hardware in `cerebrium.toml`. There is no local emulator. 3. **A `cerebrium.toml` key you leave out is reset to its default on deploy**, not left alone, - and a misspelled key is ignored in silence. Keep every value that matters in the file, spelled - as in [references/config.md](references/config.md). + and a misspelled key does nothing in the CLI while still reaching the backend. Keep every + value that matters in the file, spelled as in [references/config.md](references/config.md). 4. **Never invent config keys or GPU identifiers.** Both are validated server-side and a wrong value fails the deploy. The accepted sets are in the references. 5. **Adapt an example before writing from scratch.** `https://github.com/CerebriumAI/examples`