Removing built-in (to code) custom benchmarks

AGENTS.md

@@ -97,7 +97,7 @@ Start with the smallest useful sample limit before running a full benchmark with
 
 Ask before running any evaluation that is expected to be slow, expensive, call an external model API, network-dependent, destructive, or likely to meaningfully modify local run state. Note that demo model runs are for workflow validation only. Do not treat them as model-quality benchmark evidence.
 
-Do not run evaluations that call external model APIs unless the user explicitly requests one or provides a provider-prefixed model name. Configure providers in the `quantiles.toml` config file using the [model configuration guide](https://quantiles.io/documentation/model-configuration). Follow configuration examples in the [`cli/examples/configs`](./cli/examples/configs) directory. Before running an evaluation that calls an external model API, verify that the required provider API key is configured, but never print or expose the key value.
+Do not run evaluations that call external model APIs unless the user explicitly requests one or provides a provider-prefixed model name. Configure providers using the [model configuration guide](https://quantiles.io/documentation/model-configuration). For local evaluations, follow the [`custom_code`](./cli/examples/configs/custom_code/quantiles.toml) and [`custom_nocode`](./custom-nocode-examples/quantiles.toml) configuration examples. For registry benchmarks, pass supported run-specific overrides with `--input`. Before running an evaluation that calls an external model API, verify that the required provider API key is configured, but never print or expose the key value.
 
 Model inputs should use provider-prefixed model names, for example:
 

README.md

@@ -39,7 +39,7 @@ Evaluation workflows quickly outgrow one-off scripts once teams need caching, re
 - Write standard Python with familiar Pythonic patterns
 - Resume interrupted or failed runs without repeating completed work
 
-Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom eval code or built-in benchmarks, then inspect what changed across runs without requiring notebooks, pipelines, manual comparisons, or a hosted evaluation service.
+Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom evaluations or benchmarks from the Quantiles registry, then inspect what changed across runs without requiring notebooks, pipelines, manual comparisons, or a hosted evaluation service.
 
 ## Quickstart
 
@@ -49,13 +49,13 @@ Install the CLI:
 curl -fsSL https://cli.quantiles.io/install.sh | bash
 ```
 
-Run the [SimpleQA Verified](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) built-in benchmark:
+Run [SimpleQA Verified](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) from the Quantiles benchmark registry:
 
 ```bash
 qt run simpleqa-verified
 ```
 
-> The command above runs [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) with a demo model that generates random text. It validates the evaluation workflow without requiring provider API keys or incurring inference costs. Do not use its results to draw conclusions about model quality.
+> The command above downloads the [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) definition from the Quantiles benchmark registry and runs it locally with a demo model that generates random text. Fetching the benchmark definition and an uncached dataset requires network access, but no provider API key or paid model inference is required. Do not use demo-model results to draw conclusions about model quality.
 
 Inspect the recorded run:
 
@@ -100,21 +100,26 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for av
 
 ### Configuration and customization
 
-You can customize how the CLI executes [built-in-benchmarks](https://quantiles.io/documentation/built-in-benchmarks), [custom no-code evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations), and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) using a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the benchmark with `qt run`.
+You can define [custom no-code evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) in a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the evaluation with `qt run`.
 
 See the following resources for more details:
 
 - [Configuration guide](https://quantiles.io/documentation/configuration) - Detailed configuration instructions and reference documentation for supported fields, validation rules, and examples.
 - [Model configuration guide](https://quantiles.io/documentation/model-configuration) - Configure provider models and credentials, and troubleshoot common setup issues.
-- [Configuration examples](./cli/examples/configs) - Complete examples, including a [custom-code evaluation](./cli/examples/configs/custom_code/quantiles.toml)
+- [Custom-code configuration example](./cli/examples/configs/custom_code/quantiles.toml) - A complete Python SDK evaluation configuration.
+- [Custom no-code examples](./custom-nocode-examples/quantiles.toml) - Complete dataset, prompt, model, and scoring configurations.
 
-#### Built-in benchmarks
+#### Registry benchmarks
 
-[Built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks) are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Configuration is optional and can override execution settings such as the model and sample count. Use them to get started quickly or establish a repeatable baseline.
+Registry benchmarks are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Run one by name without adding a local configuration section. The CLI downloads its definition from the Quantiles benchmark registry and executes it locally. Supported run-specific settings such as the model and sample limit can be passed with `--input`.
+
+```bash
+qt run simpleqa-verified --input '{"model":"random","limit":10}'
+```
 
 The [benchmark hub](https://quantiles.io/benchmark-hub) describes available benchmarks, their evaluation setup, and common metrics used across AI evaluation workflows.
 
-> To request another open-source built-in benchmark, [file an issue](https://github.com/quantiles-evals/quantiles/issues) with its name, source dataset or repository, and any available reference implementation.
+> To request another registry benchmark, [file an issue](https://github.com/quantiles-evals/quantiles/issues) with its name, source dataset or repository, and any available reference implementation.
 
 #### Custom evaluations
 

cli/README.md

@@ -13,8 +13,8 @@ curl -fsSL https://cli.quantiles.io/install.sh | bash
 A few commands to see `qt` in action:
 
 ```bash
-# 1. Run a built-in evaluation using a demo model that does
-# not incur any usage charges.
+# 1. Download a registry benchmark and run it locally using a
+# demo model that does not incur any usage charges.
 #
 # You can also build and run custom evaluations.
 # See "Configure evaluations" below.
@@ -33,20 +33,21 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for a
 
 ## Configure evaluations
 
-The CLI supports three evaluation types:
+The CLI supports two locally configured evaluation types and remote registry benchmarks:
 
-- [Built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks) run predefined datasets and scoring methods. They work without configuration, but you can override settings such as the model, sample count, and concurrency.
 - [`custom_nocode` evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) define the dataset, prompt template, model, and scoring method entirely in configuration. Supported scoring styles include exact match, multiple choice, and text similarity.
 - [`custom_code` evaluations](https://quantiles.io/documentation/custom-evaluations) run your own Python evaluation through the Quantiles Python SDK.
+- Registry benchmarks are downloaded by name from the Quantiles remote benchmark service and executed locally using the native `custom_nocode` runtime.
 
-Add a `quantiles.toml` or `.quantiles.toml` file to configure an evaluation. For example:
+Add a `quantiles.toml` or `.quantiles.toml` file to configure a custom evaluation. For example:
 
 ```toml
-[benchmarks.pubmedqa]
-dataset = "hf://quantiles/PubMedQA"
-samples = 50
+[benchmarks.support-triage]
+type = "custom_code"
+command = ["uv", "run", "eval.py"]
+
+[benchmarks.support-triage.input]
 model = "openai:gpt-5.6"
-max_workers = 100
 ```
 
 Custom no-code similarity evaluations support Levenshtein distance and cosine similarity. Configuration to use Levenshtein distance is below:
@@ -69,16 +70,24 @@ prompt_template_file = "prompts/qa.txt"
 style = { type = "similarity", golden_column = "answer", metric = { type = "cosine", embedding_model = "fastembed" } }
 ```
 
-See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. See the [model configuration guide](https://quantiles.io/documentation/model-configuration) for guidance on setting up provider models, managing credentials, and troubleshooting configuration issues. Additional runnable configurations are available in [CLI configuration examples](./examples/configs) and [custom no-code examples](../custom-nocode-examples/quantiles.toml).
+See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. See the [model configuration guide](https://quantiles.io/documentation/model-configuration) for guidance on setting up provider models, managing credentials, and troubleshooting configuration issues. Additional runnable configurations are available in the [custom-code example](./examples/configs/custom_code/quantiles.toml) and [custom no-code examples](../custom-nocode-examples/quantiles.toml).
 
-### Custom evaluations and the remote benchmark service
+### Remote benchmark fallback
 
 When you run `qt run <eval_name>`, the CLI first looks in the local configuration file for an evaluation called `eval_name`. If one is found, the CLI runs it immediately. If none is found, `qt` looks in the Quantiles remote benchmark service for an evaluation of the same name. If a match is found, the CLI downloads the benchmark definition and runs it.
 
 >If you want to override the location of the remote benchmark service, use the `--remote-url` flag or the `QUANTILES_REMOTE_URL` environment variable.
 
 When `qt` uses the remote benchmark service, downloaded remote definitions and prompt templates are verified and kept in memory for the run. They will not be cached on disk.
 
+Registry benchmarks do not use local benchmark configuration sections. Apply supported run-specific overrides with `--input` instead:
+
+```bash
+qt run simpleqa-verified --input '{"model":"openai:gpt-5.6","limit":50}'
+```
+
+Resolving the benchmark and downloading an uncached dataset requires network access. Provider-backed models also require their provider credentials, and the provider may charge you for usage.
+
 If you run an eval from the remote benchmark registry, the CLI will persist the registry endpoint, immutable benchmark version, and manifest hash. If that run needs to be resumed later with `qt resume`, the CLI will re-download that exact benchmark version again and reject it if the manifest hash changed (the registry guarantees that versions are immutable once published). This behavior means that resuming runs that were started from the benchmark registry requires internet access.
 
 When `--input` overrides a remote benchmark's `prompt_template_file` with a local file, `qt` also persists the file's SHA-256 hash. On resume, `qt` reads the local prompt into memory and rejects the resume before changing the run status if its contents no longer match the stored hash.

cli/scripts/install-beta.sh

@@ -10,7 +10,7 @@ exe="$bin_dir/qt"
 installed_help_text() {
     echo "Quantiles is a full-featured local-native toolchain for running and analyzing AI evals at scale."
     echo ""
-    echo "Run your first benchmark example without calling an external model API or incurring any usage charges:"
+    echo "Download a registry benchmark and run it locally without calling an external model API or incurring model usage charges:"
     echo ""
     echo "    qt run pubmedqa"
     echo ""

cli/scripts/install.sh

@@ -10,7 +10,7 @@ exe="$bin_dir/qt"
 installed_help_text() {
     echo "Quantiles is a full-featured local-native toolchain for running and analyzing AI evals at scale."
     echo ""
-    echo "Run your first benchmark example without calling an external model API or incurring any usage charges:"
+    echo "Download a registry benchmark and run it locally without calling an external model API or incurring model usage charges:"
     echo ""
     echo "    qt run pubmedqa"
     echo ""

cli/src/builtins/common.rs

@@ -6,38 +6,11 @@ use std::time::Instant;
 use anyhow::{Context, Result};
 use sea_orm::DatabaseConnection;
 use serde::{Deserialize, Serialize};
-use serde_json::Value;
 
 use crate::db::steps::{self, StepDecision};
 use crate::llm::{LLMSampler, Sampler};
 use crate::metrics_store::MetricsStore;
 
-/// Fields shared by every builtin benchmark config. When adding a new builtin,
-/// embed this with `#[serde(flatten)]` so that `limit`, `model`, and
-/// `max_workers` are automatically supported without duplication.
-#[derive(Debug, Default, Deserialize)]
-pub(crate) struct BuiltinConfig {
-    /// Number of dataset rows to evaluate. If omitted, the entire dataset is used.
-    #[serde(default)]
-    pub(crate) limit: Option<usize>,
-    /// The dataset to use for the evaluation.
-    /// Currently `HuggingFace` is the only supported source, and all sources
-    /// must start with `hf://...` or `huggingface://...`
-    #[serde(default)]
-    pub(crate) dataset: Option<String>,
-    /// Which model sampler to use. If omitted, the builtin chooses a sensible default.
-    #[serde(default)]
-    pub(crate) model: Option<Sampler>,
-    /// Maximum number of concurrent workers. Falls back to `QUANTILES_MAX_WORKERS` env var (default 25).
-    #[serde(default)]
-    pub(crate) max_workers: Option<usize>,
-}
-
-/// Extract a string field from a JSON row.
-pub(crate) fn extract_text(row: &Value, key: &str) -> Option<String> {
-    row.get(key)?.as_str().map(String::from)
-}
-
 /// Compute a deterministic hash for step caching.
 pub(crate) fn hash_input(input: &str) -> String {
     let mut hasher = DefaultHasher::new();
@@ -137,37 +110,6 @@ pub(crate) fn resolve_sampler(
     }
 }
 
-/// Emit aggregate `accuracy`, `correct_count`, and `total_count` metrics from a
-/// collection of per-sample boolean correctness values.
-#[expect(clippy::cast_precision_loss)]
-pub(crate) async fn emit_accuracy_metrics(
-    metrics_store: &MetricsStore,
-    run_id: i64,
-    results: impl IntoIterator<Item = bool>,
-) {
-    let mut correct_count = 0usize;
-    let mut total_count = 0usize;
-    for is_correct in results {
-        total_count += 1;
-        if is_correct {
-            correct_count += 1;
-        }
-    }
-
-    if total_count > 0 {
-        let accuracy = correct_count as f64 / total_count as f64;
-        metrics_store
-            .emit(run_id, None, "accuracy", accuracy, None)
-            .await;
-        metrics_store
-            .emit(run_id, None, "correct_count", correct_count as f64, None)
-            .await;
-        metrics_store
-            .emit(run_id, None, "total_count", total_count as f64, None)
-            .await;
-    }
-}
-
 /// Statistics computed from a collection of similarity scores.
 #[derive(Debug)]
 pub(crate) struct ScoreStatistics {
@@ -233,29 +175,6 @@ mod tests {
     use super::*;
     use rstest::rstest;
 
-    #[rstest]
-    #[case("hello", Some("hello"))]
-    #[case("", Some(""))]
-    fn test_extract_text(#[case] value: &str, #[case] expected: Option<&str>) {
-        use serde_json::json;
-        let row = json!({"field": value});
-        assert_eq!(extract_text(&row, "field"), expected.map(String::from));
-    }
-
-    #[test]
-    fn test_extract_text_missing_key() {
-        use serde_json::json;
-        let row = json!({ "other": "data" });
-        assert_eq!(extract_text(&row, "field"), None);
-    }
-
-    #[test]
-    fn test_extract_text_non_string_value() {
-        use serde_json::json;
-        let row = json!({ "field": 42 });
-        assert_eq!(extract_text(&row, "field"), None);
-    }
-
     #[rstest]
     #[case("hello")]
     #[case("")]
@@ -350,74 +269,4 @@ mod tests {
         // Should get a resolved sampler, not the default.
         assert!(!result.sample("test").await.unwrap().is_empty());
     }
-
-    #[test]
-    fn test_emit_accuracy_metrics_empty() {
-        let tmpdir = tempfile::tempdir().unwrap();
-        let metrics_store =
-            crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap();
-
-        let rt = tokio::runtime::Runtime::new().unwrap();
-        rt.block_on(async {
-            emit_accuracy_metrics(&metrics_store, 1, [false; 0]).await;
-            metrics_store.flush(1).await.unwrap();
-            let agg = metrics_store.list_aggregate_for_run(1).await.unwrap();
-            assert!(
-                agg.is_empty(),
-                "no metrics should be emitted for empty results"
-            );
-        });
-    }
-
-    #[test]
-    fn test_emit_accuracy_metrics_all_correct() {
-        let tmpdir = tempfile::tempdir().unwrap();
-        let metrics_store =
-            crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap();
-
-        let rt = tokio::runtime::Runtime::new().unwrap();
-        rt.block_on(async {
-            emit_accuracy_metrics(&metrics_store, 1, [true, true, true]).await;
-            metrics_store.flush(1).await.unwrap();
-            let agg = metrics_store.list_aggregate_for_run(1).await.unwrap();
-
-            let accuracy = agg.iter().find(|m| m.metric_name == "accuracy").unwrap();
-            assert!((accuracy.metric_value - 1.0).abs() < 1e-10);
-
-            let correct = agg
-                .iter()
-                .find(|m| m.metric_name == "correct_count")
-                .unwrap();
-            assert!((correct.metric_value - 3.0).abs() < f64::EPSILON);
-
-            let total = agg.iter().find(|m| m.metric_name == "total_count").unwrap();
-            assert!((total.metric_value - 3.0).abs() < f64::EPSILON);
-        });
-    }
-
-    #[test]
-    fn test_emit_accuracy_metrics_mixed() {
-        let tmpdir = tempfile::tempdir().unwrap();
-        let metrics_store =
-            crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap();
-
-        let rt = tokio::runtime::Runtime::new().unwrap();
-        rt.block_on(async {
-            emit_accuracy_metrics(&metrics_store, 1, [true, false, true, false]).await;
-            metrics_store.flush(1).await.unwrap();
-            let agg = metrics_store.list_aggregate_for_run(1).await.unwrap();
-
-            let accuracy = agg.iter().find(|m| m.metric_name == "accuracy").unwrap();
-            assert!((accuracy.metric_value - 0.5).abs() < 1e-10);
-
-            let correct = agg
-                .iter()
-                .find(|m| m.metric_name == "correct_count")
-                .unwrap();
-            assert!((correct.metric_value - 2.0).abs() < f64::EPSILON);
-
-            let total = agg.iter().find(|m| m.metric_name == "total_count").unwrap();
-            assert!((total.metric_value - 4.0).abs() < f64::EPSILON);
-        });
-    }
 }