Built-in benchmark updates

AGENTS.md

@@ -16,7 +16,7 @@ Before attempting tasks in this repository, read these files:
 
 ## Scope
 
-These instructions apply to the Quantiles open-source repository. Quantiles is a local-first CLI and SDK toolchain for running AI evaluation workflows with fast, continuous feedback. It runs evaluations, records steps, metrics, events, inputs, and outputs, and runs eval comparisons locally so teams can inspect results, identify regressions, and iterate with confidence.
+These instructions apply to the Quantiles open-source repository. Quantiles is a local-first CLI and SDK toolchain for running AI evaluation workflows with fast, continuous feedback. It retrieves built-in benchmark configurations, executes benchmarks and evaluations locally, records steps, metrics, events, inputs, and outputs, and compares runs locally so teams can inspect results, identify regressions, and iterate with confidence.
 
 Root-level files in this repository provide project-wide orientation, contribution guidance, security policy, licensing, and agent instructions. Implementation-specific work belongs in the relevant subdirectory. If a subdirectory has its own `AGENTS.md` file, follow the nearest one first. Subdirectory instructions should override this root guide for implementation details, package managers, commands, tests, and code style. User instructions may customize the workflow for their project, environment, or preferences, but they must not override safety requirements, system instructions, repository safeguards, or security boundaries.
 
@@ -37,9 +37,9 @@ When performing tasks in this repository, always do the following:
 - Read the nearest subdirectory `AGENTS.md` if present.
 - Inspect the relevant README, package configuration, and existing tests before changing behavior.
 - Prefer the smallest change that satisfies the task.
-- Obtain explicit approval for external model calls and potentially expensive evaluation runs.
+- Obtain explicit approval for external hosted AI model calls and potentially expensive evaluation runs.
 - Keep changes small, public-safe, and reviewable.
-- Preserve Quantiles as a local-first, offline system by default.
+- Preserve local evaluation execution and storage, and keep network-dependent behavior explicit.
 - Verify commands, links, package names, benchmark and evaluation names, and release status before documenting them.
 - Update public docs when CLI behavior, SDK APIs, workflows, benchmarks, schemas, setup steps, or agent guidance changes.
 - Prefer concrete examples with commands, file paths, inputs, outputs, and expected behavior.
@@ -63,6 +63,7 @@ Do not silently change evaluation semantics. Changes to prompts, datasets, score
 Update the relevant documentation in this repository, such as `README.md`, when any of the following change:
 
 - CLI commands, flags, outputs, or setup steps.
+- Hosted benchmark registry behavior.
 - APIs, SDKs, imports, examples, or package names.
 - Benchmark names, datasets, scoring methods, or limitations.
 - DB schemas, step semantics, metrics, events, or comparison behavior.
@@ -110,7 +111,8 @@ Model inputs should use provider-prefixed model names, for example:
 Preserve Quantiles as local-first infrastructure. Follow the guidelines below to ensure the project maintains safety, security, and privacy:
 
 - The CLI and local server should store Quantiles state locally by default.
-- Evaluation workflows may call remote model providers, hosted judges, APIs, or external tools only when configured by the user. Selected benchmarks and evaluation configurations may also download remote datasets.
+- Load benchmark configurations only from a local configuration file or the hosted Quantiles benchmark registry. Do not use any other source unless explicitly requested by the user.
+- Evaluation workflows may download datasets or call remote model providers, hosted judges, APIs, or external tools only when the selected configuration requires them. Keep this network activity explicit.
 - Do not inspect, print, summarize, commit, or infer values from `.env` or `.envrc` files, secrets, tokens, private datasets, PHI, customer data, or local Quantiles databases unless the user explicitly asks and the data is safe to inspect.
 - Never commit the `.quantiles/` directory, SQLite databases, Parquet metrics, local traces, benchmark outputs, provider credentials, or temporary run artifacts.
 - Use placeholder names such as `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `QUANTILES_API_KEY` when examples need credentials.
@@ -128,16 +130,13 @@ Use these terms consistently in public docs:
 - `@quantiles/sdk`: the unreleased TypeScript SDK package.
 - `evaluation`: user-authored evaluation or agent-loop code.
 - `benchmark`: a repeatable evaluation harness with a defined dataset, scoring method, and result shape.
+- `built-in benchmark`: a benchmark from the hosted Quantiles benchmark registry.
 - `run`: one recorded execution of an evaluation or benchmark.
 - `step`: a durable recorded unit of an evaluation execution.
 - `metric`: a measured value emitted during a run.
 - `event`: recorded observability data from an evaluation.
 - `.quantiles/`: local Quantiles workspace state, including a SQLite database and metrics Parquet files.
 
-Prefer `local-first` and `offline by default` for open-source behavior.
-
-When remote model calls, hosted judges, external tools, provider APIs, or network datasets are involved, state that those calls are user-configured exceptions to the local-first default.
-
 ## Output Style For Coding Agents
 
 When adding documentation to files in this repository, follow the guidelines below:
@@ -158,6 +157,7 @@ After running, inspecting, comparing, or resuming Quantiles evaluations, report:
 - Run ID or run IDs.
 - Evaluation or benchmark name.
 - Model, including whether it was a demo model.
+- Hosted built-in benchmark version and manifest hash, when applicable.
 - Input and output JSON.
 - Status and success or failure.
 - Key metrics.

README.md

@@ -23,9 +23,11 @@ Quantiles centralizes its components in this monorepo so developers, researchers
 
 ## ![New](./docs/assets/new-badge.svg) What's New
 
+**[2026.08.12]** Added built-in benchmark support for `gpqa`, `medmcqa`, `medqa`, `mmlu-pro`, and `pubmedqa`. The new `qt add <benchmark>` command downloads a built-in benchmark’s configuration and prompt from the hosted registry, adds them to the local project, and makes the benchmark easy to customize. See [Built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks) for details and the [Quantiles Benchmark Hub](https://quantiles.io/benchmark-hub#built-in) for detailed information on each benchmark.
+
 **[2026.07.27]** Published the [model configuration guide](https://quantiles.io/documentation/model-configuration), covering the built-in demo model, supported model providers, credentials, request concurrency, cost and data handling, and troubleshooting.
 
-**[2026.07.19]** Added `custom_nocode` evaluations, which let users configure custom evals in `quantiles.toml` without writing or maintaining custom code. See the [custom no-code evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) for details.
+**[2026.07.19]** Added custom configuration (`custom_nocode`) evaluations, which let users configure custom evals in `quantiles.toml` without writing or maintaining custom code. See the [custom configuration evaluation documentation](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) for details.
 
 ## Why use Quantiles?
 
@@ -37,7 +39,7 @@ Evaluation workflows quickly outgrow one-off scripts once teams need caching, re
 - Analyze individual samples using recorded step status, outputs, and metrics
 - Inspect and compare evaluation runs directly from the same `qt` CLI
 - Write standard Python with familiar Pythonic patterns
-- Resume interrupted or failed runs without repeating completed work
+- Resume failed or interrupted evaluation runs without repeating completed work
 
 Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom evaluations or benchmarks from the Quantiles registry, then inspect what changed across runs without requiring notebooks, pipelines, manual comparisons, or a hosted evaluation service.
 
@@ -55,18 +57,22 @@ Run [SimpleQA Verified](https://quantiles.io/benchmark-hub/benchmark/simpleqa-ve
 qt run simpleqa-verified
 ```
 
-> The command above downloads the [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) definition from the Quantiles benchmark registry and runs it locally with a demo model that generates random text. Fetching the benchmark definition and an uncached dataset requires network access, but no provider API key or paid model inference is required. Do not use demo-model results to draw conclusions about model quality.
+The command above downloads the [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) definition from the hosted Quantiles benchmark registry and runs it locally with a demo model that generates random text. Fetching the benchmark definition and an uncached dataset requires network access, but no provider API key or paid model inference is required.
+
+> The demo model validates the evaluation workflow. Do not use its results to draw conclusions about model quality.
 
 Inspect the recorded run:
 
 ```bash
-# If you've run `qt run` before, you might need to pass a different integer to `qt show`
+# If you have run `qt run` previously, replace the value passed to `qt show`
+# with the ID of the evaluation run you want to inspect.
 #
-# See all your runs with `qt list`.
+# Use `qt list` to view all evaluation runs and their IDs.
+
 qt show 1
 ```
 
-Or output machine- and agent-readable JSON:
+To output machine- and agent-readable JSON:
 
 ```bash
 qt show 1 --json
@@ -80,88 +86,109 @@ qt --help
 
 ## CLI
 
-Use `qt show` to inspect a single run, `qt list` to see a list of all runs, and `qt compare` to compare behavior across runs.
+The `qt` CLI starts a local HTTP server when needed, runs evaluation workflows, stores run metadata in the local workspace, records and analyzes workflow steps and metrics, compares runs from the command line, and resumes failed or interrupted evaluation runs.
 
 Common commands:
 
 ```bash
-qt --version
-qt add <benchmark_name>
+# Import a built-in benchmark configuration and prompt from the hosted Quantiles benchmark registry into quantiles.toml:
+qt add <eval_name>
+```
+
+```bash
+# Run a benchmark or evaluation
 qt run <eval_name>
+```
+
+```bash
+# Add a one-time override to the evaluation run
+qt run <eval_name> [--input <json>]
+```
+
+```bash
+# List all evaluation runs
 qt list
+```
+
+```bash
+# Show details of a given evaluation run
 qt show <run_id>
+```
+
+```bash
+# Compare two evaluation runs
 qt compare <run_id_a> <run_id_b>
 ```
 
-When `qt run <eval_name>` does not find the requested name in the local configuration, it queries the Quantiles remote benchmark service for a benchmark that has the given name. See the [CLI's README file](./cli/README.md#remote-benchmark-fallback) for details.
+```bash
+# Resume a failed or interrupted evaluation run
+qt resume <run_id>
+```
 
-> Note: Pass `--json` to any of these commands to output machine- and agent-friendly JSON instead of human-formatted output.
+> Note: Pass `--json` to `qt add`, `qt run`, `qt list`, `qt show`, `qt compare`, or `qt resume` to request machine- and agent-friendly output.
 
 See the [CLI reference](https://quantiles.io/documentation/reference/cli) for available commands, options, and usage details.
 
 ### Configuration and customization
 
-You can define [custom no-code evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) in a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the evaluation with `qt run`.
+You can customize how the CLI executes [built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks), [custom configuration evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations), and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) using a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory. When you run a benchmark or evaluation, Quantiles first checks the configuration file for a matching local definition. If none is found, it queries the Quantiles benchmark registry (hosted at `https://api.quantiles.io`) for a built-in benchmark with that name.
 
 See the following resources for more details:
 
-- [Configuration guide](https://quantiles.io/documentation/configuration) - Detailed configuration instructions and reference documentation for supported fields, validation rules, and examples.
-- [Model configuration guide](https://quantiles.io/documentation/model-configuration) - Configure provider models and credentials, and troubleshoot common setup issues.
-- [Custom-code configuration example](./cli/examples/configs/custom_code/quantiles.toml) - A complete Python SDK evaluation configuration.
-- [Custom no-code examples](./custom-nocode-examples/quantiles.toml) - Complete dataset, prompt, model, and scoring configurations.
+- [Configuration documentation](https://quantiles.io/documentation/configuration) - Detailed configuration instructions and reference documentation for supported fields, validation rules, and examples.
+- [Custom configuration examples](./custom-nocode-examples/quantiles.toml) - Complete dataset, prompt, model, and scoring configurations.
+- [Custom-code configuration examples](./cli/examples/configs/custom_code/quantiles.toml) - A complete Python SDK evaluation configuration.
+
+#### Built-in benchmarks
 
-#### Registry benchmarks
+[Built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks) are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Run them directly from the hosted Quantiles benchmark registry with their default configuration, or configure their settings in one of two ways:
 
-Registry benchmarks are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Run one by name without adding a local configuration section. The CLI downloads its definition from the Quantiles benchmark registry and executes it locally. Supported run-specific settings such as the model and sample limit can be passed with `--input`.
+- [Apply a one-time override](https://quantiles.io/documentation/built-in-benchmarks#apply-one-time-configuration-overrides), such as the AI model or sample limit, with `--input`. For example:
 
 ```bash
-qt run simpleqa-verified --input '{"model":"random","limit":10}'
+qt run gpqa --input '{"model":"openai:gpt-5.6-luna","limit":10}'
 ```
 
-To save a registry benchmark in the local configuration, add it by name:
+- [Add a customized built-in benchmark](https://quantiles.io/documentation/built-in-benchmarks#apply-persistent-configuration-settings) to a config file to apply the same settings in future runs. For example:
 
 ```bash
-qt add simpleqa-verified
+qt add gpqa
 ```
 
-This command downloads the benchmark definition and prompt template, appends the benchmark to an existing `quantiles.toml` or `.quantiles.toml`, or creates `quantiles.toml` in the current directory. The prompt template is stored beside the configuration at `<benchmark_name>-prompt/prompt.txt`. The command returns an error if the benchmark is already configured or is not present in the registry. Pass `--json` for machine-readable output.
-
-The [benchmark hub](https://quantiles.io/benchmark-hub) describes available benchmarks, their evaluation setup, and common metrics used across AI evaluation workflows.
+The [Quantiles Benchmark Hub](https://quantiles.io/benchmark-hub) describes available benchmarks, their evaluation setup, and common metrics used across AI evaluation workflows.
 
 > To request another registry benchmark, [file an issue](https://github.com/quantiles-evals/quantiles/issues) with its name, source dataset or repository, and any available reference implementation.
 
 #### Custom evaluations
 
 Custom evaluations measure behavior specific to your product, workflow, prompt, dataset, rubric, or release process. Quantiles provides two ways to build them:
 
-- [`custom_nocode`](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations): define a custom evaluation entirely in configuration.
-- [`custom_code`](https://quantiles.io/documentation/custom-evaluations): build specialized evaluation logic with [Python](https://quantiles.io/documentation/reference/python-sdk).
+- [Custom configuration (`custom_nocode`) evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations): define a custom evaluation entirely in configuration without writing or maintaining Python.
+- [Custom code (`custom_code`) evaluations](https://quantiles.io/documentation/custom-evaluations): build specialized evaluation logic with [Python](https://quantiles.io/documentation/reference/python-sdk).
 
-Prefer to use `custom_nocode` evaluations wherever possible, since they're easier for humans and agents to create and maintain. When required, fall back to `custom_code` evaluations.
+Prefer custom configuration evaluations wherever possible because they are easier for humans and agents to create and maintain. Use a custom code evaluation when the required behavior cannot be expressed in configuration.
 
-#### Python SDK for `custom_code` evaluations
+##### Python SDK for `custom_code` evaluations
 
 Use the [official Quantiles Python SDK](https://quantiles.io/documentation/reference/python-sdk) to build `custom_code` evaluations. The SDK provides Python-native APIs for resilient, efficient evaluations, including durable steps, structured inputs and outputs, and high-performance metrics emission.
 
 The SDK integrates tightly with the `qt` CLI’s local API for running, recording, and analyzing benchmarks.
 
 The [Python SDK source code](./python) is available in this repository, and the [Python SDK reference](https://quantiles.io/documentation/reference/python-sdk) has usage instructions and API documentation.
 
-## Local-First and Offline by Default
+## Local-First Execution and Offline Workflows
 
 Quantiles is a [local-first system that supports offline workflows](https://quantiles.io/documentation/local-first-offline) and stores evaluation metadata, outputs, and metrics on your computer by default.
 
-The entire Quantiles toolchain, including the `qt` CLI, SDKs, on-disk data formats, and REST API, is optimized to use your local computing power instead of relying on cloud or other non-local resources.
-
-The CLI and Python SDK support offline evaluation workflows when the required code, datasets, and models are available locally:
+Quantiles supports fully offline evaluation through both the CLI and Python SDK when all required configurations, prompts, datasets, models, and other dependencies are available locally:
 
 - Quantiles scoring and metric aggregation are computed locally.
 - Run metadata, inputs, outputs, steps, and events are stored in a local [SQLite](https://sqlite.org/) database.
 - Metrics are stored in local [Parquet](https://parquet.apache.org/) files.
 - `qt show`, `qt list`, and `qt compare` access only local metadata and metrics stores.
 - Python evaluation code runs locally on your machine.
 
-Downloading uncached datasets and calling remote models, hosted judges, or external tools requires network access. These operations occur only when requested by the selected benchmark or evaluation configuration.
+Network access may be required to retrieve configurations from the hosted benchmark registry, retrieve datasets, or call hosted AI models.
 
 ## Coding Agents