Local-first AI evaluation for developers and coding agents.
Quantiles is open-source, local-first evaluation infrastructure for applied AI systems, designed for developer and coding-agent workflows.
Use the qt CLI and Python SDK to create, run, analyze, and compare evaluations for models, prompts, and agents with resource-efficient local execution. Quantiles records metrics, sample-level results, execution history, and evaluation traces so you can measure system behavior, detect regressions, validate changes, and ship higher-quality, more reliable AI systems.
Quantiles centralizes its components in this monorepo so developers, researchers, and coding agents can use, inspect, modify, test, and extend the system. Its reusable skills and instruction files work with Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other compatible agents.
[2026.08.12] Added built-in benchmark support for gpqa, medmcqa, medqa, mmlu-pro, and pubmedqa. The new qt add <benchmark> command downloads a built-in benchmark’s configuration and prompt from the hosted registry, adds them to the local project, and makes the benchmark easy to customize. See Built-in benchmarks for details and the Quantiles Benchmark Hub for detailed information on each benchmark.
[2026.07.27] Published the model configuration guide, covering the built-in demo model, supported model providers, credentials, request concurrency, cost and data handling, and troubleshooting.
[2026.07.19] Added custom configuration (custom_nocode) evaluations, which let users configure custom evals in quantiles.toml without writing or maintaining custom code. See the custom configuration evaluation documentation for details.
Evaluation workflows quickly outgrow one-off scripts once teams need caching, retries, dataset handling, metrics capture, and run comparison. Quantiles gives teams those primitives so they don't have to build them from scratch:
- Run evaluation workflows locally from the CLI
- Automatically record evaluation runs, steps, metrics, events, inputs, and final outputs
- Store execution history locally in open data formats
- Analyze individual samples using recorded step status, outputs, and metrics
- Inspect and compare evaluation runs directly from the same
qtCLI - Write standard Python with familiar Pythonic patterns
- Resume failed or interrupted evaluation runs without repeating completed work
Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom evaluations or benchmarks from the Quantiles registry, then inspect what changed across runs without requiring notebooks, pipelines, manual comparisons, or a hosted evaluation service.
Install the CLI:
curl -fsSL https://cli.quantiles.io/install.sh | bashRun SimpleQA Verified from the Quantiles benchmark registry:
qt run simpleqa-verifiedThe command above downloads the simpleqa-verified definition from the hosted Quantiles benchmark registry and runs it locally with a demo model that generates random text. Fetching the benchmark definition and an uncached dataset requires network access, but no provider API key or paid model inference is required.
The demo model validates the evaluation workflow. Do not use its results to draw conclusions about model quality.
Inspect the recorded run:
# If you have run `qt run` previously, replace the value passed to `qt show`
# with the ID of the evaluation run you want to inspect.
#
# Use `qt list` to view all evaluation runs and their IDs.
qt show 1To output machine- and agent-readable JSON:
qt show 1 --jsonFor the complete command reference:
qt --helpThe qt CLI starts a local HTTP server when needed, runs evaluation workflows, stores run metadata in the local workspace, records and analyzes workflow steps and metrics, compares runs from the command line, and resumes failed or interrupted evaluation runs.
Common commands:
# Import a built-in benchmark configuration and prompt from the hosted Quantiles benchmark registry into quantiles.toml:
qt add <eval_name># Run a benchmark or evaluation
qt run <eval_name># Add a one-time override to the evaluation run
qt run <eval_name> [--input <json>]# List all evaluation runs
qt list# Show details of a given evaluation run
qt show <run_id># Compare two evaluation runs
qt compare <run_id_a> <run_id_b># Resume a failed or interrupted evaluation run
qt resume <run_id>Note: Pass
--jsontoqt add,qt run,qt list,qt show,qt compare, orqt resumeto request machine- and agent-friendly output.
See the CLI reference for available commands, options, and usage details.
You can customize how the CLI executes built-in benchmarks, custom configuration evaluations, and custom code evaluations using a quantiles.toml or .quantiles.toml configuration file in the current working directory. When you run a benchmark or evaluation, Quantiles first checks the configuration file for a matching local definition. If none is found, it queries the Quantiles benchmark registry (hosted at https://api.quantiles.io) for a built-in benchmark with that name.
See the following resources for more details:
- Configuration documentation - Detailed configuration instructions and reference documentation for supported fields, validation rules, and examples.
- Custom configuration examples - Complete dataset, prompt, model, and scoring configurations.
- Custom-code configuration examples - A complete Python SDK evaluation configuration.
Built-in benchmarks are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Run them directly from the hosted Quantiles benchmark registry with their default configuration, or configure their settings in one of two ways:
- Apply a one-time override, such as the AI model or sample limit, with
--input. For example:
qt run gpqa --input '{"model":"openai:gpt-5.6-luna","limit":10}'- Add a customized built-in benchmark to a config file to apply the same settings in future runs. For example:
qt add gpqaThe Quantiles Benchmark Hub describes available benchmarks, their evaluation setup, and common metrics used across AI evaluation workflows.
To request another registry benchmark, file an issue with its name, source dataset or repository, and any available reference implementation.
Custom evaluations measure behavior specific to your product, workflow, prompt, dataset, rubric, or release process. Quantiles provides two ways to build them:
- Custom configuration (
custom_nocode) evaluations: define a custom evaluation entirely in configuration without writing or maintaining Python. - Custom code (
custom_code) evaluations: build specialized evaluation logic with Python.
Prefer custom configuration evaluations wherever possible because they are easier for humans and agents to create and maintain. Use a custom code evaluation when the required behavior cannot be expressed in configuration.
Use the official Quantiles Python SDK to build custom_code evaluations. The SDK provides Python-native APIs for resilient, efficient evaluations, including durable steps, structured inputs and outputs, and high-performance metrics emission.
The SDK integrates tightly with the qt CLI’s local API for running, recording, and analyzing benchmarks.
The Python SDK source code is available in this repository, and the Python SDK reference has usage instructions and API documentation.
Quantiles is a local-first system that supports offline workflows and stores evaluation metadata, outputs, and metrics on your computer by default.
Quantiles supports fully offline evaluation through both the CLI and Python SDK when all required configurations, prompts, datasets, models, and other dependencies are available locally:
- Quantiles scoring and metric aggregation are computed locally.
- Run metadata, inputs, outputs, steps, and events are stored in a local SQLite database.
- Metrics are stored in local Parquet files.
qt show,qt list, andqt compareaccess only local metadata and metrics stores.- Python evaluation code runs locally on your machine.
Network access may be required to retrieve configurations from the hosted benchmark registry, retrieve datasets, or call hosted AI models.
Quantiles is designed for use with coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, and OpenCode. The Quantiles llms.txt provides a concise, public, LLM-readable overview with links to agent guides and related documentation that agents can use for additional context.
The Quantiles agent skill repository provides a SKILL.md instruction file that guides coding agents through creating, running, analyzing, and comparing evaluations. Use the following agent prompt to install it:
Install the Quantiles eval skill at github.com/quantiles-evals/skill
If you want your agent to run an eval, use the following prompt:
Use the Quantiles eval skill to run the SimpleQA Verified benchmark and summarize the results.
The embedded AGENTS.md file gives agents repository-specific instructions, such as how to add features to the CLI and SDKs, ensuring that contributors can use agents of their choice to make high-quality contributions to the Quantiles open source components.
See the Quantiles documentation for comprehensive guides and reference documentation.
Start here:
Quantiles exists to make AI evaluation workflows more practical, repeatable, and useful for engineering teams. We welcome contributions from the community, whether you are fixing bugs, improving documentation, adding evaluations and benchmarks, or helping make the open-source Quantiles project more reliable for AI engineers and researchers.
Please read our contributing guide to get started.
Please do not report security vulnerabilities through public GitHub issues. Follow the security reporting guidance in SECURITY.md.
Quantiles open source is licensed under the Apache License 2.0. Hosted, enterprise, or managed Quantiles products may be offered under separate commercial terms.