Adding tau3-airline as a custom_code benchmark

README.md

@@ -106,7 +106,7 @@ See the following resources for more details:
 
 - [Configuration guide](https://quantiles.io/documentation/configuration) - Detailed configuration instructions and reference documentation for supported fields, validation rules, and examples.
 - [Model configuration guide](https://quantiles.io/documentation/model-configuration) - Configure provider models and credentials, and troubleshoot common setup issues.
-- [Custom-code configuration example](./cli/examples/configs/custom_code/quantiles.toml) - A complete Python SDK evaluation configuration.
+- [Custom-code benchmark examples](./custom-code-examples/README.md) - Runnable Python SDK evaluations with a shared configuration.
 - [Custom no-code examples](./custom-nocode-examples/quantiles.toml) - Complete dataset, prompt, model, and scoring configurations.
 
 #### Registry benchmarks

cli/README.md

@@ -70,7 +70,7 @@ prompt_template_file = "prompts/qa.txt"
 style = { type = "similarity", golden_column = "answer", metric = { type = "cosine", embedding_model = "fastembed" } }
 ```
 
-See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. See the [model configuration guide](https://quantiles.io/documentation/model-configuration) for guidance on setting up provider models, managing credentials, and troubleshooting configuration issues. Additional runnable configurations are available in the [custom-code example](./examples/configs/custom_code/quantiles.toml) and [custom no-code examples](../custom-nocode-examples/quantiles.toml).
+See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. See the [model configuration guide](https://quantiles.io/documentation/model-configuration) for guidance on setting up provider models, managing credentials, and troubleshooting configuration issues. Additional runnable configurations are available in the [custom-code example](./examples/configs/custom_code/quantiles.toml), the [custom-code benchmark examples](../custom-code-examples/README.md), and the [custom no-code examples](../custom-nocode-examples/quantiles.toml).
 
 ### Remote benchmark fallback
 

custom-code-examples/.gitignore

@@ -0,0 +1 @@
+.quantiles/

custom-code-examples/README.md

@@ -0,0 +1,15 @@
+# Quantiles custom-code examples
+
+This directory contains runnable `custom_code` benchmarks. Each benchmark has an isolated Python project, while the top-level [`quantiles.toml`](./quantiles.toml) provides a single place to run and compare them.
+
+Run commands from this directory:
+
+```bash
+qt run t3-airline
+qt run swebench-pro
+```
+
+- [`t3-airline`](./t3-airline/README.md) wraps the official τ³ airline benchmark. It requires a one-time data checkout and configured model-provider credentials, and its default run calls external models.
+- [`swebench-pro`](./swebench-pro/README.md) runs a one-task gold-patch smoke test with the official SWE-Bench Pro evaluator. Its first run downloads the public harness, dataset, Python dependencies, and a Docker image; it does not call a model.
+
+Quantiles stores both benchmarks' local run history in this directory's ignored `.quantiles/` workspace. Benchmark-specific environments, caches, and generated results stay inside their respective subdirectories and are also ignored.

custom-code-examples/quantiles.toml

@@ -0,0 +1,26 @@
+# Custom-code benchmarks in this directory run in isolated uv projects while
+# sharing the same local Quantiles workspace and run history.
+
+[benchmarks.t3-airline]
+type = "custom_code"
+command = ["uv", "run", "--project", "t3-airline", "python", "t3-airline/t3_airline.py"]
+
+[benchmarks.t3-airline.input]
+data_dir = "t3-airline/.tau3/tau2-bench/data"
+agent_model = "openai:gpt-4.1"
+user_model = "openai:gpt-4.1"
+num_tasks = 1
+num_trials = 1
+max_steps = 100
+max_concurrency = 1
+seed = 300
+
+[benchmarks.swebench-pro]
+type = "custom_code"
+command = ["uv", "run", "--project", "swebench-pro", "python", "swebench-pro/swebench_pro.py"]
+
+[benchmarks.swebench-pro.input]
+cache_dir = "swebench-pro/.swebench-pro-cache"
+output_dir = "swebench-pro/.swebench-pro-results"
+num_workers = 1
+use_local_docker = true

custom-code-examples/swebench-pro/.gitignore

@@ -0,0 +1,4 @@
+.swebench-pro-cache/
+.swebench-pro-results/
+.venv/
+__pycache__/

custom-code-examples/swebench-pro/README.md

@@ -0,0 +1,33 @@
+# SWE-Bench Pro `custom_code` example
+
+This example is a thin adapter around Scale's official SWE-Bench Pro evaluator. With Docker running, start the one-task gold-patch smoke test from the `custom-code-examples` directory:
+
+```bash
+qt run swebench-pro
+```
+
+The first run clones [`scaleapi/SWE-bench_Pro-os`](https://github.com/scaleapi/SWE-bench_Pro-os), creates an isolated environment for its dependencies, downloads the public dataset, extracts the first gold patch, and pulls the task's Docker image. These network-dependent setup operations can take several minutes. Later runs reuse `.swebench-pro-cache/` and `.swebench-pro-results/` in this directory.
+
+The adapter records the official evaluator invocation as a durable Quantiles step and emits `resolution_rate`, `resolved_count`, and `total_count`. It does not call a model or require a provider API key. The upstream project currently marks local Docker evaluation as beta, and the evaluator runs benchmark code inside containers.
+
+To grade your own predictions instead of the default gold patch, override the paths:
+
+```bash
+qt run swebench-pro --input '{
+  "repo_path": "/path/to/SWE-bench_Pro-os",
+  "evaluator_python": "/path/to/SWE-bench_Pro-os/.venv/bin/python",
+  "raw_sample_path": "/path/to/samples.jsonl",
+  "patch_path": "/path/to/predictions.json",
+  "output_dir": "swebench-pro/.swebench-pro-results",
+  "num_workers": 1,
+  "use_local_docker": true
+}'
+```
+
+Run the focused offline tests from this subdirectory:
+
+```bash
+uv run python -m unittest discover -s tests -v
+uv run ruff check .
+uv run ruff format . --check
+```

custom-code-examples/swebench-pro/pyproject.toml

@@ -0,0 +1,26 @@
+[project]
+name = "quantiles-swebench-pro-example"
+version = "0.1.0"
+description = "Minimal Quantiles custom_code adapter for SWE-Bench Pro"
+requires-python = ">=3.12"
+dependencies = [
+  "pydantic>=2.13.4",
+  "quantiles",
+]
+
+[dependency-groups]
+dev = [
+  "ruff>=0.15.12,<0.16.0",
+]
+
+[tool.uv.sources]
+quantiles = { path = "../../python", editable = true }
+
+[tool.ruff]
+target-version = "py312"
+line-length = 100
+indent-width = 2
+
+[tool.ruff.format]
+indent-style = "space"
+quote-style = "double"