diff --git a/AGENTS.md b/AGENTS.md index 637132c..cf30db1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -97,7 +97,7 @@ Start with the smallest useful sample limit before running a full benchmark with Ask before running any evaluation that is expected to be slow, expensive, call an external model API, network-dependent, destructive, or likely to meaningfully modify local run state. Note that demo model runs are for workflow validation only. Do not treat them as model-quality benchmark evidence. -Do not run evaluations that call external model APIs unless the user explicitly requests one or provides a provider-prefixed model name. Configure providers in the `quantiles.toml` config file using the [model configuration guide](https://quantiles.io/documentation/model-configuration). Follow configuration examples in the [`cli/examples/configs`](./cli/examples/configs) directory. Before running an evaluation that calls an external model API, verify that the required provider API key is configured, but never print or expose the key value. +Do not run evaluations that call external model APIs unless the user explicitly requests one or provides a provider-prefixed model name. Configure providers using the [model configuration guide](https://quantiles.io/documentation/model-configuration). For local evaluations, follow the [`custom_code`](./cli/examples/configs/custom_code/quantiles.toml) and [`custom_nocode`](./custom-nocode-examples/quantiles.toml) configuration examples. For registry benchmarks, pass supported run-specific overrides with `--input`. Before running an evaluation that calls an external model API, verify that the required provider API key is configured, but never print or expose the key value. Model inputs should use provider-prefixed model names, for example: diff --git a/README.md b/README.md index 36ead5c..8cdfebf 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ Evaluation workflows quickly outgrow one-off scripts once teams need caching, re - Write standard Python with familiar Pythonic patterns - Resume interrupted or failed runs without repeating completed work -Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom eval code or built-in benchmarks, then inspect what changed across runs without requiring notebooks, pipelines, manual comparisons, or a hosted evaluation service. +Quantiles borrows concepts from durable workflow execution systems to make evaluation runs resilient to crashes and restarts, while adding a high-throughput execution engine, rich observability, metrics, and eval reproducibility. Use it to run custom evaluations or benchmarks from the Quantiles registry, then inspect what changed across runs without requiring notebooks, pipelines, manual comparisons, or a hosted evaluation service. ## Quickstart @@ -49,13 +49,13 @@ Install the CLI: curl -fsSL https://cli.quantiles.io/install.sh | bash ``` -Run the [SimpleQA Verified](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) built-in benchmark: +Run [SimpleQA Verified](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) from the Quantiles benchmark registry: ```bash qt run simpleqa-verified ``` -> The command above runs [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) with a demo model that generates random text. It validates the evaluation workflow without requiring provider API keys or incurring inference costs. Do not use its results to draw conclusions about model quality. +> The command above downloads the [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) definition from the Quantiles benchmark registry and runs it locally with a demo model that generates random text. Fetching the benchmark definition and an uncached dataset requires network access, but no provider API key or paid model inference is required. Do not use demo-model results to draw conclusions about model quality. Inspect the recorded run: @@ -100,21 +100,26 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for av ### Configuration and customization -You can customize how the CLI executes [built-in-benchmarks](https://quantiles.io/documentation/built-in-benchmarks), [custom no-code evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations), and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) using a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the benchmark with `qt run`. +You can define [custom no-code evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) and [custom code evaluations](https://quantiles.io/documentation/custom-evaluations) in a `quantiles.toml` or `.quantiles.toml` configuration file in the current working directory or a parent directory. The CLI uses this configuration each time you run the evaluation with `qt run`. See the following resources for more details: - [Configuration guide](https://quantiles.io/documentation/configuration) - Detailed configuration instructions and reference documentation for supported fields, validation rules, and examples. - [Model configuration guide](https://quantiles.io/documentation/model-configuration) - Configure provider models and credentials, and troubleshoot common setup issues. -- [Configuration examples](./cli/examples/configs) - Complete examples, including a [custom-code evaluation](./cli/examples/configs/custom_code/quantiles.toml) +- [Custom-code configuration example](./cli/examples/configs/custom_code/quantiles.toml) - A complete Python SDK evaluation configuration. +- [Custom no-code examples](./custom-nocode-examples/quantiles.toml) - Complete dataset, prompt, model, and scoring configurations. -#### Built-in benchmarks +#### Registry benchmarks -[Built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks) are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Configuration is optional and can override execution settings such as the model and sample count. Use them to get started quickly or establish a repeatable baseline. +Registry benchmarks are ready-to-run evaluations with predefined datasets, scoring methods, and metrics. Run one by name without adding a local configuration section. The CLI downloads its definition from the Quantiles benchmark registry and executes it locally. Supported run-specific settings such as the model and sample limit can be passed with `--input`. + +```bash +qt run simpleqa-verified --input '{"model":"random","limit":10}' +``` The [benchmark hub](https://quantiles.io/benchmark-hub) describes available benchmarks, their evaluation setup, and common metrics used across AI evaluation workflows. -> To request another open-source built-in benchmark, [file an issue](https://github.com/quantiles-evals/quantiles/issues) with its name, source dataset or repository, and any available reference implementation. +> To request another registry benchmark, [file an issue](https://github.com/quantiles-evals/quantiles/issues) with its name, source dataset or repository, and any available reference implementation. #### Custom evaluations diff --git a/cli/README.md b/cli/README.md index dcfbe36..c3ff939 100644 --- a/cli/README.md +++ b/cli/README.md @@ -13,8 +13,8 @@ curl -fsSL https://cli.quantiles.io/install.sh | bash A few commands to see `qt` in action: ```bash -# 1. Run a built-in evaluation using a demo model that does -# not incur any usage charges. +# 1. Download a registry benchmark and run it locally using a +# demo model that does not incur any usage charges. # # You can also build and run custom evaluations. # See "Configure evaluations" below. @@ -33,20 +33,21 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for a ## Configure evaluations -The CLI supports three evaluation types: +The CLI supports two locally configured evaluation types and remote registry benchmarks: -- [Built-in benchmarks](https://quantiles.io/documentation/built-in-benchmarks) run predefined datasets and scoring methods. They work without configuration, but you can override settings such as the model, sample count, and concurrency. - [`custom_nocode` evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) define the dataset, prompt template, model, and scoring method entirely in configuration. Supported scoring styles include exact match, multiple choice, and text similarity. - [`custom_code` evaluations](https://quantiles.io/documentation/custom-evaluations) run your own Python evaluation through the Quantiles Python SDK. +- Registry benchmarks are downloaded by name from the Quantiles remote benchmark service and executed locally using the native `custom_nocode` runtime. -Add a `quantiles.toml` or `.quantiles.toml` file to configure an evaluation. For example: +Add a `quantiles.toml` or `.quantiles.toml` file to configure a custom evaluation. For example: ```toml -[benchmarks.pubmedqa] -dataset = "hf://quantiles/PubMedQA" -samples = 50 +[benchmarks.support-triage] +type = "custom_code" +command = ["uv", "run", "eval.py"] + +[benchmarks.support-triage.input] model = "openai:gpt-5.6" -max_workers = 100 ``` Custom no-code similarity evaluations support Levenshtein distance and cosine similarity. Configuration to use Levenshtein distance is below: @@ -69,9 +70,9 @@ prompt_template_file = "prompts/qa.txt" style = { type = "similarity", golden_column = "answer", metric = { type = "cosine", embedding_model = "fastembed" } } ``` -See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. See the [model configuration guide](https://quantiles.io/documentation/model-configuration) for guidance on setting up provider models, managing credentials, and troubleshooting configuration issues. Additional runnable configurations are available in [CLI configuration examples](./examples/configs) and [custom no-code examples](../custom-nocode-examples/quantiles.toml). +See the [configuration guide](https://quantiles.io/documentation/configuration) for file location, supported fields, validation behavior, and examples. See the [model configuration guide](https://quantiles.io/documentation/model-configuration) for guidance on setting up provider models, managing credentials, and troubleshooting configuration issues. Additional runnable configurations are available in the [custom-code example](./examples/configs/custom_code/quantiles.toml) and [custom no-code examples](../custom-nocode-examples/quantiles.toml). -### Custom evaluations and the remote benchmark service +### Remote benchmark fallback When you run `qt run `, the CLI first looks in the local configuration file for an evaluation called `eval_name`. If one is found, the CLI runs it immediately. If none is found, `qt` looks in the Quantiles remote benchmark service for an evaluation of the same name. If a match is found, the CLI downloads the benchmark definition and runs it. @@ -79,6 +80,14 @@ When you run `qt run `, the CLI first looks in the local configuratio When `qt` uses the remote benchmark service, downloaded remote definitions and prompt templates are verified and kept in memory for the run. They will not be cached on disk. +Registry benchmarks do not use local benchmark configuration sections. Apply supported run-specific overrides with `--input` instead: + +```bash +qt run simpleqa-verified --input '{"model":"openai:gpt-5.6","limit":50}' +``` + +Resolving the benchmark and downloading an uncached dataset requires network access. Provider-backed models also require their provider credentials, and the provider may charge you for usage. + If you run an eval from the remote benchmark registry, the CLI will persist the registry endpoint, immutable benchmark version, and manifest hash. If that run needs to be resumed later with `qt resume`, the CLI will re-download that exact benchmark version again and reject it if the manifest hash changed (the registry guarantees that versions are immutable once published). This behavior means that resuming runs that were started from the benchmark registry requires internet access. When `--input` overrides a remote benchmark's `prompt_template_file` with a local file, `qt` also persists the file's SHA-256 hash. On resume, `qt` reads the local prompt into memory and rejects the resume before changing the run status if its contents no longer match the stored hash. diff --git a/cli/scripts/install-beta.sh b/cli/scripts/install-beta.sh index 738663f..9591548 100755 --- a/cli/scripts/install-beta.sh +++ b/cli/scripts/install-beta.sh @@ -10,7 +10,7 @@ exe="$bin_dir/qt" installed_help_text() { echo "Quantiles is a full-featured local-native toolchain for running and analyzing AI evals at scale." echo "" - echo "Run your first benchmark example without calling an external model API or incurring any usage charges:" + echo "Download a registry benchmark and run it locally without calling an external model API or incurring model usage charges:" echo "" echo " qt run pubmedqa" echo "" diff --git a/cli/scripts/install.sh b/cli/scripts/install.sh index e6e6acb..b55512e 100755 --- a/cli/scripts/install.sh +++ b/cli/scripts/install.sh @@ -10,7 +10,7 @@ exe="$bin_dir/qt" installed_help_text() { echo "Quantiles is a full-featured local-native toolchain for running and analyzing AI evals at scale." echo "" - echo "Run your first benchmark example without calling an external model API or incurring any usage charges:" + echo "Download a registry benchmark and run it locally without calling an external model API or incurring model usage charges:" echo "" echo " qt run pubmedqa" echo "" diff --git a/cli/src/builtins/common.rs b/cli/src/builtins/common.rs index 75fc999..094d1e0 100644 --- a/cli/src/builtins/common.rs +++ b/cli/src/builtins/common.rs @@ -6,38 +6,11 @@ use std::time::Instant; use anyhow::{Context, Result}; use sea_orm::DatabaseConnection; use serde::{Deserialize, Serialize}; -use serde_json::Value; use crate::db::steps::{self, StepDecision}; use crate::llm::{LLMSampler, Sampler}; use crate::metrics_store::MetricsStore; -/// Fields shared by every builtin benchmark config. When adding a new builtin, -/// embed this with `#[serde(flatten)]` so that `limit`, `model`, and -/// `max_workers` are automatically supported without duplication. -#[derive(Debug, Default, Deserialize)] -pub(crate) struct BuiltinConfig { - /// Number of dataset rows to evaluate. If omitted, the entire dataset is used. - #[serde(default)] - pub(crate) limit: Option, - /// The dataset to use for the evaluation. - /// Currently `HuggingFace` is the only supported source, and all sources - /// must start with `hf://...` or `huggingface://...` - #[serde(default)] - pub(crate) dataset: Option, - /// Which model sampler to use. If omitted, the builtin chooses a sensible default. - #[serde(default)] - pub(crate) model: Option, - /// Maximum number of concurrent workers. Falls back to `QUANTILES_MAX_WORKERS` env var (default 25). - #[serde(default)] - pub(crate) max_workers: Option, -} - -/// Extract a string field from a JSON row. -pub(crate) fn extract_text(row: &Value, key: &str) -> Option { - row.get(key)?.as_str().map(String::from) -} - /// Compute a deterministic hash for step caching. pub(crate) fn hash_input(input: &str) -> String { let mut hasher = DefaultHasher::new(); @@ -137,37 +110,6 @@ pub(crate) fn resolve_sampler( } } -/// Emit aggregate `accuracy`, `correct_count`, and `total_count` metrics from a -/// collection of per-sample boolean correctness values. -#[expect(clippy::cast_precision_loss)] -pub(crate) async fn emit_accuracy_metrics( - metrics_store: &MetricsStore, - run_id: i64, - results: impl IntoIterator, -) { - let mut correct_count = 0usize; - let mut total_count = 0usize; - for is_correct in results { - total_count += 1; - if is_correct { - correct_count += 1; - } - } - - if total_count > 0 { - let accuracy = correct_count as f64 / total_count as f64; - metrics_store - .emit(run_id, None, "accuracy", accuracy, None) - .await; - metrics_store - .emit(run_id, None, "correct_count", correct_count as f64, None) - .await; - metrics_store - .emit(run_id, None, "total_count", total_count as f64, None) - .await; - } -} - /// Statistics computed from a collection of similarity scores. #[derive(Debug)] pub(crate) struct ScoreStatistics { @@ -233,29 +175,6 @@ mod tests { use super::*; use rstest::rstest; - #[rstest] - #[case("hello", Some("hello"))] - #[case("", Some(""))] - fn test_extract_text(#[case] value: &str, #[case] expected: Option<&str>) { - use serde_json::json; - let row = json!({"field": value}); - assert_eq!(extract_text(&row, "field"), expected.map(String::from)); - } - - #[test] - fn test_extract_text_missing_key() { - use serde_json::json; - let row = json!({ "other": "data" }); - assert_eq!(extract_text(&row, "field"), None); - } - - #[test] - fn test_extract_text_non_string_value() { - use serde_json::json; - let row = json!({ "field": 42 }); - assert_eq!(extract_text(&row, "field"), None); - } - #[rstest] #[case("hello")] #[case("")] @@ -350,74 +269,4 @@ mod tests { // Should get a resolved sampler, not the default. assert!(!result.sample("test").await.unwrap().is_empty()); } - - #[test] - fn test_emit_accuracy_metrics_empty() { - let tmpdir = tempfile::tempdir().unwrap(); - let metrics_store = - crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap(); - - let rt = tokio::runtime::Runtime::new().unwrap(); - rt.block_on(async { - emit_accuracy_metrics(&metrics_store, 1, [false; 0]).await; - metrics_store.flush(1).await.unwrap(); - let agg = metrics_store.list_aggregate_for_run(1).await.unwrap(); - assert!( - agg.is_empty(), - "no metrics should be emitted for empty results" - ); - }); - } - - #[test] - fn test_emit_accuracy_metrics_all_correct() { - let tmpdir = tempfile::tempdir().unwrap(); - let metrics_store = - crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap(); - - let rt = tokio::runtime::Runtime::new().unwrap(); - rt.block_on(async { - emit_accuracy_metrics(&metrics_store, 1, [true, true, true]).await; - metrics_store.flush(1).await.unwrap(); - let agg = metrics_store.list_aggregate_for_run(1).await.unwrap(); - - let accuracy = agg.iter().find(|m| m.metric_name == "accuracy").unwrap(); - assert!((accuracy.metric_value - 1.0).abs() < 1e-10); - - let correct = agg - .iter() - .find(|m| m.metric_name == "correct_count") - .unwrap(); - assert!((correct.metric_value - 3.0).abs() < f64::EPSILON); - - let total = agg.iter().find(|m| m.metric_name == "total_count").unwrap(); - assert!((total.metric_value - 3.0).abs() < f64::EPSILON); - }); - } - - #[test] - fn test_emit_accuracy_metrics_mixed() { - let tmpdir = tempfile::tempdir().unwrap(); - let metrics_store = - crate::metrics_store::MetricsStore::new(tmpdir.path().to_path_buf()).unwrap(); - - let rt = tokio::runtime::Runtime::new().unwrap(); - rt.block_on(async { - emit_accuracy_metrics(&metrics_store, 1, [true, false, true, false]).await; - metrics_store.flush(1).await.unwrap(); - let agg = metrics_store.list_aggregate_for_run(1).await.unwrap(); - - let accuracy = agg.iter().find(|m| m.metric_name == "accuracy").unwrap(); - assert!((accuracy.metric_value - 0.5).abs() < 1e-10); - - let correct = agg - .iter() - .find(|m| m.metric_name == "correct_count") - .unwrap(); - assert!((correct.metric_value - 2.0).abs() < f64::EPSILON); - - let total = agg.iter().find(|m| m.metric_name == "total_count").unwrap(); - assert!((total.metric_value - 4.0).abs() < f64::EPSILON); - }); - } } diff --git a/cli/src/builtins/financebench/mod.rs b/cli/src/builtins/financebench/mod.rs deleted file mode 100644 index 74925ab..0000000 --- a/cli/src/builtins/financebench/mod.rs +++ /dev/null @@ -1 +0,0 @@ -pub use crate::builtins::similarity::FINANCEBENCH as FinancebenchBuiltin; diff --git a/cli/src/builtins/input.rs b/cli/src/builtins/input.rs deleted file mode 100644 index 437a124..0000000 --- a/cli/src/builtins/input.rs +++ /dev/null @@ -1,40 +0,0 @@ -use serde::Serialize; - -use crate::llm::Sampler; - -/// Normalized run input schema for all builtins. -#[derive(Serialize)] -pub(crate) struct BuiltinRunInput { - pub(crate) dataset: String, - pub(crate) model: String, - pub(crate) num_samples: usize, - #[serde(skip_serializing_if = "Option::is_none")] - pub(crate) max_workers: Option, -} - -/// Rewrite the run record input to the normalized builtin shape. -pub(crate) async fn set_builtin_run_input( - db: &sea_orm::DatabaseConnection, - run_id: i64, - dataset: &str, - model: Option<&Sampler>, - num_samples: usize, - max_workers: Option, -) -> anyhow::Result<()> { - let input = serde_json::to_string(&BuiltinRunInput { - dataset: dataset.to_owned(), - model: builtin_model_name(model), - num_samples, - max_workers, - })?; - crate::db::set_run_input(db, run_id, &input).await -} - -/// Derive the display name for a builtin model. -/// The built-in `Random` sampler is always reported as `demo-builtin`. -fn builtin_model_name(model: Option<&Sampler>) -> String { - match model { - None | Some(Sampler::Random) => "demo-builtin".to_string(), - Some(other) => other.to_string(), - } -} diff --git a/cli/src/builtins/mod.rs b/cli/src/builtins/mod.rs index 978c581..abce446 100644 --- a/cli/src/builtins/mod.rs +++ b/cli/src/builtins/mod.rs @@ -1,12 +1,7 @@ mod common; mod custom_nocode; mod dataset_runner; -mod financebench; -mod input; mod output; -mod pubmedqa; -mod similarity; -mod simpleqa_verified; pub use custom_nocode::CustomNoCodeBuiltin; pub use custom_nocode::metrics::{ @@ -20,7 +15,7 @@ use sea_orm::DatabaseConnection; use crate::metrics_store::MetricsStore; -/// Context provided to every builtin eval execution. +/// Context provided to every native evaluation execution. pub struct BuiltinContext<'a> { pub db: &'a DatabaseConnection, pub metrics_store: &'a MetricsStore, @@ -31,25 +26,11 @@ pub struct BuiltinContext<'a> { pub quiet: bool, } -/// Trait for builtin evals that run natively inside the CLI. +/// Trait for evaluations that run natively inside the CLI. #[async_trait] pub trait BuiltinWorkflow: Send + Sync { - /// Unique name of the builtin (e.g. "simpleqa-verified", "mmlu-pro", etc...). + /// Unique name of the native workflow. fn name(&self) -> String; - /// Execute the builtin eval and persist its metrics/output. + /// Execute the native evaluation and persist its metrics/output. async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()>; } - -/// Try to resolve a builtin by its name. -#[must_use] -pub fn resolve(name: &str) -> Option> { - if name == financebench::FinancebenchBuiltin.name() { - Some(Box::new(financebench::FinancebenchBuiltin)) - } else if name == pubmedqa::PubmedqaBuiltin.name() { - Some(Box::new(pubmedqa::PubmedqaBuiltin)) - } else if name == simpleqa_verified::SimpleqaVerifiedBuiltin.name() { - Some(Box::new(simpleqa_verified::SimpleqaVerifiedBuiltin)) - } else { - None - } -} diff --git a/cli/src/builtins/pubmedqa/config.rs b/cli/src/builtins/pubmedqa/config.rs deleted file mode 100644 index 8577c40..0000000 --- a/cli/src/builtins/pubmedqa/config.rs +++ /dev/null @@ -1,25 +0,0 @@ -use serde::{Deserialize, Serialize}; - -use crate::builtins::common::BuiltinConfig; - -/// Parsed user input for the builtin. -/// -/// All common fields (`limit`, `model`, `max_workers`) live in [`BuiltinConfig`] -/// and are flattened during deserialization so that the TOML surface stays flat. -#[derive(Debug, Default, Deserialize)] -pub(crate) struct PubMedQAConfig { - #[serde(flatten)] - pub(crate) base: BuiltinConfig, -} - -/// Per-row step output stored as JSON in the step record. -#[derive(Debug, Serialize, Deserialize)] -pub(crate) struct RowOutput { - pub(crate) sample_id: String, - pub(crate) question: String, - pub(crate) context: String, - pub(crate) gold_answer: String, - pub(crate) prediction: Option, - pub(crate) is_correct: bool, - pub(crate) model_response: String, -} diff --git a/cli/src/builtins/pubmedqa/data.rs b/cli/src/builtins/pubmedqa/data.rs deleted file mode 100644 index 43df2f7..0000000 --- a/cli/src/builtins/pubmedqa/data.rs +++ /dev/null @@ -1,362 +0,0 @@ -use std::collections::hash_map::DefaultHasher; -use std::hash::{Hash, Hasher}; - -use anyhow::{Result, bail}; -use serde_json::Value; - -use crate::builtins::common::extract_text; - -/// A normalized `PubMedQA` row after messy upstream data is coerced. -#[derive(Debug)] -pub(crate) struct PubmedQARow { - pub(crate) sample_id: String, - pub(crate) question: String, - pub(crate) context: String, - pub(crate) gold_answer: String, -} - -/// Transform a raw HF row into a canonical `PubmedQARow`. -pub(crate) fn transform_pubmedqa_row(raw: &Value) -> Result { - let question = extract_text(raw, "question") - .or_else(|| extract_text(raw, "query")) - .or_else(|| extract_text(raw, "prompt")) - .or_else(|| extract_text(raw, "input")) - .unwrap_or_default(); - - let context = extract_context(raw); - - let gold_answer = normalize_label(raw.get("final_decision")) - .or_else(|| normalize_label(raw.get("finalDecision"))) - .or_else(|| normalize_label(raw.get("answer"))) - .or_else(|| normalize_label(raw.get("label"))) - .or_else(|| normalize_label(raw.get("target"))); - - if question.is_empty() || gold_answer.is_none() { - bail!("missing question or gold_answer"); - } - - let sample_id = extract_text(raw, "id") - .or_else(|| extract_text(raw, "qid")) - .or_else(|| extract_text(raw, "question_id")) - .unwrap_or_else(|| { - let mut hasher = DefaultHasher::new(); - format!("{}|{}|{}", question, context, gold_answer.as_ref().unwrap()).hash(&mut hasher); - format!("{:016x}", hasher.finish()) - }); - - Ok(PubmedQARow { - sample_id, - question, - context, - gold_answer: gold_answer.unwrap(), - }) -} - -/// Coerce arbitrary JSON into a plain string (handles nested lists/objects). -fn coerce_text(value: &Value) -> String { - match value { - Value::String(s) => s.trim().to_string(), - Value::Array(arr) => arr - .iter() - .map(coerce_text) - .filter(|s| !s.is_empty()) - .collect::>() - .join("\n"), - Value::Object(map) => map - .values() - .map(coerce_text) - .filter(|s| !s.is_empty()) - .collect::>() - .join("\n"), - _ => String::new(), - } -} - -/// Extract the context field, trying multiple upstream column names and nested structures. -fn extract_context(row: &Value) -> String { - let direct_keys = ["context", "abstract", "passage", "long_answer", "evidence"]; - for key in &direct_keys { - if let Some(text) = extract_text(row, key) - && !text.is_empty() - { - return text; - } - } - - if let Some(context) = row.get("context") { - let text = coerce_ordered_object(context, &["contexts", "labels", "meshes"]); - if !text.is_empty() { - return text; - } - } - - if let Some(contexts) = row.get("contexts") { - let text = coerce_ordered_object( - contexts, - &["label", "contexts", "context", "abstract", "title"], - ); - if !text.is_empty() { - return text; - } - } - - String::new() -} - -fn coerce_ordered_object(value: &Value, ordered_keys: &[&str]) -> String { - match value { - Value::Array(arr) => arr - .iter() - .map(coerce_text) - .filter(|s| !s.is_empty()) - .collect::>() - .join("\n"), - Value::Object(map) => { - let mut parts = Vec::new(); - for key in ordered_keys { - if let Some(v) = map.get(*key) { - let text = coerce_text(v); - if !text.is_empty() { - parts.push(text); - } - } - } - for (key, value) in map { - if !ordered_keys.contains(&key.as_str()) { - let text = coerce_text(value); - if !text.is_empty() { - parts.push(text); - } - } - } - parts.join("\n") - } - _ => String::new(), - } -} - -/// Normalize a raw label value to yes/no/maybe. -fn normalize_label(value: Option<&Value>) -> Option { - let s = value?.as_str()?; - let normalized = s.trim().to_lowercase(); - if ["yes", "no", "maybe"].contains(&normalized.as_str()) { - Some(normalized) - } else { - None - } -} - -#[cfg(test)] -mod tests { - use super::*; - use rstest::{fixture, rstest}; - use serde_json::json; - - #[fixture] - fn standard_row() -> Value { - json!({ - "question": "Is it effective?", - "context": "Study shows 90% efficacy.", - "final_decision": "yes", - "id": "q1" - }) - } - - #[fixture] - fn alias_row() -> Value { - json!({ - "query": "Does it work?", - "abstract": "Results were promising.", - "answer": "maybe", - "qid": "q2" - }) - } - - #[fixture] - fn nested_object_context_row() -> Value { - json!({ - "prompt": "Is it safe?", - "contexts": { - "label": "Primary", - "context": "No adverse events reported.", - "title": "Safety Study" - }, - "label": "yes", - "question_id": "q3" - }) - } - - #[fixture] - fn pubmedqa_context_row() -> Value { - json!({ - "question": "Do statins reduce atrial fibrillation?", - "context": { - "contexts": [ - "Preoperative statin therapy was evaluated.", - "Postoperative atrial fibrillation was less frequent." - ], - "labels": ["BACKGROUND", "RESULTS"], - "meshes": ["Atrial Fibrillation", "Hydroxymethylglutaryl-CoA Reductase Inhibitors"] - }, - "final_decision": "yes", - "pubid": 12345 - }) - } - - #[fixture] - fn list_context_row() -> Value { - json!({ - "input": "Should we use it?", - "contexts": [ - "Phase I completed.", - "Phase II in progress." - ], - "target": "no", - "id": "q4" - }) - } - - #[rstest] - fn test_transform_standard(standard_row: Value) { - let row = transform_pubmedqa_row(&standard_row).unwrap(); - assert_eq!(row.sample_id, "q1"); - assert_eq!(row.question, "Is it effective?"); - assert_eq!(row.context, "Study shows 90% efficacy."); - assert_eq!(row.gold_answer, "yes"); - } - - #[rstest] - fn test_transform_alias_fields(alias_row: Value) { - let row = transform_pubmedqa_row(&alias_row).unwrap(); - assert_eq!(row.sample_id, "q2"); - assert_eq!(row.question, "Does it work?"); - assert_eq!(row.context, "Results were promising."); - assert_eq!(row.gold_answer, "maybe"); - } - - #[rstest] - fn test_transform_nested_object_context(nested_object_context_row: Value) { - let row = transform_pubmedqa_row(&nested_object_context_row).unwrap(); - assert_eq!(row.sample_id, "q3"); - assert_eq!(row.question, "Is it safe?"); - assert_eq!( - row.context, - "Primary\nNo adverse events reported.\nSafety Study" - ); - assert_eq!(row.gold_answer, "yes"); - } - - #[rstest] - fn test_transform_pubmedqa_nested_context(pubmedqa_context_row: Value) { - let row = transform_pubmedqa_row(&pubmedqa_context_row).unwrap(); - assert_eq!(row.question, "Do statins reduce atrial fibrillation?"); - assert_eq!( - row.context, - "Preoperative statin therapy was evaluated.\n\ - Postoperative atrial fibrillation was less frequent.\n\ - BACKGROUND\n\ - RESULTS\n\ - Atrial Fibrillation\n\ - Hydroxymethylglutaryl-CoA Reductase Inhibitors" - ); - assert_eq!(row.gold_answer, "yes"); - } - - #[rstest] - fn test_transform_list_context(list_context_row: Value) { - let row = transform_pubmedqa_row(&list_context_row).unwrap(); - assert_eq!(row.sample_id, "q4"); - assert_eq!(row.question, "Should we use it?"); - assert_eq!(row.context, "Phase I completed.\nPhase II in progress."); - assert_eq!(row.gold_answer, "no"); - } - - #[rstest] - fn test_transform_missing_question() { - let raw = json!({"context": "Some context", "final_decision": "yes"}); - assert!(transform_pubmedqa_row(&raw).is_err()); - } - - #[rstest] - fn test_transform_missing_gold_answer() { - let raw = json!({"question": "Some question?", "context": "Some context"}); - assert!(transform_pubmedqa_row(&raw).is_err()); - } - - #[rstest] - fn test_transform_generates_sample_id_when_missing() { - let raw = json!({ - "question": "Q", - "context": "C", - "final_decision": "yes" - }); - let row = transform_pubmedqa_row(&raw).unwrap(); - assert!(!row.sample_id.is_empty()); - assert_eq!(row.sample_id.len(), 16); - } - - #[rstest] - #[case(json!({"context": "Direct"}), "Direct")] - #[case(json!({"abstract": "Abstract text"}), "Abstract text")] - #[case(json!({"passage": "Passage text"}), "Passage text")] - #[case(json!({"long_answer": "Long"}), "Long")] - #[case(json!({"evidence": "Evidence"}), "Evidence")] - fn test_extract_context_direct_fields(#[case] input: Value, #[case] expected: &str) { - assert_eq!(extract_context(&input), expected); - } - - #[rstest] - fn test_extract_context_list() { - let input = json!({"contexts": ["Part 1", "Part 2"]}); - assert_eq!(extract_context(&input), "Part 1\nPart 2"); - } - - #[rstest] - fn test_extract_context_nested_object() { - let input = json!({ - "contexts": { - "label": "L", - "context": "C", - "abstract": "A", - "title": "T", - "extra": "E" - } - }); - assert_eq!(extract_context(&input), "L\nC\nA\nT\nE"); - } - - #[rstest] - fn test_extract_context_nested_singular_context() { - let input = json!({ - "context": { - "contexts": ["C1", "C2"], - "labels": ["L1", "L2"], - "meshes": ["M1"], - "extra": "E" - } - }); - assert_eq!(extract_context(&input), "C1\nC2\nL1\nL2\nM1\nE"); - } - - #[rstest] - fn test_extract_context_empty() { - assert_eq!(extract_context(&json!({})), ""); - } - - #[rstest] - fn test_extract_context_prefers_direct_over_nested() { - let input = json!({"context": "Direct", "contexts": ["Nested"]}); - assert_eq!(extract_context(&input), "Direct"); - } - - #[rstest] - #[case(json!("hello"), "hello")] - #[case(json!(" hello "), "hello")] - #[case(json!(["a", "b"]), "a\nb")] - #[case(json!({"x": "a", "y": "b"}), "a\nb")] - #[case(json!(null), "")] - #[case(json!(42), "")] - fn test_coerce_text(#[case] value: Value, #[case] expected: &str) { - assert_eq!(coerce_text(&value), expected); - } -} diff --git a/cli/src/builtins/pubmedqa/eval.rs b/cli/src/builtins/pubmedqa/eval.rs deleted file mode 100644 index 08734cc..0000000 --- a/cli/src/builtins/pubmedqa/eval.rs +++ /dev/null @@ -1,91 +0,0 @@ -/// Build the single-string prompt sent to the sampler. -pub(crate) fn build_prompt(question: &str, context: &str) -> String { - format!( - "You are answering a biomedical research question.\n\ - Reply with exactly one label: yes, no, or maybe.\n\n\ - Question:\n{question}\n\n\ - Context:\n{context}\n\n\ - Answer with exactly one token: yes, no, or maybe." - ) -} - -/// Extract a `PubMedQA` label (`yes`, `no`, or `maybe`) from a model response. -/// -/// This function is intentionally tolerant of common LLM formatting variations, -/// so it can accept responses including the following: -/// -/// - "yes" -/// - "Yes." -/// - "The answer is no." -/// - "maybe, based on the abstract" -/// -/// The parser only examines the first few tokens of the response to avoid -/// accidentally matching unrelated words later in a long generation. -pub(crate) fn extract_label_from_response(content: &str) -> Option { - let lowered = content.trim().to_lowercase(); - match lowered.as_str() { - "yes" | "no" | "maybe" => { - return Some(lowered); - } - _ => {} - } - - let cleaned = lowered.trim_start_matches(|c: char| !c.is_alphabetic()); - - for token in cleaned.split_whitespace().take(5) { - let token = token.trim_matches(|c: char| !c.is_alphabetic()); - match token { - "yes" | "no" | "maybe" => { - return Some(token.to_string()); - } - _ => {} - } - } - - None -} - -#[cfg(test)] -mod tests { - use super::*; - use rstest::rstest; - - #[rstest] - #[case("yes", Some("yes"))] - #[case("no", Some("no"))] - #[case("maybe", Some("maybe"))] - #[case("Yes", Some("yes"))] - #[case("NO", Some("no"))] - #[case("Maybe", Some("maybe"))] - #[case("yes.", Some("yes"))] - #[case("no,", Some("no"))] - #[case("(maybe)", Some("maybe"))] - #[case("The answer is yes.", Some("yes"))] - #[case("I think no.", Some("no"))] - #[case("It is maybe, based on evidence.", Some("maybe"))] - #[case(" \n\nyes", Some("yes"))] - #[case("\t no", Some("no"))] - #[case("\"yes\"", Some("yes"))] - #[case("**maybe**", Some("maybe"))] - #[case("aB3fG9kL2m", None)] - #[case("hello world", None)] - #[case("", None)] - #[case("one two three four five yes", None)] - fn test_extract_label_from_response(#[case] input: &str, #[case] expected: Option<&str>) { - assert_eq!( - extract_label_from_response(input), - expected.map(String::from) - ); - } - - #[rstest] - #[case("Q1", "C1")] - #[case("A longer question?", "A longer context.")] - #[case("", "")] - fn test_build_prompt_contains_parts(#[case] question: &str, #[case] context: &str) { - let prompt = build_prompt(question, context); - assert!(prompt.contains("yes, no, or maybe")); - assert!(prompt.contains(question)); - assert!(prompt.contains(context)); - } -} diff --git a/cli/src/builtins/pubmedqa/mod.rs b/cli/src/builtins/pubmedqa/mod.rs deleted file mode 100644 index 4385097..0000000 --- a/cli/src/builtins/pubmedqa/mod.rs +++ /dev/null @@ -1,156 +0,0 @@ -use std::sync::Arc; - -use anyhow::{Context, Result, bail}; - -use crate::builtins::common::{ - emit_accuracy_metrics, get_max_workers, hash_input, resolve_sampler, run_timed_step, -}; -use crate::builtins::dataset_runner::DatasetRunner; -use crate::builtins::input::set_builtin_run_input; -use crate::builtins::output::set_builtin_run_output; -use crate::builtins::{BuiltinContext, BuiltinWorkflow}; -use crate::dataset::{DatasetManager, resolve_hf_dataset_source}; -use crate::llm::random_label::RandomLabelSampler; - -use config::{PubMedQAConfig, RowOutput}; -use data::transform_pubmedqa_row; -use eval::{build_prompt, extract_label_from_response}; - -mod config; -mod data; -mod eval; - -/// `PubMedQA` builtin using the quantiles/PubMedQA dataset. -pub struct PubmedqaBuiltin; - -const DEFAULT_DATASET_SOURCE: &str = "hf://quantiles/PubMedQA"; - -#[expect(clippy::too_many_lines)] -#[async_trait::async_trait] -impl BuiltinWorkflow for PubmedqaBuiltin { - fn name(&self) -> String { - "pubmedqa".to_string() - } - - async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()> { - let config: PubMedQAConfig = ctx - .input - .map(serde_json::from_str) - .transpose() - .context("invalid builtin input JSON")? - .unwrap_or_default(); - - if config.base.limit == Some(0) { - bail!("limit must be > 0"); - } - - let max_workers = config.base.max_workers.unwrap_or_else(get_max_workers); - - let llm = resolve_sampler(config.base.model.as_ref(), || { - Arc::new(RandomLabelSampler::new(&["yes", "no", "maybe"])) - })?; - - let manager = DatasetManager::new()?; - let dataset_source = config - .base - .dataset - .as_deref() - .unwrap_or(DEFAULT_DATASET_SOURCE); - let dataset_id = resolve_hf_dataset_source(dataset_source)?; - let info = manager - .init(dataset_id, Some("pqa_labeled"), Some("train"), None) - .await?; - - let total = info - .total_rows - .context("could not determine dataset size; pass an explicit limit")?; - let limit = config.base.limit.unwrap_or(total).min(total); - - set_builtin_run_input( - ctx.db, - ctx.run_id, - dataset_source, - config.base.model.as_ref(), - limit, - config.base.max_workers, - ) - .await?; - - let db = ctx.db.clone(); - let model = config.base.model.clone(); - let run_id = ctx.run_id; - - let name = self.name(); - let results = DatasetRunner::new(&manager, dataset_id, &info, limit) - .desc(&name) - .set_quiet(ctx.quiet) - .for_each_concurrent(max_workers, move |i, row| { - let llm = Arc::clone(&llm); - let db = db.clone(); - let model = model.clone(); - async move { - let row = transform_pubmedqa_row(&row) - .with_context(|| format!("row {i}: invalid row data"))?; - - let prompt = build_prompt(&row.question, &row.context); - let model_str = model - .as_ref() - .map_or("random_label".to_string(), std::string::ToString::to_string); - let input_hash = hash_input(&format!( - "{prompt}\nmodel={model_str}\nsampler=pubmedqa-random-label-v1" - )); - let step_key = format!("eval-{}", row.sample_id); - - let (output, step_id) = run_timed_step( - &db, - ctx.metrics_store, - run_id, - &step_key, - &input_hash, - async { - let model_response = llm - .sample(&prompt) - .await - .with_context(|| format!("failed to sample LLM for row {i}"))?; - - let prediction = extract_label_from_response(&model_response); - let is_correct = prediction.as_ref() == Some(&row.gold_answer); - - Ok(RowOutput { - sample_id: row.sample_id.clone(), - question: row.question, - context: row.context, - gold_answer: row.gold_answer, - prediction, - is_correct, - model_response, - }) - }, - ) - .await?; - - if let Some(step_id) = step_id { - ctx.metrics_store - .emit( - ctx.run_id, - Some(step_id), - "is_correct", - if output.is_correct { 1.0 } else { 0.0 }, - None, - ) - .await; - } - - Ok::<_, anyhow::Error>(output.is_correct) - } - }) - .await?; - - let total_count = results.len(); - emit_accuracy_metrics(ctx.metrics_store, ctx.run_id, results).await; - - set_builtin_run_output(ctx.db, ctx.run_id, total_count).await?; - - Ok(()) - } -} diff --git a/cli/src/builtins/similarity.rs b/cli/src/builtins/similarity.rs deleted file mode 100644 index acc5e61..0000000 --- a/cli/src/builtins/similarity.rs +++ /dev/null @@ -1,222 +0,0 @@ -use anyhow::{Context, Result, bail}; -use serde::{Deserialize, Serialize}; -use std::sync::Arc; - -use crate::builtins::common::{ - compute_statistics, extract_text, get_max_workers, hash_input, resolve_sampler, run_timed_step, -}; -use crate::builtins::dataset_runner::DatasetRunner; -use crate::builtins::input::set_builtin_run_input; -use crate::builtins::output::set_builtin_run_output; -use crate::builtins::{BuiltinContext, BuiltinWorkflow}; - -use crate::dataset::{DatasetManager, resolve_hf_dataset_source}; -use crate::llm::random::RandomSampler; -use crate::similarity::{SimilarityMetricName, build_similarity_metric}; - -/// Configuration shared by all similarity-based builtins. -/// -/// All common fields (`limit`, `model`, `max_workers`) live in [`BuiltinConfig`] -/// and are flattened during deserialization so that the TOML surface stays flat. -#[derive(Debug, Default, Deserialize)] -struct SimilarityConfig { - #[serde(flatten)] - base: crate::builtins::common::BuiltinConfig, - /// Similarity metric name. Defaults to `cosine`. - #[serde(default)] - metric: SimilarityMetricName, -} - -/// Per-row step output stored as JSON in the step record. -#[derive(Debug, Serialize, Deserialize)] -struct RowOutput { - input: String, - response: String, - target: String, - similarity_name: String, - similarity_score: f64, -} - -/// Parameterised builtin for benchmarks that score LLM responses with a -/// similarity metric. -#[derive(Clone, Copy)] -pub struct SimilarityBenchmark { - name: &'static str, - dataset_source: &'static str, - input_field: &'static str, - target_field: &'static str, -} - -/// `simpleqa-verified` builtin. -pub const SIMPLEQA: SimilarityBenchmark = SimilarityBenchmark { - name: "simpleqa-verified", - dataset_source: "hf://quantiles/simpleqa-verified", - input_field: "problem", - target_field: "answer", -}; - -/// `financebench` builtin. -pub const FINANCEBENCH: SimilarityBenchmark = SimilarityBenchmark { - name: "financebench", - dataset_source: "hf://quantiles/financebench", - input_field: "question", - target_field: "answer", -}; - -#[expect(clippy::too_many_lines)] -#[async_trait::async_trait] -impl BuiltinWorkflow for SimilarityBenchmark { - fn name(&self) -> String { - self.name.to_string() - } - - async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()> { - let config: SimilarityConfig = ctx - .input - .map(serde_json::from_str) - .transpose() - .context("invalid builtin input JSON")? - .unwrap_or_default(); - - if config.base.limit == Some(0) { - bail!("limit must be > 0"); - } - - let metric = build_similarity_metric(config.metric)?; - - let llm = resolve_sampler(config.base.model.as_ref(), || { - Arc::new(RandomSampler::new(80)) - })?; - - let manager = DatasetManager::new()?; - let dataset_source = config - .base - .dataset - .as_deref() - .unwrap_or(self.dataset_source); - let dataset_id = resolve_hf_dataset_source(dataset_source)?; - let info = manager.init(dataset_id, None, None, None).await?; - - let total = info - .total_rows - .context("could not determine dataset size; pass an explicit limit")?; - let limit = config.base.limit.unwrap_or(total).min(total); - - let db = ctx.db; - let run_id = ctx.run_id; - - set_builtin_run_input( - db, - run_id, - dataset_source, - config.base.model.as_ref(), - limit, - config.base.max_workers, - ) - .await?; - - let metric_name = config.metric.to_string(); - let llm = &llm; - let metric = &metric; - let max_workers = config.base.max_workers.unwrap_or_else(get_max_workers); - - let scores = DatasetRunner::new(&manager, dataset_id, &info, limit) - .desc(self.name) - .set_quiet(ctx.quiet) - .for_each_concurrent(max_workers, |i, row| { - let metric_name = metric_name.clone(); - async move { - let input = extract_text(&row, self.input_field) - .with_context(|| format!("row {i}: missing '{}'", self.input_field))?; - let target = extract_text(&row, self.target_field) - .with_context(|| format!("row {i}: missing '{}'", self.target_field))?; - - let input_hash = hash_input(&input); - let step_key = format!("row-{i}"); - - let (output, step_id) = run_timed_step( - db, - ctx.metrics_store, - run_id, - &step_key, - &input_hash, - async { - let response = llm - .sample(&input) - .await - .with_context(|| format!("failed to sample LLM for row {i}"))?; - - let similarity_score = - metric.compute(&response, &target).await.with_context(|| { - format!("failed to compute similarity for row {i}") - })?; - - Ok(RowOutput { - input: input.clone(), - response, - target: target.clone(), - similarity_name: metric_name.clone(), - similarity_score, - }) - }, - ) - .await?; - - if let Some(step_id) = step_id { - ctx.metrics_store - .emit( - ctx.run_id, - Some(step_id), - "similarity_score", - output.similarity_score, - None, - ) - .await; - } - - Ok(output.similarity_score) - } - }) - .await?; - - // Emit aggregate metrics - if !scores.is_empty() { - let stats = compute_statistics(&scores); - - ctx.metrics_store - .emit(ctx.run_id, None, "mean_similarity", stats.mean, None) - .await; - ctx.metrics_store - .emit(ctx.run_id, None, "stdev_similarity", stats.std, None) - .await; - ctx.metrics_store - .emit( - ctx.run_id, - None, - "variance_similarity", - stats.variance, - None, - ) - .await; - ctx.metrics_store - .emit(ctx.run_id, None, "median_similarity", stats.median, None) - .await; - ctx.metrics_store - .emit(ctx.run_id, None, "min_similarity", stats.min, None) - .await; - ctx.metrics_store - .emit(ctx.run_id, None, "max_similarity", stats.max, None) - .await; - ctx.metrics_store - .emit(ctx.run_id, None, "p99_similarity", stats.p99, None) - .await; - ctx.metrics_store - .emit(ctx.run_id, None, "p95_similarity", stats.p95, None) - .await; - } - - set_builtin_run_output(ctx.db, ctx.run_id, scores.len()).await?; - - Ok(()) - } -} diff --git a/cli/src/builtins/simpleqa_verified/mod.rs b/cli/src/builtins/simpleqa_verified/mod.rs deleted file mode 100644 index 388497c..0000000 --- a/cli/src/builtins/simpleqa_verified/mod.rs +++ /dev/null @@ -1 +0,0 @@ -pub use crate::builtins::similarity::SIMPLEQA as SimpleqaVerifiedBuiltin; diff --git a/cli/src/commands/resume.rs b/cli/src/commands/resume.rs index 1b2b3f0..a1ee4b7 100644 --- a/cli/src/commands/resume.rs +++ b/cli/src/commands/resume.rs @@ -11,7 +11,7 @@ use qt::metrics_store::MetricsStore; /// Result of planning how to resume a run. #[derive(Debug)] pub(crate) enum ResumePlan { - Builtin, + CustomNoCode, CustomCode(Vec), RemoteBenchmark, } @@ -52,20 +52,13 @@ pub(crate) fn plan_resume( qt::config::BenchmarkConfig::CustomCode(c) => { Ok(ResumePlan::CustomCode(c.command.clone())) } - qt::config::BenchmarkConfig::Builtin(_) - | qt::config::BenchmarkConfig::CustomNoCode(_) => Ok(ResumePlan::Builtin), - } - } - None => { - if builtins::resolve(workflow_name).is_some() { - Ok(ResumePlan::Builtin) - } else { - bail!( - "no config section found for benchmark `{workflow_name}`; \ - cannot resume custom eval without config" - ); + qt::config::BenchmarkConfig::CustomNoCode(_) => Ok(ResumePlan::CustomNoCode), } } + None => bail!( + "no config section found for benchmark `{workflow_name}`; \ + cannot resume custom eval without config" + ), } } @@ -219,27 +212,20 @@ async fn execute_resume_plan(args: ExecuteResumeArgs<'_>) -> Result<()> { process_start, } = args; match plan { - ResumePlan::Builtin => { - let builtin: Box = match bench_config { - Some(qt::config::BenchmarkConfig::CustomNoCode(_)) => { - Box::new(builtins::CustomNoCodeBuiltin::new(workflow_name.to_owned())) - } - _ => builtins::resolve(workflow_name) - .with_context(|| format!("builtin `{workflow_name}` not found"))?, - }; - let custom_nocode_input = match bench_config { - Some(qt::config::BenchmarkConfig::CustomNoCode(config)) => { - Some(super::run::assemble_custom_nocode_input(config, None)?) - } - _ => None, + ResumePlan::CustomNoCode => { + let Some(qt::config::BenchmarkConfig::CustomNoCode(config)) = bench_config else { + unreachable!("custom no-code resume plan requires custom no-code config"); }; + let builtin: Box = + Box::new(builtins::CustomNoCodeBuiltin::new(workflow_name.to_owned())); + let custom_nocode_input = super::run::assemble_custom_nocode_input(config, None)?; super::run::execute_builtin(super::run::ExecuteBuiltinArgs { db, metrics_store, run_id, workflow_name, builtin, - input: custom_nocode_input.as_deref().or(stored_input), + input: Some(&custom_nocode_input), json, process_start, remote_hash: None, @@ -301,40 +287,16 @@ mod tests { /// begins, because a completed run cannot be meaningfully resumed. #[test] fn plan_resume_completed_run_errors() { - let bench = qt::config::BenchmarkConfig::Builtin(qt::config::BuiltinBenchmarkConfig { - type_: "builtin".to_owned(), - samples: None, - dataset: "hf://quantiles/PubMedQA".to_owned(), - model: None, - max_workers: None, - }); + let bench = + qt::config::BenchmarkConfig::CustomCode(qt::config::CustomCodeBenchmarkConfig { + type_: "custom_code".to_owned(), + command: vec!["python".to_owned(), "eval.py".to_owned()], + input: None, + }); let err = plan_resume("demo", &RunStatus::Completed, Some(&bench), None).unwrap_err(); assert!(err.to_string().contains("already completed")); } - /// A builtin benchmark with a valid config section should plan to resume as a builtin, - /// using the stored input from the database. - #[test] - fn plan_resume_builtin_with_config() { - let bench = qt::config::BenchmarkConfig::Builtin(qt::config::BuiltinBenchmarkConfig { - type_: "builtin".to_owned(), - samples: Some(10), - dataset: "hf://quantiles/PubMedQA".to_owned(), - model: None, - max_workers: None, - }); - let plan = plan_resume("demo", &RunStatus::Failed, Some(&bench), None).unwrap(); - assert!(matches!(plan, ResumePlan::Builtin)); - } - - /// A builtin benchmark that has no config section can still be resumed by name lookup, - /// falling back to the hardcoded builtin registry. - #[test] - fn plan_resume_builtin_without_config() { - let plan = plan_resume("pubmedqa", &RunStatus::Failed, None, None).unwrap(); - assert!(matches!(plan, ResumePlan::Builtin)); - } - #[test] fn plan_resume_prefers_persisted_remote_provenance_over_builtin_name() { let provenance = qt::db::RemoteBenchmarkProvenance { @@ -378,7 +340,7 @@ mod tests { assert!(err.to_string().contains("no config section found")); } - /// An unknown workflow name with neither a config section nor a builtin match must + /// An unknown workflow name with no config section must /// fail immediately with a clear "no config section found" message. #[test] fn plan_resume_unknown_without_config_errors() { @@ -400,7 +362,7 @@ mod tests { assert!(err.to_string().contains("non-empty `command`")); } - /// A `custom_nocode` benchmark should plan to resume as a builtin so that the + /// A `custom_nocode` benchmark should plan to resume natively so that the /// CLI can re-run the no-code workflow natively without spawning an external command. #[test] fn plan_resume_custom_nocode_with_config() { @@ -427,7 +389,7 @@ mod tests { }, )); let plan = plan_resume("nocode_custom", &RunStatus::Failed, Some(&bench), None).unwrap(); - assert!(matches!(plan, ResumePlan::Builtin)); + assert!(matches!(plan, ResumePlan::CustomNoCode)); } #[tokio::test] diff --git a/cli/src/commands/run.rs b/cli/src/commands/run.rs index 323afb5..9d1ab51 100644 --- a/cli/src/commands/run.rs +++ b/cli/src/commands/run.rs @@ -52,15 +52,6 @@ pub async fn run( remote, ) .await - } else if builtins::resolve(workflow_name).is_some() { - let (effective_input, _) = assemble_builtin_input(None, cli_input); - run_builtin_workflow( - workflow_name, - effective_input.as_deref(), - json, - process_start, - ) - .await } else { bail!("no config section found for benchmark `{workflow_name}`"); } @@ -77,16 +68,6 @@ async fn run_configured_benchmark( bench: &qt::config::BenchmarkConfig, ) -> Result<()> { match bench { - qt::config::BenchmarkConfig::Builtin(config) => { - let (effective_input, _) = assemble_builtin_input(Some(config), cli_input); - run_builtin_workflow( - workflow_name, - effective_input.as_deref(), - json, - process_start, - ) - .await - } qt::config::BenchmarkConfig::CustomCode(config) => { let (merged_input, overridden_keys) = merge_inputs(config.input.as_ref(), cli_input)?; let warning = (!overridden_keys.is_empty()).then(|| { @@ -232,31 +213,6 @@ pub(super) fn remote_benchmark_builtin( }) } -fn assemble_builtin_input( - bench: Option<&qt::config::BuiltinBenchmarkConfig>, - cli_input: Option<&str>, -) -> (Option, Vec) { - if let Some(cli_str) = cli_input { - return (Some(cli_str.to_owned()), Vec::new()); - } - - if let Some(bench) = bench { - let input = BuiltinConfigInput { - limit: bench.samples, - dataset: bench.dataset.clone(), - model: bench.model.clone(), - max_workers: bench.max_workers, - }; - - let json = - serde_json::to_string(&input).expect("infallible serialization of BuiltinConfigInput"); - - (Some(json), Vec::new()) - } else { - (None, Vec::new()) - } -} - /// Serialize a custom no-code configuration to JSON after applying supported overrides /// given on the command line from the `--input` flag in the `cli_input` parameter. /// @@ -338,39 +294,6 @@ fn merge_inputs( } } -async fn run_builtin_workflow( - workflow_name: &str, - input: Option<&str>, - json: bool, - process_start: Instant, -) -> Result<()> { - let builtin = builtins::resolve(workflow_name) - .with_context(|| format!("builtin `{workflow_name}` not found"))?; - - let cwd = std::env::current_dir()?; - let root = db::resolve_workspace_root(&cwd, true).await?; - let db = db::open_workspace(&root).await?; - let metrics_store = MetricsStore::new(db::metrics_dir(&root))?; - - let run_id = db::create_run(&db, workflow_name, input).await?; - if !json { - println!("Created run {run_id}"); - } - - execute_builtin(ExecuteBuiltinArgs { - db: &db, - metrics_store: &metrics_store, - run_id, - workflow_name, - builtin, - input, - json, - process_start, - remote_hash: None, - }) - .await -} - /// Arguments for the [`execute_builtin`] function. pub struct ExecuteBuiltinArgs<'a> { pub db: &'a DatabaseConnection, @@ -590,18 +513,6 @@ struct BuiltinRunJsonOutput<'a> { remote_hash: Option<&'a str>, } -/// Config input shape auto-generated from `quantiles.toml` `[benchmarks.*]`. -#[derive(Serialize, Default)] -struct BuiltinConfigInput { - #[serde(skip_serializing_if = "Option::is_none")] - limit: Option, - dataset: String, - #[serde(skip_serializing_if = "Option::is_none")] - model: Option, - #[serde(skip_serializing_if = "Option::is_none")] - max_workers: Option, -} - /// Captured result of running the user command. struct ChildCommandOutput { /// Process exit status reported by the operating system. @@ -858,64 +769,6 @@ mod tests { assert!(overridden.is_empty()); } - /// A `--input` CLI flag should take precedence over any config fields, returning the - /// raw CLI string directly without assembling a config-based JSON object. - #[test] - fn assemble_builtin_input_with_cli_override() { - let bench = qt::config::BuiltinBenchmarkConfig { - type_: "builtin".to_owned(), - samples: Some(10), - dataset: "hf://quantiles/PubMedQA".to_owned(), - model: None, - max_workers: None, - }; - let (input, _) = super::assemble_builtin_input(Some(&bench), Some(r#"{"model":"x"}"#)); - assert_eq!(input, Some(r#"{"model":"x"}"#.to_owned())); - } - - /// When no `--input` is given but the config has builtin fields, they should be - /// assembled into a `BuiltinConfigInput` JSON object with the correct key names. - #[test] - fn assemble_builtin_input_from_config() { - let bench = qt::config::BuiltinBenchmarkConfig { - type_: "builtin".to_owned(), - samples: Some(5), - dataset: "hf://quantiles/PubMedQA".to_owned(), - model: Some(qt::llm::Sampler::Random {}), - max_workers: Some(8), - }; - let (input, _) = super::assemble_builtin_input(Some(&bench), None); - let parsed: serde_json::Value = serde_json::from_str(&input.unwrap()).unwrap(); - assert_eq!(parsed["limit"], 5); - assert_eq!(parsed["dataset"], "hf://quantiles/PubMedQA"); - assert_eq!(parsed["model"], "random"); - assert_eq!(parsed["max_workers"], 8); - } - - /// When the builtin config section only has the required dataset, the input should - /// still carry that dataset source into builtin execution. - #[test] - fn assemble_builtin_input_with_dataset_only_config() { - let bench = qt::config::BuiltinBenchmarkConfig { - type_: "builtin".to_owned(), - samples: None, - dataset: "hf://quantiles/PubMedQA".to_owned(), - model: None, - max_workers: None, - }; - let (input, _) = super::assemble_builtin_input(Some(&bench), None); - let parsed: serde_json::Value = serde_json::from_str(&input.unwrap()).unwrap(); - assert_eq!(parsed["dataset"], "hf://quantiles/PubMedQA"); - } - - /// When there is no config section at all and no CLI `--input`, builtin runs should - /// proceed with no input JSON stored in the database. - #[test] - fn assemble_builtin_input_none_when_no_bench() { - let (input, _) = super::assemble_builtin_input(None, None); - assert!(input.is_none()); - } - /// A `custom_nocode` benchmark config with all fields should serialize into the /// expected JSON shape, converting field names faithfully. #[test] diff --git a/cli/src/config/mod.rs b/cli/src/config/mod.rs index 686df37..b862fdb 100644 --- a/cli/src/config/mod.rs +++ b/cli/src/config/mod.rs @@ -3,15 +3,12 @@ use std::collections::HashMap; use anyhow::{Context, Result, bail}; use serde::{Deserialize, Deserializer}; -use crate::llm::Sampler; - /// Configuration for a single benchmark. /// /// Exactly one of the variants is deserialized based on the `type` field: -/// `builtin` (default when absent), `custom_code`, or `custom_nocode`. +/// `custom_code` or `custom_nocode`. #[derive(Debug, Clone)] pub enum BenchmarkConfig { - Builtin(BuiltinBenchmarkConfig), CustomCode(CustomCodeBenchmarkConfig), CustomNoCode(Box), } @@ -24,7 +21,6 @@ impl BenchmarkConfig { /// Returns an error when a field has an invalid value. pub fn validate(&self) -> Result<()> { match self { - BenchmarkConfig::Builtin(_) => Ok(()), BenchmarkConfig::CustomCode(c) => { if c.command.is_empty() { bail!("custom_code benchmark config must have a non-empty `command` field"); @@ -73,41 +69,16 @@ impl<'de> Deserialize<'de> for BenchmarkConfig { })?; Ok(BenchmarkConfig::CustomNoCode(Box::new(config))) } - Some("builtin") | None => { - let config = BuiltinBenchmarkConfig::deserialize(value).map_err(|e| { - serde::de::Error::custom(format!( - "failed to deserialize builtin benchmark config: {e}" - )) - })?; - Ok(BenchmarkConfig::Builtin(config)) - } Some(other) => Err(serde::de::Error::custom(format!( - "invalid benchmark type `{other}`; expected `builtin`, `custom_code`, or `custom_nocode`", + "invalid benchmark type `{other}`; expected `custom_code` or `custom_nocode`", ))), + None => Err(serde::de::Error::custom( + "benchmark config requires a `type` field; expected `custom_code` or `custom_nocode`", + )), } } } -/// Built-in benchmark configuration. -#[derive(Debug, Deserialize, Clone)] -#[serde(deny_unknown_fields)] -pub struct BuiltinBenchmarkConfig { - #[serde(default = "default_type_builtin", rename = "type")] - pub type_: String, - /// Number of samples (rows) to evaluate. - pub samples: Option, - /// Dataset source for this benchmark. - pub dataset: String, - /// Which model sampler to use for this benchmark. - pub model: Option, - /// Maximum number of concurrent workers for this benchmark. - pub max_workers: Option, -} - -fn default_type_builtin() -> String { - "builtin".to_owned() -} - /// Custom-code benchmark configuration. #[derive(Debug, Deserialize, Clone)] #[serde(deny_unknown_fields)] @@ -176,30 +147,22 @@ pub fn load() -> Result { } #[cfg(test)] -#[expect(clippy::needless_raw_string_hashes)] mod tests { use super::*; #[test] - fn deserialize_builtin_without_type() { + fn benchmark_without_type_errors() { let toml = r#" [benchmarks.demo] dataset = "hf://quantiles/demo" samples = 10 "#; - let config: WorkspaceConfig = toml::from_str(toml).unwrap(); - let bench = config.benchmarks.get("demo").unwrap(); - assert!(matches!(bench, BenchmarkConfig::Builtin(_))); - if let BenchmarkConfig::Builtin(b) = bench { - assert_eq!(b.type_, "builtin"); - assert_eq!(b.samples, Some(10)); - assert_eq!(b.dataset, "hf://quantiles/demo"); - assert!(b.model.is_none()); - } + let result: Result = toml::from_str(toml); + assert!(result.is_err()); } #[test] - fn deserialize_builtin_with_explicit_type() { + fn builtin_type_errors() { let toml = r#" [benchmarks.demo] type = "builtin" @@ -207,22 +170,8 @@ mod tests { dataset = "hf://quantiles/demo" model = "openai:gpt-4" "#; - let config: WorkspaceConfig = toml::from_str(toml).unwrap(); - let bench = config.benchmarks.get("demo").unwrap(); - assert!(matches!(bench, BenchmarkConfig::Builtin(_))); - if let BenchmarkConfig::Builtin(b) = bench { - assert_eq!(b.dataset, "hf://quantiles/demo"); - } - } - - #[test] - fn builtin_requires_dataset_field() { - let toml = r#" - [benchmarks.demo] - samples = 5 - "#; let result: Result = toml::from_str(toml); - assert!(result.is_err(), "builtin should require dataset field"); + assert!(result.is_err()); } #[test] @@ -263,30 +212,6 @@ mod tests { } } - #[test] - fn builtin_rejects_command_field() { - let toml = r#" - [benchmarks.demo] - type = "builtin" - command = ["echo", "hello"] - "#; - let result: Result = toml::from_str(toml); - assert!(result.is_err(), "builtin should reject command field"); - } - - #[test] - fn builtin_rejects_input_field() { - let toml = r#" - [benchmarks.demo] - type = "builtin" - - [benchmarks.demo.input] - foo = "bar" - "#; - let result: Result = toml::from_str(toml); - assert!(result.is_err(), "builtin should reject input field"); - } - #[test] fn custom_code_rejects_samples_field() { let toml = r#" diff --git a/python-examples/README.md b/python-examples/README.md index b1eb7c5..cfd8aa5 100644 --- a/python-examples/README.md +++ b/python-examples/README.md @@ -4,7 +4,7 @@ This directory contains examples of building [`custom_code` evaluations](https:/ The accompanying [`quantiles.toml`](./quantiles.toml) makes these examples easily runnable with `qt`: -| Evaluation | Command | Source | Notes | -| ---------------------------------- | --------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| PubMedQA | `qt run custom_pubmedqa` | [`src/pubmedqa.py`](./src/pubmedqa.py) | Implements the [PubMedQA](https://pubmedqa.github.io/) biomedical question-answering benchmark as a custom evaluation. PubMedQA is also available as a built-in benchmark with `qt run pubmedqa`. | -| Customer-support prompt evaluation | `qt run custom_prompt_eval` | [`src/prompt_eval.py`](./src/prompt_eval.py) | Demonstrates a deterministic customer-support classification evaluation with recorded steps and metrics. | +| Evaluation | Command | Source | Notes | +| --- | --- | --- | --- | +| PubMedQA | `qt run custom_pubmedqa` | [`src/pubmedqa.py`](./src/pubmedqa.py) | Implements the [PubMedQA](https://pubmedqa.github.io/) biomedical question-answering benchmark as a custom evaluation. A registry-backed definition is also available with `qt run pubmedqa`. | +| Customer-support prompt evaluation | `qt run custom_prompt_eval` | [`src/prompt_eval.py`](./src/prompt_eval.py) | Demonstrates a deterministic customer-support classification evaluation with recorded steps and metrics. | diff --git a/python-examples/quantiles.toml b/python-examples/quantiles.toml index db336b2..553c06e 100644 --- a/python-examples/quantiles.toml +++ b/python-examples/quantiles.toml @@ -1,4 +1,4 @@ -# PubMedQA built with the Python SDK (not using the built-in benchmark) +# PubMedQA built with the Python SDK instead of the registry-backed benchmark [benchmarks.custom_pubmedqa] type = "custom_code" command = ["uv", "run", "src/pubmedqa.py"] diff --git a/python-examples/src/pubmedqa.py b/python-examples/src/pubmedqa.py index a0d7ef7..20a23bf 100644 --- a/python-examples/src/pubmedqa.py +++ b/python-examples/src/pubmedqa.py @@ -1,8 +1,8 @@ """ This file demonstrates how to use the Quantiles Python SDK to implement a new benchmark for the `qt` CLI. -It uses PubMedQA as an illustrative example, implementing the benchmark from scratch even though PubMedQA is -already available as a built-in benchmark and can be run directly with `qt run pubmedqa`. +It uses PubMedQA as an illustrative example, implementing the benchmark as custom code. A registry-backed +PubMedQA definition can also be downloaded and run directly with `qt run pubmedqa`. The steps shown in this example can be used to add new benchmarks or build custom evaluation workflows in Quantiles. """