Adding t^3-bench for testing

README.md

@@ -57,6 +57,16 @@ qt run simpleqa-verified
 
 > The command above downloads the [`simpleqa-verified`](https://quantiles.io/benchmark-hub/benchmark/simpleqa-verified) definition from the Quantiles benchmark registry and runs it locally with a demo model that generates random text. Fetching the benchmark definition and an uncached dataset requires network access, but no provider API key or paid model inference is required. Do not use demo-model results to draw conclusions about model quality.
 
+The CLI also includes `tau3-mock`, a small native conformance benchmark for
+structured tool calling, model-driven user simulation, mutable task
+environments, trajectory recording, component rewards, and `pass_at_k`. It is
+not an official τ³ leaderboard domain; see the
+[`tau3-mock` implementation notes](cli/src/builtins/tau3/README.md).
+Running `qt run tau3-mock` uses a local deterministic structured-tool demo model
+to validate the harness without provider calls.
+The `tau3-airline` name is reserved for the forthcoming airline implementation
+and currently returns a not-implemented error.
+
 Inspect the recorded run:
 
 ```bash

cli/README.md

@@ -36,6 +36,8 @@ See the [CLI reference](https://quantiles.io/documentation/reference/cli) for a
 The CLI supports two locally configured evaluation types and remote registry benchmarks:
 
 - [`custom_nocode` evaluations](https://quantiles.io/documentation/custom-evaluations/custom-nocode-evaluations) define the dataset, prompt template, model, and scoring method entirely in configuration. Supported scoring styles include exact match, multiple choice, and text similarity.
+- `tau3-mock` is a native structured-tool-calling conformance benchmark for the in-process τ³-style agent harness. With no model configured, it uses a local deterministic structured-tool demo model. It records tool trajectories, isolated environment state, component rewards, and `pass_at_k`; it is not an official τ³ leaderboard domain. See [`src/builtins/tau3/README.md`](src/builtins/tau3/README.md).
+- `tau3-airline` is reserved for the forthcoming official-compatible airline implementation and currently returns a not-implemented error.
 - [`custom_code` evaluations](https://quantiles.io/documentation/custom-evaluations) run your own Python evaluation through the Quantiles Python SDK.
 - Registry benchmarks are downloaded by name from the Quantiles remote benchmark service and executed locally using the native `custom_nocode` runtime.
 

cli/src/builtins/mod.rs

@@ -2,6 +2,7 @@ mod common;
 mod custom_nocode;
 mod dataset_runner;
 mod output;
+mod tau3;
 
 pub use custom_nocode::CustomNoCodeBuiltin;
 pub use custom_nocode::metrics::{
@@ -34,3 +35,13 @@ pub trait BuiltinWorkflow: Send + Sync {
     /// Execute the native evaluation and persist its metrics/output.
     async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()>;
 }
+
+/// Resolve an evaluation that is implemented natively inside the CLI.
+#[must_use]
+pub fn resolve(name: &str) -> Option<Box<dyn BuiltinWorkflow>> {
+    match name {
+        "tau3-mock" => Some(Box::new(tau3::Tau3MockBuiltin)),
+        "tau3-airline" => Some(Box::new(tau3::Tau3AirlineBuiltin)),
+        _ => None,
+    }
+}

cli/src/builtins/tau3/README.md

@@ -0,0 +1,50 @@
+# Native τ³ harness
+
+`tau3-mock` is a built-in conformance benchmark for Quantiles' native τ³-style
+agent harness. It exercises the parts that differ from prompt/response
+benchmarks:
+
+- provider-neutral structured tool calling;
+- a model-driven user simulator;
+- a turn-based agent/user/tool orchestrator;
+- an isolated mutable environment for every task trial;
+- durable trajectories and component rewards; and
+- aggregate `pass_at_k` metrics using the unbiased estimator.
+
+It intentionally uses a small bundled mock domain. It is not a substitute for
+the official τ³-bench 1.0.1 airline, retail, telecom, banking-knowledge, or voice
+tracks, and its scores must not be submitted to or compared with the official
+leaderboard.
+
+Run it without configuration to use the local deterministic structured-tool
+demo model for both the agent and simulated user:
+
+```console
+qt run tau3-mock
+```
+
+The demo run validates the harness without making network calls and is not
+model-quality evidence. To evaluate a provider model, pass separate agent and
+user models:
+
+```console
+qt run tau3-mock --input '{
+  "model": "openai:gpt-5.2",
+  "user_model": "openai:gpt-5.2",
+  "trials": 4,
+  "max_turns": 20
+}'
+```
+
+The built-in supports OpenAI, Anthropic, and Gemini model configurations because
+those existing Quantiles backends expose structured tool calls through `genai`.
+Remote model calls occur only when the user runs the benchmark with one of those
+models. Quantiles continues to store runs, steps, trajectories, and metrics
+locally.
+
+Official domain parity requires a separately reviewable port of the versioned
+task data, domain policies, databases, exact tool behavior, banking retrieval
+corpus and graders, and full-duplex voice orchestration. Until that work lands,
+`tau3-mock` executes the bundled conformance domain, while `tau3-airline` is
+reserved and returns a not-implemented error. The general `tau3` name and other
+official domain names do not resolve.

cli/src/builtins/tau3/environment.rs

@@ -0,0 +1,124 @@
+use anyhow::{Result, bail};
+use async_trait::async_trait;
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+
+use super::model::{ToolDefinition, ToolInvocation};
+
+#[async_trait]
+pub(crate) trait ToolEnvironment: Send {
+    fn tools(&self) -> Vec<ToolDefinition>;
+    async fn invoke(&mut self, call: &ToolInvocation) -> Result<Value>;
+    fn state(&self) -> Value;
+    fn terminated(&self) -> bool;
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+pub(crate) struct MockState {
+    pub(crate) customer_id: String,
+    pub(crate) name: String,
+    pub(crate) email: String,
+}
+
+pub(crate) struct MockEnvironment {
+    state: MockState,
+    terminated: bool,
+}
+
+impl MockEnvironment {
+    pub(crate) fn new(state: MockState) -> Self {
+        Self {
+            state,
+            terminated: false,
+        }
+    }
+}
+
+#[async_trait]
+impl ToolEnvironment for MockEnvironment {
+    fn tools(&self) -> Vec<ToolDefinition> {
+        vec![
+            ToolDefinition {
+                name: "get_customer".to_owned(),
+                description: "Look up a customer record by customer ID.".to_owned(),
+                schema: json!({
+                    "type": "object",
+                    "properties": {"customer_id": {"type": "string"}},
+                    "required": ["customer_id"],
+                    "additionalProperties": false
+                }),
+            },
+            ToolDefinition {
+                name: "update_customer_email".to_owned(),
+                description: "Update the email address on a customer record.".to_owned(),
+                schema: json!({
+                    "type": "object",
+                    "properties": {
+                        "customer_id": {"type": "string"},
+                        "email": {"type": "string"}
+                    },
+                    "required": ["customer_id", "email"],
+                    "additionalProperties": false
+                }),
+            },
+            ToolDefinition {
+                name: "transfer_to_human".to_owned(),
+                description: "End the interaction and transfer the customer to a human agent."
+                    .to_owned(),
+                schema: json!({
+                    "type": "object",
+                    "properties": {"reason": {"type": "string"}},
+                    "required": ["reason"],
+                    "additionalProperties": false
+                }),
+            },
+        ]
+    }
+
+    async fn invoke(&mut self, call: &ToolInvocation) -> Result<Value> {
+        match call.name.as_str() {
+            "get_customer" => {
+                require_customer_id(&call.arguments, &self.state.customer_id)?;
+                Ok(serde_json::to_value(&self.state)?)
+            }
+            "update_customer_email" => {
+                require_customer_id(&call.arguments, &self.state.customer_id)?;
+                let email = required_string(&call.arguments, "email")?;
+                if !email.contains('@') {
+                    bail!("email must contain @");
+                }
+                email.clone_into(&mut self.state.email);
+                Ok(json!({"status": "success", "customer": self.state}))
+            }
+            "transfer_to_human" => {
+                let _ = required_string(&call.arguments, "reason")?;
+                self.terminated = true;
+                Ok(json!({"status": "transferred"}))
+            }
+            other => bail!("unknown tool `{other}`"),
+        }
+    }
+
+    fn state(&self) -> Value {
+        serde_json::to_value(&self.state).expect("MockState is serializable")
+    }
+
+    fn terminated(&self) -> bool {
+        self.terminated
+    }
+}
+
+fn require_customer_id(arguments: &Value, expected: &str) -> Result<()> {
+    let actual = required_string(arguments, "customer_id")?;
+    if actual != expected {
+        bail!("customer `{actual}` not found");
+    }
+    Ok(())
+}
+
+fn required_string<'a>(arguments: &'a Value, key: &str) -> Result<&'a str> {
+    arguments
+        .get(key)
+        .and_then(Value::as_str)
+        .ok_or_else(|| anyhow::anyhow!("missing or invalid `{key}` argument"))
+}