Skip to content

cli-reference.md

Latest commit

 

History

History
359 lines (262 loc) · 12.4 KB

cli-reference.md

File metadata and controls

359 lines (262 loc) · 12.4 KB
Raw

CLI Reference

The tau2 command provides a unified interface for all τ-bench functionality. Use tau2 <command> --help to see full details for any command.

Run tau2 intro (or just tau2) to see an overview of available domains, commands, and a quick-start guide directly in the terminal.

tau2 run — Run Evaluations

Run agent evaluations across different communication modes.

Basic Usage

tau2 run \
  --domain <domain> \
  --agent-llm <llm_name> \
  --user-llm <llm_name> \
  --num-trials <trial_count> \
  --num-tasks <task_count>

Common Options

Option Description
--domain, -d Domain to evaluate: airline, retail, telecom, mock, banking_knowledge
--agent-llm LLM model for the agent
--user-llm LLM model for the user simulator
--agent-llm-args JSON dict of extra args for agent LLM (e.g. '{"temperature": 0.5}')
--user-llm-args JSON dict of extra args for user LLM
--agent Agent implementation to use (default: llm_agent)
--user User simulator implementation to use (default: user_simulator)
--num-trials Number of evaluation trials (default: 1)
--num-tasks Number of tasks to evaluate (omit for all tasks)
--task-ids Specific task IDs to evaluate
--task-split-name Task split to use (default: base)
--task-set-name Task set to use (default: domain default)
--max-steps Maximum simulation steps (default: 200)
--max-errors Maximum consecutive tool errors allowed (default: 10)
--max-concurrency Maximum concurrent simulations (default: 3)
--seed Random seed for reproducibility (default: 300)
--save-to Custom output directory name (saved under data/simulations/)
--log-level Log level (default: ERROR)
--verbose-logs Save detailed logs (LLM calls, audio, ticks)
--audio-debug Save per-tick audio files and timing analysis (requires --audio-native)
--llm-log-mode LLM log mode when --verbose-logs is on: all or latest (default: latest)
--max-retries Max retries for failed tasks (default: 3)
--retry-delay Delay in seconds between retries (default: 1.0)
--enforce-communication-protocol Enforce protocol rules (e.g. no mixed text + tool call messages)
--user-persona User persona config as JSON dict
--xml-prompt Force XML tags in system prompt
--no-xml-prompt Force plain text system prompt (no XML tags)
--auto-resume Automatically resume from existing save file without prompting
--auto-review Automatically run LLM conversation review after each simulation
--review-mode Review mode when --auto-review is on: full or user (default: full)
--review-model LLM model to use for review calls (default: claude-opus-4-5)
--hallucination-retries Max retries when user simulator hallucination is detected (full-duplex only, default: 3). Set to 0 to disable
--timeout Maximum wallclock time in seconds per simulation (no timeout by default)
--audio-native Enable audio native mode (voice full-duplex)
--audio-taps Save WAV files at each pipeline stage for debugging (requires --audio-native)
--retrieval-config Retrieval configuration for banking_knowledge domain (e.g., alltools, bm25, terminal_use)
--retrieval-config-kwargs JSON arguments for the retrieval config constructor (e.g., '{"top_k": 10}')

Audio Native Options

Option Default Description
--audio-native false Enable audio native mode
--audio-native-provider openai Provider: openai, gemini, xai
--audio-native-model (per-provider) Model to use (defaults to provider-specific model if not set)
--tick-duration 0.2 Tick duration in seconds (simulation timestep)
--max-steps-seconds 600 Maximum conversation duration in seconds
--speech-complexity regular Speech complexity: control, regular, or ablation variants (control_audio, control_accents, control_behavior, control_audio_accents, control_audio_behavior, control_accents_behavior)
--pcm-sample-rate 16000 User simulator PCM synthesis rate
--telephony-rate 8000 API/agent telephony rate

Turn-taking thresholds:

Option Default Description
--wait-to-respond-other 1.0 Min seconds since agent spoke before user responds
--wait-to-respond-self 5.0 Min seconds since user spoke before responding again
--yield-when-interrupted 1.0 How long user keeps speaking when agent interrupts
--yield-when-interrupting 5.0 How long user keeps speaking when interrupting agent
--interruption-check-interval 2.0 Interval for checking interruptions
--integration-duration 0.5 Integration duration for linearization
--silence-annotation-threshold 4.0 Silence threshold for annotations

Examples

# Standard text evaluation
tau2 run --domain airline --agent-llm gpt-4.1 --user-llm gpt-4.1 --num-trials 1 --num-tasks 5

# Audio native (voice full-duplex)
tau2 run --domain retail --audio-native --num-tasks 1 --verbose-logs

# Audio native with custom provider and settings
tau2 run --domain retail --audio-native --audio-native-provider gemini \
  --tick-duration 0.2 --max-steps-seconds 240 --speech-complexity control \
  --verbose-logs --save-to my_audio_native_run

# Audio native with hallucination retries disabled
tau2 run --domain retail --audio-native --hallucination-retries 0 --num-tasks 1

# Knowledge retrieval with BM25
tau2 run --domain banking_knowledge --retrieval-config bm25 \
  --agent-llm gpt-4.1 --user-llm gpt-4.1 --num-tasks 5

# Knowledge retrieval with embeddings and reranker
tau2 run --domain banking_knowledge --retrieval-config openai_embeddings_reranker \
  --agent-llm gpt-4.1 --user-llm gpt-4.1 --num-tasks 5

tau2 play — Interactive Play Mode

Experience τ-bench interactively from either perspective.

tau2 play

Play mode allows you to:

  • Play as Agent: Manually control the agent's responses and tool calls
  • Play as User: Control the user while an LLM agent handles requests (available in domains with user tools like telecom)
  • Understand tasks by walking through scenarios step-by-step
  • Test strategies before implementing them in code
  • Choose task splits to practice on training data or test on held-out tasks

See the Gym Documentation for using the gymnasium interface programmatically.

tau2 view — View Results

Browse and analyze simulation results.

tau2 view
Option Description
--dir Directory containing simulation files (defaults to data/simulations/)
--file Path to a specific results file to view
--only-show-failed Only show failed tasks
--only-show-all-failed Only show tasks that failed in all trials
--expanded-ticks Show expanded tick view (for full-duplex simulations)

tau2 domain — View Domain Documentation

View domain policy and API documentation.

tau2 domain <domain>

Then visit http://127.0.0.1:8004/redoc to see the domain policy and available tools.

tau2 check-data — Check Data Configuration

Verify that your data directory is properly configured.

tau2 check-data

tau2 start — Start All Servers

Start all domain servers.

tau2 start

tau2 evaluate-trajs — Evaluate Trajectories

Re-evaluate trajectory files and optionally update rewards.

tau2 evaluate-trajs <paths...>
Option Description
<paths> Paths to trajectory files, directories, or glob patterns
-o, --output-dir Directory to save updated trajectories. If omitted, only displays metrics

tau2 review — LLM Conversation Review

Run LLM-based review on simulation results to detect agent and/or user errors.

tau2 review <path>
Option Description
<path> Path to a results.json file or directory containing them
-m, --mode Review mode: full (agent + user, default) or user (user simulator only)
-o, --output Output path for reviewed results (single file only)
--interruption-enabled Flag indicating interruption was enabled in these simulations
--show-details Show detailed review for each simulation
-c, --max-concurrency Max concurrent reviews (default: 32)
--limit Limit review to first N simulations
--task-ids Only review simulations for these task IDs
--log-llm Log LLM request/response for each review call
--review-model LLM model to use for review calls (default: claude-opus-4-5)

tau2 convert-results — Convert Results Format

Convert simulation results between monolithic JSON and directory-based formats.

tau2 convert-results <path> [--to {json,dir}] [--no-backup]
Option Description
<path> Path to a results.json file or directory containing one
--to Target format: json (monolithic) or dir (directory with individual sim files). If omitted, converts to the opposite of the current format
--no-backup Skip creating a backup before conversion

Text runs default to monolithic JSON; voice runs default to directory-based format. Use this command to convert between them when needed.

tau2 leaderboard — View Leaderboard

Show the τ-bench leaderboard in the terminal.

tau2 leaderboard
Option Description
--domain, -d Show leaderboard for a specific domain: retail, airline, telecom, or banking_knowledge
--metric, -m Metric to rank by: pass_1, pass_2, pass_3, pass_4, cost (default: pass_1)
--limit, -n Limit the number of entries shown

tau2 submit — Leaderboard Submission

See the full Leaderboard Submission Guide.

# Prepare a submission
tau2 submit prepare <paths...> --output ./my_submission

# Prepare a voice submission (auto-detected, or force with --voice)
tau2 submit prepare <paths...> --output ./my_submission --voice

# Skip trajectory verification during preparation
tau2 submit prepare <paths...> --output ./my_submission --no-verify

# Validate a submission
tau2 submit validate <submission_dir> [--mode public|private]

# Verify trajectory files
tau2 submit verify-trajs <paths...> [--mode public|private]

Environment CLI (beta)

An interactive CLI for directly querying and testing domain environments.

make env-cli

Commands:

  • :q — quit
  • :d — change domain
  • :n — start new session (clears history)

Example:

$ make env-cli

Welcome to the Environment CLI!
Connected to airline domain.

Query (:n new session, :d change domain, :q quit)> What flights are available from SF to LA tomorrow?
Assistant: Let me check the flight availability for you...

Useful for testing domain tools, debugging environment responses, and exploring domain functionality without starting the full server stack.

Running Tests

make test              # Core tests (requires: uv sync --extra dev)
make test-voice        # Voice + streaming tests (requires: uv sync --extra dev --extra voice)
make test-knowledge    # Banking knowledge tests (requires: uv sync --extra dev --extra knowledge)
make test-gym          # Gymnasium tests (requires: uv sync --extra dev --extra gym)
make test-all          # All tests (requires: uv sync --all-extras)

Advanced: Ablation Studies

The telecom domain supports ablation studies for research purposes.

No-user mode

The LLM is given all tools and information upfront (no user interaction):

tau2 run \
  --domain telecom \
  --agent llm_agent_solo \
  --agent-llm gpt-4.1 \
  --user dummy_user

Oracle-plan mode

The LLM is given an oracle plan, removing the need for action planning:

tau2 run \
  --domain telecom \
  --agent llm_agent_gt \
  --agent-llm gpt-4.1 \
  --user-llm gpt-4.1

Workflow policy format

Test the impact of policy format using the workflow policy for telecom:

tau2 run \
  --domain telecom-workflow \
  --agent-llm gpt-4.1 \
  --user-llm gpt-4.1