Benchmarking Large Language Models as Orchestrators in Multi-Agent Pipelines
PrincipalBench is the first benchmark specifically designed to evaluate LLMs as orchestrators (principals) in multi-agent pipelines. Rather than measuring a model's ability to answer questions or write code directly, PrincipalBench measures:
- Task Decomposition (DQ) — Can the model break a complex multi-hop task into correctly ordered atomic subtasks?
- Failure Detection (FDR) — Can the model detect when a worker agent returns a plausible-but-wrong output?
- Adversarial Recovery (ARS) — Can the model produce the correct final answer after catching a worker failure?
- Context Coherence (CC) — Does the model maintain coherent context across many pipeline hops?
The best worker model is not the best orchestrator. Models that dominate SWE-Bench (Claude-3.5-Sonnet #1, GPT-4o #2) fall to ranks #3 and #4 in PrincipalBench, while Llama-3.1-70B achieves the highest PrincipalScore (0.785). The Spearman rank correlation between SWE-Bench and PrincipalBench is ρ = 0.57 (p = 0.14).
principal-bench/
├── paper/ # LaTeX source and compiled PDF
│ ├── main.tex # Paper source
│ ├── main.pdf # Compiled PDF
│ ├── references.bib # Bibliography
│ └── arxiv.sty # arXiv style file
├── harness/ # Benchmark evaluation engine
│ ├── pipeline.py # Core orchestrator harness
│ ├── worker_sim.py # Mock worker simulator
│ ├── model_client.py # LLM API client
│ └── run_benchmark.py # CLI entry point
├── tasks/ # Task dataset
│ ├── tasks.jsonl # 500 multi-hop tasks
│ ├── schema.json # Task schema definition
│ └── generate_tasks.py # Task generation script
├── failures/ # Failure injection library
│ ├── failure_library.jsonl # 105 failure specifications
│ └── generate_failures.py # Failure generation script
├── scoring/ # Metrics and statistics
│ ├── metrics.py # DQ, FDR, ARS, CC scoring
│ └── statistical.py # Bootstrap CIs, rank comparison
├── test/ # User-facing test harness
│ └── run_test.py # Standalone test runner
├── scripts/ # Figure generation
│ └── regenerate_figures.py
├── viz/ # Generated figures (PDF + PNG)
├── results/ # Benchmark results (CSV)
├── requirements.txt # Python dependencies
├── LICENSE # MIT License
└── README.md # This file
pip install -r requirements.txtCreate a .env file in the root directory:
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key # optional
GOOGLE_API_KEY=your_google_key # optionalpython test/run_test.py --model gpt-4o --domain finance --num-tasks 10This will:
- Load 10 finance-domain tasks from the dataset
- Run each through your chosen orchestrator model
- Inject adversarial failures at the second-to-last hop
- Report DQ, FDR, ARS, CC, and PrincipalScore
python harness/run_benchmark.py \
--models gpt-4o claude-3-5-sonnet llama-3.1-70b \
--tasks tasks/tasks.jsonl \
--failures failures/failure_library.jsonl \
--seeds 0 1 2 \
--output results/my_results.csvThe test/ directory provides a standalone test harness designed for users who want to:
- Evaluate their own orchestrator models
- Run cost analyses on their own API keys
- Contribute results back to improve the benchmark
python test/run_test.py \
--model gpt-4o \
--domain all \
--num-tasks 50 \
--seeds 3 \
--output test_results.jsonThe test runner outputs:
- Per-task metric breakdowns
- Aggregated PrincipalScore with confidence intervals
- Estimated API cost for the run
- A
test_results.jsonfile that can be contributed back
We invite the community to run evaluations and submit results. After running:
python test/run_test.py --model your-model --num-tasks 50Share your results by opening an issue with the test_results.json attached, or submit a PR adding your model's results to results/community/.
All four metrics are LLM-free — computed from ground truth without requiring an LLM judge:
| Metric | Formula | Range |
|---|---|---|
| Decomposition Quality (DQ) | F₁ ∩ (1 − λ·τ) | [0, 1] |
| Failure Detection Rate (FDR) | Accuracy of detection flags | [0, 1] |
| Adversarial Recovery Score (ARS) | Edit similarity to ground truth | [0, 1] |
| Context Coherence (CC) | max(0, 1 + β̂₁), where β̂₁ = per-hop degradation | [0, 1] |
PrincipalScore = 0.25·DQ + 0.30·FDR + 0.25·ARS + 0.20·CC
The failure injection library contains 105 specifications across three types:
| Type | Count | Description |
|---|---|---|
| Factual Error (FE) | 35 | Wrong number, name, date, or classification |
| Structural Error (SE) | 35 | Wrong format, missing fields, truncated output |
| Adversarial Plausible (AP) | 35 | Near-correct but wrong; hardest to detect |
| Domain | Tasks | Tier 1 | Tier 2 | Tier 3 | Mean Hops |
|---|---|---|---|---|---|
| Finance | 167 | 40 | 80 | 47 | 4.1 |
| Legal | 167 | 30 | 90 | 47 | 3.8 |
| Code | 166 | 30 | 100 | 36 | 4.3 |
| Total | 500 | 100 | 270 | 130 | 4.1 |
Running the full benchmark (500 tasks × 8 models × 3 seeds = 12,000 calls) costs approximately:
| Model | Per-Task Cost | Full Benchmark |
|---|---|---|
| GPT-4o | $0.018 | ~$210 |
| Claude-3.5-Sonnet | $0.014 | ~$162 |
| Llama-3.1-70B (Together) | $0.002 | ~$27 |
Actual costs depend on task token lengths, prompt design, and API pricing tiers. Use test/run_test.py --model <model> --num-tasks 5 to estimate your costs before running the full benchmark.
If you use PrincipalBench in your research, please cite:
@misc{chowdhury2025principalbench,
title={{PrincipalBench}: Benchmarking Large Language Models
as Orchestrators in Multi-Agent Pipelines},
author={Sayan Chowdhury},
year={2025},
howpublished={arXiv preprint},
url={https://github.com/saynchowdhury/Principal-Bench}
}MIT License — see LICENSE for details.
- Author: Sayan Chowdhury
- Email: sayanshytech.20@gmail.com
- GitHub: saynchowdhury/Principal-Bench