A multi-agent research crew that tells you what it cost, what it dropped, and whether it beat one agent.
$ python -m research_crew --compare --seeds 60
found bogus prec cost fails found/cost
-----------------------------------------------------
crew 9.50 4.55 0.69 18.0 0.00 0.528
solo 4.50 1.65 0.74 4.0 0.00 1.125
the crew found 2.11x as much and cost 4.50x as much.
per unit of spend, solo wins.
Read that table again. The crew found more than twice as many real facts — and was less precise than the single agent it replaced, at 4.5x the price.
Both halves of that are true at once, and almost no multi-agent demo shows you the second half. That is the entire reason this repo exists.
Part of Agent Lab · Vol 1 · Project 04. Pure standard library. No API key, no network, no model — the whole thing runs offline and deterministically, which is what makes the numbers above a measurement rather than a screenshot of one lucky run.
git clone https://github.com/dev48v/research-crew
cd research-crew
python -m venv .venv && . .venv/Scripts/activate # Linux/macOS: . .venv/bin/activate
pip install -e ".[dev]"
python -m research_crew --compare # the table above
python -m research_crew "your question" # one run, itemised
pytest -q # 23 testsThe crew is the standard shape — plan, fan out, critique, edit:
wave 1 plan (planner)
wave 2 research:0 research:1 research:2 research:3 <- the only parallel part
wave 3 critique (critic)
wave 4 edit (editor)
A single run is itemised, including the parts that did not work:
$ python -m research_crew "why" -n 4 --failure-rate 0.25
budget 10000, spent 26, 4 wave(s)
ok plan planner cost 2
ok research:0 researcher cost 6 <- 6, not 3: it failed once and retried
ok research:1 researcher cost 6
ok research:2 researcher cost 3
ok research:3 researcher cost 3
ok critique critic cost 2
FAIL edit editor cost 4 (editor failed on edit)
-> failed=1, ok=6
Three things in that output are deliberate and unusual:
Retries are billed. research:0 cost 6 for a task priced at 3. It failed, it
retried, and both attempts consumed budget. A framework that reports only the
successful attempt's cost is under-reporting your bill by exactly your failure rate.
A task whose dependency failed is skipped, never run. If every researcher
fails, the editor does not run on an empty context and produce a fluent summary of
nothing. That is the multi-agent failure mode that costs you credibility rather
than money, and it is a two-line check:
broken = [d for d in task.depends_on if d in failed]
if broken:
report.results.append(TaskResult(task.id, task.role, "skipped",
reason=f"depends on {', '.join(broken)}"))The budget is hard. When it runs out the remaining tasks are reported
over-budget with what they needed and what was left. It stops; it does not
quietly truncate and hand you a shorter answer that looks complete.
Every number below comes from python -m research_crew --compare with the flags
shown, 60 seeds each, and every one is pinned as a test so it cannot rot.
| critic strictness | facts found | invented surviving | crew precision | solo precision |
|---|---|---|---|---|
| 0.0 | 10.28 | 6.33 | 0.628 | 0.736 |
| 0.3 | 9.50 | 4.55 | 0.685 | 0.736 |
| 0.5 | 8.98 | 3.58 | 0.724 | 0.736 |
| 0.7 | 8.35 | 2.47 | 0.778 | 0.736 |
| 1.0 | 7.62 | 0.70 | 0.920 | 0.736 |
Fan-out multiplies hallucinations linearly — four researchers invent roughly four times as much — while real facts saturate, because researchers largely rediscover the same things. So the crew's precision starts below the single agent's and only crosses it at strictness ~ 0.56.
The critic is not a nice-to-have you add later. Below that threshold, the crew is a machine for generating confident noise at 4.5x the price.
A filter has two error rates, and only one of them gets discussed:
catch_rate = 0.9 * strictness # invented material it removes
collateral_rate = 0.25 * strictness # CORRECT material it removes on the wayThat second line is why the strictness table trades recall for precision at every
step: going 0.0 to 1.0 buys 29 points of precision and pays 2.7 facts for them.
There is no setting that gets both. test_the_critic_removes_correct_claims_too
fails if the critic ever stops paying that price — because a filter with no
collateral damage is a filter that is not filtering.
| researchers | facts found | precision | cost | facts per unit cost |
|---|---|---|---|---|
| 1 | 3.95 | 0.842 | 9.0 | 0.439 |
| 2 | 6.35 | 0.794 | 12.0 | 0.529 |
| 4 | 8.98 | 0.724 | 18.0 | 0.499 |
| 8 | 10.22 | 0.589 | 30.0 | 0.341 |
| 16 | 10.30 | 0.397 | 54.0 | 0.191 |
From 8 researchers to 16: +0.08 facts for +24 cost, and precision falls off a cliff. The efficiency peak is at two. Scaling the crew past the point where recall saturates converts money directly into noise.
At a 20% per-task failure rate, a crew run ends with 0.53 tasks failed, skipped or over budget. The single agent: 0.05. Ten times the exposure — which is just what seven tasks instead of one predicts. Reliability engineering does not stop applying because the components are models.
Most descriptions of a "crew" are a list of personas. The personas are the least load-bearing part. What decides whether a crew works is the dependency graph:
- Dependencies are declared, never inferred from list order. A crew whose correctness depends on the order somebody appended tasks breaks the first time it is reordered.
- Cycles are refused at build time, not discovered at run time when two agents are waiting on each other.
- The scheduler returns waves, not a flat order. Flattening a topological sort throws away the parallelism the graph was built to express — and the fan-out is the only genuinely parallel part of a research crew.
def waves(self) -> list[list[Task]]:
remaining = {t.id: set(t.depends_on) for t in self.tasks}
done, out = set(), []
while remaining:
ready = sorted(tid for tid, deps in remaining.items() if deps <= done)
if not ready:
raise CycleError(f"dependency cycle among: {', '.join(sorted(remaining))}")
out.append([known[tid] for tid in ready])
done.update(ready)
for tid in ready:
del remaining[tid]
return outcritical_path() follows directly and is the number to check before adding
workers: plan(2) + research(3) + critique(2) + edit(2) = 9, whatever the
fan-out width. Sixteen researchers do not make it finish sooner than two.
ScriptedAgent is not a model. It is a deterministic stand-in with the four dials
that actually change the answer:
ScriptedAgent(seed=1, recall=0.35, hallucination_rate=0.15,
failure_rate=0.0, critic_strictness=0.5)Ground truth is a fixed list of 12 facts, so quality is counted, not judged — no LLM-as-judge in the measurement loop. That is what lets the sweeps above run 600 crews in 0.16 seconds and be reproducible on your machine.
Swapping in a real model is one callable:
def nim_agent(task: Task, context: dict[str, str]) -> str:
... # raise AgentFailure on a recoverable error; return the text otherwise
run(research_crew("your question", n_researchers=4), nim_agent, budget=50_000)The runner, the budget, the retry accounting and the skip-on-broken-dependency logic are all model-agnostic.
src/research_crew/
graph.py Task, TaskGraph, waves(), critical_path() - cycles refused at build time
runner.py hard budget, billed retries, skip-on-broken-dependency
agents.py deterministic stand-ins + score() against known ground truth
evaluate.py the crew-vs-solo sweep
__main__.py python -m research_crew
tests/ 23 tests, including every claim in this README
Not a conclusion this repo can hand you, but the sweeps narrow it:
- Use one agent when spend is the constraint. It wins on facts-per-unit-cost at every setting tested.
- Use a crew when coverage matters more than the bill — it found 2.11x as much — and only with a critic above the break-even strictness, or you have bought noise.
- Keep the crew small. Two to four. The efficiency peak is at two and the recall ceiling arrives by eight.
- Count the failures. Seven tasks fail seven times as often as one.
MIT