VulcanBench

Fully open-source benchmarking for LLMs on realistic, multi-file software engineering tasks. VulcanBench measures how models perform across reasoning effort, language, codebase scale, and task complexity — with full traces, reproducible scoring, and a local dashboard.

v0.8.0 — adds the Voice Eval Suite v1 (vulcanbench voice): text-vs-audio delta measurement ("voice tax") across OpenAI Realtime, Gemini Live, and Qwen3-Omni, with a 200-question held-out set, a voices/rate/noise audio matrix, and modality-blind scoring. See docs/VOICE_EVAL.md. Previous: v0.7.0 — adds a Qwen / DashScope provider (qwen:qwen3.7-plus and friends) so Alibaba Cloud models can be benchmarked like OpenAI / Anthropic / Z.ai / Kimi. Builds on v0.6's frontier-hard task tier and cost-efficient reporting (--max-run-cost, compare, regrade, --only-missing), and on v0.5's 52 gold-verified tasks, tool-calling agent, Docker sandbox, pre-run cost estimates, five-metric scoring, and HTML replay. See docs/QUICKSTART.md to get started.

One-command setup

git clone https://github.com/morganlinton/VulcanBench.git
cd VulcanBench
make setup
source .venv/bin/activate
vulcanbench --help

Dashboard + backend (the dashboard reads live data from the API):

pip install -e ".[backend]"
uvicorn backend.app:app --port 8000          # serves ./runs at /api/*
cd dashboard && npm install && npm run dev    # http://localhost:3000

The dashboard falls back to a friendly empty state if the backend isn't running. Point it elsewhere with NEXT_PUBLIC_API_BASE (see dashboard/.env.example).

By default the API reads ./runs/ directly. For a durable, queryable store, set DATABASE_URL (Postgres or SQLite) and the API switches to a database — POST /api/runs and /api/feedback become writable, and python scripts/ingest_runs.py bulk-loads existing runs. A Postgres is provided by docker compose up db.

Example run

# Offline, deterministic (no API key) — drives the real agent loop end to end.
# Real runs default to the Docker sandbox; --sandbox local is fine for the
# deterministic mock model.
vulcanbench run --task hello-world --model mock:synthetic --sandbox local

# Any real model via the generic provider interface:
export OPENAI_API_KEY=...      # or ANTHROPIC_API_KEY=...
vulcanbench run --task hello-world --model openai:gpt-4o
vulcanbench run --task hello-world --model anthropic:claude-opus-4-8
vulcanbench run --task hello-world --model zai:glm-5.2
vulcanbench run --task hello-world --model qwen:qwen3.7-plus
vulcanbench run --task hello-world --model deepseek:deepseek-v4-flash

# Each run prints all five metrics + cost, e.g.:
#   functional=1.0 quality=1.0 security=1.0 human_like=0.8 total=0.974 cost=$0.0
# and writes ./runs/<id>/{trace.jsonl, summary.json, replay.html, final.patch}

# Run a whole suite: repeat for signal, parallelize, and cap the spend.
vulcanbench run --suite v1 --model openai:gpt-4o --repeat 5 --max-concurrency 4 --max-cost 20.00

# Compare normalized reasoning effort on the same suite/model.
vulcanbench run --suite v1 --model openai:gpt-4o --effort low
vulcanbench effort-sweep --suite v1 --model openai:gpt-5.1 --efforts low,medium,high --repeat 3 --sandbox docker

# Fast micro/small sweep vs navigation-heavy medium/large tasks:
vulcanbench run --suite v1-micro --model openai:gpt-4o
vulcanbench run --suite v1-large --model openai:gpt-4o --repeat 5 --sandbox docker

# Diamond tier: rubric-graded *mergeability* (not just correctness). Use a judge
# model different from the one under test to avoid self-grading:
vulcanbench run --suite v1-diamond --model anthropic:claude-opus-4-8 --judge-model openai:gpt-5.5
# Carbyne tier: harder still — terse prompts where the naive solution is subtly wrong:
vulcanbench run --suite v1-carbyne --model anthropic:claude-opus-4-8 --judge-model openai:gpt-5.5
vulcanbench leaderboard            # by model: pass@1 ± stderr, pass@k, cost, latency
vulcanbench leaderboard --by run   # per-run drill-down
vulcanbench report -o report.md    # shareable Markdown/JSON report (ranking,
                                   #   model-separation/discrimination, effort
                                   #   sensitivity, per-task breakdown,
                                   #   environment, drift flags)
vulcanbench calibrate              # empirical difficulty calibration from recorded runs
vulcanbench replay <id>

# Runs execute in an isolated container by default (see Sandbox below);
# build the image once with `make sandbox-image`.

# Use it as a CI regression gate (threshold must be in [0, 1]):
vulcanbench run --suite v1 --model openai:gpt-4o --repeat 5 --fail-under 0.8

The gate fails closed: it exits 4 if pass@1 is below the threshold, if pass@1 is unavailable, or if any suite run errored — a CI gate never goes green on a partial or unknown result.

Exit codes: 0 ok · 1 usage/error · 2 provider · 3 sandbox · 4 gate failed (below --fail-under, or a run errored).

final.patch is a real git diff of the agent's edits; replay.html is fully self-contained (open in any browser). Use --no-judges to skip the LLM judge ensemble, --timeout SECONDS to cap a run's wall-clock. Traces, summaries, and patches are secret-redacted and size-capped before they're written, so run artifacts are safe to publish. See make ci, make docker-up, docs/.

Cost-efficient reporting

Grading is deterministic and every run records the task_hash it was scored against, so comparisons are queries over ./runs, not re-runs. Four commands turn a "$70 full-matrix re-run" into a "~$10 one new column":

# Per-run hard cost ceiling: stop a single agent run once its own spend crosses
# the value (records cost_capped; the partial result is still graded). Turns a
# failing run that would ruminate to the step cap into a bounded "DNF at cap".
vulcanbench run --suite v2 --model anthropic:claude-fable-5 --max-run-cost 2.50

# Resume / fill only the gaps: reuse fresh cached runs for this model+effort and
# launch only the missing tasks (stale runs, scored against an older task
# definition, are ignored and re-run). Pairs with --max-run-cost.
vulcanbench run --suite v2 --model anthropic:claude-opus-4-8 --effort high \
  --only-missing --max-run-cost 2.50

# Assemble the model × effort matrix for a frozen suite from cached runs only —
# baselines are never re-run. Add a model = run that one model, then re-compare.
vulcanbench compare --suite v2                 # complete cells (prints a frozen version id)
vulcanbench compare --suite v2 --incomplete    # show gaps + the command to fill them

# Re-score existing runs against the current task definition at $0 API cost
# (rebuilds base + captured agent patch + current hidden tests, re-verifies).
# Use after editing a task's tests/thresholds instead of re-running the model.
vulcanbench regrade runs/<run-id> --sandbox docker
vulcanbench regrade runs/ --sandbox docker     # every run under a directory

Cache reuse (--only-missing) and comparison (compare) only see runs under the directory they scan (--output-dir / --runs-dir, default ./runs, recursive), so keep every run under one root. See docs/QUICKSTART.md for the full workflow.

Models

Specify a model as provider:model:

• mock:synthetic — deterministic, offline; used by tests and demos.
• openai:<model> — OpenAI Chat Completions for normal runs, or the Responses API when --effort is supplied. Needs OPENAI_API_KEY.
• anthropic:<model> — Anthropic Messages API. Needs ANTHROPIC_API_KEY.
• zai:<model> — Z.ai (Zhipu) OpenAI-compatible Chat Completions API. Needs ZAI_API_KEY. Reasoning effort is not supported; --effort is recorded as metadata only.
• kimi:<model> — Moonshot AI (Kimi) OpenAI-compatible Chat Completions API. Needs MOONSHOT_API_KEY. extra-high maps to Kimi's reasoning_effort=max; other effort levels are recorded as metadata only.
• qwen:<model> — Alibaba Cloud DashScope (Qwen) OpenAI-compatible Chat Completions API. Needs DASHSCOPE_API_KEY. Default base URL is the international endpoint; set DASHSCOPE_BASE_URL for China or another region. low/medium map to Qwen's reasoning_effort and extra-high maps to its xhigh (Qwen3.8+; the documented enum is low/medium/xhigh, default xhigh). high is recorded as metadata only — Qwen has no such level, and an unset request runs at the xhigh default. Pre-3.8 models may ignore the field.
• deepseek:<model> — DeepSeek OpenAI-compatible Chat Completions API. Needs DEEPSEEK_API_KEY. low/high map to DeepSeek's reasoning_effort field and extra-high maps to its max; medium is recorded as metadata only (DeepSeek's enum is low/high/max — it silently coerces medium to high, so the harness never sends it).

--effort accepts low, medium, high, or extra-high. OpenAI runs map it to the Responses API reasoning.effort field; Anthropic runs map it to the Messages API output_config.effort field. extra-high maps to xhigh on both providers and is opt-in for sweeps because support is model-dependent (e.g. Claude Opus 4.7+). Mock, Z.ai, and Qwen runs accept the field as no-op metadata. Effort labels are each provider's own scale — a cross-provider comparison at the same label compares each model at its own setting, not a calibrated equivalence.

Sandbox

The agent's tool execution can run in an isolated Docker container instead of on the host:

# Build the base image once (git, ripgrep, ruff, bandit, radon, pytest):
docker build -t vulcanbench/sandbox:base -f sandbox/Dockerfile.base .

vulcanbench run --task hello-world --model openai:gpt-4o
# --sandbox local|docker|auto   (default: docker)
# --image vulcanbench/sandbox:base   (default: per-task metadata or vulcanbench/sandbox:base)
# --network                     (off by default; opt in for dependency installs)

• docker (default) runs tools in a non-root, network-off, resource-limited container (workspace bind-mounted, cleaned up after each run). It errors out if the daemon is unreachable — it never silently falls back to host execution.
• local runs the model's commands directly on the host — fast and Docker-free, but unsandboxed; opt in deliberately (fine for mock:synthetic and trusted dev loops).
• auto uses Docker when available. Falling back to host execution additionally requires VULCANBENCH_ALLOW_HOST_EXEC=1; otherwise it errors out.

File operations (read/edit/search) always run host-side over the shared mount; command execution (run_command/run_tests/run_lint) and the functional verifier run inside the container, so the whole run is reproduced in one isolated environment. Build the all-language image with docker build -t vulcanbench/sandbox:base -f sandbox/Dockerfile.base . (Python + Go + Node).

Tasks

The tasks/v1/ suite holds 52 gold-verified tasks across Python, Go, TypeScript, and Rust, plus the hello-world demo. Each task ships a starting repo/, hidden tests/ (never shown to the agent), declarative fail_to_pass/pass_to_pass test commands in metadata.json, and a gold_patch.diff reference solution.

The corpus spans three difficulty tiers (easy / medium / hard) across all four languages. Most tasks today are localized single-file fixes that set a floor; a growing set raise the ceiling with genuine subtlety — operator precedence and associativity (py-expr-eval, hard), a race-free, order-preserving parallel map verified under go test -race (go-parallel-map, hard), an RFC 6901 JSON Pointer resolver (py-jsonpointer, hard), and a prototype-pollution-safe deep merge (ts-deep-merge, hard). Broader task_complexity (multi_file / system / architecture) and larger repo_scale coverage is active work — see ROADMAP. Because the task_complexity and repo_scale fields are validated against the repo, a task's declared scale is checked, not just asserted. vulcanbench report includes a discrimination section so you can see which tasks actually separate the models you run (and which carry no signal).

make validate-tasks                              # validate every task
vulcanbench validate-task tasks/v1/<id>          # one task

Grading: hidden tests or an agentic grader

By default a task's functional score comes from hidden tests — deterministic and exact, but it requires the issue to fully specify the expected behavior.

A task can instead opt into an agentic grader (metadata.grader: "agentic") that judges the agent's diff against a list of plain-English acceptance_criteria (never shown to the agent), so the prompt can be terse and realistic — closer to how developers actually ask. The grader, not the prompt, holds the spec.

# Use a strong, independent grader model to avoid a model grading its own work:
vulcanbench run --task py-slugify-terse --model openai:gpt-5.5 \
  --judge-model anthropic:claude-opus-4-8

Agentic grading is non-deterministic, so it is opt-in and never the default; tasks that need exact, reproducible scoring keep the test verifier. See tasks/v1/py-slugify-terse for an example.

Trust the grader before you rely on it. An LLM grader is only worth using if it agrees with ground truth and doesn't flip its verdict run to run. Two tools:

• Set metadata.grader_samples: N to grade by majority vote over N calls (ties resolve to incorrect); each grade reports its self_consistency.
• python scripts/grader_eval.py --task tasks/v1/<id> --model <grader> --samples 5 grades a task's labeled grader_cases.json (known-correct and known-incorrect changes) and reports accuracy, false-pass rate (graded correct but actually wrong — the dangerous one), and self-consistency. Don't ship an agentic task whose grader posts a non-zero false-pass rate.

Validation proves each task is real: the gold patch must solve it (functional == 1.0), the fail_to_pass tests must genuinely fail before the fix, and scoring must be deterministic over repeated runs.

Provenance is labeled and checked. Every task declares source (hand-authored or oss) and an explicit decontaminated boolean. Hand-authored tasks are written now (post-cutoff, so decontaminated: true); the validator enforces that. An oss task (e.g. oss-inflection-titleize, sourced verbatim from a real MIT-licensed repo with its LICENSE preserved) is honestly labeled decontaminated: false — its fix predates model cutoffs — and the vulcanbench report integrity section flags every run scored against it. Scaffold one with python scripts/import_oss_issues.py. Format details: docs/TASK_CONTRIBUTION.md.

Voice Eval Suite

Measures the voice tax: how many points a model loses when the same question arrives as speech instead of text. 200 held-out questions (tasks/voice-v1/), rendered via TTS under a voices × rate × noise matrix, answered through each model's realtime/audio endpoint, and scored by a modality-blind scorer (results have no numbers to show yet — none are published until the first full run).

export OPENAI_API_KEY=...     # TTS + Realtime + STT fallback
export GEMINI_API_KEY=...     # Gemini Live
export DASHSCOPE_API_KEY=...  # Qwen3-Omni
export XAI_API_KEY=...        # Grok Voice (grok-voice-think-fast-2.0)

vulcanbench voice run -m openai-realtime --dry-run     # 5-question pipeline check
vulcanbench voice run -m openai-realtime,gemini-live,qwen-omni,grok-voice
vulcanbench voice report runs/voice-<id> -o report.md

Full methodology: docs/VOICE_EVAL.md.

VulcanCyber Eval Suite

A defensive cybersecurity suite (tasks/vulcancyber-v1/): each task is a real merged open-source PR that fixes a security weakness, sourced post model cutoff (upstream_merged >= 2026-06-01) and graded by the project's own deterministic security regression tests — "here is vulnerable code, produce the fix." v1 ships 16 validated tasks across Python/TS/Go/JS/Rust and a dozen vulnerability classes (Family A vuln-fixes plus one Family-B security-tool fix): prototype pollution, CRLF/host injection, encoded-separator auth bypass, filename & ref-name spoofing, unpaired-surrogate crash, several algorithmic/resource DoS classes (YAML merge-key, urlencoded field-count, int-URL, integer-overflow), unescaped control-character serialization, and a gosec scanner detection-gap (Family B) — spanning hono, zod, undici, validator.js, content-disposition, echo, gosec, pyyaml, tornado, werkzeug, urllib3, quick-xml, toml, and gitoxide.

make sandbox-image-all       # base + Rust + Go 1.26 images the suite needs
make validate-cyber          # gold-solves, fail-to-pass real, deterministic x3
vulcanbench run --suite vulcancyber-v1 --model mock:synthetic --sandbox local   # $0 smoke
vulcanbench run --suite vulcancyber-v1 --model anthropic:claude-opus-5 --sandbox docker

Posture is defensive only (no offensive tooling). Candidate PRs are surfaced by scripts/mine_security_prs.py (a read-only gh search). Full methodology, task list, and sourcing recipe: docs/CYBER_EVAL.md; curation discipline: tasks/vulcancyber-v1/CHARTER.md.

Architecture & Reproducibility

• Standardized tools (list/read/edit/search/run) behind one protocol, with interchangeable local and Docker executors (see Sandbox)
• Every step captured as JSONL (llm, tool, diff, test, metric) + token usage
• Each run records its vulcanbench replay <id> command for reproduction
• Docker sandbox runs untrusted command execution in a non-root, network-off, resource-limited container

Full details: docs/ARCHITECTURE.md, docs/METRICS.md, docs/REPRODUCIBILITY.md

Documentation

Doc	Purpose
QUICKSTART	Setup, smoke test, first real run
METRICS	How the five scores are computed
DEPLOYMENT	Hosted API + dashboard (optional)
CONTRIBUTING	Add tasks, run CI locally
CYBER_EVAL	VulcanCyber v1 — defensive cybersecurity suite
ROADMAP	Planned follow-ups
results/	Published benchmark snapshots (MD, JSON, PDF)

Quality and security analyzers run when the relevant toolchains are on your host (e.g. bandit for Python via the venv; gosec for Go if installed). Otherwise those metrics report null with a reason — never a fabricated score. Use --no-judges to skip the LLM judge ensemble and cut cost roughly threefold.

License

Apache 2.0 (see LICENSE and NOTICE).

Provider terms & data usage

VulcanBench is an independent evaluation harness. A few boundaries keep its use consistent with the model providers' terms — please read these before running or publishing results.

• You bring your own keys, under your own agreement. VulcanBench never bundles or shares API credentials. Each run uses the keys in your environment (OPENAI_API_KEY, ANTHROPIC_API_KEY, ZAI_API_KEY, MOONSHOT_API_KEY, DASHSCOPE_API_KEY, DEEPSEEK_API_KEY), so every call is made under your account and your commercial/API agreement with that provider. You are responsible for staying within your provider's terms and usage policies.

• Outputs are for evaluation, not training. Recorded run artifacts (traces, patches, summaries) capture model outputs solely for scoring, inspection, and reproducibility. Both OpenAI and Anthropic prohibit using their outputs to develop or train competing models — do not use VulcanBench artifacts, or any published corpus of them, for that purpose. VulcanBench intentionally has no "export outputs as a training dataset" feature.

• Trademarks & independence. "OpenAI" and "GPT" are trademarks of OpenAI; "Anthropic" and "Claude" are trademarks of Anthropic; "Z.ai" and "GLM" are trademarks of Zhipu AI; "Kimi" and "Moonshot" are trademarks of Moonshot AI; "Qwen" and "DashScope" are trademarks of Alibaba Cloud; "DeepSeek" is a trademark of DeepSeek. VulcanBench is not affiliated with, sponsored by, or endorsed by any of these companies. Model and provider names are used only to identify the systems under test.

This is not legal advice; consult the current provider terms for authoritative guidance.