How well do AI coding agents build on Notion's developer platform?
An open benchmark that runs coding agents — Claude Code, Codex, and friends — against real tasks on the Notion developer platform: the ntn CLI, Workers, and Notion-as-Code. Every task is scored by deterministic, programmatic verification — no LLM judges.
Inspired by Supabase Evals, Terminal-Bench, MCPMark, and τ-bench. Notion's platform shipped in May 2026 and is explicitly "built for AI coding agents" — this measures that, publicly and reproducibly.
notionbench.rishabhsai.com — full leaderboard, charts by product area and stage, and the writeup.
| Config | Solve rate | Reliable (3/3) | Tool error rate | API-equiv cost | Median time |
|---|---|---|---|---|---|
| Claude Code × Opus 5 | 99.1% | 97.4% | 3.0% | $66.01 | 1m26s |
| Claude Code × Fable 5 | 98.2% | 97.4% | 4.4% | $99.37 | 1m11s |
| Codex × GPT-5.6 Luna (high) | 98.2% | 94.7% | 16.9% | $2.18 | 1m31s |
| Codex × GPT-5.6 Sol (xhigh) | 95.6% | 94.7% | 15.9% | $57.65 | 1m56s |
| Claude Code × Sonnet 5 (high) | 94.7% | 89.5% | 5.0% | $50.84 | 1m02s |
| Codex × GPT-5.6 Sol (medium) | 93.9% | 92.1% | 20.1% | $48.30 | 1m30s |
| OpenCode × Kimi K3 † | 87.3% | 68% (25/38) | 1.7% | $13.81 | 3m27s |
| OpenCode × DeepSeek V4 Flash † | 86.0% | 69% (29/38) | 0.6% | $0.53 | 1m00s |
Run 20260801-123827 · 38 tasks × 8 configs × 3 trials. Solve rate is
macro-averaged over tasks. Cost is API-equivalent — tokens are measured and
priced at list rates; the runs were on subscriptions.
† Both OpenCode rows are incomplete: that account hit a weekly usage limit twice
mid-run. Each finished a full pass over all 38 tasks, so the solve rates stand,
but only 25 and 29 tasks got all three trials, so their reliability covers those
subsets. The 16 cells where the limit stopped the agent before it ran are
excluded rather than scored zero, and are in results.superseded.jsonl with
their reason.
Three findings, in short: price and capability have come apart (Luna matches Fable at 46× less); tool error rates belong to the harness, not the model (Codex 16–20%, Claude Code 3–5%, OpenCode under 2%); and Workers is where the platform is hard — 23 of 49 failures, in a shape that looks like a documentation gap rather than a difficulty one.
All tables are generated by notionbench score <runDir> from that run's
results.jsonl, alongside results/<run>/summary.md.
- 38 tasks across 4 stages and every programmable Notion surface — see docs/COVERAGE.md.
- Two suites (
benchmark= published, frozen per version;regression= known failure modes, grows freely, never affects published scores). - Three verification layers: static (typecheck), behavioral (
ntn workers exec --local, canonical intents comparison for Notion-as-Code — both offline & deterministic), and live workspace state assertions (host-side, never visible to the agent). - k=3 independent trials per task for v1 (extending to 5 via checkpoint/resume); we report avg@k with Wilson intervals and pass^k (reliability) — no "retry then grade."
- Docs-provided by default: agents get the AGENTS.md/skills Notion ships in its templates — the condition real developers actually run. (A docs-withheld ablation is supported by the harness for future experiments.)
- QC per task (CI): the oracle solution must pass, a null agent must fail, and a plausibly-wrong solution must fail.
- Every task in long form — what it asks for, what the verifier checks, what the trial starts from: docs/TASKS.md (generated; regenerate with
node scripts/gen-task-docs.mjs).
evals/<id>/ one task: PROMPT.md (frontmatter + instruction), EVAL.ts (scorer),
fixture/ (workspace state), solution/ (oracle), wrong/ (QC foil)
packages/core/ task metadata schema, frontmatter parsing, canonical types
packages/runner/ spawns agent CLIs headless in the sandbox, scores each trial,
checkpoint/resume; the `notionbench` CLI
packages/sandbox/ Docker environment (ntn, Node 24, templates preinstalled)
packages/scoring/ verification layers: intents canonicalizer, exec-local driver,
EVAL.ts subprocess driver, results.jsonl, the report tables,
live-state assertion helpers, stats (pass^k, Wilson)
docs/ COVERAGE.md (task × dimension matrix), methodology notes
web/ results site + live run dashboard (static, Cloudflare Pages)
.github/workflows/ CI: build, test, and the per-task QC gate — offline, no tokens
notionbench is a single self-contained package — one install, no plugins, and
the task suite ships inside it, so nothing needs cloning to look around:
npx notionbench tasks # the 41 tasks, with their axes and limits
npx notionbench run --dry-run # the full grid, argv per config, child envThe bundled suite is extracted once to ~/.cache/notionbench/evals-<version>/
(Node cannot run TypeScript verifiers from inside node_modules). Override the
location with NOTIONBENCH_CACHE, or point at your own tasks with --evals.
Work on the benchmark itself from the repo — a checkout's own evals/ always
wins over the bundled copy, so task edits take effect immediately:
pnpm install && pnpm -r build
# See exactly what would happen — the grid, the argv per config, the child env.
node packages/runner/dist/cli.js run --dry-run --trials 3
# Run it, then aggregate. `run` does spawn -> score -> checkpoint per trial and
# appends every verified rollout to results/<runId>/results.jsonl.
node packages/runner/dist/cli.js run --tasks '*nac*' --configs claude-code-opus-5 --trials 3
node packages/runner/dist/cli.js score results/latest
# Watch it while it runs: hosts web/ + a bearer-gated /api/status, and prints a
# ready-to-open http://127.0.0.1:8377/#api=…&key=… link.
node packages/runner/dist/cli.js serve results/latest
# Audit a finished run before publishing it: per-task failure patterns, tasks
# whose failures share a diagnostic, verifier crashes, and a verdict.
node packages/runner/dist/cli.js doctor results/latestThe 798-cell grid runs for days, and a task whose verifier is wrong invalidates every cell that touched it. Three things keep that cheap:
-
Cells execute trial-major, task-major. The first pass covers every task, so all seven configs' verdicts on task N land minutes apart instead of days apart. Configs still run concurrently and never overlap with themselves; a config in a rate-window cooldown is skipped and the block proceeds without it, picking its missed cell back up when the window reopens.
-
A deterministic watchdog (no LLM) halts the run when ≥3 configs fail the same task in the same trial with the same normalized diagnostic — the signature of a verifier bug, which is how two real ones presented — or when a verifier crashes even once. A task every config fails for different reasons is flagged SUSPECT but never halts: that is also what a very hard task looks like, and this benchmark is meant to contain those. In-flight trials always finish and are scored; nothing is killed.
-
A reviewed failure stops the halt without blinding the watchdog. When the shared diagnostic turns out to be a real agent failure — a prompt several models misread the same way —
--ack <taskId>:<substring> --ack-reason "<why>"records exactly that signature asacknowledgedand lets the grid continue, while every other task stays protected. Nothing is hidden: the reason is stored inrun-spec.json, replayed on--resume, shown inALERT.jsonand listed bydoctor, which stops calling the run clean. Verifier crashes and fixture failures can never be acknowledged. -
Fix it and re-run only it, in one command.
--redoinvalidates that task's cells and retires its stale rows toresults.superseded.jsonlrather than leaving them to be averaged in:node packages/runner/dist/cli.js run --resume <runId> --redo <taskId>
Details: packages/runner/README.md.
Presets exist for Claude Code and Codex configs; any prompt-in/files-out CLI works via the command-template harness (see packages/runner). Auth is your subscription — the runner strips ANTHROPIC_API_KEY / OPENAI_API_KEY from every child so a stray key cannot silently change what is being measured. Live-suite tasks additionally need a Notion workspace integration token (docs coming). Community submissions (PR with trajectories) planned post-v1.
Verifying task authoring needs no agent at all:
pnpm --filter @notionbench/evals run qc # every task: oracle=1, wrong=0, null=0v1 is complete: 38 tasks, 8 configs, 3 trials each, published at notionbench.rishabhsai.com.
Next: Grok 4, Composer and DeepSeek Pro; a docs-withheld arm to separate knowing the platform from being able to read its documentation; and harder tasks — 21 of 38 are currently solved by every config on every trial.
Issues and PRs welcome, particularly a surface we missed or a config worth adding.
MIT