leaderboard: Claude Code / Claude Opus 5 (k=10, 140 trials, 82.86%)

[mrojas54]

Adds a leaderboard row for Claude Code / Claude Opus 5 from a complete k=10 run of the Enterprise-Bench L1-L2 suite.

Run configuration

Field	Value
Agent	claude-code (Claude Code)
Model	claude-opus-5 (Claude Opus 5, Anthropic direct API)
Harbor version	0.20.0
Repository commit	b653af3eb89f505feb86169fb6ae10de3430ec86 (unmodified main)
Attempts per task (-k)	10
Concurrency (-n)	3
Max retries (-r)	0 (default — no trial was retried)
Judge	openai/gpt-5
Tasks	all 14 (tasks/, unmodified)
MCP	repo mcp.json — pm/crm/file-server on host.docker.internal:8011-8013

Exact command

harbor run -p tasks \
  -a claude-code \
  -m claude-opus-5 \
  -k 10 \
  -n 3 \
  --mcp-config mcp.json \
  --ae ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
  --ae OPENAI_API_KEY="$OPENAI_API_KEY" \
  --jobs-dir jobs/jobs-opus5 \
  --upload \
  --public \
  --yes

Environment

• macOS (Darwin 25.5.0), Apple Silicon
• Docker Desktop: 8 CPUs, 7.75 GiB memory (-n 3 per the dataset's concurrency guidance)
• Base image enterprise-bench/conversational-base:latest built locally via make build-image (sha256:a7eda4c0…, 1.32 GB)
• MCP servers via make start-servers (6 containers), healthy for the full run
• No AWS Bedrock override (CLAUDE_CODE_USE_BEDROCK unset) — direct Anthropic API

Results

140/140 trials completed. Denominator is 140.

Outcome	Count
Passed (reward 1.0)	116
Verifier failures (reward 0.0)	24
Execution exceptions	0
Retries	0
Cancelled / errored / pending	0

Accuracy 82.86% (116/140). Every trial produced a scored result — no trial was excluded, retried, or removed.

pass@k: 2 → 0.9063 · 3 → 0.9238 · 4 → 0.9279 · 5 → 0.9286 · 8 → 0.9286 · 10 → 0.9286

pass_at_3 is not emitted by Harbor's summary; it was computed with the same unbiased estimator, which reproduces Harbor's reported values for k = 2, 4, 5, 8, 10 exactly.

Total Anthropic agent cost $173.37 over 3h 16m 39s (avg 248.51 s/trial). Tokens: 11,029,955 uncached input + 125,514,814 cached input + 1,667,088 output = 138,211,857 total (reconciles with the job summary).

Per-task breakdown

Task	Pass	Task	Pass
eng-l1-a	0/10	sales-l2-b	10/10
eng-l1-b	10/10	sales-l2-c	10/10
eng-l1-c	6/10	sales-l2-d	9/10
eng-l2-a	8/10	support-l1-a	10/10
eng-l2-b	6/10	support-l1-b	9/10
sales-l1-a	10/10	support-l1-c	10/10
sales-l2-a	8/10	support-l2-a	10/10

eng-l1-a failed all 10 attempts on the ticket → component → open-issue join (Criterion 6). This matches the difficulty signal documented in SKILL.md, where the same criterion fails across agents and models; it is not specific to this run.

Provenance

Public Harbor Hub job: https://hub.harborframework.com/jobs/08ef17c4-5c9d-4bfd-a0a4-3525ff5d5e9f

Verified reachable while logged out — both the job page and an individual trial's trajectory page return HTTP 200 without credentials. Visibility is public. The 140 trial_ids in the entry are the trial UUIDs from that job, all unique.

Notes

• Tasks and dataset are unmodified; this run is from main at the commit above. (My other open PR, fix(eng-l1-a): distinguish exact and approximate issue matches #12, proposes an eng-l1-a spec change — it is not applied here.)
• No credentials appear in logs; keys were injected from 1Password at runtime via op run.
• An earlier attempt at this run was halted at 73/140 when the API key hit a configured spend limit. It was never uploaded and is not used here; this submission is from a single clean 140-trial run.

🤖 Generated with Claude Code

[nimit2801]

Hey @mrojas54 thank you so much for sharing Opus 5, we're taking a look at the runs, would get back to you in 2 days. Could you also share your email and discordId?

Thank you

[mrojas54]

Hey @mrojas54 thank you so much for sharing Opus 5, we're taking a look at the runs, would get back to you in 2 days. Could you also share your email and discordId?

Thank you! I will send them to you via discord.

Here is my discord ID: 665979180621627413

BTW, the Engineering number is mostly eng-l1-a scoring 0/10 and theres a PR up. I'm okay with this number being on the leaderboard for now since it will be using the same test base as all others. But I wanted to just note while you're reviewing: it's not a model error.