Score variance of different runs

[hanningzhang]

Dear Authors,

Thank you so much for your benchmark. I am testing my Qwen3.5-35B-A3B with Opus-4.5 as the judge. I ran the benchmark 3 times, and the mean scores are 65.71%, 69.13%, and 74.26%, respectively. And the token cost varies a lot. I am wondering if this is considered normal? And is there any methods to reduce the variance? I am also curious how you calculate the AVG score on your leaderboard?

Here are the logs of my run:

2026-04-03 19:14:12,124 - INFO - Total tokens used: 1,662,453 (input: 1,625,709, output: 36,744)
2026-04-03 19:14:12,124 - INFO - Total API requests: 115
2026-04-03 19:14:12,124 - INFO - Avg tokens/task: 69,269
2026-04-03 19:14:12,124 - INFO - Mean score: 0.6571
2026-04-03 19:14:12,124 - INFO - Score per 1K tokens: 0.0095 (higher = more efficient)

2026-04-03 20:29:21,516 - INFO - Total tokens used: 1,549,510 (input: 1,514,401, output: 35,109)
2026-04-03 20:29:21,516 - INFO - Total API requests: 100
2026-04-03 20:29:21,516 - INFO - Avg tokens/task: 64,563
2026-04-03 20:29:21,516 - INFO - Mean score: 0.6913
2026-04-03 20:29:21,521 - INFO - Score per 1K tokens: 0.0107 (higher = more efficient)

2026-04-03 21:02:03,418 - INFO - Total tokens used: 1,524,989 (input: 1,497,458, output: 27,531)
2026-04-03 21:02:03,418 - INFO - Total API requests: 104
2026-04-03 21:02:03,418 - INFO - Avg tokens/task: 63,541
2026-04-03 21:02:03,418 - INFO - Mean score: 0.7426
2026-04-03 21:02:03,418 - INFO - Score per 1K tokens: 0.0117 (higher = more efficient)

Thank you!

[ScuttleBot]

👋 Hi! I'm @olearycrew's OpenClaw bot — helping triage and respond to PinchBench issues.

Great data — thanks for sharing the detailed logs across all 3 runs. The short answer: yes, this level of variance is normal, and here's why.

Why the variance happens

1. LLM non-determinism. Even at temperature 0, most LLM APIs don't guarantee deterministic outputs. Qwen3.5-35B-A3B running through an API will produce different token sequences across runs, which means different tool calls, different reasoning paths, and different final outputs — all from the same prompt.

2. Task sensitivity. Some PinchBench tasks have binary or near-binary outcomes — get the approach right and you score high, miss a key step and you score low. A single task swinging from 0.4 to 0.9 across runs can shift your mean score by ~2 percentage points over 24 tasks. Your range of 65.7%–74.3% (~8.5pp spread) suggests a handful of tasks are landing differently each run.

3. Judge variance. You're using Opus 4.5 as the judge, which itself is non-deterministic. The same agent output can receive slightly different scores across judge evaluations, especially on subjective tasks (creative writing, market research, etc.).

4. Token usage tells the story. Your token counts vary from 1.52M to 1.66M across runs — the agent is taking meaningfully different paths each time. More tokens generally means more exploration/retries, which can go either way on final score.

How to reduce it

• Run multiple times and average. The benchmark supports --runs N for exactly this reason. 3 runs is a solid start; 5 gives you a tighter confidence interval. Report the mean ± std dev.
• Temperature 0 on the agent. If you're not already, set temperature to 0 for the agent model. Won't eliminate variance (see above) but reduces it.
• Check per-task scores. The results JSON includes per-task breakdowns. Compare which tasks swing across your 3 runs — that'll tell you whether it's a few volatile tasks or broad instability.

Leaderboard methodology

For leaderboard scores, the standard practice is to report the mean across multiple runs. The results payload already captures mean, std, min, and max per task when using --runs N.

Related issues

This connects to two open discussions:

• Leaderboard scoring bias: runs with fewer tasks rank higher than comprehensive runs #53 — Leaderboard scoring bias: how mean scores can be misleading without context on task count and variance
• Feature: Publish per-task variance and retry counts on leaderboard #8 — Per-task variance on leaderboard: feature request to surface std dev, consistency scores, and per-run breakdowns in the UI

Your experience is exactly why #8 matters — a single aggregate number doesn't capture reliability. Surfacing variance data would let people distinguish "consistently good" from "occasionally great."

Hope that helps! 🦀