Designing the first AgentMeasure benchmark: what should it measure?

[roy-tong]

Measurement Report #1 measured our own pipeline (42 synthetic calls, 0 qualified usage - by design). The next step is a benchmark that external tools can run against, so claims become comparable.

Draft shape (from benchmark/BENCHMARK-DRAFT.md):

1. A synthetic agent workload: defined attempts, retries, timeouts, and a few genuine operations mixed in.
2. Instrument a tool with @agentmeasure/mcp, run the workload, and see what the metrics report.
3. Scoring: Operation Resolution, Qualified Usage (Strict), and the gap between raw calls and resolved operations.

Open questions:

• Should the benchmark be pass/fail on definitions, or a scored report?
• How do we avoid teaching to the test (tools that game the definitions)?
• Is a synthetic workload enough, or does credibility require one real runtime (Claude Code / Codex) trace?

I lean toward: scored report + one real-runtime trace as a stretch goal. What breaks in that design?

[roy-tong]

First data point is in: Benchmark Run #1 (claim audit) — six real ecosystem claims graded on the evidence ladder. The pattern: every number is self-reported; the strongest claim came from the only independent observer; nobody publishes the unit definition. Full scorecard: reports/benchmark-run-001.md. The rubric held up on real claims, and exposed that 'unit definition' is the highest-value field to standardize next.

Open question: should Run #2 re-audit x402 against its public docs, or pick a fresh claim set from registry badges?