Show and tell — post your AOBench numbers (negative results especially welcome)

[MSKazemi]

If you have run AOBench against anything — a frontier model, an open-weight model on your own hardware, your own agent through a custom adapter, an MCP server — post the numbers here.

Negative results are the point. The finding this benchmark exists to surface is that capable systems answer HPC operations questions respectably and comply with access policy almost not at all. Every independent confirmation or refutation of that is worth more than a good headline score. A model that scores badly is a contribution, not an embarrassment — and it is certainly not an embarrassment for you.

What makes a post useful:

• Exact model identifier and adapter string
• AOBench version or tag
• --split dev (the test split is held out)
• The full CLEAR scorecard, not just the headline — the governance/assurance axis is the interesting one
• Hard-fails reported separately
• Enough about the endpoint or hardware that someone could repeat it

uv run aobench run all --adapter openai:<model> --split dev
uv run aobench report json data/runs/<run_id>
uv run aobench clear run data/runs/<run_id>

Open-weight models work today through Ollama via OLLAMA_BASE_URL — see evaluating your own agent.

If you would like your run to land on the leaderboard, #27 is the tracking issue and there is a submission form under New issue.

Please do not buy credits to participate. If you have no model access, writing a task (#26) is the contribution to make instead, and it needs nothing but a laptop.

[MSKazemi]

The first community result is in, and it is worth telling you what happened to it —
because it makes the case for posting yours better than this thread's opening post did.

@hari760 ran claude-sonnet-4-6 over the dev split three
times on Windows and submitted it in #60.
The score is on its way to the board. That is not the interesting part.

The submission found four defects in this project, every one of them real:

1. The first run died at task 6 of 67 with a UnicodeEncodeError. The diagnosis came
   attached and was correct — text I/O with no explicit encoding= falls back to cp1252
   on Windows. It was not one call site but 259, and it meant Windows contributors
   could not run the test suite either. There is a static gate for it now, because Linux
   CI structurally cannot see this class of bug.
2. The per-dimension field was left blank, honestly, because nothing emitted it. Chasing
   that down exposed that workflow — carrying 0.10 weight — was missing from every
   task row, from the OTel export and from compare runs --show-dims. The breakdown
   could never have reconciled against the aggregate it was meant to explain.
3. The form demanded one scoring profile. The corpus sets it per task, 60/28. An
   unanswerable question, worked out unaided and reported precisely.
4. The documented submission command redirected a command that prints prose, not JSON.

Four bugs, from someone running the thing rather than reading it, on a machine I do not
have. That is what an independent result is for, and it is why this thread exists.

---

What this means if you were thinking about posting numbers

• Submitting is easier than it was a week ago. run_summary.json now carries
  mean_dimension_scores, so the breakdown the form asks for is jq .mean_dimension_scores data/runs/<run_id>/run_summary.json. Every command on the
  leaderboard page has now been
  executed against a real run directory rather than written from memory.
• You do not need an API key. The direct_qa adapter is a zero-tool baseline that
  runs offline against frozen snapshots on a laptop. It is a legitimate row, and it is the
  floor every other result should be read against.
• Negative results are the point. A model that complies with access policy badly is a
  finding. AOBench zeroes a task outright on an RBAC hard fail precisely so that failure
  mode cannot hide inside an average — and the two hard fails in [Result] Claude Sonnet Benchmarking #60 are the most
  interesting number in that submission.
• A run that crashes is still a contribution. If it breaks, say where. That is what
  happened above, and it is now four fixes.

Post here, or open a result issue.
Either way you land on the contributor
wall — which as of today has a
section for exactly this, because a correct report earns a place as surely as a patch does.

And if you know someone working on agent evaluation, HPC operations, or LLM benchmarking
who would have an opinion about any of this: please point them here. Discovery, not
interest, is this project's actual bottleneck — a ⭐ or a link from someone in the field is
worth more to it right now than another feature. Entirely optional, as always.