Roadmap for v0.5 and beyond — what should we build next?

[MSKazemi]

AOBench has four milestones and about thirty open issues, but the ordering is currently one person's opinion. If you operate a cluster, evaluate agents, or are thinking about giving an AI assistant access to real infrastructure, your opinion is better than mine on what matters next.

The question: of the things below, which two would change whether you actually use AOBench?

• More corpus breadth. 26 of the 50 QCAT × role cells have exactly one task (Write a second task for a thin QCAT × role cell (32 of 50 have only one) #26). More tasks means finer-grained findings, but it is slow work.
• More models callable. A LiteLLM adapter (LiteLLM adapter — one file, ~100 providers #33) would open ~100 providers in one file; Gemini (Google Gemini adapter #34) and Bedrock (AWS Bedrock adapter #35) are the specific gaps people have hit.
• Real facility grounding. Six environments are grounded in Marconi100 ExaData today. Extending that — or grounding against a second real facility — is the biggest credibility lever and needs someone with data access.
• Reporting depth. The reporting pipeline epic ([Epic] Reporting suite — report renderers on the data→stats→model→render pipeline #16): richer scorecards, comparisons, statistics.
• Deployment surfaces. Durable run queue (F2 — durable async run queue (arq/Redis worker) #10), experiment persistence (F5 — persist experiment results in a database #13), OIDC→RBAC for the MCP server (F7 — full OIDC to RBAC mapping for the MCP server #14), live A2A transport (F12 — live A2A HTTP transport #15).
• Something not on this list. Genuinely — say so.

Second question, more important than the first: what would you need AOBench to measure that it currently does not? The scoring dimensions were chosen from HPC operations experience, not from a survey, and I would like to know where that shows.

No wrong answers, and "I looked at this and bounced off because X" is the single most useful reply in the thread.