Start here — where to contribute

[MSKazemi]

👋 Welcome. This issue is the map. Everything on the roadmap that is not yet built has a tracking issue, grouped so you can find something that fits the time you have.

You do not need HPC access, a cluster, or an API key to contribute. The whole benchmark runs against frozen snapshots on a laptop, and the direct_qa adapter needs no model provider.

Pick by how much time you have

Time	Browse
A few hours	effort: small · good first issue
About a day	effort: medium
A substantial project	effort: large

Pick by what you like working on

Interest	Label
Command-line tools and ergonomics	area: cli
HTTP APIs, job queues, auth	area: api
MCP / agent protocols	area: mcp · area: a2a
Evaluation and scoring methodology	area: scorers
Tasks, environments, HPC data	area: benchmark
Docs, examples, developer experience	area: docs
Reports, statistics, leaderboards	area: reporting
CI, packaging, typing, lint	area: infra · type: tech-debt

Pick by milestone

• v0.5 — Contributor on-ramp & DX — small, well-specified work. Start here.
• v0.6 — Infrastructure — durable queue, sandboxed executor, live A2A transport, experiment persistence, OIDC, SDK.
• v0.7 — Reporting & scoring depth — the reporting pipeline and remaining scoring depth.
• Blocked — external dependency — real work that cannot start yet. Marked blocked: upstream. Discussion welcome, PRs not yet.

The most valuable thing you can do — no code required

AOBench is a benchmark. Its two highest-value contributions are data, not code, and
neither needs cluster access, an API key, or any knowledge of the internals:

• Write a second task for a thin QCAT × role cell (32 of 50 have only one) #26 — write a task. 31 of the 50 QCAT × role cells currently have exactly one task,
  which means the benchmark cannot tell "understands the category" from "got lucky".
  One JSON file against an existing environment snapshot. This is the single most useful
  contribution to this project.
• Run a model and submit a leaderboard entry — negative results welcome #27 — run a model and submit its numbers. Independent results from independent
  hardware are what make a benchmark credible rather than a claim. Negative results are
  more useful than good ones — a model that complies with access policy badly is the
  finding, not an embarrassment.

Suggested first contributions

Every issue named here was checked against main on 2026-09-11 and is genuinely open.

No code required

• Record a 30-second demo GIF for the README #9 — record a 30-second demo GIF for the README. No Python at all. The README asks
  people to picture what a run looks like and then does not show them.
• Run a model and submit a leaderboard entry — negative results welcome #27 — run a model and submit its numbers. This is now easier than it was: the run
  summary emits mean_dimension_scores, so the per-dimension breakdown the form asks for
  is one jq away. Negative results are more useful than good ones. The first
  community submission ([Result] Claude Sonnet Benchmarking #60) found five bugs in this project on its way to a score, and
  its numbers are now the first community row on the
  leaderboard.

A few hours

• Write a second task for a thin QCAT × role cell (32 of 50 have only one) #26 — write a second task for a thin cell. One JSON file against an existing
  environment snapshot. Still the single most useful contribution to this project, and
  aobench new task --thinnest now picks the emptiest cell and writes a valid spec for you.
  DOCS_USR is filled — by @userfypp, the project's first
  corpus contribution (#78); 31 cells are not.
• Run a model and submit a leaderboard entry — negative results welcome #27 — run a model and submit its numbers. Listed twice on this page on purpose. It
  is the one contribution nobody else can make for you: independent numbers from
  independent hardware. Negative results are more useful than good ones.
• examples/ is outside the encoding gate — and has 2 live Windows-breaking calls #69 — a quality gate with a blind spot aimed at newcomers.
  scripts/check_text_encoding.py keeps encoding-less text I/O out of the tree, and its own
  docstring argues that a contributor who cannot run the tests is as blocked as one whose
  run dies — but it scans src/aobench, tests and scripts and not examples/, which
  is the first thing a new user opens. Add one tuple entry and it goes red on two real
  calls in the CI-gate example. Two-word fix; the point is closing the hole.
• A mocked asyncio.run leaks a coroutine, and the warning accuses an innocent test #70 — CLOSED. Fixed by @motodriver
  (PR #80).
• Ten tasks grant tool families that do not exist — three dev tasks get zero tools, and validate passes #74 — CLAIMED by @motodriver, see the table below.
  Ten tasks grant tool families that do not exist (topology, inventory, filesystem,
  and bms in the RBAC policy files), silently dropped by ToolRegistry — three of the
  ten hand the agent no tools at all.
• check_facts.py guards every number except the ones a newcomer checks first #71 — CLOSED. Fixed by @motodriver
  (PR #83).
• A freshly scaffolded task is scored in dev runs and gets a vacuous tool_use 1.0 — the top score in the run #75 — CLOSED. Fixed by @motodriver
  (PR #82).
• Flaky release gate: test_report_json_flag_emits_clean_json_on_stdout fails ~1 run in 8 #72 — a flaky release gate. test_report_json_flag_emits_clean_json_on_stdout.
  Flakiness is worth fixing for its own sake: a test that fails at random teaches everyone
  to re-run instead of read.
• Verify every command in the docs actually does what the page says — from a clean checkout #62 — check that the documented commands actually work. 136 fenced shell commands
  across 24 pages, and nothing verifies any of them; two have already been caught doing
  something other than what the page said. One page per PR, no internals knowledge needed,
  and being new to the project is an advantage — the places you get stuck are the finding.

(Three issues stood in this section this morning and all three are now closed, because
contributors finished them within nine hours of each other on 2026-09-11: #61 —
the tools/ mypy --strict slice, the last package-sized piece of #7 — by
@Akimbo92i (PR #64);
#65 — the patch.dict test-isolation bug that #61 surfaced on its way through — by
@motodriver (PR #67);
and #66 — the MockSlurmTool._load_json return type that three of the seven
job_details.json snapshots disagreed with — by
@QIU-Guanzong (PR #68).
The tree is now at a single mypy --strict error and it is a third-party
import-untyped. Each of those three was found by the previous one — which is the honest
reason the small-issue shelf keeps emptying.)

About a day

• Google Gemini adapter #34 / AWS Bedrock adapter #35 — a Gemini or AWS Bedrock adapter. Nobody can benchmark a model AOBench
  cannot call, so every missing provider is a result that never gets submitted.
• Triage the 96 CodeQL findings — path injection on the server surfaces first #23 — triage the CodeQL findings. 80 open alerts, of which 25 are
  py/path-injection on the server surfaces and the rest is largely noise. The value is
  in the triage, not the patching; a dismissed alert with a one-line reason is a real
  deliverable here.

Claimed right now — do not duplicate these

Kept current here so an issue someone has spoken for is visibly taken, rather than taken
on trust that I am reading my notifications.

Issue	Claimed by	Since
#33 — LiteLLM adapter	@hoti-code	2026-09-01
#62 — docs/getting-started/quickstart.md (other pages still free)	@aawhan0	2026-09-10
#23 — CodeQL triage	offered to @lorenzo-benites	2026-09-08
#9 — demo GIF for the README	@BillP313	2026-09-11
#74 — ghost tool families in the corpus	@motodriver	2026-09-16

Everything else on this page is free.

Claiming an issue protects it. If you comment to say you are taking something, it is
yours and nothing will be merged over you. That promise exists because it was broken:
someone claimed #32 at 07:29 one morning, a parallel PR doing the same work was opened
four and a half hours later, and the claim was never even acknowledged. Details and
apology are on that issue.
If you claim something and I go quiet for more than a couple of days, ping it — that is my
failure, not impatience on your part.

If you have claimed something and changed your mind, just say so. Nobody minds, and it
frees it for the next person.

Housekeeping note — 2026-08-10

Three issues that were labelled good first issue (#1, #4, #5) described work that was
already on main when they were opened. They have been closed with an explanation and
replaced with a verified set (#26–#39). If you were looking at one of them, sorry — each
closing comment names its live replacement.

Every issue under the label is now checked against main. If you find another that is
already done, say so on the issue and I will close it and credit you for the catch.

Housekeeping note — 2026-09-10

This map had drifted into exactly the failure it apologises for above: four of the five
issues it recommended as first contributions had already shipped — #29, #31 and #32
merged from external contributors, #37 likewise. Anyone arriving here since late August
clicked four suggestions and found closed issues. That is a bad first impression and it
was mine to prevent, not yours to work around. The list above is rebuilt from issues
verified open today, and the thin-cell count is corrected from 26 to 32 of 50, which
is what the corpus actually shows.

If you find another stale issue, say so on it — I will close it and credit you for the
catch, and that catch goes on the contributor
wall the same as a patch does.

Housekeeping note — 2026-09-11

This list was rebuilt yesterday and went stale again the same day: #65 and #66 were
both recommended here as first contributions this morning, and both were merged and closed
by the afternoon. Fixed above, with the finished work credited rather than deleted.

Then, four minutes after that rebuild, the shelf was refilled — #69 and #70, both
good first issue / effort: small, both free, both listed above. Both were found the
honest way, by running this project's own quality gates and reading what they printed, and
both were proved against main before the issue was typed: #69's gate genuinely reports OK
today while examples/04_ci_gate.py has two live encoding-less read_text() calls, and
#70's root cause was reproduced and reverted to confirm the warning comes back. The
diagnosis in each is tested rather than guessed, so neither sends you chasing a premise that
was never checked.

That churn is worth being straight about, because it is the honest shape of this project
right now: the shelf of small, self-contained issues empties faster than one maintainer
refills it, and it refills mostly because finishing one issue surfaces the next — #61
surfaced #65 and #66; #70 is the same class of defect as #65, found by looking for more of
it. If the small list looks short on the day you arrive, check again, and check the
good first issue label
directly rather than trusting this page alone.

The two most valuable contributions here have never been the small ones, though, and
neither needs cluster access, an API key, or any knowledge of the internals:

• Write a second task for a thin QCAT × role cell (32 of 50 have only one) #26 — write one task. 31 of the 50 QCAT × role cells hold exactly one task, so on
  those cells the benchmark cannot distinguish "understands the category" from "got lucky".
  One JSON file fixes one cell.
• Run a model and submit a leaderboard entry — negative results welcome #27 — run a model and post the numbers. A benchmark whose only results come from its
  own author is a claim, not a benchmark.

If small and mechanical is what you have time for, #62 is the deepest seam left: 136
documented commands across 24 pages, one page per PR, and being new here is an advantage,
because the places you get stuck are the finding.

Housekeeping note — 2026-09-12

Seven issues were opened in the last day and every one of them came out of doing the work
rather than from a planning session: #69 and #70 fell out of finishing #61;
#71 and #72 out of re-running the gates; #73 out of testing aobench review task while it was two hours old; and #74 and #75 out of sweeping every spec in the
corpus with it. #73 is already closed — @mgalore fixed it
in PR #77 within hours of it being filed.
Of the six still open, none is claimed and four are effort: small (#69, #70, #71,
#74).

(Corrected 2026-09-12: this note first said six issues, that #73 was filed by a
contributor rather than fixed by one, and that four were unclaimed with three small. The
counts are now checked against the tracker rather than written from memory.)

That is the honest shape of this project: the backlog refills from contact with the code,
so the list above goes stale within hours of a merge rather than within weeks. If something
here is already closed, say so on it — that catch is credited on the
contributor wall exactly like a
patch.

Housekeeping note — 2026-09-19

Three issues suggested above as open work are now closed, all three by
@motodriver: #70 (the leaked-coroutine warning,
PR #80), #75 (the scaffold-scoring bug,
PR #82), and #71 (the test-count fact
gate, PR #83). PERF_RES_002
(PR #81 by
@yangziao56) also merged. #74 remains open and is now
claimed — see the table below.

If you click through from this page and find something already closed, say so on it; that
catch is credited exactly like a patch, as noted above.

Housekeeping note — 2026-09-12 (evening)

The first corpus contribution landed. @userfypp filled the
DOCS_USR cell (#78) — the first task AOBench
has ever received from outside the project. Thin cells: 32 → 31.

Two things changed as a result, both worth knowing before you pick something:

• Write a second task for a thin QCAT × role cell (32 of 50 have only one) #26 was auto-closed and has been reopened. The PR said Closes #26, which was correct
  of the author and wrong for the backlog — Write a second task for a thin QCAT × role cell (32 of 50 have only one) #26 is a standing call across all 50 cells. If
  you see it closed again after a task merges, that is the same mechanic, not a decision.
• Corpus work is much cheaper now than when this page was written. aobench new task --thinnest writes a valid spec for the emptiest cell, and make review runs the same
  checklist CI runs on your PR. Neither existed a week ago.

How it works

1. Comment on the issue to say you are taking it. No need to wait for permission on anything labelled good first issue.
2. Read CONTRIBUTING.md — clone to green tests is three commands.
3. Open a PR. Prefer under ~300 changed lines.

You will get a first response within 3 working days, even if it is only "seen, I will look properly on Friday". If a PR of yours goes quiet for more than a week, ping it — that is our failure, not rudeness on your part.

Questions are welcome and expected. Ask here, in the issue itself, or in Discussions.