Write a second task for a thin QCAT × role cell (32 of 50 have only one)

[MSKazemi]

This issue stays open until every cell has more than one task. It is a standing call
across all 50 cells, not a single task — GitHub closed it automatically when the first one
landed (PR #78 by
@userfypp, which filled DOCS_USR), and it has been
reopened. 31 cells are still thin. If you are looking for the most useful thing you
can do here, it is still this.

Problem

Corpus breadth is the main thing limiting what AOBench can measure, and coverage is
lopsided. MON_SYS and ENERGY_FAC have 6 tasks each, while 31 of the 50
QCAT × role cells have exactly one task. A cell with a single task cannot distinguish
"the agent understands this category" from "the agent got lucky on this item".

Cells with exactly one task today:

AIOPS_DES  AIOPS_RES  AIOPS_USR  ARCH_FAC  ARCH_RES  ARCH_SYS  ARCH_USR
DATA_DES   DATA_FAC   DATA_RES   DATA_SYS  DATA_USR  DOCS_DES  DOCS_FAC
DOCS_RES   DOCS_SYS   ENERGY_DES ENERGY_RES FAC_DES   FAC_FAC   FAC_RES
FAC_SYS    FAC_USR    JOB_DES    JOB_RES    MON_DES   PERF_DES  PERF_FAC
PERF_RES   SEC_DES    SEC_RES

Regenerate the list yourself at any time:

ls benchmark/tasks/specs/*.json | xargs -n1 basename \
  | sed -E 's/^M100_//; s/_[0-9]+\.json$//' | sort | uniq -c | sort -n

# The `s/^M100_//` matters: an M100 task spec is named M100_<QCAT>_<ROLE>_NNN.json,
# so without stripping that prefix the count shows spurious "M100_ENERGY_SYS" style
# rows instead of folding those tasks into the real cell they belong to.

Desired result

One new task spec in a cell that currently has one task, reusing an existing environment
bundle.

You do not need cluster access, an API key, or a GPU to do this. That is the entire
point of the snapshot design, and it is why this is the most valuable contribution
anyone can make to AOBench.

Where to look

Path	Why
docs/guides/adding-a-task.md	Read this first. The complete path from idea to merged, including the reviewer checklist. 159 lines, all of them worth it
src/aobench/schemas/task.py	The TaskSpec model — authoritative, more so than the guide
benchmark/tasks/specs/	Where your new <QCAT>_<ROLE>_<NNN>.json goes; the existing task in your chosen cell is the model to copy
docs/reference/environment-catalog.md	Pick an environment that already supports your task

Implementation hints

1. Pick a cell from the list above. Open its one existing task and read it end to end.
2. Reuse that task's environment bundle. Authoring a new environment is a separate,
   larger contribution — see docs/guides/adding-an-environment.md.
3. Your gold answer must be derivable from evidence that actually exists in the
   snapshot. If answering requires knowledge the agent has no way to obtain, the task
   measures memorisation and it will not merge.
4. The task must be role-sensitive: a competent operator in a different role should
   answer differently, or need different tool access. If not, it is a knowledge question
   and belongs in DOCS at best.
5. Set benchmark_split to dev. The test split is held out and is not open for
   contribution.

Acceptance criteria

• uv run aobench validate benchmark passes with the new task included
• The F1–F7 fidelity gate passes (run without AOBENCH_SKIP_FIDELITY=1)
• Every tool call in the gold trajectory is permitted for the declared role under the
  environment's rbac_policy.yaml
• Every claim in the gold answer traces to a file in the environment snapshot
• uv run aobench run task --task <NEW_ID> --env <ENV> --adapter direct_qa completes
• The reviewer checklist at the end of adding-a-task.md is satisfied

Tests

No new test file is normally needed — the corpus validators cover it:

uv run aobench validate benchmark
uv run python -m pytest tests/unit/ -k "task or spec or validate"

Difficulty

Roughly a day for your first one, most of it spent reading the environment snapshot
rather than writing JSON. Much faster after that — which is why this is the lane we most
want people in.

What you get for it

Stated plainly, because this project asks for corpus work more than for code and the credit
for it is easy to leave implicit:

• Your name lands on a public page with a stable URL. The contributor
  wall and
  AUTHORS.md describe what each
  person actually did, in specific terms, not as a row of avatars. If that is useful to you
  as third-party evidence — a CV, a profile, a funding application — please use it. That is
  what it is for.
• Release notes name contributors for the version their change shipped in.
• Co-authorship is on the table for this kind of work. The
  recognition policy
  says it in its own words: "Substantial corpus or methodological contributions may warrant
  co-authorship on a paper that depends on them. If you believe that applies to your work,
  say so — the awkwardness of asking should not decide who gets credit." It says may, and
  it means may — it is a conversation, not a promise attached to a single merged file. But
  corpus work is exactly the category the policy was written for, and asking is explicitly
  welcome.
• Contributions stay listed even if you later step away.

If you would rather not be listed at all, say so in the PR and you will not be. Opting out
is honoured immediately and without being asked why.

Getting started

Comment here with the cell you want to take. I will confirm it is unclaimed and
sanity-check the idea before you write anything — that costs you ten minutes and can
save you a day. Questions are expected, not a nuisance.

[userfypp]

Hi! I'd like to take the DOCS_USR cell.

I was thinking of adding a task using env_21 around the PII storage policy: a scientific user asks whether patient data can be stored on /lustre/scratch, where it is allowed to be stored instead, and what happens if the policy is violated.

The answer would be grounded in docs/data_management_policy.md. Does that sound like a suitable second DOCS_USR task?

[userfypp]

I started checking the task against env_21 before writing the spec and found a blocker.

The PII policy is in docs/data_management_policy.md, but MockDocsTool._retrieve() currently only returns the first 500 characters of each matched document. The ## 5. Compliance section is later, so queries for patient, PII, tier4_sensitive, or immediate account suspension don’t actually expose the relevant policy text to the agent.

I also checked the RBAC tool, but it only returns role permissions, not that policy content.

I haven’t changed anything yet because the task as proposed would have a gold answer the agent can’t retrieve with the current tools.

Would you prefer that I adjust the task idea to something the agent can retrieve today, or should the docs retrieval limitation be fixed first and then I add the PII task?

[MSKazemi]

You found a real bug, and a worse one than you described. Thank you — this is exactly the
kind of thing that only surfaces when someone actually tries to build against the tool.

I reproduced it against env_21. data_management_policy.md is 1090 characters, and
every term you named lives past character 900:

Term	Offset
compliance	901
patient	928
pii	974
tier4	1000
suspension	1078

_retrieve matched on the full document text but returned content[:500] as the
snippet. So the match succeeded and the evidence was then thrown away — the tool reported
a hit whose snippet contained none of the query terms. Silent, and worse than a plain
truncation: an agent could be scored on grounding against evidence the tool would never
surface, which quietly caps the grounding dimension on any task whose answer lives late
in a long document. That is a scoring-integrity bug, not a retrieval nicety, and it was
sitting under a comment that read # Return the first 500 chars as a snippet as though it
were intentional.

Fixed on main. Snippets are now a window centred on the match and snapped to line
boundaries, anchored on the most selective matched term — otherwise "where can patient
data be stored" centres on data, which appears in the title, and lands you back at
character 0. Retrieval order is unchanged, so it stays deterministic, which matters here.
Regression tests are in tests/unit/test_docs_tool_snippet.py, including your exact
queries.

Your queries now behave:

'where can patient data be stored'  -> patient=True  suspension=True
'PII'                               -> patient=True  suspension=True
'immediate account suspension'      -> patient=True  suspension=True
'scratch retention'                 -> returns the retention section, not compliance

So you are unblocked, and the DOCS_USR cell is yours. To answer the question you
originally asked: yes, that task is a good fit — a scientific user asking whether patient
data may live on /lustre/scratch, where it must go instead, and what happens if the
policy is violated, grounded in data_management_policy.md. It is a good second task for
that cell precisely because it is a refusal-shaped answer: the correct response is "no,
and here is the alternative", which is a different failure mode from the existing task and
therefore worth having.

Two suggestions now that the tool actually returns the clause:

• Put all three parts in gold_answer — the prohibition, the correct destination, and
  the consequence. An agent that says "no" without naming the alternative has half-answered
  a user-support question, and the scoring should reflect that.
• Set required_evidence_refs to the compliance section specifically. That is the part
  that was unreachable, so a task pinned to it is also a permanent regression test for the
  bug you just found.

Sorry you lost time to this before writing a line of the task. Ping me on the PR and I
will review quickly.

Optional and no obligation: if you found the project worth the dig, a ⭐ genuinely helps
it get discovered, and do feel free to post about the find — "traced a silent grounding
bug in an HPC agent benchmark" is a nice thing to have your name on, and the credit is
entirely yours.

[MSKazemi]

Reopened — this issue is not done. GitHub closed it automatically because
PR #78 said Closes #26, which was the right
thing for the author to write and the wrong outcome for the backlog: #26 is a standing call
across all 50 QCAT × role cells, not a request for one task.

DOCS_USR is filled — by @userfypp, with AOBench's first
corpus contribution from outside the project. 31 cells are still thin, and the issue body
now lists exactly which ones, with the count corrected from 32 to 31.

Two commands exist now that did not when this issue was written, and together they remove
most of the friction that made corpus work feel harder than it is:

aobench new task --thinnest      # picks the emptiest cell and writes a valid spec
make review                      # the same checklist CI will run on your PR

The scaffolder allocates the task ID, suggests the environment that comparable tasks already
use, and lists the files that actually exist in that snapshot so you can cite evidence
without going exploring. It deliberately leaves the question, the gold answer and the
evidence refs as TODO — those are the parts worth a human writing, and a generated gold
answer would only measure the generator.

Worth reading #78 as the model for what a good one looks like: the gold answer states that
the snapshot does not name a concrete permitted storage path rather than inventing one to
look complete. That restraint is the whole game here.

[yangziao56]

Hi Mohsen — I'm Ziao Yang, a PhD candidate at Brandeis University. I'd like to claim PERF_RES for a second development-set task, provisionally PERF_RES_002, reusing env_03.

The pilot would ask a researcher to diagnose the checkpoint performance of job 902117 from the available job evidence: approximately 180 MB/s versus an expected 480 MB/s is a 62.5% throughput reduction (37.5% of expected throughput). It would distinguish that calculation from the reported 25% CPU-frequency reduction and 71% CPU efficiency, attribute the thermal explanation to the snapshot's performance notes, and explicitly avoid identifying a slowest ensemble variant when no per-variant measurements are available. The workflow would stay within the researcher's permitted read-only tools.

If this direction is useful, I would propose a bounded six-task PERF coverage package, two additional tasks for each of PERF_RES, PERF_DES and PERF_FAC, with distinct role-dependent questions and a source-evidence consistency audit. I would first verify that each proposed answer is exposed by the actual tools and that existing snapshots support it; I would not invent data or change published tasks/environments to make a new answer work.

Could you confirm that PERF_RES is still unclaimed, and whether you would want this broader package? I am also interested in the paper co-authorship route described in your recognition policy: which planned paper would depend on such a contribution, and what concrete scope, quality and review responsibilities would qualify me for consideration as an individual co-author? I understand this is not promised by a single merged file, so I would like to agree expectations before expanding the pilot.

Work method disclosure: Codex is acting with my authorization to inspect the evidence, prepare the task and validation code, and handle technical revisions on my behalf. Substantial AI assistance will be disclosed in any PR. Please let me know if this agent-led workflow needs a particular additional review arrangement for corpus acceptance or authorship.

Thank you!

[yangziao56]

The single PERF_RES_002 pilot is now available for concrete scope review at commit fe95c6a. This is a reviewable fork branch, not a claim of assignment, acceptance, or agreed co-authorship.

It adds one development task against unchanged env_03, with an explicitly requested canonical answer format and the existing exact_match scorer. The task-local audit recomputes throughput loss from the actual researcher-bound slurm.job_details output. Wrong arithmetic, invented variants and unsupported causal claims fail outcome/CuP; a forbidden facility query is denied and fails RBAC/CuP (the aggregate can retain partial dimension credit).

Full local make check passed (1,655 passed, 24 skipped). A separate Codex diagnostic, given only the visible query and role-bound tools, reached the correct answer in eight tool calls. This was not an official frontier-model benchmark or a human-only trial, and its token usage was unavailable. Implementation and AI assistance are documented in the audit note, with a reproducible audit script.

The strict format is a deliberate tradeoff: an earlier semantic-match prototype gave substantial credit to incorrect numbers. I have not changed global scoring, existing tasks, or snapshots. I am keeping the broader six-task expansion on hold pending your scope/recognition response, and can revise this pilot in the existing branch once you have reviewed the direction.