Skip to content

Generate and send a real RBAC-hard-fail fixture to the EvalPort spec (discussion #51) #85

Open

Description

@MSKazemi

Context

In discussion #51, we offered
@adhabnr-ux a real conformance fixture for their
EvalPort interchange spec: an actual Result
document emitted from a genuine aobench run that RBAC-hard-fails, rather than a
hand-authored example.

What to do

  1. Find or construct a task/trace combination that RBAC-hard-fails for a legitimate
    reason — an agent actually attempting a forbidden tool call or crossing a role boundary.
    Do not use AIOPS_USR_001 or PERF_USR_001 — those hard-fail because of the
    "own-job" premise mismatch tracked in #63,
    which is a corpus defect, not a legitimate policy violation, and would misrepresent what
    a real hard fail looks like to a third party building against it.
  2. Run it, capture the real BenchmarkResult / grader output, and verify by hand that
    hard_fail=True with a genuine hard_fail_reason and real dimension_scores.
  3. Write a small generation script (in the style of scripts/audit_perf_res_002.py) so the
    fixture is regeneratable against a pinned AOBench version rather than a one-off paste.
  4. Open it as a PR against evalport (external repo — needs Mohsen's own review/approval
    before opening, since it's outward-facing to a project we don't own).
  5. Reply on discussion Idea: aobench-openeval-adapter for portable eval interchange (EvalPort) #51 linking the PR.

Acceptance criteria

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: docsDocumentation and examples

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions