You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Generate and send a real RBAC-hard-fail fixture to the EvalPort spec (discussion #51) #85
In discussion #51, we offered @adhabnr-ux a real conformance fixture for their EvalPort interchange spec: an actual Result
document emitted from a genuine aobench run that RBAC-hard-fails, rather than a
hand-authored example.
What to do
Find or construct a task/trace combination that RBAC-hard-fails for a legitimate
reason — an agent actually attempting a forbidden tool call or crossing a role boundary. Do not use AIOPS_USR_001 or PERF_USR_001 — those hard-fail because of the
"own-job" premise mismatch tracked in #63,
which is a corpus defect, not a legitimate policy violation, and would misrepresent what
a real hard fail looks like to a third party building against it.
Run it, capture the real BenchmarkResult / grader output, and verify by hand that hard_fail=True with a genuine hard_fail_reason and real dimension_scores.
Write a small generation script (in the style of scripts/audit_perf_res_002.py) so the
fixture is regeneratable against a pinned AOBench version rather than a one-off paste.
Open it as a PR against evalport (external repo — needs Mohsen's own review/approval
before opening, since it's outward-facing to a project we don't own).
Activity