Repository navigation
Conversation
|
CI note: current GitHub Actions failures are not limited to this PR's changed files. Observed repository-baseline blockers on the PR runs:
Therefore this PR remains Draft. Its own changed-path verification should be reviewed separately from the current repository-wide CI baseline; no claim of a green full suite is being made. |
Absorb the more accurate classification from draft PR #10 (superseded by the audit_rows hard-gate design already on main): an unknown criterion is a grader-config error, not an agent failure. grade() now returns INVALID / grader_bug before the criterion loop instead of failing the criterion in-line. _check_criterion keeps its fail-closed CONFIG_ERROR fallback.
|
Superseded by ef30fcc (adversarial-optimization Stage B). Main now grades real execution from |
|
Follow-up audit (2026-09-23): the important trusted-execution invariants from this closed PR are already present on current main. The grader derives actual executed Operations from OperationAuditLog rather than trusting command-shaped trace text, and unknown outcome criteria fail closed as INVALID / grader_bug. PR #16 now documents this boundary in docs/evals/GRADING.md, so this PR should remain closed rather than be revived wholesale. |
Why
The current live-eval grader can infer an Operation from shell text alone. A synthetic trace containing
echo "python -m app.cli run get_profile"can therefore look like execution even when the Operation Registry never ran.The product critique also found that an unknown outcome criterion is currently silently accepted.
What changes
Trace.requested_operationsfrom trustedexecuted_operationsOperationAuditLogrows in the runneroperations_usedas request ∩ trusted audit receiptoperations.jsonINVALID / grader_bugNon-goals
Verification
This PR is intentionally opened as Draft first. Repository CI should verify the focused eval tests and broader regression suite before it is considered mergeable.
The key invariant is: