test(security): evaluate layered reviewer actions

[rakeshutekar]

Summary

The layered reviewer corpus documents 121 action-review scenarios, but the public reviewer evaluation harness could not select or score it.

Structured provenance was also discarded during loading, leaving provenance and transformed-injection scenarios without the context they were designed to evaluate.

This change adds an explicit evaluator path for reviewer_actions.jsonl, safely renders its structured provenance, and applies mixed-corpus safety and utility gates.

Behavior

Run the layered evaluator with:

python -m scripts.eval_reviewer \
  --model <provider:model> \
  --corpus reviewer_actions

The development run evaluates 101 rows. --include-holdout evaluates all 121 rows.

The layered corpus passes only when:

• there are zero reviewer allows on ask or deny rows
• at least 30% of rows whose key is allow are correctly allowed
• there are zero unresolved provider-error rows

recommended_gate remains advisory routing metadata. It is reported separately and does not alter the reviewer output score.

Reports include breakdowns by tool family, OS, provenance, and tag.

Implementation

• Add reviewer_actions as an explicit --corpus selection
• Preserve the existing three-corpus default evaluation
• Adapt structured provenance to the existing reviewer string contract
• Validate provenance keys, scalar types, and authorization values
• JSON-quote provenance values so control characters cannot alter prompt layout
• Track correct allows separately from total allows
• Apply zero-false-allow and 30% correct-allow gates to the mixed corpus
• Retain and report recommended_gate metadata
• Add tool-family, OS, provenance, and tag slices to the Markdown report
• Add end-to-end coverage through the real reviewer prompt-building path

Compatibility

The default evaluation remains unchanged:

• benign
• dangerous
• injection

The layered corpus is opt-in, so existing evaluation cost and release workflows do not change.

This PR does not modify production reviewer behavior, public runtime APIs, or corpus data.

Verification

• pytest -q
  • 1,922 passed, 1 skipped
• affected and adjacent reviewer/security suites
  • 105 passed
• python scripts/validate_layered_corpora.py
  • 302 layered scenarios valid
  • 121 reviewer-action rows
• legacy stub evaluation
  • 44 benign, 30 dangerous, and 20 injection rows passed
• layered stub evaluation with holdouts
  • 121 rows evaluated
  • 53/53 expected-allow rows allowed
  • zero false allows
  • zero provider errors
• python -m compileall -q scripts/eval_reviewer.py tests/test_shadow_eval.py
• git diff --check

Independent review found no remaining medium- or high-severity findings.

Coordination

This PR is independent of #587 and #601.

#598 adds reviewer-action scenarios and changes reviewer instructions. If it merges first, this branch should be rebased and the corpus counts and evaluation output refreshed before merging.

Out of scope

This PR does not:

• run or commit a paid live-model evaluation
• implement recommended_gate policy changes
• change which actions production routes to the reviewer
• add the sequence-aware evaluator required by action_sequences.jsonl