Review agent outputs approve despite detecting protected-path constraint

[fullsend-ai-retro]

What happened

On PR #195, the review agent detected protected paths and stated it cannot approve — yet output approve. Same pattern on PR #37 (#52).

What could go better

The agent generates narrative text about the constraint but no structured finding to force the verdict. The pr-review SKILL.md step 6e rule isn't mechanically enforced. Confidence: HIGH.

Proposed change

In skills/pr-review/SKILL.md, require that protected-path matches produce a structured finding with source: 'protected-path' and severity critical, forcing action to comment. Consider a schema constraint in schemas/review-result.schema.json rejecting action=approve with protected-path findings.

Validation criteria

Next 3 review runs on PRs touching protected paths should never output action=approve. Verify via agent-result.json artifacts.

---

Generated by retro agent from #195

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 11:51 AM UTC · Completed 11:56 AM UTC
Commit: 7d0bfa9 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: High — governance bypass allowing automated approval of protected-path changes

Problem

The review agent detects protected-path modifications and generates narrative text stating it cannot approve, but still emits action: "approve" in structured output. This has occurred on at least two PRs (PR #195 and PR #37, previously documented in #52).

The SKILL.md step 6e already has explicit rules ("the outcome MUST NOT be approve" when protected-path findings exist, and "Never approve when any protected-path finding exists"), but these are narrative guidance only — no structured mechanism enforces them.

Root Cause Hypothesis

The protected-path check produces narrative text but no structured finding object that mechanically forces the verdict. The LLM-based verdict selection can ignore or override narrative constraints. Additionally, the schema (review-result.schema.json) lacks a constraint preventing action=approve when protected-path findings are present, so no validation catches the violation.

Recommended Fix

1. SKILL.md: Require that protected-path matches produce a structured finding with category: 'protected-path' and severity: 'critical', mechanically forcing the verdict away from approve.
2. Schema: Add a conditional constraint in review-result.schema.json that rejects action=approve when any finding has category: 'protected-path'. This provides defense-in-depth at validation time.

Proposed Test

# Schema validation tests:
# 1. REJECT: action=approve + finding with category='protected-path' → schema validation fails
# 2. ACCEPT: action=comment + finding with category='protected-path' → valid
# 3. ACCEPT: action=approve + no protected-path findings → valid
#
# Integration test:
# Run review agent on a PR touching .github/ or CODEOWNERS
# Assert agent-result.json has action != 'approve'
# Assert findings array contains entry with category='protected-path'

Related Issues

• Review agent verdict contradicted its own analysis on PR #32: chose comment despite saying findings should be addressed before merge #45 (broader verdict-inconsistency issue, ready-to-code) — complementary but addresses general medium-finding escalation, not protected-path enforcement specifically
• Evidence for #45: first review approved despite protected-path detection on PR #37 #52 (closed as duplicate of Review agent verdict contradicted its own analysis on PR #32: chose comment despite saying findings should be addressed before merge #45) — documented the same protected-path approval pattern on PR ci: add Makefile and workflow for agent script tests #37
• Review agent silently approves when pr-review sub-agent orchestration fails #285 — related pattern where the review agent silently approves on orchestration failure

---

Labels: This is a review-agent bug with high severity — it bypasses governance controls on protected paths.

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 11:57 AM UTC · Completed 12:04 PM UTC
Commit: 7d0bfa9 · View workflow run →