Review agent: verify diff completeness against PR's stated scope for bulk-change PRs

[fullsend-ai-retro]

What happened

On PR #2508, the author stated the goal was to "pin all GitHub Actions to full-length commit SHAs." The fullsend review agent ran 4 times across 3 push cycles. It correctly flagged 3 unpinned actions in renovate.yml (which was in the diff) but missed two categories of omission that a human reviewer caught:

1. Out-of-diff composite action: .github/actions/setup-gcp/action.yml contained an unpinned google-github-actions/auth@v3 — the same action that was SHA-pinned in e2e.yml within the diff. This composite action is called by 6+ reusable workflows, giving it a wider attack surface. (HIGH finding by waynesun09, review comment)

2. Missing file from branch: functional-tests.yml existed on main but was absent from the PR branch (merged after branch was cut). It contained 6 unpinned third-party actions including WIF credential handling. (HIGH finding by waynesun09, second review)

Both gaps required the author to push additional fix commits, adding 1 full rework cycle that could have been avoided.

What could go better

The review agent's correctness and intent-coherence sub-agents should detect when a PR's title/description signals a bulk or exhaustive change (patterns like "pin all", "rename all", "migrate all X to Y") and verify that the diff actually covers all matching files in the repo.

In this case, a simple grep -r 'uses:' .github/ internal/scaffold/ filtered against the diff's file list would have revealed both the composite action gap and any workflow files not included in the PR.

I'm fairly confident this is a recurring pattern — bulk-change PRs are inherently scope-sensitive, and the current review agent architecture evaluates the diff in isolation without checking whether the diff fulfills the PR's stated intent. The intent-coherence sub-agent currently checks whether changes are within scope (not out-of-scope), but does not check whether the scope is complete.

Existing issue #1275 (trace workflow_call chains) would partially help by detecting the composite action gap, but would not catch the missing functional-tests.yml or generalize to non-Actions bulk changes. Issue #1774 is about suppressing false positives for incomplete multi-site changes — the inverse of what's needed here.

Proposed change

Add an intent-completeness check to the intent-coherence sub-agent definition at skills/pr-review/sub-agents/intent-coherence.md.

The check should:

1. Parse the PR title and description for bulk-change intent signals (e.g., "all", "every", "each", "migrate", "rename across", "pin all").
2. When detected, infer a file pattern or search query from the PR's stated scope (e.g., for "pin all GitHub Actions" → search for uses: in **/*.yml under .github/ and scaffold dirs).
3. Compare files matching that pattern against the PR's changed-file list.
4. If files exist in the repo that match the scope but are not in the diff, emit a medium+ finding listing the omitted files.

This should be scoped narrowly to avoid false positives — only trigger when the PR description explicitly claims exhaustive coverage. The finding should be informational ("these files may also need changes") rather than asserting they definitely do, since some omissions may be intentional.

Validation criteria

On the next 3 PRs to this repo that claim to make a bulk/exhaustive change (e.g., "update all X", "pin all Y"), the review agent should identify any files matching the stated scope that are not in the diff. Measure: (1) the review agent flags at least one omitted file that a human would also flag, and (2) no more than 1 false positive per review where the omission was intentional.

---

Generated by retro agent from #2508

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 6:28 PM UTC · Completed 6:31 PM UTC
Commit: 6bd5f3a · View workflow run →

[fullsend-ai-triage]

Triage Summary

Category: Feature · Severity: Medium · Component: Review agent intent-coherence sub-agent

Problem

The intent-coherence sub-agent verifies that PR changes are within scope of the stated intent, but does not verify that the diff covers the full scope. For bulk-change PRs ("pin all X", "rename all Y"), files matching the stated scope but missing from the diff go undetected — as demonstrated on PR #2508 where an unpinned composite action and a missing workflow file were both caught by a human reviewer instead.

Recommended Approach

Add an intent-completeness check to skills/pr-review/sub-agents/intent-coherence.md:

1. Detect bulk-change intent signals in PR title/description ("all", "every", "each", "migrate", "pin all", etc.)
2. Infer a file pattern or search query from the stated scope
3. Compare matching repo files against the PR's changed-file list
4. Emit a medium+ informational finding listing omitted files

Keep the trigger narrow — only activate when the PR explicitly claims exhaustive coverage. Findings should suggest ("these files may also need changes") rather than assert.

Related Issues

• Review agent: add completeness check for multi-site patterns #1213 — trace workflow_call chains (would catch composite action gaps for Actions-specific cases)
• Code agent should grep-verify completeness after rename/terminology tasks #1794 — code agent grep-verify after rename tasks (analogous feature for the code agent)
• Review agent: verify functional impact before flagging incomplete multi-site changes #1774 — suppress false positives for incomplete multi-site changes (inverse concern)

Proposed Test Case

Scenario: Bulk-change PR with omitted files
  Given a PR with description "pin all GitHub Actions to commit SHAs"
  And the repo has 5 workflow files containing `uses:` directives
  And the diff modifies only 3 of those files
  Then the intent-completeness check should flag the 2 omitted files

Scenario: Non-bulk PR (no false positive)
  Given a PR with description "update checkout action version"
  And only one workflow file is modified
  Then no intent-completeness finding should be emitted

Edge case: composite actions under .github/actions/ should be
included when the scope is "all GitHub Actions" but may be
excluded when the scope specifies a narrower path.

---

Labels: Retro-generated feature request targeting the review agent's intent-coherence sub-agent, which is part of the harness-loaded skill infrastructure.

[fullsend-ai-prioritize]

RICE Priority Score: 1.2

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	The review agent runs on all PRs across the platform, but this enhancement only triggers on bulk-change PRs (those claiming exhaustive scope like 'pin all', 'rename all'). These PRs occur regularly across multiple teams and repos, and the security implications (e.g., unpinned GitHub Actions with credential handling) affect the whole org. Score reflects multiple strategic users being affected by the gap in review coverage.
Impact	1.5	When triggered, this check catches completeness gaps that led to a full rework cycle on PR #2508 and could have security implications (unpinned actions handling WIF credentials). The impact per affected user is significant — it prevents missed files in security-sensitive bulk changes and saves review round-trips — but it's an enhancement to an existing review flow rather than fixing a broken core feature.
Confidence	0.8	High confidence due to the extremely well-documented issue with a concrete real-world example (PR #2508), clear proposed implementation scoped to a single sub-agent definition file, specific validation criteria, and test cases. The triage summary also confirmed the approach. Minor uncertainty around false-positive tuning for the bulk-change signal detection.
Effort	1.5	Moderate complexity — requires modifying the intent-coherence sub-agent definition, implementing pattern detection for bulk-change signals in PR titles/descriptions, inferring file search patterns from stated scope, and comparing against the changed-file list. Careful tuning is needed to keep the trigger narrow and avoid false positives. Related issues (#1275, #1774) suggest this interacts with other sub-agent concerns.

Formula: (1.5 x 1.5 x 0.8) / 1.5 = 1.2