Review agent should self-report coverage gaps on large PRs

[fullsend-ai-retro]

What happened

On PR #4100, the review agent ran 6 passes on a 3,900-line greenfield PR implementing a GitLab cron poller (16 files across internal/poll/ and internal/dispatch/). On every pass it approved the PR, finding only API contract mismatches, error handling gaps, and surface-level issues (8 findings on first pass, 4 on second). A separate review-squad with ADR-0067 context then found 14 additional issues — 1 CRITICAL (unbounded dedup key growth hitting the 10K CI variable limit), 5 HIGH (transient read clobber, dispatch failure bypassing retry budget, poison event stall, bot detection blind spots) — all in business logic areas the standard agent never examined in depth: dedup state management, watermark advancement, label state diffing, and failure retry semantics.

What could go better

The review agent provided no signal about what it did and did not check. Its APPROVED verdict implied comprehensive coverage, but it had only reviewed a subset of the code's concerns. The human author had no way to know which areas lacked deep review until the review-squad independently discovered CRITICAL bugs in unchecked areas 6 days later. If the agent had reported 'I reviewed API contracts, error handling, and YAML generation in depth; I did not deeply verify dedup logic, state persistence, or watermark advancement,' the author could have triggered a targeted deeper review or checked those areas manually. This is distinct from existing proposals: #3007 proposes cross-referencing ADRs (which helps the agent find more bugs), while this proposal helps the human understand what the agent missed regardless of whether ADR context is available. Confidence: high — the coverage gap is clearly demonstrated by the delta between agent findings and review-squad findings on the same code.

Proposed change

Add a coverage self-assessment section to the review agent's output for PRs above a configurable size threshold (suggested: >1000 lines changed or >8 files). The review agent definition in fullsend-ai/agents should be updated to instruct the agent to include a 'Coverage' block in its structured output listing: (1) which code areas / concerns it reviewed in depth, (2) which areas it only skimmed or did not check, and (3) a confidence level for its overall verdict (high/medium/low). The post-review script should surface this coverage block in the review summary comment so the human author can see it. For example: 'Reviewed in depth: API types, CLI wiring, error handling, YAML generation. Not deeply checked: dedup state management, watermark persistence logic, label diffing edge cases. Verdict confidence: medium.'

Validation criteria

On the next 5 large PRs (>1000 lines) reviewed by the review agent on fullsend-ai/fullsend, the review output should include a coverage self-assessment section. Spot-check 2-3 of these to verify the coverage report is accurate (the areas listed as 'not deeply checked' should correlate with areas where human reviewers or review-squad agents subsequently find issues). The coverage section should not be present on small PRs (<200 lines) to avoid noise.

---

Generated by retro agent from fullsend-ai/fullsend#4100

[fullsend-ai-prioritize]

RICE Priority Score: 1.8

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	The review agent is used across the organization on all PRs. This change specifically targets large PRs (>1000 lines or >8 files), which are a meaningful subset where coverage gaps matter most. Multiple teams and authors submitting large PRs would benefit from coverage transparency, affecting multiple active users across the org.
Impact	1.5	This addresses a significant workflow gap: the review agent's APPROVED verdict currently implies comprehensive coverage when it may have only checked a subset of concerns. The concrete example shows CRITICAL bugs (unbounded dedup key growth) and 5 HIGH issues were missed because the author had no signal about unchecked areas. Adding coverage self-assessment gives authors actionable information to trigger targeted deeper reviews, meaningfully reducing the risk of critical bugs shipping undetected.
Confidence	0.8	High confidence — the issue is exceptionally well-documented with a specific real-world example (PR #4100), a clear delta between agent findings and review-squad findings, concrete proposed changes, explicit validation criteria, and a clear distinction from the related ADR-context proposal (#3007). The scope and expected behavior are unambiguous.
Effort	1	Medium effort — requires updating the review agent prompt/definition to produce a structured coverage self-assessment block, modifying the post-review script to parse and surface it in the summary comment, and adding a configurable size threshold to gate the behavior. Touches the agent definition and post-review pipeline but is well-scoped and doesn't require architectural changes.

Formula: (1.5 x 1.5 x 0.8) / 1 = 1.8