What happened
PR #2659 replaced ubuntu-latest with ubuntu-24.04 across 37 files. The review agent ran 3 successful reviews and produced 4 medium findings, all at the design/architecture level (protected-path, runner injection, scope-creep, architectural-misalignment). Meanwhile, the Qodo bot identified that several documentation files (ADRs, plan docs, specs) still contained ubuntu-latest references that were not updated. The PR was merged with both sets of findings unaddressed.
What could go better
When a PR applies a mechanical pattern change (replacing string X with string Y across many files), the review agent should verify completeness by checking whether instances of the old value remain in files outside the diff. This class of finding — 'you replaced X in 30 files but missed it in 5 others' — is high-signal and low-false-positive. The review agent's existing sub-agents focus on correctness, security, and design, but none performs a consistency/completeness check for bulk mechanical changes. The Qodo bot caught this gap, suggesting it's detectable with a straightforward grep-based check. Confidence: high that this is a real gap. The review agent's sub-agent architecture is well-suited to add this without disrupting existing review dimensions. Issue #1222 proposes a similar check for the triage agent (recommending old-value searches for rename tasks), but no equivalent exists for the review agent to verify the change was applied completely.
Proposed change
Add a review sub-agent or extend an existing one (e.g., the correctness or completeness sub-agent) to detect mechanical migration patterns in PRs. When a PR contains many files with the same before/after string replacement (detectable from the diff), the sub-agent should grep the full repo for remaining instances of the old value, filtering to relevant file types (docs, configs, templates, examples). If unreplaced instances are found, emit a medium-severity finding listing the files and suggesting they be updated for consistency. This could be implemented as a new review dimension in the review orchestrator's sub-agent dispatch, or as an additional check in an existing sub-agent definition.
Validation criteria
On the next 5 PRs that apply a mechanical string replacement across 5+ files, the review agent should check for remaining instances of the old value and flag any that were missed. False positive rate should be below 20% (some old-value instances may be intentionally kept, e.g., in changelogs or migration guides).
Generated by retro agent from #2659
What happened
PR #2659 replaced
ubuntu-latestwithubuntu-24.04across 37 files. The review agent ran 3 successful reviews and produced 4 medium findings, all at the design/architecture level (protected-path, runner injection, scope-creep, architectural-misalignment). Meanwhile, the Qodo bot identified that several documentation files (ADRs, plan docs, specs) still containedubuntu-latestreferences that were not updated. The PR was merged with both sets of findings unaddressed.What could go better
When a PR applies a mechanical pattern change (replacing string X with string Y across many files), the review agent should verify completeness by checking whether instances of the old value remain in files outside the diff. This class of finding — 'you replaced X in 30 files but missed it in 5 others' — is high-signal and low-false-positive. The review agent's existing sub-agents focus on correctness, security, and design, but none performs a consistency/completeness check for bulk mechanical changes. The Qodo bot caught this gap, suggesting it's detectable with a straightforward grep-based check. Confidence: high that this is a real gap. The review agent's sub-agent architecture is well-suited to add this without disrupting existing review dimensions. Issue #1222 proposes a similar check for the triage agent (recommending old-value searches for rename tasks), but no equivalent exists for the review agent to verify the change was applied completely.
Proposed change
Add a review sub-agent or extend an existing one (e.g., the correctness or completeness sub-agent) to detect mechanical migration patterns in PRs. When a PR contains many files with the same before/after string replacement (detectable from the diff), the sub-agent should grep the full repo for remaining instances of the old value, filtering to relevant file types (docs, configs, templates, examples). If unreplaced instances are found, emit a medium-severity finding listing the files and suggesting they be updated for consistency. This could be implemented as a new review dimension in the review orchestrator's sub-agent dispatch, or as an additional check in an existing sub-agent definition.
Validation criteria
On the next 5 PRs that apply a mechanical string replacement across 5+ files, the review agent should check for remaining instances of the old value and flag any that were missed. False positive rate should be below 20% (some old-value instances may be intentionally kept, e.g., in changelogs or migration guides).
Generated by retro agent from #2659