What happened
PR #609 migrated documentation away from the deprecated .fullsend/customized/skills/ overlay. The code agent updated 3 files and claimed in the PR description: "Verified zero remaining customized/skills references via grep -r 'customized/skills'." However, FEATURES.md:37 still contained the deprecated reference and FEATURES.md:137 used stale "agent frontmatter" terminology.
The review agent's initial run (30654105629, Jul 31) did not catch the FEATURES.md references. The second review (31423700078, Aug 10, after rebase) found both issues. This gap added a third fix iteration and ~2 days of calendar time before the PR could be approved.
What could go better
The docs-currency sub-agent already has guidance for a "deprecation pattern strategy using bare-word and syntax-specific grep patterns," but the initial review did not apply it to search beyond the diff. For migration PRs — where the explicit goal is removing all references to a deprecated pattern — the sub-agent should systematically grep the entire repo for remaining references, not rely on stochastic exploration.
The second review caught the issue, confirming the capability exists but is applied inconsistently. Making it deterministic for migration PRs would eliminate the variance.
The code agent's grep verification claim ("zero remaining references") was also inaccurate. Consistent with the review agent's zero-trust principle ("does not trust other agents or claims about changes"), the review agent should independently verify such completeness claims rather than accepting them at face value.
Confidence: High that this is a real gap — the evidence is direct. Moderate confidence the proposed fix will work, since the underlying search capability already exists and just needs a stronger trigger.
Proposed change
In skills/pr-review/sub-agents/docs-currency.md, add explicit guidance for migration/deprecation PRs: when the PR title or description indicates migration away from a deprecated pattern (keywords like "migrate," "deprecate," "remove references," "per ADR"), the sub-agent should grep the entire repo for remaining references to the deprecated string before finalizing findings. If references remain outside the diff, emit an incomplete-migration finding.
Additionally, add a check to independently verify grep-based completeness claims in the PR description — when the PR claims "zero remaining X references," run the same grep and confirm.
Validation criteria
On the next 3 migration/deprecation PRs in this repo, the review agent's initial pass should detect any remaining references to the deprecated pattern outside the diff. Zero false negatives on deprecated-pattern references findable via a simple grep. Measurable by comparing the initial review's findings against a manual grep -r for the deprecated string.
Generated by retro agent from #609
What happened
PR #609 migrated documentation away from the deprecated
.fullsend/customized/skills/overlay. The code agent updated 3 files and claimed in the PR description: "Verified zero remaining customized/skills references viagrep -r 'customized/skills'." However, FEATURES.md:37 still contained the deprecated reference and FEATURES.md:137 used stale "agent frontmatter" terminology.The review agent's initial run (30654105629, Jul 31) did not catch the FEATURES.md references. The second review (31423700078, Aug 10, after rebase) found both issues. This gap added a third fix iteration and ~2 days of calendar time before the PR could be approved.
What could go better
The docs-currency sub-agent already has guidance for a "deprecation pattern strategy using bare-word and syntax-specific grep patterns," but the initial review did not apply it to search beyond the diff. For migration PRs — where the explicit goal is removing all references to a deprecated pattern — the sub-agent should systematically grep the entire repo for remaining references, not rely on stochastic exploration.
The second review caught the issue, confirming the capability exists but is applied inconsistently. Making it deterministic for migration PRs would eliminate the variance.
The code agent's grep verification claim ("zero remaining references") was also inaccurate. Consistent with the review agent's zero-trust principle ("does not trust other agents or claims about changes"), the review agent should independently verify such completeness claims rather than accepting them at face value.
Confidence: High that this is a real gap — the evidence is direct. Moderate confidence the proposed fix will work, since the underlying search capability already exists and just needs a stronger trigger.
Proposed change
In
skills/pr-review/sub-agents/docs-currency.md, add explicit guidance for migration/deprecation PRs: when the PR title or description indicates migration away from a deprecated pattern (keywords like "migrate," "deprecate," "remove references," "per ADR"), the sub-agent should grep the entire repo for remaining references to the deprecated string before finalizing findings. If references remain outside the diff, emit anincomplete-migrationfinding.Additionally, add a check to independently verify grep-based completeness claims in the PR description — when the PR claims "zero remaining X references," run the same grep and confirm.
Validation criteria
On the next 3 migration/deprecation PRs in this repo, the review agent's initial pass should detect any remaining references to the deprecated pattern outside the diff. Zero false negatives on deprecated-pattern references findable via a simple grep. Measurable by comparing the initial review's findings against a manual
grep -rfor the deprecated string.Generated by retro agent from #609