Review agent should check git history before recommending re-addition of deliberately removed content

[fullsend-ai-retro]

What happened

On PR #5888, which restructured tracing documentation, the review agent recommended adding an ADR 0050 link to docs/guides/infrastructure/distributed-tracing.md for "discoverability per issue #5659." However, commit f058f7dd had deliberately removed that same ADR link from the infrastructure doc as a policy decision about audience separation. The human reviewer (waynesun09) caught this as a HIGH-severity finding, noting it reversed a prior deliberate decision. The author acknowledged and fixed it.

This is worse than a missed finding — the review agent's recommendation would have actively created the policy violation. The agent optimized for local consistency (the dev guide links ADR 0050, so the infra guide should too) without awareness that a prior commit established a different pattern for this specific file.

The review agent (resolved from fullsend-ai/agents@747566d) ran via dispatch run 30895237350 and subsequent iterations.

What could go better

The review agent lacks repository history awareness. It evaluates the current diff for internal consistency but has no mechanism to detect that content was previously present in a file and deliberately removed. When the agent sees a cross-reference in file A but not in file B, it recommends adding it to file B — a reasonable consistency heuristic in most cases, but harmful when file B deliberately omits that reference per a prior decision.

I am confident this is a real gap: the agent literally recommended the exact change the human identified as a policy violation. The agent's docs-review skill (skills/docs-review/SKILL.md) does not include any instruction to consult git history before recommending content additions.

Uncertainty: Implementing a full git-history check for every recommendation could be expensive and may generate false positives (content removed for cleanup, not policy reasons). The signal-to-noise ratio of git log -S results would need tuning.

Proposed change

Add a git-history awareness step to the review agent's documentation review skill (skills/docs-review/SKILL.md in fullsend-ai/agents). When the agent considers recommending additions of cross-references, links, or section content to a modified file, it should first run git log --oneline -S '<content_snippet>' -- <file_path> to check whether that content was previously present and removed. If the history shows a prior deliberate removal (especially with a descriptive commit message), the agent should note the prior removal context in its review rather than recommending re-addition.

Alternatively, a lighter-weight approach: when reviewing documentation restructuring PRs (detected by large section deletions paired with new file additions), instruct the agent to run git diff HEAD~1 -- <file> and flag any recommendation that would re-add content visible in the removal side of that diff, with a note to verify whether the removal was intentional.

Validation criteria

On the next 5 documentation PRs that modify files with prior deliberate content removals in git history, the review agent should not recommend re-adding that content. Instead, it should note the prior removal and defer to the author's judgment. Monitor for false positives where the agent incorrectly suppresses a valid consistency recommendation due to an unrelated prior removal.

---

Generated by retro agent from fullsend-ai/fullsend#5888

[fullsend-ai-prioritize]

RICE Priority Score: 0.53

Score breakdown

Dimension	Score	Reasoning
Reach	1	The review agent runs on PRs across the organization, but this specific failure mode — recommending re-addition of deliberately removed content — occurs only on documentation restructuring PRs where prior deliberate removals exist. This affects a moderate number of users (authors and reviewers of doc PRs processed by the agent), not a single edge case but not platform-wide either.
Impact	1	This is a meaningful workflow improvement: the agent actively produced a harmful false positive (recommending a policy violation) rather than merely missing something. However, the impact per affected user is medium — human reviewers are already catching these, the domain is documentation cross-references (not code correctness), and a workaround (human review) exists and is functioning. Fixing this reduces review noise and prevents potential policy regressions when human reviewers are less attentive.
Confidence	0.8	High confidence. The issue is exceptionally well-documented with specific PR, commit, and discussion references, a clear reproduction case, two proposed implementation approaches, and explicit validation criteria. The one uncertainty (acknowledged in the issue itself) is around implementation complexity — git-log checks could be expensive and may produce false positives from unrelated prior removals, requiring tuning.
Effort	1.5	Moderate effort. The change requires modifying the docs-review skill to add conditional git-history lookups (git log -S or git diff HEAD~1), designing heuristics to distinguish deliberate policy removals from routine cleanup, handling false positives, and validating across multiple documentation PRs. Two implementation approaches are proposed (full history check vs. lighter-weight diff check), indicating some design work is needed.

Formula: (1 x 1 x 0.8) / 1.5 = 0.53