Review agent should itemize and verify content redistribution in documentation restructuring PRs

[fullsend-ai-retro]

What happened

On PR #5888, which split distributed-tracing.md into three audience-targeted documents, the review agent noticed content loss at LOW severity with the generic observation: "The old document contained MLflow-specific examples... These sections were removed. If intentionally simplified... this is fine — verify no content was accidentally dropped." This put the verification burden back on the author.

The human reviewer (waynesun09) did the actual verification work across 4 review sweeps and identified 5 specific pieces of dropped content:

1. Trace-replay tooling (hack/upload-traces.sh) — undocumented in any of the new files
2. Production-deployment checklist (HA/auth/retention/cost guidance) — dropped entirely
3. Network-reachability prerequisite — missing from "Before you begin" section
4. OTEL_EXPORTER_OTLP_CERTIFICATE environment variable — omitted from the env var catalog
5. Backend-agnostic org guidance — now siloed under an MLflow-titled page

The review agent ran via dispatch runs in fullsend-ai/.fullsend, resolved from fullsend-ai/agents@747566d.

What could go better

The review agent's current approach to content loss in restructuring PRs is to make a vague observation and ask the author to verify — this provides almost no value over not flagging it at all. The human reviewer demonstrated what a useful review looks like: systematically diffing old vs. new content, searching for each topic across all output files, and reporting exactly what was found and what was missing.

I am confident this is a real gap because: (a) the agent's vague finding provided no actionable information, (b) the human found 5 specific items the agent missed, and (c) 4 of those 5 were accepted by the author and fixed. The agent has access to git show and grep — the tools needed for this verification — but its docs-review skill does not instruct it to use them for content redistribution checks.

Uncertainty: Defining "restructuring PR" precisely (vs. simple deletion) may be tricky. A heuristic like "PR deletes >50 lines from a doc file AND adds new doc files" could work but may need tuning.

Proposed change

Enhance the review agent's documentation review skill (skills/docs-review/SKILL.md in fullsend-ai/agents) with a content-redistribution verification pass. Trigger condition: when a PR removes significant content from a documentation file (e.g., >50 lines deleted) and adds or rewrites other documentation files.

The verification pass should:

1. Retrieve the pre-change version of modified docs using git show HEAD~1:<file>
2. Extract distinct sections/topics from the old version (by heading or thematic block).
3. Search for each topic across all files in the post-change PR using grep or content matching.
4. Report a content-redistribution table: each old section, where it landed (file + section), or "NOT FOUND" if missing.
5. Rate missing content as MEDIUM severity with specific remediation rather than making a vague "content may have been lost" observation at LOW severity.

This should be a dedicated sub-step in the docs-review skill, not a general instruction, so it activates reliably on restructuring PRs.

Validation criteria

On the next 3 documentation restructuring PRs in fullsend-ai/fullsend (PRs that split, merge, or substantially reorganize doc files), the review agent should produce a content-redistribution report that itemizes each section from the original document and identifies where it landed or flags it as missing. Measure whether human reviewers still independently discover dropped content that the agent did not flag. Success: the agent catches at least 80% of content omissions that a human reviewer would identify.

---

Generated by retro agent from fullsend-ai/fullsend#5888

[fullsend-ai-prioritize]

RICE Priority Score: 0.8

Score breakdown

Dimension	Score	Reasoning
Reach	1	This affects all users who submit or review documentation restructuring PRs (splitting, merging, or reorganizing docs) across fullsend-ai repos. While the review agent runs on all PRs, doc restructuring is a niche but recurring scenario — not every PR triggers this, but when it does, both authors and reviewers are affected. Scores as a moderate number of users across orgs.
Impact	1.5	The issue demonstrates that the current behavior provides almost no value — the agent makes a vague LOW-severity observation that puts the verification burden back on the author, while a human reviewer independently found 5 specific content drops (4 of which were accepted and fixed). A proper content-redistribution table would be a significant improvement, turning an unhelpful flag into actionable, systematic verification that prevents content loss in shipped docs.
Confidence	0.8	High confidence — the issue is exceptionally well-described with a concrete real-world example (PR #5888), 5 specific evidence points, a clear proposed implementation using tools the agent already has (git show, grep), defined trigger heuristics, and measurable validation criteria (80% catch rate over next 3 restructuring PRs).
Effort	1.5	Moderate complexity — requires designing a restructuring-PR detection heuristic, implementing section extraction from pre-change docs, cross-file search for redistributed content, building a report table format, and integrating this as a dedicated sub-step in the docs-review skill. The tools needed (git show, grep) are already available, but the section extraction and matching logic requires design work and tuning.

Formula: (1 x 1.5 x 0.8) / 1.5 = 0.8