What happened
On PR #5201, the review agent ran 4 completed reviews on a PR that added 55 new lines of user-facing documentation to docs/guides/user/running-agents-locally.md (a container-based CLI usage section). The review agent’s docs-currency sub-agent found zero issues with the documentation. The human reviewer (rh-hemartin) then posted 12 inline comments — all on the documentation file — requesting structural changes:
- The container prerequisites section was placed before the commands section, referencing concepts not yet introduced. The human said: “This section is way too early, it is mentioning a lot of things that get explained after.”
- Multiple paragraphs were unnecessarily verbose. The human said: “Not sure this explanation is needed” and “make it a lot shorter.”
- The macOS limitation note was placed mid-section rather than in the platform-notes/troubleshooting section where it belongs.
- Multiple gcloud credential alternatives were offered when a single path would suffice. The human said: “I won’t even offer this, don’t add more complexity.”
- The documentation presented a macOS container workflow as viable (with caveats) when it was completely non-functional.
The review agent’s docs-currency sub-agent (skills/pr-review/sub-agents/docs-currency.md) and the docs-review skill (skills/docs-review/SKILL.md) focus exclusively on detecting whether code changes made existing documentation stale. They extract identifiers from the diff, grep documentation files, and check for staleness. None of these steps evaluate the quality of new documentation added in the PR.
The human’s review resulted in a CHANGES_REQUESTED verdict, 4 follow-up commits from the author restructuring the documentation, and a final “Much better, thanks” approval — a full rework cycle that could have been caught earlier.
What could go better
The review agent’s documentation dimension has a structural blind spot: it evaluates whether code changes broke existing docs but never evaluates whether newly added documentation is well-structured. On PR #5201, the human reviewer caught 8 issues that all fell outside the docs-currency scope:
- Section ordering / pedagogical flow: content placed before its prerequisites are explained.
- Content density: verbose explanations where concise ones would serve the reader better.
- Audience-appropriate complexity: multiple alternatives offered where a single recommended path reduces cognitive load.
- Procedural validity: documented procedures that have not been tested and are actually broken.
These are information architecture concerns — they require evaluating the document as a reader would experience it, top-to-bottom, assessing whether each section appears at the right point in the reader’s journey. The staleness-detection approach (grep for changed identifiers in doc files) cannot surface these issues by design.
This is a recurring pattern. Existing issues #1480 (shallow docs feedback), #2199 (fact-checking claims), and #4838 (semantic placement for root context files) each address specific facets of documentation review quality but none covers the information-architecture dimension for user-facing guides. Issue #4838 is scoped to root context files (AGENTS.md, CLAUDE.md); #1480 focuses on code example validation and non-empty review bodies; #2199 focuses on verifying claims against the codebase. The gap identified here — evaluating section ordering, content density, and audience appropriateness in new documentation — is distinct.
Confidence: high that this is a real gap. The evidence is direct: 8 human findings, all outside the docs-currency scope, resulting in a full rework cycle. Confidence is medium that this is a high-frequency pattern, since not every PR adds substantial new documentation. The impact is highest on PRs that add 50+ lines of new user-facing docs.
Proposed change
Add a documentation quality evaluation to the review agent’s documentation dimension, alongside the existing staleness detection. Two implementation options:
Option A: Extend the docs-currency sub-agent. In skills/pr-review/sub-agents/docs-currency.md, add a second responsibility after the staleness check: when the PR adds substantial new documentation (e.g., 30+ new lines in .md files under docs/), evaluate the added content for information architecture concerns. Specifically:
- Section ordering: Check whether new sections reference concepts, tools, or procedures that appear later in the document. Flag cases where a section’s prerequisites are explained after the section itself.
- Content density: Flag sections that explain things already implicit in the surrounding context (e.g., explaining what a directory path does when the usage example makes it clear).
- Audience-appropriate complexity: Flag cases where multiple alternative approaches are offered when a single recommended path with a brief mention of alternatives would reduce reader cognitive load.
- Procedural completeness: Flag documented procedures described as untested, non-functional, or may-not-work as medium-severity findings rather than accepting them silently.
Update the “Own” section to include: “When the PR adds substantial new documentation: information architecture (section ordering, content density, audience-appropriate complexity, procedural completeness).”
Option B: Add a new sub-agent. Create a docs-quality.md sub-agent in skills/pr-review/sub-agents/ that runs only when the PR adds 30+ lines of new documentation. This keeps the staleness check (docs-currency) and quality check (docs-quality) separate. The new sub-agent would use Sonnet and be dispatched conditionally.
Option A is simpler but risks making the docs-currency agent’s prompt too long. Option B is cleaner but adds dispatch complexity. Recommend starting with Option A and splitting if the combined prompt exceeds the model’s effective instruction-following capacity.
The docs-review skill (skills/docs-review/SKILL.md) would also need a corresponding update to its process steps (adding a step 4b or similar for quality evaluation on new content), since the docs-currency sub-agent follows that skill inline.
Validation criteria
On the next 5 PRs that add 30+ lines of new user-facing documentation (files under docs/ or similar guide paths), the review agent should:
- Produce at least one information-architecture finding per PR when the human reviewer subsequently identifies section-ordering, verbosity, or complexity issues — i.e., the agent catches these before the human does.
- Not generate false-positive information-architecture findings on PRs where the human reviewer approves the documentation structure without comment.
- Flag any documented procedures described as untested or non-functional as at least medium severity.
Proxy metric: track the ratio of human documentation-structure review comments that were not anticipated by the review agent’s docs dimension. The current baseline on PR #5201 is 8/8 (100% missed). Target: below 50% on the measured PRs.
Generated by retro agent from fullsend-ai/fullsend#5201
What happened
On PR #5201, the review agent ran 4 completed reviews on a PR that added 55 new lines of user-facing documentation to
docs/guides/user/running-agents-locally.md(a container-based CLI usage section). The review agent’s docs-currency sub-agent found zero issues with the documentation. The human reviewer (rh-hemartin) then posted 12 inline comments — all on the documentation file — requesting structural changes:The review agent’s docs-currency sub-agent (
skills/pr-review/sub-agents/docs-currency.md) and the docs-review skill (skills/docs-review/SKILL.md) focus exclusively on detecting whether code changes made existing documentation stale. They extract identifiers from the diff, grep documentation files, and check for staleness. None of these steps evaluate the quality of new documentation added in the PR.The human’s review resulted in a CHANGES_REQUESTED verdict, 4 follow-up commits from the author restructuring the documentation, and a final “Much better, thanks” approval — a full rework cycle that could have been caught earlier.
What could go better
The review agent’s documentation dimension has a structural blind spot: it evaluates whether code changes broke existing docs but never evaluates whether newly added documentation is well-structured. On PR #5201, the human reviewer caught 8 issues that all fell outside the docs-currency scope:
These are information architecture concerns — they require evaluating the document as a reader would experience it, top-to-bottom, assessing whether each section appears at the right point in the reader’s journey. The staleness-detection approach (grep for changed identifiers in doc files) cannot surface these issues by design.
This is a recurring pattern. Existing issues #1480 (shallow docs feedback), #2199 (fact-checking claims), and #4838 (semantic placement for root context files) each address specific facets of documentation review quality but none covers the information-architecture dimension for user-facing guides. Issue #4838 is scoped to root context files (AGENTS.md, CLAUDE.md); #1480 focuses on code example validation and non-empty review bodies; #2199 focuses on verifying claims against the codebase. The gap identified here — evaluating section ordering, content density, and audience appropriateness in new documentation — is distinct.
Confidence: high that this is a real gap. The evidence is direct: 8 human findings, all outside the docs-currency scope, resulting in a full rework cycle. Confidence is medium that this is a high-frequency pattern, since not every PR adds substantial new documentation. The impact is highest on PRs that add 50+ lines of new user-facing docs.
Proposed change
Add a documentation quality evaluation to the review agent’s documentation dimension, alongside the existing staleness detection. Two implementation options:
Option A: Extend the docs-currency sub-agent. In
skills/pr-review/sub-agents/docs-currency.md, add a second responsibility after the staleness check: when the PR adds substantial new documentation (e.g., 30+ new lines in.mdfiles underdocs/), evaluate the added content for information architecture concerns. Specifically:Update the “Own” section to include: “When the PR adds substantial new documentation: information architecture (section ordering, content density, audience-appropriate complexity, procedural completeness).”
Option B: Add a new sub-agent. Create a
docs-quality.mdsub-agent inskills/pr-review/sub-agents/that runs only when the PR adds 30+ lines of new documentation. This keeps the staleness check (docs-currency) and quality check (docs-quality) separate. The new sub-agent would use Sonnet and be dispatched conditionally.Option A is simpler but risks making the docs-currency agent’s prompt too long. Option B is cleaner but adds dispatch complexity. Recommend starting with Option A and splitting if the combined prompt exceeds the model’s effective instruction-following capacity.
The docs-review skill (
skills/docs-review/SKILL.md) would also need a corresponding update to its process steps (adding a step 4b or similar for quality evaluation on new content), since the docs-currency sub-agent follows that skill inline.Validation criteria
On the next 5 PRs that add 30+ lines of new user-facing documentation (files under
docs/or similar guide paths), the review agent should:Proxy metric: track the ratio of human documentation-structure review comments that were not anticipated by the review agent’s docs dimension. The current baseline on PR #5201 is 8/8 (100% missed). Target: below 50% on the measured PRs.
Generated by retro agent from fullsend-ai/fullsend#5201