fix(review): strengthen line number verification in review prompts

[rh-hemartin]

Summary

• Replace the advisory "re-read the file" paragraph in the meta-prompt with a mandatory 4-step verification procedure that names the root cause (diff-relative offset vs file-absolute line number confusion)
• Add line number verification as step 2 in the challenger's procedure, so misplaced lines get caught and corrected before reaching the harness
• Add verification reminders to the correctness and security sub-agents (the two opus agents that produce the most line-specific findings)

Motivation

Ref: fullsend-ai/fullsend#2687 — retro agent filed after run 28229558711 placed 3/4 inline comments at wrong line numbers on konflux-ci/architecture#367. That issue was closed after 3 days without its "next 10 runs" validation window elapsing.

The problem persists in functional tests. Recent Functional Tests (review) runs in this repo all fail with github api: 422 Unprocessable Entity when submitting the review — consistent with line numbers outside valid diff hunk ranges:

Run	Date	Result
29502300145	2026-07-16 13:28	422
29490975129	2026-07-16 10:29	422
29489838926	2026-07-16 10:09	422
29488554026	2026-07-16 09:49	422
29486683277	2026-07-16 09:18	422
29480790119	2026-07-16 07:42	422
29480112981	2026-07-16 07:30	422
29443765398	2026-07-15 19:15	422
29439726895	2026-07-15 18:14	422
29439240811	2026-07-15 18:07	422

See also fullsend-ai/fullsend#2688 (harness-side fix for content-anchored comments).

Test plan

• Verify changed prompts are internally consistent across meta-prompt, code-review skill, and sub-agent definitions
• Run a review on a PR with known inline findings and check line placement accuracy against baseline
• Monitor next 10 review runs for misplaced inline comment rate

🤖 Generated with Claude Code

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 10:30 AM UTC · Completed 10:46 AM UTC
Commit: cebfed8 · View workflow run →

[qodo-code-review]

PR Summary by Qodo

Fix review prompts: enforce verified file-absolute line numbers

🐞 Bug fix ⚙️ Configuration changes 📝 Documentation 🕐 10-20 Minutes

AI Description

• Replace advisory guidance with a mandatory 4-step line-number verification procedure.
• Add challenger backstop to correct or remove incorrect line fields before output.
• Reinforce line verification rules in correctness and security sub-agent instructions.

Diagram

graph TD
  ReviewEngine(["PR Review runtime"]) --> MetaPrompt["meta-prompt.md"] --> Findings["Findings (JSON)"]
  ReviewEngine --> SkillDoc["SKILL.md"]
  ReviewEngine --> Challenger["challenger.md"] --> Findings
  ReviewEngine --> Correctness["correctness.md"] --> Findings
  ReviewEngine --> Security["security.md"] --> Findings

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Harness-level line validation (post-processing)

• ➕ Catches incorrect line fields regardless of agent compliance
• ➕ Can auto-correct or strip invalid lines deterministically
• ➕ Reduces reliance on prompt-following for correctness
• ➖ Requires implementing file lookup/search logic in the runtime
• ➖ Harder to validate semantic intent (which of multiple matches is correct)

2. Require quoted source snippet with each `line`

• ➕ Makes mismatches obvious during review and in automated checks
• ➕ Enables lightweight validation: snippet-at-line must match
• ➖ Increases output verbosity and token usage
• ➖ Snippet selection can still be wrong or overly broad

3. Drop line numbers unless high confidence

• ➕ Eliminates mis-anchored inline comments entirely
• ➕ Encourages findings that reference stable symbols/blocks instead
• ➖ Reduces usefulness of inline commenting and triage speed
• ➖ Shifts effort to humans to locate issues

Recommendation: The PR’s approach (mandatory verification steps + challenger backstop + targeted sub-agent reminders) is the fastest, lowest-risk way to reduce mis-anchored inline comments. If misplacements persist, add a harness-level validator as a deterministic safety net that strips/repairs invalid line fields.

Files changed (5) +54 / -18

Documentation (1) +9 / -6

SKILL.mdClarify line numbers must be file-derived, not diff-counted +9/-6

Clarify line numbers must be file-derived, not diff-counted

• Replaces a soft “re-read the file” advisory with an explicit 4-step verification checklist. Explicitly calls out diff-relative offsets vs file-absolute line numbers and instructs omitting 'line' when uncertain.

skills/code-review/SKILL.md

Other (4) +45 / -12

meta-prompt.mdMake line-number verification mandatory in the meta-prompt +19/-9

Make line-number verification mandatory in the meta-prompt

• Replaces the prior advisory paragraph with a mandatory procedure for emitting any finding that includes a 'line' field. Documents the root cause (diff-relative counting vs file-absolute line numbers) and provides a concrete 4-step verification workflow.

skills/pr-review/meta-prompt.md

challenger.mdAdd line-number accuracy verification as challenger step 2 +10/-3

Add line-number accuracy verification as challenger step 2

• Inserts a dedicated step for validating that the cited line contains the described code. Requires correcting the line or removing the 'line' field when it cannot be verified, and renumbers subsequent challenger steps.

skills/pr-review/sub-agents/challenger.md

correctness.mdAdd correctness sub-agent reminder to verify cited lines +8/-0

Add correctness sub-agent reminder to verify cited lines

• Appends a new “Line number verification” section instructing the agent to Read the file at the cited line and avoid deriving line numbers from diff line counting. Advises omitting 'line' when exact verification is not possible.

skills/pr-review/sub-agents/correctness.md

security.mdAdd security sub-agent reminder to verify cited lines +8/-0

Add security sub-agent reminder to verify cited lines

• Appends a new “Line number verification” section mirroring the correctness agent guidance: verify by reading the file, avoid diff-counted offsets, and omit 'line' if unverified.

skills/pr-review/sub-agents/security.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (1)

Context used

✅ Compliance rules (platform): 55 rules

✅ Skills: 4 invoked

  code-review

  code-implementation

  pr-review

  docs-review

1. Protected skills/ files modified 📜 Skill insight § Compliance

Description

This PR modifies files under the protected skills/ path, which must not be auto-approved and
requires explicit human review per governance controls. Although the PR links an authorizing issue,
the protected-path requirement still applies.

Code

skills/pr-review/meta-prompt.md[R26-44]

+**Line number verification (mandatory before emitting any finding
+with a `line` field):**
+
+Line numbers in diff hunk headers (`@@ -X,N +Y,M @@`) are file-
+absolute, but your position within the diff output is not — counting
+lines from the top of a hunk gives a diff-relative offset, not a file
+line number. Always derive line numbers from the file itself, never
+from counting diff lines.
+
+Before emitting a finding with a `line` value:
+
+1. Read the file at the line you intend to cite.
+2. Confirm the content at that line is the specific code or text your
+   finding describes — not a nearby line in the same function or block.
+3. If the content does not match, grep or search the file for the
+   expected content and use the correct line number.
+4. If you cannot locate the exact line, omit the `line` field. A
+   finding with no line number is always better than one that points
+   to the wrong code.

Relevance

⭐⭐ Medium

Similar protected-path justification finding was only partially accepted (PR#184); no clear history
enforcing “skills/ must never auto-approve”.

PR-#184
PR-#59

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1538392 requires raising a finding whenever protected paths (including skills/)
are modified, and indicates the PR must not be auto-approved. The diff shows modifications inside
skills/pr-review/meta-prompt.md, which is within the protected skills/ path.

skills/pr-review/meta-prompt.md[26-44]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
This PR changes protected governance/infrastructure content under `skills/`, which must not be auto-approved and requires explicit human review.

## Issue Context
The PR appears authorized (it links `fullsend-ai/fullsend#2688`), so the key requirement is ensuring the PR is routed for mandatory human approval (e.g., CODEOWNERS / reviewer enforcement) rather than changing logic.

## Fix Focus Areas
- skills/pr-review/meta-prompt.md[26-44]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[fullsend-ai-review]

Review

Findings

Medium

• [protected-path] skills/code-review/SKILL.md, skills/pr-review/meta-prompt.md, skills/pr-review/sub-agents/challenger.md — All 3 modified skill files are under the skills/ protected path. The PR references Review skill should instruct agents to verify line numbers against file content fullsend#2687 and Post-review harness should anchor inline comments with content snippets fullsend#2688 with clear rationale (strengthening line number verification to fix persistent 422 errors from misplaced inline comments). The changes are well-scoped, internally consistent prompt reinforcements: code-review and meta-prompt define a mandatory 4-step verification procedure that names the root cause (diff-relative offset vs file-absolute line number confusion), while the challenger adds line number verification as a new procedural step with a corresponding line-corrected adjudication action. Human approval is always required for protected-path changes, regardless of context.

---

Labels: PR modifies review agent prompt files under skills/pr-review/ and skills/code-review/, and adds an eval test case for line number accuracy

Previous run

Review

Findings

Medium

• [protected-path] skills/code-review/SKILL.md, skills/docs-review/SKILL.md, skills/pr-review/meta-prompt.md, skills/pr-review/sub-agents/challenger.md, skills/pr-review/sub-agents/correctness.md, skills/pr-review/sub-agents/security.md — All 6 changed files are under the skills/ protected path. The PR references Review skill should instruct agents to verify line numbers against file content fullsend#2687 and Post-review harness should anchor inline comments with content snippets fullsend#2688 with clear rationale (improving line number accuracy to fix persistent 422 errors from misplaced inline comments). The changes are well-scoped prompt reinforcements consistent across the meta-prompt, code-review skill, and all sub-agent definitions. Human approval is always required for protected-path changes, regardless of context.

Low

• [heading-level-inconsistency] skills/pr-review/sub-agents/security.md:147 — The new ## Line number verification section uses an H2 heading, while the structurally analogous section in correctness.md uses ### Line number verification (H3). Both files are internally consistent with their own heading hierarchies — security.md uses H2 for all top-level sections, correctness.md uses H3 for procedural subsections — but the cross-file inconsistency creates a different structural signal for identical content. Remediation: change ## Line number verification to ### Line number verification in security.md, or elevate correctness.md to H2, for cross-file consistency.

---

Labels: PR modifies review agent prompt files under skills/pr-review/ and skills/code-review/

Previous run (2)

Review

Findings

Medium

• [protected-path] skills/code-review/SKILL.md, skills/docs-review/SKILL.md, skills/pr-review/meta-prompt.md, skills/pr-review/sub-agents/challenger.md, skills/pr-review/sub-agents/correctness.md, skills/pr-review/sub-agents/security.md — All 6 changed files are under the skills/ protected path. The PR references Post-review harness should anchor inline comments with content snippets fullsend#2688 and provides clear rationale for the changes (improving line number accuracy in review findings). Human approval is always required for protected-path changes, regardless of context.

Low

• [heading-level-inconsistency] skills/pr-review/sub-agents/security.md:142 — The new ## Line number verification section uses an H2 heading, while the structurally similar ### Cross-file verification section at line 86 in the same file uses H3. The correctness.md file uses H3 for both analogous sections, making this an inconsistency across the two sub-agent files. Remediation: change ## Line number verification to ### Line number verification.

Previous run

Review

Findings

Medium

• [protected-path] skills/code-review/SKILL.md, skills/pr-review/meta-prompt.md, skills/pr-review/sub-agents/challenger.md, skills/pr-review/sub-agents/correctness.md, skills/pr-review/sub-agents/security.md — All 5 changed files are under the skills/ protected path. The PR references Post-review harness should anchor inline comments with content snippets fullsend#2688 and provides clear rationale for the changes (improving line number accuracy in review findings). Human approval is always required for protected-path changes, regardless of context.

Low

• [stale-doc] skills/docs-review/SKILL.md:183 — The docs-review skill describes the Line: field as "line number of the stale reference, if identifiable" without the line number verification procedure this PR adds elsewhere. When docs-review runs as a pr-review sub-agent, it already receives verification guidance via the meta-prompt (Part 2 of dispatch). The gap only affects standalone invocations. The PR deliberately scoped reinforcement to the two opus agents (correctness, security) that produce the most line-specific findings — three other sonnet sub-agents have the same gap.

---

Labels: PR modifies review agent prompt files under skills/pr-review/ and skills/code-review/

Previous run (3)

Review

Findings

Medium

• [protected-path] skills/code-review/SKILL.md, skills/docs-review/SKILL.md, skills/pr-review/meta-prompt.md, skills/pr-review/sub-agents/challenger.md, skills/pr-review/sub-agents/correctness.md, skills/pr-review/sub-agents/security.md — All 6 changed files are under the skills/ protected path. The PR references Post-review harness should anchor inline comments with content snippets fullsend#2688 and provides clear rationale for the changes (improving line number accuracy in review findings). Human approval is always required for protected-path changes, regardless of context.

Low

• [heading-level-inconsistency] skills/pr-review/sub-agents/security.md:142 — The new ## Line number verification section uses an H2 heading, while the structurally similar ### Cross-file verification section at line 86 in the same file uses H3. The correctness.md file uses H3 for both analogous sections, making this an inconsistency across the two sub-agent files. Remediation: change ## Line number verification to ### Line number verification.

Previous run (4)

Review

Findings

Medium

• [protected-path] skills/code-review/SKILL.md, skills/pr-review/meta-prompt.md, skills/pr-review/sub-agents/challenger.md, skills/pr-review/sub-agents/correctness.md, skills/pr-review/sub-agents/security.md — All 5 changed files are under the skills/ protected path. The PR references Post-review harness should anchor inline comments with content snippets fullsend#2688 and provides clear rationale for the changes (improving line number accuracy in review findings). Human approval is always required for protected-path changes, regardless of context.

Low

• [stale-doc] skills/docs-review/SKILL.md:183 — The docs-review skill describes the Line: field as "line number of the stale reference, if identifiable" without the line number verification procedure this PR adds elsewhere. When docs-review runs as a pr-review sub-agent, it already receives verification guidance via the meta-prompt (Part 2 of dispatch). The gap only affects standalone invocations. The PR deliberately scoped reinforcement to the two opus agents (correctness, security) that produce the most line-specific findings — three other sonnet sub-agents have the same gap.

---

Labels: PR modifies review agent prompt files under skills/pr-review/ and skills/code-review/

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 1:28 PM UTC · Completed 1:45 PM UTC
Commit: eec6949 · View workflow run →

[waynesun09]

Review Squad — 3 agents (Claude × 2, Grok) · 6 verified MEDIUM+ findings

2 HIGH · 4 MEDIUM. No duplicate findings from the existing qodo-bot comment (unrelated protected-path note). Note: this PR was force-pushed mid-review (cebfed8f → eec6949a, adding skills/docs-review/SKILL.md back into the diff) — all findings below are verified against current head eec6949a.

The diagnostic core of this fix (naming the diff-relative-vs-file-absolute confusion, mandating Read-tool-based verification) correctly targets the root cause in fullsend-ai/fullsend#2687. The problems below are in the implementation, not the diagnosis. Three MEDIUM findings that don't anchor to a single line in this diff are included below rather than as inline comments:

MEDIUM (premature-decision) — Round-2 fix asserts recurrence with no evidence trail

This repo has an established pattern for documenting a fix's failure to hold — "Evidence for #NNNN" issues (e.g. #4275, #4506, #4498). No such issue exists for #2687, which was closed 3 days after opening — not enough time for its own "next 10 runs" validation window to elapse. This PR's commit message asserts wrong line numbers are recurring as present-tense fact with no cited run or PR.
Suggestion: Link the specific review run(s) that motivated this PR, or open an "Evidence for #2687" issue per repo convention.

MEDIUM (premature-decision) — No measurement mechanism exists to check whether this fix works

The only eval harness for this skill (eval/review/) has one case with zero expected findings — it cannot exercise line-citation accuracy at all. The PR's "monitor next 10 runs" test-plan item and #2687's "<10% mismatch" validation criteria have no instrumentation behind them. Separately, #2687 itself diagnoses the root cause as a harness gap ("cannot validate semantic correctness"), and #2688 (deliberately deferred by this PR) exists because prompt-only fixes were anticipated to be insufficient — round 1 already tried stronger wording once.
Suggestion: Add an eval case with a known-correct line-citation scenario and a judge that checks it; consider pairing with the harness-side backstop from #2688 rather than betting twice on wording alone.

MEDIUM — No auditable artifact proves line verification actually occurred (self-report only)

All the new instructions rely entirely on the agent's self-report — no line_content snippet, no verified flag, no record of the original (possibly wrong) line before correction. Contrast with the challenger's existing removed_findings array, which requires original_category/original_description/removal_reason and is fully auditable. The challenger's challenger_action enum (kept|downgraded|merged|removed) has no line-corrected value, so a silent line fix produces no visible signal it happened.
Suggestion: Add a line-corrected challenger action with an original_line field; consider requiring a line_content snippet in the finding schema now, ahead of #2688's harness-side fuzzy-match.

Assisted-by: Claude (review), Grok (review)

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 6:42 AM UTC · Completed 6:48 AM UTC
Commit: 9a9633a · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 6:51 AM UTC · Completed 6:59 AM UTC
Commit: e66621f · View workflow run →

[waynesun09]

Review Squad — 3 agents (Claude × 2, Grok-attempted) · re-review at head e66621f

1 CRITICAL · 3 HIGH · 1 MEDIUM, all new since the prior round. Note on tooling: the Grok subprocess (cursor-agent --model grok-4.5-high) never returned output in either dispatch (timed out after 10+ min); the third agent's findings below are its own direct GitHub-API verification, not an independent Grok read — flagged for transparency, not counted as model diversity.

Good news first: all 3 issues from the prior round are genuinely fixed — the redundant duplicate sections in security.md/correctness.md were dropped entirely (not just reworded), the old-file/new-file hunk-header ambiguity is now resolved (+Y called out explicitly), and the evidence-trail gap is closed (PR body now cites 10 specific failed CI runs). The LOW heading-inconsistency finding is moot — the section it was about no longer exists.

The findings below are new, found during this re-review pass, and verified directly against the current head and live CI/issue data before posting.

CRITICAL — This PR's own CI, run at its exact head commit, still reproduces the 422 error it claims to fix — despite provably correct line citations

Evidence: CI run 29561223892 ran at head_sha = e66621fe4d3baa3df29f4fdfab92fc106b85e5cd (confirmed via gh api .../runs/29561223892 --jq .head_sha — exact match to this PR's current head) and finished conclusion: failure, with the functional-tests (review) job failing. Its case 002-line-accuracy — the case this PR added specifically to validate the fix — failed with github api: 422 Unprocessable Entity submitting the review. The agent's agent-result.json from that run was independently re-checked against input.yaml's fixture content: all six cited line numbers are file-absolute-correct (line 65 = the SQL-injection f-string, line 21 = average_value's divide, line 31 = the unclosed open(), etc.) and review_quality scored 5.0/5. Since the line numbers are correct, this 422 cannot be the diff-relative/file-absolute confusion this PR's prompt changes target — it's consistent with the still-open companion issue fullsend-ai/fullsend#2688, which notes the harness only validates line numbers against diff hunk ranges (a correct-but-out-of-hunk line still 422s) and frames the harness-side fix as a separate, not-yet-built "defense in depth" layer.
Suggestion: Before merging, reproduce and root-cause this specific 422 independently of line-number correctness (inspect the exact request body fullsend post-review sends — commit_id, side, position vs line). Don't present this PR as resolving the linked CI failures until an actual run on this branch posts successfully end-to-end.

[verified] · flagged by 1/3 agents (Claude-researcher), independently reproduced against live CI run 29561223892

MEDIUM (premature-decision) — PR silently doesn't fulfill the explicit reintroduction request from issue #202

Finding: #202 ("Remove review eval case 001-clean-approve until post-review 422 errors are resolved", closed 2026-07-16) states: "Once the line number accuracy issue is fixed (branch fix/review-line-number-accuracy, PR forthcoming), reintroduce the test case." This PR is that branch/PR, yet neither the diff nor the description mentions 001-clean-approve or #202. (The branch is currently 54 commits behind main, so the deleted case still shows in stale CI runs against this branch — that's not the same as this PR reintroducing it.)
Suggestion: Explicitly decide and state whether reintroducing 001-clean-approve is in scope here or a tracked follow-up, rather than leaving #202's stated expectation silently unaddressed.

[verified] · flagged by 1/3 agents (Claude-researcher), confirmed against issue #202 and branch compare (ahead_by:1, behind_by:54)

Agent Agreement

• line_accuracy judge referenced in the new eval case does not exist anywhere in the repo — flagged by 3/3 agents (full consensus, posted inline below)
• Prior-round fixes (redundant sections dropped, hunk-header disambiguation, evidence table) are genuine and complete — confirmed by 2/3 agents (Claude-researcher, third agent) via direct diff/force-push-timeline inspection

Assisted-by: Claude (review), Claude (review), Grok (attempted, non-functional this round)

[waynesun09]

HIGH — line_accuracy judge referenced here does not exist; this eval case currently gives zero signal on line-citation accuracy

eval/review/eval.yaml registers exactly five judges — review_quality, required_labels, forbidden_labels, max_turns, max_cost — confirmed both by reading the file at this head and by the CI run's own log line: Scoring 2 cases with 5 judges: [...]. No line_accuracy judge exists anywhere in the repo, and a repo-wide search for line_accuracy/expected_findings finds no consumer of the expected_findings block above (lines 13-41) at all. Worse, the one judge that does run, review_quality, is explicitly told by this same paragraph to ignore line accuracy ("should focus on whether the agent identified the right issues, not whether the line citations are exact") — so this eval case, added specifically to validate this PR's own fix, measures nothing about the thing it's named for. This is also an untested design choice shipped as final with no fallback marker (no TODO, no "not yet wired" note) if the judge is never built.

Suggestion: Add a check-type line_accuracy judge to eval.yaml that matches posted findings against expected_findings by file+line (or file+line_content) with a registered threshold — or, if that's out of scope for this PR, delete this claim and the dead expected_findings block, or mark them explicitly as "NOT YET WIRED" so the gap is visible.

[verified] · flagged by 3/3 agents (full consensus — independently confirmed by Claude-coder, Claude-researcher, and the third agent's own direct verification)

[waynesun09]

HIGH — original_line isn't stripped by the orchestrator and violates the strict output schema; it's also silently dropped whenever line-correction co-occurs with another challenger action

Two compounding issues on this field:

1. Not stripped, fails schema. skills/pr-review/SKILL.md step 6d.4 (unchanged by this PR) instructs: "Strip the challenger-specific fields (challenger_action, challenger_reason) before merging" — it does not list original_line. schemas/review-result.schema.json's finding definition is additionalProperties: false with a fixed field list that does not include original_line. Any finding that goes through a line-corrected challenger action will carry this field into fullsend-check-output validation and fail it — per agents/review.md, "Schema validation failures trigger a harness retry iteration," so this wastes turns/cost on exactly the review path this PR is trying to make more common.
2. Can't represent composite outcomes. challenger_action is a single enum value (kept|downgraded|merged|removed|line-corrected), and original_line is populated "only when challenger_action is line-corrected." If a finding is both line-corrected and downgraded/merged/removed in the same pass, the challenger can only report one action — so the auditable-correction record this PR added (to close a prior review's "no auditable artifact" finding) silently disappears whenever a correction co-occurs with any other adjudication.

Suggestion: Update SKILL.md step 6d.4 to also strip original_line (or whitelist fields to keep, rather than listing fields to drop, so future challenger-specific fields don't leak by default). Separately, decouple original_line from challenger_action — populate it whenever a line was corrected regardless of the primary action, or document explicit precedence for the composite case.

[verified] · flagged by 2/3 agents (Claude-coder, Claude-researcher), confirmed against current SKILL.md and schemas/review-result.schema.json

[waynesun09]

HIGH — Carve-out lumps "small" scope (which permits reading source files) in with "trivial" (which doesn't), reintroducing the exact failure mode this PR fixes for small-scope PRs

This reads: "If your scope constraint prohibits reading source files (e.g. trivial/small), derive line numbers from the diff hunk headers... on a best-effort basis." But per skills/pr-review/SKILL.md's scope-constraint table (unchanged by this PR): trivial is "≤5 tool calls... Do NOT read... surrounding files" (does prohibit source reads), while small is "≤15 tool calls. Read the diff, linked issue, and up to 3 context files directly relevant to the change" — which explicitly permits reading source files, including in nearly all cases the very file a finding targets. As written, a small-scoped sub-agent (PRs under 20 changed lines) now has an explicit, sanctioned excuse to skip the mandatory read-and-verify procedure and fall back to less-accurate hunk-header arithmetic even when it has budget to just read the file — which is the same "mandatory verification vs. tool-call budget" contradiction the prior review round flagged as HIGH, just reintroduced for the small tier specifically by this new carve-out.

Suggestion: Scope the carve-out to trivial only, and for small scope require using one of the 3 permitted context-file reads to verify the line, falling back to hunk-header math only if that budget is exhausted.

[verified] · flagged by 2/3 agents (Claude-coder, Claude-researcher), confirmed against SKILL.md's scope-constraint table

[rh-hemartin]

422 is produced by another system, closing this will open again in the future.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 2:16 PM UTC · Completed 2:29 PM UTC
Commit: e66621f · View workflow run →

[fullsend-ai-retro]

Retro: PR #197 — strengthen line number verification in review prompts

What happened

PR #197 was a human-authored change (rh-hemartin + Claude co-author) that attempted to fix review agent line number accuracy by replacing advisory "re-read the file" guidance with a mandatory 4-step verification procedure across meta-prompt.md, challenger.md, and SKILL.md. It also added a new eval case (002-line-accuracy) with planted bugs and expected findings.

The PR went through 4 automated review runs (29490984206, 29502313148, 29560787322, 29561234734) and 2 human Review Squad rounds over 6 days before being closed without merging on 2026-07-22.

Review quality gap

The automated review agent found only procedural/cosmetic issues across all 4 runs: a protected-path governance flag (MEDIUM) and a heading-level inconsistency (LOW). The human Review Squad found 11 unique substantive findings including:

• CRITICAL: The PR's own CI (run 29561223892) still produced 422 errors at the PR's head commit with provably correct line numbers, proving the root cause is in the harness post-review script's diff-hunk validation — not in the prompts this PR modified.
• HIGH: The new eval case referenced a line_accuracy judge that does not exist in eval/review/eval.yaml (which defines only 5 judges: review_quality, required_labels, forbidden_labels, max_turns, max_cost).
• HIGH: A new original_line field in challenger output was not added to the strip-list or review-result.schema.json (additionalProperties: false), meaning it would fail schema validation at runtime.
• HIGH: The scope-constraint carve-out incorrectly lumped "small" scope with "trivial," giving small-scoped sub-agents an excuse to skip mandatory verification even when they have tool-call budget.

The automated agent never evaluated whether the PR's fundamental thesis was correct, never verified cross-file consistency between the new procedures and existing scope constraints, and never checked whether referenced artifacts (judges, schema fields) existed.

Evidence for existing open issues

• agents#193 (post-review 422 on inline comments outside diff hunks): This PR provides further evidence that the 422 root cause is harness-side, not prompt-side. The PR author's closing comment confirms this: "422 is produced by another system."
• fullsend#2667 / fullsend#369 / fullsend#4164 (review agent should check CI status): The CRITICAL finding — CI fails at the PR's head commit, directly refuting the PR's thesis — was caught only by human reviewers. The automated agent has no mechanism to cross-reference CI results against the PR's stated goals.
• agents#209 / agents#245 (review eval suite has zero coverage): The eval cases directory remains empty after this PR's closure. The proposed 002-line-accuracy case had structural issues that would have prevented it from working even if merged.

What went well

• The requires-manual-review label and CODEOWNERS enforcement correctly gated this PR for human review, preventing a flawed change from merging.
• The human Review Squad process was thorough and well-structured, producing actionable findings with specific file/line references and evidence.
• The PR author responded constructively to review feedback, dropping redundant sub-agent sections and closing the PR when the root cause was identified elsewhere.

Proposals filed

• Add CI validation that eval case annotations reference valid judges (in fullsend-ai/agents)
• Add reference-validity check to correctness sub-agent guidance (in fullsend-ai/agents)