Fix agent should self-check new code for same-defect-class regressions

[fullsend-ai-retro]

What happened

On PR #708, the fix agent addressed a review finding about static metadata fields by adding dynamic truncation detection in fix iteration 1 (commit 9007375, 2026-08-09). The new code used gh pr list --limit 100 to count open PRs for the truncation calculation. However, --limit 100 silently caps results at 100 — the exact same defect class as the original issue #705, which was about gh issue list --limit 1000 truncating backlog results. The review agent caught this in review iteration 2, identifying that repos with >100 open PRs would get false-positive backlog_truncated=true. This required fix iteration 2 to restructure the approach, and fix iteration 3 to add tolerance for API timing races — two additional review-fix cycles that could have been avoided if the fix agent had verified its new code against the defect class it was addressing.

What could go better

The fix agent introduced a bug of the same class as the bug being fixed. The original issue was about using a capped API call (--limit 1000) as if it returned complete data. The fix agent's new code used another capped API call (--limit 100) as if it returned complete data. This pattern — where the fix introduces a new instance of the same defect class — is distinct from existing proposals: #665 covers verifying that fixes resolve findings (backward-looking), #549 covers adversarial verification for security findings specifically, and #129 covers checking for parallel instances of existing bugs in the codebase. None address the fix agent checking its own new code for the same defect class as the finding being addressed (forward-looking). Confidence is high that this is a genuine gap — the defect-class recurrence is clearly documented in the review comments, and the pattern cost 2 additional review-fix cycles (~4 agent runs). The risk is that defect-class identification is subjective and hard to operationalize generically, but the fix-review skill's step 4 (planning) already includes validity checking, making this a natural extension.

Proposed change

Add defect-class awareness to the fix-review skill (skills/fix-review/SKILL.md) in two locations:

1. Step 4 (Plan fixes): After the existing validity checking, add guidance: "Before implementing, identify the defect class of the original issue and each review finding being addressed (e.g., truncated/capped API results, unescaped user input, race conditions, hardcoded limits). When planning new code to address a finding, verify that your approach does not rely on the same flawed assumption. For example, if the finding is about an API call that silently truncates results, verify that any new API calls you introduce return complete results."

2. Step 7d (Self-review): After the existing checks for untraced changes, debug prints, and secrets, add: "Re-read the defect class of the original issue and the finding you addressed. Scan your diff for any new code that could exhibit the same class of defect — particularly new API calls, limit parameters, or assumptions about completeness that mirror the original bug."

Validation criteria

On the next 5 fix agent runs where the original issue or review finding involves a capped/truncated/limited data source, the fix agent should not introduce new code that uses capped API calls as complete counts. More broadly: the fix agent's self-review step should produce a log line or structured output field indicating it checked for same-class defect recurrence, and the rework rate for same-class bugs should decrease (baseline from this PR: 1 same-class bug in 3 iterations). A functional eval case could be constructed from this PR's scenario: given a finding about --limit 100 truncation, verify the fix agent does not introduce another --limit N call for counting.

---

Generated by retro agent from #708

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 7:20 PM UTC · Completed 7:23 PM UTC

Commit: 1cf646e · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: Medium — causes wasted review-fix cycles but is caught by existing review agent.

Related issues: #665 (verify fixes resolve findings — backward-looking), #549 (adversarial verification for security findings), #129 (check for parallel instances of pattern-based bugs in existing code). This issue is distinct: it targets the fix agent checking its own new code for same-class defects (forward-looking).

Note: PR #751 currently modifies the same skills/fix-review/SKILL.md file (language tightening, trigger phrases). The changes target different sections and should not conflict, but the implementer should rebase if #751 merges first.

Proposed Test Case

Scenario: Fix agent does not introduce same-class defect
  Given a review finding about "gh issue list --limit 1000" silently
        truncating results and being used as a complete count
  When the fix agent plans and implements a fix
  Then the fix agent's self-review output includes a same-class
       defect check
  And the fix diff does not contain new "--limit N" calls used
      for counting without pagination or explicit truncation handling

---

Labels: This is a feature request to enhance the fix-review skill with defect-class awareness, filed by the retro agent against the fix agent's behavior.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[fullsend-ai-prioritize]

RICE Priority Score: 1.2

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	The fix agent is a core platform component used across all repos and customers. Improving its self-review quality benefits all users who trigger fix agent runs. However, the specific failure mode (same-class defect recurrence) is relatively infrequent — it was observed once in this PR's 3 iterations. Scoring 1.5 because multiple strategic customers benefit from fix agent quality improvements, even though the specific trigger condition is uncommon.
Impact	1	When same-class defects slip through, they cause ~2 additional review-fix cycles (~4 agent runs), wasting compute and time. However, the existing review agent catches these before they merge, so there's no production impact. This is a noticeable workflow efficiency improvement (fewer wasted cycles, faster PR resolution) but not a blocking issue. Scoring 1 for medium impact.
Confidence	0.8	High confidence at 0.8. The issue is exceptionally well-documented with a concrete real-world example from PR #708, specific commit references, clear proposed changes to two identified sections of SKILL.md, and well-defined validation criteria. The only uncertainty is whether 'defect-class awareness' can be reliably operationalized as prompt guidance — the issue itself acknowledges defect-class identification is subjective.
Effort	1	The change is scoped to adding guidance text at two specific locations in skills/fix-review/SKILL.md (Step 4 and Step 7d). No architectural changes required. However, it also needs eval case construction for validation, and the implementer should be aware of potential rebase with PR #751. Scoring 1 for medium effort — well-scoped but requires understanding the fix-review skill's flow and creating meaningful validation.

Formula: (1.5 x 1 x 0.8) / 1 = 1.2