What happened
On issue #88 (redhat-community-ai-tools/ci-failure-tracker#88), the user reported AI misclassification of a test failure where fix commits existed on master but not on the release-4.18 branch. The triage agent identified this branch-specific detection as the core problem. The code agent's reasoning trace (run 29503182816) at 13:49 UTC explicitly noted: 'The critical detail is that the classifier needs to check for fixes on the specific release branch the failing job targets, not just on main' and 'Simply counting fix commits across all branches is not enough.' However, at 13:51 UTC the code agent chose: 'I will go with the immediate override approach... though I recognize this is a first iteration and the reporter's point about fix commits on main not necessarily being on the relevant branch is worth revisiting later.' The resulting PR #100 searched all commits without branch filtering. The human reviewer closed the PR stating: 'The core issue remains that this PR does not check which branch fix commits are on, which is the entire point of issue #88.'
What could go better
The code agent had the correct understanding of the core requirement (branch-specific detection) from both the triage output and the issue comments, but chose to defer it as a 'first iteration.' This led to a PR that was rejected for not solving the stated problem, wasting the code agent run, the review cycle, the fix agent run, and human reviewer time. The pattern of implementing a partial solution that defers the hard part is particularly wasteful because the review will predictably flag the gap, and the fix agent cannot easily add the deferred feature. Confidence is medium-high: the reasoning trace clearly shows the agent acknowledged and then deferred the requirement. The uncertainty is whether stronger guidance would change the agent's behavior or whether it would still defer complex requirements that involve external API design decisions.
Proposed change
In the code agent definition (agents/code.md) or the code-implementation skill, add guidance distinguishing core requirements from enhancements. When the issue or triage output identifies a specific gap as the primary problem being reported (not a nice-to-have or enhancement), the code agent must implement it rather than deferring it to a text suggestion. Suggested wording: 'If the issue describes a specific failure mode and the triage output identifies a root cause, your implementation must address that root cause. Do not implement a partial solution that defers the key requirement to a suggested_action or TODO comment. If implementing the core requirement is genuinely infeasible within one PR (e.g., requires new infrastructure or external service setup), explain why in the PR description, scope the PR as a prerequisite rather than a solution, and do not claim to close the issue.' This is distinct from issue #2131 (code agent should receive triage context) because the context was available and acknowledged here; the problem is that the agent chose to defer rather than implement.
Validation criteria
On the next 5 code agent runs where the triage output identifies a specific core requirement: (1) the code agent's implementation should address the core requirement rather than deferring it, (2) the PR should not be closed by human reviewers for missing the core requirement, and (3) if the core requirement is genuinely infeasible, the PR description should explicitly explain why and not claim to close the issue.
Generated by retro agent from redhat-community-ai-tools/ci-failure-tracker#100
What happened
On issue #88 (redhat-community-ai-tools/ci-failure-tracker#88), the user reported AI misclassification of a test failure where fix commits existed on master but not on the release-4.18 branch. The triage agent identified this branch-specific detection as the core problem. The code agent's reasoning trace (run 29503182816) at 13:49 UTC explicitly noted: 'The critical detail is that the classifier needs to check for fixes on the specific release branch the failing job targets, not just on main' and 'Simply counting fix commits across all branches is not enough.' However, at 13:51 UTC the code agent chose: 'I will go with the immediate override approach... though I recognize this is a first iteration and the reporter's point about fix commits on main not necessarily being on the relevant branch is worth revisiting later.' The resulting PR #100 searched all commits without branch filtering. The human reviewer closed the PR stating: 'The core issue remains that this PR does not check which branch fix commits are on, which is the entire point of issue #88.'
What could go better
The code agent had the correct understanding of the core requirement (branch-specific detection) from both the triage output and the issue comments, but chose to defer it as a 'first iteration.' This led to a PR that was rejected for not solving the stated problem, wasting the code agent run, the review cycle, the fix agent run, and human reviewer time. The pattern of implementing a partial solution that defers the hard part is particularly wasteful because the review will predictably flag the gap, and the fix agent cannot easily add the deferred feature. Confidence is medium-high: the reasoning trace clearly shows the agent acknowledged and then deferred the requirement. The uncertainty is whether stronger guidance would change the agent's behavior or whether it would still defer complex requirements that involve external API design decisions.
Proposed change
In the code agent definition (agents/code.md) or the code-implementation skill, add guidance distinguishing core requirements from enhancements. When the issue or triage output identifies a specific gap as the primary problem being reported (not a nice-to-have or enhancement), the code agent must implement it rather than deferring it to a text suggestion. Suggested wording: 'If the issue describes a specific failure mode and the triage output identifies a root cause, your implementation must address that root cause. Do not implement a partial solution that defers the key requirement to a suggested_action or TODO comment. If implementing the core requirement is genuinely infeasible within one PR (e.g., requires new infrastructure or external service setup), explain why in the PR description, scope the PR as a prerequisite rather than a solution, and do not claim to close the issue.' This is distinct from issue #2131 (code agent should receive triage context) because the context was available and acknowledged here; the problem is that the agent chose to defer rather than implement.
Validation criteria
On the next 5 code agent runs where the triage output identifies a specific core requirement: (1) the code agent's implementation should address the core requirement rather than deferring it, (2) the PR should not be closed by human reviewers for missing the core requirement, and (3) if the core requirement is genuinely infeasible, the PR description should explicitly explain why and not claim to close the issue.
Generated by retro agent from redhat-community-ai-tools/ci-failure-tracker#100