Triage agent should check sibling PR discussions for follow-up plans before promoting retro-filed issues

[fullsend-ai-retro]

What happened

The retro agent filed issue #116 (redhat-community-ai-tools/ci-failure-tracker#116) at 16:27 UTC on 2026-07-16, flagging that PR #114 removed _extract_operator_version but left 14 test call sites. The triage agent ran at 16:28 and labeled the issue ready-to-code at 16:31. The code agent then spent ~13 minutes producing PR #117 (146 additions, 87 deletions). Three days later, the maintainer (rrasouli) closed issue #116 saying: 'This is not a bug, it's expected cleanup after a refactor. The replacement tests from PR #115 will be adapted and merged separately.'

Critically, PR #115 had been closed at 16:23 — 5 minutes before triage started — with rrasouli's comment: 'The test file from this PR (test_csv_operator_version.py) will be adapted for the merged implementation in a follow-up.' This signal was available but not consulted by the triage agent.

What could go better

When triaging a retro-filed issue about test breakage from a recent merge, the triage agent should cross-reference related PRs for evidence of planned follow-up work. In this case, PR #115 (which touched the same test area) was closed with an explicit follow-up plan 5 minutes before triage ran. Had the triage checked comments on sibling PRs (PRs linked to the same issue #111 that PR #114 fixed), it could have flagged the planned follow-up and waited for maintainer confirmation instead of auto-promoting to ready-to-code. This is related to agents#212 (triage should not label ready-to-code when analysis identifies human-declined findings) but distinct: here the signal is in a sibling PR's closing comment, not in the issue itself. Confidence: medium — this context would have helped here, but it's not always available. The pattern is most relevant for retro issues about breakage from very recent merges where competing PRs exist.

Proposed change

Extend the triage agent's issue-analysis skill to check for recently-closed sibling PRs when the issue references a specific PR as the source of breakage. Specifically: (1) Extract the PR number from the retro issue body (e.g., 'PR #114 removed...'). (2) Find other PRs linked to the same root issue (issue #111 in this case). (3) Read closing comments on those sibling PRs for signals like 'will be adapted', 'follow-up', 'separately'. (4) If follow-up work is planned, use a label like needs-maintainer-input instead of ready-to-code. This belongs in the triage agent's issue-labels skill or as a new pre-check in the triage flow.

Validation criteria

On a retro-filed issue about test breakage where a sibling PR's closing comment mentions planned follow-up work, the triage agent does NOT apply ready-to-code and instead flags the issue for maintainer confirmation. The code agent does not start work until a human confirms.

---

Generated by retro agent from redhat-community-ai-tools/ci-failure-tracker#117

[fullsend-ai-prioritize]

RICE Priority Score: 0.47

Score breakdown

Dimension	Score	Reasoning
Reach	1	This affects all users of the retro→triage→code agent pipeline, but only in the specific scenario where a retro-filed issue references breakage from a recent merge and sibling PRs exist with follow-up plans. The pattern is moderately common across repos using the agent system but not every-issue frequent.
Impact	1	When triggered, the failure mode wastes code agent compute (~13 min in the cited incident), creates noise for maintainers who must close unnecessary PRs/issues, and erodes trust in the agent system. However, maintainers can easily close the spurious work — no data loss or blocking occurs.
Confidence	0.7	The issue is very well-documented with a specific incident, clear root cause analysis, a concrete proposed solution, and validation criteria. The issue itself notes 'Confidence: medium' on the pattern's generalizability, and the frequency of this exact scenario across all managed repos is uncertain, which slightly lowers confidence.
Effort	1.5	Requires multiple components: extracting referenced PR numbers from retro issue bodies, discovering sibling PRs via linked root issues using the GitHub API, analyzing closing comments for follow-up signals (e.g., 'will be adapted', 'follow-up'), and modifying the triage agent's labeling logic. Some design work is needed for reliable signal detection, and testing must cover the new pre-check path.

Formula: (1 x 1 x 0.7) / 1.5 = 0.47