Review agent should flag test file references to removed methods as high-severity

[fullsend-ai-retro]

What happened

PR #114 removed three methods from gcsweb.py (_extract_operator_version, _fetch_build_log_text, _DEFAULT_OPERATOR_VERSION_PATTERNS) but did not update the test file test_build_health.py, which has 14 call sites referencing _extract_operator_version. The review agent was dispatched (run 29514681265) but could not review because the PR was already merged (9-second merge window). If the review had run, the correctness sub-agent should have detected that methods deleted in the PR are still referenced in test files — this is a mechanical check (deleted symbol referenced elsewhere) that is high-confidence and high-impact.

What could go better

The review agent's correctness sub-agent should include a check for deleted functions/methods/classes that are still referenced in test files (or any other files in the repo). This is a straightforward static analysis pattern: when a PR deletes a function definition, search for call sites in unchanged files. If found, flag as high-severity (test breakage). This check would have caught the bug in PR #114 had the review agent been able to run. It would also complement the proposal to add test CI — reviews run even when CI is absent. Confidence: medium — this check is mechanically sound but the review agent's ability to perform cross-file reference analysis depends on its tool access and prompting.

Proposed change

Add guidance to the review agent's correctness sub-agent skill (or the review agent definition in fullsend-ai/agents) to check for 'orphaned references' when a PR deletes function/method definitions. Specifically: when a PR removes a def method_name or class ClassName, the sub-agent should search test files and other unchanged files for references to the deleted symbol. If found, flag as HIGH severity with the finding: 'Method X was removed in file A but is still referenced in file B at lines N'. This check applies to Python, JavaScript, TypeScript, and other languages where function names are used as identifiers.

Validation criteria

When a PR deletes a function and test files still reference it, the review agent flags it as a high-severity correctness finding before the PR can be merged.

---

Generated by retro agent from redhat-community-ai-tools/ci-failure-tracker#114

[fullsend-ai-prioritize]

RICE Priority Score: 2.1

Score breakdown

Dimension	Score	Reasoning
Reach	2	The review agent runs on PRs across all repos using fullsend-ai agents. Orphaned references from deleted functions is a common pattern in any codebase, so this check would benefit most active users across orgs. The specific example affected the redhat-community-ai-tools org but the fix is platform-wide.
Impact	1.5	Catching test breakage before merge is a significant pain point — PR #114 broke 14 test call sites that went undetected. However, CI tests (if present) or manual review can also catch this, so it's not fully blocking. This is a high-value quality gate that prevents real bugs from landing.
Confidence	0.7	The issue is very well-described with a concrete real-world example (PR #114), a clear proposed change, and explicit validation criteria. The mechanical nature of the check (deleted symbol still referenced elsewhere) is well-understood. Some uncertainty remains around the sub-agent's cross-file analysis capabilities and prompting approach.
Effort	1	This requires updating the review agent's correctness sub-agent skill definition to add guidance for detecting orphaned references when a PR deletes function/method/class definitions. The check is conceptually straightforward (search for deleted symbol names in unchanged files) but requires understanding the agent architecture and ensuring it works across Python, JS, and TS.

Formula: (2 x 1.5 x 0.7) / 1 = 2.1