[ARCHIVED] Benchmark issue 359

[guyoron1]

What happened

PR #2947 added tryAgentsRepoFallback in internal/cli/run.go (1495+ lines, 12 files changed), fundamentally altering agent resolution behavior. The functional-tests workflow (.github/workflows/functional-tests.yml) exercises this exact code path (triage eval cases go through runAgent → resolveAgentSource → tryAgentsRepoFallback). However, the workflow's paths filter only included eval/**, internal/scaffold/**, .github/workflows/functional-tests.yml, and .github/scripts/** — not internal/cli/. So functional tests never ran on PR fullsend-ai#2947. The allowlist propagation bug shipped undetected and broke main within 18 hours. PR #3425 added internal/cli/run.go to the path filter as part of the fix. The human reviewer waynesun09 further recommended adding internal/harness/** as well, noting internal/cli/** would be too broad (triggering on ~18% of commits).

What could go better

When a PR modifies code that is exercised by path-filtered CI workflows, the review agent should check whether the modified files are in the workflow's path filter. If they aren't, the review agent should flag that relevant integration tests won't run. This is particularly important when a PR adds a new code path (like tryAgentsRepoFallback) that changes the behavior of code already covered by integration tests. The review agent reviewed PR fullsend-ai#2947 across 5+ completed runs and never flagged the path filter gap. Confidence: medium — detecting this requires the review agent to understand which CI workflows exercise which code paths, which may be complex. However, for cases where a PR adds a new function call to a file that is already part of a test pipeline's execution path, a simpler heuristic could work: 'If the PR modifies files under paths not in any CI workflow path filter, and those files contain functions called by code that IS in a filtered path, flag the gap.'

Proposed change

Add a review heuristic: when reviewing a PR that modifies Go files under internal/cli/ or internal/harness/, check whether .github/workflows/functional-tests.yml includes those paths in its trigger filter. If not, flag it as a potential CI coverage gap. More broadly, the review agent should cross-reference modified files against path-filtered CI workflow triggers and flag files that are exercised by tests but not in the filter. For this repo specifically, waynesun09 recommended adding internal/cli/run.go (done in PR fullsend-ai#3425) and internal/harness/** to the functional-tests paths filter — the internal/harness/** addition should still be made.

Validation criteria

On a future PR that modifies internal/cli/run.go or internal/harness/ files: (1) functional-tests workflow triggers on the PR (verifying the path filter is correct), (2) the review agent flags if a modified file is not in any relevant CI path filter. For the immediate repo fix: verify that internal/harness/** is added to the functional-tests path filter (per waynesun09's recommendation that hasn't been implemented yet).

---

Generated by retro agent from fullsend-ai#3425

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 6:32 AM UTC · Completed 6:34 AM UTC
Commit: a2afa8b · View workflow run →

[fullsend-ai-triage]

This issue appears to be a duplicate of #329 — both are mirrors of the same upstream issue (fullsend-ai#3444) and contain identical content. Issue #329 has already been triaged with a detailed summary covering both the immediate path filter fix and the review agent enhancement.

Closing in favor of #329 to keep discussion in one place.

[guyoron1]

/fs-triage

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 8:43 AM UTC · Completed 8:45 AM UTC
Commit: aa6585d · View workflow run →

[fullsend-ai-triage]

This issue is a duplicate of #329 — both are mirrors of the same upstream issue (fullsend-ai#3444) with identical content.

#329 is already open, triaged, and has had code work dispatched against it. Closing in favor of #329 to keep discussion consolidated.

[guyoron1]

/fs-code

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 8:52 AM UTC · Completed 8:59 AM UTC
Commit: aa6585d · View workflow run →

[fullsend-ai-coder]

This issue is a duplicate of #329. The review heuristic for CI path filter coverage gaps is implemented in PR #405 (agent/329-ci-path-filter-heuristic), which now also closes this issue.

The second part of the issue — adding internal/harness/** to the functional-tests.yml path filter — is not applicable to this repository. This repo has no functional-tests.yml; the e2e.yml workflow already uses **/*.go which covers all Go files including internal/harness/. The path filter gap described in the issue applies to the upstream fullsend-ai/fullsend repo.