test(#5493): fold PR-to-issue negative into PR-label positive scenario - #5498
Conversation
|
🤖 Finished Review · ✅ Success · Started 4:54 AM UTC · Completed 5:04 AM UTC |
Site previewPreview: https://a75663be-site.fullsend-ai.workers.dev Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
ReviewFindingsMedium
Low
Previous runReviewFindingsMedium
Low
Previous run (2)Looks good to me Labels: PR modifies e2e behaviour dispatch feature tests |
|
/fs-fix Behaviour CI is flaking against the 30m job ceiling; the folded dispatch scenario itself is fine. Evidence
Fix (in this PR)
Out of scope / noteTransient e2e admin failure ( |
|
/fs-fix Please rebase this branch onto current Then apply the prior timeout reliability fix if not already present after rebase:
After push, leave the branch based on up-to-date main so behaviour CI can re-run cleanly with |
|
🤖 Finished Fix · ❌ Failure · Started 7:54 AM UTC · Completed 8:05 AM UTC |
|
The fix agent completed, but the post-fix script failed before finishing.
Workflow run: https://github.com/fullsend-ai/.fullsend/actions/runs/29989530182 Details: |
|
/fs-fix Previous fix run failed at push: the GitHub App cannot update Do only
A maintainer will apply the behaviour |
|
/fs-retro This fix agent comment regarding failure within the sandbox regarding workflow permissions is not properly formatted: #5498 (comment), open an issue |
|
🤖 Finished Fix · ✅ Success · Started 8:12 AM UTC · Completed 8:20 AM UTC |
|
🤖 Finished Retro · ✅ Success · Started 8:13 AM UTC · Completed 8:34 AM UTC |
🔧 Fix agent — iteration 1 (human-triggered)Rebased branch onto current main (0dcbe1a). Rebase resolved cleanly — no conflicts. Only dispatch.feature changed (the PR's intended change). No workflow files, Makefile timeouts, or scenario logic were modified. The post-script needs to force-push since history was rewritten by the rebase. Fixed (2):
Tests: passed Updated by fullsend fix agent |
|
/fs-review Branch is rebased onto current main. Human follow-up commit bumps behaviour job |
|
🤖 Review · ❌ Terminated · Started 8:24 AM UTC · Ended 8:40 AM UTC |
Retro: PR #5498 — fold PR-to-issue negative into PR-label positive scenarioWorkflow overviewPR #5498 was opened by the code agent to resolve issue #5493 (part of #5490). It folds two e2e dispatch scenarios into one, eliminating a separate ~90s settle wait. The change is small (+10/−12 lines, single file Timeline:
What went well
Existing issue corroborationThe fix agent's attempt to modify New proposalOne net-new issue identified: the post-failure comment's Details section uses 4-space indented code blocks instead of fenced code blocks, causing rendering issues on GitHub. See proposal below. Proposals filed
|
|
🤖 Finished Review · ✅ Success · Started 8:24 AM UTC · Completed 8:40 AM UTC |
a238b8e to
870d8aa
Compare
The behaviour test dispatch detection polling loop used dispatchMaxTry=24 × dispatchPoll=5s = 120s (2 minutes) to detect whether GitHub Actions dispatched a workflow in response to an issues event. When GitHub webhook delivery is under load, this 2-minute window is exceeded, causing false test failures that require human intervention to retry (observed on PRs #5497 and #5498 on the same day with unrelated code changes). Raise dispatchMaxTry from 24 to 48, extending the detection window to 48 × 5s = 240s (4 minutes). This stays well within the existing dispatchWait of 12 minutes for workflow completion. Add a regression test that asserts the dispatch detection window is at least 4 minutes. Note: pre-commit could not run (network 403 in sandbox). Manual linting verification is required. Closes #5503
|
/fs-fix Branch drifted again — now ~6 commits behind Do only
After push, maintainer will re-apply |
|
🤖 Finished Fix · ❌ Failure · Started 9:05 AM UTC · Completed 9:10 AM UTC |
|
The fix agent completed, but the post-fix script failed before finishing. Workflow run: https://github.com/fullsend-ai/.fullsend/actions/runs/29993761849 Details: |
Remove the standalone "PR does not trigger issue-only harness" scenario and fold its negative assertion into the "PR label dispatches PR-only harness" scenario. The positive scenario already opens a PR, labels it, and waits for pr-ping to complete — the negative check piggy-backs on the same dispatch round-trip without a separate 90s settle wait. The merged scenario now: - Registers both pr-ping (change_proposal) and issue-only-ping (work_item) with the same label, verifying entity kind filtering - Asserts pr-ping workflow completes and dummy proves execution - Asserts pr-ping was dispatched exactly 1 time - Asserts issue-only-ping agent did not run This follows the enabled/disabled pattern already established by the "Disabled harness is not dispatched while enabled one triggers" scenario. Closes #5493 Signed-off-by: Barak Korren <bkorren@redhat.com>
Healthy behaviour suites run ~24–29m and were flaking against the 30m job ceiling after the folded dispatch scenario already passed. Align the Makefile go test -timeout with the workflow budget. Signed-off-by: Barak Korren <bkorren@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
1e1f61b to
034f846
Compare
|
/fs-review Behaviour is green on Note for findings triage:
|
|
🤖 Finished Review · ✅ Success · Started 9:52 AM UTC · Completed 10:02 AM UTC |
Ready for CODEOWNERS re-approvalAll CI green on Latest review findings (≤ done bar):
Sibling timeout-only PR: #5519 (same bump for landing on main so @rh-hemartin could you re-approve current head for the |
|
🤖 Finished Retro · ✅ Success · Started 10:17 AM UTC · Completed 10:35 AM UTC |
Retro: PR #5498 — fold PR-to-issue negative into PR-label positive scenarioWorkflow overviewPR #5498 was a small e2e test consolidation (+12/−14 lines, 3 files) opened by the code agent at 04:53 UTC to resolve issue #5493. It merged at 10:15 UTC (~5h22m total). The code change itself was clean and correct, but the PR required substantial human intervention due to two known infrastructure bugs that caused fix agent failures. Timeline
What went well
Evidence for existing issues (no new proposals needed)
Agents repoAgent definitions resolved from |
Summary
Fold the standalone "PR does not trigger issue-only harness" negative scenario into the existing "PR label dispatches PR-only harness" positive scenario, eliminating a separate ~90s settle wait and GHA round-trip. The merged scenario registers both a PR-only harness (
pr-ping) and an issue-only harness (issue-only-ping) with the same label, then asserts thatpr-pingdispatched exactly once whileissue-only-pingdid not run.Related Issue
Fixes #5493 (part of #5490)
Changes
dispatch.featureissue-only-ping) withwork_itementity kind and the same labelpr-pingwas dispatched exactly 1 timeissue-only-pingagent did not runTesting
make e2e-testor behaviour subset)Checklist
!for breaking changes)Closes #5493
Post-script verification
agent/5493-fold-pr-issue-negative)c9c7540c2995d9a294ede00c2444d2eb600e4852..HEAD)