Skip to content

fix(#2432): retry merge on 409 after updating PR branch - #2434

Merged
rh-hemartin merged 4 commits into
mainfrom
fix/2432-merge-409-retry
Jul 21, 2026
Merged

fix(#2432): retry merge on 409 after updating PR branch#2434
rh-hemartin merged 4 commits into
mainfrom
fix/2432-merge-409-retry

Conversation

@ralphbean

Copy link
Copy Markdown
Member

Summary

  • When MergeChangeProposal gets a 409 "Head branch is out of date", it now calls GitHub's PUT .../pulls/{n}/update-branch to sync the PR with the base, waits 3s, then retries the merge (up to 3 attempts).
  • Non-409 errors are returned immediately (no behavior change).
  • Fixes the flaky TestAdminInstallUninstall failure at the enrollment PR merge step.

Closes #2432

Test plan

  • TestMergeChangeProposal_Success — happy path unchanged
  • TestMergeChangeProposal_409UpdatesBranchAndRetries — 409 triggers update-branch then successful retry
  • TestMergeChangeProposal_NonConflictErrorNotRetried — 422 not retried
  • TestMergeChangeProposal_409PersistsAfterRetries — gives up after max attempts with clear error

🤖 Generated with Claude Code

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Retry PR merge after updating branch on 409 “out of date” conflicts
🐞 Bug fix 🧪 Tests 🕐 20-40 Minutes

Grey Divider

Description

• Retry squash-merge when GitHub returns 409 “head branch is out of date”.
• Auto-call update-branch, wait briefly, then re-attempt merge (max 3 tries).
• Add focused HTTP tests covering success, retry, non-retry errors, and retry exhaustion.
Diagram

sequenceDiagram
  participant Caller as Caller
  participant GHClient as LiveClient
  participant GitHub as GitHub API

  Caller->>GHClient: MergeChangeProposal(owner, repo, number)
  GHClient->>GitHub: PUT /pulls/{n}/merge (squash)
  alt Merge succeeds
    GitHub-->>GHClient: 200 OK
    GHClient-->>Caller: nil
  else 409 Head out of date
    GitHub-->>GHClient: 409 Conflict
    GHClient->>GitHub: PUT /pulls/{n}/update-branch
    GitHub-->>GHClient: 202 Accepted
    GHClient-->>GHClient: wait 3s (or ctx cancel)
    GHClient->>GitHub: PUT /pulls/{n}/merge (retry, up to 3)
    alt Eventually succeeds
      GitHub-->>GHClient: 200 OK
      GHClient-->>Caller: nil
    else Still 409 after max attempts
      GitHub-->>GHClient: 409 Conflict
      GHClient-->>Caller: error (out of date after retries)
    end
  else Non-409 error
    GitHub-->>GHClient: 4xx/5xx
    GHClient-->>Caller: error (no retry)
  end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Poll update-branch completion instead of fixed 3s sleep
  • ➕ More reliable under load/queueing (avoids retrying before branch update finishes).
  • ➕ Can reduce unnecessary delay when updates complete quickly.
  • ➖ Requires an additional API to check update status (or repeated merge attempts anyway).
  • ➖ Adds complexity (poll loop, backoff, timeout semantics).
2. Exponential backoff with jitter for retries
  • ➕ More resilient to variable GitHub processing time and transient errors.
  • ➕ Reduces thundering-herd behavior if many merges hit 409 simultaneously.
  • ➖ Slightly more complex logic and test expectations.
  • ➖ May increase total time-to-fail in persistent conflict cases.
3. Use GitHub Auto-merge (enable and let GitHub merge when up-to-date)
  • ➕ Offloads retry/merge timing to GitHub.
  • ➕ Potentially fewer client-side heuristics (sleep/retry).
  • ➖ May require repo/PR settings and additional permissions.
  • ➖ Behavioral change: merge becomes asynchronous and harder to reason about in workflows.

Recommendation: The PR’s approach is a pragmatic fix for flaky merges: detect the specific 409, trigger update-branch, and retry a small fixed number of times. The main improvement to consider is replacing the fixed 3s delay with a bounded poll/backoff strategy (or backoff+jitter) to better match GitHub’s actual update latency, but the current implementation is a reasonable minimal change given the added test coverage.

Files changed (2) +154 / -5

Bug fix (1) +33 / -5
github.goRetry merge on 409 by updating PR branch via update-branch endpoint +33/-5

Retry merge on 409 by updating PR branch via update-branch endpoint

• Adds a bounded retry loop to 'MergeChangeProposal' that specifically handles HTTP 409 conflicts by calling GitHub’s 'update-branch' endpoint, waiting briefly, and retrying the merge. Non-409 errors still return immediately, and a final error is returned if the branch remains out of date after all attempts.

internal/forge/github/github.go

Tests (1) +121 / -0
github_merge_test.goAdd tests for merge retry/update-branch behavior and retry limits +121/-0

Add tests for merge retry/update-branch behavior and retry limits

• Introduces an 'httptest'-based suite validating: normal merge success, 409-triggered update-branch then successful retry, non-409 errors not retried, and persistent 409s eventually failing after retries.

internal/forge/github/github_merge_test.go

@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown

Site preview

Preview: https://67ea9b9e-site.fullsend-ai.workers.dev

Commit: d5e1f4cd457cdc6b4c24842178483ef25e59d62d

@qodo-code-review

qodo-code-review Bot commented Jun 18, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 51 rules
✅ Skills: writing-user-docs, writing-adrs

Grey Divider


Action required

1. range maxAttempts invalid loop ✗ Dismissed 📘 Rule violation ≡ Correctness
Description
MergeChangeProposal uses for attempt := range maxAttempts, which is invalid Go syntax and will
fail compilation. This will cause make go-vet (and therefore linting in CI) to fail for this PR.
Code

internal/forge/github/github.go[2064]

+	for attempt := range maxAttempts {
Evidence
The compliance checklist requires make go-vet and make lint to pass. The new code introduces
invalid Go syntax at internal/forge/github/github.go:2064, which will fail compilation and
therefore fail go vet (and linting that includes vet).

Rule 1062050: Go code must pass make go-vet without issues
Rule 1062039: All code changes must pass make lint without failures
internal/forge/github/github.go[2060-2066]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`MergeChangeProposal` contains an invalid Go loop: `for attempt := range maxAttempts { ... }`, which will not compile.

## Issue Context
This change is intended to retry merge attempts up to `maxAttempts` times, so the loop should iterate `attempt` from 0 to `maxAttempts-1`.

## Fix Focus Areas
- internal/forge/github/github.go[2060-2091]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Update-branch errors ignored ✓ Resolved 🐞 Bug ≡ Correctness
Description
MergeChangeProposal calls the PR update-branch endpoint via do() but never checks the HTTP
status code nor returns updateErr, so it can keep retrying merges even when the branch update was
rejected/rate-limited and ultimately return a misleading “branch remained out of date” error. It
also performs an update-branch call on the final 409 attempt even though no subsequent merge retry
will occur.
Code

internal/forge/github/github.go[R2076-2088]

+		// Update the PR branch to incorporate base branch changes.
+		updateResp, updateErr := c.do(ctx, http.MethodPut, updatePath, map[string]string{})
+		if updateErr == nil {
+			updateResp.Body.Close()
+		}
+
+		if attempt < maxAttempts-1 {
+			select {
+			case <-time.After(3 * time.Second):
+			case <-ctx.Done():
+				return ctx.Err()
+			}
+		}
Evidence
In MergeChangeProposal, the new update-branch request is made with do() and its error/status
are not acted upon. The do() helper explicitly returns responses without status checking, while
checkStatus() is the mechanism used elsewhere to turn non-success statuses into APIErrors—so
skipping it here means update failures are silently ignored.

internal/forge/github/github.go[2056-2092]
internal/forge/github/github.go[95-140]
internal/forge/github/github.go[216-236]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`MergeChangeProposal` uses `c.do()` for `PUT .../update-branch` and ignores both (a) non-2xx HTTP status codes and (b) `updateErr`. Since `do()` does not treat non-2xx as errors, the code can proceed as if the branch update succeeded when it actually failed, and it may perform an unnecessary final `update-branch` call even when no further merge retry will happen.

## Issue Context
- `do()` returns `(*http.Response, nil)` for non-2xx responses; status validation must be done via `checkStatus` (or an equivalent helper).
- `update-branch` commonly returns `202 Accepted`, so the status checker should accept 202.

## Fix Focus Areas
- internal/forge/github/github.go[2059-2092]
- internal/forge/github/github.go[95-165]
- internal/forge/github/github.go[216-236]

## Suggested fix approach
1. Only call `update-branch` when `attempt < maxAttempts-1` (i.e., when a retry will actually occur).
2. After `updateResp, updateErr := c.do(...)`:
  - If `updateErr != nil`, return a wrapped error (e.g., `fmt.Errorf("update pull request #%d branch: %w", number, updateErr)`).
  - Otherwise `defer updateResp.Body.Close()` and validate status with `checkStatus(updateResp, http.StatusAccepted, http.StatusOK, http.StatusNoContent)` (whatever is correct for your usage).
  - If status validation fails, return a wrapped error so the caller sees the real cause.
3. Consider preserving the last merge 409 `err` and including it in the final failure message for better diagnostics (optional but helpful).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Comment thread internal/forge/github/github.go
Comment thread internal/forge/github/github.go Outdated
@fullsend-ai-review

fullsend-ai-review Bot commented Jun 18, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:34 PM UTC · Completed 4:56 PM UTC
Commit: fd6cbd9 · View workflow run →

@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.57143% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/forge/github/github.go 88.57% 3 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 18, 2026

Copy link
Copy Markdown

Review

Findings

Low

  • [redundant-retry] e2e/admin/admin_test.go:292 — The outer retry loops in mergeEnrollmentPR (lines 292–316) and mergeScaffoldPR (lines 354–373) still catch 409 errors, call UpdatePullRequestBranch, and retry MergeChangeProposal. Since MergeChangeProposal now handles 409 retries internally (3 attempts with branch updates and SHA polling), the two retry layers compose multiplicatively: up to 3×3=9 merge attempts. The outer loops' UpdatePullRequestBranch calls are also redundant. The outer retry logic should be removed or simplified to a plain call now that the method handles retries internally.
    Remediation: Remove the retry-with-409-handling loops in mergeEnrollmentPR and mergeScaffoldPR and replace each with a single MergeChangeProposal call.
Previous run

Review

The change correctly adds 409 retry logic to MergeChangeProposal: on conflict, it updates the PR branch via the existing UpdatePullRequestBranch method, waits 3 seconds (context-aware), and retries up to 3 times. Non-409 errors return immediately. Error wrapping with %w preserves errors.As compatibility throughout. The implementation properly guards against updating the branch on the final attempt, handles context cancellation during the sleep, and returns a descriptive error on exhaustion that wraps the last merge error.

The four new unit tests cover the key paths: happy path, 409→update→retry→success, non-409 immediate failure, and retry exhaustion. Test assertions are precise (exact counts, not weak inequalities).

No security, intent, documentation, or cross-repo contract concerns.

Findings

Low

  • [incomplete-scope] e2e/admin/admin_test.go:292mergeEnrollmentPR (lines 292–316) and mergeScaffoldPR (lines 354–373) still contain their own 3-attempt retry loops with UpdatePullRequestBranch and 5-second sleeps for 409 errors. Now that MergeChangeProposal handles 409 retries internally (3 attempts, 3-second delays), both layers retry independently. Since the final error wraps the original *APIError with %w, errors.As in the e2e code still matches StatusConflict, creating a 3×3=9 attempt scenario with cumulative delays of ~22s. The outer retry is redundant and masks internal retry failures.
    Remediation: In a follow-up change, simplify both e2e merge helpers to call MergeChangeProposal directly without their own retry loops.

  • [test-inadequate] internal/forge/github/github_merge_test.go — No test covers the scenario where the update-branch endpoint fails (e.g., returns 422 or a network error) or where the context is cancelled during the 3-second inter-retry sleep. The production code has distinct error paths for both ("update branch failed" wrapper, and ctx.Err() return), but neither is exercised.
    Remediation: Add tests for: (1) update-branch returning a non-success status after a 409 merge; (2) context cancellation during the retry delay.

Previous run (2)

Review

Findings

Low

  • [incomplete-scope] e2e/admin/admin_test.go:297mergeEnrollmentPR (lines 292–316) and mergeScaffoldPR (lines 354–373) still contain their own 3-attempt retry loops with UpdatePullRequestBranch and 5-second sleeps for 409 errors. Now that MergeChangeProposal handles 409 retries internally (3 attempts, 3-second delays), both layers retry independently. Since the final error wraps the original *APIError with %w, errors.As in the e2e code still matches StatusConflict, creating a 3×3=9 attempt scenario with cumulative delays of ~24s. The outer retry is redundant and masks internal retry failures.
    Remediation: In a follow-up change, simplify both e2e merge helpers to call MergeChangeProposal directly without their own retry loops.

  • [test-inadequate] internal/forge/github/github_merge_test.go — No test covers the scenario where the update-branch endpoint fails (e.g., returns 422 or a network error). The production code has two distinct error paths for update-branch failures (c.do error and checkStatus failure), but neither is exercised. Context cancellation during the 3-second inter-retry sleep is also untested.
    Remediation: Add tests for: (1) update-branch returning a non-success status after a 409 merge; (2) context cancellation during the retry delay.

  • [error-message-format] internal/forge/github/github.go — The error messages use "update pull request #%d branch:" but the existing UpdatePullRequestBranch method uses "update pull request branch #%d:". Inconsistent word order for the same operation.
    Remediation: Align the error message format to match UpdatePullRequestBranch: "update pull request branch #%d:".

  • [doc-style] internal/forge/github/github.go — The function comment says "retries up to 3 times with a short delay between attempts" but omits the specific delay duration (3 seconds) and total potential delay (~9s). The comparable retryOnRepoRace function documents its timing precisely: "linear backoff (2s between attempts) and up to 5 attempts (~10s total)."
    Remediation: Specify the delay: "retries up to 3 times with a 3-second delay between attempts (~9s maximum)."

Previous run (3)

Review

Findings

Low

  • [incomplete-scope] e2e/admin/admin_test.go:297mergeEnrollmentPR (lines 292–316) and mergeScaffoldPR (lines 354–373) still contain their own 3-attempt retry loops with UpdatePullRequestBranch and 5-second sleeps for 409 errors. Now that MergeChangeProposal handles 409 retries internally (3 attempts, 3-second delays), both layers retry independently. Since the final error wraps the original *APIError with %w, errors.As in the e2e code still matches StatusConflict, creating a 3×3=9 attempt scenario with cumulative delays of ~24s. The outer retry is redundant and masks internal retry failures.
    Remediation: In a follow-up change, simplify both e2e merge helpers to call MergeChangeProposal directly without their own retry loops.

  • [test-inadequate] internal/forge/github/github_merge_test.go — No test covers the scenario where the update-branch endpoint fails (e.g., returns 422 or a network error). The production code has two distinct error paths for update-branch failures (c.do error and checkStatus failure), but neither is exercised. Context cancellation during the 3-second inter-retry sleep is also untested.
    Remediation: Add tests for: (1) update-branch returning a non-success status after a 409 merge; (2) context cancellation during the retry delay.

  • [error-message-format] internal/forge/github/github.go — The error messages use "update pull request #%d branch:" but the existing UpdatePullRequestBranch method uses "update pull request branch #%d:". Inconsistent word order for the same operation.
    Remediation: Align the error message format to match UpdatePullRequestBranch: "update pull request branch #%d:".

  • [doc-style] internal/forge/github/github.go — The function comment says "retries up to 3 times with a short delay between attempts" but omits the specific delay duration (3 seconds) and total potential delay (~9s). The comparable retryOnRepoRace function documents its timing precisely: "linear backoff (2s between attempts) and up to 5 attempts (~10s total)."
    Remediation: Specify the delay: "retries up to 3 times with a 3-second delay between attempts (~9s maximum)."

Previous run (4)

Review

Findings

Medium

  • [incomplete-scope] e2e/admin/admin_test.go:318 — The e2e test mergeEnrollmentPR (lines 318–338) still contains its own 3-attempt retry loop with UpdatePullRequestBranch and 5-second sleeps for 409 errors. Now that MergeChangeProposal handles this internally, both layers retry independently. Since the final error from MergeChangeProposal wraps the original *APIError with %w, errors.As in the e2e code will still match, creating a 3×3=9 merge attempt scenario with cumulative delays of ~24s. The outer retry loop is now redundant at best and masks failures at worst.
    Remediation: In a follow-up change, simplify the e2e retry call sites to call MergeChangeProposal directly without their own retry loops.

Previous run (5)

Review

Findings

Medium

  • [error-handling-gap] internal/forge/github/github.go:2078 — The update-branch call's result is never inspected for success. Since c.do() returns (resp, nil) for all non-retryable HTTP responses (including 4xx errors), the code only closes the body but never checks updateResp.StatusCode. If the update-branch endpoint returns 403 (permissions), 422 (merge conflict), or any other error status, the code silently proceeds to sleep and retry the merge. When all retries are exhausted, the final error says "branch remained out of date" with no context about update-branch failures. The existing UpdatePullRequestBranch method demonstrates the correct pattern: checkStatus(resp, http.StatusAccepted).
    Remediation: After the c.do call, check updateResp.StatusCode for non-success codes (the endpoint returns 202 on success). Include update-branch failure context in the final error message.

  • [error-message-format] internal/forge/github/github.go:2093 — The final error on retry exhaustion uses fmt.Errorf without %w, so it does not wrap the last merge error. This prevents callers from using errors.As/errors.Is on the result. Compare with retryOnRepoRace (line 597) which wraps the last error with %w.
    Remediation: Capture the last merge error and wrap it: fmt.Errorf("merge pull request #%d: branch out of date after %d attempts: %w", number, maxAttempts, lastErr).

Low

  • [edge-case] internal/forge/github/github.go:2078 — On the final loop iteration (attempt == maxAttempts-1), when the merge fails with 409, the code still calls update-branch even though the loop will not execute another merge attempt. This is a wasted API call.
    Remediation: Guard the update-branch call with if attempt < maxAttempts-1.

  • [test-inadequate] internal/forge/github/github_merge_test.go:111TestMergeChangeProposal_409PersistsAfterRetries asserts mergeAttempts.Load() > 1 but the contract is exactly 3 attempts (maxAttempts). A weaker assertion would still pass if the retry logic were accidentally changed.
    Remediation: Use assert.Equal(t, int32(3), mergeAttempts.Load()).

  • [test-inadequate] internal/forge/github/github_merge_test.go — No test covers the scenario where the update-branch endpoint fails (e.g., returns 422 or a network error). This is the scenario where the silent error handling in production code matters most.

  • [doc-style] internal/forge/github/github.go:2060 — The function comment documents the retry behavior but omits the delay duration (3 seconds) and total potential delay (~9s). Compare with retryOnRepoRace which documents timing details.

Previous run (6)

Review

Findings

Medium

  • [error-handling-gap] internal/forge/github/github.go:2078 — The update-branch call's result is never inspected for success. Since c.do() returns (resp, nil) for all non-retryable HTTP responses (including 4xx errors), the code only closes the body but never checks updateResp.StatusCode. If the update-branch endpoint returns 403 (permissions), 422 (merge conflict), or any other error status, the code silently proceeds to sleep and retry the merge. When all retries are exhausted, the final error says "branch remained out of date" with no context about update-branch failures. The existing UpdatePullRequestBranch method demonstrates the correct pattern: checkStatus(resp, http.StatusAccepted).
    Remediation: After the c.do call, check updateResp.StatusCode for non-success codes (the endpoint returns 202 on success). Include update-branch failure context in the final error message.

  • [error-message-format] internal/forge/github/github.go:2093 — The final error on retry exhaustion uses fmt.Errorf without %w, so it does not wrap the last merge error. This prevents callers from using errors.As/errors.Is on the result. Compare with retryOnRepoRace (line 597) which wraps the last error with %w.
    Remediation: Capture the last merge error and wrap it: fmt.Errorf("merge pull request #%d: branch out of date after %d attempts: %w", number, maxAttempts, lastErr).

Low

  • [edge-case] internal/forge/github/github.go:2078 — On the final loop iteration (attempt == maxAttempts-1), when the merge fails with 409, the code still calls update-branch even though the loop will not execute another merge attempt. This is a wasted API call.
    Remediation: Guard the update-branch call with if attempt < maxAttempts-1.

  • [test-inadequate] internal/forge/github/github_merge_test.go:111TestMergeChangeProposal_409PersistsAfterRetries asserts mergeAttempts.Load() > 1 but the contract is exactly 3 attempts (maxAttempts). A weaker assertion would still pass if the retry logic were accidentally changed.
    Remediation: Use assert.Equal(t, int32(3), mergeAttempts.Load()).

  • [test-inadequate] internal/forge/github/github_merge_test.go — No test covers the scenario where the update-branch endpoint fails (e.g., returns 422 or a network error). This is the scenario where the silent error handling in production code matters most.

  • [doc-style] internal/forge/github/github.go:2060 — The function comment documents the retry behavior but omits the delay duration (3 seconds) and total potential delay (~9s). Compare with retryOnRepoRace which documents timing details.


Labels: PR modifies the GitHub forge client to fix an e2e test flake.

Previous run (7)

Review

Findings

Medium

  • [error-handling-gap] internal/forge/github/github.go:2076 — The update-branch call's error is silently discarded. If c.do() returns an error (network failure, context cancellation, rate limit exhaustion), the code proceeds to sleep and retry the merge with no indication that the update failed. When all retries are exhausted, the final error message says "branch remained out of date" with no context about update-branch failures, making the issue difficult to diagnose in production.
    Remediation: At minimum, log the update error (if updateErr != nil). Consider including the last update-branch error in the final returned error message when all retries are exhausted.

Low

  • [edge-case] internal/forge/github/github.go:2076 — On the final loop iteration (attempt == maxAttempts-1), when the merge fails with 409, the code still calls update-branch even though the loop will not execute another merge attempt. This is wasted work and an unnecessary API call.
    Remediation: Move the update-branch call inside the if attempt < maxAttempts-1 block.

  • [test-inadequate] internal/forge/github/github_merge_test.go:111TestMergeChangeProposal_409PersistsAfterRetries asserts mergeAttempts.Load() > 1 (at least 2), but the contract is exactly 3 attempts (maxAttempts). A weaker assertion would pass even if the retry logic were accidentally changed to only retry once.
    Remediation: Use assert.Equal(t, int32(3), mergeAttempts.Load()) for precise verification.

  • [test-inadequate] internal/forge/github/github_merge_test.go — No test for update-branch failure (e.g., the update endpoint returning 422 or a network error). This is the scenario where the silent error swallowing in production code matters most.

  • [pattern-inconsistency] internal/forge/github/github.go:2080 — Body close is conditional (if updateErr == nil). The safer defensive pattern is if updateResp != nil { updateResp.Body.Close() }, which handles cases where the response object exists alongside a non-nil error.

  • [doc-style] internal/forge/github/github.go:2056 — The function comment documents the retry behavior but omits the delay duration (3 seconds) and total potential delay (~9s). Compare with retryOnTransient which documents timing details.


Labels: PR fixes a bug in the GitHub forge client that caused flaky e2e test failures

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 22, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 2:47 PM UTC · Completed 3:13 PM UTC
Commit: c140351 · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment component/e2e End-to-end tests and removed requires-manual-review Review requires human judgment labels Jun 22, 2026

@ifireball ifireball left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment by Qodo

@ralphbean ralphbean left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed fc5c076. Qodo's first item (range over int) is a false positive — for i := range n has been valid since Go 1.22 and we use it in do() already. The second item about ignoring update-branch errors is legit though, and the fullsend review flagged the same thing plus a few more.

Changes:

  • update-branch response now goes through checkStatus(), same pattern as UpdatePullRequestBranch
  • skips the update-branch call on the final iteration where no retry follows
  • wraps the last merge error with %w in the exhaustion message
  • tightened the retry test to assert exactly 3 attempts
  • added a test for update-branch failure

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 24, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:55 PM UTC · Completed 5:08 PM UTC
Commit: fc5c076 · View workflow run →

@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment and removed requires-manual-review Review requires human judgment labels Jun 24, 2026
@ralphbean
ralphbean requested a review from ifireball June 29, 2026 17:55
ralphbean added 2 commits July 9, 2026 16:24
When MergeChangeProposal gets a 409 "Head branch is out of date", call
GitHub's update-branch endpoint to sync the PR branch with the base,
wait briefly, then retry the merge. Up to 3 attempts before giving up.
Non-409 errors are still returned immediately.

Closes #2432

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
Check the update-branch response status code with checkStatus (matching
the pattern in UpdatePullRequestBranch) and return updateErr if the
request itself fails. Skip the update-branch call on the final attempt
since no retry will follow. Wrap the last merge error with %w so
callers can inspect the underlying APIError. Tighten the test assertion
from > 1 to exactly 3 attempts.

Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
@ralphbean
ralphbean force-pushed the fix/2432-merge-409-retry branch from fc5c076 to ddb9b37 Compare July 9, 2026 20:32
fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed requires-manual-review Review requires human judgment labels Jul 9, 2026

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] redundant-retry — e2e/admin/admin_test.go

Both mergeEnrollmentPR (lines 296-316) and mergeScaffoldPR (lines 370-389) have their own 3-attempt retry loops that catch 409 errors, call UpdatePullRequestBranch, and sleep 5 seconds. With MergeChangeProposal now retrying internally (3 attempts, 3s delays), these layers compound to 3×3 = 9 merge attempts with up to ~24s of cumulative delay. The outer retry still matches the 409 via errors.As because the final error wraps the *APIError with %w.

Suggested fix: Remove the outer retry loops from both helpers — a single MergeChangeProposal call now handles the full retry sequence. Can be a tracked follow-up if preferred.


Assisted-by: Claude (review), Gemini (review), Codex (review)

Comment thread internal/forge/github/github.go Outdated
Replace inline update-branch logic with a call to the existing
UpdatePullRequestBranch method, eliminating inconsistencies between
the two call sites.

Signed-off-by: Ralph Bean <rbean@redhat.com>
Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 9, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 9:11 PM UTC · Completed 9:23 PM UTC
Commit: d5e1f4c · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jul 9, 2026

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review squad (5 agents: Claude, Gemini, Codex) findings, deduplicated against existing PR comments. Most MEDIUM+ items from this pass were already raised and resolved in earlier commits (ddb9b37, d5e1f4c) or already open in the prior review (the redundant e2e retry-loop finding at #4666709137 — not re-posted here). Three new unposted findings below.

Assisted-by: Claude (review), Gemini (review), Codex (review)

Comment thread internal/forge/github/github.go Outdated
Comment thread internal/forge/github/github.go Outdated
Comment thread internal/forge/github/github_merge_test.go
Replace the flat 3-second sleep after update-branch with SHA polling
that confirms the async branch update actually landed before retrying
the merge. Extract retry timing into package vars so tests don't pay
real wall-clock delays.

Add tests for update-branch failure mid-retry and context cancellation
during the poll loop.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ralph Bean <rbean@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 10, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:26 PM UTC · Completed 6:40 PM UTC
Commit: d069686 · View workflow run →

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed ready-for-merge All reviewers approved — ready to merge labels Jul 10, 2026

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All prior review-squad findings (async update-branch race, hardcoded retry delay, missing test coverage) were fixed correctly in d069686 — verified by reading the fix and running go test ./internal/forge/github/... -race -run TestMergeChangeProposal -v (6/6 pass, race-clean, ~1s).

Non-blocking: the PR description is stale — it still says "waits 3s, then retries" but that was replaced by SHA polling, and the test plan doesn't list the two new tests (TestMergeChangeProposal_UpdateBranchFailsMidRetry, TestMergeChangeProposal_ContextCancelledDuringPoll). Worth a quick update before merge for anyone reading the PR later.

Assisted-by: Claude (review)

@ifireball

Copy link
Copy Markdown
Member

Is this still needed, given that #2432 is closed?

@ralphbean

Copy link
Copy Markdown
Member Author

#2435 was a tactical fix — it added retry logic at the e2e test call site so the test stops flaking. This one moves the retry into MergeChangeProposal itself so every caller gets it automatically. It also adds SHA polling to confirm the async update-branch actually landed before retrying, which #2435 doesn't do (it just sleeps 5s and hopes).

So yeah, the bug that motivated it is fixed at the test level, but the underlying library call still can't handle a 409 on its own. If any other caller hits the same race, they'd get the same failure.

@rh-hemartin

Copy link
Copy Markdown
Member

Merging as Ralph is on PTO

@rh-hemartin
rh-hemartin added this pull request to the merge queue Jul 21, 2026
Merged via the queue into main with commit c088a3c Jul 21, 2026
29 of 31 checks passed
@rh-hemartin
rh-hemartin deleted the fix/2432-merge-409-retry branch July 21, 2026 11:47
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jul 21, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 11:49 AM UTC · Completed 12:00 PM UTC
Commit: d069686 · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #2434 — Retry merge on 409 after updating PR branch

Timeline: Issue #2432 filed Jun 18 (flaky 409 on enrollment PR merge). Triage agent ran (run 27773271387), diagnosed correctly, recommended fixing at the mergeEnrollmentPR call-site. Code agent (run 27773660122) followed triage guidance and opened PR #2435 (call-site fix, merged same day). Human (ralphbean) independently opened PR #2434 fixing at the MergeChangeProposal implementation level — a more general solution protecting all callers.

PR #2434 went through 6 review agent runs over 33 days (Jun 18 – Jul 21). The review agent consistently found legitimate low/medium findings (error handling gaps, edge cases, test assertion weakness) with zero false positives. However, three important findings were caught only by human reviewer waynesun09 (using a multi-model review squad): (1) [high] race conditionUpdatePullRequestBranch returns 202 Accepted (async) but the code used a flat 3s sleep without verifying the update completed; (2) [medium] code reuse — new code reimplemented the update-branch API call inline instead of calling the existing UpdatePullRequestBranch method; (3) [medium] hardcoded retry delay — no jitter, not injectable for tests, inconsistent with existing retryDelay() patterns.

What went well: The review agent produced zero false positives across 6 runs (vs Qodo's 50% false-positive rate). Its error-handling and test-adequacy findings were all actionable and drove real fixes. The triage agent's root-cause diagnosis was accurate.

What could go better: Two key gaps emerged — the triage agent's fix-location recommendation (call-site vs implementation level) and the review agent's inability to detect inline reimplementation of existing methods.

Existing issue evidence:

  • The async API semantics gap (review agent missing 202 vs 200 distinction) is well-covered by fullsend-ai/agents#314, which was filed from a nearly identical finding on PR feat(forge): add GitLab forge client implementation #4101 (GitLab forge). This PR provides additional supporting evidence for that issue.
  • Redundant review runs (6 dispatches for this PR) are covered by 17+ existing issues including #5139, #2599, and #2587.
  • The code agent following triage too faithfully overlaps with fullsend-ai/agents#267 (code agent should critically evaluate suggested fixes). The proposals below address the complementary triage-side and review-side gaps.

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/e2e End-to-end tests ready-for-merge All reviewers approved — ready to merge type/bug Confirmed defect in existing behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(e2e): flaky 409 "Head branch is out of date" when merging enrollment PR

4 participants