What happened
PR #2870 was ejected from the merge queue 5 times over 72 minutes before merging on the 6th attempt. Every ejection was caused by TestAdminInstallUninstall (e2e/admin/admin_test.go:271) failing with Expected value not to be nil. The underlying cause was GitHub API rate limit exhaustion on e2e pool orgs: logs show creating lock repo in test-org-1: rate limited, github api: 403 API rate limit exceeded for user ID, and Hit rate limit, skipping remaining orgs this round. A concurrent PR (#2811) was also in the merge queue, compounding the rate limit pressure. The PR's actual changes (a validation schema fix) were unrelated to the test failure.
What could go better
The test lacks resilience to rate limiting on the e2e pool orgs. When concurrent merge queue runs hit the same pool orgs, rate limits compound and cause cascading failures. This is a recurring pattern — issue #1879 tracks general e2e flakiness, and issue #2618 addresses a related (but different) transient auth failure in the same test file. Neither addresses rate limit resilience specifically. Confidence: high — the failure mode is clear from the logs and the 6th attempt succeeded after enough time for rate limits to reset.
Proposed change
In e2e/admin/admin_test.go, add rate-limit-aware retry logic to TestAdminInstallUninstall. Specifically: (1) detect 403 rate-limit responses from the GitHub API and retry with exponential backoff rather than failing immediately, (2) add a t.Skip fallback when all pool orgs are rate-limited after retries so the test is marked as skipped rather than failed, preventing merge queue ejection for infrastructure flakiness. Additionally, consider staggering e2e pool org assignment across concurrent merge queue entries to reduce rate limit contention.
Validation criteria
The next 5 merge queue runs that encounter GitHub API rate limiting on pool orgs should either retry successfully or skip gracefully instead of failing the test suite and ejecting the PR. Measure by checking merge queue ejection events over the next 2 weeks — ejections caused by TestAdminInstallUninstall rate limiting should drop to zero.
Generated by retro agent from #2870
What happened
PR #2870 was ejected from the merge queue 5 times over 72 minutes before merging on the 6th attempt. Every ejection was caused by
TestAdminInstallUninstall(e2e/admin/admin_test.go:271) failing withExpected value not to be nil. The underlying cause was GitHub API rate limit exhaustion on e2e pool orgs: logs showcreating lock repo in test-org-1: rate limited,github api: 403 API rate limit exceeded for user ID, andHit rate limit, skipping remaining orgs this round. A concurrent PR (#2811) was also in the merge queue, compounding the rate limit pressure. The PR's actual changes (a validation schema fix) were unrelated to the test failure.What could go better
The test lacks resilience to rate limiting on the e2e pool orgs. When concurrent merge queue runs hit the same pool orgs, rate limits compound and cause cascading failures. This is a recurring pattern — issue #1879 tracks general e2e flakiness, and issue #2618 addresses a related (but different) transient auth failure in the same test file. Neither addresses rate limit resilience specifically. Confidence: high — the failure mode is clear from the logs and the 6th attempt succeeded after enough time for rate limits to reset.
Proposed change
In
e2e/admin/admin_test.go, add rate-limit-aware retry logic toTestAdminInstallUninstall. Specifically: (1) detect 403 rate-limit responses from the GitHub API and retry with exponential backoff rather than failing immediately, (2) add at.Skipfallback when all pool orgs are rate-limited after retries so the test is marked as skipped rather than failed, preventing merge queue ejection for infrastructure flakiness. Additionally, consider staggering e2e pool org assignment across concurrent merge queue entries to reduce rate limit contention.Validation criteria
The next 5 merge queue runs that encounter GitHub API rate limiting on pool orgs should either retry successfully or skip gracefully instead of failing the test suite and ejecting the PR. Measure by checking merge queue ejection events over the next 2 weeks — ejections caused by
TestAdminInstallUninstallrate limiting should drop to zero.Generated by retro agent from #2870