test(behaviour): inference provision hits IAM policy conflict 409 under parallel ensure

[guyoron1]

Upstream: fullsend-ai/issues/5672

What happens

When the behaviour suite leases numbered pool repos (test-repo-NN) and RepoEnsurer runs fullsend inference provision for several repos at once, ensure sometimes fails with:

provisioning WIF for inference: granting Agent Platform access for repo <org>/test-repo-NN: IAM policy conflict (status 409)

The leased scenario then fails in the after-scenario / background ensure path before the Gherkin steps under test run.

Seen repeatedly on PR fullsend-ai#5407 behaviour jobs (e.g. https://github.com/fullsend-ai/fullsend/actions/runs/30350962576 on test-repo-08, https://github.com/fullsend-ai/fullsend/actions/runs/30354560530 on test-repo-10). Re-running the failed job often passes once contention clears.

What should happen

Parallel ensure / inference provision for distinct pool repos should not leave scenarios failing permanently on a transient IAM policy conflict. Either provision succeeds after contention, or ensure surfaces a clear, recoverable path so the suite is not flaky.

How to reproduce

1. Run behaviour CI (make behaviour-test / E2E behaviour job) with default GODOG_CONCURRENCY.
2. Force several repos through create + inference provision in the same window (fresh test-repo-NN missing post-install files).
3. Intermittently observe IAM policy conflict (status 409) during Agent Platform access grant.

Context

Follow-up from fullsend-ai#5407. Orthogonal to URL-sourced harness FetchPolicy / host-repo work; infra flake in pool ensure. Always-re-vendor on already-installed repos (also from fullsend-ai#5407) may increase concurrent GCP IAM traffic and make this more visible, but the conflict appears during new repo provision as well.

No dedicated open issue found for this WIF/IAM 409 in behaviour ensure.

[guyoron1]

/fs-triage

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 10:42 AM UTC · Completed 10:45 AM UTC
Commit: 9394865 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: medium · Category: bug (test infrastructure flake)

Problem

Parallel behaviour scenarios trigger concurrent fullsend inference provision calls for distinct pool repos. GCP IAM policy updates use optimistic concurrency — when two provision calls try to grant Agent Platform access simultaneously, the second sees a stale etag and gets a 409 conflict. The provision code does not retry on this transient error, so the ensure step fails before the scenario under test runs.

Root Cause Hypothesis

The IAM read-modify-write cycle in the inference provision path lacks retry logic for 409 responses. GCP IAM policies are etag-guarded; concurrent writes to the same policy resource conflict even when granting access for different repos (if they share an IAM policy ancestor).

Recommended Fix

Add retry with exponential backoff and jitter on HTTP 409 in the IAM policy grant path. On conflict, re-read the current policy (fresh etag) and re-apply. 3–5 retries with sub-second initial backoff should resolve the transient contention.

Proposed Test

Test: IAM grant retries on 409 conflict
- Stub IAM API to return 409 on first 2 calls, succeed on 3rd
- Assert: grant function succeeds after retries
- Assert: non-409 errors (e.g. 403, 500) are NOT retried
- Assert: if all retries exhausted (all 409), returns error

Cross-references

• Upstream tracking: test(behaviour): inference provision hits IAM policy conflict 409 under parallel ensure fullsend-ai/fullsend#5672 (open, no PR in flight)
• Not a duplicate of test(behaviour): fork branch create fails with 409 empty repository #697 (different 409 — that one is empty-repo during fork branch creation)
• Orthogonal to test(behaviour): "good-local" harness flaky timeout in url-dispatch integrity scenario #706 (harness timeout flake)

---

Labels: This is a confirmed bug in test infrastructure (behaviour suite flake from missing retry logic). Marking as type/bug and triaged.

[fullsend-ai-coder]

🤖 Code · ❌ Terminated · Started 10:46 AM UTC · Ended 10:59 AM UTC
Commit: 9394865 · View workflow run →

[guyoron1]

/fs-code

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 10:46 AM UTC · Completed 10:59 AM UTC
Commit: 9394865 · View workflow run →

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 11:06 AM UTC · Completed 11:11 AM UTC
Commit: 9394865 · View workflow run →

[fullsend-ai-coder]

An open PR already addresses this issue — skipping automated implementation.

• fix(#698): add exponential backoff with jitter to IAM retry #712 by @app/fullsend-ai-coder

To override, comment /fs-code --force on this issue.

_{Posted by fullsend pre-code check}