diff --git a/docs/guides/dev/behaviour-testing.md b/docs/guides/dev/behaviour-testing.md index 865285573..2853bd751 100644 --- a/docs/guides/dev/behaviour-testing.md +++ b/docs/guides/dev/behaviour-testing.md @@ -231,6 +231,8 @@ The After hook calls `Driver.DeallocateRepo` to return the slot. `Driver.Finaliz Concurrent callers for the same repo are serialized via `singleflight.Group` — only one goroutine runs the create+install flow while others wait. This removes the requirement for numbered `test-repo-NN` repos to be pre-provisioned in the pool org. +**Credential context separation:** The suite's e2e installation token and dispatch's per-repo `GITHUB_TOKEN` are distinct credential contexts with independent permission propagation graphs. After a pool repo is deleted and recreated, the suite can confirm the repo exists (via `GetRepo`), but it **cannot** observe or predict when dispatch-side collaborator permissions will be ready. Do not add `GetCollaboratorPermission` polling to the suite-side readiness checks — the suite's token resolves permissions through a different GitHub subsystem than dispatch's token. See the [package doc comment](../../../pkg/behaviourtest/drivers/install/doc.go) for details and the empirical evidence from [#6701](https://github.com/fullsend-ai/fullsend/issues/6701). + **Suite duration:** Because each leased `test-repo-NN` pays create + inference provision + `github setup` on first use in a run, serial godog suites take longer than the old shared-`test-repo` model. CI budgets **45 minutes** for the behaviour job (`timeout-minutes` and `go test -timeout`) to match. Runner env (defaults shown): diff --git a/pkg/behaviourtest/drivers/install/doc.go b/pkg/behaviourtest/drivers/install/doc.go new file mode 100644 index 000000000..8dceb3634 --- /dev/null +++ b/pkg/behaviourtest/drivers/install/doc.go @@ -0,0 +1,47 @@ +// Package install provides pool-repo lifecycle management for behaviour +// tests: lazy repo creation, fullsend installation, pool leasing, and +// teardown. +// +// # Credential context separation +// +// Two distinct credential contexts operate on pool repos. Understanding +// this separation is essential when working on repo-readiness checks +// after pool repo recreation. +// +// The test suite holds an e2e GitHub App installation token, minted via +// OIDC at CI job start. This token drives all suite-side API calls: +// repo creation/deletion, label application, PR management, and the +// readiness-polling functions in this package (awaitCreation, +// awaitDeletion, awaitWorkflowReady). +// +// Dispatch workflows (harness-dispatch, harness-run) operate with the +// pool repo's own GITHUB_TOKEN — a per-workflow-run token scoped to the +// repository where the workflow executes. This token is used by +// harness-dispatch to call GetCollaboratorPermission when authorizing +// the triggering actor. +// +// These are independent credential contexts with independent permission +// propagation graphs. When a pool repo is deleted and recreated +// (resetRepo), the suite's e2e token can observe the new repo via +// GetRepo (awaitCreation) as soon as the repo object propagates through +// GitHub's eventual-consistency layer. However, the dispatch-side +// GITHUB_TOKEN's view of collaborator permissions on the new repo ID +// propagates on a separate, slower schedule that the suite cannot +// observe or predict. +// +// Consequence: the suite CANNOT reliably probe or wait for dispatch-side +// permission readiness by calling GetCollaboratorPermission from the +// suite side. The suite's e2e installation token and dispatch's +// GITHUB_TOKEN resolve permissions through different GitHub subsystems. +// Polling GetCollaboratorPermission with the suite's token tells you +// nothing about whether dispatch's token will see the correct +// permissions. This was empirically validated in issue #6701: both a +// human developer (PR #6703) and an autonomous agent (PR #6709) +// independently attempted suite-side permission polling with +// exponential backoff, and both failed — 0 of 12 repos resolved over +// 63 seconds. +// +// Any fix for dispatch-side permission propagation delays on freshly +// recreated repos must operate on the dispatch side (e.g., retry within +// the dispatch workflow itself) rather than from the suite side. +package install diff --git a/pkg/behaviourtest/drivers/install/ensure.go b/pkg/behaviourtest/drivers/install/ensure.go index 78d1b936d..b1ca5abc2 100644 --- a/pkg/behaviourtest/drivers/install/ensure.go +++ b/pkg/behaviourtest/drivers/install/ensure.go @@ -292,6 +292,12 @@ func (e *repoEnsurer) ensureRepoExists(ctx context.Context, org, repoName, targe // newly created repo is visible via the API. GitHub's eventual // consistency means operations on a just-created repo can return 404 // until propagation completes. +// +// NOTE: This function confirms repo visibility only — not dispatch-side +// permission readiness. Do not add GetCollaboratorPermission polling +// here; it will not work. See the package doc comment in doc.go for the +// credential context separation that makes suite-side permission +// probing unreliable. func (e *repoEnsurer) awaitCreation(ctx context.Context, org, repoName, target string) error { e.logf("[ensure] waiting for %s creation to propagate", target) delay := resetRetryDelay