Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/guides/dev/behaviour-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,8 @@ The After hook calls `Driver.DeallocateRepo` to return the slot. `Driver.Finaliz

Concurrent callers for the same repo are serialized via `singleflight.Group` — only one goroutine runs the create+install flow while others wait. This removes the requirement for numbered `test-repo-NN` repos to be pre-provisioned in the pool org.

**Credential context separation:** The suite's e2e installation token and dispatch's per-repo `GITHUB_TOKEN` are distinct credential contexts with independent permission propagation graphs. After a pool repo is deleted and recreated, the suite can confirm the repo exists (via `GetRepo`), but it **cannot** observe or predict when dispatch-side collaborator permissions will be ready. Do not add `GetCollaboratorPermission` polling to the suite-side readiness checks — the suite's token resolves permissions through a different GitHub subsystem than dispatch's token. See the [package doc comment](../../../pkg/behaviourtest/drivers/install/doc.go) for details and the empirical evidence from [#6701](https://github.com/fullsend-ai/fullsend/issues/6701).

**Suite duration:** Because each leased `test-repo-NN` pays create + inference provision + `github setup` on first use in a run, serial godog suites take longer than the old shared-`test-repo` model. CI budgets **45 minutes** for the behaviour job (`timeout-minutes` and `go test -timeout`) to match.

Runner env (defaults shown):
Expand Down
47 changes: 47 additions & 0 deletions pkg/behaviourtest/drivers/install/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
// Package install provides pool-repo lifecycle management for behaviour
// tests: lazy repo creation, fullsend installation, pool leasing, and
// teardown.
//
// # Credential context separation
//
// Two distinct credential contexts operate on pool repos. Understanding
// this separation is essential when working on repo-readiness checks
// after pool repo recreation.
//
// The test suite holds an e2e GitHub App installation token, minted via
// OIDC at CI job start. This token drives all suite-side API calls:
// repo creation/deletion, label application, PR management, and the
// readiness-polling functions in this package (awaitCreation,
// awaitDeletion, awaitWorkflowReady).
//
// Dispatch workflows (harness-dispatch, harness-run) operate with the
// pool repo's own GITHUB_TOKEN — a per-workflow-run token scoped to the
// repository where the workflow executes. This token is used by
// harness-dispatch to call GetCollaboratorPermission when authorizing
// the triggering actor.
//
// These are independent credential contexts with independent permission
// propagation graphs. When a pool repo is deleted and recreated
// (resetRepo), the suite's e2e token can observe the new repo via
// GetRepo (awaitCreation) as soon as the repo object propagates through
// GitHub's eventual-consistency layer. However, the dispatch-side
// GITHUB_TOKEN's view of collaborator permissions on the new repo ID
// propagates on a separate, slower schedule that the suite cannot
// observe or predict.
//
// Consequence: the suite CANNOT reliably probe or wait for dispatch-side
// permission readiness by calling GetCollaboratorPermission from the
// suite side. The suite's e2e installation token and dispatch's
// GITHUB_TOKEN resolve permissions through different GitHub subsystems.
// Polling GetCollaboratorPermission with the suite's token tells you
// nothing about whether dispatch's token will see the correct
// permissions. This was empirically validated in issue #6701: both a
// human developer (PR #6703) and an autonomous agent (PR #6709)
// independently attempted suite-side permission polling with
// exponential backoff, and both failed — 0 of 12 repos resolved over
// 63 seconds.
//
// Any fix for dispatch-side permission propagation delays on freshly
// recreated repos must operate on the dispatch side (e.g., retry within
// the dispatch workflow itself) rather than from the suite side.
package install
6 changes: 6 additions & 0 deletions pkg/behaviourtest/drivers/install/ensure.go
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,12 @@ func (e *repoEnsurer) ensureRepoExists(ctx context.Context, org, repoName, targe
// newly created repo is visible via the API. GitHub's eventual
// consistency means operations on a just-created repo can return 404
// until propagation completes.
//
// NOTE: This function confirms repo visibility only — not dispatch-side
// permission readiness. Do not add GetCollaboratorPermission polling
// here; it will not work. See the package doc comment in doc.go for the
// credential context separation that makes suite-side permission
// probing unreliable.
func (e *repoEnsurer) awaitCreation(ctx context.Context, org, repoName, target string) error {
e.logf("[ensure] waiting for %s creation to propagate", target)
delay := resetRetryDelay
Expand Down
Loading