fullsend-ai · fullsend-ai-coder · Sep 1, 2026
@@ -273,6 +273,7 @@ jobs:
           BEHAVIOUR_SCM: github
           BEHAVIOUR_CI: githubactions
           BEHAVIOUR_INSTALL_MODE: per-repo
+          BEHAVIOUR_FULLSEND_REF: ${{ github.event.pull_request.head.sha || github.sha }}
           BEHAVIOUR_ARTIFACT_DIR: ${{ runner.temp }}/behaviour-artifacts
           E2E_GCP_PROJECT_ID: ${{ secrets.E2E_GCP_PROJECT_ID }}
           E2E_GCP_WIF_PROVIDER: ${{ secrets.E2E_GCP_WIF_PROVIDER }}

@@ -29,17 +29,17 @@ Runtime selection is shared with production via `defaults.runtime` in org `confi
 - Add **behaviour tests** under `e2e/behaviour/` using **godog** and portable Gherkin feature files.
 - Exercise **real SCM + real CI** through **driver interfaces** (`scm.Driver`, `ci.Driver`, `install.Driver`); v1 implementations target GitHub and GitHub Actions.
 - Substitute inference with a **dummy runtime** (`runtime: dummy` in per-repo config, or `defaults.runtime: dummy` for per-org) that executes scripted operations in the real OpenShell sandbox and emits `behaviour-results.json`.
-- Select backends via **runner env** (`BEHAVIOUR_SCM`, `BEHAVIOUR_CI`, `BEHAVIOUR_INSTALL_MODE`); feature files stay install-mode agnostic. v1 runs **per-repo** against the halfsend org pool; the suite provisions fullsend via `fullsend github setup` rather than requiring pre-installed orgs.
+- Select backends via **runner env** (`BEHAVIOUR_SCM`, `BEHAVIOUR_CI`, `BEHAVIOUR_INSTALL_MODE`); feature files stay install-mode agnostic. ~~v1 runs **per-repo** against the halfsend org pool; the suite provisions fullsend via `fullsend github setup` rather than requiring pre-installed orgs.~~ **Note (2026-09, #6815):** Behaviour tests now use the dedicated `fullsend-ai-test` org with ephemeral `bt-{uuid}-{slot}` repos and `repos install --fullsend-ref`.
 - Use **compatibility tags** (`@skip:*`, `@requires:*`) to filter scenarios for future backends; tags do not select configuration.
 
 ## Consequences
 
 - Behaviour tests can pass while prompt quality regresses; LLM evals remain necessary for instruction coverage.
 - Behaviour orgs are provisioned at suite start with `--runtime dummy`; production orgs must not use dummy unintentionally.
-- **Note (2026-07, #5439 / PR #5489):** Numbered behaviour pool repos (`test-repo-NN`) are lazily created and installed on first scenario use via the ensurer internal to `install.Driver`; suite-start provisioning still applies to the shared admin/`test-repo` install path where used.
+- ~~**Note (2026-07, #5439 / PR #5489):** Numbered behaviour pool repos (`test-repo-NN`) are lazily created and installed on first scenario use via the ensurer internal to `install.Driver`; suite-start provisioning still applies to the shared admin/`test-repo` install path where used.~~ **Note (2026-09, #6815):** Repos are now ephemeral `bt-{uuid}-{slot}` in `fullsend-ai-test`, created per scenario and deleted on deallocation.
 - Adding GitLab or Tekton requires new drivers and runner env values, not feature file rewrites.
 - Dummy runtime op vocabulary stays minimal; new ops require runtime + docs updates when scenarios need them.
-- Behaviour tests depend on live external infrastructure: GitHub API, GitHub Actions runners, GCP WIF/mint, and the shared halfsend org pool. Transient outages, API rate limits, or pool org state corruption can fail the suite; CI distinguishes infrastructure failures from regressions via workflow logs and artifact inspection, but there is no offline fallback.
-- Behaviour tests share the halfsend org pool and lock mechanism with admin e2e tests (`e2e.yml` runs both jobs). Lock hold time scales with scenario count; pool size was doubled to absorb the additional load and can be increased again if contention appears.
+- Behaviour tests depend on live external infrastructure: GitHub API, GitHub Actions runners, GCP WIF/mint, and ~~the shared halfsend org pool~~ the dedicated `fullsend-ai-test` org. Transient outages, API rate limits, or ~~pool org state corruption~~ ephemeral repo lifecycle failures can fail the suite; CI distinguishes infrastructure failures from regressions via workflow logs and artifact inspection, but there is no offline fallback. **Note (2026-09, #6815):** Behaviour tests migrated from the shared halfsend pool to `fullsend-ai-test` with ephemeral repos; pool-specific failure modes (lock contention, state corruption) no longer apply.
+- ~~Behaviour tests share the halfsend org pool and lock mechanism with admin e2e tests (`e2e.yml` runs both jobs). Lock hold time scales with scenario count; pool size was doubled to absorb the additional load and can be increased again if contention appears.~~ **Note (2026-09, #6815):** Behaviour tests no longer share the halfsend org pool or lock mechanism. They use `fullsend-ai-test` with unique `bt-{uuid}-{slot}` repos per CI run, eliminating pool contention.
 
-> **Note (2026-07):** Shared live-test infrastructure (org pool, CLI runner, cleanup) lives in `pkg/e2etest/`; the Gherkin framework lives in `pkg/behaviourtest/`. In-repo runners remain under `e2e/behaviour/` and `e2e/admin/`.
+> **Note (2026-07):** Shared live-test infrastructure (~~org pool, CLI runner, cleanup~~ `TokenForBehaviourOrg`, `BehaviourTestOrg`) lives in `pkg/e2etest/`; the Gherkin framework lives in `pkg/behaviourtest/`. In-repo runners remain under `e2e/behaviour/` and `e2e/admin/`. **(2026-09, #6815):** Behaviour tests use `fullsend-ai-test` with ephemeral repos; org pool and lock helpers are used only by admin e2e.
@@ -28,23 +28,25 @@ Set when starting the suite (not in feature files):
 BEHAVIOUR_SCM=github              # also: gitlab; future: forgejo
 BEHAVIOUR_CI=githubactions        # also: gitlabci; future: tekton
 BEHAVIOUR_INSTALL_MODE=per-repo   # v1 default and only supported value
+BEHAVIOUR_FULLSEND_REF=<sha>      # head SHA for repos install --fullsend-ref (CI: pr head sha)
+E2E_KEEP_REPOS=false              # set true to preserve ephemeral repos for debugging
 ENVIRONMENT=dev                   # mint/infra target: dev (default) or stage
 ```
 
-The suite in `e2e/behaviour/suite_test.go` (or an external runner) acquires a pool org via `pkg/e2etest`, runs pre-install cleanup, calls an `install.Factory` (e.g. `install.NewRepoPoolCFMintPreviews(...)`) to get a unified `install.Driver` that owns mint deploy, pool allocation, repo ensure, and teardown. The suite constructs SCM and CI drivers, then runs godog with `pkg/behaviourtest/suite.InitScenario`. `InitScenario` clones a template `*world.World` per scenario. When a scenario calls "Given the enrolled test repository", `Driver.AllocateRepo` leases a unique repo name and ensures it is created and installed. `Driver.DeallocateRepo` returns the name in the After hook. `Driver.Finalize` tears down suite-scoped resources (e.g. preview mint) and reclaims outstanding leases. Unsupported `BEHAVIOUR_INSTALL_MODE` or `ENVIRONMENT` values fail at suite startup. `ENVIRONMENT` is `dev` or `stage` (empty defaults to `dev`).
+The suite in `e2e/behaviour/suite_test.go` (or an external runner) uses the dedicated `fullsend-ai-test` org, calls an `install.Factory` (e.g. `install.NewRepoPoolCFMintPreviews(...)`) to get a unified `install.Driver` that owns mint deploy, ephemeral repo allocation, repo ensure, and teardown. The suite constructs SCM and CI drivers, then runs godog with `pkg/behaviourtest/suite.InitScenario`. `InitScenario` clones a template `*world.World` per scenario. When a scenario calls "Given the enrolled test repository", `Driver.AllocateRepo` leases a unique ephemeral repo name (`bt-{uuid}-{slot}`) and ensures it is created and installed. `Driver.DeallocateRepo` deletes the ephemeral repo and returns the slot in the After hook. `Driver.Finalize` tears down suite-scoped resources (e.g. preview mint) and reclaims outstanding leases. Unsupported `BEHAVIOUR_INSTALL_MODE` or `ENVIRONMENT` values fail at suite startup. `ENVIRONMENT` is `dev` or `stage` (empty defaults to `dev`).
 
 ### Install driver (unified)
 
 The suite uses a single unified `install.Driver` constructed via `install.Factory` (e.g. `install.NewRepoPoolCFMintPreviews` or `install.NewRepoPoolExternalMint`). Each concrete driver owns the full lifecycle:
 
 1. Deploys the mint (RepoPoolCFMintPreviews: CF Worker preview; RepoPoolExternalMint: pre-configured URL).
-2. Manages an internal channel-based pool of repo names (`test-repo-01` … `test-repo-12`).
-3. Lazily creates and installs numbered pool repos on demand via an internal ensurer (concurrent-safe via singleflight).
+2. Manages an internal channel-based pool of ephemeral repo slots (`bt-{uuid}-{slot}`).
+3. Lazily creates and installs ephemeral repos on demand via an internal ensurer (concurrent-safe via singleflight).
 4. Exposes `AllocateRepo` / `DeallocateRepo` / `Finalize` / `Capacity`.
 
 The Factory takes the allocated org name plus runtime dependencies (forge client, token, CLI binary, GCP project, logger). Driver-specific inputs (PEMs, allowlists, pool size, mint URL) come from env or are computed inside the driver. The suite does not construct or thread pool, ensurer, or mint driver types directly — all internal lifecycle is encapsulated inside the concrete driver returned by the factory. Default concurrency is `driver.Capacity()`; `GODOG_CONCURRENCY` overrides it (warn, do not fail, if concurrency > Capacity).
 
-Pool orgs must already have shared GitHub Apps, org-level mint enrollment, and per-repo mint enrollment for each numbered repo (one-time GCP admin step on the hosted mint project). The driver does not run `fullsend admin install` or `fullsend mint enroll`. See [e2e-testing.md](e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).
+The `fullsend-ai-test` org must have shared GitHub Apps and org-level mint enrollment. Per-repo mint enrollment for ephemeral repos is pre-provisioned by a GCP admin on the hosted mint project. The driver does not run `fullsend admin install` or `fullsend mint enroll`. See [e2e-testing.md](e2e-testing.md#behaviour-tests-and-per-repo-mint-enrollment).
 
 `Finalize` (RepoPoolCFMintPreviews) abandons the preview alias via `fullsend mint delete --platform=cloudflare` and reclaims any outstanding leases with an error. The RepoPoolExternalMint driver's teardown is a no-op.
 
@@ -71,7 +73,7 @@ Steps use `w.Org` and `w.RepoName` (the allocated repo name) plus per-repo const
 
 ## Testing drivers
 
-Prefer unit tests with `httptest` for REST helpers. Optional smoke scenarios against live backends mirror admin e2e credentials (`GITHUB_TOKEN`, halfsend org pool).
+Prefer unit tests with `httptest` for REST helpers. Optional smoke scenarios against live backends mirror admin e2e credentials (`GITHUB_TOKEN`, `fullsend-ai-test` org).
 
 ## Future backends checklist