What happened
On PR #6028, human reviewer @waynesun09 had to query the GitHub API live to verify that fstest-write, fstest-triage, and fstest-outsider are not fullsend-ai org members, all three have only read permission on fullsend-ai/fullsend and fullsend-ai/agents, and their write capability is scoped to disposable pool-org test-repo infrastructure. This blast-radius verification was the key analysis that justified approving the PR. The existing documentation in docs/guides/dev/behaviour-testing.md describes intended permission levels ('write-level', 'triage-level', 'outsider') but does not document the actual production-repo permissions or org membership status of these accounts.
What could go better
Every reviewer of a PR involving test actor PATs must currently re-derive the blast-radius assessment from scratch by querying GitHub APIs. This information is stable (it changes only when accounts are re-provisioned) and should be documented so both agents and humans can reference it. The review agent could not perform the external verification (it lacks API access to check account permissions), but if the documentation existed, it could reference the documented scope to assess blast radius and flag any changes that would violate the containment assumptions. Confidence is high that this documentation gap exists — the behaviour-testing guide was read during this analysis and confirmed to lack this information.
Proposed change
Add a subsection to docs/guides/dev/behaviour-testing.md under the test actor accounts section documenting: (1) fstest-write, fstest-triage, and fstest-outsider are NOT members of the fullsend-ai org; (2) all three have only read permission on fullsend-ai/fullsend and fullsend-ai/agents; (3) their write capability is scoped to disposable pool-org test-repo-NN infrastructure only; (4) classic PATs for these accounts therefore cannot write to production repos even if exfiltrated; (5) this containment should be re-verified whenever new test actor accounts are added, existing accounts are granted additional access, or the pool-org infrastructure changes. Include a 'last verified' date so reviewers know how stale the information is.
Validation criteria
On subsequent PRs involving test actor PATs (such as #6027 which wires up consumption), the review agent should be able to reference this documentation to assess blast radius. The human reviewer's approval comment should not need to independently re-verify account permissions via API — the documented scope should be sufficient unless the PR changes account provisioning. Measurable within the next 2 PRs touching test actor infrastructure.
Generated by retro agent from #6028
What happened
On PR #6028, human reviewer @waynesun09 had to query the GitHub API live to verify that fstest-write, fstest-triage, and fstest-outsider are not fullsend-ai org members, all three have only
readpermission on fullsend-ai/fullsend and fullsend-ai/agents, and their write capability is scoped to disposable pool-org test-repo infrastructure. This blast-radius verification was the key analysis that justified approving the PR. The existing documentation indocs/guides/dev/behaviour-testing.mddescribes intended permission levels ('write-level', 'triage-level', 'outsider') but does not document the actual production-repo permissions or org membership status of these accounts.What could go better
Every reviewer of a PR involving test actor PATs must currently re-derive the blast-radius assessment from scratch by querying GitHub APIs. This information is stable (it changes only when accounts are re-provisioned) and should be documented so both agents and humans can reference it. The review agent could not perform the external verification (it lacks API access to check account permissions), but if the documentation existed, it could reference the documented scope to assess blast radius and flag any changes that would violate the containment assumptions. Confidence is high that this documentation gap exists — the behaviour-testing guide was read during this analysis and confirmed to lack this information.
Proposed change
Add a subsection to
docs/guides/dev/behaviour-testing.mdunder the test actor accounts section documenting: (1) fstest-write, fstest-triage, and fstest-outsider are NOT members of the fullsend-ai org; (2) all three have onlyreadpermission on fullsend-ai/fullsend and fullsend-ai/agents; (3) their write capability is scoped to disposable pool-org test-repo-NN infrastructure only; (4) classic PATs for these accounts therefore cannot write to production repos even if exfiltrated; (5) this containment should be re-verified whenever new test actor accounts are added, existing accounts are granted additional access, or the pool-org infrastructure changes. Include a 'last verified' date so reviewers know how stale the information is.Validation criteria
On subsequent PRs involving test actor PATs (such as #6027 which wires up consumption), the review agent should be able to reference this documentation to assess blast radius. The human reviewer's approval comment should not need to independently re-verify account permissions via API — the documented scope should be sufficient unless the PR changes account provisioning. Measurable within the next 2 PRs touching test actor infrastructure.
Generated by retro agent from #6028