Skip to content

test(e2e): require verified runtime and cluster cleanup - #660

Merged
Sertaç Özercan (sozercan) merged 2 commits into
mainfrom
test/e2e-cleanup-evidence
Sep 23, 2026
Merged

Sertaç Özercan (sozercan) merged 2 commits into
mainfrom
test/e2e-cleanup-evidence

Conversation

@sozercan

@sozercan Sertaç Özercan (sozercan) commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Replace best-effort E2E AfterSuite cleanup with recorded evidence, split out of #589. Nothing in the approval work depends on it.

Every Task still present at AfterSuite, plus the specs that opt into the harness, gets an observer finalizer, is deleted with a UID precondition, and has its controller cleanup receipt recomputed and released before the suite ends. Kind teardown is recorded the same way, and a jq gate fails CI when any boundary is unproved. The Gateway lane verifies terminal retention before fixture teardown, and ephemeral clusters run the full undeploy and namespace delete before kind delete.

Compared with the original branch, the two build-tagged unit-test files for the test helpers (1,063 lines that only ran inside the live lane) are dropped; the live specs exercise the same helpers.

Cost to be aware of: the Gateway lane now waits the real retention loop, so its go test timeout moves from 30 to 55 minutes and the job from 60 to 75.

Validation: go vet -tags e2e ./test/e2e/, bash -n on the scripts, actionlint on the three workflows, make lint-fix.

Replace best-effort AfterSuite cleanup with recorded evidence. Every
Task gets an observer finalizer, is deleted with a UID precondition, and
has its controller cleanup receipt recomputed and released before the
suite ends. Kind teardown is recorded the same way, and a jq gate fails
CI when any boundary is unproved.

The Gateway lane now verifies terminal retention before fixture
teardown, and ephemeral clusters run the full undeploy and namespace
delete before kind delete.
Copilot AI balanced review requested due to automatic review settings September 23, 2026 06:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-23T06:35:38.236016Z a1bdc3d PR opened
🔒 Security Review ✅ Completed 2026-09-23T06:38:18.961731Z a1bdc3d PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The gate can miss Tasks deleted before AfterSuite, and teardown stops after the first cleanup failure.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity

Open (3)
What changed in this PR

Adds verifiable E2E resource and Kind-cluster cleanup evidence, including Gateway retention validation.

Changes:

  • Records Task, Session, suite, and Kind cleanup evidence.
  • Enforces evidence gates and uploads artifacts in CI.
  • Extends Gateway E2E retention coverage and timeouts.
File Description
test/​e2e/​live_agent_runtime_matrix_test.go Uses evidence-backed runtime cleanup.
test/​e2e/​gateway_test.go Verifies Gateway retention cleanup.
test/​e2e/​gateway_interim_test.go Configures fixture retention and ingress tracking.
test/​e2e/​gateway_cleanup_test.go Adds Gateway cleanup evidence collection.
test/​e2e/​e2e_suite_test.go Makes suite teardown recorded and strict.
test/​e2e/​cleanup_evidence_test.go Implements Task and Session cleanup verification.
scripts/​tests/​e2e-cleanup-test.sh Tests cleanup gates and exit behavior.
scripts/​live-copilot-proxy-e2e.sh Requires evidence before retrying.
scripts/​lib/​e2e-cleanup.sh Validates evidence and Kind deletion.
.github/​workflows/​test-e2e.yml Gates general E2E on cleanup proof.
.github/​workflows/​live-copilot-proxy-e2e.yml Uploads per-attempt cleanup evidence.
.github/​workflows/​gateway-e2e.yml Adds retention time and cleanup gating.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/e2e/cleanup_evidence_test.go
Comment thread test/e2e/cleanup_evidence_test.go
Comment thread test/e2e/e2e_suite_test.go
Mark the evidence record as receipt-required as soon as the observed
Task carries a runtime session identity, so an admitted Task that loses
its execution status during deletion fails closed instead of passing as
a non-ACP Task.

Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
Copilot AI review requested due to automatic review settings September 23, 2026 06:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The claimed cleanup scope is overstated, and documentation still describes removed behavior.

Review effort: Balanced
Findings: None

Resolved since last review (3)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Remove stale ephemeral cleanup documentation and workflow setting

test/​e2e/​e2e_suite_test.go:302

This now performs normal per-resource cleanup for all lanes, but website/docs/development/testing.md:212 still states that E2E_EPHEMERAL_CLUSTER skips that cleanup, and the Gateway workflow still exports the now-unused variable. Update the documentation and remove the stale setting so operators do not rely on behavior that no longer exists.

@sozercan
Sertaç Özercan (sozercan) merged commit be6b083 into main Sep 23, 2026
27 checks passed
@sozercan
Sertaç Özercan (sozercan) deleted the test/e2e-cleanup-evidence branch September 23, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants