feat(runtime): recover Foundry boots after supervisor loss - #659
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4fa15a76af
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d1faa9dac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Previous-epoch write-workspace cleanup cannot finalize a publication-prepared session because its cleanup authority permits deletion only.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
What changed in this PR
Adds default-off Foundry supervisor-loss recovery using combined Kubernetes termination and broker retirement evidence.
Changes:
- Extends harness v2 with broker identity and boot-retirement protocols.
- Adds controller-side enrollment, relay validation, and durable recovery proofs.
- Expands cleanup recovery tests and operator documentation.
| File | Description |
|---|---|
workers/acp/supervisor/status_auth_test.go |
Covers retirement-route authentication. |
workers/acp/supervisor/server.go |
Exposes identity and retirement handling. |
workers/acp/supervisor/server_test.go |
Refines drain behavior coverage. |
workers/acp/supervisor/foundry_recovery.go |
Implements broker recovery relay. |
workers/acp/supervisor/foundry_recovery_test.go |
Tests relay authorization and proofs. |
workers/acp/supervisor/foundry_legacy_wire_test.go |
Protects legacy wire compatibility. |
workers/acp/supervisor/foundry_cleanup_timeout_test.go |
Tests cleanup timeout recovery. |
workers/acp/supervisor/foundry_broker_test.go |
Adds recovery-aware fixtures. |
workers/acp/supervisor/env.go |
Adds profile-digest qualification. |
workers/acp/supervisor/env_test.go |
Tests qualification configuration. |
workers/acp/supervisor/config.go |
Validates recovery prerequisites. |
website/docs/guides/bring-your-own-agent-runtime.md |
Documents broker-backed cleanup. |
website/docs/development/agent-runtime-adapter-contract.md |
Defines the recovery contract. |
internal/harness/v2/protocol.go |
Adds capability, status, and error fields. |
internal/harness/v2/foundry_recovery.go |
Defines recovery requests, proofs, and client. |
internal/harness/v2/foundry_recovery_test.go |
Tests protocol validation and transport. |
internal/harness/v2/client.go |
Maps cleanup-unproven errors. |
internal/controller/agent_runtime_recovery.go |
Enrolls authenticated broker identity. |
internal/controller/agent_runtime_recovery_witness.go |
Persists broker identity in witnesses. |
internal/controller/agent_runtime_recovery_topology_test.go |
Updates Foundry recovery fixtures. |
internal/controller/agent_runtime_recovery_qualification_test.go |
Tests qualification gates. |
internal/controller/agent_runtime_recovery_dispatch.go |
Revalidates Foundry admission identity. |
internal/controller/agent_runtime_proxy_test.go |
Tests full-duplex proxy behavior. |
internal/controller/agent_runtime_foundry_recovery.go |
Combines local and broker retirement proof. |
internal/controller/agent_runtime_foundry_recovery_test.go |
Exercises recovery and fail-closed cases. |
internal/controller/acp_task_runtime_cleanup.go |
Adds old-epoch standalone cleanup fences. |
internal/controller/acp_task_runtime_cleanup_test.go |
Tests immutable cleanup authority. |
internal/controller/acp_session_runtime_cleanup.go |
Adds Task authority revalidation. |
internal/controller/acp_recovery.go |
Integrates standalone runtime cleanup. |
internal/controller/acp_external_runtime_dispatch_test.go |
Updates proxy and replacement-runtime tests. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Security-sensitive distributed recovery semantics warrant final human review, and two authority comments contradict the implemented permissions.
Review effort: Balanced
Findings: 2
Open (2)
Resolved since last review (1)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 98613fb726
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
Qualified Foundry supervisors advertise supportsFoundryRecovery and report the authenticated broker identity in status. After Kubernetes proves the witnessed supervisor container terminated, an authenticated replacement relays PUT /v2/recovery/foundry/retire-boot to the same broker ledger, and the controller persists the broker's sealed-boot proof as a third boot retirement kind. A replacement ledger, missing enrollment evidence, pending creation, or ambiguous invocation cannot supply that proof. Authenticate the replacement supervisor as a retirement-only relay across image rollouts while preserving exact broker identity and pre/post-request authority checks. Gate the extension behind ORKA_ACP_FOUNDRY_RECOVERY_PROFILE_DIGEST so unqualified Foundry boots keep the previous wire contract. Standalone Tasks that end under an older runtime epoch get an exact cleanup fence derived from their terminal projection and frozen binding, and a replacement AgentRuntime without observed capabilities now fails closed instead of recording a cleanup receipt.
…ntity Require the retirement relay's fence to carry the same registered runtime instance as the retired boot; pool UID, generation, and profile alone allowed a different frozen runtime identity to relay retirement. Validate standalone old-epoch cleanup against the frozen binding UID instead of the live object UID, which differs for a restored Task incarnation and left its external RuntimeSession stuck as not ready. Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
…iguration Enrollment witnessed whatever agent configuration digest a qualified broker reported. A broker partitioned under a different configuration could then certify retirement of sessions that never belonged to the registered profile. Reject enrollment unless the broker's configuration digest equals the AgentRuntime profile's digest. Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
Standalone previous-epoch cleanup reused the Session teardown fence, whose client permits only runtime session deletion. A write-workspace Task whose session was still publication_prepared therefore had its evidence-bound finalization refused locally and could never reach deletion or a cleanup receipt. Mark the standalone fence as permitting finalization while Session teardown authority stays delete-only. Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
The fence comments still said deletion-only after standalone cleanup gained evidence-bound publication finalization. Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
b3acddb to
e973ebc
Compare
Prove cleanup after a Foundry supervisor is lost, split out of #589. This extension is default-off and independent of brokered tool approvals.
Qualified Foundry supervisors advertise
supportsFoundryRecoveryand report the authenticated broker identity in/v2/status. After Kubernetes proves the witnessed supervisor container terminated, an authenticated replacement relaysPUT /v2/recovery/foundry/retire-bootto the same broker ledger, and the controller persists the broker's sealed-boot proof as a third boot-retirement kind. A replacement ledger, missing enrollment evidence, pending creation, or ambiguous invocation cannot supply that proof.ORKA_ACP_FOUNDRY_RECOVERY_PROFILE_DIGESTgates the extension to an exact runtime profile digest; unqualified Foundry boots keep the previous wire contract, which the legacy-wire test pins.Companion revision: Foundry runtime
32cd33f5.Known simplification follow-ups, deliberately not done here to keep the diff identical to the live-validated code: reuse
poolOpsfor retirement replay instead of a second map with its own sweep; extendcontrolOnceinstead of the parallelfoundryRecoveryJSONtransport; drop the proofUnmarshalJSONrequired-field loop thatValidateForalready covers.Stack:
fix/acp-cancellation-and-cleanup→ this PR → #589.Validation:
make lint-fix,make test, and the Foundry-focused suites ininternal/harness/v2,workers/acp/supervisor, andinternal/controller. The live Foundry supervisor-loss and image-rollout scenarios were exercised on the original combined branch (see #589).