docs: Document RevertActor and drop "terminal" from CRASHED - #1711
Open
shrutiyam-glitch wants to merge 3 commits into
Open
shrutiyam-glitch wants to merge 3 commits into
shrutiyam-glitch wants to merge 3 commits into
Conversation
RevertActor returns a RUNNING, PAUSED, or CRASHED actor to SUSPENDED at its last external snapshot, so CRASHED is no longer a dead end that only DeleteActor can clear. Docs and code comments still described it as terminal and told operators to delete and recreate the actor, losing its state. Update the api-guide, architecture, upgrade guide, and kubectl-ate README to cover the new verb, and correct the comments that justified keeping a partial external snapshot by naming actor deletion as the only remaining collector -- revert collects it too.
2 tasks
shrutiyam-glitch
marked this pull request as ready for review
September 17, 2026 17:14
Bowei Du (bowei)
added
kind/docs
| } | ||
|
|
||
| // The actor is CRASHED and can only be deleted from here. | ||
| // The actor is CRASHED; delete reclaims its partial snapshot. |
Collaborator
There was a problem hiding this comment.
delete reclaims its partial snapshot.
what does this mean?
| // every object that suspend wrote. CRASHED is terminal, so the actor delete is | ||
| // the only collector left: whatever it cannot name is leaked for good. | ||
| // every object that suspend wrote. When an actor crashes mid-suspend, only | ||
| // delete or revert can collect the partial snapshot: whatever they cannot name |
Collaborator
There was a problem hiding this comment.
only delete or revert can collect the partial snapshot
we are not collecting partial snapshot anywhere right?
Collaborator
Author
There was a problem hiding this comment.
by collecting, I meant only those two actions can get rid of the in progress snapshots.
will reword it.
Zoe Zhao (zoez7)
enabled auto-merge
September 18, 2026 20:20
Julian Gutierrez Oschmann (juli4n)
pushed a commit
to juli4n/substrate
that referenced
this pull request
Sep 18, 2026
#### Summary This PR introduces the `RevertActor` RPC for actor lifecycle management. It includes the API definition, the corresponding workflow execution logic, observability metrics, and updates to the authorization model to support reverting actors. Fixes agent-substrate#1556 Docs updated in agent-substrate#1711 #### Commit-wise Changes **1. Add RevertActor RPC (`1145d99`)** * Introduces the new `RevertActor` RPC to the API definitions. * Updates the corresponding protobuf bindings (affecting files like `ateapi.pb.go` and `ateapi_pb2.py`). **2. Add the revertActor workflow and observability metrics (`dcb1fae`)** * Implements the core `revertActor` workflow logic, designed to be idempotent and re-enterable. It progresses through the following steps: * **Mark Reverting:** Validates that the actor is in a revertable state (`RUNNING`, `PAUSED`, or `CRASHED`) and transitions its state to `REVERTING`. * **Discard Worker:** Safely tears down the execution environment by terminating the workload, detaching volumes, and releasing the assigned worker. * **Collect In-Progress Snapshot:** Cleans up external object storage by deleting any objects a previous suspend operation was partway through writing. * **Finalize:** Commits the actor to `SUSPENDED` and strips all node-local and in-progress state pointers (clearing `WorkerAssignment`, `LocalSnapshotInfo`, etc.), returning the actor to its untouched external snapshot. * Instruments the workflow with lifecycle operation metrics (e.g., updating `ate.actor.lifecycle.operation.duration` to track `revert` operations). * *Note/TODO:* Currently, when reverting a paused actor, the workflow drops the pointer to the node-local state but does *not* actually prune the local checkpoint bytes from the node (this is tracked in agent-substrate#641). **3. Add `can_revert` to the authorization model (`2fc53ae`)** * Adds the `can_revert` permission to the auth model, mirroring the shape of `can_suspend` (editor tier of the parent atespace, plus a direct grant so a machine identity can revert the actor it drives without holding an atespace role). **4. Serve RevertActor and add the CLI verb (`f04fab4`)** * Wires the `Control.RevertActor` service method to the workflow (replacing the generated stub that previously answered `Unimplemented`). * Adds the `"ate revert actor"` CLI command, making the feature usable end-to-end. * Implements `Terminate` for the fake atelet. This was necessary because reverting an actor from the `RUNNING` state is the first path to reach this call in functional tests (previously, delete tests skipped this step as they ran against actors with no worker assignment). **5. Add a manual verify script for RevertActor (`8022db3`)** * Adds a script to manually exercise `RevertActor` against a real control plane, since unit and functional tests only run against a fake atelet. * Tests reverting from `CRASHED`, `RUNNING`, and `PAUSED` states, and verifies that attempting to revert a `SUSPENDED` actor is properly rejected. * Simulates a crash by deleting the worker pod the actor runs on to verify the workflow can handle the absence of a worker to terminate. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RevertActorreturns a RUNNING, PAUSED, or CRASHED actor to SUSPENDED at its last external snapshot, so CRASHED is no longer a dead end that onlyDeleteActorcan clear. Docs and code comments still described it as terminal and told operators to delete and recreate the actor, losing its state.Update the api-guide, architecture, upgrade guide, and kubectl-ate README to cover the new verb, and correct the comments that justified keeping a partial external snapshot by naming actor deletion as the only remaining collector -- revert collects it too.
Follow up for the PR - #1675
Issue - #1556