[Bug]: ateapi: a plain actor delete never tells the node, so the actor's state directory is never reclaimed

[chw120]

What happened?

create → resume → suspend → delete, every step successful, nothing force-deleted and nothing crashed — and the node still keeps /var/lib/ateom-gvisor/actors/<actor-UID>/ with the actor's durable-dir
data, checkpoints and bundle overlays in it.

Only atelet can delete that directory, and Terminate is the only RPC that asks it to. A plain delete never sends one.

Why

Two independent gates, either enough on its own.

1. No worker assignment. ensureAteletTerminated returns early when status.worker_assignment is unset:

substrate/cmd/ateapi/internal/controlapi/workflow_delete.go

Lines 121 to 125 in 27bf344

	assignment := actor.GetStatus().GetWorkerAssignment()
	if assignment == nil {
	slog.InfoContext(ctx, "actor has no worker assignment, skipping atlet terminate request", slog.Any("actor", actorRef))
	return nil
	}

That is the normal state, not an edge case. A plain delete only accepts SUSPENDED or CRASHED:

substrate/cmd/ateapi/internal/controlapi/workflow_delete.go

Lines 309 to 320 in 27bf344

	shouldDelete := false
	switch st {
	case ateapipb.ActorState_ACTOR_STATE_SUSPENDED,
	ateapipb.ActorState_ACTOR_STATE_CRASHED:
	shouldDelete = true
	default:
	// This allows deletion for any state
	shouldDelete = anyState
	}
	if !shouldDelete {
	return nil, status.Errorf(codes.FailedPrecondition, "Actor %s is not in a deletable state (state: %v)", actorRef, st)
	}

and every transition into those states clears the assignment in the same commit —workflow_suspend.go#L412, crash.go#L101, workflow_pause.go#L282, workflow_revert.go#L251,
workflow_worker_delete.go#L217. Every actor that can be deleted has already lost the pointer the delete needs.

2. Worker no longer hosts the actor. Even with the assignment retained, the delete still skips:

substrate/cmd/ateapi/internal/controlapi/workflow_delete.go

Lines 127 to 141 in 27bf344

	if workerName := assignment.GetWorker().GetName(); workerName != "" {
	// Ask whether the worker still HOSTS this actor, not whether its one
	// assignment happens to be this actor: a worker hosting several is the
	// ordinary case, and the others are none of this delete's business.
	hosted, err := workerHostsActor(ctx, w.store, workerName, actor.GetMetadata().GetUid())
	if err != nil {
	return err
	}
	if !hosted {
	slog.InfoContext(ctx, "worker is no longer assigned to this actor, skipping atelet terminate request",
	slog.String("worker", workerName),
	slog.Any("actor", actorRef))
	return nil
	}
	}

ensureSuspendedFinalized calls releaseWorker before committing SUSPENDED, which drops the worker↔actor assignment row.

After a suspend commits, ateapi holds no pointer to the node at all: status.local_snapshot_info.node_vms_with_local_snapshots is the only other record of which nodes hold an actor's files, and suspend clears it in the same commit (workflow_suspend.go#L413).

Expected Behavior

Deleting an Actor reclaims its state directory on the node that holds it — whatever state it was deleted from, and whether or not the control plane still has a pointer to that node.

Steps to Reproduce

Reproduction

Two tests, no fault injection, branched off the commit above:

https://github.com/chw120/substrate/tree/probe-delete-assignment

TestDeleteActor_AteletTerminateGatedOnWorkerAssignment holds everything but status.worker_assignment fixed, against a fake atelet that records Terminate:

--- PASS: no worker assignment: the node is never told    (0 Terminate received)
--- PASS: worker assignment present: the node is told     (1 Terminate received)

The second row is the control — the fake atelet is reachable and does get called when an assignment exists.

TestSuspendThenDeleteNeverReachesAtelet runs the real ensureSuspendedFinalized, then deletes from SUSPENDED:

after suspend: state = SUSPENDED, worker_assignment = nil
delete logs:   "actor has no worker assignment, skipping atlet terminate request"
atelet received: 0 Terminate

Commenting out toUpdate.Status.WorkerAssignment = nil at workflow_suspend.go#L412 does not change the outcome — the second gate fires instead:

delete logs:   "worker is no longer assigned to this actor, skipping atelet terminate request"
atelet received: 0 Terminate

Sandbox Runtime

gVisor (runsc)

Agent Substrate Version / Commit SHA

27bf344

Kubernetes Version & Environment

No response

Host OS & Architecture

No response

Relevant Logs and Diagnostic Output

Additional Context

No response

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.

[chw120]

cc: Zoe Zhao (@zoez7)