Reclaim actor node state when a worker pod goes away

[Sairaj Pokale (sairajp-rewind)]

Fixes #1688

Nothing removes actors/<actor-UID>/ once the worker pod hosting the actor is gone. The directory holds the actor's durable-dir contents, checkpoint/restore state and bundle overlay, so it is as large as the workload made it and it persists for the life of the node. This adds reclaim on the delete path, a sweep for the cases no delete path visits, and a wipe on uninstall.

Three commits, reviewable in order:

commit	what
ateapi,atelet: reclaim actor node state when a worker's pod is gone	the delete path
atelet: sweep orphaned actor state directories	the backstop
hack: wipe the node state hostPath on uninstall	uninstall leaves nothing behind

1. Reclaim on the worker-delete path

WorkerWorkflow.releaseBoundActor now calls Terminate before releasing each bound actor — the last moment anything still knows which node holds the state. It dials by node (DialForAteletOnNode), since the pod it would otherwise dial through is what just disappeared, and is best-effort: an unreachable node must not wedge worker deregistration.

Terminate needed fixing too. It tore the workload down first and returned early on failure, never reaching the unmounts or the directory removal. A gone ateom (missing socket, no sandbox record, UNAVAILABLE) now skips that step and still reclaims. A reachable ateom that rejects the call still fails — that sandbox is live.

2. Sweep as a backstop

Evictions, node crashes and uninstalls leave directories no delete path will ever visit. cmd/atelet/actorgc.go runs at startup and on a period, like the existing imagegc.go, and compares actors/ against the actors the control plane says belong on this node.

It does not delete:

• anything, if that list cannot be read in full — it fails closed;
• actors this atelet is currently running;
• directories younger than --actor-gc-min-age;
• directories with a local pause snapshot. A PAUSED actor has one, and has no worker assignment to show up in the list by.

It renames a directory before deleting it, so a crash mid-delete leaves debris the next pass cleans up instead of a half-empty directory that still looks live.

Flags, all in cmd/atelet/actorgc.go: --actor-gc-period (default 5m; 0 disables the periodic pass, startup still runs), --actor-gc-min-age (default 10m), --actor-gc-dry-run (default false).

3. Wipe on uninstall

--delete-all tore down the control plane but never touched the hostPath it wrote to, so the data outlived the software — and fresh atelets would start on top of it. delete_ate_system now runs a short-lived DaemonSet that empties the state directory on every node, sequenced after the atelet DaemonSet is gone so nothing is writing to it. Readiness is the wipe having finished, so a completed rollout means every node is clean. --keep-node-state opts out.

Verification

3-node cluster, durable-dir workload: 15 concurrent clients each writing a 1 GiB file into its durable dir and suspending/resuming for 10 minutes.

	before	after
delete path — scale the WorkerPool to 0	+0.00 GB, 12 dirs remain	26.23 GB → 0, 0 dirs
sweep — start atelets over pre-existing orphans	untouched	23 dirs / 29.1 GB reclaimed, 0 failures
uninstall — --delete-all	+0.00 GB, 16 dirs remain	26.24 GB → 0 on every node

On the delete path, 12 reclaim calls produced 12 matching Terminates that took the gone-ateom branch — a per-node match against the directories present (5/5/2). The sweep's first pass ran before ateapi was serving and correctly reclaimed nothing (reading the live actor set failed; skipping this pass).

Reproduced and fixed on both sandbox classes; they share ateompath.BasePath, so the layout and the leak are identical.

Relation to #1677 / #1678

Same pattern, different directory. #1678 removes ateoms/<pod-UID>/ on graceful ateom shutdown; this is actors/<actor-UID>/, written by atelet, and the exits that strand it are exactly the ungraceful ones a defer does not cover.

The sweep here and the janitor proposed in #1677 landed on the same shape — both modeled on imagegc.go, node-scoped, fail-closed, min-age. If both land, worth deciding whether they should be one loop.

Not covered

The graceful path. Reclaiming during Checkpoint would keep steady state near zero and shrink the sweep's caseload; that is a separate change.

Terminate idempotency generally. Only one case improves here: Terminate with no live sandbox now succeeds instead of erroring.

• Tests pass
• Appropriate changes to documentation are included in the PR

[Sairaj Pokale (sairajp-rewind)]

cc: shrutiyam-glitch Julian Gutierrez Oschmann (@juli4n)