Proposal: Add a shim-managed Kata backend for Ateom checkpoint/restore

[Apokleos]

Background

Agent Substrate does not currently have an Ateom backend where the Kata shim owns the sandbox and checkpoint/restore lifecycle.

I propose adding a dedicated ateom-kata backend behind the existing Ateom workload lifecycle API. It would use the Kata Containers runtime-rs shim-v2 to manage sandboxes, tasks, checkpoints, and restores. This does not require a containerd daemon, and no Kata-specific objects need to leak into the public Actor API.

Proposed design

atelet
  -> ateom-kata
      -> in-process Kata runtime adapter (internal/sandboxd)
          -> runtime-rs shim-v2
              -> VMM + kata-agent
                  -> Actor workload

ateom-kata talks to the Kata shim directly over TTRPC. It handles shim startup, Sandbox and Task calls, checkpoint/restore, runtime configuration, compatibility checks, rollback, and cleanup.

The integration point is the Kata shim, not a particular hypervisor. The current POC uses QEMU/KVM, but that is only the first configuration I have validated. Other Kata-supported VMM profiles can be added once their runtime-rs and checkpoint/restore paths have been tested.

Previous discussion and related work

• Agent Substrate Issue #123
• Checkpoint/restore API discussion in Issue #123
• Recent Slack discussion about the Kata shim lifecycle

Two points from the Slack discussion are directly relevant here:

1. Ateom should reuse the Sandbox, Task, and checkpoint/restore interfaces exposed by the Kata shim instead of rebuilding the same runtime setup. This does not mean running containerd inside Ateom.
2. Kata should remain an internal runtime detail. Once the shim exposes suitable checkpoint/restore APIs, Substrate can use those APIs instead of driving a VMM directly.

Upstream Pod-level checkpoint/restore work is moving forward as well:

• The CheckpointPod and RestorePod CRI APIs landed in kubernetes/kubernetes#140366 and shipped in Kubernetes v1.37.
• The Kubernetes control path is still under review in kubernetes/kubernetes#140186.
• The containerd-side work is tracked in containerd/containerd#13822 and #13979.
• The experimental Kata shim work is tracked in kata-containers#13653 and draft PR #13654.

That work is useful context, but ateom-kata does not depend on the Kubernetes Pod checkpoint/restore path or on a containerd daemon. Its direct dependency is a pinned version of the experimental Kata shim API. That API is not a stable part of Kata 4.0 or 4.1.

A few existing Agent Substrate threads also affect the initial scope:

• The runtime modularity roadmap, Issue #1695, and PR #1708 provide context for the Kata runtime and guest asset work.
• Issue #1724 tracks guest CPU and memory sizing. PR #1136 covers restart and runtime adoption concerns.
• PR #1524 calls out entropy requirements after guest restore. Issue #1284 tracks restore-time policy reconciliation.

Current POC

I have a working POC rebased onto a recent Substrate main. It currently includes:

• POC Branch：ateom-kata POC
• SandboxClass=kata control-plane and WorkerPool wiring, runtime asset handling, and a standalone ateom-kata entry point;
• shim-managed run, checkpoint, restore, terminate, restored-task adoption, workload statistics, and cleanup;
• operation and runtime compatibility metadata, same-node restore checks, read-only checkpoint input during restore, and file size/SHA-256 validation;
• Worker CNI networking, an Actor-owned network namespace, direct egress, mTLS ingress, graceful shutdown, and failure rollback.

I tested the POC with real QEMU/KVM and normal Worker Pod CNI networking, including application state continuity after restore. The current branch also passes the targeted tests for ateom-kata, sandboxd, atelet, WorkerPool controller integration, and resource validation.

Before asking for merge, I still need to add a repository-owned, reproducible real-runtime E2E test for the exact public commit. The experimental shim protobufs and version-specific handling will stay inside the Kata adapter, with the Kata source and API version pinned and documented.

Temporary staging of the experimental Sandbox API

The Sandbox checkpoint/restore API used by this POC has not yet been published in a released upstream Go module. For now, the
matching protobuf and TTRPC bindings are kept under internal/sandboxd/sandboxapi.

This package is a temporary compatibility layer, not a Substrate-owned API and not a long-term fork of the containerd Sandbox API.
Keeping it under internal/ prevents the experimental contract from becoming part of Substrate's public API.

The POC pins the exact upstream Kata commit that implements this contract. Once a compatible Sandbox checkpoint/restore API is
released upstream and supported by the containerd/Kata shim, the local package will be removed. Substrate will then import the upstream
package through the normal Go module workflow and regenerate vendor/.

The intended migration is:

1. update the containerd/Kata shim to the released upstream contract;
2. replace internal/sandboxd/sandboxapi imports with the upstream package;
3. update go.mod and regenerate vendor/;
4. remove the local proto and generated bindings.

This migration should not require changes to the public Actor or Ateom APIs because the experimental protocol remains isolated
inside the Kata runtime adapter.

Proposed MVP

To keep the first PR series small enough to review, I propose the following MVP:

• Kata runtime-rs shim-v2 with a pinned experimental checkpoint/restore API;
• QEMU/KVM as the first validated runtime profile;
• cold start and standard Worker CNI networking;
• volume-free, Full-scope, same-node checkpoint and restore;
• application state continuity and runtime compatibility checks after restore;
• rollback, cleanup, and a reproducible real-runtime E2E test.

The first MVP will not support:

• workload volumes or Data/Data-on-Golden snapshots;
• cross-node restore or runtime adoption after an ateom-kata process restart;
• tunneled egress or dynamic per-Actor VM sizing;
• Kata VMM profiles that have not been validated yet;
• a stable upstream Kata checkpoint/restore API, since one is not available yet.

Before the implementation is ready to merge, requests for unsupported snapshot scopes, workload volumes, or egress gateways will return an explicit error instead of being ignored.

If this scope looks reasonable, I will split the existing POC into a small series of reviewable PRs.

Questions for maintainers

1. Is a dedicated ateom-kata backend, with the Kata shim owning the lifecycle, a good fit for the current Ateom model?
2. Is direct TTRPC communication between ateom-kata and the Kata shim an acceptable integration point?
3. Is pinning and isolating the experimental Kata checkpoint/restore API inside the adapter an acceptable compatibility strategy?
4. Is a volume-free, Full-scope, same-node MVP appropriate for the first contribution?

The POC is ready, and I would like to continue driving the implementation if maintainers agree with this direction. Could a maintainer assign this issue to me and suggest reviewers? If related work is already underway, please let me know so we can coordinate it here.

[BenTheElder]

Recent Slack discussion about the Kata shim lifecycle

I wouldn't call this recent ... July 12th is forever ago in the pace of this project, and I actually think we might want to take control of the guest as well, as we're really aiming for fast snapshot/resume and currently the default kata guest is bloated versus what we're actually using, which has impact on snapshot/restore times.

Kata should remain an internal runtime detail. Once the shim exposes suitable checkpoint/restore APIs, Substrate can use those APIs instead of driving a VMM directly.

We also didn't switch gvisor to the containerd shim, it's cheaper to drive runsc directly and hasn't been a big problem.

I'm not convinced that the containerd shim API is a better integration point than driving the hypervisor. We are going to want to customize quite a bit, e.g. #1657 or custom volumes.

We've also been exploring integrating very closely with the memory snapshot features, which I think will be essential for performance.

Is a volume-free, Full-scope, same-node MVP appropriate for the first contribution?

We need more discussion first. We will have to maintain this long term, CI is already a bit overloaded with the two existing runtimes.

I'd rather not merge an MVP without a clearer consensus that this is something the project desires.

---

Also: Keep in mind #1266 is close to landing and substantially changes what an ateom is supposed to do. #1689 seems relatively incompatible with the CNI model.