/vmm.log` 持久化 stdout/stderr。新增命令只公开受控读取能力：
-
-```text
-CLI resolve name/ID
- → core validates sandbox ownership
- → VMM backend opens owned log stream
- → tail/follow renderer copies to stdout
-```
-
-合同：
-
-- `logs SANDBOX` 输出全部；`--tail N` 从最后 N 行开始；`-f` 等待增长。
-- VMM restart 截断日志后 follower 从新文件头继续，不能卡在旧 offset。
-- stop 后仍可读；从未 start 返回明确 unavailable/not-found 诊断。
-- cancel 关闭 watcher/file；不泄漏 goroutine 或 FD。
-- CLI 不拼接 log path，`core` 不实现 tail 算法，VMM 模块拥有其日志。
-- `rm` 在最终释放 metadata/name/image pin 前删除 backend 拥有的 log dir；失败保留 `deleting` 并允许相同命令重试。
-
-实现使用同步轮询跟随受管文件，不创建 watcher goroutine。文件 inode 替换时重新打开；同一 inode 被截断时通过 size 和稳定头部签名回到 offset 0。`rm` 已在 metadata finalize 前执行 backend log cleanup，失败保留 `deleting`。
-
-## 为什么网络在 `run` 之前
-
-参考实现的 create/run 流程先 reserve identity，再配置网络，最后把 network facts 交给 VMM。start、stop 和 rm 都依赖相同事实。如果 KumaBox 现在先做无网络 `run`，之后必须再次修改：
-
-- create/run flags 和 request；
-- sandbox metadata schema；
-- launch plan 和 Cloud Hypervisor argv；
-- start rollback、stop quiesce、rm cleanup；
-- JSON 输出和 runbook。
-
-因此当前直接进入网络；网络闭环验收后实现 `run`。
-
-## 网络第一版边界
-
-- 只做 CNI backend；不做 bridge 和 hot resize。
-- 支持 0 或多个 NIC，具体默认 NIC 数在开工前再次核对 Cocoon 当前 CLI 默认。
-- MAC、IP、gateway、DNS、conflist、ifname、queue 数和 cleanup intent 持久化。
-- netns/TAP/TC/CNI 操作属于 `network` 模块；`core.SandboxService` 编排其与 catalog/VMM 的顺序。
-- `types` 保存跨模块 NetworkConfig 值；不保存 Network 接口。
-- create 在 reserve 后配网；start recover/unquiesce；stop quiesce；rm 全量 cleanup。
-- partial failure 必须可重试，不能因为 CNI DEL 失败就忘掉 NIC record。
-
-## `run` 与 `status`
-
-网络完成后，`run` 组合现有 create/start，不复制它们。随后实现一次性 `status`，将持久 state 与 VMM/process/cgroup/network observed facts 并列展示。watch/event 和节点级后台收敛留到一次性合同稳定后。
-
-## 验收
-
-每个切片必须通过 `make verify`、`make race`、`make lint`。`logs` 可在本地用受管文件验证；网络必须增加 Linux CNI runbook，并与 Cocoon 的同资源配置对比 create/start/stop/rm 行为。
diff --git a/docs/releasing.md b/docs/releasing.md
deleted file mode 100644
index 851ab7a..0000000
--- a/docs/releasing.md
+++ /dev/null
@@ -1,39 +0,0 @@
-# Releasing KumaBox
-
-A release requires both repository gates and Linux acceptance evidence.
-
-## 1. Repository gates
-
-```bash
-make verify
-make race
-make lint
-```
-
-The worktree must contain no generated binary, coverage output, or unrelated artifact. Documentation links must pass `make docs-check`.
-
-## 2. Linux acceptance
-
-Run the applicable checked-in runbooks on a Linux/KVM host. Record:
-
-- commit and version;
-- kernel and distribution;
-- Cloud Hypervisor, `mkfs.erofs`, `mkfs.ext4`, and Go versions;
-- CPU architecture and cgroup mode;
-- exact commands, exit codes, image digests, and any retained cleanup state.
-
-At minimum, a lifecycle release must cover image import/verify, create/start/exec/console/stop/restart/rm, cancellation, and host process identity checks. Network releases must also cover CNI ADD/DEL, outbound connectivity, quiesce/recover, and partial-failure retry.
-
-## 3. Build
-
-```bash
-make clean
-make build
-make agent
-```
-
-`bin/kumabox` is the host CLI, `bin/kumabox-check` is the host checker, and `bin/kumabox-agent` is the Linux guest agent.
-
-## 4. Version
-
-Build metadata is injected through `version.Version`, `version.Commit`, and `version.BuildTime`. Create a signed or annotated version tag only after the release commit and Linux evidence are final.
diff --git a/docs/runbooks/s2-oci.md b/docs/runbooks/s2-oci.md
deleted file mode 100644
index 9e8f851..0000000
--- a/docs/runbooks/s2-oci.md
+++ /dev/null
@@ -1,82 +0,0 @@
-# S2 OCI Linux 手动验收
-
-状态：2026-09-15 项目负责人已完成测试机验证并明确审批 S2 通过。按 DEC-022，本阶段不启动 VMM、不需要 KVM。
-
-## 准备
-
-需要 Go、mkfs.erofs >= 1.8、fsck.erofs、jq 和一份包含真实 regular kernel/initrd 的 KumaBox-compatible OCI 镜像。alpine 等普通容器镜像通常不满足启动合同。
-
-```bash
-make verify
-make lint
-make build
-export PATH="$PWD/bin:$PATH"
-mkfs.erofs --version
-kumabox doctor
-export KUMABOX_S2_REFERENCE='填写真实兼容镜像引用'
-KUMABOX_S2_WORK=$(mktemp -d /var/tmp/kumabox-s2.XXXXXX)
-kb() { kumabox --root-dir "$KUMABOX_S2_WORK/data" --run-dir "$KUMABOX_S2_WORK/run" --log-dir "$KUMABOX_S2_WORK/log" "$@"; }
-kb image ls --json
-```
-
-空列表必须是 `[]`。本节只写隔离的临时 root；系统 doctor 的 fix/upgrade 由负责人在验收机上显式运行。
-
-## 真转换、registry 与幂等
-
-```bash
-kb image pull "$KUMABOX_S2_REFERENCE" --platform linux/amd64
-kb image inspect "$KUMABOX_S2_REFERENCE" > "$KUMABOX_S2_WORK/first.json"
-kb image verify "$KUMABOX_S2_REFERENCE"
-find "$KUMABOX_S2_WORK/data/images/layers" -name '*.erofs' -exec fsck.erofs '{}' \;
-find "$KUMABOX_S2_WORK/data/images" -type f -exec sha256sum '{}' \; | sort > "$KUMABOX_S2_WORK/before.sha256"
-kb image pull "$KUMABOX_S2_REFERENCE" --platform linux/amd64
-kb image inspect "$KUMABOX_S2_REFERENCE" > "$KUMABOX_S2_WORK/second.json"
-find "$KUMABOX_S2_WORK/data/images" -type f -exec sha256sum '{}' \; | sort > "$KUMABOX_S2_WORK/after.sha256"
-diff "$KUMABOX_S2_WORK/before.sha256" "$KUMABOX_S2_WORK/after.sha256"
-diff "$KUMABOX_S2_WORK/first.json" "$KUMABOX_S2_WORK/second.json"
-test ! -e "$KUMABOX_S2_WORK/data/images/blobs"
-test -z "$(find "$KUMABOX_S2_WORK/data/staging/imports" -mindepth 1 -print -quit)"
-```
-
-每个 source layer 只有一份 EROFS；重复 pull 不出现转换进程，产物、digest 和 created_at 不变。inspect 包含 compressed source digest、EROFS digest、boot candidates 的 digest/size 与最终选择。
-
-## Layout/archive 与别名
-
-```bash
-kb image import tiny ./testdata/oci-layout --platform linux/amd64
-kb image verify tiny
-KUMABOX_S2_ARCHIVE="$KUMABOX_S2_WORK/tiny.bin"
-tar -C testdata/oci-layout -czf "$KUMABOX_S2_ARCHIVE" .
-kb image import tiny-alias "$KUMABOX_S2_ARCHIVE" --platform linux/amd64
-kb image inspect tiny | jq '.names'
-kb image rm tiny
-kb image verify tiny-alias
-kb image rm tiny-alias
-```
-
-gzip 通过 magic 检测，与扩展名无关。fixture 启动文件是占位数据，只验证转换/完整性；不能用来启动 VM。实际 Ubuntu 等镜像另外验证 versioned boot basename、whiteout 和 arm64 gzip kernel。
-
-## 损坏与恢复
-
-针对真实已拉取镜像：
-
-```bash
-KUMABOX_S2_LAYER=$(kb image inspect "$KUMABOX_S2_REFERENCE" | jq -r '.boot.kernel_layer | sub("^sha256:"; "")')
-KUMABOX_S2_KERNEL=$(kb image inspect "$KUMABOX_S2_REFERENCE" | jq -r '.boot.kernel_file')
-printf x >> "$KUMABOX_S2_WORK/data/images/boot/sha256/$KUMABOX_S2_LAYER/$KUMABOX_S2_KERNEL"
-kb image verify "$KUMABOX_S2_REFERENCE"; test "$?" -eq 5
-kb image pull "$KUMABOX_S2_REFERENCE" --platform linux/amd64
-kb image verify "$KUMABOX_S2_REFERENCE"
-```
-
-verify 报 ARTIFACT_CORRUPT；重拉只重建损坏 layer，恢复原摘要。缺失文件报 ARTIFACT_UNAVAILABLE（6），缺镜像为 NOT_FOUND（3），用法错误为 2。
-
-## 并发、取消与崩溃
-
-使用另一个空 root 对同一镜像并发 pull 两次；两进程都成功，允许 staging 重复转换，最终只有一份 EROFS。两边都 verify，通过后删除检查共享引用。
-
-对空 root 中慢下载/大镜像 import 发送 SIGINT/SIGTERM：进程终止下载和 mkfs.erofs，staging 清理完毕，没有半成品 image 记录。原样重试成功。
-
-在转换期间和发布/提交窗口分别 kill -9：inspect 要么 NOT_FOUND，要么完整且 verify 通过；绝不显示 importing/半成品。重试时未知最终文件重建；已提交产物校验后复用。kill -9 的旧 staging 可以保留为不可见孤儿，后续 GC 阶段回收。
-
-最后记录宿主架构、mkfs.erofs 版本、镜像完整 manifest digest、各命令退出码与产物文件列表，附到 ROADMAP 的进度日志。不要把 cache、bin 或 coverage 产物加入 Git；本目录中的规范和 runbook 应随代码提交。
diff --git a/docs/runbooks/s3-create.md b/docs/runbooks/s3-create.md
deleted file mode 100644
index 9495e24..0000000
--- a/docs/runbooks/s3-create.md
+++ /dev/null
@@ -1,110 +0,0 @@
-# S3 Sandbox Linux 验收
-
-本 runbook 验证当前无网络生命周期：create、start、exec、logs、console、stop、restart 和 rm。需要 Linux、KVM、cgroup v2、Cloud Hypervisor、`mkfs.erofs`、`mkfs.ext4`、jq，以及一份带 `overlay-v1` profile 和 `kumabox-agent` 的真实镜像。
-
-## 准备
-
-```bash
-make verify
-make lint
-make build
-make agent
-export PATH="$PWD/bin:$PATH"
-export KUMABOX_S3_IMAGE='填写真实兼容镜像引用'
-KUMABOX_S3_WORK=$(mktemp -d /var/tmp/kumabox-s3.XXXXXX)
-kb() { sudo kumabox --root-dir "$KUMABOX_S3_WORK/data" --run-dir "$KUMABOX_S3_WORK/run" --log-dir "$KUMABOX_S3_WORK/log" "$@"; }
-
-sudo kumabox doctor
-kb image pull "$KUMABOX_S3_IMAGE" --platform linux/amd64
-kb image verify "$KUMABOX_S3_IMAGE"
-test "$(kb image inspect "$KUMABOX_S3_IMAGE" | jq -r '.boot.profile')" = overlay-v1
-```
-
-## Create 和查询
-
-```bash
-kb create "$KUMABOX_S3_IMAGE" --name lifecycle --cpus 2 --memory 1GiB --storage 10GiB --json \
- | tee "$KUMABOX_S3_WORK/create.json"
-KUMABOX_S3_ID=$(jq -r '.id' "$KUMABOX_S3_WORK/create.json")
-test "$(jq -r '.state' "$KUMABOX_S3_WORK/create.json")" = created
-kb ps -a
-kb inspect lifecycle | jq -e --arg id "$KUMABOX_S3_ID" '.id == $id and .state == "created"'
-test "$(stat -c %s "$KUMABOX_S3_WORK/data/sandboxes/$KUMABOX_S3_ID/cow.raw")" = 10737418240
-blkid "$KUMABOX_S3_WORK/data/sandboxes/$KUMABOX_S3_ID/cow.raw" | grep 'TYPE="ext4"'
-```
-
-COW apparent size 为 10 GiB，实际占用应明显更小。`ps` 默认不显示 created，`ps -a` 显示完整 UUID 和列标题。
-
-## Start、exec 和 console
-
-```bash
-kb start lifecycle --json | tee "$KUMABOX_S3_WORK/start.json"
-test "$(jq -r '.state' "$KUMABOX_S3_WORK/start.json")" = running
-kb exec lifecycle -- uname -a
-kb exec lifecycle -- hostname
-echo hello | kb exec -i lifecycle -- cat
-kb exec lifecycle -- sh -c 'exit 17'; test "$?" -eq 17
-kb console lifecycle
-```
-
-console 中确认 guest 完成启动；使用 `Ctrl-]` 后 `.` 断开。检查：
-
-```bash
-cat "$KUMABOX_S3_WORK/run/sandboxes/$KUMABOX_S3_ID/process.json" | jq .
-cat "$KUMABOX_S3_WORK/run/sandboxes/$KUMABOX_S3_ID/cmdline"
-test -S "$KUMABOX_S3_WORK/run/sandboxes/$KUMABOX_S3_ID/api.sock"
-test -S "$KUMABOX_S3_WORK/run/sandboxes/$KUMABOX_S3_ID/vsock.uds"
-cat "$KUMABOX_S3_WORK/log/sandboxes/$KUMABOX_S3_ID/vmm.log"
-```
-
-## Logs、tail 和 follow
-
-```bash
-kb logs lifecycle | tee "$KUMABOX_S3_WORK/log-all.txt"
-kb logs --tail 20 lifecycle | tee "$KUMABOX_S3_WORK/log-tail.txt"
-kb logs -f lifecycle
-```
-
-确认全量输出包含启动日志，tail 不超过最后 20 行。保持 `logs -f` 运行，在另一个终端执行 `kb stop lifecycle && kb start lifecycle`；follower 应显示新一轮启动日志且不重复旧文件尾部。按 `Ctrl-C` 后命令正常退出，sandbox 继续运行。
-
-## Stop、持久数据和 restart
-
-```bash
-kb exec lifecycle -- sh -c 'echo persisted >/persist-check'
-kb stop lifecycle --json | tee "$KUMABOX_S3_WORK/stop.json"
-test "$(jq -r '.state' "$KUMABOX_S3_WORK/stop.json")" = stopped
-test ! -e "$KUMABOX_S3_WORK/run/sandboxes/$KUMABOX_S3_ID"
-kb stop lifecycle
-kb start lifecycle
-kb exec lifecycle -- cat /persist-check | grep -Fx persisted
-kb stop lifecycle
-```
-
-stop 后 VMM 进程不存在、runtime dir 被清理、对应 cgroup 为空并删除。第二次 stop 幂等成功。restart 保持相同 sandbox ID、image digest 和 COW 数据。
-
-## 删除与引用
-
-```bash
-kb image rm "$KUMABOX_S3_IMAGE"; test "$?" -eq 4
-kb rm lifecycle --json | tee "$KUMABOX_S3_WORK/remove.json"
-test "$(jq -r '.id' "$KUMABOX_S3_WORK/remove.json")" = "$KUMABOX_S3_ID"
-test ! -e "$KUMABOX_S3_WORK/data/sandboxes/$KUMABOX_S3_ID"
-test ! -e "$KUMABOX_S3_WORK/log/sandboxes/$KUMABOX_S3_ID"
-kb image rm "$KUMABOX_S3_IMAGE"
-test "$(kb ps -a --json)" = "[]"
-```
-
-## 恢复与安全边界
-
-至少验证：
-
-1. start 过程中发送 SIGINT，重试 start 能收敛为唯一 VMM；
-2. running 时 kill VMM，stop 不向复用 PID 的无关进程发信号；
-3. stop 在 TERM 等待阶段中断，重试继续 `stopping`；
-4. rm 清理中断，重试继续 `deleting`；
-5. 破坏 process.json 的 boot ID、binary 或 socket 后，stop 返回冲突且不发送信号；
-6. agent 未启动时 exec 有界失败，VMM 保持可 inspect/stop；
-7. logs follow 在 truncate/reopen 后继续，取消后无残留进程或 FD；
-8. 非 TTY 重定向无 ANSI 控制字符，JSON 保持缩进。
-
-记录 commit、内核、架构、Cloud Hypervisor、cgroup mode、formatter 版本、完整 image digest、每个命令退出码和失败后的持久状态。
diff --git a/scripts/check-doc-links.sh b/scripts/check-doc-links.sh
deleted file mode 100755
index a039479..0000000
--- a/scripts/check-doc-links.sh
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env bash
-
-set -euo pipefail
-
-failed=0
-while IFS= read -r document; do
- while IFS= read -r markdown_link; do
- target="${markdown_link#](}"
- target="${target%)}"
- target="${target%%#*}"
- case "${target}" in
- ""|http://*|https://*|mailto:*) continue ;;
- esac
- if [[ ! -e "$(dirname "${document}")/${target}" ]]; then
- printf 'broken Markdown link: %s -> %s\n' "${document}" "${target}" >&2
- failed=1
- fi
- done < <(grep -Eo '\]\([^)]+(\.md|LICENSE)(#[^)]*)?\)' "${document}" || true)
-done < <(git ls-files '*.md')
-
-exit "${failed}"
From 50ce5761f468900c9c0622f233accb8e10f74fcb Mon Sep 17 00:00:00 2001
From: YoungLH <974840768@qq.com>
Date: Tue, 22 Sep 2026 09:41:24 +0800
Subject: [PATCH 38/45] feat: add CNI network foundation
---
config/config.go | 83 +++++-
config/config_test.go | 25 ++
core/images.go | 4 +-
go.mod | 8 +-
go.sum | 38 ++-
metadata/sqlite/store.go | 21 +-
metadata/sqlite/store_test.go | 52 ++++
network/cni/cni.go | 388 ++++++++++++++++++++++++++
network/cni/cni_test.go | 275 +++++++++++++++++++
network/cni/lifecycle.go | 494 ++++++++++++++++++++++++++++++++++
network/cni/platform_linux.go | 243 +++++++++++++++++
network/cni/platform_other.go | 36 +++
network/network.go | 103 +++++++
network/network_test.go | 37 +++
types/network.go | 152 +++++++++++
types/network_test.go | 31 +++
16 files changed, 1969 insertions(+), 21 deletions(-)
create mode 100644 network/cni/cni.go
create mode 100644 network/cni/cni_test.go
create mode 100644 network/cni/lifecycle.go
create mode 100644 network/cni/platform_linux.go
create mode 100644 network/cni/platform_other.go
create mode 100644 network/network.go
create mode 100644 network/network_test.go
create mode 100644 types/network.go
create mode 100644 types/network_test.go
diff --git a/config/config.go b/config/config.go
index f46190f..7c3e66f 100644
--- a/config/config.go
+++ b/config/config.go
@@ -6,6 +6,8 @@ package config
import (
"errors"
"fmt"
+ "net"
+ "path/filepath"
"runtime"
"strings"
"time"
@@ -29,6 +31,8 @@ type Config struct {
Metadata Metadata `mapstructure:"metadata"`
// Sandbox controls writable disk preparation and compensation.
Sandbox Sandbox `mapstructure:"sandbox"`
+ // Network controls host CNI discovery and lifecycle recovery.
+ Network Network `mapstructure:"network"`
// VMM selects and configures process backends.
VMM VMM `mapstructure:"vmm"`
}
@@ -65,6 +69,57 @@ type Sandbox struct {
CleanupTimeout time.Duration `mapstructure:"cleanup_timeout"`
}
+// Network contains host networking policy shared by provider implementations.
+type Network struct {
+ // CNI locates network configuration and plugin executables installed by the
+ // host administrator.
+ CNI CNI `mapstructure:"cni"`
+ // DNS is a comma- or semicolon-separated list injected into guest network
+ // configuration by the boot protocol.
+ DNS string `mapstructure:"dns"`
+ // Scope is an optional two-character installation identifier used in host
+ // network namespace names.
+ Scope string `mapstructure:"scope"`
+ // CleanupTimeout bounds detached compensation after caller cancellation.
+ CleanupTimeout time.Duration `mapstructure:"cleanup_timeout"`
+}
+
+// CNI contains host-owned CNI discovery paths.
+type CNI struct {
+ // ConfDir contains .conflist network definitions.
+ ConfDir string `mapstructure:"conf_dir"`
+ // BinDir contains CNI plugin executables.
+ BinDir string `mapstructure:"bin_dir"`
+}
+
+// DNSServers parses and validates the configured guest DNS server list.
+func (n Network) DNSServers() ([]string, error) {
+ if strings.TrimSpace(n.DNS) == "" {
+ return nil, nil
+ }
+ var result []string
+ for value := range strings.SplitSeq(strings.ReplaceAll(n.DNS, ";", ","), ",") {
+ value = strings.TrimSpace(value)
+ if value == "" {
+ continue
+ }
+ if net.ParseIP(value) == nil {
+ return nil, fmt.Errorf("invalid DNS server %q", value)
+ }
+ result = append(result, value)
+ }
+ return result, nil
+}
+
+// NamespacePrefix returns the installation-specific prefix for named network
+// namespaces. An empty scope preserves the readable product default.
+func (n Network) NamespacePrefix() string {
+ if n.Scope == "" {
+ return "kumabox-"
+ }
+ return n.Scope + "-"
+}
+
// VMM contains backend selection and host process policy.
type VMM struct {
// Default selects the backend for newly created sandboxes.
@@ -98,6 +153,10 @@ func Default() Config {
},
Metadata: Metadata{BusyTimeout: 50 * time.Millisecond, RetryLimit: 5 * time.Second},
Sandbox: Sandbox{Ext4Binary: "mkfs.ext4", CleanupTimeout: 10 * time.Second},
+ Network: Network{
+ CNI: CNI{ConfDir: "/etc/cni/net.d", BinDir: "/opt/cni/bin"},
+ DNS: "8.8.8.8,1.1.1.1", CleanupTimeout: 30 * time.Second,
+ },
VMM: VMM{
Default: types.VMMCloudHypervisor, CgroupParent: "/sys/fs/cgroup/kumabox.slice",
CloudHypervisor: CloudHypervisor{
@@ -130,6 +189,25 @@ func (c *Config) Validate() error {
if strings.TrimSpace(c.Sandbox.Ext4Binary) == "" || c.Sandbox.CleanupTimeout <= 0 {
return errors.New("sandbox requires an ext4 binary and positive cleanup timeout")
}
+ if !filepath.IsAbs(c.Network.CNI.ConfDir) || !filepath.IsAbs(c.Network.CNI.BinDir) {
+ return errors.New("network CNI configuration and binary directories must be absolute")
+ }
+ if c.Network.CleanupTimeout <= 0 {
+ return errors.New("network cleanup timeout must be positive")
+ }
+ if _, err := c.Network.DNSServers(); err != nil {
+ return fmt.Errorf("network DNS: %w", err)
+ }
+ if c.Network.Scope != "" {
+ if len(c.Network.Scope) != 2 {
+ return errors.New("network scope must contain exactly two ASCII letters or digits")
+ }
+ for _, character := range c.Network.Scope {
+ if (character < 'a' || character > 'z') && (character < 'A' || character > 'Z') && (character < '0' || character > '9') {
+ return errors.New("network scope must contain exactly two ASCII letters or digits")
+ }
+ }
+ }
if err := c.VMM.Default.Validate(); err != nil {
return fmt.Errorf("vmm default: %w", err)
}
@@ -167,7 +245,10 @@ func NewLoader() *Loader {
"images.boot_size": defaults.Images.BootSize, "images.archive_size": defaults.Images.ArchiveSize,
"metadata.busy_timeout": defaults.Metadata.BusyTimeout, "metadata.retry_limit": defaults.Metadata.RetryLimit,
"sandbox.ext4_binary": defaults.Sandbox.Ext4Binary, "sandbox.cleanup_timeout": defaults.Sandbox.CleanupTimeout,
- "vmm.default": defaults.VMM.Default, "vmm.cgroup_parent": defaults.VMM.CgroupParent,
+ "network.cni.conf_dir": defaults.Network.CNI.ConfDir, "network.cni.bin_dir": defaults.Network.CNI.BinDir,
+ "network.dns": defaults.Network.DNS, "network.scope": defaults.Network.Scope,
+ "network.cleanup_timeout": defaults.Network.CleanupTimeout,
+ "vmm.default": defaults.VMM.Default, "vmm.cgroup_parent": defaults.VMM.CgroupParent,
"vmm.cloud_hypervisor.binary": defaults.VMM.CloudHypervisor.Binary,
"vmm.cloud_hypervisor.startup_timeout": defaults.VMM.CloudHypervisor.StartupTimeout,
"vmm.cloud_hypervisor.stop_grace": defaults.VMM.CloudHypervisor.StopGrace,
diff --git a/config/config_test.go b/config/config_test.go
index 5e106e8..f09381d 100644
--- a/config/config_test.go
+++ b/config/config_test.go
@@ -117,3 +117,28 @@ func TestValidateRejectsOverlappingRoots(t *testing.T) {
t.Fatal("Validate() accepted overlapping roots")
}
}
+
+func TestNetworkConfigParsesDNSAndScope(t *testing.T) {
+ config := Default()
+ config.Network.DNS = "10.0.0.2; 2001:4860:4860::8888"
+ config.Network.Scope = "k1"
+ if err := config.Validate(); err != nil {
+ t.Fatal(err)
+ }
+ servers, err := config.Network.DNSServers()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(servers) != 2 || servers[0] != "10.0.0.2" || config.Network.NamespacePrefix() != "k1-" {
+ t.Fatalf("servers=%v prefix=%q", servers, config.Network.NamespacePrefix())
+ }
+ config.Network.Scope = "unsafe/"
+ if err := config.Validate(); err == nil {
+ t.Fatal("invalid network scope was accepted")
+ }
+ config = Default()
+ config.Network.DNS = "not-an-address"
+ if err := config.Validate(); err == nil {
+ t.Fatal("invalid DNS server was accepted")
+ }
+}
diff --git a/core/images.go b/core/images.go
index 971187a..6855e9f 100644
--- a/core/images.go
+++ b/core/images.go
@@ -21,6 +21,7 @@ import (
"github.com/kumabox/kumabox/images/source"
"github.com/kumabox/kumabox/metadata"
"github.com/kumabox/kumabox/metadata/sqlite"
+ networkcni "github.com/kumabox/kumabox/network/cni"
sandboxcatalog "github.com/kumabox/kumabox/sandbox/catalog"
"github.com/kumabox/kumabox/types"
)
@@ -124,5 +125,6 @@ func NewRegistrySource(reference string) (images.Source, string, error) {
// the database shape without an explicit migration.
func metadataCollections() []metadata.Collection {
result := catalog.Collections()
- return append(result, sandboxcatalog.Collections()...)
+ result = append(result, sandboxcatalog.Collections()...)
+ return append(result, networkcni.Collections()...)
}
diff --git a/go.mod b/go.mod
index 0405991..c16a9c9 100644
--- a/go.mod
+++ b/go.mod
@@ -3,6 +3,8 @@ module github.com/kumabox/kumabox
go 1.24.4
require (
+ github.com/containernetworking/cni v1.3.0
+ github.com/containernetworking/plugins v1.9.1
github.com/gofrs/flock v0.13.0
github.com/google/go-containerregistry v0.20.6
github.com/klauspost/compress v1.18.0
@@ -12,6 +14,8 @@ require (
github.com/spf13/cobra v1.10.2
github.com/spf13/pflag v1.0.10
github.com/spf13/viper v1.21.0
+ github.com/vishvananda/netlink v1.3.1
+ github.com/vishvananda/netns v0.0.5
golang.org/x/sync v0.16.0
modernc.org/sqlite v1.38.2
)
@@ -27,7 +31,7 @@ require (
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
- github.com/mdlayher/socket v0.4.1 // indirect
+ github.com/mdlayher/socket v0.5.1 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
@@ -44,7 +48,7 @@ require (
github.com/vbatts/tar-split v0.12.1 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b // indirect
- golang.org/x/net v0.9.0 // indirect
+ golang.org/x/net v0.43.0 // indirect
golang.org/x/sys v0.37.0 // indirect
golang.org/x/text v0.28.0 // indirect
modernc.org/libc v1.66.3 // indirect
diff --git a/go.sum b/go.sum
index e4426a6..1cf5386 100644
--- a/go.sum
+++ b/go.sum
@@ -1,7 +1,13 @@
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
+github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
+github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
github.com/containerd/stargz-snapshotter/estargz v0.16.3 h1:7evrXtoh1mSbGj/pfRccTampEyKpjpOnS3CyiV1Ebr8=
github.com/containerd/stargz-snapshotter/estargz v0.16.3/go.mod h1:uyr4BfYfOj3G9WBVE8cOlQmXAbPN9VEQpBBeJIuOipU=
+github.com/containernetworking/cni v1.3.0 h1:v6EpN8RznAZj9765HhXQrtXgX+ECGebEYEmnuFjskwo=
+github.com/containernetworking/cni v1.3.0/go.mod h1:Bs8glZjjFfGPHMw6hQu82RUgEPNGEaBb9KS5KtNMnJ4=
+github.com/containernetworking/plugins v1.9.1 h1:8oU6WsIsU3bpnNZuvHp74a6cE1MJwbj2P7s4/yTUNlA=
+github.com/containernetworking/plugins v1.9.1/go.mod h1:fj7kS55qg3o/RgS+WGsF3+ZxwIImMPusQZKzBpcSr4c=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY=
github.com/creack/pty v1.1.18/go.mod h1:MOBLtS5ELjhRRrroQr9kyvTxUAFNvYEK993ew/Vr4O4=
@@ -20,6 +26,10 @@ github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHk
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
+github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/gofrs/flock v0.13.0 h1:95JolYOvGMqeH31+FC7D2+uULf6mG61mEZ/A8dRYMzw=
@@ -28,8 +38,8 @@ github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/go-containerregistry v0.20.6 h1:cvWX87UxxLgaH76b4hIvya6Dzz9qHB31qAwjAohdSTU=
github.com/google/go-containerregistry v0.20.6/go.mod h1:T0x8MuoAoKX/873bkeSfLD2FAkwCDf9/HZgsFJ02E2Y=
-github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
-github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
+github.com/google/pprof v0.0.0-20250820193118-f64d9cf942d6 h1:EEHtgt9IwisQ2AZ4pIsMjahcegHh6rmhqxzIRQIyepY=
+github.com/google/pprof v0.0.0-20250820193118-f64d9cf942d6/go.mod h1:I6V7YzU0XDpsHqbsyrghnFZLO1gwK6NPTNvmetQIk9U=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
@@ -42,8 +52,8 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
-github.com/mdlayher/socket v0.4.1 h1:eM9y2/jlbs1M615oshPQOHZzj6R6wMT7bX5NPiQvn2U=
-github.com/mdlayher/socket v0.4.1/go.mod h1:cAqeGjoufqdxWkD7DkpyS+wcefOtmu5OQ8KuoJGIReA=
+github.com/mdlayher/socket v0.5.1 h1:VZaqt6RkGkt2OE9l3GcC6nZkqD3xKeQLyfleW/uBcos=
+github.com/mdlayher/socket v0.5.1/go.mod h1:TjPLHI1UgwEv5J1B5q0zTZq12A/6H7nKmtTanQE37IQ=
github.com/mdlayher/vsock v1.2.1 h1:pC1mTJTvjo1r9n9fbm7S1j04rCgCzhCOS5DY0zqHlnQ=
github.com/mdlayher/vsock v1.2.1/go.mod h1:NRfCibel++DgeMD8z/hP+PPTjlNJsdPOmxcnENvE+SE=
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
@@ -52,6 +62,10 @@ github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ=
github.com/moby/term v0.5.2/go.mod h1:d3djjFCrjnB+fl8NJux+EJzu0msscUP+f8it8hPkFLc=
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
+github.com/onsi/ginkgo/v2 v2.25.1 h1:Fwp6crTREKM+oA6Cz4MsO8RhKQzs2/gOIVOUscMAfZY=
+github.com/onsi/ginkgo/v2 v2.25.1/go.mod h1:ppTWQ1dh9KM/F1XgpeRqelR+zHVwV81DGRSDnFxK7Sk=
+github.com/onsi/gomega v1.38.1 h1:FaLA8GlcpXDwsb7m0h2A9ew2aTk3vnZMlzFgg5tz/pk=
+github.com/onsi/gomega v1.38.1/go.mod h1:LfcV8wZLvwcYRwPiJysphKAEsmcFnLMK/9c+PjvlX8g=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
@@ -92,25 +106,33 @@ github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/vbatts/tar-split v0.12.1 h1:CqKoORW7BUWBe7UL/iqTVvkTBOF8UvOMKOIZykxnnbo=
github.com/vbatts/tar-split v0.12.1/go.mod h1:eF6B6i6ftWQcDqEn3/iGFRFRo8cBIMSJVOpnNdfTMFA=
+github.com/vishvananda/netlink v1.3.1 h1:3AEMt62VKqz90r0tmNhog0r/PpWKmrEShJU0wJW6bV0=
+github.com/vishvananda/netlink v1.3.1/go.mod h1:ARtKouGSTGchR8aMwmkzC0qiNPrrWO5JS/XMVl45+b4=
+github.com/vishvananda/netns v0.0.5 h1:DfiHV+j8bA32MFM7bfEunvT8IAqQ/NzSJHtcmW5zdEY=
+github.com/vishvananda/netns v0.0.5/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM=
+go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs=
+go.uber.org/automaxprocs v1.6.0/go.mod h1:ifeIMSnPZuznNm6jmdzmU3/bfk01Fe2fotchwEFJ8r8=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b h1:M2rDM6z3Fhozi9O7NWsxAkg/yqS/lQJ6PmkyIV3YP+o=
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b/go.mod h1:3//PLf8L/X+8b4vuAfHzxeRUl04Adcb341+IGKfnqS8=
golang.org/x/mod v0.26.0 h1:EGMPT//Ezu+ylkCijjPc+f4Aih7sZvaAr+O3EHBxvZg=
golang.org/x/mod v0.26.0/go.mod h1:/j6NAhSk8iQ723BGAUyoAcn7SlD7s15Dp9Nd/SfeaFQ=
-golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM=
-golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns=
+golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
+golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.37.0 h1:fdNQudmxPjkdUTPnLn5mdQv7Zwvbvpaxqs831goi9kQ=
golang.org/x/sys v0.37.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
-golang.org/x/tools v0.35.0 h1:mBffYraMEf7aa0sB+NuKnuCy8qI/9Bughn8dC2Gu5r0=
-golang.org/x/tools v0.35.0/go.mod h1:NKdj5HkL/73byiZSJjqJgKn3ep7KjFkBOkR/Hps3VPw=
+golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
+golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
diff --git a/metadata/sqlite/store.go b/metadata/sqlite/store.go
index 482aa42..3e7e4e5 100644
--- a/metadata/sqlite/store.go
+++ b/metadata/sqlite/store.go
@@ -28,7 +28,7 @@ const (
// applicationID distinguishes KumaBox metadata from unrelated SQLite files.
applicationID = 0x4B554D41
// schemaVersion identifies the current application collection contract.
- schemaVersion = 2
+ schemaVersion = 3
// firstSchemaVersion is the oldest metadata version with an in-place migration.
firstSchemaVersion = 1
// initLockName serializes schema initialization across processes in this directory.
@@ -231,8 +231,8 @@ func initialize(ctx context.Context, path string, collections []metadata.Collect
switch version {
case schemaVersion:
return nil
- case firstSchemaVersion:
- return migrateVersionOne(ctx, db, collections)
+ case 1, 2:
+ return migrateCollections(ctx, db, collections, version)
default:
return errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, fmt.Errorf("metadata schema version %d is unsupported; this binary supports versions %d through %d", version, firstSchemaVersion, schemaVersion))
}
@@ -268,14 +268,17 @@ func initialize(ctx context.Context, path string, collections []metadata.Collect
return commit(ctx, tx)
}
-// migrateVersionOne adds the collections introduced with sandbox management
-// and publishes version 2 only after every declaration is durable. Version 1
-// already uses the same collections and records tables, so record payloads and
-// image artifacts remain unchanged.
+// migrateCollections adds collections introduced after the stored version and
+// publishes the current version only after every declaration is durable. All
+// supported versions use the same collections and records tables, so existing
+// module payloads remain unchanged.
//
-// BEGIN IMMEDIATE -> register missing collections -> user_version=2 -> COMMIT
+// BEGIN IMMEDIATE -> register missing collections -> publish version -> COMMIT
// \---------------- any failure: ROLLBACK -----------------/
-func migrateVersionOne(ctx context.Context, db *sql.DB, collections []metadata.Collection) (returnErr error) {
+func migrateCollections(ctx context.Context, db *sql.DB, collections []metadata.Collection, from int) (returnErr error) {
+ if from < firstSchemaVersion || from >= schemaVersion {
+ return errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, fmt.Errorf("cannot migrate metadata schema version %d", from))
+ }
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return mapError(err)
diff --git a/metadata/sqlite/store_test.go b/metadata/sqlite/store_test.go
index f70eb7e..5dd9783 100644
--- a/metadata/sqlite/store_test.go
+++ b/metadata/sqlite/store_test.go
@@ -167,6 +167,58 @@ func TestStoreMigratesVersionOneAndPreservesRecords(t *testing.T) {
}
}
+func TestStoreMigratesVersionTwoAndPreservesSandboxRecords(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "meta.db")
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ statements := []string{
+ "CREATE TABLE collections (name TEXT NOT NULL PRIMARY KEY)",
+ "CREATE TABLE records (collection TEXT NOT NULL, id TEXT NOT NULL, data BLOB NOT NULL, PRIMARY KEY(collection, id), FOREIGN KEY(collection) REFERENCES collections(name))",
+ fmt.Sprintf("PRAGMA application_id = %d", applicationID),
+ "PRAGMA user_version = 2",
+ "INSERT INTO collections(name) VALUES ('sandboxes')",
+ "INSERT INTO records(collection,id,data) VALUES ('sandboxes','sandbox-id',x'6b656570')",
+ }
+ for _, statement := range statements {
+ if _, err := db.Exec(statement); err != nil {
+ _ = db.Close()
+ t.Fatal(err)
+ }
+ }
+ if err := db.Close(); err != nil {
+ t.Fatal(err)
+ }
+
+ store, err := Open(t.Context(), path, []metadata.Collection{"sandboxes", "network_records"}, DefaultOptions())
+ if err != nil {
+ t.Fatalf("Open migrated v2 database: %v", err)
+ }
+ t.Cleanup(func() {
+ if err := store.Close(); err != nil {
+ t.Error(err)
+ }
+ })
+ if err := store.View(t.Context(), func(reader metadata.Reader) error {
+ value, exists, err := reader.Get(t.Context(), "sandboxes", "sandbox-id")
+ if err != nil {
+ return err
+ }
+ if !exists || string(value) != "keep" {
+ return fmt.Errorf("sandbox record = %q, %t", value, exists)
+ }
+ return nil
+ }); err != nil {
+ t.Fatal(err)
+ }
+ if err := store.Update(t.Context(), func(writer metadata.Writer) error {
+ return writer.Put(t.Context(), "network_records", "network-id", []byte("network"))
+ }); err != nil {
+ t.Fatalf("write migrated network collection: %v", err)
+ }
+}
+
func TestStoreMigrationFailureRollsBackVersionAndCollections(t *testing.T) {
path := filepath.Join(t.TempDir(), "meta.db")
writeVersionOneDatabase(t, path, "CREATE TABLE collections (name TEXT NOT NULL PRIMARY KEY CHECK(name <> 'sandboxes'))")
diff --git a/network/cni/cni.go b/network/cni/cni.go
new file mode 100644
index 0000000..15b61a8
--- /dev/null
+++ b/network/cni/cni.go
@@ -0,0 +1,388 @@
+// Package cni implements network.Provider with CNI plugins, one named network
+// namespace per sandbox, and TAP devices connected through traffic-control
+// redirects.
+package cni
+
+import (
+ "cmp"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "maps"
+ "path/filepath"
+ "slices"
+ "strings"
+ "time"
+
+ "github.com/containernetworking/cni/libcni"
+ cnitypes "github.com/containernetworking/cni/pkg/types"
+
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/metadata"
+ "github.com/kumabox/kumabox/network"
+ "github.com/kumabox/kumabox/storage"
+ "github.com/kumabox/kumabox/types"
+)
+
+// CollectionRecords stores one crash-recoverable aggregate per sandbox.
+const CollectionRecords metadata.Collection = "network_records"
+
+const (
+ recordVersion = 1
+ defaultTAPPrefix = "tap"
+ namedNamespaceDir = "/var/run/netns"
+)
+
+// Collections declares the metadata owned by the CNI adapter.
+func Collections() []metadata.Collection { return []metadata.Collection{CollectionRecords} }
+
+// Options contains immutable host paths and cleanup policy for one provider.
+type Options struct {
+ // ConfDir contains host-installed .conflist files.
+ ConfDir string
+ // BinDir contains host-installed CNI plugin executables.
+ BinDir string
+ // CacheDir is managed persistent state used by the CNI library.
+ CacheDir string
+ // NamespacePrefix separates named namespaces owned by this installation.
+ NamespacePrefix string
+ // CleanupTimeout bounds rollback after caller cancellation.
+ CleanupTimeout time.Duration
+}
+
+// Validate rejects ambiguous or unsafe provider configuration.
+func (o Options) Validate() error {
+ for name, path := range map[string]string{"configuration": o.ConfDir, "binary": o.BinDir, "cache": o.CacheDir} {
+ if !filepath.IsAbs(path) {
+ return fmt.Errorf("CNI %s directory must be absolute", name)
+ }
+ }
+ if o.NamespacePrefix == "" || len(o.NamespacePrefix) > 32 || strings.ContainsAny(o.NamespacePrefix, "/\x00") {
+ return errors.New("CNI namespace prefix is invalid")
+ }
+ for _, character := range o.NamespacePrefix {
+ if (character < 'a' || character > 'z') && (character < 'A' || character > 'Z') &&
+ (character < '0' || character > '9') && character != '-' && character != '_' {
+ return errors.New("CNI namespace prefix is invalid")
+ }
+ }
+ if o.CleanupTimeout <= 0 {
+ return errors.New("CNI cleanup timeout must be positive")
+ }
+ return nil
+}
+
+// pluginRuntime executes one parsed CNI network list. The narrow seam keeps lifecycle
+// recovery testable without requiring root privileges or plugin binaries.
+type pluginRuntime interface {
+ AddNetworkList(context.Context, *libcni.NetworkConfigList, *libcni.RuntimeConf) (cnitypes.Result, error)
+ DelNetworkList(context.Context, *libcni.NetworkConfigList, *libcni.RuntimeConf) error
+}
+
+// platform owns Linux namespace, link, TAP, and traffic-control operations.
+type platform interface {
+ EnsureNamespace(string, string) (bool, error)
+ RemoveNamespace(context.Context, string) error
+ NamespaceExists(string) error
+ SetupRedirect(string, string, string, int, string) (string, error)
+ DeleteTAP(string, string) error
+ SetLinkState(string, []string, bool) error
+ VerifyTAP(string, string) error
+}
+
+// Provider is the CNI implementation of network.Provider.
+type Provider struct {
+ options Options
+ store metadata.Store
+ lists map[string]*libcni.NetworkConfigList
+ defaultName string
+ runtime pluginRuntime
+ platform platform
+ loadErr error
+}
+
+var _ network.Provider = (*Provider)(nil)
+
+// New creates a provider. Conflist discovery is intentionally best-effort so a
+// command can still open metadata and report or retry retained cleanup state
+// after host configuration has temporarily disappeared.
+func New(options Options, store metadata.Store) (*Provider, error) {
+ if err := options.Validate(); err != nil {
+ return nil, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ if store == nil {
+ return nil, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, errors.New("CNI metadata store is required"))
+ }
+ if err := storage.EnsureDir(options.CacheDir); err != nil {
+ return nil, errdefs.New(errdefs.ClassUnavailable, errdefs.CodeArtifactUnavailable, fmt.Errorf("create CNI cache: %w", err))
+ }
+ provider := &Provider{
+ options: options, store: store, platform: newPlatform(),
+ lists: make(map[string]*libcni.NetworkConfigList),
+ }
+ lists, defaultName, err := loadConfLists(options.ConfDir)
+ if err != nil {
+ provider.loadErr = err
+ return provider, nil
+ }
+ provider.lists = lists
+ provider.defaultName = defaultName
+ provider.runtime = libcni.NewCNIConfigWithCacheDir([]string{options.BinDir}, options.CacheDir, nil)
+ return provider, nil
+}
+
+// Type returns the durable provider identity.
+func (*Provider) Type() types.NetworkBackend { return types.NetworkBackendCNI }
+
+// confList resolves an explicit conflist name or the deterministic first file.
+func (p *Provider) confList(name string) (*libcni.NetworkConfigList, error) {
+ if p == nil || p.runtime == nil || len(p.lists) == 0 {
+ if p != nil && p.loadErr != nil {
+ return nil, fmt.Errorf("%w: load .conflist files from %s: %w", network.ErrNotConfigured, p.options.ConfDir, p.loadErr)
+ }
+ return nil, fmt.Errorf("%w: no .conflist files in %s", network.ErrNotConfigured, p.options.ConfDir)
+ }
+ resolved := cmp.Or(name, p.defaultName)
+ list, exists := p.lists[resolved]
+ if !exists {
+ return nil, fmt.Errorf("CNI network %q not found; available networks: %s", resolved, strings.Join(slices.Sorted(maps.Keys(p.lists)), ", "))
+ }
+ return list, nil
+}
+
+// loadConfLists loads only explicit CNI list files. A single-plugin .conf is
+// not silently treated as an application network contract.
+func loadConfLists(dir string) (map[string]*libcni.NetworkConfigList, string, error) {
+ files, err := libcni.ConfFiles(dir, []string{".conflist"})
+ if err != nil {
+ return nil, "", err
+ }
+ if len(files) == 0 {
+ return nil, "", fmt.Errorf("no .conflist files in %s", dir)
+ }
+ slices.Sort(files)
+ result := make(map[string]*libcni.NetworkConfigList, len(files))
+ defaultName := ""
+ for _, path := range files {
+ list, err := libcni.ConfListFromFile(path)
+ if err != nil {
+ return nil, "", fmt.Errorf("parse %s: %w", path, err)
+ }
+ if _, exists := result[list.Name]; exists {
+ return nil, "", fmt.Errorf("CNI network name %q is declared more than once", list.Name)
+ }
+ result[list.Name] = list
+ if defaultName == "" {
+ defaultName = list.Name
+ }
+ }
+ return result, defaultName, nil
+}
+
+type (
+ recordPhase string
+ interfacePhase string
+)
+
+const (
+ phasePreparing recordPhase = "preparing"
+ phaseReady recordPhase = "ready"
+ phaseDeleting recordPhase = "deleting"
+
+ interfaceStaged interfacePhase = "staged"
+ interfaceAdding interfacePhase = "adding"
+ interfaceReady interfacePhase = "ready"
+)
+
+// recordData is an adapter-owned cleanup journal. The aggregate is written
+// before namespace creation, and each NIC reaches adding before plugin code can
+// produce host-side effects.
+type recordData struct {
+ Version int `json:"version"`
+ SandboxID string `json:"sandbox_id"`
+ Network string `json:"network,omitempty"`
+ NamespaceName string `json:"namespace_name"`
+ NamespacePath string `json:"namespace_path"`
+ Phase recordPhase `json:"phase"`
+ Interfaces []interfaceData `json:"interfaces"`
+}
+
+type interfaceData struct {
+ Index int `json:"index"`
+ Name string `json:"name"`
+ TAP string `json:"tap"`
+ Phase interfacePhase `json:"phase"`
+ MAC string `json:"mac,omitempty"`
+ Queues int `json:"queues"`
+ QueueSize int `json:"queue_size"`
+ IPv4 *ipv4Data `json:"ipv4,omitempty"`
+}
+
+type ipv4Data struct {
+ Address string `json:"address"`
+ Gateway string `json:"gateway,omitempty"`
+ Prefix int `json:"prefix"`
+}
+
+func (p *Provider) namespace(id types.SandboxID) (string, string) {
+ name := p.options.NamespacePrefix + id.String()
+ return name, filepath.Join(namedNamespaceDir, name)
+}
+
+func (p *Provider) view(ctx context.Context, id types.SandboxID) (*recordData, error) {
+ var result *recordData
+ err := p.store.View(ctx, func(reader metadata.Reader) error {
+ raw, exists, err := reader.Get(ctx, CollectionRecords, id.String())
+ if err != nil || !exists {
+ return err
+ }
+ result, err = decodeRecord(raw)
+ return err
+ })
+ return result, err
+}
+
+func (p *Provider) update(ctx context.Context, id types.SandboxID, mutate func(*recordData) (*recordData, error)) error {
+ return p.store.Update(ctx, func(writer metadata.Writer) error {
+ raw, exists, err := writer.Get(ctx, CollectionRecords, id.String())
+ if err != nil {
+ return err
+ }
+ var record *recordData
+ if exists {
+ record, err = decodeRecord(raw)
+ if err != nil {
+ return err
+ }
+ }
+ next, err := mutate(record)
+ if err != nil {
+ return err
+ }
+ if next == nil {
+ return writer.Delete(ctx, CollectionRecords, id.String())
+ }
+ if next.SandboxID != id.String() {
+ return errors.New("network record ID differs from its metadata key")
+ }
+ return putRecord(ctx, writer, next)
+ })
+}
+
+func putRecord(ctx context.Context, writer metadata.Writer, record *recordData) error {
+ if err := validateRecord(record); err != nil {
+ return err
+ }
+ raw, err := json.Marshal(record)
+ if err != nil {
+ return err
+ }
+ return writer.Put(ctx, CollectionRecords, record.SandboxID, raw)
+}
+
+func decodeRecord(raw []byte) (*recordData, error) {
+ var record recordData
+ if err := json.Unmarshal(raw, &record); err != nil {
+ return nil, corrupt(err)
+ }
+ if err := validateRecord(&record); err != nil {
+ return nil, corrupt(err)
+ }
+ return &record, nil
+}
+
+func validateRecord(record *recordData) error {
+ if record == nil {
+ return errors.New("network record is missing")
+ }
+ if record.Version != recordVersion {
+ return fmt.Errorf("network record version %d is unsupported", record.Version)
+ }
+ if _, err := types.ParseSandboxID(record.SandboxID); err != nil {
+ return err
+ }
+ if record.NamespaceName == "" || record.NamespacePath != filepath.Join(namedNamespaceDir, record.NamespaceName) {
+ return errors.New("network record namespace is invalid")
+ }
+ if len(record.Interfaces) > 0 && record.Network == "" {
+ return errors.New("network record with interfaces requires a conflist name")
+ }
+ switch record.Phase {
+ case phasePreparing, phaseReady, phaseDeleting:
+ default:
+ return fmt.Errorf("network record phase %q is invalid", record.Phase)
+ }
+ seen := make(map[int]struct{}, len(record.Interfaces))
+ for _, item := range record.Interfaces {
+ if item.Index < 0 || item.Name != interfaceName(item.Index) || item.TAP == "" || item.Queues < 2 || item.Queues%2 != 0 || item.QueueSize <= 0 {
+ return fmt.Errorf("network record interface %d is invalid", item.Index)
+ }
+ switch item.Phase {
+ case interfaceStaged, interfaceAdding:
+ case interfaceReady:
+ if _, err := item.toType(record.Network); err != nil {
+ return err
+ }
+ default:
+ return fmt.Errorf("network record interface phase %q is invalid", item.Phase)
+ }
+ if _, exists := seen[item.Index]; exists {
+ return fmt.Errorf("network record interface index %d is duplicated", item.Index)
+ }
+ seen[item.Index] = struct{}{}
+ }
+ if record.Phase == phaseReady {
+ for _, item := range record.Interfaces {
+ if item.Phase != interfaceReady {
+ return errors.New("ready network record contains an incomplete interface")
+ }
+ }
+ }
+ return nil
+}
+
+func (item interfaceData) toType(networkName string) (types.NetworkInterface, error) {
+ result := types.NetworkInterface{
+ Index: item.Index, Name: item.Name, TAP: item.TAP, MAC: item.MAC,
+ Queues: item.Queues, QueueSize: item.QueueSize, Network: networkName,
+ }
+ if item.IPv4 != nil {
+ result.IPv4 = &types.IPv4Config{Address: item.IPv4.Address, Gateway: item.IPv4.Gateway, Prefix: item.IPv4.Prefix}
+ }
+ return result, result.Validate()
+}
+
+func fromType(value types.NetworkInterface, phase interfacePhase) interfaceData {
+ result := interfaceData{
+ Index: value.Index, Name: value.Name, TAP: value.TAP, Phase: phase,
+ MAC: value.MAC, Queues: value.Queues, QueueSize: value.QueueSize,
+ }
+ if value.IPv4 != nil {
+ result.IPv4 = &ipv4Data{Address: value.IPv4.Address, Gateway: value.IPv4.Gateway, Prefix: value.IPv4.Prefix}
+ }
+ return result
+}
+
+func corrupt(cause error) error {
+ return errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, fmt.Errorf("decode CNI network record: %w", cause))
+}
+
+func interfaceName(index int) string { return fmt.Sprintf("eth%d", index) }
+
+func findInterface(record *recordData, index int) int {
+ return slices.IndexFunc(record.Interfaces, func(item interfaceData) bool { return item.Index == index })
+}
+
+func removeInterface(record *recordData, index int) {
+ position := findInterface(record, index)
+ if position >= 0 {
+ record.Interfaces = slices.Delete(record.Interfaces, position, position+1)
+ }
+}
+
+// newTestProvider constructs a provider around injected side-effect seams. It
+// stays unexported so production composition always uses New.
+func newTestProvider(options Options, store metadata.Store, lists map[string]*libcni.NetworkConfigList, defaultName string, executor pluginRuntime, host platform) *Provider {
+ return &Provider{options: options, store: store, lists: lists, defaultName: defaultName, runtime: executor, platform: host}
+}
diff --git a/network/cni/cni_test.go b/network/cni/cni_test.go
new file mode 100644
index 0000000..2b28cec
--- /dev/null
+++ b/network/cni/cni_test.go
@@ -0,0 +1,275 @@
+package cni
+
+import (
+ "context"
+ "errors"
+ "net"
+ "os"
+ "path/filepath"
+ "slices"
+ "testing"
+ "time"
+
+ "github.com/containernetworking/cni/libcni"
+ cnitypes "github.com/containernetworking/cni/pkg/types"
+ current "github.com/containernetworking/cni/pkg/types/100"
+
+ "github.com/kumabox/kumabox/metadata"
+ "github.com/kumabox/kumabox/network"
+ "github.com/kumabox/kumabox/types"
+)
+
+type fakeRuntime struct {
+ addError error
+ delError error
+ adds []string
+ dels []string
+}
+
+func (f *fakeRuntime) AddNetworkList(_ context.Context, _ *libcni.NetworkConfigList, runtime *libcni.RuntimeConf) (cnitypes.Result, error) {
+ f.adds = append(f.adds, runtime.IfName)
+ if f.addError != nil {
+ return nil, f.addError
+ }
+ return ¤t.Result{
+ CNIVersion: "1.0.0",
+ IPs: []*current.IPConfig{{
+ Address: net.IPNet{IP: net.ParseIP("10.42.0.7"), Mask: net.CIDRMask(24, 32)},
+ Gateway: net.ParseIP("10.42.0.1"),
+ }},
+ }, nil
+}
+
+func (f *fakeRuntime) DelNetworkList(_ context.Context, _ *libcni.NetworkConfigList, runtime *libcni.RuntimeConf) error {
+ f.dels = append(f.dels, runtime.IfName)
+ return f.delError
+}
+
+type fakePlatform struct {
+ namespace bool
+ removeError error
+ linksUp []bool
+ deletedTAPs []string
+ verifiedTAPs []string
+ ensuredNames []string
+ removedNames []string
+ redirectedNames []string
+}
+
+func (f *fakePlatform) EnsureNamespace(name, _ string) (bool, error) {
+ created := !f.namespace
+ f.namespace = true
+ f.ensuredNames = append(f.ensuredNames, name)
+ return created, nil
+}
+
+func (f *fakePlatform) RemoveNamespace(_ context.Context, name string) error {
+ f.removedNames = append(f.removedNames, name)
+ if f.removeError != nil {
+ return f.removeError
+ }
+ f.namespace = false
+ return nil
+}
+
+func (f *fakePlatform) NamespaceExists(string) error {
+ if !f.namespace {
+ return os.ErrNotExist
+ }
+ return nil
+}
+
+func (f *fakePlatform) SetupRedirect(_, interfaceName, _ string, _ int, overrideMAC string) (string, error) {
+ f.redirectedNames = append(f.redirectedNames, interfaceName)
+ if overrideMAC != "" {
+ return overrideMAC, nil
+ }
+ return "02:00:00:00:00:07", nil
+}
+
+func (f *fakePlatform) DeleteTAP(_, tap string) error {
+ f.deletedTAPs = append(f.deletedTAPs, tap)
+ return nil
+}
+
+func (f *fakePlatform) SetLinkState(_ string, _ []string, up bool) error {
+ f.linksUp = append(f.linksUp, up)
+ return nil
+}
+
+func (f *fakePlatform) VerifyTAP(_, tap string) error {
+ f.verifiedTAPs = append(f.verifiedTAPs, tap)
+ return nil
+}
+
+func TestProviderLifecyclePersistsCleanupIntent(t *testing.T) {
+ provider, executor, host, id := testProvider(t)
+ namespace, err := provider.Prepare(t.Context(), id)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if namespace != filepath.Join(namedNamespaceDir, "kb-"+id.String()) {
+ t.Fatalf("namespace = %q", namespace)
+ }
+ interfaces, err := provider.Add(t.Context(), id, "bridge", network.AddSpec{Index: 0, Queues: 4})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(interfaces) != 1 || interfaces[0].MAC != "02:00:00:00:00:07" || interfaces[0].IPv4.Address != "10.42.0.7" {
+ t.Fatalf("interfaces = %+v", interfaces)
+ }
+ record, err := provider.view(t.Context(), id)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record == nil || record.Phase != phaseReady || record.Interfaces[0].Phase != interfaceReady {
+ t.Fatalf("record = %+v", record)
+ }
+ if err := provider.Quiesce(t.Context(), id); err != nil {
+ t.Fatal(err)
+ }
+ if err := provider.Unquiesce(t.Context(), id); err != nil {
+ t.Fatal(err)
+ }
+ if err := provider.Verify(t.Context(), id, interfaces); err != nil {
+ t.Fatal(err)
+ }
+ if err := provider.Delete(t.Context(), id); err != nil {
+ t.Fatal(err)
+ }
+ record, err = provider.view(t.Context(), id)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record != nil || host.namespace {
+ t.Fatalf("delete retained record=%+v namespace=%t", record, host.namespace)
+ }
+ if !slices.Equal(executor.adds, []string{"eth0"}) || !slices.Equal(executor.dels, []string{"eth0"}) {
+ t.Fatalf("CNI calls add=%v del=%v", executor.adds, executor.dels)
+ }
+ if !slices.Equal(host.linksUp, []bool{false, true}) {
+ t.Fatalf("link states = %v", host.linksUp)
+ }
+}
+
+func TestAddFailureCompensatesWithoutLosingNamespaceOwnership(t *testing.T) {
+ provider, executor, _, id := testProvider(t)
+ executor.addError = errors.New("injected ADD failure")
+ if _, err := provider.Prepare(t.Context(), id); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := provider.Add(t.Context(), id, "bridge", network.AddSpec{Index: 0, Queues: 2}); err == nil {
+ t.Fatal("Add unexpectedly succeeded")
+ }
+ record, err := provider.view(t.Context(), id)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record == nil || record.Phase != phasePreparing || len(record.Interfaces) != 0 {
+ t.Fatalf("rollback record = %+v", record)
+ }
+ if !slices.Equal(executor.dels, []string{"eth0"}) {
+ t.Fatalf("rollback DEL calls = %v", executor.dels)
+ }
+}
+
+func TestDeleteFailureRetainsOnlyRetryableCleanupState(t *testing.T) {
+ provider, executor, _, id := testProvider(t)
+ if _, err := provider.Prepare(t.Context(), id); err != nil {
+ t.Fatal(err)
+ }
+ interfaces, err := provider.Add(t.Context(), id, "bridge", network.AddSpec{Index: 0, Queues: 2})
+ if err != nil || len(interfaces) != 1 {
+ t.Fatalf("Add = %+v, %v", interfaces, err)
+ }
+ executor.delError = errors.New("injected DEL failure")
+ if err := provider.Delete(t.Context(), id); err == nil {
+ t.Fatal("Delete unexpectedly succeeded")
+ }
+ record, err := provider.view(t.Context(), id)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record == nil || record.Phase != phaseDeleting || len(record.Interfaces) != 1 {
+ t.Fatalf("failed delete record = %+v", record)
+ }
+ executor.delError = nil
+ if err := provider.Delete(t.Context(), id); err != nil {
+ t.Fatalf("Delete retry: %v", err)
+ }
+ if record, err := provider.view(t.Context(), id); err != nil || record != nil {
+ t.Fatalf("retry retained record=%+v error=%v", record, err)
+ }
+}
+
+func TestLoadConfListsUsesFirstFilenameAndRejectsDuplicateNames(t *testing.T) {
+ directory := t.TempDir()
+ writeConflist(t, directory, "20-second.conflist", "second")
+ writeConflist(t, directory, "10-first.conflist", "first")
+ lists, defaultName, err := loadConfLists(directory)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if defaultName != "first" || len(lists) != 2 {
+ t.Fatalf("default=%q lists=%v", defaultName, lists)
+ }
+ writeConflist(t, directory, "30-duplicate.conflist", "first")
+ if _, _, err := loadConfLists(directory); err == nil {
+ t.Fatal("duplicate CNI network name was accepted")
+ }
+}
+
+func TestNewWithoutConflistAllowsInspectionButRejectsAdd(t *testing.T) {
+ store, err := metadata.NewMemory(Collections())
+ if err != nil {
+ t.Fatal(err)
+ }
+ options := Options{
+ ConfDir: filepath.Join(t.TempDir(), "missing"), BinDir: "/opt/cni/bin",
+ CacheDir: filepath.Join(t.TempDir(), "cache"), NamespacePrefix: "kb-", CleanupTimeout: time.Second,
+ }
+ provider, err := New(options, store)
+ if err != nil {
+ t.Fatal(err)
+ }
+ id := mustID(t)
+ if namespace, err := provider.Prepare(t.Context(), id); err != nil || namespace != "" {
+ t.Fatalf("Prepare = %q, %v", namespace, err)
+ }
+ if _, err := provider.Add(t.Context(), id, "", network.AddSpec{Index: 0, Queues: 2}); !errors.Is(err, network.ErrNotConfigured) {
+ t.Fatalf("Add error = %v", err)
+ }
+}
+
+func testProvider(t *testing.T) (*Provider, *fakeRuntime, *fakePlatform, types.SandboxID) {
+ t.Helper()
+ store, err := metadata.NewMemory(Collections())
+ if err != nil {
+ t.Fatal(err)
+ }
+ executor := &fakeRuntime{}
+ host := &fakePlatform{}
+ list := &libcni.NetworkConfigList{Name: "bridge", CNIVersion: "1.0.0"}
+ options := Options{
+ ConfDir: "/etc/cni/net.d", BinDir: "/opt/cni/bin", CacheDir: filepath.Join(t.TempDir(), "cache"),
+ NamespacePrefix: "kb-", CleanupTimeout: time.Second,
+ }
+ return newTestProvider(options, store, map[string]*libcni.NetworkConfigList{"bridge": list}, "bridge", executor, host), executor, host, mustID(t)
+}
+
+func mustID(t *testing.T) types.SandboxID {
+ t.Helper()
+ id, err := types.ParseSandboxID("123e4567-e89b-42d3-a456-426614174000")
+ if err != nil {
+ t.Fatal(err)
+ }
+ return id
+}
+
+func writeConflist(t *testing.T, directory, name, networkName string) {
+ t.Helper()
+ contents := []byte(`{"cniVersion":"1.0.0","name":"` + networkName + `","plugins":[{"type":"bridge"}]}`)
+ if err := os.WriteFile(filepath.Join(directory, name), contents, 0o600); err != nil {
+ t.Fatal(err)
+ }
+}
diff --git a/network/cni/lifecycle.go b/network/cni/lifecycle.go
new file mode 100644
index 0000000..77dd55d
--- /dev/null
+++ b/network/cni/lifecycle.go
@@ -0,0 +1,494 @@
+package cni
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io/fs"
+ "slices"
+
+ "github.com/containernetworking/cni/libcni"
+ cnitypes "github.com/containernetworking/cni/pkg/types"
+ current "github.com/containernetworking/cni/pkg/types/100"
+
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/network"
+ "github.com/kumabox/kumabox/types"
+)
+
+// Prepare records namespace ownership before asking the kernel to create it.
+// If no conflist is installed, it returns an empty namespace; Add will report
+// the actionable configuration error when networking is actually requested.
+func (p *Provider) Prepare(ctx context.Context, id types.SandboxID) (string, error) {
+ if err := validID(id); err != nil {
+ return "", err
+ }
+ if _, err := p.confList(""); err != nil {
+ if errors.Is(err, network.ErrNotConfigured) {
+ return "", nil
+ }
+ return "", err
+ }
+ name, path := p.namespace(id)
+ if err := p.update(ctx, id, func(record *recordData) (*recordData, error) {
+ if record == nil {
+ return &recordData{
+ Version: recordVersion, SandboxID: id.String(), NamespaceName: name,
+ NamespacePath: path, Phase: phasePreparing, Interfaces: []interfaceData{},
+ }, nil
+ }
+ if record.Phase == phaseDeleting {
+ return nil, errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, fmt.Errorf("sandbox %s network deletion is incomplete", id))
+ }
+ if record.NamespaceName != name || record.NamespacePath != path {
+ return nil, corrupt(errors.New("stored namespace differs from configured namespace"))
+ }
+ return record, nil
+ }); err != nil {
+ return "", fmt.Errorf("record network namespace intent: %w", err)
+ }
+ if _, err := p.platform.EnsureNamespace(name, path); err != nil {
+ return "", fmt.Errorf("ensure network namespace %s: %w", name, err)
+ }
+ return path, nil
+}
+
+// Add stages every NIC before plugin execution, then advances one interface at
+// a time through adding to ready. A crash during ADD therefore leaves enough
+// information for Delete to issue the matching DEL.
+//
+// staged -> adding -> CNI ADD -> TAP/TC -> ready
+// \---- failure ----> CNI DEL -> sweep
+func (p *Provider) Add(ctx context.Context, id types.SandboxID, networkName string, specs ...network.AddSpec) (result []types.NetworkInterface, returnErr error) {
+ if err := validID(id); err != nil {
+ return nil, err
+ }
+ if len(specs) == 0 {
+ return []types.NetworkInterface{}, nil
+ }
+ list, err := p.confList(networkName)
+ if err != nil {
+ return nil, err
+ }
+ if _, err := p.Prepare(ctx, id); err != nil {
+ return nil, err
+ }
+ if err := validateSpecs(specs); err != nil {
+ return nil, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ if err := p.stage(ctx, id, list.Name, specs); err != nil {
+ return nil, err
+ }
+
+ touched := make([]int, 0, len(specs))
+ defer func() {
+ if returnErr == nil || len(touched) == 0 {
+ return
+ }
+ rollbackCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), p.options.CleanupTimeout)
+ defer cancel()
+ returnErr = errors.Join(returnErr, p.rollback(rollbackCtx, id, list, touched))
+ }()
+
+ result = make([]types.NetworkInterface, 0, len(specs))
+ for _, spec := range specs {
+ item, err := p.interfaceRecord(ctx, id, spec.Index)
+ if err != nil {
+ return nil, err
+ }
+ if item.Phase == interfaceReady {
+ ready, err := item.toType(list.Name)
+ if err != nil {
+ return nil, corrupt(err)
+ }
+ result = append(result, ready)
+ continue
+ }
+ touched = append(touched, spec.Index)
+ if item.Phase == interfaceAdding {
+ if err := p.deleteOne(ctx, id, list, item, true); err != nil {
+ return nil, fmt.Errorf("recover interrupted CNI ADD for %s/%s: %w", id, item.Name, err)
+ }
+ }
+ if err := p.setInterfacePhase(ctx, id, spec.Index, interfaceAdding); err != nil {
+ return nil, err
+ }
+ ready, err := p.addOne(ctx, id, list, item, spec.Existing)
+ if err != nil {
+ return nil, err
+ }
+ if err := p.storeReady(ctx, id, ready); err != nil {
+ return nil, err
+ }
+ result = append(result, ready)
+ }
+ if err := p.update(ctx, id, func(record *recordData) (*recordData, error) {
+ if record == nil {
+ return nil, corrupt(errors.New("network record disappeared while completing ADD"))
+ }
+ record.Phase = phaseReady
+ return record, nil
+ }); err != nil {
+ return nil, fmt.Errorf("commit network readiness: %w", err)
+ }
+ touched = nil
+ slices.SortFunc(result, func(left, right types.NetworkInterface) int { return left.Index - right.Index })
+ return result, nil
+}
+
+func (p *Provider) stage(ctx context.Context, id types.SandboxID, networkName string, specs []network.AddSpec) error {
+ return p.update(ctx, id, func(record *recordData) (*recordData, error) {
+ if record == nil {
+ return nil, corrupt(errors.New("network namespace intent is missing"))
+ }
+ if record.Phase == phaseDeleting {
+ return nil, errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, fmt.Errorf("sandbox %s network deletion is incomplete", id))
+ }
+ if record.Network != "" && record.Network != networkName {
+ return nil, errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, fmt.Errorf("sandbox %s is already bound to CNI network %q", id, record.Network))
+ }
+ record.Network = networkName
+ for _, spec := range specs {
+ position := findInterface(record, spec.Index)
+ if position >= 0 {
+ continue
+ }
+ tap, err := network.TAPName(defaultTAPPrefix, id, spec.Index)
+ if err != nil {
+ return nil, err
+ }
+ value := types.NetworkInterface{
+ Index: spec.Index, Name: interfaceName(spec.Index), TAP: tap,
+ Queues: spec.Queues, QueueSize: network.DefaultQueueSize, Network: networkName,
+ }
+ if spec.Existing != nil {
+ value.MAC = spec.Existing.MAC
+ value.IPv4 = spec.Existing.IPv4
+ }
+ record.Interfaces = append(record.Interfaces, fromType(value, interfaceStaged))
+ }
+ slices.SortFunc(record.Interfaces, func(left, right interfaceData) int { return left.Index - right.Index })
+ return record, nil
+ })
+}
+
+func (p *Provider) addOne(ctx context.Context, id types.SandboxID, list *libcni.NetworkConfigList, item interfaceData, existing *types.NetworkInterface) (types.NetworkInterface, error) {
+ record, err := p.view(ctx, id)
+ if err != nil || record == nil {
+ return types.NetworkInterface{}, errors.Join(err, errors.New("network record is missing"))
+ }
+ runtimeConfig := &libcni.RuntimeConf{ContainerID: id.String(), NetNS: record.NamespacePath, IfName: item.Name}
+ if existing != nil && existing.IPv4 != nil && existing.IPv4.Address != "" {
+ runtimeConfig.Args = [][2]string{{"IgnoreUnknown", "1"}, {"IP", existing.IPv4.Address}}
+ }
+ cniResult, err := p.runtime.AddNetworkList(ctx, list, runtimeConfig)
+ if err != nil {
+ return types.NetworkInterface{}, fmt.Errorf("CNI ADD %s/%s: %w", id, item.Name, err)
+ }
+ ipv4, err := extractIPv4(cniResult)
+ if err != nil {
+ return types.NetworkInterface{}, fmt.Errorf("parse CNI result for %s/%s: %w", id, item.Name, err)
+ }
+ overrideMAC := item.MAC
+ if existing != nil && existing.MAC != "" {
+ overrideMAC = existing.MAC
+ }
+ mac, err := p.platform.SetupRedirect(record.NamespacePath, item.Name, item.TAP, item.Queues, overrideMAC)
+ if err != nil {
+ return types.NetworkInterface{}, fmt.Errorf("connect TAP for %s/%s: %w", id, item.Name, err)
+ }
+ ready := types.NetworkInterface{
+ Index: item.Index, Name: item.Name, TAP: item.TAP, MAC: mac,
+ Queues: item.Queues, QueueSize: item.QueueSize, Network: list.Name, IPv4: ipv4,
+ }
+ if err := ready.Validate(); err != nil {
+ return types.NetworkInterface{}, err
+ }
+ return ready, nil
+}
+
+func (p *Provider) rollback(ctx context.Context, id types.SandboxID, list *libcni.NetworkConfigList, indices []int) error {
+ var failures []error
+ released := make(map[int]bool, len(indices))
+ for _, index := range indices {
+ item, err := p.interfaceRecord(ctx, id, index)
+ if err != nil {
+ failures = append(failures, err)
+ continue
+ }
+ if err := p.deleteOne(ctx, id, list, item, true); err != nil {
+ failures = append(failures, fmt.Errorf("rollback %s: %w", item.Name, err))
+ continue
+ }
+ released[index] = true
+ }
+ if len(released) > 0 {
+ if err := p.update(ctx, id, func(record *recordData) (*recordData, error) {
+ if record == nil {
+ return nil, nil
+ }
+ for index := range released {
+ removeInterface(record, index)
+ }
+ return record, nil
+ }); err != nil {
+ failures = append(failures, fmt.Errorf("release rollback records: %w", err))
+ }
+ }
+ return errors.Join(failures...)
+}
+
+func (p *Provider) deleteOne(ctx context.Context, id types.SandboxID, list *libcni.NetworkConfigList, item interfaceData, deleteTAP bool) error {
+ record, err := p.view(ctx, id)
+ if err != nil || record == nil {
+ return errors.Join(err, errors.New("network record is missing"))
+ }
+ if item.Phase != interfaceStaged {
+ runtimeConfig := &libcni.RuntimeConf{ContainerID: id.String(), NetNS: record.NamespacePath, IfName: item.Name}
+ if err := p.runtime.DelNetworkList(ctx, list, runtimeConfig); err != nil {
+ return fmt.Errorf("CNI DEL %s/%s: %w", id, item.Name, err)
+ }
+ }
+ if deleteTAP {
+ if err := p.platform.DeleteTAP(record.NamespacePath, item.TAP); err != nil {
+ return fmt.Errorf("delete TAP %s: %w", item.TAP, err)
+ }
+ }
+ return nil
+}
+
+func (p *Provider) setInterfacePhase(ctx context.Context, id types.SandboxID, index int, phase interfacePhase) error {
+ return p.update(ctx, id, func(record *recordData) (*recordData, error) {
+ if record == nil {
+ return nil, corrupt(errors.New("network record is missing"))
+ }
+ position := findInterface(record, index)
+ if position < 0 {
+ return nil, corrupt(fmt.Errorf("network interface %d is missing", index))
+ }
+ record.Interfaces[position].Phase = phase
+ return record, nil
+ })
+}
+
+func (p *Provider) storeReady(ctx context.Context, id types.SandboxID, ready types.NetworkInterface) error {
+ return p.update(ctx, id, func(record *recordData) (*recordData, error) {
+ if record == nil {
+ return nil, corrupt(errors.New("network record is missing"))
+ }
+ position := findInterface(record, ready.Index)
+ if position < 0 {
+ return nil, corrupt(fmt.Errorf("network interface %d is missing", ready.Index))
+ }
+ record.Interfaces[position] = fromType(ready, interfaceReady)
+ return record, nil
+ })
+}
+
+func (p *Provider) interfaceRecord(ctx context.Context, id types.SandboxID, index int) (interfaceData, error) {
+ record, err := p.view(ctx, id)
+ if err != nil {
+ return interfaceData{}, err
+ }
+ if record == nil {
+ return interfaceData{}, corrupt(errors.New("network record is missing"))
+ }
+ position := findInterface(record, index)
+ if position < 0 {
+ return interfaceData{}, corrupt(fmt.Errorf("network interface %d is missing", index))
+ }
+ return record.Interfaces[position], nil
+}
+
+// Verify checks both the namespace and every expected TAP. Metadata alone is
+// never accepted as proof that host plumbing survived a reboot.
+func (p *Provider) Verify(_ context.Context, id types.SandboxID, expected []types.NetworkInterface) error {
+ if err := validID(id); err != nil {
+ return err
+ }
+ _, path := p.namespace(id)
+ if err := p.platform.NamespaceExists(path); err != nil {
+ return fmt.Errorf("network namespace %s: %w", path, err)
+ }
+ for _, item := range expected {
+ if err := item.Validate(); err != nil {
+ return err
+ }
+ if err := p.platform.VerifyTAP(path, item.TAP); err != nil {
+ return fmt.Errorf("verify TAP %s: %w", item.TAP, err)
+ }
+ }
+ return nil
+}
+
+// Recover rebuilds missing host plumbing from the durable guest identities.
+func (p *Provider) Recover(ctx context.Context, id types.SandboxID, networkName string, expected []types.NetworkInterface) ([]types.NetworkInterface, error) {
+ if err := p.Verify(ctx, id, expected); err == nil {
+ if err := p.Unquiesce(ctx, id); err != nil {
+ return nil, err
+ }
+ return slices.Clone(expected), nil
+ }
+ if err := p.Delete(ctx, id); err != nil {
+ return nil, fmt.Errorf("delete incomplete network before recovery: %w", err)
+ }
+ if _, err := p.Prepare(ctx, id); err != nil {
+ return nil, err
+ }
+ specs := make([]network.AddSpec, len(expected))
+ for index := range expected {
+ current := expected[index]
+ specs[index] = network.AddSpec{Index: current.Index, Queues: current.Queues, Existing: ¤t}
+ if networkName == "" {
+ networkName = current.Network
+ }
+ }
+ return p.Add(ctx, id, networkName, specs...)
+}
+
+// Quiesce brings CNI-side veth devices down while retaining identity and TAPs.
+func (p *Provider) Quiesce(ctx context.Context, id types.SandboxID) error {
+ return p.setLinkState(ctx, id, false)
+}
+
+// Unquiesce brings retained CNI-side veth devices back up before launch.
+func (p *Provider) Unquiesce(ctx context.Context, id types.SandboxID) error {
+ return p.setLinkState(ctx, id, true)
+}
+
+func (p *Provider) setLinkState(ctx context.Context, id types.SandboxID, up bool) error {
+ record, err := p.view(ctx, id)
+ if err != nil || record == nil {
+ return err
+ }
+ names := make([]string, 0, len(record.Interfaces))
+ for _, item := range record.Interfaces {
+ if item.Phase == interfaceReady {
+ names = append(names, item.Name)
+ }
+ }
+ if len(names) == 0 {
+ return nil
+ }
+ if err := p.platform.SetLinkState(record.NamespacePath, names, up); err != nil {
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ return fmt.Errorf("set sandbox %s network links up=%t: %w", id, up, err)
+ }
+ return nil
+}
+
+// Delete advances the aggregate to deleting before slow host operations. Each
+// successful DEL is swept independently; failures keep exactly the remaining
+// release context for the next retry.
+//
+// ready -> deleting -> per-NIC DEL -> remove netns -> delete record
+// \ failure: retain only unfinished NICs /
+func (p *Provider) Delete(ctx context.Context, id types.SandboxID) error {
+ if err := validID(id); err != nil {
+ return err
+ }
+ if err := p.update(ctx, id, func(record *recordData) (*recordData, error) {
+ if record == nil {
+ return nil, nil
+ }
+ record.Phase = phaseDeleting
+ return record, nil
+ }); err != nil {
+ return fmt.Errorf("mark network deleting: %w", err)
+ }
+ record, err := p.view(ctx, id)
+ if err != nil || record == nil {
+ return err
+ }
+ released := make(map[int]bool, len(record.Interfaces))
+ var failures []error
+ for _, item := range record.Interfaces {
+ list, listErr := p.confList(record.Network)
+ if item.Phase == interfaceStaged {
+ listErr = nil
+ }
+ if listErr != nil {
+ failures = append(failures, listErr)
+ continue
+ }
+ if err := p.deleteOne(ctx, id, list, item, false); err != nil {
+ failures = append(failures, err)
+ continue
+ }
+ released[item.Index] = true
+ }
+ if len(released) > 0 {
+ if err := p.update(ctx, id, func(current *recordData) (*recordData, error) {
+ if current == nil {
+ return nil, nil
+ }
+ for index := range released {
+ removeInterface(current, index)
+ }
+ return current, nil
+ }); err != nil {
+ failures = append(failures, fmt.Errorf("sweep released network records: %w", err))
+ }
+ }
+ if len(failures) > 0 {
+ return errors.Join(failures...)
+ }
+ if err := p.platform.RemoveNamespace(ctx, record.NamespaceName); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return fmt.Errorf("remove network namespace %s: %w", record.NamespaceName, err)
+ }
+ if err := p.update(ctx, id, func(*recordData) (*recordData, error) { return nil, nil }); err != nil {
+ return fmt.Errorf("finalize network deletion: %w", err)
+ }
+ return nil
+}
+
+func validateSpecs(specs []network.AddSpec) error {
+ seen := make(map[int]struct{}, len(specs))
+ for _, spec := range specs {
+ if spec.Index < 0 || spec.Queues < 2 || spec.Queues%2 != 0 {
+ return fmt.Errorf("NIC %d requires an even queue count of at least two", spec.Index)
+ }
+ if _, exists := seen[spec.Index]; exists {
+ return fmt.Errorf("NIC index %d is duplicated", spec.Index)
+ }
+ seen[spec.Index] = struct{}{}
+ if spec.Existing != nil {
+ if spec.Existing.Index != spec.Index {
+ return fmt.Errorf("NIC %d recovery identity belongs to index %d", spec.Index, spec.Existing.Index)
+ }
+ if err := spec.Existing.Validate(); err != nil {
+ return err
+ }
+ }
+ }
+ return nil
+}
+
+func validID(id types.SandboxID) error {
+ _, err := types.ParseSandboxID(id.String())
+ if err != nil {
+ return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ return nil
+}
+
+func extractIPv4(result cnitypes.Result) (*types.IPv4Config, error) {
+ converted, err := current.NewResultFromResult(result)
+ if err != nil {
+ return nil, err
+ }
+ for _, configuration := range converted.IPs {
+ if configuration == nil || configuration.Address.IP.To4() == nil {
+ continue
+ }
+ prefix, _ := configuration.Address.Mask.Size()
+ result := &types.IPv4Config{Address: configuration.Address.IP.String(), Prefix: prefix}
+ if configuration.Gateway != nil {
+ result.Gateway = configuration.Gateway.String()
+ }
+ return result, result.Validate()
+ }
+ return nil, nil
+}
diff --git a/network/cni/platform_linux.go b/network/cni/platform_linux.go
new file mode 100644
index 0000000..1460dc6
--- /dev/null
+++ b/network/cni/platform_linux.go
@@ -0,0 +1,243 @@
+//go:build linux
+
+package cni
+
+import (
+ "cmp"
+ "context"
+ "errors"
+ "fmt"
+ "io/fs"
+ "net"
+ "os"
+ "runtime"
+ "syscall"
+ "time"
+
+ cns "github.com/containernetworking/plugins/pkg/ns"
+ "github.com/vishvananda/netlink"
+ "github.com/vishvananda/netns"
+)
+
+const (
+ tapTXQueueLength = 10000
+ tapGROMaxSize = 65536
+)
+
+type linuxPlatform struct{}
+
+func newPlatform() platform { return linuxPlatform{} }
+
+func (linuxPlatform) EnsureNamespace(name, path string) (_ bool, returnErr error) {
+ if _, err := os.Stat(path); err == nil {
+ return false, nil
+ } else if !errors.Is(err, fs.ErrNotExist) {
+ return false, err
+ }
+ runtime.LockOSThread()
+ defer runtime.UnlockOSThread()
+ original, err := netns.Get()
+ if err != nil {
+ return false, fmt.Errorf("get current network namespace: %w", err)
+ }
+ defer func() {
+ returnErr = errors.Join(returnErr, netns.Set(original), original.Close())
+ }()
+ created, err := netns.NewNamed(name)
+ if err != nil {
+ return false, fmt.Errorf("create named network namespace %s: %w", name, err)
+ }
+ if err := created.Close(); err != nil {
+ return false, fmt.Errorf("close network namespace %s: %w", name, err)
+ }
+ return true, nil
+}
+
+func (linuxPlatform) RemoveNamespace(ctx context.Context, name string) error {
+ deadline := time.NewTimer(time.Second)
+ defer deadline.Stop()
+ ticker := time.NewTicker(100 * time.Millisecond)
+ defer ticker.Stop()
+ for {
+ err := netns.DeleteNamed(name)
+ if err == nil || errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ select {
+ case <-ctx.Done():
+ return ctx.Err()
+ case <-deadline.C:
+ return err
+ case <-ticker.C:
+ }
+ }
+}
+
+func (linuxPlatform) NamespaceExists(path string) error {
+ _, err := os.Stat(path)
+ return err
+}
+
+func (linuxPlatform) VerifyTAP(namespacePath, tapName string) error {
+ return cns.WithNetNSPath(namespacePath, func(_ cns.NetNS) error {
+ _, err := netlink.LinkByName(tapName)
+ return err
+ })
+}
+
+func (linuxPlatform) SetupRedirect(namespacePath, interfaceName, tapName string, queues int, overrideMAC string) (string, error) {
+ var mac string
+ err := cns.WithNetNSPath(namespacePath, func(_ cns.NetNS) error {
+ var err error
+ mac, err = setupRedirect(interfaceName, tapName, queues, overrideMAC)
+ return err
+ })
+ return mac, err
+}
+
+func setupRedirect(interfaceName, tapName string, queues int, overrideMAC string) (string, error) {
+ source, err := netlink.LinkByName(interfaceName)
+ if err != nil {
+ return "", fmt.Errorf("find CNI link %s: %w", interfaceName, err)
+ }
+ if overrideMAC != "" {
+ hardwareAddress, err := net.ParseMAC(overrideMAC)
+ if err != nil {
+ return "", fmt.Errorf("parse MAC %s: %w", overrideMAC, err)
+ }
+ if err := netlink.LinkSetHardwareAddr(source, hardwareAddress); err != nil {
+ return "", fmt.Errorf("set MAC on %s: %w", interfaceName, err)
+ }
+ }
+ mac := cmp.Or(overrideMAC, source.Attrs().HardwareAddr.String())
+ addresses, err := netlink.AddrList(source, netlink.FAMILY_ALL)
+ if err != nil {
+ return "", fmt.Errorf("list addresses on %s: %w", interfaceName, err)
+ }
+ for _, address := range addresses {
+ if err := netlink.AddrDel(source, &address); err != nil {
+ return "", fmt.Errorf("remove address %s from %s: %w", address.IPNet, interfaceName, err)
+ }
+ }
+ tap, err := createTAP(tapName, queues)
+ if err != nil {
+ return "", err
+ }
+ if source.Attrs().MTU > 0 {
+ if err := netlink.LinkSetMTU(tap, source.Attrs().MTU); err != nil {
+ return "", fmt.Errorf("set TAP %s MTU: %w", tapName, err)
+ }
+ }
+ for _, link := range []netlink.Link{source, tap} {
+ if err := netlink.LinkSetUp(link); err != nil {
+ return "", fmt.Errorf("set link %s up: %w", link.Attrs().Name, err)
+ }
+ qdisc := &netlink.Ingress{QdiscAttrs: netlink.QdiscAttrs{LinkIndex: link.Attrs().Index, Parent: netlink.HANDLE_INGRESS}}
+ if err := netlink.QdiscAdd(qdisc); err != nil {
+ return "", fmt.Errorf("add ingress qdisc to %s: %w", link.Attrs().Name, err)
+ }
+ }
+ if err := redirect(source, tap); err != nil {
+ return "", fmt.Errorf("redirect %s to %s: %w", interfaceName, tapName, err)
+ }
+ if err := redirect(tap, source); err != nil {
+ return "", fmt.Errorf("redirect %s to %s: %w", tapName, interfaceName, err)
+ }
+ return mac, nil
+}
+
+func createTAP(name string, queues int) (netlink.Link, error) {
+ queuePairs := max(1, queues/2)
+ flags := netlink.TUNTAP_VNET_HDR | netlink.TUNTAP_NO_PI
+ if queuePairs == 1 {
+ flags |= netlink.TUNTAP_ONE_QUEUE
+ } else {
+ flags |= netlink.TUNTAP_MULTI_QUEUE_DEFAULTS
+ }
+ tap := &netlink.Tuntap{
+ LinkAttrs: netlink.LinkAttrs{Name: name},
+ Mode: netlink.TUNTAP_MODE_TAP,
+ Queues: queuePairs,
+ Flags: flags,
+ }
+ if err := netlink.LinkAdd(tap); err != nil {
+ return nil, fmt.Errorf("create TAP %s: %w", name, err)
+ }
+ for _, descriptor := range tap.Fds {
+ _ = descriptor.Close()
+ }
+ link, err := netlink.LinkByName(name)
+ if err != nil {
+ return nil, fmt.Errorf("resolve TAP %s: %w", name, err)
+ }
+ // Queue and GRO tuning improve throughput but are not supported by every
+ // kernel. The functional network path must remain available in that case.
+ _ = netlink.LinkSetTxQLen(link, tapTXQueueLength)
+ _ = netlink.LinkSetGROMaxSize(link, tapGROMaxSize)
+ return link, nil
+}
+
+func redirect(source, target netlink.Link) error {
+ return netlink.FilterAdd(&netlink.U32{
+ FilterAttrs: netlink.FilterAttrs{
+ LinkIndex: source.Attrs().Index, Parent: netlink.HANDLE_INGRESS,
+ Priority: 1, Protocol: syscall.ETH_P_ALL,
+ },
+ Sel: &netlink.TcU32Sel{
+ Flags: netlink.TC_U32_TERMINAL,
+ Keys: []netlink.TcU32Key{{Mask: 0, Val: 0, Off: 0, OffMask: 0}},
+ },
+ Actions: []netlink.Action{&netlink.MirredAction{
+ ActionAttrs: netlink.ActionAttrs{Action: netlink.TC_ACT_STOLEN},
+ MirredAction: netlink.TCA_EGRESS_REDIR, Ifindex: target.Attrs().Index,
+ }},
+ })
+}
+
+func (linuxPlatform) DeleteTAP(namespacePath, tapName string) error {
+ err := cns.WithNetNSPath(namespacePath, func(_ cns.NetNS) error {
+ link, err := netlink.LinkByName(tapName)
+ if err != nil {
+ var notFound netlink.LinkNotFoundError
+ if errors.As(err, ¬Found) {
+ return nil
+ }
+ return err
+ }
+ return netlink.LinkDel(link)
+ })
+ var namespaceMissing cns.NSPathNotExistErr
+ if errors.As(err, &namespaceMissing) {
+ return nil
+ }
+ return err
+}
+
+func (linuxPlatform) SetLinkState(namespacePath string, names []string, up bool) error {
+ err := cns.WithNetNSPath(namespacePath, func(_ cns.NetNS) error {
+ for _, name := range names {
+ link, err := netlink.LinkByName(name)
+ if err != nil {
+ var notFound netlink.LinkNotFoundError
+ if errors.As(err, ¬Found) {
+ continue
+ }
+ return err
+ }
+ if up {
+ err = netlink.LinkSetUp(link)
+ } else {
+ err = netlink.LinkSetDown(link)
+ }
+ if err != nil {
+ return fmt.Errorf("set link %s state: %w", name, err)
+ }
+ }
+ return nil
+ })
+ var namespaceMissing cns.NSPathNotExistErr
+ if errors.As(err, &namespaceMissing) {
+ return nil
+ }
+ return err
+}
diff --git a/network/cni/platform_other.go b/network/cni/platform_other.go
new file mode 100644
index 0000000..512ada4
--- /dev/null
+++ b/network/cni/platform_other.go
@@ -0,0 +1,36 @@
+//go:build !linux
+
+package cni
+
+import (
+ "context"
+ "errors"
+)
+
+var errPlatformUnsupported = errors.New("CNI network namespace operations require Linux")
+
+type unsupportedPlatform struct{}
+
+func newPlatform() platform { return unsupportedPlatform{} }
+
+func (unsupportedPlatform) EnsureNamespace(string, string) (bool, error) {
+ return false, errPlatformUnsupported
+}
+
+func (unsupportedPlatform) RemoveNamespace(context.Context, string) error {
+ return errPlatformUnsupported
+}
+
+func (unsupportedPlatform) NamespaceExists(string) error { return errPlatformUnsupported }
+
+func (unsupportedPlatform) SetupRedirect(string, string, string, int, string) (string, error) {
+ return "", errPlatformUnsupported
+}
+
+func (unsupportedPlatform) DeleteTAP(string, string) error { return errPlatformUnsupported }
+
+func (unsupportedPlatform) SetLinkState(string, []string, bool) error {
+ return errPlatformUnsupported
+}
+
+func (unsupportedPlatform) VerifyTAP(string, string) error { return errPlatformUnsupported }
diff --git a/network/network.go b/network/network.go
new file mode 100644
index 0000000..ae831fc
--- /dev/null
+++ b/network/network.go
@@ -0,0 +1,103 @@
+// Package network defines the host network provider contract used by the
+// sandbox service. Concrete CNI and bridge implementations live in child
+// packages and provider-private cleanup state never crosses this boundary.
+package network
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "strconv"
+ "strings"
+
+ "github.com/kumabox/kumabox/types"
+)
+
+const (
+ // DefaultQueueSize is the descriptor count used by supported VMM backends.
+ DefaultQueueSize = 512
+ // linuxInterfaceNameLimit excludes the terminating NUL byte.
+ linuxInterfaceNameLimit = 15
+)
+
+// ErrNotConfigured reports that no usable infrastructure network definition
+// is installed on the host.
+var ErrNotConfigured = errors.New("network provider is not configured")
+
+// AddSpec describes one NIC allocation. Existing is set during host recovery
+// so the provider can preserve the durable MAC and IP identity.
+type AddSpec struct {
+ // Index is the stable zero-based NIC position.
+ Index int
+ // Queues overrides the CPU-derived queue count when positive.
+ Queues int
+ // Existing carries the identity that recovery must preserve.
+ Existing *types.NetworkInterface
+}
+
+// Provider owns host network namespaces, CNI allocations, and TAP plumbing for
+// a sandbox. Callers serialize operations for one sandbox identifier.
+type Provider interface {
+ // Type returns the durable backend identity.
+ Type() types.NetworkBackend
+ // Prepare creates or recovers the sandbox network namespace.
+ Prepare(context.Context, types.SandboxID) (string, error)
+ // Add allocates and wires the requested interfaces.
+ Add(context.Context, types.SandboxID, string, ...AddSpec) ([]types.NetworkInterface, error)
+ // Verify proves that the namespace and expected TAP devices are present.
+ Verify(context.Context, types.SandboxID, []types.NetworkInterface) error
+ // Recover reconstructs missing host state while preserving guest identity.
+ Recover(context.Context, types.SandboxID, string, []types.NetworkInterface) ([]types.NetworkInterface, error)
+ // Quiesce disables CNI-side links while a VMM is stopped.
+ Quiesce(context.Context, types.SandboxID) error
+ // Unquiesce restores links immediately before a VMM launch.
+ Unquiesce(context.Context, types.SandboxID) error
+ // Delete releases every allocation and the private namespace. It is
+ // retryable after partial failure.
+ Delete(context.Context, types.SandboxID) error
+}
+
+// AddRange builds fresh NIC requests for a contiguous index range.
+func AddRange(first, count int) []AddSpec {
+ if first < 0 || count <= 0 {
+ return nil
+ }
+ result := make([]AddSpec, count)
+ for offset := range result {
+ result[offset] = AddSpec{Index: first + offset}
+ }
+ return result
+}
+
+// QueueCount returns two virtio queues per vCPU with a minimum RX/TX pair.
+func QueueCount(cpus uint32) int { return max(2, int(cpus)*2) }
+
+// ResolveQueues returns an explicit valid queue count or the CPU-derived
+// default. Invalid explicit values are rejected by the provider.
+func ResolveQueues(requested int, cpus uint32) int {
+ if requested > 0 {
+ return requested
+ }
+ return QueueCount(cpus)
+}
+
+// TAPName derives a deterministic Linux interface name within IFNAMSIZ. The
+// UUID prefix plus NIC index remains unique within a sandbox namespace.
+func TAPName(prefix string, id types.SandboxID, index int) (string, error) {
+ if prefix == "" || index < 0 || strings.ContainsAny(prefix, "/\x00") {
+ return "", errors.New("TAP prefix and NIC index are invalid")
+ }
+ for _, character := range prefix {
+ if (character < 'a' || character > 'z') && (character < 'A' || character > 'Z') &&
+ (character < '0' || character > '9') && character != '-' && character != '_' {
+ return "", errors.New("TAP prefix and NIC index are invalid")
+ }
+ }
+ suffix := "-" + strconv.Itoa(index)
+ compact := strings.ReplaceAll(id.String(), "-", "")
+ const identityLength = 8
+ if len(prefix)+identityLength+len(suffix) > linuxInterfaceNameLimit || len(compact) < identityLength {
+ return "", fmt.Errorf("TAP prefix %q and NIC index %d exceed Linux name limits", prefix, index)
+ }
+ return prefix + compact[:identityLength] + suffix, nil
+}
diff --git a/network/network_test.go b/network/network_test.go
new file mode 100644
index 0000000..a3f1b21
--- /dev/null
+++ b/network/network_test.go
@@ -0,0 +1,37 @@
+package network
+
+import (
+ "testing"
+
+ "github.com/kumabox/kumabox/types"
+)
+
+func TestQueueCountAndTAPName(t *testing.T) {
+ if got := QueueCount(0); got != 2 {
+ t.Fatalf("QueueCount(0) = %d, want 2", got)
+ }
+ if got := QueueCount(4); got != 8 {
+ t.Fatalf("QueueCount(4) = %d, want 8", got)
+ }
+ id, err := types.ParseSandboxID("123e4567-e89b-42d3-a456-426614174000")
+ if err != nil {
+ t.Fatal(err)
+ }
+ name, err := TAPName("tap", id, 12)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if name != "tap123e4567-12" || len(name) > linuxInterfaceNameLimit {
+ t.Fatalf("TAPName = %q", name)
+ }
+}
+
+func TestAddRangeRejectsInvalidBounds(t *testing.T) {
+ if got := AddRange(-1, 1); got != nil {
+ t.Fatalf("AddRange(-1, 1) = %#v", got)
+ }
+ got := AddRange(2, 2)
+ if len(got) != 2 || got[0].Index != 2 || got[1].Index != 3 {
+ t.Fatalf("AddRange(2, 2) = %#v", got)
+ }
+}
diff --git a/types/network.go b/types/network.go
new file mode 100644
index 0000000..c852acb
--- /dev/null
+++ b/types/network.go
@@ -0,0 +1,152 @@
+package types
+
+import (
+ "errors"
+ "fmt"
+ "net"
+ "path/filepath"
+ "strconv"
+)
+
+// NetworkBackend identifies the host networking implementation that owns a
+// sandbox's durable network resources.
+type NetworkBackend string
+
+const (
+ // NetworkBackendCNI selects a CNI plugin chain running in a private network
+ // namespace.
+ NetworkBackendCNI NetworkBackend = "cni"
+)
+
+// Validate rejects backend names that cannot be routed to an implementation.
+func (b NetworkBackend) Validate() error {
+ switch b {
+ case NetworkBackendCNI:
+ return nil
+ default:
+ return fmt.Errorf("unsupported network backend %q", b)
+ }
+}
+
+// IPv4Config is the guest-visible address returned by an infrastructure
+// network provider.
+type IPv4Config struct {
+ // Address is one IPv4 address without its prefix length.
+ Address string
+ // Gateway is an optional IPv4 default gateway.
+ Gateway string
+ // Prefix is the CIDR prefix length in bits.
+ Prefix int
+}
+
+// Validate rejects malformed or non-IPv4 addresses before they are persisted
+// or rendered into the guest boot contract.
+func (c IPv4Config) Validate() error {
+ ip := net.ParseIP(c.Address)
+ if ip == nil || ip.To4() == nil {
+ return fmt.Errorf("network address %q is not IPv4", c.Address)
+ }
+ if c.Prefix < 0 || c.Prefix > 32 {
+ return fmt.Errorf("network prefix %d is outside 0..32", c.Prefix)
+ }
+ if c.Gateway != "" {
+ gateway := net.ParseIP(c.Gateway)
+ if gateway == nil || gateway.To4() == nil {
+ return fmt.Errorf("network gateway %q is not IPv4", c.Gateway)
+ }
+ }
+ return nil
+}
+
+// NetworkInterface contains the durable handoff from host networking to a VMM.
+// Provider-private cleanup phases and CNI record identifiers are deliberately
+// excluded from this shared value object.
+type NetworkInterface struct {
+ // Index is the zero-based NIC position used to derive the guest name.
+ Index int
+ // Name is the interface name created by CNI inside the private namespace.
+ Name string
+ // TAP is the device opened by the VMM.
+ TAP string
+ // MAC is the stable guest hardware address.
+ MAC string
+ // Queues is the total RX and TX virtio queue count.
+ Queues int
+ // QueueSize is the descriptor count for each virtio queue.
+ QueueSize int
+ // Network is the resolved CNI conflist name.
+ Network string
+ // IPv4 is nil when a plugin intentionally returns no IPv4 address.
+ IPv4 *IPv4Config
+}
+
+// Validate checks the provider-to-VMM handoff independently of persistence and
+// command presentation.
+func (c NetworkInterface) Validate() error {
+ if c.Index < 0 {
+ return errors.New("network interface index must not be negative")
+ }
+ if c.Name != "eth"+strconv.Itoa(c.Index) {
+ return fmt.Errorf("network interface %d must be named eth%d", c.Index, c.Index)
+ }
+ if c.TAP == "" || c.Network == "" {
+ return errors.New("network interface requires TAP and network names")
+ }
+ if _, err := net.ParseMAC(c.MAC); err != nil {
+ return fmt.Errorf("network interface MAC %q: %w", c.MAC, err)
+ }
+ if c.Queues < 2 || c.Queues%2 != 0 || c.QueueSize <= 0 {
+ return errors.New("network interface requires an even queue count of at least two and a positive queue size")
+ }
+ if c.IPv4 != nil {
+ if err := c.IPv4.Validate(); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// NetworkSetup is the complete durable network state of one sandbox. Its zero
+// value represents a sandbox created without networking.
+type NetworkSetup struct {
+ // Backend selects the provider used by later lifecycle operations.
+ Backend NetworkBackend
+ // Namespace is the absolute Linux network namespace path containing the
+ // CNI interfaces and TAP devices.
+ Namespace string
+ // Interfaces are ordered by their stable NIC index.
+ Interfaces []NetworkInterface
+}
+
+// Validate accepts the disabled zero value and otherwise checks a complete,
+// deterministic provider handoff.
+func (s NetworkSetup) Validate() error {
+ if s.Backend == "" {
+ if s.Namespace != "" || len(s.Interfaces) != 0 {
+ return errors.New("network setup without a backend must be empty")
+ }
+ return nil
+ }
+ if err := s.Backend.Validate(); err != nil {
+ return err
+ }
+ if !filepath.IsAbs(s.Namespace) {
+ return errors.New("network namespace must be an absolute path")
+ }
+ seen := make(map[int]struct{}, len(s.Interfaces))
+ previous := -1
+ for _, networkInterface := range s.Interfaces {
+ if err := networkInterface.Validate(); err != nil {
+ return fmt.Errorf("network interface %d: %w", networkInterface.Index, err)
+ }
+ if _, exists := seen[networkInterface.Index]; exists {
+ return fmt.Errorf("network interface index %d is duplicated", networkInterface.Index)
+ }
+ if networkInterface.Index <= previous {
+ return errors.New("network interfaces must be ordered by increasing index")
+ }
+ seen[networkInterface.Index] = struct{}{}
+ previous = networkInterface.Index
+ }
+ return nil
+}
diff --git a/types/network_test.go b/types/network_test.go
new file mode 100644
index 0000000..b6902d2
--- /dev/null
+++ b/types/network_test.go
@@ -0,0 +1,31 @@
+package types
+
+import "testing"
+
+func TestNetworkSetupValidatesDurableHandoff(t *testing.T) {
+ setup := NetworkSetup{
+ Backend: NetworkBackendCNI,
+ Namespace: "/var/run/netns/kb-sandbox",
+ Interfaces: []NetworkInterface{{
+ Index: 0, Name: "eth0", TAP: "tap12345678-0", MAC: "02:00:00:00:00:01",
+ Queues: 4, QueueSize: 512, Network: "bridge",
+ IPv4: &IPv4Config{Address: "10.42.0.7", Gateway: "10.42.0.1", Prefix: 24},
+ }},
+ }
+ if err := setup.Validate(); err != nil {
+ t.Fatal(err)
+ }
+ setup.Interfaces = append(setup.Interfaces, setup.Interfaces[0])
+ if err := setup.Validate(); err == nil {
+ t.Fatal("duplicate network interface was accepted")
+ }
+}
+
+func TestNetworkSetupZeroValueDisablesNetworking(t *testing.T) {
+ if err := (NetworkSetup{}).Validate(); err != nil {
+ t.Fatal(err)
+ }
+ if err := (NetworkSetup{Namespace: "/var/run/netns/unowned"}).Validate(); err == nil {
+ t.Fatal("namespace without backend was accepted")
+ }
+}
From a536d924c8db75c8a125aaa314d678182c5c4544 Mon Sep 17 00:00:00 2001
From: YoungLH <974840768@qq.com>
Date: Tue, 22 Sep 2026 10:06:58 +0800
Subject: [PATCH 39/45] feat: provision sandbox networks
---
cli/root.go | 4 +
cli/root_test.go | 13 ++++
cli/sandbox/create.go | 15 +++-
cli/sandbox/create_test.go | 35 ++++++++-
cli/sandbox/output.go | 84 +++++++++++++++++++--
cli/sandbox/remove_test.go | 2 +-
core/sandbox.go | 28 ++++++-
core/sandbox_runtime.go | 7 ++
core/sandbox_runtime_test.go | 24 ++++++
core/sandbox_storage.go | 85 ++++++++++++++++-----
core/sandbox_storage_test.go | 116 +++++++++++++++++++++++++++++
core/sandbox_test.go | 68 ++++++++++++++++-
sandbox/catalog/store.go | 135 ++++++++++++++++++++++++++++++++--
sandbox/catalog/store_test.go | 51 ++++++++++++-
types/sandbox.go | 46 +++++++++++-
types/sandbox_test.go | 3 +
16 files changed, 672 insertions(+), 44 deletions(-)
diff --git a/cli/root.go b/cli/root.go
index 16c06c9..bae0f45 100644
--- a/cli/root.go
+++ b/cli/root.go
@@ -125,8 +125,12 @@ func newRootCommand() (*cobra.Command, error) {
flags.String("root-dir", configuration.Paths.Data, "persistent data directory")
flags.String("run-dir", configuration.Paths.Run, "runtime state directory")
flags.String("log-dir", configuration.Paths.Log, "log directory")
+ flags.String("cni-conf-dir", configuration.Network.CNI.ConfDir, "CNI .conflist directory")
+ flags.String("cni-bin-dir", configuration.Network.CNI.BinDir, "CNI plugin binary directory")
+ flags.String("dns", configuration.Network.DNS, "comma-separated guest DNS servers")
for key, name := range map[string]string{
"paths.data": "root-dir", "paths.run": "run-dir", "paths.log": "log-dir",
+ "network.cni.conf_dir": "cni-conf-dir", "network.cni.bin_dir": "cni-bin-dir", "network.dns": "dns",
} {
if err := loader.BindFlag(key, flags.Lookup(name)); err != nil {
return nil, fmt.Errorf("bind --%s: %w", name, err)
diff --git a/cli/root_test.go b/cli/root_test.go
index 695ac35..a31fd2a 100644
--- a/cli/root_test.go
+++ b/cli/root_test.go
@@ -112,6 +112,19 @@ func TestInvalidConfigurationUsesDomainExitCode(t *testing.T) {
}
}
+func TestNetworkFlagsAreBoundIntoConfiguration(t *testing.T) {
+ for _, args := range [][]string{
+ {"--cni-conf-dir", "relative", "version"},
+ {"--cni-bin-dir", "relative", "version"},
+ {"--dns", "not-an-ip", "version"},
+ } {
+ err := Execute(t.Context(), args, &bytes.Buffer{}, &bytes.Buffer{})
+ if got := ExitCode(err); got != 5 {
+ t.Fatalf("Execute(%v) exit = %d, want 5; error = %v", args, got, err)
+ }
+ }
+}
+
func TestImageAndUsageExitCodes(t *testing.T) {
base := t.TempDir()
flags := []string{"--root-dir", filepath.Join(base, "data"), "--run-dir", filepath.Join(base, "run"), "--log-dir", filepath.Join(base, "log")}
diff --git a/cli/sandbox/create.go b/cli/sandbox/create.go
index 09fd0e2..495ca99 100644
--- a/cli/sandbox/create.go
+++ b/cli/sandbox/create.go
@@ -26,6 +26,8 @@ func NewCreateCommand(configuration configProvider) *cobra.Command {
cpus := types.DefaultSandboxCPUs
memory := "1GiB"
storageSize := "10GiB"
+ nics := 1
+ networkName := ""
asJSON := false
command := &cobra.Command{
Use: "create IMAGE",
@@ -49,7 +51,16 @@ func NewCreateCommand(configuration configProvider) *cobra.Command {
if storageBytes < types.MinSandboxStorage {
return invalidFlag("storage", fmt.Errorf("must be at least %d bytes", types.MinSandboxStorage))
}
- sandboxConfig := types.SandboxConfig{Name: name, CPUs: cpus, Memory: memoryBytes, Storage: storageBytes}
+ if nics < 0 || nics > types.MaxSandboxNICs {
+ return invalidFlag("nics", fmt.Errorf("must be between 0 and %d", types.MaxSandboxNICs))
+ }
+ if nics == 0 && networkName != "" {
+ return invalidFlag("network", errors.New("requires at least one NIC"))
+ }
+ sandboxConfig := types.SandboxConfig{
+ Name: name, CPUs: cpus, Memory: memoryBytes, Storage: storageBytes,
+ NICs: nics, NetworkName: networkName,
+ }
if err := sandboxConfig.Validate(); err != nil {
return err
}
@@ -82,6 +93,8 @@ func NewCreateCommand(configuration configProvider) *cobra.Command {
command.Flags().Uint32Var(&cpus, "cpus", cpus, "number of virtual CPUs")
command.Flags().StringVar(&memory, "memory", memory, "guest memory (for example 1GiB)")
command.Flags().StringVar(&storageSize, "storage", storageSize, "logical sparse COW size (minimum 10GiB)")
+ command.Flags().IntVar(&nics, "nics", nics, "number of network interfaces (0 disables networking)")
+ command.Flags().StringVar(&networkName, "network", networkName, "CNI network name (empty selects the default)")
command.Flags().BoolVar(&asJSON, "json", false, "print the created sandbox as indented JSON")
return command
}
diff --git a/cli/sandbox/create_test.go b/cli/sandbox/create_test.go
index 399ecd2..c74e44e 100644
--- a/cli/sandbox/create_test.go
+++ b/cli/sandbox/create_test.go
@@ -53,6 +53,8 @@ func TestCreateCommandMapsResourceValidationToFlags(t *testing.T) {
{name: "CPUs", args: []string{"demo", "--name", "box", "--cpus", "0"}, flag: "--cpus"},
{name: "memory", args: []string{"demo", "--name", "box", "--memory", "1MiB"}, flag: "--memory"},
{name: "storage", args: []string{"demo", "--name", "box", "--storage", "1GiB"}, flag: "--storage"},
+ {name: "NICs", args: []string{"demo", "--name", "box", "--nics", "-1"}, flag: "--nics"},
+ {name: "network without NIC", args: []string{"demo", "--name", "box", "--nics", "0", "--network", "bridge"}, flag: "--network"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
@@ -78,10 +80,21 @@ func TestWriteResultUsesFullIDAndIndentedJSON(t *testing.T) {
t.Fatal(err)
}
record := types.Sandbox{
- ID: types.SandboxID("123e4567-e89b-42d3-a456-426614174000"),
- Config: types.SandboxConfig{Name: "box", CPUs: 2, Memory: types.DefaultSandboxMemory, Storage: types.DefaultSandboxStorage},
+ ID: types.SandboxID("123e4567-e89b-42d3-a456-426614174000"),
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 2, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1, NetworkName: "bridge",
+ },
ImageDigest: digest, VMM: types.VMMCloudHypervisor, State: types.SandboxStateCreated, Generation: 2,
CreatedAt: time.Date(2026, 9, 15, 10, 0, 0, 0, time.UTC), UpdatedAt: time.Date(2026, 9, 15, 10, 0, 1, 0, time.UTC),
+ Network: types.NetworkSetup{
+ Backend: types.NetworkBackendCNI, Namespace: "/var/run/netns/kumabox-test",
+ Interfaces: []types.NetworkInterface{{
+ Index: 0, Name: "eth0", TAP: "tap0", MAC: "02:00:00:00:00:01",
+ Queues: 4, QueueSize: 512, Network: "bridge",
+ IPv4: &types.IPv4Config{Address: "10.42.0.2", Gateway: "10.42.0.1", Prefix: 24},
+ }},
+ },
}
var text bytes.Buffer
if err := writeSandboxResult(&text, record, false); err != nil {
@@ -97,6 +110,22 @@ func TestWriteResultUsesFullIDAndIndentedJSON(t *testing.T) {
if !strings.Contains(jsonOut.String(), "\n \"id\":") || !strings.Contains(jsonOut.String(), "\"state\": \"created\"") || !strings.HasSuffix(jsonOut.String(), "\n") {
t.Fatalf("JSON result = %q", jsonOut.String())
}
+ var output sandboxOutput
+ if err := json.Unmarshal(jsonOut.Bytes(), &output); err != nil {
+ t.Fatal(err)
+ }
+ if output.NICs != 1 || output.NetworkName != "bridge" || output.Network == nil ||
+ len(output.Network.Interfaces) != 1 || output.Network.Interfaces[0].IPv4 == nil {
+ t.Fatalf("network JSON output = %+v", output)
+ }
+}
+
+func TestCreateCommandDefaultsToOneNIC(t *testing.T) {
+ command := NewCreateCommand(func() config.Config { return config.Config{} })
+ flag := command.Flags().Lookup("nics")
+ if flag == nil || flag.DefValue != "1" {
+ t.Fatalf("--nics default = %+v, want 1", flag)
+ }
}
func TestCreateProgressReportsCommittedOutputFailure(t *testing.T) {
@@ -123,7 +152,7 @@ func TestCreateCommandPersistsCreatedSandboxAndFinalCOW(t *testing.T) {
seedImage(t, roots)
installFakeMKFS(t, base)
command := NewCreateCommand(func() config.Config { return sandboxTestConfig(roots) })
- command.SetArgs([]string{"demo", "--name", "box", "--cpus", "1", "--json"})
+ command.SetArgs([]string{"demo", "--name", "box", "--cpus", "1", "--nics", "0", "--json"})
var stdout, stderr bytes.Buffer
command.SetOut(&stdout)
command.SetErr(&stderr)
diff --git a/cli/sandbox/output.go b/cli/sandbox/output.go
index 87ee411..8e71c55 100644
--- a/cli/sandbox/output.go
+++ b/cli/sandbox/output.go
@@ -30,6 +30,12 @@ type sandboxOutput struct {
Memory int64 `json:"memory"`
// Storage is the logical sparse COW size in bytes.
Storage int64 `json:"storage"`
+ // NICs is the requested network interface count.
+ NICs int `json:"nics"`
+ // NetworkName is the resolved CNI network name.
+ NetworkName string `json:"network_name,omitempty"`
+ // Network is the resolved provider-to-VMM handoff.
+ Network *networkOutput `json:"network,omitempty"`
// Generation fences stale lifecycle transitions.
Generation uint64 `json:"generation"`
// Failure explains retained cleanup work for an error-state sandbox.
@@ -48,6 +54,46 @@ type sandboxFailureOutput struct {
Message string `json:"message"`
}
+// networkOutput is the stable JSON projection of resolved sandbox networking.
+type networkOutput struct {
+ // Backend identifies the provider that owns host network resources.
+ Backend string `json:"backend"`
+ // Namespace is the absolute network namespace path.
+ Namespace string `json:"namespace"`
+ // Interfaces lists NICs in stable guest index order.
+ Interfaces []networkInterfaceOutput `json:"interfaces"`
+}
+
+// networkInterfaceOutput describes one guest NIC and its host TAP endpoint.
+type networkInterfaceOutput struct {
+ // Index is the stable zero-based guest NIC position.
+ Index int `json:"index"`
+ // Name is the guest interface name.
+ Name string `json:"name"`
+ // TAP is the host-side device opened by the VMM.
+ TAP string `json:"tap"`
+ // MAC is the durable guest hardware address.
+ MAC string `json:"mac"`
+ // Queues is the total virtio RX and TX queue count.
+ Queues int `json:"queues"`
+ // QueueSize is the descriptor count for each queue.
+ QueueSize int `json:"queue_size"`
+ // Network is the resolved CNI conflist name.
+ Network string `json:"network"`
+ // IPv4 is the optional guest-visible IPv4 assignment.
+ IPv4 *ipv4Output `json:"ipv4,omitempty"`
+}
+
+// ipv4Output is the stable JSON projection of a guest IPv4 assignment.
+type ipv4Output struct {
+ // Address is the guest IPv4 address without a prefix.
+ Address string `json:"address"`
+ // Gateway is the optional default gateway.
+ Gateway string `json:"gateway,omitempty"`
+ // Prefix is the CIDR prefix length.
+ Prefix int `json:"prefix"`
+}
+
// removeOutput is the stable JSON result for a completed sandbox removal.
type removeOutput struct {
// ID is the immutable identity whose resources were deleted.
@@ -61,8 +107,12 @@ func sandboxResult(sandbox types.Sandbox) sandboxOutput {
result := sandboxOutput{
ID: sandbox.ID.String(), Name: sandbox.Config.Name, ImageDigest: sandbox.ImageDigest.String(), VMM: string(sandbox.VMM),
State: string(sandbox.State), CPUs: sandbox.Config.CPUs, Memory: sandbox.Config.Memory,
- Storage: sandbox.Config.Storage, Generation: sandbox.Generation,
- CreatedAt: sandbox.CreatedAt.UTC(), UpdatedAt: sandbox.UpdatedAt.UTC(),
+ Storage: sandbox.Config.Storage, NICs: sandbox.Config.NICs, NetworkName: sandbox.Config.NetworkName,
+ Generation: sandbox.Generation,
+ CreatedAt: sandbox.CreatedAt.UTC(), UpdatedAt: sandbox.UpdatedAt.UTC(),
+ }
+ if sandbox.Network.Backend != "" {
+ result.Network = networkResult(sandbox.Network)
}
if sandbox.Failure != nil {
result.Failure = &sandboxFailureOutput{Phase: sandbox.Failure.Phase, Message: sandbox.Failure.Message}
@@ -70,6 +120,29 @@ func sandboxResult(sandbox types.Sandbox) sandboxOutput {
return result
}
+func networkResult(setup types.NetworkSetup) *networkOutput {
+ result := &networkOutput{
+ Backend: string(setup.Backend), Namespace: setup.Namespace,
+ Interfaces: make([]networkInterfaceOutput, 0, len(setup.Interfaces)),
+ }
+ for _, networkInterface := range setup.Interfaces {
+ item := networkInterfaceOutput{
+ Index: networkInterface.Index, Name: networkInterface.Name, TAP: networkInterface.TAP,
+ MAC: networkInterface.MAC, Queues: networkInterface.Queues, QueueSize: networkInterface.QueueSize,
+ Network: networkInterface.Network,
+ }
+ if networkInterface.IPv4 != nil {
+ item.IPv4 = &ipv4Output{
+ Address: networkInterface.IPv4.Address,
+ Gateway: networkInterface.IPv4.Gateway,
+ Prefix: networkInterface.IPv4.Prefix,
+ }
+ }
+ result.Interfaces = append(result.Interfaces, item)
+ }
+ return result
+}
+
// writeSandboxJSON emits one complete sandbox as indented JSON.
func writeSandboxJSON(writer io.Writer, sandbox types.Sandbox) error {
encoder := json.NewEncoder(writer)
@@ -121,13 +194,14 @@ func writeSandboxIDs(writer io.Writer, records []types.Sandbox) error {
// writeSandboxTable renders headers for empty results and keeps IDs actionable.
func writeSandboxTable(writer io.Writer, records []types.Sandbox) error {
table := tabwriter.NewWriter(writer, 0, 4, 2, ' ', 0)
- if _, err := fmt.Fprintln(table, "SANDBOX ID\tNAME\tIMAGE ID\tVMM\tSTATE\tCPUS\tMEMORY\tSTORAGE\tCREATED"); err != nil {
+ if _, err := fmt.Fprintln(table, "SANDBOX ID\tNAME\tIMAGE ID\tVMM\tSTATE\tCPUS\tMEMORY\tSTORAGE\tNICS\tNETWORK\tCREATED"); err != nil {
return err
}
for _, record := range records {
- if _, err := fmt.Fprintf(table, "%s\t%s\t%s\t%s\t%s\t%d\t%s\t%s\t%s\n",
+ if _, err := fmt.Fprintf(table, "%s\t%s\t%s\t%s\t%s\t%d\t%s\t%s\t%d\t%s\t%s\n",
record.ID, record.Config.Name, record.ImageDigest.Hex()[:12], record.VMM, record.State, record.Config.CPUs,
- formatIECBytes(record.Config.Memory), formatIECBytes(record.Config.Storage),
+ formatIECBytes(record.Config.Memory), formatIECBytes(record.Config.Storage), record.Config.NICs,
+ record.Config.NetworkName,
record.CreatedAt.UTC().Format(time.RFC3339),
); err != nil {
return err
diff --git a/cli/sandbox/remove_test.go b/cli/sandbox/remove_test.go
index 32812e9..a65be28 100644
--- a/cli/sandbox/remove_test.go
+++ b/cli/sandbox/remove_test.go
@@ -111,7 +111,7 @@ func TestRemoveCommandClosesCreateAndImageReferenceLifecycle(t *testing.T) {
func executeCreate(t *testing.T, roots storage.Roots, name string) types.SandboxID {
t.Helper()
command := NewCreateCommand(func() config.Config { return sandboxTestConfig(roots) })
- command.SetArgs([]string{"demo", "--name", name, "--cpus", "1"})
+ command.SetArgs([]string{"demo", "--name", name, "--cpus", "1", "--nics", "0"})
var stdout, stderr bytes.Buffer
command.SetOut(&stdout)
command.SetErr(&stderr)
diff --git a/core/sandbox.go b/core/sandbox.go
index 4a14608..d64a2a4 100644
--- a/core/sandbox.go
+++ b/core/sandbox.go
@@ -12,8 +12,11 @@ import (
imagecatalog "github.com/kumabox/kumabox/images/catalog"
"github.com/kumabox/kumabox/metadata"
"github.com/kumabox/kumabox/metadata/sqlite"
+ "github.com/kumabox/kumabox/network"
+ "github.com/kumabox/kumabox/network/cni"
"github.com/kumabox/kumabox/sandbox"
sandboxcatalog "github.com/kumabox/kumabox/sandbox/catalog"
+ "github.com/kumabox/kumabox/storage"
"github.com/kumabox/kumabox/types"
"github.com/kumabox/kumabox/vmm"
)
@@ -38,7 +41,7 @@ type imageGuard interface {
// so the service receives it once instead of under several role aliases.
type sandboxCatalog interface {
Reserve(context.Context, string, types.Digest, types.Sandbox) error
- MarkCreated(context.Context, types.SandboxID, uint64, time.Time) (types.Sandbox, error)
+ MarkCreated(context.Context, types.SandboxID, uint64, types.NetworkSetup, time.Time) (types.Sandbox, error)
MarkError(context.Context, types.SandboxID, uint64, types.SandboxFailure, time.Time) (types.Sandbox, error)
Forget(context.Context, types.SandboxID, uint64) error
Resolve(context.Context, string) (types.Sandbox, error)
@@ -74,6 +77,8 @@ type sandboxDependencies struct {
catalog sandboxCatalog
// disks prepares and cleans sandbox-owned writable disks.
disks disk.Backend
+ // networks owns sandbox network namespaces, CNI allocations, and TAP devices.
+ networks network.Provider
// imagePaths derives immutable artifacts after the image guard verifies them.
imagePaths images.Paths
// runtimes route persisted VMM identities to process adapters.
@@ -99,7 +104,7 @@ type SandboxService struct {
// newSandboxService validates and records the explicit capabilities needed by
// sandbox commands. Defaults are limited to deterministic process-local seams.
func newSandboxService(dependencies sandboxDependencies) (*SandboxService, error) {
- if dependencies.images == nil || dependencies.catalog == nil || dependencies.disks == nil || dependencies.runtimes.Len() == 0 {
+ if dependencies.images == nil || dependencies.catalog == nil || dependencies.disks == nil || dependencies.networks == nil || dependencies.runtimes.Len() == 0 {
return nil, errors.New("sandbox service adapters are incomplete")
}
if dependencies.cleanupTimeout <= 0 {
@@ -160,12 +165,27 @@ func OpenSandbox(ctx context.Context, configuration config.Config, reporter Sand
if err != nil {
return nil, err
}
+ cacheDir, err := storage.Join(configuration.Paths.Data, "cni", "cache")
+ if err != nil {
+ return nil, errors.Join(err, store.Close())
+ }
+ networks, err := cni.New(cni.Options{
+ ConfDir: configuration.Network.CNI.ConfDir,
+ BinDir: configuration.Network.CNI.BinDir,
+ CacheDir: cacheDir,
+ NamespacePrefix: configuration.Network.NamespacePrefix(),
+ CleanupTimeout: configuration.Network.CleanupTimeout,
+ }, store)
+ if err != nil {
+ return nil, errors.Join(err, store.Close())
+ }
imageCatalog := imagecatalog.New(store, imagecatalog.WithImageUsage(sandboxcatalog.Usage{}))
sandboxCatalog := sandboxcatalog.New(store, imagecatalog.Reader{})
service, err := newSandboxService(sandboxDependencies{
paths: sandboxPaths, imagePaths: imagePaths, images: images.NewGuard(imagePaths, imageCatalog),
- catalog: sandboxCatalog, disks: disks, runtimes: runtimes, reporter: reporter,
- store: store, defaultVMM: defaultVMM, cleanupTimeout: configuration.Sandbox.CleanupTimeout,
+ catalog: sandboxCatalog, disks: disks, networks: networks, runtimes: runtimes, reporter: reporter,
+ store: store, defaultVMM: defaultVMM,
+ cleanupTimeout: max(configuration.Sandbox.CleanupTimeout, configuration.Network.CleanupTimeout),
})
if err != nil {
return nil, errors.Join(err, store.Close())
diff --git a/core/sandbox_runtime.go b/core/sandbox_runtime.go
index 524486a..66ad391 100644
--- a/core/sandbox_runtime.go
+++ b/core/sandbox_runtime.go
@@ -59,6 +59,13 @@ func (s *SandboxService) Start(ctx context.Context, reference string) (result ty
if err != nil {
return types.Sandbox{}, err
}
+ if record.Config.NICs > 0 || record.Network.Backend != "" {
+ return record, errdefs.New(
+ errdefs.ClassInvalid,
+ errdefs.CodeHostIncompatible,
+ errors.New("starting a networked sandbox is not supported until VMM network attachment is available"),
+ )
+ }
backend, err := s.dependencies.runtimes.Backend(record.VMM)
if err != nil {
return record, err
diff --git a/core/sandbox_runtime_test.go b/core/sandbox_runtime_test.go
index 594a598..2d6c65f 100644
--- a/core/sandbox_runtime_test.go
+++ b/core/sandbox_runtime_test.go
@@ -76,6 +76,30 @@ func TestStartCommitsRunningOnlyAfterLaunchReadiness(t *testing.T) {
}
}
+func TestStartRejectsNetworkedSandboxBeforeRuntimeRecovery(t *testing.T) {
+ service, steps := newTestSandboxService(t, nil)
+ if _, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ }); err != nil {
+ t.Fatal(err)
+ }
+ *steps = nil
+ if _, err := service.Start(t.Context(), "box"); err == nil {
+ t.Fatal("Start accepted a networked sandbox before VMM network attachment exists")
+ } else if code, ok := errdefs.CodeOf(err); !ok || code != errdefs.CodeHostIncompatible {
+ t.Fatalf("Start error code = %q, %v; want %q", code, err, errdefs.CodeHostIncompatible)
+ }
+ if got := *steps; !reflect.DeepEqual(got, []string{
+ "status:resolving sandbox", "resolve", "status:waiting for sandbox operation lock", "resolve",
+ }) {
+ t.Fatalf("Start touched runtime state before rejection: %v", got)
+ }
+}
+
func TestStartRecoversRunningProcessFromStartingState(t *testing.T) {
service, steps := newTestSandboxService(t, nil)
if _, err := service.Create(t.Context(), CreateSandboxRequest{
diff --git a/core/sandbox_storage.go b/core/sandbox_storage.go
index 42d7dcc..45265d0 100644
--- a/core/sandbox_storage.go
+++ b/core/sandbox_storage.go
@@ -8,17 +8,19 @@ import (
"github.com/kumabox/kumabox/errdefs"
filelock "github.com/kumabox/kumabox/lock/flock"
+ "github.com/kumabox/kumabox/network"
"github.com/kumabox/kumabox/types"
)
-// Create reserves identity and image usage before preparing the private disk.
-// Only the final generation-fenced transition makes the disk startable.
+// Create reserves identity and image usage before preparing private host
+// resources. Only the final generation-fenced transition publishes the
+// resolved network handoff and makes the disk startable.
//
-// validate -> ID lock -> image locks + reservation -> sparse ext4 COW -> Created
-// | |
-// +---- failure cleanup <---+
+// validate -> reserve -> CNI namespace + NICs -> sparse ext4 COW -> Created
+// | | |
+// +<------ detached failure cleanup <-------+
func (s *SandboxService) Create(ctx context.Context, request CreateSandboxRequest) (result types.Sandbox, returnErr error) {
- if s == nil || s.dependencies.images == nil || s.dependencies.catalog == nil || s.dependencies.disks == nil || s.dependencies.runtimes.Len() == 0 || s.dependencies.reporter == nil || s.dependencies.newID == nil || s.dependencies.now == nil || s.dependencies.cleanupTimeout <= 0 {
+ if s == nil || s.dependencies.images == nil || s.dependencies.catalog == nil || s.dependencies.disks == nil || s.dependencies.networks == nil || s.dependencies.runtimes.Len() == 0 || s.dependencies.reporter == nil || s.dependencies.newID == nil || s.dependencies.now == nil || s.dependencies.cleanupTimeout <= 0 {
return types.Sandbox{}, errors.New("sandbox service is not configured")
}
if request.ImageReference == "" {
@@ -81,6 +83,37 @@ func (s *SandboxService) Create(ctx context.Context, request CreateSandboxReques
}
return types.Sandbox{}, errdefs.Context(err, "create sandbox", request.Config.Name, "reserve", "check the image and sandbox name", false)
}
+ setup := types.NetworkSetup{}
+ if request.Config.NICs > 0 {
+ if err := s.dependencies.reporter.Status("preparing sandbox network"); err != nil {
+ return types.Sandbox{}, s.compensate(ctx, record, "report", err)
+ }
+ namespace, err := s.dependencies.networks.Prepare(ctx, id)
+ if err != nil {
+ return types.Sandbox{}, s.compensate(ctx, record, "network prepare", err)
+ }
+ if err := s.dependencies.reporter.Status("allocating sandbox network interfaces"); err != nil {
+ return types.Sandbox{}, s.compensate(ctx, record, "report", err)
+ }
+ specs := network.AddRange(0, request.Config.NICs)
+ queues := network.QueueCount(request.Config.CPUs)
+ for index := range specs {
+ specs[index].Queues = queues
+ }
+ interfaces, err := s.dependencies.networks.Add(ctx, id, request.Config.NetworkName, specs...)
+ if err != nil {
+ return types.Sandbox{}, s.compensate(ctx, record, "network add", err)
+ }
+ setup = types.NetworkSetup{Backend: s.dependencies.networks.Type(), Namespace: namespace, Interfaces: interfaces}
+ if err := setup.Validate(); err != nil {
+ return types.Sandbox{}, s.compensate(ctx, record, "network result", err)
+ }
+ if len(interfaces) != request.Config.NICs {
+ return types.Sandbox{}, s.compensate(ctx, record, "network result", fmt.Errorf("network provider returned %d interfaces, expected %d", len(interfaces), request.Config.NICs))
+ }
+ record.Network = setup
+ record.Config.NetworkName = interfaces[0].Network
+ }
if err := s.dependencies.reporter.Status("creating sparse ext4 disk"); err != nil {
return types.Sandbox{}, s.compensate(ctx, record, "report", err)
}
@@ -90,7 +123,7 @@ func (s *SandboxService) Create(ctx context.Context, request CreateSandboxReques
if err := s.dependencies.reporter.Status("committing created state"); err != nil {
return types.Sandbox{}, s.compensate(ctx, record, "report", err)
}
- created, err := s.dependencies.catalog.MarkCreated(ctx, id, record.Generation, s.dependencies.now().UTC())
+ created, err := s.dependencies.catalog.MarkCreated(ctx, id, record.Generation, setup, s.dependencies.now().UTC())
if err != nil {
return types.Sandbox{}, s.compensate(ctx, record, "commit", err)
}
@@ -101,14 +134,14 @@ func (s *SandboxService) Create(ctx context.Context, request CreateSandboxReques
return created, nil
}
-// Remove records cleanup intent before deleting the COW directory and releases
-// the name and image reference only after filesystem cleanup succeeds.
+// Remove records cleanup intent before deleting every owned host resource and
+// releases the name and image reference only after cleanup succeeds.
//
-// resolve -> sandbox lock -> Deleting -> remove files -> forget record + name
-// | |
-// +---- retry resumes here <---+
+// resolve -> sandbox lock -> Deleting -> disk -> network -> logs -> finalize
+// | |
+// +-------- retry resumes ---------+
func (s *SandboxService) Remove(ctx context.Context, reference string) (result types.Sandbox, returnErr error) {
- if s == nil || s.dependencies.catalog == nil || s.dependencies.disks == nil || s.dependencies.runtimes.Len() == 0 || s.dependencies.reporter == nil || s.dependencies.now == nil {
+ if s == nil || s.dependencies.catalog == nil || s.dependencies.disks == nil || s.dependencies.networks == nil || s.dependencies.runtimes.Len() == 0 || s.dependencies.reporter == nil || s.dependencies.now == nil {
return types.Sandbox{}, errors.New("sandbox service is not configured")
}
if reference == "" {
@@ -164,6 +197,14 @@ func (s *SandboxService) Remove(ctx context.Context, reference string) (result t
if err := s.dependencies.disks.Remove(ctx, deleting.ID); err != nil {
return deleting, errdefs.Context(err, "remove sandbox", reference, "disk cleanup", "retry removal to finish cleanup", true)
}
+ if deleting.Config.NICs > 0 || deleting.Network.Backend != "" {
+ if err := s.dependencies.reporter.Status("removing sandbox network"); err != nil {
+ return deleting, errdefs.Context(err, "remove sandbox", reference, "report", "retry removal to finish cleanup", true)
+ }
+ if err := s.dependencies.networks.Delete(ctx, deleting.ID); err != nil {
+ return deleting, errdefs.Context(err, "remove sandbox", reference, "network cleanup", "retry removal to finish cleanup", true)
+ }
+ }
if err := s.dependencies.reporter.Status("removing VMM logs"); err != nil {
return deleting, errdefs.Context(err, "remove sandbox", reference, "report", "retry removal to finish cleanup", true)
}
@@ -182,20 +223,24 @@ func (s *SandboxService) Remove(ctx context.Context, reference string) (result t
return deleting, nil
}
-// compensate removes the owned disk before forgetting the Creating reservation.
-// If cleanup cannot be proven complete, Error retains the resource owner and image pin.
+// compensate removes every potentially owned resource before forgetting the
+// Creating reservation. If cleanup cannot be proven complete, Error retains
+// the resource owner and image pin.
func (s *SandboxService) compensate(ctx context.Context, record types.Sandbox, phase string, cause error) error {
cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), s.dependencies.cleanupTimeout)
defer cancel()
- removeErr := s.dependencies.disks.Remove(cleanupCtx, record.ID)
- if removeErr == nil {
+ cleanupErr := s.dependencies.disks.Remove(cleanupCtx, record.ID)
+ if record.Config.NICs > 0 || record.Network.Backend != "" {
+ cleanupErr = errors.Join(cleanupErr, s.dependencies.networks.Delete(cleanupCtx, record.ID))
+ }
+ if cleanupErr == nil {
forgetErr := s.dependencies.catalog.Forget(cleanupCtx, record.ID, record.Generation)
if forgetErr == nil {
return errdefs.Context(cause, "create sandbox", record.Config.Name, phase, "fix the failure and retry", false)
}
- removeErr = forgetErr
+ cleanupErr = forgetErr
}
- failure := types.SandboxFailure{Phase: phase, Message: errors.Join(cause, removeErr).Error()}
+ failure := types.SandboxFailure{Phase: phase, Message: errors.Join(cause, cleanupErr).Error()}
_, markErr := s.dependencies.catalog.MarkError(cleanupCtx, record.ID, record.Generation, failure, s.dependencies.now().UTC())
- return errdefs.Context(errors.Join(cause, removeErr, markErr), "create sandbox", record.Config.Name, phase, "inspect or remove the retained error sandbox", false)
+ return errdefs.Context(errors.Join(cause, cleanupErr, markErr), "create sandbox", record.Config.Name, phase, "inspect or remove the retained error sandbox", false)
}
diff --git a/core/sandbox_storage_test.go b/core/sandbox_storage_test.go
index fa77c6d..76d652b 100644
--- a/core/sandbox_storage_test.go
+++ b/core/sandbox_storage_test.go
@@ -30,6 +30,83 @@ func TestCreateCommitsCreatedAfterDiskPreparation(t *testing.T) {
}
}
+func TestCreatePublishesResolvedNetworkWithCreatedState(t *testing.T) {
+ service, steps := newTestSandboxService(t, nil)
+ record, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 2, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 2,
+ },
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record.State != types.SandboxStateCreated || record.Config.NetworkName != "default" ||
+ record.Network.Backend != types.NetworkBackendCNI || len(record.Network.Interfaces) != 2 {
+ t.Fatalf("created network record = %+v", record)
+ }
+ networks := service.dependencies.networks.(*fakeNetwork)
+ if len(networks.specs) != 2 || networks.specs[0].Queues != 4 || networks.specs[1].Queues != 4 {
+ t.Fatalf("network specs = %+v", networks.specs)
+ }
+ want := []string{
+ "status:resolving and checking image", "verify", "reserve",
+ "status:preparing sandbox network", "network-prepare",
+ "status:allocating sandbox network interfaces", "network-add",
+ "status:creating sparse ext4 disk", "disk",
+ "status:committing created state", "created", "report",
+ }
+ if !reflect.DeepEqual(*steps, want) {
+ t.Fatalf("steps = %v, want %v", *steps, want)
+ }
+}
+
+func TestCreateNetworkFailureCleansResourcesBeforeForgettingReservation(t *testing.T) {
+ failure := errors.New("CNI add failed")
+ service, steps := newTestSandboxService(t, nil)
+ networks := service.dependencies.networks.(*fakeNetwork)
+ networks.addErr = failure
+ if _, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ }); !errors.Is(err, failure) {
+ t.Fatalf("Create error = %v", err)
+ }
+ wantTail := []string{"network-add", "remove", "network-delete", "forget"}
+ if got := (*steps)[len(*steps)-len(wantTail):]; !reflect.DeepEqual(got, wantTail) {
+ t.Fatalf("cleanup steps = %v, want %v", got, wantTail)
+ }
+}
+
+func TestCreateRetainsNetworkOwnerWhenCleanupFails(t *testing.T) {
+ addFailure := errors.New("CNI add failed")
+ deleteFailure := errors.New("CNI delete failed")
+ service, steps := newTestSandboxService(t, nil)
+ networks := service.dependencies.networks.(*fakeNetwork)
+ networks.addErr, networks.deleteErr = addFailure, deleteFailure
+ if _, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ }); !errors.Is(err, addFailure) || !errors.Is(err, deleteFailure) {
+ t.Fatalf("Create error = %v", err)
+ }
+ catalog := service.dependencies.catalog.(*fakeCatalog)
+ if catalog.record.State != types.SandboxStateError || catalog.record.Failure == nil || catalog.record.Failure.Phase != "network add" {
+ t.Fatalf("retained record = %+v", catalog.record)
+ }
+ wantTail := []string{"network-add", "remove", "network-delete", "error"}
+ if got := (*steps)[len(*steps)-len(wantTail):]; !reflect.DeepEqual(got, wantTail) {
+ t.Fatalf("cleanup steps = %v, want %v", got, wantTail)
+ }
+}
+
func TestCreateRejectsUnavailableVMMBeforeReservation(t *testing.T) {
service, steps := newTestSandboxService(t, nil)
_, err := service.Create(t.Context(), CreateSandboxRequest{
@@ -172,6 +249,45 @@ func TestRemoveFailureRetainsDeletingAndRetryFinishes(t *testing.T) {
}
}
+func TestRemoveNetworkFailureRetainsDeletingUntilRetry(t *testing.T) {
+ service, steps := newTestSandboxService(t, nil)
+ if _, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ }); err != nil {
+ t.Fatal(err)
+ }
+ failure := errors.New("network cleanup failed")
+ networks := service.dependencies.networks.(*fakeNetwork)
+ networks.deleteErr = failure
+ *steps = nil
+ if _, err := service.Remove(t.Context(), "box"); !errors.Is(err, failure) {
+ t.Fatalf("Remove error = %v", err)
+ }
+ catalog := service.dependencies.catalog.(*fakeCatalog)
+ if catalog.record.State != types.SandboxStateDeleting || catalog.deleted {
+ t.Fatalf("retained delete record = %+v, deleted=%v", catalog.record, catalog.deleted)
+ }
+ if got := strings.Join(*steps, ","); !strings.Contains(got, "remove,status:removing sandbox network,network-delete") || strings.Contains(got, "finalize") {
+ t.Fatalf("network cleanup ordering = %v", *steps)
+ }
+
+ networks.deleteErr = nil
+ *steps = nil
+ if _, err := service.Remove(t.Context(), "box"); err != nil {
+ t.Fatal(err)
+ }
+ if !catalog.deleted {
+ t.Fatal("retry did not finalize metadata")
+ }
+ if got := strings.Join(*steps, ","); !strings.Contains(got, "remove,status:removing sandbox network,network-delete") || !strings.Contains(got, "finalize") {
+ t.Fatalf("retry did not repeat idempotent cleanup: %v", *steps)
+ }
+}
+
func TestRemoveLogFailureRetainsDeletingUntilRetry(t *testing.T) {
service, steps := newTestSandboxService(t, nil)
if _, err := service.Create(t.Context(), CreateSandboxRequest{
diff --git a/core/sandbox_test.go b/core/sandbox_test.go
index 1cf2b1c..f1a3bea 100644
--- a/core/sandbox_test.go
+++ b/core/sandbox_test.go
@@ -3,6 +3,7 @@ package core
import (
"context"
"errors"
+ "fmt"
"io"
"path/filepath"
"runtime"
@@ -13,6 +14,7 @@ import (
"github.com/kumabox/kumabox/config"
"github.com/kumabox/kumabox/errdefs"
"github.com/kumabox/kumabox/images"
+ "github.com/kumabox/kumabox/network"
"github.com/kumabox/kumabox/sandbox"
"github.com/kumabox/kumabox/storage"
"github.com/kumabox/kumabox/types"
@@ -47,11 +49,15 @@ func (f *fakeCatalog) Reserve(_ context.Context, _ string, _ types.Digest, recor
return nil
}
-func (f *fakeCatalog) MarkCreated(_ context.Context, _ types.SandboxID, expected uint64, updated time.Time) (types.Sandbox, error) {
+func (f *fakeCatalog) MarkCreated(_ context.Context, _ types.SandboxID, expected uint64, setup types.NetworkSetup, updated time.Time) (types.Sandbox, error) {
*f.steps = append(*f.steps, "created")
if expected != f.record.Generation {
return types.Sandbox{}, errors.New("wrong generation")
}
+ f.record.Network = setup
+ if len(setup.Interfaces) > 0 {
+ f.record.Config.NetworkName = setup.Interfaces[0].Network
+ }
f.record.State, f.record.Generation, f.record.UpdatedAt = types.SandboxStateCreated, expected+1, updated
return f.record, nil
}
@@ -174,6 +180,62 @@ type fakeDisk struct {
remove error
}
+type fakeNetwork struct {
+ steps *[]string
+ prepareErr error
+ addErr error
+ deleteErr error
+ namespace string
+ interfaces []types.NetworkInterface
+ specs []network.AddSpec
+}
+
+func (*fakeNetwork) Type() types.NetworkBackend { return types.NetworkBackendCNI }
+
+func (f *fakeNetwork) Prepare(context.Context, types.SandboxID) (string, error) {
+ *f.steps = append(*f.steps, "network-prepare")
+ return f.namespace, f.prepareErr
+}
+
+func (f *fakeNetwork) Add(_ context.Context, _ types.SandboxID, networkName string, specs ...network.AddSpec) ([]types.NetworkInterface, error) {
+ *f.steps = append(*f.steps, "network-add")
+ f.specs = append([]network.AddSpec(nil), specs...)
+ if f.addErr != nil {
+ return nil, f.addErr
+ }
+ if len(f.interfaces) > 0 {
+ return append([]types.NetworkInterface(nil), f.interfaces...), nil
+ }
+ if networkName == "" {
+ networkName = "default"
+ }
+ result := make([]types.NetworkInterface, 0, len(specs))
+ for _, spec := range specs {
+ result = append(result, types.NetworkInterface{
+ Index: spec.Index, Name: fmt.Sprintf("eth%d", spec.Index), TAP: fmt.Sprintf("tap%d", spec.Index),
+ MAC: fmt.Sprintf("02:00:00:00:00:%02x", spec.Index+1), Queues: spec.Queues,
+ QueueSize: network.DefaultQueueSize, Network: networkName,
+ })
+ }
+ return result, nil
+}
+
+func (*fakeNetwork) Verify(context.Context, types.SandboxID, []types.NetworkInterface) error {
+ return nil
+}
+
+func (*fakeNetwork) Recover(context.Context, types.SandboxID, string, []types.NetworkInterface) ([]types.NetworkInterface, error) {
+ return nil, nil
+}
+
+func (*fakeNetwork) Quiesce(context.Context, types.SandboxID) error { return nil }
+func (*fakeNetwork) Unquiesce(context.Context, types.SandboxID) error { return nil }
+
+func (f *fakeNetwork) Delete(context.Context, types.SandboxID) error {
+ *f.steps = append(*f.steps, "network-delete")
+ return f.deleteErr
+}
+
func (f fakeDisk) Prepare(context.Context, types.SandboxID, int64) error {
*f.steps = append(*f.steps, "disk")
return f.prepare
@@ -344,7 +406,8 @@ func newTestSandboxService(t *testing.T, diskError error) (*SandboxService, *[]s
}
service, err := newSandboxService(sandboxDependencies{
paths: paths, imagePaths: imagePaths, images: fakeGuard{image: image, steps: &steps},
- catalog: catalog, disks: fakeDisk{steps: &steps, prepare: diskError}, runtimes: runtimes,
+ catalog: catalog, disks: fakeDisk{steps: &steps, prepare: diskError},
+ networks: &fakeNetwork{steps: &steps, namespace: "/var/run/netns/kumabox-test"}, runtimes: runtimes,
defaultVMM: types.VMMCloudHypervisor, cleanupTimeout: 10 * time.Second,
reporter: fakeReporter{steps: &steps},
newID: func() (types.SandboxID, error) { return fixedID, nil },
@@ -391,6 +454,7 @@ func TestNewSandboxServiceValidatesNamedDependencies(t *testing.T) {
{name: "image guard", mutate: func(dependencies *sandboxDependencies) { dependencies.images = nil }},
{name: "catalog", mutate: func(dependencies *sandboxDependencies) { dependencies.catalog = nil }},
{name: "disk backend", mutate: func(dependencies *sandboxDependencies) { dependencies.disks = nil }},
+ {name: "network provider", mutate: func(dependencies *sandboxDependencies) { dependencies.networks = nil }},
{name: "VMM registry", mutate: func(dependencies *sandboxDependencies) { dependencies.runtimes = nil }},
{name: "cleanup timeout", mutate: func(dependencies *sandboxDependencies) { dependencies.cleanupTimeout = 0 }},
{name: "default VMM", mutate: func(dependencies *sandboxDependencies) { dependencies.defaultVMM = types.VMMFirecracker }},
diff --git a/sandbox/catalog/store.go b/sandbox/catalog/store.go
index d97f979..158cdc9 100644
--- a/sandbox/catalog/store.go
+++ b/sandbox/catalog/store.go
@@ -61,6 +61,12 @@ type recordData struct {
Memory int64 `json:"memory"`
// Storage is logical COW capacity in bytes.
Storage int64 `json:"storage"`
+ // NICs is the immutable requested network interface count.
+ NICs int `json:"nics,omitempty"`
+ // NetworkName is the resolved CNI conflist name.
+ NetworkName string `json:"network_name,omitempty"`
+ // Network is the resolved provider-to-VMM handoff.
+ Network *networkData `json:"network,omitempty"`
// ImageDigest pins the canonical manifest record.
ImageDigest string `json:"image_digest"`
// VMM identifies the backend that owns runtime artifacts. Empty legacy
@@ -86,6 +92,33 @@ type failureData struct {
Message string `json:"message"`
}
+// networkData is the stable persisted form of one resolved network setup.
+type networkData struct {
+ Backend string `json:"backend"`
+ Namespace string `json:"namespace"`
+ Interfaces []networkInterfaceData `json:"interfaces"`
+}
+
+// networkInterfaceData stores one NIC without exposing adapter encoding tags
+// through the shared types package.
+type networkInterfaceData struct {
+ Index int `json:"index"`
+ Name string `json:"name"`
+ TAP string `json:"tap"`
+ MAC string `json:"mac"`
+ Queues int `json:"queues"`
+ QueueSize int `json:"queue_size"`
+ Network string `json:"network"`
+ IPv4 *ipv4Data `json:"ipv4,omitempty"`
+}
+
+// ipv4Data stores the optional guest-visible IPv4 assignment.
+type ipv4Data struct {
+ Address string `json:"address"`
+ Gateway string `json:"gateway,omitempty"`
+ Prefix int `json:"prefix"`
+}
+
// nameData is deliberately small so names can be checked without decoding aggregates.
type nameData struct {
// ID is the owner in CollectionSandboxes.
@@ -134,9 +167,49 @@ func (c *Store) Reserve(ctx context.Context, imageReference string, expected typ
return errdefs.Context(err, "reserve sandbox", record.Config.Name, "metadata", "choose another name or retry", false)
}
-// MarkCreated performs the create commit only when state and generation still match.
-func (c *Store) MarkCreated(ctx context.Context, id types.SandboxID, expected uint64, updated time.Time) (types.Sandbox, error) {
- return c.transition(ctx, id, expected, types.SandboxStateCreating, types.SandboxStateCreated, nil, updated)
+// MarkCreated atomically publishes resolved network state and the Created
+// transition only when state and generation still match.
+func (c *Store) MarkCreated(ctx context.Context, id types.SandboxID, expected uint64, setup types.NetworkSetup, updated time.Time) (types.Sandbox, error) {
+ if err := setup.Validate(); err != nil {
+ return types.Sandbox{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ var result types.Sandbox
+ err := c.store.Update(ctx, func(writer metadata.Writer) error {
+ record, err := load(ctx, writer, id)
+ if err != nil {
+ return err
+ }
+ if record.Generation != expected || record.State != types.SandboxStateCreating {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, fmt.Errorf("sandbox %s changed from expected Creating generation %d", id, expected))
+ }
+ if record.Config.NICs == 0 {
+ if setup.Backend != "" {
+ return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, errors.New("sandbox without NICs cannot commit network setup"))
+ }
+ } else {
+ if setup.Backend == "" || len(setup.Interfaces) != record.Config.NICs {
+ return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, errors.New("networked sandbox requires one resolved interface per requested NIC"))
+ }
+ resolved := setup.Interfaces[0].Network
+ if record.Config.NetworkName != "" && record.Config.NetworkName != resolved {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, errors.New("resolved network differs from the requested network"))
+ }
+ record.Config.NetworkName = resolved
+ }
+ record.Network = setup
+ record.State = types.SandboxStateCreated
+ record.Generation++
+ record.UpdatedAt = updated
+ if err := record.Validate(); err != nil {
+ return corrupt("sandbox create transition", err)
+ }
+ if err := putJSON(ctx, writer, CollectionSandboxes, id.String(), encode(record)); err != nil {
+ return err
+ }
+ result = record
+ return nil
+ })
+ return result, errdefs.Context(err, "create sandbox", id.String(), "mark created", "inspect the sandbox state before retrying", false)
}
// MarkError retains ownership and diagnostics when create cleanup cannot finish.
@@ -478,10 +551,14 @@ func deleteRecord(ctx context.Context, writer metadata.Writer, record types.Sand
func encode(record types.Sandbox) recordData {
data := recordData{
ID: record.ID.String(), Name: record.Config.Name, CPUs: record.Config.CPUs,
- Memory: record.Config.Memory, Storage: record.Config.Storage,
+ Memory: record.Config.Memory, Storage: record.Config.Storage, NICs: record.Config.NICs,
+ NetworkName: record.Config.NetworkName,
ImageDigest: record.ImageDigest.String(), VMM: string(record.VMM), State: string(record.State),
Generation: record.Generation, CreatedAt: record.CreatedAt, UpdatedAt: record.UpdatedAt,
}
+ if record.Network.Backend != "" {
+ data.Network = encodeNetwork(record.Network)
+ }
if record.Failure != nil {
data.Failure = &failureData{Phase: record.Failure.Phase, Message: record.Failure.Message}
}
@@ -506,10 +583,16 @@ func decode(raw []byte) (types.Sandbox, error) {
data.VMM = string(types.VMMCloudHypervisor)
}
record := types.Sandbox{
- ID: id, Config: types.SandboxConfig{Name: data.Name, CPUs: data.CPUs, Memory: data.Memory, Storage: data.Storage},
+ ID: id, Config: types.SandboxConfig{
+ Name: data.Name, CPUs: data.CPUs, Memory: data.Memory, Storage: data.Storage,
+ NICs: data.NICs, NetworkName: data.NetworkName,
+ },
ImageDigest: digest, VMM: types.VMMType(data.VMM), State: types.SandboxState(data.State), Generation: data.Generation,
CreatedAt: data.CreatedAt, UpdatedAt: data.UpdatedAt,
}
+ if data.Network != nil {
+ record.Network = decodeNetwork(*data.Network)
+ }
if data.Failure != nil {
record.Failure = &types.SandboxFailure{Phase: data.Failure.Phase, Message: data.Failure.Message}
}
@@ -519,6 +602,48 @@ func decode(raw []byte) (types.Sandbox, error) {
return record, nil
}
+func encodeNetwork(setup types.NetworkSetup) *networkData {
+ result := &networkData{
+ Backend: string(setup.Backend), Namespace: setup.Namespace,
+ Interfaces: make([]networkInterfaceData, 0, len(setup.Interfaces)),
+ }
+ for _, networkInterface := range setup.Interfaces {
+ data := networkInterfaceData{
+ Index: networkInterface.Index, Name: networkInterface.Name, TAP: networkInterface.TAP,
+ MAC: networkInterface.MAC, Queues: networkInterface.Queues, QueueSize: networkInterface.QueueSize,
+ Network: networkInterface.Network,
+ }
+ if networkInterface.IPv4 != nil {
+ data.IPv4 = &ipv4Data{
+ Address: networkInterface.IPv4.Address, Gateway: networkInterface.IPv4.Gateway,
+ Prefix: networkInterface.IPv4.Prefix,
+ }
+ }
+ result.Interfaces = append(result.Interfaces, data)
+ }
+ return result
+}
+
+func decodeNetwork(data networkData) types.NetworkSetup {
+ result := types.NetworkSetup{
+ Backend: types.NetworkBackend(data.Backend), Namespace: data.Namespace,
+ Interfaces: make([]types.NetworkInterface, 0, len(data.Interfaces)),
+ }
+ for _, item := range data.Interfaces {
+ networkInterface := types.NetworkInterface{
+ Index: item.Index, Name: item.Name, TAP: item.TAP, MAC: item.MAC,
+ Queues: item.Queues, QueueSize: item.QueueSize, Network: item.Network,
+ }
+ if item.IPv4 != nil {
+ networkInterface.IPv4 = &types.IPv4Config{
+ Address: item.IPv4.Address, Gateway: item.IPv4.Gateway, Prefix: item.IPv4.Prefix,
+ }
+ }
+ result.Interfaces = append(result.Interfaces, networkInterface)
+ }
+ return result
+}
+
// putJSON keeps all record writes consistently encoded.
func putJSON(ctx context.Context, writer metadata.Writer, collection metadata.Collection, key string, value any) error {
raw, err := json.Marshal(value)
diff --git a/sandbox/catalog/store_test.go b/sandbox/catalog/store_test.go
index 532493d..078e047 100644
--- a/sandbox/catalog/store_test.go
+++ b/sandbox/catalog/store_test.go
@@ -60,6 +60,53 @@ func TestResolveRejectsDanglingNameBinding(t *testing.T) {
}
}
+func TestMarkCreatedAtomicallyPublishesResolvedNetwork(t *testing.T) {
+ store, err := metadata.NewMemory(Collections())
+ if err != nil {
+ t.Fatal(err)
+ }
+ id := types.SandboxID("123e4567-e89b-42d3-a456-426614174000")
+ created := time.Date(2026, 9, 15, 10, 0, 0, 0, time.UTC)
+ record := types.Sandbox{
+ ID: id,
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 2, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ ImageDigest: testDigest(t, 'a'), VMM: types.VMMCloudHypervisor,
+ State: types.SandboxStateCreating, Generation: 1, CreatedAt: created, UpdatedAt: created,
+ }
+ if err := store.Update(t.Context(), func(writer metadata.Writer) error {
+ return putJSON(t.Context(), writer, CollectionSandboxes, id.String(), encode(record))
+ }); err != nil {
+ t.Fatal(err)
+ }
+ setup := types.NetworkSetup{
+ Backend: types.NetworkBackendCNI, Namespace: "/var/run/netns/kumabox-test",
+ Interfaces: []types.NetworkInterface{{
+ Index: 0, Name: "eth0", TAP: "tap0", MAC: "02:00:00:00:00:01",
+ Queues: 4, QueueSize: 512, Network: "bridge",
+ IPv4: &types.IPv4Config{Address: "10.42.0.2", Gateway: "10.42.0.1", Prefix: 24},
+ }},
+ }
+ createdRecord, err := New(store, nil).MarkCreated(t.Context(), id, 1, setup, created.Add(time.Second))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if createdRecord.State != types.SandboxStateCreated || createdRecord.Generation != 2 ||
+ createdRecord.Config.NetworkName != "bridge" || createdRecord.Network.Namespace != setup.Namespace ||
+ len(createdRecord.Network.Interfaces) != 1 || createdRecord.Network.Interfaces[0].IPv4 == nil {
+ t.Fatalf("created network record = %+v", createdRecord)
+ }
+ resolved, err := New(store, nil).Resolve(t.Context(), id.String())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if resolved.Config.NetworkName != "bridge" || resolved.Network.Interfaces[0].IPv4.Address != "10.42.0.2" {
+ t.Fatalf("persisted network record = %+v", resolved)
+ }
+}
+
func TestListReturnsValidatedRecordsNewestFirst(t *testing.T) {
store, err := metadata.NewMemory(Collections())
if err != nil {
@@ -158,14 +205,14 @@ func TestReservationPinsImageInsideRemovalTransaction(t *testing.T) {
if _, err := imageStore.Resolve(t.Context(), "demo"); err != nil {
t.Fatalf("referenced image removal did not roll back: %v", err)
}
- createdRecord, err := sandboxStore.MarkCreated(t.Context(), id, 1, created.Add(time.Second))
+ createdRecord, err := sandboxStore.MarkCreated(t.Context(), id, 1, types.NetworkSetup{}, created.Add(time.Second))
if err != nil {
t.Fatal(err)
}
if createdRecord.State != types.SandboxStateCreated || createdRecord.Generation != 2 {
t.Fatalf("created record = %+v", createdRecord)
}
- if _, err := sandboxStore.MarkCreated(t.Context(), id, 1, created.Add(2*time.Second)); err == nil {
+ if _, err := sandboxStore.MarkCreated(t.Context(), id, 1, types.NetworkSetup{}, created.Add(2*time.Second)); err == nil {
t.Fatal("stale generation transition succeeded")
} else if code, _ := errdefs.CodeOf(err); code != errdefs.CodeStateConflict {
t.Fatalf("stale transition error = %v", err)
diff --git a/types/sandbox.go b/types/sandbox.go
index cdf1d2e..05d6dc1 100644
--- a/types/sandbox.go
+++ b/types/sandbox.go
@@ -28,9 +28,14 @@ const (
MinSandboxStorage int64 = 10 << 30
// MaxSandboxCPUs bounds conversion to host-native integer APIs and unreasonable shapes.
MaxSandboxCPUs uint32 = 1024
+ // MaxSandboxNICs bounds host resource allocation from one create request.
+ MaxSandboxNICs = 64
)
-var validSandboxName = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}$`)
+var (
+ validSandboxName = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}$`)
+ validNetworkName = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._-]{0,62}$`)
+)
// VMMType identifies the virtual machine monitor that owns a sandbox's
// runtime. It is persisted so every later lifecycle operation selects the same
@@ -132,6 +137,11 @@ type SandboxConfig struct {
Memory int64
// Storage is the logical size of the sparse ext4 COW disk in bytes.
Storage int64
+ // NICs is the requested network interface count; zero disables networking.
+ NICs int
+ // NetworkName selects one CNI conflist. Empty selects the provider default
+ // and is replaced by the resolved name when creation commits.
+ NetworkName string
}
// Validate enforces the resource and naming contract before any persistent change.
@@ -148,6 +158,15 @@ func (c SandboxConfig) Validate() error {
if c.Storage < MinSandboxStorage {
return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, fmt.Errorf("storage must be at least %d bytes", MinSandboxStorage))
}
+ if c.NICs < 0 || c.NICs > MaxSandboxNICs {
+ return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, fmt.Errorf("NIC count must be between 0 and %d", MaxSandboxNICs))
+ }
+ if c.NICs == 0 && c.NetworkName != "" {
+ return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, errors.New("network name requires at least one NIC"))
+ }
+ if c.NetworkName != "" && !validNetworkName.MatchString(c.NetworkName) {
+ return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, fmt.Errorf("network name %q must match %s", c.NetworkName, validNetworkName))
+ }
return nil
}
@@ -203,6 +222,9 @@ type Sandbox struct {
ImageDigest Digest
// VMM selects the backend that owns this sandbox's runtime artifacts.
VMM VMMType
+ // Network is the resolved provider-to-VMM handoff. It remains empty while a
+ // networked sandbox is still Creating and cleanup may be incomplete.
+ Network NetworkSetup
// State controls which operations may consume owned resources.
State SandboxState
// Generation increments on every state transition and fences stale operations.
@@ -229,6 +251,28 @@ func (s Sandbox) Validate() error {
if err := s.VMM.Validate(); err != nil {
return err
}
+ if err := s.Network.Validate(); err != nil {
+ return err
+ }
+ if s.Config.NICs == 0 && s.Network.Backend != "" {
+ return errors.New("sandbox without NICs must not contain network setup")
+ }
+ if s.Network.Backend != "" {
+ if len(s.Network.Interfaces) != s.Config.NICs {
+ return fmt.Errorf("sandbox has %d network interfaces, expected %d", len(s.Network.Interfaces), s.Config.NICs)
+ }
+ for _, networkInterface := range s.Network.Interfaces {
+ if networkInterface.Network != s.Config.NetworkName {
+ return errors.New("sandbox network interface differs from the resolved network name")
+ }
+ }
+ } else if s.Config.NICs > 0 {
+ switch s.State {
+ case SandboxStateCreating, SandboxStateError, SandboxStateDeleting:
+ default:
+ return errors.New("networked sandbox state requires resolved network setup")
+ }
+ }
switch s.State {
case SandboxStateCreating, SandboxStateCreated, SandboxStateStarting, SandboxStateRunning,
SandboxStateStopping, SandboxStateStopped, SandboxStateError, SandboxStateDeleting:
diff --git a/types/sandbox_test.go b/types/sandbox_test.go
index 1fecf82..6f9ea23 100644
--- a/types/sandbox_test.go
+++ b/types/sandbox_test.go
@@ -31,6 +31,9 @@ func TestSandboxConfigValidationMatchesCreateContract(t *testing.T) {
{"cpus", SandboxConfig{Name: "demo", Memory: MinSandboxMemory, Storage: MinSandboxStorage}},
{"memory", SandboxConfig{Name: "demo", CPUs: 1, Memory: MinSandboxMemory - 1, Storage: MinSandboxStorage}},
{"storage", SandboxConfig{Name: "demo", CPUs: 1, Memory: MinSandboxMemory, Storage: MinSandboxStorage - 1}},
+ {"NIC count", SandboxConfig{Name: "demo", CPUs: 1, Memory: MinSandboxMemory, Storage: MinSandboxStorage, NICs: MaxSandboxNICs + 1}},
+ {"network without NIC", SandboxConfig{Name: "demo", CPUs: 1, Memory: MinSandboxMemory, Storage: MinSandboxStorage, NetworkName: "default"}},
+ {"network name", SandboxConfig{Name: "demo", CPUs: 1, Memory: MinSandboxMemory, Storage: MinSandboxStorage, NICs: 1, NetworkName: "bad/name"}},
} {
t.Run(test.name, func(t *testing.T) {
err := test.config.Validate()
From fbd0fe6fabf15af50a7444a63ab8d5508276f7df Mon Sep 17 00:00:00 2001
From: YoungLH <974840768@qq.com>
Date: Tue, 22 Sep 2026 14:08:18 +0800
Subject: [PATCH 40/45] feat: connect sandbox networking to VMM
---
Makefile | 1 +
core/sandbox.go | 39 +++++++-
core/sandbox_runtime.go | 68 ++++++++++---
core/sandbox_runtime_test.go | 140 +++++++++++++++++++++++++--
core/sandbox_storage.go | 34 +++++--
core/sandbox_storage_test.go | 9 +-
core/sandbox_test.go | 50 ++++++++--
network/registry.go | 69 +++++++++++++
network/registry_test.go | 66 +++++++++++++
oci-images/ubuntu/Dockerfile | 7 ++
oci-images/ubuntu/network.sh | 75 ++++++++++++++
oci-images/ubuntu/overlay.sh | 8 ++
types/network.go | 5 +-
types/network_test.go | 14 +++
vmm/cloudhypervisor/args.go | 14 +++
vmm/cloudhypervisor/args_test.go | 9 ++
vmm/cloudhypervisor/driver.go | 2 +-
vmm/cloudhypervisor/process_linux.go | 36 +++++++
vmm/cloudhypervisor/process_other.go | 2 +
vmm/vmm.go | 60 +++++++++++-
vmm/vmm_test.go | 20 ++++
21 files changed, 684 insertions(+), 44 deletions(-)
create mode 100644 network/registry.go
create mode 100644 network/registry_test.go
create mode 100644 oci-images/ubuntu/network.sh
diff --git a/Makefile b/Makefile
index 46576c7..d6b8880 100644
--- a/Makefile
+++ b/Makefile
@@ -81,6 +81,7 @@ test: vet ## Run tests with race detection and coverage
doctor-check: ## Check host and guest shell script syntax
bash -n scripts/kumabox-check.sh
sh -n oci-images/ubuntu/overlay.sh
+ sh -n oci-images/ubuntu/network.sh
race: ## Run all Go tests with race detection
go test -race ./...
diff --git a/core/sandbox.go b/core/sandbox.go
index d64a2a4..89485c4 100644
--- a/core/sandbox.go
+++ b/core/sandbox.go
@@ -77,8 +77,10 @@ type sandboxDependencies struct {
catalog sandboxCatalog
// disks prepares and cleans sandbox-owned writable disks.
disks disk.Backend
- // networks owns sandbox network namespaces, CNI allocations, and TAP devices.
- networks network.Provider
+ // networks routes persisted network identities to provider adapters.
+ networks *network.Registry
+ // defaultNetwork selects the provider for newly created networked sandboxes.
+ defaultNetwork types.NetworkBackend
// imagePaths derives immutable artifacts after the image guard verifies them.
imagePaths images.Paths
// runtimes route persisted VMM identities to process adapters.
@@ -87,6 +89,8 @@ type sandboxDependencies struct {
defaultVMM types.VMMType
// cleanupTimeout bounds compensation that outlives caller cancellation.
cleanupTimeout time.Duration
+ // dnsServers are rendered into static guest boot network parameters.
+ dnsServers []string
// reporter emits progress independently of command results.
reporter SandboxReporter
// newID and now are replaceable in same-package tests.
@@ -113,6 +117,9 @@ func newSandboxService(dependencies sandboxDependencies) (*SandboxService, error
if _, err := dependencies.runtimes.Backend(dependencies.defaultVMM); err != nil {
return nil, err
}
+ if _, err := dependencies.networks.Provider(dependencies.defaultNetwork); err != nil {
+ return nil, err
+ }
if dependencies.reporter == nil {
dependencies.reporter = discardReporter{}
}
@@ -135,6 +142,10 @@ func OpenSandbox(ctx context.Context, configuration config.Config, reporter Sand
if err := configuration.Validate(); err != nil {
return nil, err
}
+ dnsServers, err := configuration.Network.DNSServers()
+ if err != nil {
+ return nil, err
+ }
imagePaths, err := images.NewPaths(configuration.Paths)
if err != nil {
return nil, err
@@ -169,7 +180,7 @@ func OpenSandbox(ctx context.Context, configuration config.Config, reporter Sand
if err != nil {
return nil, errors.Join(err, store.Close())
}
- networks, err := cni.New(cni.Options{
+ cniProvider, err := cni.New(cni.Options{
ConfDir: configuration.Network.CNI.ConfDir,
BinDir: configuration.Network.CNI.BinDir,
CacheDir: cacheDir,
@@ -179,13 +190,18 @@ func OpenSandbox(ctx context.Context, configuration config.Config, reporter Sand
if err != nil {
return nil, errors.Join(err, store.Close())
}
+ networks, err := network.NewRegistry(cniProvider)
+ if err != nil {
+ return nil, errors.Join(err, store.Close())
+ }
imageCatalog := imagecatalog.New(store, imagecatalog.WithImageUsage(sandboxcatalog.Usage{}))
sandboxCatalog := sandboxcatalog.New(store, imagecatalog.Reader{})
service, err := newSandboxService(sandboxDependencies{
paths: sandboxPaths, imagePaths: imagePaths, images: images.NewGuard(imagePaths, imageCatalog),
catalog: sandboxCatalog, disks: disks, networks: networks, runtimes: runtimes, reporter: reporter,
- store: store, defaultVMM: defaultVMM,
+ store: store, defaultVMM: defaultVMM, defaultNetwork: types.NetworkBackendCNI,
cleanupTimeout: max(configuration.Sandbox.CleanupTimeout, configuration.Network.CleanupTimeout),
+ dnsServers: dnsServers,
})
if err != nil {
return nil, errors.Join(err, store.Close())
@@ -201,6 +217,21 @@ func (s *SandboxService) Close() error {
return s.dependencies.store.Close()
}
+// networkProvider resolves the provider that owns a sandbox's durable network
+// state. Creating or retained-error records without a published setup fall
+// back to the configured creation backend so cleanup can still resume.
+func (s *SandboxService) networkProvider(record types.Sandbox) (network.Provider, bool, error) {
+ if record.Config.NICs == 0 && record.Network.Backend == "" {
+ return nil, false, nil
+ }
+ backend := record.Network.Backend
+ if backend == "" {
+ backend = s.dependencies.defaultNetwork
+ }
+ provider, err := s.dependencies.networks.Provider(backend)
+ return provider, true, err
+}
+
// List returns a consistent sandbox snapshot. Unless includeAll is true, only
// states associated with an active VMM operation are returned.
func (s *SandboxService) List(ctx context.Context, includeAll bool) ([]types.Sandbox, error) {
diff --git a/core/sandbox_runtime.go b/core/sandbox_runtime.go
index 66ad391..9274864 100644
--- a/core/sandbox_runtime.go
+++ b/core/sandbox_runtime.go
@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"io"
+ "reflect"
"runtime"
"github.com/kumabox/kumabox/agent"
@@ -23,7 +24,7 @@ import (
// |
// abort + retained Error
func (s *SandboxService) Start(ctx context.Context, reference string) (result types.Sandbox, returnErr error) {
- if s == nil || s.dependencies.catalog == nil || s.dependencies.images == nil || s.dependencies.disks == nil || s.dependencies.runtimes.Len() == 0 || s.dependencies.reporter == nil || s.dependencies.now == nil {
+ if s == nil || s.dependencies.catalog == nil || s.dependencies.images == nil || s.dependencies.disks == nil || s.dependencies.networks == nil || s.dependencies.runtimes.Len() == 0 || s.dependencies.reporter == nil || s.dependencies.now == nil {
return types.Sandbox{}, errors.New("sandbox service is not configured")
}
if reference == "" {
@@ -59,13 +60,6 @@ func (s *SandboxService) Start(ctx context.Context, reference string) (result ty
if err != nil {
return types.Sandbox{}, err
}
- if record.Config.NICs > 0 || record.Network.Backend != "" {
- return record, errdefs.New(
- errdefs.ClassInvalid,
- errdefs.CodeHostIncompatible,
- errors.New("starting a networked sandbox is not supported until VMM network attachment is available"),
- )
- }
backend, err := s.dependencies.runtimes.Backend(record.VMM)
if err != nil {
return record, err
@@ -115,7 +109,6 @@ func (s *SandboxService) Start(ctx context.Context, reference string) (result ty
if err != nil {
return record, failBeforeLaunch("validate artifacts", err)
}
-
if err := s.dependencies.reporter.Status("committing starting state"); err != nil {
return record, failBeforeLaunch("report", err)
}
@@ -129,6 +122,9 @@ func (s *SandboxService) Start(ctx context.Context, reference string) (result ty
if err := plan.Validate(); err != nil {
return starting, s.failStart(ctx, backend, starting, "build launch plan", err, vmm.Process{})
}
+ if err := s.recoverNetwork(ctx, starting); err != nil {
+ return starting, s.failStart(ctx, backend, starting, "recover network", err, vmm.Process{})
+ }
if err := s.dependencies.reporter.Status("launching " + string(backend.Type())); err != nil {
return starting, s.failStart(ctx, backend, starting, "report", err, vmm.Process{})
}
@@ -185,6 +181,9 @@ func (s *SandboxService) recoverStart(ctx context.Context, backend vmm.Backend,
if err := backend.Cleanup(ctx, record.ID); err != nil {
return record, false, err
}
+ if err := s.quiesceNetwork(ctx, record); err != nil {
+ return record, false, err
+ }
stopped, err := s.dependencies.catalog.MarkStopped(ctx, record.ID, record.Generation, types.SandboxStateRunning, s.dependencies.now().UTC())
return stopped, false, err
}
@@ -240,7 +239,10 @@ func (s *SandboxService) launchPlan(record types.Sandbox, image types.Image) (vm
if err != nil {
return vmm.LaunchPlan{}, err
}
- cmdline, err := vmm.OverlayV1Cmdline(vmm.OverlayV1Config{LayerCount: len(image.Layers), Hostname: record.Config.Name})
+ cmdline, err := vmm.OverlayV1Cmdline(vmm.OverlayV1Config{
+ LayerCount: len(image.Layers), Hostname: record.Config.Name,
+ Interfaces: record.Network.Interfaces, DNSServers: s.dependencies.dnsServers,
+ })
if err != nil {
return vmm.LaunchPlan{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeImageIncompatible, err)
}
@@ -256,9 +258,46 @@ func (s *SandboxService) launchPlan(record types.Sandbox, image types.Image) (vm
return vmm.LaunchPlan{
SandboxID: record.ID, CPUs: record.Config.CPUs, Memory: record.Config.Memory,
BootProfile: image.Boot.Profile, Kernel: kernel, Initrd: initrd, Cmdline: cmdline, Disks: disks,
+ Network: record.Network,
}, nil
}
+// recoverNetwork verifies retained host plumbing or rebuilds it with the
+// persisted guest MAC and IP identity before the VMM opens any TAP.
+func (s *SandboxService) recoverNetwork(ctx context.Context, record types.Sandbox) error {
+ provider, hasNetwork, err := s.networkProvider(record)
+ if err != nil || !hasNetwork {
+ return err
+ }
+ if record.Network.Backend == "" {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, errors.New("sandbox network creation is incomplete"))
+ }
+ if err := s.dependencies.reporter.Status("recovering sandbox network"); err != nil {
+ return err
+ }
+ recovered, err := provider.Recover(ctx, record.ID, record.Config.NetworkName, record.Network.Interfaces)
+ if err != nil {
+ return err
+ }
+ if !reflect.DeepEqual(recovered, record.Network.Interfaces) {
+ return errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, errors.New("recovered network identity differs from persisted sandbox state"))
+ }
+ return nil
+}
+
+// quiesceNetwork lowers retained CNI-side links after process absence. Keeping
+// this inside the sandbox operation lock prevents a queued start from racing
+// with a late link-down operation.
+func (s *SandboxService) quiesceNetwork(ctx context.Context, record types.Sandbox) error {
+ provider, hasNetwork, err := s.networkProvider(record)
+ if err != nil || !hasNetwork || record.Network.Backend == "" {
+ return err
+ }
+ reportErr := s.dependencies.reporter.Status("quiescing sandbox network")
+ // Presentation failure must not leave an otherwise stoppable host link up.
+ return errors.Join(reportErr, provider.Quiesce(ctx, record.ID))
+}
+
// failStart cleans only the exact process identity (when available) and retains
// an Error record so the next start or removal has an explicit owner.
func (s *SandboxService) failStart(ctx context.Context, backend vmm.Backend, starting types.Sandbox, phase string, cause error, process vmm.Process) error {
@@ -270,6 +309,7 @@ func (s *SandboxService) failStart(ctx context.Context, backend vmm.Backend, sta
} else {
cleanupErr = backend.Cleanup(cleanupCtx, starting.ID)
}
+ cleanupErr = errors.Join(cleanupErr, s.quiesceNetwork(cleanupCtx, starting))
failureCause := errors.Join(cause, cleanupErr)
failure := types.SandboxFailure{Phase: phase, Message: failureCause.Error()}
_, markErr := s.dependencies.catalog.MarkStartError(cleanupCtx, starting.ID, starting.Generation, failure, s.dependencies.now().UTC())
@@ -283,7 +323,7 @@ func (s *SandboxService) failStart(ctx context.Context, backend vmm.Backend, sta
// Starting/Stopping ----- retry resumes the owned process generation -----^
// Running + no VMM --------------------- cleanup ------------------------^
func (s *SandboxService) Stop(ctx context.Context, reference string) (result types.Sandbox, returnErr error) {
- if s == nil || s.dependencies.catalog == nil || s.dependencies.runtimes.Len() == 0 || s.dependencies.reporter == nil || s.dependencies.now == nil {
+ if s == nil || s.dependencies.catalog == nil || s.dependencies.networks == nil || s.dependencies.runtimes.Len() == 0 || s.dependencies.reporter == nil || s.dependencies.now == nil {
return types.Sandbox{}, errors.New("sandbox service is not configured")
}
if reference == "" {
@@ -334,6 +374,9 @@ func (s *SandboxService) Stop(ctx context.Context, reference string) (result typ
if err := backend.Cleanup(ctx, record.ID); err != nil {
return record, errdefs.Context(err, "stop sandbox", reference, "cleanup runtime", "inspect the runtime scope before retrying", false)
}
+ if err := s.quiesceNetwork(ctx, record); err != nil {
+ return record, errdefs.Context(err, "stop sandbox", reference, "quiesce network", "retry the stop to finish network cleanup", false)
+ }
if err := s.dependencies.reporter.Committed(record); err != nil {
return record, errdefs.Context(err, "stop sandbox", reference, "report", "sandbox is not running", false)
}
@@ -377,6 +420,9 @@ func (s *SandboxService) Stop(ctx context.Context, reference string) (result typ
if err := backend.Cleanup(ctx, record.ID); err != nil {
return record, errdefs.Context(err, "stop sandbox", reference, "cleanup runtime", "retry the stop to finish cleanup", committed)
}
+ if err := s.quiesceNetwork(ctx, record); err != nil {
+ return record, errdefs.Context(err, "stop sandbox", reference, "quiesce network", "retry the stop to finish network cleanup", committed)
+ }
// Error retains the original start/create diagnostic after any residual VMM
// is gone. It can be removed or started explicitly by the next command.
diff --git a/core/sandbox_runtime_test.go b/core/sandbox_runtime_test.go
index 2d6c65f..6fcc09c 100644
--- a/core/sandbox_runtime_test.go
+++ b/core/sandbox_runtime_test.go
@@ -76,7 +76,7 @@ func TestStartCommitsRunningOnlyAfterLaunchReadiness(t *testing.T) {
}
}
-func TestStartRejectsNetworkedSandboxBeforeRuntimeRecovery(t *testing.T) {
+func TestStartRecoversNetworkBeforeLaunchingInItsNamespace(t *testing.T) {
service, steps := newTestSandboxService(t, nil)
if _, err := service.Create(t.Context(), CreateSandboxRequest{
ImageReference: "demo",
@@ -88,15 +88,76 @@ func TestStartRejectsNetworkedSandboxBeforeRuntimeRecovery(t *testing.T) {
t.Fatal(err)
}
*steps = nil
- if _, err := service.Start(t.Context(), "box"); err == nil {
- t.Fatal("Start accepted a networked sandbox before VMM network attachment exists")
- } else if code, ok := errdefs.CodeOf(err); !ok || code != errdefs.CodeHostIncompatible {
- t.Fatalf("Start error code = %q, %v; want %q", code, err, errdefs.CodeHostIncompatible)
+ record, err := service.Start(t.Context(), "box")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record.State != types.SandboxStateRunning {
+ t.Fatalf("started record = %+v", record)
+ }
+ plan := testRuntime(t, service).plan
+ if plan.Network.Namespace != "/var/run/netns/kumabox-test" || len(plan.Network.Interfaces) != 1 {
+ t.Fatalf("launch network = %+v", plan.Network)
}
if got := *steps; !reflect.DeepEqual(got, []string{
"status:resolving sandbox", "resolve", "status:waiting for sandbox operation lock", "resolve",
+ "status:checking existing runtime", "observe", "cleanup",
+ "status:checking host runtime", "preflight",
+ "status:verifying image and sandbox disk", "verify", "check",
+ "status:committing starting state", "starting",
+ "status:recovering sandbox network", "network-recover",
+ "status:launching cloud-hypervisor", "launch",
+ "status:committing running state", "running", "report",
}) {
- t.Fatalf("Start touched runtime state before rejection: %v", got)
+ t.Fatalf("Start steps = %v", got)
+ }
+}
+
+func TestStartNetworkRecoveryFailureRetainsErrorAndQuiesces(t *testing.T) {
+ service, steps := newTestSandboxService(t, nil)
+ if _, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ }); err != nil {
+ t.Fatal(err)
+ }
+ failure := errors.New("network recovery failed")
+ testNetwork(t, service).recoverErr = failure
+ *steps = nil
+ if _, err := service.Start(t.Context(), "box"); !errors.Is(err, failure) {
+ t.Fatalf("Start error = %v", err)
+ }
+ record := service.dependencies.catalog.(*fakeCatalog).record
+ if record.State != types.SandboxStateError || record.Failure == nil || record.Failure.Phase != "recover network" {
+ t.Fatalf("failed start record = %+v", record)
+ }
+ if got := strings.Join(*steps, ","); !strings.Contains(got,
+ "starting,status:recovering sandbox network,network-recover,cleanup,status:quiescing sandbox network,network-quiesce,start-error") {
+ t.Fatalf("recovery compensation steps = %v", *steps)
+ }
+}
+
+func TestStartRejectsRecoveredNetworkIdentityDrift(t *testing.T) {
+ service, _ := newTestSandboxService(t, nil)
+ if _, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ }); err != nil {
+ t.Fatal(err)
+ }
+ networkAdapter := testNetwork(t, service)
+ networkAdapter.recovered = append([]types.NetworkInterface(nil), service.dependencies.catalog.(*fakeCatalog).record.Network.Interfaces...)
+ networkAdapter.recovered[0].MAC = "02:00:00:00:00:fe"
+ if _, err := service.Start(t.Context(), "box"); err == nil {
+ t.Fatal("Start accepted a recovered network with changed guest identity")
+ } else if code, ok := errdefs.CodeOf(err); !ok || code != errdefs.CodeArtifactCorrupt {
+ t.Fatalf("Start error = %v, want %s", err, errdefs.CodeArtifactCorrupt)
}
}
@@ -204,6 +265,73 @@ func TestStopRecordsIntentBeforeTerminatingRunningVMM(t *testing.T) {
}
}
+func TestStopQuiescesNetworkAfterRuntimeCleanup(t *testing.T) {
+ service, steps := newTestSandboxService(t, nil)
+ if _, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ }); err != nil {
+ t.Fatal(err)
+ }
+ catalog := service.dependencies.catalog.(*fakeCatalog)
+ catalog.record.State, catalog.record.Generation = types.SandboxStateRunning, 4
+ runtimeAdapter := testRuntime(t, service)
+ runtimeAdapter.observation = vmm.Observation{State: vmm.ProcessRunning, Process: vmm.Process{PID: 42}}
+ *steps = nil
+ if _, err := service.Stop(t.Context(), "box"); err != nil {
+ t.Fatal(err)
+ }
+ if got := strings.Join(*steps, ","); !strings.Contains(got,
+ "stop,status:cleaning runtime state,cleanup,status:quiescing sandbox network,network-quiesce,status:committing stopped state,stopped") {
+ t.Fatalf("network stop ordering = %v", *steps)
+ }
+}
+
+func TestStopRetriesNetworkQuiesceFromStoppingState(t *testing.T) {
+ service, steps := newTestSandboxService(t, nil)
+ if _, err := service.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1,
+ },
+ }); err != nil {
+ t.Fatal(err)
+ }
+ catalog := service.dependencies.catalog.(*fakeCatalog)
+ catalog.record.State, catalog.record.Generation = types.SandboxStateRunning, 4
+ runtimeAdapter := testRuntime(t, service)
+ runtimeAdapter.observation = vmm.Observation{State: vmm.ProcessRunning, Process: vmm.Process{PID: 42}}
+ networkAdapter := testNetwork(t, service)
+ failure := errors.New("link state failed")
+ networkAdapter.quiesceErr = failure
+ *steps = nil
+ if _, err := service.Stop(t.Context(), "box"); !errors.Is(err, failure) {
+ t.Fatalf("Stop error = %v", err)
+ }
+ if catalog.record.State != types.SandboxStateStopping || catalog.record.Generation != 5 {
+ t.Fatalf("retained record = %+v", catalog.record)
+ }
+
+ networkAdapter.quiesceErr = nil
+ runtimeAdapter.observation = vmm.Observation{State: vmm.ProcessAbsent}
+ *steps = nil
+ record, err := service.Stop(t.Context(), "box")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record.State != types.SandboxStateStopped || record.Generation != 6 {
+ t.Fatalf("retried stop record = %+v", record)
+ }
+ if got := strings.Join(*steps, ","); strings.Contains(got, ",stop,") || !strings.Contains(got,
+ "cleanup,status:quiescing sandbox network,network-quiesce,status:committing stopped state,stopped") {
+ t.Fatalf("retried stop steps = %v", *steps)
+ }
+}
+
func TestStopResumesStoppingAndRecoversStarting(t *testing.T) {
for _, test := range []struct {
name string
diff --git a/core/sandbox_storage.go b/core/sandbox_storage.go
index 45265d0..1399cd9 100644
--- a/core/sandbox_storage.go
+++ b/core/sandbox_storage.go
@@ -35,6 +35,14 @@ func (s *SandboxService) Create(ctx context.Context, request CreateSandboxReques
if _, err := s.dependencies.runtimes.Backend(request.VMM); err != nil {
return types.Sandbox{}, err
}
+ var networkProvider network.Provider
+ if request.Config.NICs > 0 {
+ var providerErr error
+ networkProvider, providerErr = s.dependencies.networks.Provider(s.dependencies.defaultNetwork)
+ if providerErr != nil {
+ return types.Sandbox{}, providerErr
+ }
+ }
if int(request.Config.CPUs) > runtime.NumCPU() { //nolint:gosec // Config validation bounds CPUs to a small positive value
return types.Sandbox{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeHostIncompatible, fmt.Errorf("requested %d vCPUs exceeds available host CPUs (%d)", request.Config.CPUs, runtime.NumCPU()))
}
@@ -88,7 +96,7 @@ func (s *SandboxService) Create(ctx context.Context, request CreateSandboxReques
if err := s.dependencies.reporter.Status("preparing sandbox network"); err != nil {
return types.Sandbox{}, s.compensate(ctx, record, "report", err)
}
- namespace, err := s.dependencies.networks.Prepare(ctx, id)
+ namespace, err := networkProvider.Prepare(ctx, id)
if err != nil {
return types.Sandbox{}, s.compensate(ctx, record, "network prepare", err)
}
@@ -100,11 +108,11 @@ func (s *SandboxService) Create(ctx context.Context, request CreateSandboxReques
for index := range specs {
specs[index].Queues = queues
}
- interfaces, err := s.dependencies.networks.Add(ctx, id, request.Config.NetworkName, specs...)
+ interfaces, err := networkProvider.Add(ctx, id, request.Config.NetworkName, specs...)
if err != nil {
return types.Sandbox{}, s.compensate(ctx, record, "network add", err)
}
- setup = types.NetworkSetup{Backend: s.dependencies.networks.Type(), Namespace: namespace, Interfaces: interfaces}
+ setup = types.NetworkSetup{Backend: networkProvider.Type(), Namespace: namespace, Interfaces: interfaces}
if err := setup.Validate(); err != nil {
return types.Sandbox{}, s.compensate(ctx, record, "network result", err)
}
@@ -113,6 +121,12 @@ func (s *SandboxService) Create(ctx context.Context, request CreateSandboxReques
}
record.Network = setup
record.Config.NetworkName = interfaces[0].Network
+ if err := s.dependencies.reporter.Status("quiescing sandbox network"); err != nil {
+ return types.Sandbox{}, s.compensate(ctx, record, "report", err)
+ }
+ if err := networkProvider.Quiesce(ctx, id); err != nil {
+ return types.Sandbox{}, s.compensate(ctx, record, "network quiesce", err)
+ }
}
if err := s.dependencies.reporter.Status("creating sparse ext4 disk"); err != nil {
return types.Sandbox{}, s.compensate(ctx, record, "report", err)
@@ -182,6 +196,10 @@ func (s *SandboxService) Remove(ctx context.Context, reference string) (result t
if err != nil {
return record, err
}
+ networkProvider, hasNetwork, err := s.networkProvider(record)
+ if err != nil {
+ return record, err
+ }
if err := s.dependencies.reporter.Status("marking sandbox for deletion"); err != nil {
return types.Sandbox{}, err
}
@@ -197,11 +215,11 @@ func (s *SandboxService) Remove(ctx context.Context, reference string) (result t
if err := s.dependencies.disks.Remove(ctx, deleting.ID); err != nil {
return deleting, errdefs.Context(err, "remove sandbox", reference, "disk cleanup", "retry removal to finish cleanup", true)
}
- if deleting.Config.NICs > 0 || deleting.Network.Backend != "" {
+ if hasNetwork {
if err := s.dependencies.reporter.Status("removing sandbox network"); err != nil {
return deleting, errdefs.Context(err, "remove sandbox", reference, "report", "retry removal to finish cleanup", true)
}
- if err := s.dependencies.networks.Delete(ctx, deleting.ID); err != nil {
+ if err := networkProvider.Delete(ctx, deleting.ID); err != nil {
return deleting, errdefs.Context(err, "remove sandbox", reference, "network cleanup", "retry removal to finish cleanup", true)
}
}
@@ -230,8 +248,10 @@ func (s *SandboxService) compensate(ctx context.Context, record types.Sandbox, p
cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), s.dependencies.cleanupTimeout)
defer cancel()
cleanupErr := s.dependencies.disks.Remove(cleanupCtx, record.ID)
- if record.Config.NICs > 0 || record.Network.Backend != "" {
- cleanupErr = errors.Join(cleanupErr, s.dependencies.networks.Delete(cleanupCtx, record.ID))
+ if provider, hasNetwork, providerErr := s.networkProvider(record); providerErr != nil {
+ cleanupErr = errors.Join(cleanupErr, providerErr)
+ } else if hasNetwork {
+ cleanupErr = errors.Join(cleanupErr, provider.Delete(cleanupCtx, record.ID))
}
if cleanupErr == nil {
forgetErr := s.dependencies.catalog.Forget(cleanupCtx, record.ID, record.Generation)
diff --git a/core/sandbox_storage_test.go b/core/sandbox_storage_test.go
index 76d652b..c5a6807 100644
--- a/core/sandbox_storage_test.go
+++ b/core/sandbox_storage_test.go
@@ -46,7 +46,7 @@ func TestCreatePublishesResolvedNetworkWithCreatedState(t *testing.T) {
record.Network.Backend != types.NetworkBackendCNI || len(record.Network.Interfaces) != 2 {
t.Fatalf("created network record = %+v", record)
}
- networks := service.dependencies.networks.(*fakeNetwork)
+ networks := testNetwork(t, service)
if len(networks.specs) != 2 || networks.specs[0].Queues != 4 || networks.specs[1].Queues != 4 {
t.Fatalf("network specs = %+v", networks.specs)
}
@@ -54,6 +54,7 @@ func TestCreatePublishesResolvedNetworkWithCreatedState(t *testing.T) {
"status:resolving and checking image", "verify", "reserve",
"status:preparing sandbox network", "network-prepare",
"status:allocating sandbox network interfaces", "network-add",
+ "status:quiescing sandbox network", "network-quiesce",
"status:creating sparse ext4 disk", "disk",
"status:committing created state", "created", "report",
}
@@ -65,7 +66,7 @@ func TestCreatePublishesResolvedNetworkWithCreatedState(t *testing.T) {
func TestCreateNetworkFailureCleansResourcesBeforeForgettingReservation(t *testing.T) {
failure := errors.New("CNI add failed")
service, steps := newTestSandboxService(t, nil)
- networks := service.dependencies.networks.(*fakeNetwork)
+ networks := testNetwork(t, service)
networks.addErr = failure
if _, err := service.Create(t.Context(), CreateSandboxRequest{
ImageReference: "demo",
@@ -86,7 +87,7 @@ func TestCreateRetainsNetworkOwnerWhenCleanupFails(t *testing.T) {
addFailure := errors.New("CNI add failed")
deleteFailure := errors.New("CNI delete failed")
service, steps := newTestSandboxService(t, nil)
- networks := service.dependencies.networks.(*fakeNetwork)
+ networks := testNetwork(t, service)
networks.addErr, networks.deleteErr = addFailure, deleteFailure
if _, err := service.Create(t.Context(), CreateSandboxRequest{
ImageReference: "demo",
@@ -261,7 +262,7 @@ func TestRemoveNetworkFailureRetainsDeletingUntilRetry(t *testing.T) {
t.Fatal(err)
}
failure := errors.New("network cleanup failed")
- networks := service.dependencies.networks.(*fakeNetwork)
+ networks := testNetwork(t, service)
networks.deleteErr = failure
*steps = nil
if _, err := service.Remove(t.Context(), "box"); !errors.Is(err, failure) {
diff --git a/core/sandbox_test.go b/core/sandbox_test.go
index f1a3bea..da6a2b2 100644
--- a/core/sandbox_test.go
+++ b/core/sandbox_test.go
@@ -184,9 +184,12 @@ type fakeNetwork struct {
steps *[]string
prepareErr error
addErr error
+ recoverErr error
+ quiesceErr error
deleteErr error
namespace string
interfaces []types.NetworkInterface
+ recovered []types.NetworkInterface
specs []network.AddSpec
}
@@ -224,12 +227,23 @@ func (*fakeNetwork) Verify(context.Context, types.SandboxID, []types.NetworkInte
return nil
}
-func (*fakeNetwork) Recover(context.Context, types.SandboxID, string, []types.NetworkInterface) ([]types.NetworkInterface, error) {
- return nil, nil
+func (f *fakeNetwork) Recover(_ context.Context, _ types.SandboxID, _ string, expected []types.NetworkInterface) ([]types.NetworkInterface, error) {
+ *f.steps = append(*f.steps, "network-recover")
+ if f.recovered != nil {
+ return append([]types.NetworkInterface(nil), f.recovered...), f.recoverErr
+ }
+ return append([]types.NetworkInterface(nil), expected...), f.recoverErr
+}
+
+func (f *fakeNetwork) Quiesce(context.Context, types.SandboxID) error {
+ *f.steps = append(*f.steps, "network-quiesce")
+ return f.quiesceErr
}
-func (*fakeNetwork) Quiesce(context.Context, types.SandboxID) error { return nil }
-func (*fakeNetwork) Unquiesce(context.Context, types.SandboxID) error { return nil }
+func (f *fakeNetwork) Unquiesce(context.Context, types.SandboxID) error {
+ *f.steps = append(*f.steps, "network-unquiesce")
+ return nil
+}
func (f *fakeNetwork) Delete(context.Context, types.SandboxID) error {
*f.steps = append(*f.steps, "network-delete")
@@ -404,14 +418,20 @@ func newTestSandboxService(t *testing.T, diskError error) (*SandboxService, *[]s
if err != nil {
t.Fatal(err)
}
+ networkAdapter := &fakeNetwork{steps: &steps, namespace: "/var/run/netns/kumabox-test"}
+ networks, err := network.NewRegistry(networkAdapter)
+ if err != nil {
+ t.Fatal(err)
+ }
service, err := newSandboxService(sandboxDependencies{
paths: paths, imagePaths: imagePaths, images: fakeGuard{image: image, steps: &steps},
catalog: catalog, disks: fakeDisk{steps: &steps, prepare: diskError},
- networks: &fakeNetwork{steps: &steps, namespace: "/var/run/netns/kumabox-test"}, runtimes: runtimes,
+ networks: networks, runtimes: runtimes,
defaultVMM: types.VMMCloudHypervisor, cleanupTimeout: 10 * time.Second,
- reporter: fakeReporter{steps: &steps},
- newID: func() (types.SandboxID, error) { return fixedID, nil },
- now: func() time.Time { return time.Date(2026, 9, 15, 10, 0, 0, 0, time.UTC) },
+ defaultNetwork: types.NetworkBackendCNI,
+ reporter: fakeReporter{steps: &steps},
+ newID: func() (types.SandboxID, error) { return fixedID, nil },
+ now: func() time.Time { return time.Date(2026, 9, 15, 10, 0, 0, 0, time.UTC) },
})
if err != nil {
t.Fatal(err)
@@ -419,6 +439,19 @@ func newTestSandboxService(t *testing.T, diskError error) (*SandboxService, *[]s
return service, &steps
}
+func testNetwork(t *testing.T, service *SandboxService) *fakeNetwork {
+ t.Helper()
+ provider, err := service.dependencies.networks.Provider(types.NetworkBackendCNI)
+ if err != nil {
+ t.Fatal(err)
+ }
+ networkAdapter, ok := provider.(*fakeNetwork)
+ if !ok {
+ t.Fatalf("network provider = %T, want *fakeNetwork", provider)
+ }
+ return networkAdapter
+}
+
func testRuntime(t *testing.T, service *SandboxService) *fakeRuntime {
t.Helper()
backend, err := service.dependencies.runtimes.Backend(types.VMMCloudHypervisor)
@@ -458,6 +491,7 @@ func TestNewSandboxServiceValidatesNamedDependencies(t *testing.T) {
{name: "VMM registry", mutate: func(dependencies *sandboxDependencies) { dependencies.runtimes = nil }},
{name: "cleanup timeout", mutate: func(dependencies *sandboxDependencies) { dependencies.cleanupTimeout = 0 }},
{name: "default VMM", mutate: func(dependencies *sandboxDependencies) { dependencies.defaultVMM = types.VMMFirecracker }},
+ {name: "default network", mutate: func(dependencies *sandboxDependencies) { dependencies.defaultNetwork = "missing" }},
} {
t.Run(test.name, func(t *testing.T) {
dependencies := valid
diff --git a/network/registry.go b/network/registry.go
new file mode 100644
index 0000000..feb3384
--- /dev/null
+++ b/network/registry.go
@@ -0,0 +1,69 @@
+package network
+
+import (
+ "errors"
+ "fmt"
+ "reflect"
+
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/types"
+)
+
+// Registry routes durable network backend identities to provider adapters.
+// Construction freezes the available set so lifecycle operations never rely
+// on package initialization or registration order.
+type Registry struct {
+ providers map[types.NetworkBackend]Provider
+}
+
+// NewRegistry validates and freezes the supplied provider set.
+func NewRegistry(providers ...Provider) (*Registry, error) {
+ registered := make(map[types.NetworkBackend]Provider, len(providers))
+ for _, provider := range providers {
+ if provider == nil || isNilProvider(provider) {
+ return nil, errors.New("network registry contains a nil provider")
+ }
+ backend := provider.Type()
+ if err := backend.Validate(); err != nil {
+ return nil, fmt.Errorf("register network provider: %w", err)
+ }
+ if _, exists := registered[backend]; exists {
+ return nil, fmt.Errorf("network provider %q is registered more than once", backend)
+ }
+ registered[backend] = provider
+ }
+ return &Registry{providers: registered}, nil
+}
+
+func isNilProvider(provider Provider) bool {
+ value := reflect.ValueOf(provider)
+ switch value.Kind() {
+ case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
+ return value.IsNil()
+ default:
+ return false
+ }
+}
+
+// Provider returns the adapter for a persisted backend identity.
+func (r *Registry) Provider(backend types.NetworkBackend) (Provider, error) {
+ if err := backend.Validate(); err != nil {
+ return nil, errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, err)
+ }
+ if r == nil {
+ return nil, errdefs.New(errdefs.ClassInvalid, errdefs.CodeHostIncompatible, errors.New("network registry is not configured"))
+ }
+ provider, exists := r.providers[backend]
+ if !exists || provider == nil {
+ return nil, errdefs.New(errdefs.ClassInvalid, errdefs.CodeHostIncompatible, fmt.Errorf("network backend %q is not available", backend))
+ }
+ return provider, nil
+}
+
+// Len returns the number of providers frozen into the registry.
+func (r *Registry) Len() int {
+ if r == nil {
+ return 0
+ }
+ return len(r.providers)
+}
diff --git a/network/registry_test.go b/network/registry_test.go
new file mode 100644
index 0000000..64127a1
--- /dev/null
+++ b/network/registry_test.go
@@ -0,0 +1,66 @@
+package network
+
+import (
+ "context"
+ "testing"
+
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/types"
+)
+
+type registryProvider struct{ backend types.NetworkBackend }
+
+func (p *registryProvider) Type() types.NetworkBackend { return p.backend }
+func (*registryProvider) Prepare(context.Context, types.SandboxID) (string, error) {
+ return "", nil
+}
+
+func (*registryProvider) Add(context.Context, types.SandboxID, string, ...AddSpec) ([]types.NetworkInterface, error) {
+ return nil, nil
+}
+
+func (*registryProvider) Verify(context.Context, types.SandboxID, []types.NetworkInterface) error {
+ return nil
+}
+
+func (*registryProvider) Recover(context.Context, types.SandboxID, string, []types.NetworkInterface) ([]types.NetworkInterface, error) {
+ return nil, nil
+}
+func (*registryProvider) Quiesce(context.Context, types.SandboxID) error { return nil }
+func (*registryProvider) Unquiesce(context.Context, types.SandboxID) error { return nil }
+func (*registryProvider) Delete(context.Context, types.SandboxID) error { return nil }
+
+func TestRegistryRoutesPersistedBackend(t *testing.T) {
+ provider := ®istryProvider{backend: types.NetworkBackendCNI}
+ registry, err := NewRegistry(provider)
+ if err != nil {
+ t.Fatal(err)
+ }
+ resolved, err := registry.Provider(types.NetworkBackendCNI)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if resolved != provider || registry.Len() != 1 {
+ t.Fatalf("resolved provider = %T, len = %d", resolved, registry.Len())
+ }
+}
+
+func TestRegistryRejectsInvalidSetsAndUnavailableBackends(t *testing.T) {
+ var typedNil *registryProvider
+ if _, err := NewRegistry(typedNil); err == nil {
+ t.Fatal("NewRegistry accepted a typed nil provider")
+ }
+ provider := ®istryProvider{backend: types.NetworkBackendCNI}
+ if _, err := NewRegistry(provider, provider); err == nil {
+ t.Fatal("NewRegistry accepted a duplicate provider")
+ }
+ registry, err := NewRegistry()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := registry.Provider(types.NetworkBackendCNI); err == nil {
+ t.Fatal("Provider resolved an unavailable backend")
+ } else if code, ok := errdefs.CodeOf(err); !ok || code != errdefs.CodeHostIncompatible {
+ t.Fatalf("Provider error code = %q, %v; want %q", code, err, errdefs.CodeHostIncompatible)
+ }
+}
diff --git a/oci-images/ubuntu/Dockerfile b/oci-images/ubuntu/Dockerfile
index 4618aba..dc46dab 100644
--- a/oci-images/ubuntu/Dockerfile
+++ b/oci-images/ubuntu/Dockerfile
@@ -25,6 +25,7 @@ ENV DEBIAN_FRONTEND=noninteractive
COPY --from=agent-builder /out/kumabox-agent /usr/local/bin/kumabox-agent
COPY oci-images/ubuntu/overlay.sh /usr/local/lib/kumabox/initramfs/kumabox-overlay
+COPY oci-images/ubuntu/network.sh /usr/local/lib/kumabox/initramfs/kumabox-network
COPY oci-images/ubuntu/kumabox-agent.service /etc/systemd/system/kumabox-agent.service
RUN set -eu; \
@@ -32,15 +33,20 @@ RUN set -eu; \
&& apt-get install -y --no-install-recommends \
ca-certificates \
initramfs-tools \
+ iproute2 \
+ iputils-ping \
kmod \
linux-image-virtual \
systemd \
systemd-sysv \
systemd-timesyncd \
systemd-resolved \
+ curl \
udev \
&& install -m 0755 /usr/local/lib/kumabox/initramfs/kumabox-overlay \
/etc/initramfs-tools/scripts/kumabox-overlay \
+ && install -m 0755 /usr/local/lib/kumabox/initramfs/kumabox-network \
+ /etc/initramfs-tools/scripts/init-bottom/kumabox-network \
&& printf '%s\n' \
erofs overlay ext4 virtio_blk virtio_pci virtio_ring virtio_net \
vsock vmw_vsock_virtio_transport \
@@ -57,6 +63,7 @@ RUN set -eu; \
test -s "$initrd"; \
lsinitramfs "$initrd" > "$contents"; \
grep -qx 'scripts/kumabox-overlay' "$contents"; \
+ grep -qx 'scripts/init-bottom/kumabox-network' "$contents"; \
for module in erofs overlay ext4 virtio_blk virtio_pci virtio_net vsock vmw_vsock_virtio_transport; do \
filename=$(modinfo -k "$kernel" -F filename "$module"); \
if [ "$filename" != '(builtin)' ]; then \
diff --git a/oci-images/ubuntu/network.sh b/oci-images/ubuntu/network.sh
new file mode 100644
index 0000000..06b1484
--- /dev/null
+++ b/oci-images/ubuntu/network.sh
@@ -0,0 +1,75 @@
+#!/bin/sh
+# Persists initramfs static network facts into the assembled Ubuntu root.
+#
+# kernel ip= parameters
+# |
+# v
+# /run/net-ethN.conf
+# |
+# v
+# MAC-matched systemd-networkd files in the writable overlay
+
+PREREQ=""
+
+prereqs() {
+ printf '%s\n' "$PREREQ"
+}
+
+case "$1" in
+prereqs)
+ prereqs
+ exit 0
+ ;;
+esac
+
+. /scripts/functions
+
+[ -n "$rootmnt" ] || exit 0
+
+for config_file in /run/net-*.conf; do
+ [ -f "$config_file" ] || continue
+ unset DEVICE IPV4ADDR IPV4NETMASK IPV4GATEWAY IPV4DNS0 IPV4DNS1 HWADDR
+ . "$config_file"
+ [ -n "$DEVICE" ] || continue
+ [ -n "$IPV4ADDR" ] || continue
+
+ if [ -z "$HWADDR" ] && [ -r "/sys/class/net/$DEVICE/address" ]; then
+ HWADDR=$(cat "/sys/class/net/$DEVICE/address")
+ fi
+ [ -n "$HWADDR" ] || continue
+
+ prefix=0
+ old_ifs=$IFS
+ IFS=.
+ set -- $IPV4NETMASK
+ IFS=$old_ifs
+ for octet in "$@"; do
+ case "$octet" in
+ 255) prefix=$((prefix + 8)) ;;
+ 254) prefix=$((prefix + 7)) ;;
+ 252) prefix=$((prefix + 6)) ;;
+ 248) prefix=$((prefix + 5)) ;;
+ 240) prefix=$((prefix + 4)) ;;
+ 224) prefix=$((prefix + 3)) ;;
+ 192) prefix=$((prefix + 2)) ;;
+ 128) prefix=$((prefix + 1)) ;;
+ esac
+ done
+
+ identifier=$(printf '%s' "$HWADDR" | tr -d ':')
+ directory="$rootmnt/etc/systemd/network"
+ mkdir -p "$directory"
+ {
+ printf '[Match]\nMACAddress=%s\n\n' "$HWADDR"
+ printf '[Network]\nAddress=%s/%s\n' "$IPV4ADDR" "$prefix"
+ if [ -n "$IPV4GATEWAY" ] && [ "$IPV4GATEWAY" != "0.0.0.0" ]; then
+ printf 'Gateway=%s\n' "$IPV4GATEWAY"
+ fi
+ if [ -n "$IPV4DNS0" ] && [ "$IPV4DNS0" != "0.0.0.0" ]; then
+ printf 'DNS=%s\n' "$IPV4DNS0"
+ fi
+ if [ -n "$IPV4DNS1" ] && [ "$IPV4DNS1" != "0.0.0.0" ]; then
+ printf 'DNS=%s\n' "$IPV4DNS1"
+ fi
+ } >"$directory/10-kumabox-$identifier.network"
+done
diff --git a/oci-images/ubuntu/overlay.sh b/oci-images/ubuntu/overlay.sh
index edf728f..38d7267 100755
--- a/oci-images/ubuntu/overlay.sh
+++ b/oci-images/ubuntu/overlay.sh
@@ -46,6 +46,7 @@ mountroot() {
KUMABOX_LAYERS=
KUMABOX_COW=
KUMABOX_HOSTNAME=
+ KUMABOX_NETWORK=false
KUMABOX_DEVICE_TIMEOUT=10
for argument in $(cat /proc/cmdline); do
case "$argument" in
@@ -53,6 +54,7 @@ mountroot() {
kumabox.cow=*) KUMABOX_COW=${argument#kumabox.cow=} ;;
kumabox.hostname=*) KUMABOX_HOSTNAME=${argument#kumabox.hostname=} ;;
kumabox.timeout=*) KUMABOX_DEVICE_TIMEOUT=${argument#kumabox.timeout=} ;;
+ ip=*) KUMABOX_NETWORK=true ;;
esac
done
@@ -73,6 +75,12 @@ mountroot() {
*[!A-Za-z0-9_.-]*) panic "kumabox.hostname contains an invalid character" ;;
esac
+ # configure_networking parses every static ip= entry into /run/net-*.conf.
+ # Skipping it for a zero-NIC sandbox avoids the initramfs DHCP wait.
+ if [ "$KUMABOX_NETWORK" = true ] && ! ls /run/net-*.conf >/dev/null 2>&1; then
+ configure_networking
+ fi
+
modprobe erofs 2>/dev/null || true
modprobe overlay 2>/dev/null || true
modprobe ext4 2>/dev/null || true
diff --git a/types/network.go b/types/network.go
index c852acb..ce96371 100644
--- a/types/network.go
+++ b/types/network.go
@@ -135,7 +135,7 @@ func (s NetworkSetup) Validate() error {
}
seen := make(map[int]struct{}, len(s.Interfaces))
previous := -1
- for _, networkInterface := range s.Interfaces {
+ for position, networkInterface := range s.Interfaces {
if err := networkInterface.Validate(); err != nil {
return fmt.Errorf("network interface %d: %w", networkInterface.Index, err)
}
@@ -145,6 +145,9 @@ func (s NetworkSetup) Validate() error {
if networkInterface.Index <= previous {
return errors.New("network interfaces must be ordered by increasing index")
}
+ if networkInterface.Index != position {
+ return errors.New("network interface indices must be contiguous from zero")
+ }
seen[networkInterface.Index] = struct{}{}
previous = networkInterface.Index
}
diff --git a/types/network_test.go b/types/network_test.go
index b6902d2..ade63a1 100644
--- a/types/network_test.go
+++ b/types/network_test.go
@@ -29,3 +29,17 @@ func TestNetworkSetupZeroValueDisablesNetworking(t *testing.T) {
t.Fatal("namespace without backend was accepted")
}
}
+
+func TestNetworkSetupRejectsNonContiguousInterfaceIndices(t *testing.T) {
+ setup := NetworkSetup{
+ Backend: NetworkBackendCNI,
+ Namespace: "/var/run/netns/kb-sandbox",
+ Interfaces: []NetworkInterface{{
+ Index: 1, Name: "eth1", TAP: "tap12345678-1", MAC: "02:00:00:00:00:02",
+ Queues: 2, QueueSize: 512, Network: "bridge",
+ }},
+ }
+ if err := setup.Validate(); err == nil {
+ t.Fatal("NetworkSetup accepted an interface sequence that does not begin at zero")
+ }
+}
diff --git a/vmm/cloudhypervisor/args.go b/vmm/cloudhypervisor/args.go
index 2066ee6..b5b7dbd 100644
--- a/vmm/cloudhypervisor/args.go
+++ b/vmm/cloudhypervisor/args.go
@@ -35,6 +35,20 @@ func buildArgs(plan vmm.LaunchPlan, apiSocket, vsock string) []string {
}
args = append(args, strings.Join(parts, ","))
}
+ if len(plan.Network.Interfaces) > 0 {
+ args = append(args, "--net")
+ for _, networkInterface := range plan.Network.Interfaces {
+ args = append(args, strings.Join([]string{
+ "tap=" + networkInterface.TAP,
+ "mac=" + networkInterface.MAC,
+ fmt.Sprintf("num_queues=%d", networkInterface.Queues),
+ fmt.Sprintf("queue_size=%d", networkInterface.QueueSize),
+ "offload_tso=on",
+ "offload_ufo=on",
+ "offload_csum=on",
+ }, ","))
+ }
+ }
args = append(args,
"--kernel", plan.Kernel,
"--initramfs", plan.Initrd,
diff --git a/vmm/cloudhypervisor/args_test.go b/vmm/cloudhypervisor/args_test.go
index 9f506db..67eeed7 100644
--- a/vmm/cloudhypervisor/args_test.go
+++ b/vmm/cloudhypervisor/args_test.go
@@ -107,6 +107,13 @@ func TestBuildArgsMatchesDirectBootContract(t *testing.T) {
{Path: "/layers/1.erofs", Serial: "kumabox-layer1", ReadOnly: true},
{Path: "/sandbox/cow.raw", Serial: vmm.COWSerial},
},
+ Network: types.NetworkSetup{
+ Backend: types.NetworkBackendCNI, Namespace: "/var/run/netns/kumabox-test",
+ Interfaces: []types.NetworkInterface{{
+ Index: 0, Name: "eth0", TAP: "tap12345678-0", MAC: "02:00:00:00:00:01",
+ Queues: 4, QueueSize: 512, Network: "bridge",
+ }},
+ },
}
args := buildArgs(plan, "/run/api.sock", "/run/vsock.uds")
want := []string{
@@ -117,6 +124,8 @@ func TestBuildArgsMatchesDirectBootContract(t *testing.T) {
"path=/layers/0.erofs,image_type=raw,num_queues=2,queue_size=512,serial=kumabox-layer0,readonly=on",
"path=/layers/1.erofs,image_type=raw,num_queues=2,queue_size=512,serial=kumabox-layer1,readonly=on",
"path=/sandbox/cow.raw,image_type=raw,num_queues=2,queue_size=512,serial=kumabox-cow,direct=on,sparse=on",
+ "--net",
+ "tap=tap12345678-0,mac=02:00:00:00:00:01,num_queues=4,queue_size=512,offload_tso=on,offload_ufo=on,offload_csum=on",
"--kernel", "/boot/vmlinuz",
"--initramfs", "/boot/initrd.img",
"--cmdline", "boot=kumabox-overlay",
diff --git a/vmm/cloudhypervisor/driver.go b/vmm/cloudhypervisor/driver.go
index a612ca4..87e4521 100644
--- a/vmm/cloudhypervisor/driver.go
+++ b/vmm/cloudhypervisor/driver.go
@@ -173,7 +173,7 @@ func (d *Driver) Launch(ctx context.Context, plan vmm.LaunchPlan) (result vmm.Pr
command.Stdout, command.Stderr = logFile, logFile
configureProcess(command, scope)
- if err := command.Start(); err != nil {
+ if err := startProcess(command, plan.Network.Namespace); err != nil {
return vmm.Process{}, fmt.Errorf("exec cloud-hypervisor: %w", err)
}
result, err = captureProcess(command.Process.Pid, plan.SandboxID, plan.Generation, filepath.Base(d.binary), apiSocket)
diff --git a/vmm/cloudhypervisor/process_linux.go b/vmm/cloudhypervisor/process_linux.go
index 3d9beec..13626a7 100644
--- a/vmm/cloudhypervisor/process_linux.go
+++ b/vmm/cloudhypervisor/process_linux.go
@@ -9,11 +9,14 @@ import (
"os"
"os/exec"
"path/filepath"
+ "runtime"
"strconv"
"strings"
"syscall"
"time"
+ "github.com/vishvananda/netns"
+
"github.com/kumabox/kumabox/types"
"github.com/kumabox/kumabox/vmm"
)
@@ -37,6 +40,39 @@ func configureProcess(command *exec.Cmd, scope *os.File) {
}
}
+// startProcess starts the child in the requested network namespace. setns is
+// thread-local, so the caller thread is pinned until the original namespace is
+// restored after fork and exec.
+func startProcess(command *exec.Cmd, namespacePath string) (returnErr error) {
+ if namespacePath == "" {
+ return command.Start()
+ }
+ if !filepath.IsAbs(namespacePath) {
+ return errors.New("VMM network namespace path must be absolute")
+ }
+ runtime.LockOSThread()
+ defer runtime.UnlockOSThread()
+ original, err := netns.Get()
+ if err != nil {
+ return fmt.Errorf("get current network namespace: %w", err)
+ }
+ defer func() { returnErr = errors.Join(returnErr, original.Close()) }()
+ target, err := netns.GetFromPath(namespacePath)
+ if err != nil {
+ return fmt.Errorf("open VMM network namespace %s: %w", namespacePath, err)
+ }
+ defer func() { returnErr = errors.Join(returnErr, target.Close()) }()
+ if err := netns.Set(target); err != nil {
+ return fmt.Errorf("enter VMM network namespace %s: %w", namespacePath, err)
+ }
+ defer func() {
+ if err := netns.Set(original); err != nil {
+ returnErr = errors.Join(returnErr, fmt.Errorf("restore host network namespace: %w", err))
+ }
+ }()
+ return command.Start()
+}
+
func captureProcess(pid int, id types.SandboxID, generation uint64, binary, apiSocket string) (vmm.Process, error) {
start, err := processStartTicks(pid)
if err != nil {
diff --git a/vmm/cloudhypervisor/process_other.go b/vmm/cloudhypervisor/process_other.go
index f4f9170..00cc1fd 100644
--- a/vmm/cloudhypervisor/process_other.go
+++ b/vmm/cloudhypervisor/process_other.go
@@ -19,6 +19,8 @@ func platformPreflight() error { return errLinuxRequired }
func configureProcess(*exec.Cmd, *os.File) {}
+func startProcess(*exec.Cmd, string) error { return errLinuxRequired }
+
func captureProcess(int, types.SandboxID, uint64, string, string) (vmm.Process, error) {
return vmm.Process{}, errLinuxRequired
}
diff --git a/vmm/vmm.go b/vmm/vmm.go
index c15c2b6..03e5a3f 100644
--- a/vmm/vmm.go
+++ b/vmm/vmm.go
@@ -6,6 +6,7 @@ package vmm
import (
"errors"
"fmt"
+ "net"
"path/filepath"
"strings"
@@ -51,6 +52,9 @@ type LaunchPlan struct {
Cmdline string
// Disks are attached base-to-top followed by the private COW disk.
Disks []Disk
+ // Network is the validated host-to-VMM handoff. Its zero value disables
+ // network attachment and namespace entry.
+ Network types.NetworkSetup
}
// Validate rejects incomplete plans before an adapter creates runtime state.
@@ -81,6 +85,9 @@ func (p LaunchPlan) Validate() error {
return errors.New("image disks must be read-only and serialed by manifest position")
}
}
+ if err := p.Network.Validate(); err != nil {
+ return fmt.Errorf("launch network: %w", err)
+ }
return nil
}
@@ -91,6 +98,10 @@ type OverlayV1Config struct {
LayerCount int
// Hostname is the validated sandbox name applied by early userspace.
Hostname string
+ // Interfaces contains persisted guest identities in eth index order.
+ Interfaces []types.NetworkInterface
+ // DNSServers supplies up to two IPv4 resolvers to static kernel IP entries.
+ DNSServers []string
}
// OverlayV1Cmdline renders the public KumaBox boot ABI. Layer disks attach in
@@ -106,8 +117,53 @@ func OverlayV1Cmdline(config OverlayV1Config) (string, error) {
for position := config.LayerCount - 1; position >= 0; position-- {
serials = append(serials, fmt.Sprintf("%s%d", LayerSerialPrefix, position))
}
- return "console=hvc0 loglevel=3 boot=kumabox-overlay kumabox.layers=" + strings.Join(serials, ",") +
- " kumabox.cow=" + COWSerial + " kumabox.hostname=" + config.Hostname + " clocksource=kvm-clock rw", nil
+ var commandLine strings.Builder
+ commandLine.WriteString("console=hvc0 loglevel=3 boot=kumabox-overlay kumabox.layers=")
+ commandLine.WriteString(strings.Join(serials, ","))
+ commandLine.WriteString(" kumabox.cow=" + COWSerial + " kumabox.hostname=" + config.Hostname + " clocksource=kvm-clock rw")
+ if len(config.Interfaces) == 0 {
+ return commandLine.String(), nil
+ }
+ commandLine.WriteString(" net.ifnames=0")
+ dns, err := ipv4DNSServers(config.DNSServers)
+ if err != nil {
+ return "", err
+ }
+ for _, networkInterface := range config.Interfaces {
+ if err := networkInterface.Validate(); err != nil {
+ return "", err
+ }
+ if networkInterface.IPv4 == nil {
+ continue
+ }
+ mask := net.IP(net.CIDRMask(networkInterface.IPv4.Prefix, 32)).String()
+ parameter := fmt.Sprintf(" ip=%s::%s:%s:%s:%s:off",
+ networkInterface.IPv4.Address, networkInterface.IPv4.Gateway,
+ mask, config.Hostname, networkInterface.Name,
+ )
+ if len(dns) > 0 {
+ parameter += ":" + dns[0]
+ if len(dns) > 1 {
+ parameter += ":" + dns[1]
+ }
+ }
+ commandLine.WriteString(parameter)
+ }
+ return commandLine.String(), nil
+}
+
+func ipv4DNSServers(configured []string) ([]string, error) {
+ result := make([]string, 0, min(2, len(configured)))
+ for _, server := range configured {
+ address := net.ParseIP(server)
+ if address == nil || address.To4() == nil {
+ return nil, fmt.Errorf("overlay-v1 DNS server %q is not IPv4", server)
+ }
+ if len(result) < 2 {
+ result = append(result, server)
+ }
+ }
+ return result, nil
}
// Process identifies one Linux process generation independently of PID reuse.
diff --git a/vmm/vmm_test.go b/vmm/vmm_test.go
index 161c3b7..f998fbf 100644
--- a/vmm/vmm_test.go
+++ b/vmm/vmm_test.go
@@ -17,6 +17,26 @@ func TestOverlayV1CmdlineListsLayersTopToBase(t *testing.T) {
}
}
+func TestOverlayV1CmdlineRendersStaticNetworkAndDNS(t *testing.T) {
+ cmdline, err := OverlayV1Cmdline(OverlayV1Config{
+ LayerCount: 1,
+ Hostname: "demo",
+ Interfaces: []types.NetworkInterface{{
+ Index: 0, Name: "eth0", TAP: "tap12345678-0", MAC: "02:00:00:00:00:01",
+ Queues: 4, QueueSize: 512, Network: "bridge",
+ IPv4: &types.IPv4Config{Address: "10.42.0.7", Gateway: "10.42.0.1", Prefix: 24},
+ }},
+ DNSServers: []string{"8.8.8.8", "1.1.1.1", "9.9.9.9"},
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := " net.ifnames=0 ip=10.42.0.7::10.42.0.1:255.255.255.0:demo:eth0:off:8.8.8.8:1.1.1.1"
+ if !strings.Contains(cmdline, want) {
+ t.Fatalf("cmdline = %q, want suffix %q", cmdline, want)
+ }
+}
+
func TestLaunchPlanRequiresBaseToTopReadOnlyLayersAndFinalCOW(t *testing.T) {
plan := LaunchPlan{
SandboxID: "123e4567-e89b-42d3-a456-426614174000", Generation: 3,
From d391c9a986ed9faf97d543bd04869ddb90359eb3 Mon Sep 17 00:00:00 2001
From: YoungLH <974840768@qq.com>
Date: Tue, 22 Sep 2026 14:32:52 +0800
Subject: [PATCH 41/45] fix: include guest agent build dependencies
---
.dockerignore | 5 +++++
oci-images/ubuntu/Dockerfile | 6 ++++--
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index a966139..0127677 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -3,6 +3,10 @@
!go.sum
!agent/
!agent/**
+!errdefs/
+!errdefs/**
+!types/
+!types/**
!version/
!version/**
!cmd/
@@ -11,4 +15,5 @@
!oci-images/
!oci-images/ubuntu/
!oci-images/ubuntu/overlay.sh
+!oci-images/ubuntu/network.sh
!oci-images/ubuntu/kumabox-agent.service
diff --git a/oci-images/ubuntu/Dockerfile b/oci-images/ubuntu/Dockerfile
index dc46dab..882ca79 100644
--- a/oci-images/ubuntu/Dockerfile
+++ b/oci-images/ubuntu/Dockerfile
@@ -8,6 +8,8 @@ ARG GOPROXY=https://proxy.golang.org,direct
WORKDIR /src
COPY go.mod go.sum ./
COPY agent ./agent
+COPY errdefs ./errdefs
+COPY types ./types
COPY version ./version
COPY cmd/kumabox-agent ./cmd/kumabox-agent
RUN --mount=type=cache,target=/go/pkg/mod \
@@ -29,8 +31,8 @@ COPY oci-images/ubuntu/network.sh /usr/local/lib/kumabox/initramfs/kumabox-netwo
COPY oci-images/ubuntu/kumabox-agent.service /etc/systemd/system/kumabox-agent.service
RUN set -eu; \
- apt-get update \
- && apt-get install -y --no-install-recommends \
+ apt-get -o Acquire::Retries=5 update \
+ && apt-get -o Acquire::Retries=5 install -y --no-install-recommends \
ca-certificates \
initramfs-tools \
iproute2 \
From c74dd064663336dfdf57ebcc12f9871c146eddd0 Mon Sep 17 00:00:00 2001
From: YoungLH <974840768@qq.com>
Date: Tue, 22 Sep 2026 14:54:15 +0800
Subject: [PATCH 42/45] feat: add run command
---
cli/root.go | 1 +
cli/root_test.go | 1 +
cli/sandbox/create.go | 114 ++++++++++++++++++++++-------------
cli/sandbox/create_test.go | 32 ++++++++++
cli/sandbox/progress.go | 5 ++
cli/sandbox/run.go | 58 ++++++++++++++++++
core/sandbox.go | 22 +++++++
core/sandbox_runtime_test.go | 63 +++++++++++++++++++
8 files changed, 253 insertions(+), 43 deletions(-)
create mode 100644 cli/sandbox/run.go
diff --git a/cli/root.go b/cli/root.go
index bae0f45..49fdf61 100644
--- a/cli/root.go
+++ b/cli/root.go
@@ -147,6 +147,7 @@ func newRootCommand() (*cobra.Command, error) {
root.AddCommand(sandboxcmd.NewListCommand(provideConfig))
root.AddCommand(sandboxcmd.NewLogsCommand(provideConfig))
root.AddCommand(sandboxcmd.NewRemoveCommand(provideConfig))
+ root.AddCommand(sandboxcmd.NewRunCommand(provideConfig))
root.AddCommand(sandboxcmd.NewStartCommand(provideConfig))
root.AddCommand(sandboxcmd.NewStopCommand(provideConfig))
root.AddCommand(newVersionCommand())
diff --git a/cli/root_test.go b/cli/root_test.go
index a31fd2a..ee45103 100644
--- a/cli/root_test.go
+++ b/cli/root_test.go
@@ -142,6 +142,7 @@ func TestImageAndUsageExitCodes(t *testing.T) {
{"missing inspect sandbox", []string{"inspect"}, 2},
{"missing logs sandbox", []string{"logs"}, 2},
{"missing remove sandbox", []string{"rm"}, 2},
+ {"missing run image", []string{"run", "--name", "box"}, 2},
{"missing start sandbox", []string{"start"}, 2},
{"missing stop sandbox", []string{"stop"}, 2},
{"unexpected ps argument", []string{"ps", "box"}, 2},
diff --git a/cli/sandbox/create.go b/cli/sandbox/create.go
index 495ca99..2fc46a8 100644
--- a/cli/sandbox/create.go
+++ b/cli/sandbox/create.go
@@ -20,51 +20,84 @@ import (
// configProvider reads immutable configuration only after Cobra parses flags.
type configProvider func() config.Config
+// createOptions contains the resource flags shared by create and run. Keeping
+// parsing here gives both commands one validation contract and one set of
+// defaults.
+type createOptions struct {
+ name string
+ cpus uint32
+ memory string
+ storageSize string
+ nics int
+ networkName string
+}
+
+// defaultCreateOptions returns the public resource defaults for a new sandbox.
+func defaultCreateOptions() createOptions {
+ return createOptions{
+ cpus: types.DefaultSandboxCPUs, memory: "1GiB", storageSize: "10GiB", nics: 1,
+ }
+}
+
+// addFlags registers the resource shape accepted by sandbox creation commands.
+func (o *createOptions) addFlags(command *cobra.Command) {
+ command.Flags().StringVar(&o.name, "name", o.name, "required sandbox name")
+ command.Flags().Uint32Var(&o.cpus, "cpus", o.cpus, "number of virtual CPUs")
+ command.Flags().StringVar(&o.memory, "memory", o.memory, "guest memory (for example 1GiB)")
+ command.Flags().StringVar(&o.storageSize, "storage", o.storageSize, "logical sparse COW size (minimum 10GiB)")
+ command.Flags().IntVar(&o.nics, "nics", o.nics, "number of network interfaces (0 disables networking)")
+ command.Flags().StringVar(&o.networkName, "network", o.networkName, "CNI network name (empty selects the default)")
+}
+
+// request validates CLI values before any persistent service is opened.
+func (o createOptions) request(imageReference string) (core.CreateSandboxRequest, error) {
+ if o.cpus == 0 || o.cpus > types.MaxSandboxCPUs {
+ return core.CreateSandboxRequest{}, invalidFlag("cpus", fmt.Errorf("must be between 1 and %d", types.MaxSandboxCPUs))
+ }
+ memoryBytes, err := parseBytes(o.memory)
+ if err != nil {
+ return core.CreateSandboxRequest{}, invalidFlag("memory", err)
+ }
+ if memoryBytes < types.MinSandboxMemory {
+ return core.CreateSandboxRequest{}, invalidFlag("memory", fmt.Errorf("must be at least %d bytes", types.MinSandboxMemory))
+ }
+ storageBytes, err := parseBytes(o.storageSize)
+ if err != nil {
+ return core.CreateSandboxRequest{}, invalidFlag("storage", err)
+ }
+ if storageBytes < types.MinSandboxStorage {
+ return core.CreateSandboxRequest{}, invalidFlag("storage", fmt.Errorf("must be at least %d bytes", types.MinSandboxStorage))
+ }
+ if o.nics < 0 || o.nics > types.MaxSandboxNICs {
+ return core.CreateSandboxRequest{}, invalidFlag("nics", fmt.Errorf("must be between 0 and %d", types.MaxSandboxNICs))
+ }
+ if o.nics == 0 && o.networkName != "" {
+ return core.CreateSandboxRequest{}, invalidFlag("network", errors.New("requires at least one NIC"))
+ }
+ sandboxConfig := types.SandboxConfig{
+ Name: o.name, CPUs: o.cpus, Memory: memoryBytes, Storage: storageBytes,
+ NICs: o.nics, NetworkName: o.networkName,
+ }
+ if err := sandboxConfig.Validate(); err != nil {
+ return core.CreateSandboxRequest{}, err
+ }
+ return core.CreateSandboxRequest{ImageReference: imageReference, Config: sandboxConfig}, nil
+}
+
// NewCreateCommand builds the top-level create command.
func NewCreateCommand(configuration configProvider) *cobra.Command {
- name := ""
- cpus := types.DefaultSandboxCPUs
- memory := "1GiB"
- storageSize := "10GiB"
- nics := 1
- networkName := ""
+ options := defaultCreateOptions()
asJSON := false
command := &cobra.Command{
Use: "create IMAGE",
Short: "create a sandbox without starting it",
Args: cobra.ExactArgs(1),
RunE: func(command *cobra.Command, args []string) (returnErr error) {
- if cpus == 0 || cpus > types.MaxSandboxCPUs {
- return invalidFlag("cpus", fmt.Errorf("must be between 1 and %d", types.MaxSandboxCPUs))
- }
- memoryBytes, err := parseBytes(memory)
- if err != nil {
- return invalidFlag("memory", err)
- }
- if memoryBytes < types.MinSandboxMemory {
- return invalidFlag("memory", fmt.Errorf("must be at least %d bytes", types.MinSandboxMemory))
- }
- storageBytes, err := parseBytes(storageSize)
+ request, err := options.request(args[0])
if err != nil {
- return invalidFlag("storage", err)
- }
- if storageBytes < types.MinSandboxStorage {
- return invalidFlag("storage", fmt.Errorf("must be at least %d bytes", types.MinSandboxStorage))
- }
- if nics < 0 || nics > types.MaxSandboxNICs {
- return invalidFlag("nics", fmt.Errorf("must be between 0 and %d", types.MaxSandboxNICs))
- }
- if nics == 0 && networkName != "" {
- return invalidFlag("network", errors.New("requires at least one NIC"))
- }
- sandboxConfig := types.SandboxConfig{
- Name: name, CPUs: cpus, Memory: memoryBytes, Storage: storageBytes,
- NICs: nics, NetworkName: networkName,
- }
- if err := sandboxConfig.Validate(); err != nil {
return err
}
- progress, err := startCreateProgress(command, name)
+ progress, err := startCreateProgress(command, options.name)
if err != nil {
return err
}
@@ -76,25 +109,20 @@ func NewCreateCommand(configuration configProvider) *cobra.Command {
committed := false
defer func() {
closeErr := service.Close()
- returnErr = errors.Join(returnErr, errdefs.Context(closeErr, "create sandbox", name, "close metadata", "inspect the sandbox before retrying", committed))
+ returnErr = errors.Join(returnErr, errdefs.Context(closeErr, "create sandbox", options.name, "close metadata", "inspect the sandbox before retrying", committed))
}()
- record, err := service.Create(command.Context(), core.CreateSandboxRequest{ImageReference: args[0], Config: sandboxConfig})
+ record, err := service.Create(command.Context(), request)
if err != nil {
return err
}
committed = true
if err := writeSandboxResult(progress.Output(command.OutOrStdout()), record, asJSON); err != nil {
- return errdefs.Context(err, "create sandbox", name, "output", "sandbox was created; inspect it before retrying", true)
+ return errdefs.Context(err, "create sandbox", options.name, "output", "sandbox was created; inspect it before retrying", true)
}
return nil
},
}
- command.Flags().StringVar(&name, "name", name, "required sandbox name")
- command.Flags().Uint32Var(&cpus, "cpus", cpus, "number of virtual CPUs")
- command.Flags().StringVar(&memory, "memory", memory, "guest memory (for example 1GiB)")
- command.Flags().StringVar(&storageSize, "storage", storageSize, "logical sparse COW size (minimum 10GiB)")
- command.Flags().IntVar(&nics, "nics", nics, "number of network interfaces (0 disables networking)")
- command.Flags().StringVar(&networkName, "network", networkName, "CNI network name (empty selects the default)")
+ options.addFlags(command)
command.Flags().BoolVar(&asJSON, "json", false, "print the created sandbox as indented JSON")
return command
}
diff --git a/cli/sandbox/create_test.go b/cli/sandbox/create_test.go
index c74e44e..489af0d 100644
--- a/cli/sandbox/create_test.go
+++ b/cli/sandbox/create_test.go
@@ -128,6 +128,38 @@ func TestCreateCommandDefaultsToOneNIC(t *testing.T) {
}
}
+func TestRunCommandUsesCreateResourceContract(t *testing.T) {
+ createCommand := NewCreateCommand(func() config.Config { return config.Config{} })
+ runCommand := NewRunCommand(func() config.Config { return config.Config{} })
+ for _, name := range []string{"name", "cpus", "memory", "storage", "nics", "network"} {
+ createFlag, runFlag := createCommand.Flags().Lookup(name), runCommand.Flags().Lookup(name)
+ if createFlag == nil || runFlag == nil {
+ t.Fatalf("shared flag --%s is missing", name)
+ }
+ if runFlag.DefValue != createFlag.DefValue {
+ t.Fatalf("run --%s default = %q, want create default %q", name, runFlag.DefValue, createFlag.DefValue)
+ }
+ }
+ if flag := runCommand.Flags().Lookup("json"); flag == nil {
+ t.Fatal("run --json is missing")
+ }
+}
+
+func TestRunCommandValidatesResourcesBeforeOpeningService(t *testing.T) {
+ command := NewRunCommand(func() config.Config {
+ t.Fatal("resource validation opened the sandbox service")
+ return config.Config{}
+ })
+ command.SetArgs([]string{"demo", "--name", "box", "--memory", "1MiB"})
+ err := command.ExecuteContext(t.Context())
+ if err == nil || !strings.Contains(err.Error(), "--memory") {
+ t.Fatalf("run error = %v, want --memory", err)
+ }
+ if code, ok := errdefs.CodeOf(err); !ok || code != errdefs.CodeInvalidArgument {
+ t.Fatalf("run error code = %q, %v", code, ok)
+ }
+}
+
func TestCreateProgressReportsCommittedOutputFailure(t *testing.T) {
var stderr bytes.Buffer
progress, err := newTestProgress(&stderr)
diff --git a/cli/sandbox/progress.go b/cli/sandbox/progress.go
index 72a63cf..7e96391 100644
--- a/cli/sandbox/progress.go
+++ b/cli/sandbox/progress.go
@@ -41,6 +41,11 @@ func startCreateProgress(command *cobra.Command, name string) (*sandboxProgress,
return startProgress(command, "create sandbox", fmt.Sprintf("Create %q", name), "preparing sandbox", "inspect the sandbox state")
}
+// startRunProgress starts progress for one create-and-launch operation.
+func startRunProgress(command *cobra.Command, name string) (*sandboxProgress, error) {
+ return startProgress(command, "run sandbox", fmt.Sprintf("Run %q", name), "preparing sandbox", "inspect the sandbox state and VMM log")
+}
+
// startRemoveProgress starts progress for one remove operation.
func startRemoveProgress(command *cobra.Command, reference string) (*sandboxProgress, error) {
return startProgress(command, "remove sandbox", fmt.Sprintf("Remove %q", reference), "preparing removal", "retry removal or inspect retained state")
diff --git a/cli/sandbox/run.go b/cli/sandbox/run.go
new file mode 100644
index 0000000..18f7051
--- /dev/null
+++ b/cli/sandbox/run.go
@@ -0,0 +1,58 @@
+package sandbox
+
+import (
+ "errors"
+
+ "github.com/spf13/cobra"
+
+ "github.com/kumabox/kumabox/core"
+ "github.com/kumabox/kumabox/errdefs"
+)
+
+// NewRunCommand builds the top-level create-and-start command.
+func NewRunCommand(configuration configProvider) *cobra.Command {
+ options := defaultCreateOptions()
+ asJSON := false
+ command := &cobra.Command{
+ Use: "run IMAGE",
+ Short: "create and start a sandbox",
+ Args: cobra.ExactArgs(1),
+ RunE: func(command *cobra.Command, args []string) (returnErr error) {
+ request, err := options.request(args[0])
+ if err != nil {
+ return err
+ }
+ progress, err := startRunProgress(command, options.name)
+ if err != nil {
+ return err
+ }
+ defer func() { returnErr = errors.Join(returnErr, progress.Finish(returnErr)) }()
+
+ service, err := core.OpenSandbox(command.Context(), configuration(), progress)
+ if err != nil {
+ return err
+ }
+ committed := false
+ defer func() {
+ closeErr := service.Close()
+ returnErr = errors.Join(returnErr, errdefs.Context(
+ closeErr, "run sandbox", options.name, "close metadata",
+ "inspect the sandbox before retrying", committed,
+ ))
+ }()
+
+ record, err := service.Run(command.Context(), request)
+ if err != nil {
+ return err
+ }
+ committed = true
+ if err := writeSandboxResult(progress.Output(command.OutOrStdout()), record, asJSON); err != nil {
+ return errdefs.Context(err, "run sandbox", options.name, "output", "sandbox is running; inspect it before retrying", true)
+ }
+ return nil
+ },
+ }
+ options.addFlags(command)
+ command.Flags().BoolVar(&asJSON, "json", false, "print the running sandbox as indented JSON")
+ return command
+}
diff --git a/core/sandbox.go b/core/sandbox.go
index 89485c4..07cc4a9 100644
--- a/core/sandbox.go
+++ b/core/sandbox.go
@@ -232,6 +232,28 @@ func (s *SandboxService) networkProvider(record types.Sandbox) (network.Provider
return provider, true, err
}
+// Run creates and starts one sandbox as a single application use case. Create
+// owns compensation until Created is durable; after that point a failed start
+// retains the sandbox and its failure state for inspection and retry.
+//
+// image + config -> Create -> Created -> Start -> Running
+// | |
+// +----------+-> retained on start failure
+func (s *SandboxService) Run(ctx context.Context, request CreateSandboxRequest) (types.Sandbox, error) {
+ created, err := s.Create(ctx, request)
+ if err != nil {
+ return types.Sandbox{}, err
+ }
+ running, err := s.Start(ctx, created.ID.String())
+ if err != nil {
+ return created, errdefs.Context(
+ err, "run sandbox", request.Config.Name, "start",
+ "inspect the retained sandbox and VMM log before retrying", true,
+ )
+ }
+ return running, nil
+}
+
// List returns a consistent sandbox snapshot. Unless includeAll is true, only
// states associated with an active VMM operation are returned.
func (s *SandboxService) List(ctx context.Context, includeAll bool) ([]types.Sandbox, error) {
diff --git a/core/sandbox_runtime_test.go b/core/sandbox_runtime_test.go
index 6fcc09c..a034a80 100644
--- a/core/sandbox_runtime_test.go
+++ b/core/sandbox_runtime_test.go
@@ -15,6 +15,69 @@ import (
"github.com/kumabox/kumabox/vmm"
)
+func TestRunCreatesAndStartsSandbox(t *testing.T) {
+ service, steps := newTestSandboxService(t, nil)
+ record, err := service.Run(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage,
+ },
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record.State != types.SandboxStateRunning || record.Generation != 4 {
+ t.Fatalf("running record = %+v", record)
+ }
+ want := []string{
+ "status:resolving and checking image", "verify", "reserve",
+ "status:creating sparse ext4 disk", "disk",
+ "status:committing created state", "created", "report",
+ "status:resolving sandbox", "resolve",
+ "status:waiting for sandbox operation lock", "resolve",
+ "status:checking existing runtime", "observe", "cleanup",
+ "status:checking host runtime", "preflight",
+ "status:verifying image and sandbox disk", "verify", "check",
+ "status:committing starting state", "starting",
+ "status:launching cloud-hypervisor", "launch",
+ "status:committing running state", "running", "report",
+ }
+ if !reflect.DeepEqual(*steps, want) {
+ t.Fatalf("steps = %v, want %v", *steps, want)
+ }
+}
+
+func TestRunRetainsSandboxWhenStartFails(t *testing.T) {
+ service, steps := newTestSandboxService(t, nil)
+ failure := errors.New("VMM exited")
+ testRuntime(t, service).launchErr = failure
+ _, err := service.Run(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 1, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage,
+ },
+ })
+ if !errors.Is(err, failure) {
+ t.Fatalf("Run error = %v", err)
+ }
+ var classified *errdefs.Error
+ if !errors.As(err, &classified) || !classified.Committed || classified.Operation != "run sandbox" {
+ t.Fatalf("Run did not report retained state: %v", err)
+ }
+ record := service.dependencies.catalog.(*fakeCatalog).record
+ if record.State != types.SandboxStateError || record.Failure == nil || record.Failure.Phase != "launch VMM" {
+ t.Fatalf("retained record = %+v", record)
+ }
+ joined := strings.Join(*steps, ",")
+ if !strings.Contains(joined, "created,report,status:resolving sandbox") ||
+ !strings.Contains(joined, "launch,abort,start-error") ||
+ strings.Contains(joined, "forget") {
+ t.Fatalf("Run failure steps = %v", *steps)
+ }
+}
+
func TestSandboxLifecycleRoutesToPersistedVMM(t *testing.T) {
service, steps := newTestSandboxService(t, nil)
firecracker := &fakeRuntime{typ: types.VMMFirecracker, steps: steps, observation: vmm.Observation{State: vmm.ProcessAbsent}}
From a798d3215a8acb75d5affbaf9d479d3611abeb31 Mon Sep 17 00:00:00 2001
From: YoungLH <974840768@qq.com>
Date: Tue, 22 Sep 2026 15:16:22 +0800
Subject: [PATCH 43/45] feat: add live snapshot management
---
cli/root.go | 2 +
cli/root_test.go | 4 +
cli/snapshot/command.go | 134 +++++++++++++
cli/snapshot/output.go | 101 ++++++++++
cli/snapshot/output_test.go | 68 +++++++
cli/snapshot/progress.go | 64 ++++++
core/images.go | 4 +-
core/sandbox_test.go | 20 ++
core/snapshot.go | 294 +++++++++++++++++++++++++++
core/snapshot_test.go | 144 ++++++++++++++
metadata/sqlite/store.go | 4 +-
metadata/sqlite/store_test.go | 38 ++++
snapshot/catalog/store.go | 343 ++++++++++++++++++++++++++++++++
snapshot/catalog/store_test.go | 49 +++++
snapshot/paths.go | 172 ++++++++++++++++
storage/copy_linux.go | 75 +++++++
storage/copy_linux_test.go | 42 ++++
storage/copy_other.go | 29 +++
storage/directory.go | 69 +++++++
types/snapshot.go | 85 ++++++++
types/snapshot_test.go | 31 +++
vmm/backend.go | 24 +++
vmm/cloudhypervisor/snapshot.go | 88 ++++++++
vmm/vmm.go | 21 ++
24 files changed, 1902 insertions(+), 3 deletions(-)
create mode 100644 cli/snapshot/command.go
create mode 100644 cli/snapshot/output.go
create mode 100644 cli/snapshot/output_test.go
create mode 100644 cli/snapshot/progress.go
create mode 100644 core/snapshot.go
create mode 100644 core/snapshot_test.go
create mode 100644 snapshot/catalog/store.go
create mode 100644 snapshot/catalog/store_test.go
create mode 100644 snapshot/paths.go
create mode 100644 storage/copy_linux.go
create mode 100644 storage/copy_linux_test.go
create mode 100644 storage/copy_other.go
create mode 100644 storage/directory.go
create mode 100644 types/snapshot.go
create mode 100644 types/snapshot_test.go
create mode 100644 vmm/cloudhypervisor/snapshot.go
diff --git a/cli/root.go b/cli/root.go
index 49fdf61..76dd20e 100644
--- a/cli/root.go
+++ b/cli/root.go
@@ -13,6 +13,7 @@ import (
doctorcmd "github.com/kumabox/kumabox/cli/doctor"
imagecmd "github.com/kumabox/kumabox/cli/image"
sandboxcmd "github.com/kumabox/kumabox/cli/sandbox"
+ snapshotcmd "github.com/kumabox/kumabox/cli/snapshot"
"github.com/kumabox/kumabox/config"
"github.com/kumabox/kumabox/errdefs"
"github.com/kumabox/kumabox/version"
@@ -150,6 +151,7 @@ func newRootCommand() (*cobra.Command, error) {
root.AddCommand(sandboxcmd.NewRunCommand(provideConfig))
root.AddCommand(sandboxcmd.NewStartCommand(provideConfig))
root.AddCommand(sandboxcmd.NewStopCommand(provideConfig))
+ root.AddCommand(snapshotcmd.NewCommand(provideConfig))
root.AddCommand(newVersionCommand())
classifyArguments(root)
return root, nil
diff --git a/cli/root_test.go b/cli/root_test.go
index ee45103..7375f72 100644
--- a/cli/root_test.go
+++ b/cli/root_test.go
@@ -145,6 +145,10 @@ func TestImageAndUsageExitCodes(t *testing.T) {
{"missing run image", []string{"run", "--name", "box"}, 2},
{"missing start sandbox", []string{"start"}, 2},
{"missing stop sandbox", []string{"stop"}, 2},
+ {"missing snapshot save sandbox", []string{"snapshot", "save"}, 2},
+ {"missing snapshot inspect reference", []string{"snapshot", "inspect"}, 2},
+ {"missing snapshot remove reference", []string{"snapshot", "rm"}, 2},
+ {"unexpected snapshot list argument", []string{"snapshot", "ls", "extra"}, 2},
{"unexpected ps argument", []string{"ps", "box"}, 2},
{"unsupported inspect flag", []string{"inspect", "box", "--json"}, 2},
{"unknown flag", []string{"image", "ls", "--wrong"}, 2},
diff --git a/cli/snapshot/command.go b/cli/snapshot/command.go
new file mode 100644
index 0000000..e75d9a5
--- /dev/null
+++ b/cli/snapshot/command.go
@@ -0,0 +1,134 @@
+// Package snapshot exposes snapshot lifecycle commands through Cobra.
+package snapshot
+
+import (
+ "errors"
+
+ "github.com/spf13/cobra"
+
+ "github.com/kumabox/kumabox/config"
+ "github.com/kumabox/kumabox/core"
+ "github.com/kumabox/kumabox/errdefs"
+)
+
+type configProvider func() config.Config
+
+// NewCommand builds the snapshot command group.
+func NewCommand(configuration configProvider) *cobra.Command {
+ command := &cobra.Command{Use: "snapshot", Short: "manage sandbox snapshots"}
+ command.AddCommand(newSaveCommand(configuration), newListCommand(configuration), newInspectCommand(configuration), newRemoveCommand(configuration))
+ return command
+}
+
+func newSaveCommand(configuration configProvider) *cobra.Command {
+ var name, description string
+ var asJSON bool
+ command := &cobra.Command{
+ Use: "save SANDBOX",
+ Short: "save a live snapshot of a running sandbox",
+ Args: cobra.ExactArgs(1),
+ RunE: func(command *cobra.Command, args []string) (returnErr error) {
+ progress, err := newProgress(command, args[0])
+ if err != nil {
+ return err
+ }
+ defer func() { returnErr = errors.Join(returnErr, progress.Finish(returnErr)) }()
+ service, err := core.OpenSnapshots(command.Context(), configuration(), progress)
+ if err != nil {
+ return err
+ }
+ committed := false
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(service.Close(), "save snapshot", args[0], "close metadata", "inspect the snapshot before retrying", committed))
+ }()
+ record, err := service.Save(command.Context(), core.SaveSnapshotRequest{
+ SandboxReference: args[0], Name: name, Description: description,
+ })
+ if err != nil {
+ return err
+ }
+ committed = true
+ return writeResult(progress.Output(command.OutOrStdout()), record, asJSON)
+ },
+ }
+ command.Flags().StringVar(&name, "name", "", "optional unique snapshot name")
+ command.Flags().StringVar(&description, "description", "", "optional snapshot description")
+ command.Flags().BoolVar(&asJSON, "json", false, "print the saved snapshot as indented JSON")
+ return command
+}
+
+func newListCommand(configuration configProvider) *cobra.Command {
+ var asJSON bool
+ command := &cobra.Command{
+ Use: "ls",
+ Aliases: []string{"list"},
+ Short: "list snapshots",
+ Args: cobra.NoArgs,
+ RunE: func(command *cobra.Command, _ []string) (returnErr error) {
+ service, err := core.OpenSnapshots(command.Context(), configuration(), nil)
+ if err != nil {
+ return err
+ }
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(service.Close(), "list snapshots", "", "close metadata", "retry the query", false))
+ }()
+ records, err := service.List(command.Context())
+ if err != nil {
+ return err
+ }
+ if asJSON {
+ return writeListJSON(command.OutOrStdout(), records)
+ }
+ return writeTable(command.OutOrStdout(), records)
+ },
+ }
+ command.Flags().BoolVar(&asJSON, "json", false, "print snapshots as indented JSON")
+ return command
+}
+
+func newInspectCommand(configuration configProvider) *cobra.Command {
+ return &cobra.Command{
+ Use: "inspect SNAPSHOT",
+ Short: "show detailed snapshot information as JSON",
+ Args: cobra.ExactArgs(1),
+ RunE: func(command *cobra.Command, args []string) (returnErr error) {
+ service, err := core.OpenSnapshots(command.Context(), configuration(), nil)
+ if err != nil {
+ return err
+ }
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(service.Close(), "inspect snapshot", args[0], "close metadata", "retry the query", false))
+ }()
+ record, err := service.Inspect(command.Context(), args[0])
+ if err != nil {
+ return err
+ }
+ return writeJSON(command.OutOrStdout(), record)
+ },
+ }
+}
+
+func newRemoveCommand(configuration configProvider) *cobra.Command {
+ var asJSON bool
+ command := &cobra.Command{
+ Use: "rm SNAPSHOT",
+ Short: "remove a snapshot",
+ Args: cobra.ExactArgs(1),
+ RunE: func(command *cobra.Command, args []string) (returnErr error) {
+ service, err := core.OpenSnapshots(command.Context(), configuration(), nil)
+ if err != nil {
+ return err
+ }
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(service.Close(), "remove snapshot", args[0], "close metadata", "retry snapshot removal", true))
+ }()
+ record, err := service.Remove(command.Context(), args[0])
+ if err != nil {
+ return err
+ }
+ return writeResult(command.OutOrStdout(), record, asJSON)
+ },
+ }
+ command.Flags().BoolVar(&asJSON, "json", false, "print the removed snapshot as indented JSON")
+ return command
+}
diff --git a/cli/snapshot/output.go b/cli/snapshot/output.go
new file mode 100644
index 0000000..81c9cda
--- /dev/null
+++ b/cli/snapshot/output.go
@@ -0,0 +1,101 @@
+package snapshot
+
+import (
+ "encoding/json"
+ "fmt"
+ "io"
+ "text/tabwriter"
+ "time"
+
+ "github.com/kumabox/kumabox/types"
+)
+
+type output struct {
+ ID string `json:"id"`
+ Name string `json:"name,omitempty"`
+ Description string `json:"description,omitempty"`
+ SandboxID string `json:"sandbox_id"`
+ SourceGeneration uint64 `json:"source_generation"`
+ ImageDigest string `json:"image_digest"`
+ VMM string `json:"vmm"`
+ Config configOutput `json:"config"`
+ Size int64 `json:"size"`
+ CreatedAt time.Time `json:"created_at"`
+}
+
+type configOutput struct {
+ Name string `json:"name"`
+ CPUs uint32 `json:"cpus"`
+ Memory int64 `json:"memory"`
+ Storage int64 `json:"storage"`
+ NICs int `json:"nics"`
+ NetworkName string `json:"network_name,omitempty"`
+}
+
+func result(snapshot types.Snapshot) output {
+ return output{
+ ID: snapshot.ID.String(), Name: snapshot.Name, Description: snapshot.Description,
+ SandboxID: snapshot.SandboxID.String(), SourceGeneration: snapshot.SourceGeneration,
+ ImageDigest: snapshot.ImageDigest.String(), VMM: string(snapshot.VMM),
+ Config: configOutput{
+ Name: snapshot.Config.Name, CPUs: snapshot.Config.CPUs, Memory: snapshot.Config.Memory,
+ Storage: snapshot.Config.Storage, NICs: snapshot.Config.NICs, NetworkName: snapshot.Config.NetworkName,
+ },
+ Size: snapshot.Size, CreatedAt: snapshot.CreatedAt.UTC(),
+ }
+}
+
+func writeJSON(writer io.Writer, snapshot types.Snapshot) error {
+ encoder := json.NewEncoder(writer)
+ encoder.SetIndent("", " ")
+ return encoder.Encode(result(snapshot))
+}
+
+func writeResult(writer io.Writer, snapshot types.Snapshot, asJSON bool) error {
+ if asJSON {
+ return writeJSON(writer, snapshot)
+ }
+ _, err := fmt.Fprintln(writer, snapshot.ID)
+ return err
+}
+
+func writeListJSON(writer io.Writer, snapshots []types.Snapshot) error {
+ results := make([]output, 0, len(snapshots))
+ for _, snapshot := range snapshots {
+ results = append(results, result(snapshot))
+ }
+ encoder := json.NewEncoder(writer)
+ encoder.SetIndent("", " ")
+ return encoder.Encode(results)
+}
+
+func writeTable(writer io.Writer, snapshots []types.Snapshot) error {
+ table := tabwriter.NewWriter(writer, 0, 4, 2, ' ', 0)
+ if _, err := fmt.Fprintln(table, "SNAPSHOT ID\tNAME\tSANDBOX ID\tCPUS\tMEMORY\tSIZE\tDESCRIPTION\tCREATED"); err != nil {
+ return err
+ }
+ for _, snapshot := range snapshots {
+ if _, err := fmt.Fprintf(table, "%s\t%s\t%s\t%d\t%s\t%s\t%s\t%s\n",
+ snapshot.ID, snapshot.Name, snapshot.SandboxID, snapshot.Config.CPUs,
+ formatIECBytes(snapshot.Config.Memory), formatIECBytes(snapshot.Size), snapshot.Description,
+ snapshot.CreatedAt.UTC().Format(time.RFC3339),
+ ); err != nil {
+ return err
+ }
+ }
+ return table.Flush()
+}
+
+func formatIECBytes(size int64) string {
+ if size < 1024 {
+ return fmt.Sprintf("%dB", size)
+ }
+ value := float64(size)
+ for _, unit := range []string{"KiB", "MiB", "GiB", "TiB", "PiB", "EiB"} {
+ value /= 1024
+ if value < 1024 || unit == "EiB" {
+ return fmt.Sprintf("%.1f%s", value, unit)
+ }
+ }
+ return fmt.Sprintf("%dB", size)
+}
diff --git a/cli/snapshot/output_test.go b/cli/snapshot/output_test.go
new file mode 100644
index 0000000..7147a5f
--- /dev/null
+++ b/cli/snapshot/output_test.go
@@ -0,0 +1,68 @@
+package snapshot
+
+import (
+ "bytes"
+ "encoding/json"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/kumabox/kumabox/types"
+)
+
+func TestSnapshotOutputIsIndentedAndTableHasHeaders(t *testing.T) {
+ digest, err := types.ParseDigest("sha256:" + strings.Repeat("a", 64))
+ if err != nil {
+ t.Fatal(err)
+ }
+ record := types.Snapshot{
+ ID: types.SnapshotID("223e4567-e89b-42d3-a456-426614174000"), Name: "checkpoint",
+ SandboxID: types.SandboxID("123e4567-e89b-42d3-a456-426614174000"), SourceGeneration: 4,
+ ImageDigest: digest, VMM: types.VMMCloudHypervisor,
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 2, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage, NICs: 1, NetworkName: "default",
+ },
+ Size: 42, CreatedAt: time.Date(2026, 9, 22, 12, 0, 0, 0, time.UTC),
+ }
+ var jsonOutput bytes.Buffer
+ if err := writeJSON(&jsonOutput, record); err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(jsonOutput.String(), "\n \"id\"") || !strings.Contains(jsonOutput.String(), "\"cpus\": 2") {
+ t.Fatalf("snapshot JSON = %q", jsonOutput.String())
+ }
+ var decoded output
+ if err := json.Unmarshal(jsonOutput.Bytes(), &decoded); err != nil {
+ t.Fatal(err)
+ }
+ if decoded.Config.Name != "box" || decoded.Config.NetworkName != "default" {
+ t.Fatalf("snapshot output = %+v", decoded)
+ }
+ var table bytes.Buffer
+ if err := writeTable(&table, []types.Snapshot{record}); err != nil {
+ t.Fatal(err)
+ }
+ for _, text := range []string{"SNAPSHOT ID", "SANDBOX ID", record.ID.String(), "checkpoint", "1.0GiB"} {
+ if !strings.Contains(table.String(), text) {
+ t.Fatalf("snapshot table missing %q:\n%s", text, table.String())
+ }
+ }
+}
+
+func TestEmptySnapshotOutputsUseHeadersAndArray(t *testing.T) {
+ var table bytes.Buffer
+ if err := writeTable(&table, nil); err != nil {
+ t.Fatal(err)
+ }
+ if strings.Count(table.String(), "\n") != 1 {
+ t.Fatalf("empty table = %q", table.String())
+ }
+ var jsonOutput bytes.Buffer
+ if err := writeListJSON(&jsonOutput, nil); err != nil {
+ t.Fatal(err)
+ }
+ if jsonOutput.String() != "[]\n" {
+ t.Fatalf("empty JSON = %q", jsonOutput.String())
+ }
+}
diff --git a/cli/snapshot/progress.go b/cli/snapshot/progress.go
new file mode 100644
index 0000000..e28c341
--- /dev/null
+++ b/cli/snapshot/progress.go
@@ -0,0 +1,64 @@
+package snapshot
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "sync"
+
+ "github.com/spf13/cobra"
+
+ cliprogress "github.com/kumabox/kumabox/cli/progress"
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/types"
+)
+
+type progress struct {
+ mu sync.Mutex
+ renderer *cliprogress.Renderer
+ label string
+ committed bool
+}
+
+func newProgress(command *cobra.Command, reference string) (*progress, error) {
+ label := fmt.Sprintf("Snapshot %q", reference)
+ renderer, err := cliprogress.New(command.Context(), command.ErrOrStderr(), label+" · preparing snapshot")
+ if err != nil {
+ return nil, err
+ }
+ return &progress{renderer: renderer, label: label}, nil
+}
+
+func (p *progress) Status(status string) error {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ return p.renderer.Update(p.label + " · " + status)
+}
+
+func (p *progress) Committed(types.Snapshot) error {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ p.committed = true
+ return p.renderer.Update(p.label + " · finishing")
+}
+
+func (p *progress) Output(writer io.Writer) io.Writer { return p.renderer.Output(writer) }
+
+func (p *progress) Finish(operationErr error) error {
+ p.mu.Lock()
+ var classified *errdefs.Error
+ if errors.As(operationErr, &classified) && classified.Committed {
+ p.committed = true
+ }
+ outcome := cliprogress.Succeeded
+ if operationErr != nil || p.renderer.Err() != nil {
+ if p.committed {
+ outcome = cliprogress.CommittedWithErrors
+ } else {
+ outcome = cliprogress.Failed
+ }
+ }
+ label := p.label
+ p.mu.Unlock()
+ return p.renderer.Finish(label, outcome, "")
+}
diff --git a/core/images.go b/core/images.go
index 6855e9f..8ffc9d8 100644
--- a/core/images.go
+++ b/core/images.go
@@ -23,6 +23,7 @@ import (
"github.com/kumabox/kumabox/metadata/sqlite"
networkcni "github.com/kumabox/kumabox/network/cni"
sandboxcatalog "github.com/kumabox/kumabox/sandbox/catalog"
+ snapshotcatalog "github.com/kumabox/kumabox/snapshot/catalog"
"github.com/kumabox/kumabox/types"
)
@@ -126,5 +127,6 @@ func NewRegistrySource(reference string) (images.Source, string, error) {
func metadataCollections() []metadata.Collection {
result := catalog.Collections()
result = append(result, sandboxcatalog.Collections()...)
- return append(result, networkcni.Collections()...)
+ result = append(result, networkcni.Collections()...)
+ return append(result, snapshotcatalog.Collections()...)
}
diff --git a/core/sandbox_test.go b/core/sandbox_test.go
index da6a2b2..7d10062 100644
--- a/core/sandbox_test.go
+++ b/core/sandbox_test.go
@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"io"
+ "os"
"path/filepath"
"runtime"
"strings"
@@ -291,6 +292,25 @@ type fakeRuntime struct {
logsErr error
removeLogsErr error
logOptions vmm.LogOptions
+ snapshotPlan vmm.SnapshotPlan
+ snapshotErr error
+}
+
+func (f *fakeRuntime) Snapshot(_ context.Context, plan vmm.SnapshotPlan) error {
+ *f.steps = append(*f.steps, "snapshot")
+ f.snapshotPlan = plan
+ if f.snapshotErr != nil {
+ return f.snapshotErr
+ }
+ if err := os.WriteFile(filepath.Join(plan.Destination, "config.json"), []byte("{}"), 0o600); err != nil {
+ return err
+ }
+ for _, file := range plan.WritableFiles {
+ if err := os.WriteFile(file.Destination, []byte("cow"), 0o600); err != nil {
+ return err
+ }
+ }
+ return nil
}
func (f *fakeRuntime) Type() types.VMMType {
diff --git a/core/snapshot.go b/core/snapshot.go
new file mode 100644
index 0000000..2b22e6e
--- /dev/null
+++ b/core/snapshot.go
@@ -0,0 +1,294 @@
+package core
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "time"
+
+ "github.com/kumabox/kumabox/config"
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/images"
+ imagecatalog "github.com/kumabox/kumabox/images/catalog"
+ filelock "github.com/kumabox/kumabox/lock/flock"
+ "github.com/kumabox/kumabox/metadata"
+ "github.com/kumabox/kumabox/metadata/sqlite"
+ sandboxfs "github.com/kumabox/kumabox/sandbox"
+ sandboxcatalog "github.com/kumabox/kumabox/sandbox/catalog"
+ "github.com/kumabox/kumabox/snapshot"
+ snapshotcatalog "github.com/kumabox/kumabox/snapshot/catalog"
+ "github.com/kumabox/kumabox/types"
+ "github.com/kumabox/kumabox/vmm"
+)
+
+// SaveSnapshotRequest contains operator labels for one live sandbox capture.
+type SaveSnapshotRequest struct {
+ // SandboxReference is the source sandbox name or complete ID.
+ SandboxReference string
+ // Name is an optional unique snapshot lookup key.
+ Name string
+ // Description is optional operator context stored with the snapshot.
+ Description string
+}
+
+// SnapshotReporter receives capture stages without controlling the workflow.
+type SnapshotReporter interface {
+ Status(string) error
+ Committed(types.Snapshot) error
+}
+
+type snapshotCatalog interface {
+ Reserve(context.Context, types.Snapshot) error
+ Commit(context.Context, types.SnapshotID, int64) (types.Snapshot, error)
+ Forget(context.Context, types.SnapshotID) error
+ Resolve(context.Context, string) (types.Snapshot, error)
+ List(context.Context) ([]types.Snapshot, error)
+ BeginDelete(context.Context, string) (types.Snapshot, error)
+ FinalizeDelete(context.Context, types.SnapshotID) error
+}
+
+// SnapshotService coordinates sandbox locking, VMM capture, artifact
+// publication, and snapshot metadata.
+type SnapshotService struct {
+ paths snapshot.Paths
+ sandboxPaths sandboxfs.Paths
+ sandboxes sandboxCatalog
+ snapshots snapshotCatalog
+ runtimes *vmm.Registry
+ reporter SnapshotReporter
+ newID func() (types.SnapshotID, error)
+ now func() time.Time
+ store metadata.Store
+}
+
+// OpenSnapshots assembles the local snapshot service. The caller must close it.
+func OpenSnapshots(ctx context.Context, configuration config.Config, reporter SnapshotReporter) (*SnapshotService, error) {
+ if err := configuration.Validate(); err != nil {
+ return nil, err
+ }
+ imagePaths, err := images.NewPaths(configuration.Paths)
+ if err != nil {
+ return nil, err
+ }
+ sandboxPaths, err := sandboxfs.NewPaths(configuration.Paths)
+ if err != nil {
+ return nil, err
+ }
+ snapshotPaths, err := snapshot.NewPaths(configuration.Paths)
+ if err != nil {
+ return nil, err
+ }
+ if err := errors.Join(imagePaths.Ensure(), sandboxPaths.Ensure(), snapshotPaths.Ensure()); err != nil {
+ return nil, err
+ }
+ store, err := sqlite.Open(ctx, imagePaths.MetadataDB(), metadataCollections(), sqlite.Options{
+ BusyTimeout: configuration.Metadata.BusyTimeout,
+ RetryLimit: configuration.Metadata.RetryLimit,
+ })
+ if err != nil {
+ return nil, err
+ }
+ runtimes, err := openVMMRegistry(configuration)
+ if err != nil {
+ return nil, errors.Join(err, store.Close())
+ }
+ if reporter == nil {
+ reporter = discardSnapshotReporter{}
+ }
+ return &SnapshotService{
+ paths: snapshotPaths, sandboxPaths: sandboxPaths,
+ sandboxes: sandboxcatalog.New(store, imagecatalog.Reader{}), snapshots: snapshotcatalog.New(store),
+ runtimes: runtimes, reporter: reporter, newID: types.NewSnapshotID, now: time.Now, store: store,
+ }, nil
+}
+
+// Close releases the shared metadata engine.
+func (s *SnapshotService) Close() error {
+ if s == nil || s.store == nil {
+ return nil
+ }
+ return s.store.Close()
+}
+
+// Save captures native VMM state and the writable COW disk at one paused point.
+// The source resumes before artifact publication and metadata commit.
+//
+// Running -> lock -> reserve -> stage -> pause/capture/resume -> publish -> ready
+// \--- failure: clean stage + reservation ---/
+func (s *SnapshotService) Save(ctx context.Context, request SaveSnapshotRequest) (result types.Snapshot, returnErr error) {
+ if s == nil || s.sandboxes == nil || s.snapshots == nil || s.runtimes == nil || s.reporter == nil || s.newID == nil || s.now == nil {
+ return types.Snapshot{}, errors.New("snapshot service is not configured")
+ }
+ if request.SandboxReference == "" {
+ return types.Snapshot{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, errors.New("SANDBOX must not be empty"))
+ }
+ if err := s.reporter.Status("resolving sandbox"); err != nil {
+ return types.Snapshot{}, err
+ }
+ record, err := s.sandboxes.Resolve(ctx, request.SandboxReference)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ lockPath, err := s.sandboxPaths.Lock(record.ID)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ if err := s.reporter.Status("waiting for sandbox operation lock"); err != nil {
+ return types.Snapshot{}, err
+ }
+ lock := filelock.New(lockPath)
+ if err := lock.Lock(ctx); err != nil {
+ return types.Snapshot{}, errdefs.Context(err, "save snapshot", request.SandboxReference, "lock", "retry the snapshot", false)
+ }
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(lock.Unlock(context.WithoutCancel(ctx)), "save snapshot", request.SandboxReference, "unlock", "inspect the snapshot before retrying", result.ID != ""))
+ }()
+
+ record, err = s.sandboxes.Resolve(ctx, record.ID.String())
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ if record.State != types.SandboxStateRunning || record.Generation < 2 {
+ return types.Snapshot{}, errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, fmt.Errorf("sandbox %s in state %s cannot be snapshotted", record.ID, record.State))
+ }
+ backend, err := s.runtimes.Backend(record.VMM)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ snapshotter, ok := backend.(vmm.Snapshotter)
+ if !ok {
+ return types.Snapshot{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeHostIncompatible, fmt.Errorf("VMM backend %q does not support snapshots", record.VMM))
+ }
+ observation, err := backend.Observe(ctx, record.ID, record.Generation-1)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ if observation.State != vmm.ProcessRunning {
+ return types.Snapshot{}, errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, errors.New("sandbox has no ready VMM process to snapshot"))
+ }
+ id, err := s.newID()
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ pending := types.Snapshot{
+ ID: id, Name: request.Name, Description: request.Description,
+ SandboxID: record.ID, SourceGeneration: record.Generation,
+ ImageDigest: record.ImageDigest, VMM: record.VMM, Config: record.Config,
+ CreatedAt: s.now().UTC(),
+ }
+ if err := pending.Validate(); err != nil {
+ return types.Snapshot{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ if err := s.reporter.Status("reserving snapshot identity"); err != nil {
+ return types.Snapshot{}, err
+ }
+ if err := s.snapshots.Reserve(ctx, pending); err != nil {
+ return types.Snapshot{}, err
+ }
+ reserved, published := true, false
+ defer func() {
+ if returnErr == nil || !reserved || published {
+ return
+ }
+ cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
+ defer cancel()
+ returnErr = errors.Join(returnErr, snapshot.IgnoreAbsence(s.paths.RemoveStage(id)), s.snapshots.Forget(cleanupCtx, id))
+ }()
+ if err := s.paths.PrepareStage(id); err != nil {
+ return types.Snapshot{}, err
+ }
+ cowSource, err := s.sandboxPaths.COW(record.ID)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ cowDestination, err := s.paths.StageCOW(id)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ stage, err := s.paths.Stage(id)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ if err := s.reporter.Status("capturing VMM and writable disk"); err != nil {
+ return types.Snapshot{}, err
+ }
+ if err := snapshotter.Snapshot(ctx, vmm.SnapshotPlan{
+ Process: observation.Process, Destination: stage,
+ WritableFiles: []vmm.SnapshotFile{{Source: cowSource, Destination: cowDestination}},
+ }); err != nil {
+ return types.Snapshot{}, errdefs.Context(err, "save snapshot", request.SandboxReference, "capture", "inspect the running sandbox and retry", false)
+ }
+ if err := s.reporter.Status("publishing snapshot artifacts"); err != nil {
+ return types.Snapshot{}, err
+ }
+ if err := s.paths.Publish(id); err != nil {
+ return types.Snapshot{}, errdefs.Context(err, "save snapshot", request.SandboxReference, "publish", "inspect snapshot storage before retrying", false)
+ }
+ published = true
+ size, err := s.paths.Size(id)
+ if err != nil {
+ return types.Snapshot{}, errdefs.Context(err, "save snapshot", request.SandboxReference, "measure", "inspect snapshot storage before retrying", true)
+ }
+ if err := s.reporter.Status("committing snapshot metadata"); err != nil {
+ return types.Snapshot{}, errdefs.Context(err, "save snapshot", request.SandboxReference, "report", "inspect snapshot storage before retrying", true)
+ }
+ result, err = s.snapshots.Commit(ctx, id, size)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ if err := s.reporter.Committed(result); err != nil {
+ return result, errdefs.Context(err, "save snapshot", request.SandboxReference, "report", "snapshot was saved; inspect it before retrying", true)
+ }
+ return result, nil
+}
+
+// List returns every ready snapshot.
+func (s *SnapshotService) List(ctx context.Context) ([]types.Snapshot, error) {
+ if s == nil || s.snapshots == nil {
+ return nil, errors.New("snapshot service is not configured")
+ }
+ return s.snapshots.List(ctx)
+}
+
+// Inspect resolves one ready snapshot by name or complete ID.
+func (s *SnapshotService) Inspect(ctx context.Context, reference string) (types.Snapshot, error) {
+ if s == nil || s.snapshots == nil {
+ return types.Snapshot{}, errors.New("snapshot service is not configured")
+ }
+ return s.snapshots.Resolve(ctx, reference)
+}
+
+// Remove records deletion intent before removing artifacts, then releases the
+// metadata name. A failure after intent is retryable with the same reference.
+func (s *SnapshotService) Remove(ctx context.Context, reference string) (result types.Snapshot, returnErr error) {
+ if s == nil || s.snapshots == nil {
+ return types.Snapshot{}, errors.New("snapshot service is not configured")
+ }
+ record, err := s.snapshots.BeginDelete(ctx, reference)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ lockPath, err := s.paths.Lock(record.ID)
+ if err != nil {
+ return record, err
+ }
+ lock := filelock.New(lockPath)
+ if err := lock.Lock(ctx); err != nil {
+ return record, errdefs.Context(err, "remove snapshot", reference, "lock", "retry snapshot removal", true)
+ }
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(lock.Unlock(context.WithoutCancel(ctx)), "remove snapshot", reference, "unlock", "retry snapshot removal", true))
+ }()
+ if err := snapshot.IgnoreAbsence(s.paths.Remove(record.ID)); err != nil {
+ return record, errdefs.Context(err, "remove snapshot", reference, "remove artifacts", "retry snapshot removal", true)
+ }
+ if err := s.snapshots.FinalizeDelete(ctx, record.ID); err != nil {
+ return record, err
+ }
+ return record, nil
+}
+
+type discardSnapshotReporter struct{}
+
+func (discardSnapshotReporter) Status(string) error { return nil }
+func (discardSnapshotReporter) Committed(types.Snapshot) error { return nil }
diff --git a/core/snapshot_test.go b/core/snapshot_test.go
new file mode 100644
index 0000000..4e486e8
--- /dev/null
+++ b/core/snapshot_test.go
@@ -0,0 +1,144 @@
+package core
+
+import (
+ "errors"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/kumabox/kumabox/metadata"
+ "github.com/kumabox/kumabox/snapshot"
+ snapshotcatalog "github.com/kumabox/kumabox/snapshot/catalog"
+ "github.com/kumabox/kumabox/storage"
+ "github.com/kumabox/kumabox/types"
+ "github.com/kumabox/kumabox/vmm"
+)
+
+var fixedSnapshotID = types.SnapshotID("223e4567-e89b-42d3-a456-426614174000")
+
+type fakeSnapshotReporter struct{ steps *[]string }
+
+func (r fakeSnapshotReporter) Status(status string) error {
+ *r.steps = append(*r.steps, "snapshot-status:"+status)
+ return nil
+}
+
+func (r fakeSnapshotReporter) Committed(types.Snapshot) error {
+ *r.steps = append(*r.steps, "snapshot-report")
+ return nil
+}
+
+func newTestSnapshotService(t *testing.T) (*SnapshotService, *SandboxService, *[]string) {
+ t.Helper()
+ sandboxService, steps := newTestSandboxService(t, nil)
+ if _, err := sandboxService.Create(t.Context(), CreateSandboxRequest{
+ ImageReference: "demo",
+ Config: types.SandboxConfig{
+ Name: "box", CPUs: 2, Memory: types.DefaultSandboxMemory,
+ Storage: types.DefaultSandboxStorage,
+ },
+ }); err != nil {
+ t.Fatal(err)
+ }
+ catalog := sandboxService.dependencies.catalog.(*fakeCatalog)
+ catalog.record.State = types.SandboxStateRunning
+ catalog.record.Generation = 4
+ testRuntime(t, sandboxService).observation = vmm.Observation{
+ State: vmm.ProcessRunning,
+ Process: vmm.Process{
+ PID: 42, StartTicks: 10, BootID: "boot", SandboxID: fixedID,
+ Generation: 3, Binary: "cloud-hypervisor", APISocket: "/run/kumabox/api.sock",
+ },
+ }
+ roots := storage.Roots{
+ Data: filepath.Join(t.TempDir(), "data"), Run: filepath.Join(t.TempDir(), "run"), Log: filepath.Join(t.TempDir(), "log"),
+ }
+ paths, err := snapshot.NewPaths(roots)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := paths.Ensure(); err != nil {
+ t.Fatal(err)
+ }
+ memory, err := metadata.NewMemory(snapshotcatalog.Collections())
+ if err != nil {
+ t.Fatal(err)
+ }
+ service := &SnapshotService{
+ paths: paths, sandboxPaths: sandboxService.dependencies.paths,
+ sandboxes: catalog, snapshots: snapshotcatalog.New(memory), runtimes: sandboxService.dependencies.runtimes,
+ reporter: fakeSnapshotReporter{steps: steps}, newID: func() (types.SnapshotID, error) { return fixedSnapshotID, nil },
+ now: func() time.Time { return time.Date(2026, 9, 22, 12, 0, 0, 0, time.UTC) }, store: memory,
+ }
+ return service, sandboxService, steps
+}
+
+func TestSaveSnapshotPublishesCompleteCapture(t *testing.T) {
+ service, sandboxService, _ := newTestSnapshotService(t)
+ record, err := service.Save(t.Context(), SaveSnapshotRequest{
+ SandboxReference: "box", Name: "checkpoint/one", Description: "before upgrade",
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record.ID != fixedSnapshotID || record.SandboxID != fixedID || record.Name != "checkpoint/one" || record.Size != 5 {
+ t.Fatalf("snapshot = %+v", record)
+ }
+ directory, err := service.paths.Dir(record.ID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, name := range []string{"config.json", "cow.raw"} {
+ if _, err := os.Stat(filepath.Join(directory, name)); err != nil {
+ t.Fatalf("snapshot artifact %s: %v", name, err)
+ }
+ }
+ plan := testRuntime(t, sandboxService).snapshotPlan
+ if plan.Process.Generation != 3 || plan.Destination == "" || len(plan.WritableFiles) != 1 {
+ t.Fatalf("snapshot plan = %+v", plan)
+ }
+ listed, err := service.List(t.Context())
+ if err != nil || len(listed) != 1 || listed[0].ID != record.ID {
+ t.Fatalf("List = %+v, %v", listed, err)
+ }
+}
+
+func TestSaveSnapshotFailureCleansReservationAndStage(t *testing.T) {
+ service, sandboxService, _ := newTestSnapshotService(t)
+ failure := errors.New("capture failed")
+ testRuntime(t, sandboxService).snapshotErr = failure
+ request := SaveSnapshotRequest{SandboxReference: "box", Name: "retryable"}
+ if _, err := service.Save(t.Context(), request); !errors.Is(err, failure) {
+ t.Fatalf("Save error = %v", err)
+ }
+ if records, err := service.List(t.Context()); err != nil || len(records) != 0 {
+ t.Fatalf("List after failure = %+v, %v", records, err)
+ }
+ stage, err := service.paths.Stage(fixedSnapshotID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := os.Stat(stage); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("snapshot stage remains: %v", err)
+ }
+ testRuntime(t, sandboxService).snapshotErr = nil
+ if _, err := service.Save(t.Context(), request); err != nil {
+ t.Fatalf("retry after compensation: %v", err)
+ }
+}
+
+func TestRemoveSnapshotDeletesArtifactsAndName(t *testing.T) {
+ service, _, _ := newTestSnapshotService(t)
+ record, err := service.Save(t.Context(), SaveSnapshotRequest{SandboxReference: "box", Name: "remove-me"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ removed, err := service.Remove(t.Context(), "remove-me")
+ if err != nil || removed.ID != record.ID {
+ t.Fatalf("Remove = %+v, %v", removed, err)
+ }
+ if _, err := service.Inspect(t.Context(), "remove-me"); err == nil {
+ t.Fatal("removed snapshot still resolves")
+ }
+}
diff --git a/metadata/sqlite/store.go b/metadata/sqlite/store.go
index 3e7e4e5..12caffc 100644
--- a/metadata/sqlite/store.go
+++ b/metadata/sqlite/store.go
@@ -28,7 +28,7 @@ const (
// applicationID distinguishes KumaBox metadata from unrelated SQLite files.
applicationID = 0x4B554D41
// schemaVersion identifies the current application collection contract.
- schemaVersion = 3
+ schemaVersion = 4
// firstSchemaVersion is the oldest metadata version with an in-place migration.
firstSchemaVersion = 1
// initLockName serializes schema initialization across processes in this directory.
@@ -231,7 +231,7 @@ func initialize(ctx context.Context, path string, collections []metadata.Collect
switch version {
case schemaVersion:
return nil
- case 1, 2:
+ case 1, 2, 3:
return migrateCollections(ctx, db, collections, version)
default:
return errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, fmt.Errorf("metadata schema version %d is unsupported; this binary supports versions %d through %d", version, firstSchemaVersion, schemaVersion))
diff --git a/metadata/sqlite/store_test.go b/metadata/sqlite/store_test.go
index 5dd9783..4128ed2 100644
--- a/metadata/sqlite/store_test.go
+++ b/metadata/sqlite/store_test.go
@@ -219,6 +219,44 @@ func TestStoreMigratesVersionTwoAndPreservesSandboxRecords(t *testing.T) {
}
}
+func TestStoreMigratesVersionThreeAndAddsSnapshotCollections(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "meta.db")
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ statements := []string{
+ "CREATE TABLE collections (name TEXT NOT NULL PRIMARY KEY)",
+ "CREATE TABLE records (collection TEXT NOT NULL, id TEXT NOT NULL, data BLOB NOT NULL, PRIMARY KEY(collection, id), FOREIGN KEY(collection) REFERENCES collections(name))",
+ fmt.Sprintf("PRAGMA application_id = %d", applicationID),
+ "PRAGMA user_version = 3",
+ "INSERT INTO collections(name) VALUES ('sandboxes')",
+ }
+ for _, statement := range statements {
+ if _, err := db.Exec(statement); err != nil {
+ _ = db.Close()
+ t.Fatal(err)
+ }
+ }
+ if err := db.Close(); err != nil {
+ t.Fatal(err)
+ }
+ store, err := Open(t.Context(), path, []metadata.Collection{"sandboxes", "snapshots", "snapshot_names"}, DefaultOptions())
+ if err != nil {
+ t.Fatalf("Open migrated v3 database: %v", err)
+ }
+ t.Cleanup(func() {
+ if err := store.Close(); err != nil {
+ t.Error(err)
+ }
+ })
+ if err := store.Update(t.Context(), func(writer metadata.Writer) error {
+ return writer.Put(t.Context(), "snapshots", "snapshot-id", []byte("snapshot"))
+ }); err != nil {
+ t.Fatalf("write migrated snapshot collection: %v", err)
+ }
+}
+
func TestStoreMigrationFailureRollsBackVersionAndCollections(t *testing.T) {
path := filepath.Join(t.TempDir(), "meta.db")
writeVersionOneDatabase(t, path, "CREATE TABLE collections (name TEXT NOT NULL PRIMARY KEY CHECK(name <> 'sandboxes'))")
diff --git a/snapshot/catalog/store.go b/snapshot/catalog/store.go
new file mode 100644
index 0000000..575b17d
--- /dev/null
+++ b/snapshot/catalog/store.go
@@ -0,0 +1,343 @@
+// Package catalog persists snapshot identities, optional names, and publication
+// state. Artifact capture and removal remain in the snapshot and core packages.
+package catalog
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "slices"
+ "strings"
+ "time"
+
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/metadata"
+ "github.com/kumabox/kumabox/types"
+)
+
+const (
+ // CollectionSnapshots stores ready and pending records by immutable ID.
+ CollectionSnapshots metadata.Collection = "snapshots"
+ // CollectionNames maps optional human-readable names to snapshot IDs.
+ CollectionNames metadata.Collection = "snapshot_names"
+)
+
+// Collections declares the record sets required by this adapter.
+func Collections() []metadata.Collection {
+ return []metadata.Collection{CollectionSnapshots, CollectionNames}
+}
+
+// Store adapts shared metadata transactions to snapshot persistence.
+type Store struct{ store metadata.Store }
+
+// New constructs a snapshot catalog without taking ownership of the engine.
+func New(store metadata.Store) *Store { return &Store{store: store} }
+
+type recordData struct {
+ ID string `json:"id"`
+ Name string `json:"name,omitempty"`
+ Description string `json:"description,omitempty"`
+ SandboxID string `json:"sandbox_id"`
+ SandboxName string `json:"sandbox_name"`
+ SourceGeneration uint64 `json:"source_generation"`
+ ImageDigest string `json:"image_digest"`
+ VMM string `json:"vmm"`
+ CPUs uint32 `json:"cpus"`
+ Memory int64 `json:"memory"`
+ Storage int64 `json:"storage"`
+ NICs int `json:"nics,omitempty"`
+ NetworkName string `json:"network_name,omitempty"`
+ Size int64 `json:"size"`
+ CreatedAt time.Time `json:"created_at"`
+ Ready bool `json:"ready"`
+ Deleting bool `json:"deleting,omitempty"`
+}
+
+type nameData struct {
+ ID string `json:"id"`
+}
+
+// Reserve atomically holds an ID and optional name before large capture I/O.
+func (s *Store) Reserve(ctx context.Context, snapshot types.Snapshot) error {
+ if s == nil || s.store == nil {
+ return errors.New("snapshot catalog is not configured")
+ }
+ if err := snapshot.Validate(); err != nil {
+ return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ err := s.store.Update(ctx, func(writer metadata.Writer) error {
+ if _, exists, err := writer.Get(ctx, CollectionSnapshots, snapshot.ID.String()); err != nil {
+ return err
+ } else if exists {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeNameTaken, fmt.Errorf("snapshot ID %s already exists", snapshot.ID))
+ }
+ if snapshot.Name != "" {
+ if _, exists, err := writer.Get(ctx, CollectionNames, snapshot.Name); err != nil {
+ return err
+ } else if exists {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeNameTaken, fmt.Errorf("snapshot name %q already exists", snapshot.Name))
+ }
+ rawName, err := json.Marshal(nameData{ID: snapshot.ID.String()})
+ if err != nil {
+ return err
+ }
+ if err := writer.Put(ctx, CollectionNames, snapshot.Name, rawName); err != nil {
+ return err
+ }
+ }
+ raw, err := json.Marshal(encode(snapshot, false))
+ if err != nil {
+ return err
+ }
+ return writer.Put(ctx, CollectionSnapshots, snapshot.ID.String(), raw)
+ })
+ return errdefs.Context(err, "save snapshot", snapshot.Name, "reserve", "choose another snapshot name", false)
+}
+
+// Commit publishes size and readiness after artifacts are atomically visible.
+func (s *Store) Commit(ctx context.Context, id types.SnapshotID, size int64) (types.Snapshot, error) {
+ var result types.Snapshot
+ err := s.store.Update(ctx, func(writer metadata.Writer) error {
+ record, err := load(ctx, writer, id)
+ if err != nil {
+ return err
+ }
+ if record.Ready {
+ result, err = decodeSnapshot(record)
+ if err != nil {
+ return err
+ }
+ return nil
+ }
+ record.Size = size
+ result, err = decodeSnapshot(record)
+ if err != nil {
+ return err
+ }
+ record.Ready = true
+ raw, err := json.Marshal(record)
+ if err != nil {
+ return err
+ }
+ if err := writer.Put(ctx, CollectionSnapshots, id.String(), raw); err != nil {
+ return err
+ }
+ return nil
+ })
+ return result, errdefs.Context(err, "save snapshot", id.String(), "commit", "inspect snapshot storage before retrying", true)
+}
+
+// Forget releases a pending reservation during pre-publication compensation.
+func (s *Store) Forget(ctx context.Context, id types.SnapshotID) error {
+ err := s.store.Update(ctx, func(writer metadata.Writer) error {
+ record, err := load(ctx, writer, id)
+ if err != nil {
+ return err
+ }
+ if record.Ready {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, errors.New("ready snapshot cannot be forgotten"))
+ }
+ if record.Name != "" {
+ if err := writer.Delete(ctx, CollectionNames, record.Name); err != nil {
+ return err
+ }
+ }
+ return writer.Delete(ctx, CollectionSnapshots, id.String())
+ })
+ return err
+}
+
+// Resolve returns one ready snapshot by exact name or complete ID.
+func (s *Store) Resolve(ctx context.Context, reference string) (types.Snapshot, error) {
+ if s == nil || s.store == nil {
+ return types.Snapshot{}, errors.New("snapshot catalog is not configured")
+ }
+ var result types.Snapshot
+ err := s.store.View(ctx, func(reader metadata.Reader) error {
+ record, err := resolve(ctx, reader, reference)
+ if err != nil {
+ return err
+ }
+ if !record.Ready || record.Deleting {
+ return notFound(reference)
+ }
+ result, err = decodeSnapshot(record)
+ if err != nil {
+ return err
+ }
+ return nil
+ })
+ return result, errdefs.Context(err, "resolve snapshot", reference, "metadata", "check the snapshot name or ID", false)
+}
+
+// List returns ready snapshots ordered newest first.
+func (s *Store) List(ctx context.Context) ([]types.Snapshot, error) {
+ var result []types.Snapshot
+ err := s.store.View(ctx, func(reader metadata.Reader) error {
+ return reader.Scan(ctx, CollectionSnapshots, func(id string, raw []byte) error {
+ record, err := decode(raw)
+ if err != nil {
+ return err
+ }
+ if record.ID != id {
+ return corrupt(errors.New("snapshot record key differs from ID"))
+ }
+ if record.Ready && !record.Deleting {
+ snapshot, err := decodeSnapshot(record)
+ if err != nil {
+ return err
+ }
+ result = append(result, snapshot)
+ }
+ return nil
+ })
+ })
+ slices.SortFunc(result, func(left, right types.Snapshot) int {
+ if order := right.CreatedAt.Compare(left.CreatedAt); order != 0 {
+ return order
+ }
+ return strings.Compare(left.ID.String(), right.ID.String())
+ })
+ return result, errdefs.Context(err, "list snapshots", "", "metadata", "inspect snapshot metadata", false)
+}
+
+// BeginDelete records durable deletion intent and returns the artifact owner.
+func (s *Store) BeginDelete(ctx context.Context, reference string) (types.Snapshot, error) {
+ var result types.Snapshot
+ err := s.store.Update(ctx, func(writer metadata.Writer) error {
+ record, err := resolve(ctx, writer, reference)
+ if err != nil {
+ return err
+ }
+ if !record.Ready {
+ return notFound(reference)
+ }
+ result, err = decodeSnapshot(record)
+ if err != nil {
+ return err
+ }
+ if record.Deleting {
+ return nil
+ }
+ record.Deleting = true
+ raw, err := json.Marshal(record)
+ if err != nil {
+ return err
+ }
+ return writer.Put(ctx, CollectionSnapshots, record.ID, raw)
+ })
+ return result, errdefs.Context(err, "remove snapshot", reference, "mark deleting", "retry snapshot removal", false)
+}
+
+// FinalizeDelete releases metadata and the optional name after artifacts are absent.
+func (s *Store) FinalizeDelete(ctx context.Context, id types.SnapshotID) error {
+ err := s.store.Update(ctx, func(writer metadata.Writer) error {
+ record, err := load(ctx, writer, id)
+ if err != nil {
+ return err
+ }
+ if !record.Deleting {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, errors.New("snapshot is not deleting"))
+ }
+ if record.Name != "" {
+ if err := writer.Delete(ctx, CollectionNames, record.Name); err != nil {
+ return err
+ }
+ }
+ return writer.Delete(ctx, CollectionSnapshots, id.String())
+ })
+ return errdefs.Context(err, "remove snapshot", id.String(), "finalize", "retry snapshot removal", true)
+}
+
+func resolve(ctx context.Context, reader metadata.Reader, reference string) (recordData, error) {
+ if reference == "" {
+ return recordData{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, errors.New("SNAPSHOT must not be empty"))
+ }
+ if raw, exists, err := reader.Get(ctx, CollectionNames, reference); err != nil {
+ return recordData{}, err
+ } else if exists {
+ var name nameData
+ if err := json.Unmarshal(raw, &name); err != nil || name.ID == "" {
+ return recordData{}, corrupt(errors.New("invalid snapshot name binding"))
+ }
+ id, err := types.ParseSnapshotID(name.ID)
+ if err != nil {
+ return recordData{}, corrupt(err)
+ }
+ return load(ctx, reader, id)
+ }
+ id, err := types.ParseSnapshotID(reference)
+ if err != nil {
+ return recordData{}, notFound(reference)
+ }
+ return load(ctx, reader, id)
+}
+
+func load(ctx context.Context, reader metadata.Reader, id types.SnapshotID) (recordData, error) {
+ raw, exists, err := reader.Get(ctx, CollectionSnapshots, id.String())
+ if err != nil {
+ return recordData{}, err
+ }
+ if !exists {
+ return recordData{}, notFound(id.String())
+ }
+ return decode(raw)
+}
+
+func decode(raw []byte) (recordData, error) {
+ var record recordData
+ if err := json.Unmarshal(raw, &record); err != nil {
+ return recordData{}, corrupt(err)
+ }
+ if _, err := decodeSnapshot(record); err != nil {
+ return recordData{}, corrupt(err)
+ }
+ return record, nil
+}
+
+func encode(snapshot types.Snapshot, ready bool) recordData {
+ return recordData{
+ ID: snapshot.ID.String(), Name: snapshot.Name, Description: snapshot.Description,
+ SandboxID: snapshot.SandboxID.String(), SandboxName: snapshot.Config.Name,
+ SourceGeneration: snapshot.SourceGeneration,
+ ImageDigest: snapshot.ImageDigest.String(), VMM: string(snapshot.VMM),
+ CPUs: snapshot.Config.CPUs, Memory: snapshot.Config.Memory, Storage: snapshot.Config.Storage,
+ NICs: snapshot.Config.NICs, NetworkName: snapshot.Config.NetworkName,
+ Size: snapshot.Size, CreatedAt: snapshot.CreatedAt.UTC(), Ready: ready,
+ }
+}
+
+func decodeSnapshot(record recordData) (types.Snapshot, error) {
+ id, err := types.ParseSnapshotID(record.ID)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ sandboxID, err := types.ParseSandboxID(record.SandboxID)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ digest, err := types.ParseDigest(record.ImageDigest)
+ if err != nil {
+ return types.Snapshot{}, err
+ }
+ result := types.Snapshot{
+ ID: id, Name: record.Name, Description: record.Description,
+ SandboxID: sandboxID, SourceGeneration: record.SourceGeneration,
+ ImageDigest: digest, VMM: types.VMMType(record.VMM), Size: record.Size,
+ Config: types.SandboxConfig{
+ Name: record.SandboxName, CPUs: record.CPUs, Memory: record.Memory, Storage: record.Storage,
+ NICs: record.NICs, NetworkName: record.NetworkName,
+ },
+ CreatedAt: record.CreatedAt.UTC(),
+ }
+ return result, result.Validate()
+}
+
+func notFound(reference string) error {
+ return errdefs.New(errdefs.ClassNotFound, errdefs.CodeNotFound, fmt.Errorf("snapshot %q was not found", reference))
+}
+
+func corrupt(cause error) error {
+ return errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, cause)
+}
diff --git a/snapshot/catalog/store_test.go b/snapshot/catalog/store_test.go
new file mode 100644
index 0000000..e67941c
--- /dev/null
+++ b/snapshot/catalog/store_test.go
@@ -0,0 +1,49 @@
+package catalog
+
+import (
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/kumabox/kumabox/metadata"
+ "github.com/kumabox/kumabox/types"
+)
+
+func TestSnapshotCatalogPublishesAndDeletesNameAtomically(t *testing.T) {
+ memory, err := metadata.NewMemory(Collections())
+ if err != nil {
+ t.Fatal(err)
+ }
+ store := New(memory)
+ digest, err := types.ParseDigest("sha256:" + strings.Repeat("a", 64))
+ if err != nil {
+ t.Fatal(err)
+ }
+ record := types.Snapshot{
+ ID: types.SnapshotID("223e4567-e89b-42d3-a456-426614174000"), Name: "checkpoint",
+ SandboxID: types.SandboxID("123e4567-e89b-42d3-a456-426614174000"), SourceGeneration: 4,
+ ImageDigest: digest, VMM: types.VMMCloudHypervisor,
+ Config: types.SandboxConfig{Name: "box", CPUs: 2, Memory: types.DefaultSandboxMemory, Storage: types.DefaultSandboxStorage},
+ CreatedAt: time.Date(2026, 9, 22, 12, 0, 0, 0, time.UTC),
+ }
+ if err := store.Reserve(t.Context(), record); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := store.Resolve(t.Context(), "checkpoint"); err == nil {
+ t.Fatal("pending snapshot was visible")
+ }
+ ready, err := store.Commit(t.Context(), record.ID, 42)
+ if err != nil || ready.Size != 42 || ready.Config.Name != "box" {
+ t.Fatalf("Commit = %+v, %v", ready, err)
+ }
+ deleting, err := store.BeginDelete(t.Context(), "checkpoint")
+ if err != nil || deleting.ID != record.ID {
+ t.Fatalf("BeginDelete = %+v, %v", deleting, err)
+ }
+ if err := store.FinalizeDelete(t.Context(), record.ID); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := store.Resolve(t.Context(), "checkpoint"); err == nil {
+ t.Fatal("deleted name still resolves")
+ }
+}
diff --git a/snapshot/paths.go b/snapshot/paths.go
new file mode 100644
index 0000000..2fef025
--- /dev/null
+++ b/snapshot/paths.go
@@ -0,0 +1,172 @@
+// Package snapshot owns persistent snapshot artifacts and their storage
+// contracts. Application ordering lives in core and metadata encoding lives in
+// snapshot/catalog.
+package snapshot
+
+import (
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/storage"
+ "github.com/kumabox/kumabox/types"
+)
+
+const cowName = "cow.raw"
+
+// Paths derives final, staging, and lock paths for snapshot artifacts.
+type Paths struct {
+ roots storage.Roots
+}
+
+// NewPaths validates shared roots without touching the filesystem.
+func NewPaths(roots storage.Roots) (Paths, error) {
+ validated, err := roots.Validate()
+ if err != nil {
+ return Paths{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ return Paths{roots: validated}, nil
+}
+
+// Ensure creates shared artifact, staging, and lock parents.
+func (p Paths) Ensure() error {
+ for _, path := range []string{p.DataDir(), p.StagingDir(), p.LocksDir()} {
+ if err := storage.EnsureDir(path); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// DataDir contains one immutable directory per ready snapshot.
+func (p Paths) DataDir() string { return filepath.Join(p.roots.Data, "snapshots") }
+
+// StagingDir contains unpublished captures safe to remove after failure.
+func (p Paths) StagingDir() string { return filepath.Join(p.roots.Data, "staging", "snapshots") }
+
+// LocksDir contains stable snapshot operation locks.
+func (p Paths) LocksDir() string { return filepath.Join(p.roots.Run, "locks", "snapshots") }
+
+// Dir returns the published snapshot directory.
+func (p Paths) Dir(id types.SnapshotID) (string, error) { return p.idDir(p.DataDir(), id) }
+
+// Stage returns the private unpublished capture directory.
+func (p Paths) Stage(id types.SnapshotID) (string, error) { return p.idDir(p.StagingDir(), id) }
+
+// Lock returns the stable operation lock path for one snapshot.
+func (p Paths) Lock(id types.SnapshotID) (string, error) {
+ if _, err := types.ParseSnapshotID(id.String()); err != nil {
+ return "", err
+ }
+ return storage.Join(p.LocksDir(), id.String()+".lock")
+}
+
+// COW returns the captured writable overlay path inside a snapshot directory.
+func (p Paths) COW(id types.SnapshotID) (string, error) {
+ dir, err := p.Dir(id)
+ if err != nil {
+ return "", err
+ }
+ return storage.Join(dir, cowName)
+}
+
+// StageCOW returns the unpublished writable overlay path.
+func (p Paths) StageCOW(id types.SnapshotID) (string, error) {
+ dir, err := p.Stage(id)
+ if err != nil {
+ return "", err
+ }
+ return storage.Join(dir, cowName)
+}
+
+// PrepareStage creates an empty private capture directory.
+func (p Paths) PrepareStage(id types.SnapshotID) error {
+ dir, err := p.Stage(id)
+ if err != nil {
+ return err
+ }
+ if err := os.Mkdir(dir, 0o700); err != nil {
+ return fmt.Errorf("create snapshot staging directory: %w", err)
+ }
+ return nil
+}
+
+// Publish atomically makes a fully synchronized capture visible.
+func (p Paths) Publish(id types.SnapshotID) error {
+ stage, err := p.Stage(id)
+ if err != nil {
+ return err
+ }
+ final, err := p.Dir(id)
+ if err != nil {
+ return err
+ }
+ return storage.PublishDir(stage, final)
+}
+
+// RemoveStage removes an unpublished capture after a failed save.
+func (p Paths) RemoveStage(id types.SnapshotID) error {
+ stage, err := p.Stage(id)
+ if err != nil {
+ return err
+ }
+ if err := storage.CheckPath(stage); err != nil {
+ return err
+ }
+ return os.RemoveAll(stage)
+}
+
+// Remove deletes one published artifact directory.
+func (p Paths) Remove(id types.SnapshotID) error {
+ dir, err := p.Dir(id)
+ if err != nil {
+ return err
+ }
+ if err := storage.CheckPath(dir); err != nil {
+ return err
+ }
+ if err := os.RemoveAll(dir); err != nil {
+ return fmt.Errorf("remove snapshot artifacts: %w", err)
+ }
+ return nil
+}
+
+// Size returns the sum of regular-file logical sizes.
+func (p Paths) Size(id types.SnapshotID) (int64, error) {
+ dir, err := p.Dir(id)
+ if err != nil {
+ return 0, err
+ }
+ var size int64
+ err = filepath.WalkDir(dir, func(path string, entry os.DirEntry, walkErr error) error {
+ if walkErr != nil {
+ return walkErr
+ }
+ if entry.Type().IsRegular() {
+ info, err := entry.Info()
+ if err != nil {
+ return err
+ }
+ size += info.Size()
+ }
+ return nil
+ })
+ return size, err
+}
+
+func (p Paths) idDir(root string, id types.SnapshotID) (string, error) {
+ if _, err := types.ParseSnapshotID(id.String()); err != nil {
+ return "", err
+ }
+ return storage.Join(root, id.String())
+}
+
+// IgnoreAbsence converts cleanup of an already absent path into success.
+func IgnoreAbsence(err error) error {
+ if errors.Is(err, os.ErrNotExist) {
+ return nil
+ }
+ return err
+}
diff --git a/storage/copy_linux.go b/storage/copy_linux.go
new file mode 100644
index 0000000..5f14a86
--- /dev/null
+++ b/storage/copy_linux.go
@@ -0,0 +1,75 @@
+//go:build linux
+
+package storage
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "os"
+ "syscall"
+
+ "golang.org/x/sys/unix"
+)
+
+// CopySparse copies data extents while preserving holes and the source's
+// logical size. The destination must not already exist.
+func CopySparse(destination, source string) (returnErr error) {
+ input, err := os.Open(source) //nolint:gosec // callers supply validated managed paths
+ if err != nil {
+ return fmt.Errorf("open sparse source: %w", err)
+ }
+ defer func() { returnErr = errors.Join(returnErr, input.Close()) }()
+ info, err := input.Stat()
+ if err != nil || !info.Mode().IsRegular() {
+ return errors.Join(err, errors.New("sparse source must be a regular file"))
+ }
+ output, err := os.OpenFile(destination, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600) //nolint:gosec // managed staging path
+ if err != nil {
+ return fmt.Errorf("create sparse destination: %w", err)
+ }
+ defer func() { returnErr = errors.Join(returnErr, output.Close()) }()
+ if err := output.Truncate(info.Size()); err != nil {
+ return err
+ }
+ for offset := int64(0); offset < info.Size(); {
+ data, err := unix.Seek(int(input.Fd()), offset, unix.SEEK_DATA)
+ if errors.Is(err, syscall.ENXIO) {
+ break
+ }
+ if errors.Is(err, syscall.EINVAL) {
+ return copyDense(output, input)
+ }
+ if err != nil {
+ return fmt.Errorf("seek sparse data: %w", err)
+ }
+ hole, err := unix.Seek(int(input.Fd()), data, unix.SEEK_HOLE)
+ if err != nil {
+ return fmt.Errorf("seek sparse hole: %w", err)
+ }
+ if _, err := input.Seek(data, io.SeekStart); err != nil {
+ return err
+ }
+ if _, err := output.Seek(data, io.SeekStart); err != nil {
+ return err
+ }
+ if _, err := io.CopyN(output, input, hole-data); err != nil {
+ return fmt.Errorf("copy sparse extent: %w", err)
+ }
+ offset = hole
+ }
+ return output.Sync()
+}
+
+func copyDense(destination, source *os.File) error {
+ if _, err := source.Seek(0, io.SeekStart); err != nil {
+ return err
+ }
+ if _, err := destination.Seek(0, io.SeekStart); err != nil {
+ return err
+ }
+ if _, err := io.Copy(destination, source); err != nil {
+ return err
+ }
+ return destination.Sync()
+}
diff --git a/storage/copy_linux_test.go b/storage/copy_linux_test.go
new file mode 100644
index 0000000..9f6c638
--- /dev/null
+++ b/storage/copy_linux_test.go
@@ -0,0 +1,42 @@
+//go:build linux
+
+package storage
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+)
+
+func TestCopySparsePreservesLogicalData(t *testing.T) {
+ directory := t.TempDir()
+ source := filepath.Join(directory, "source.raw")
+ file, err := os.OpenFile(source, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := file.WriteAt([]byte("first"), 0); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := file.WriteAt([]byte("last"), 16<<20); err != nil {
+ t.Fatal(err)
+ }
+ if err := file.Close(); err != nil {
+ t.Fatal(err)
+ }
+ destination := filepath.Join(directory, "destination.raw")
+ if err := CopySparse(destination, source); err != nil {
+ t.Fatal(err)
+ }
+ want, err := os.ReadFile(source)
+ if err != nil {
+ t.Fatal(err)
+ }
+ got, err := os.ReadFile(destination)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if string(got) != string(want) {
+ t.Fatal("sparse copy changed file data")
+ }
+}
diff --git a/storage/copy_other.go b/storage/copy_other.go
new file mode 100644
index 0000000..095820d
--- /dev/null
+++ b/storage/copy_other.go
@@ -0,0 +1,29 @@
+//go:build !linux
+
+package storage
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "os"
+)
+
+// CopySparse provides a portable development-host fallback. Production Linux
+// builds use extent-aware copying to preserve holes.
+func CopySparse(destination, source string) (returnErr error) {
+ input, err := os.Open(source) //nolint:gosec // callers supply validated managed paths
+ if err != nil {
+ return fmt.Errorf("open sparse source: %w", err)
+ }
+ defer func() { returnErr = errors.Join(returnErr, input.Close()) }()
+ output, err := os.OpenFile(destination, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600) //nolint:gosec // managed staging path
+ if err != nil {
+ return fmt.Errorf("create sparse destination: %w", err)
+ }
+ defer func() { returnErr = errors.Join(returnErr, output.Close()) }()
+ if _, err := io.Copy(output, input); err != nil {
+ return err
+ }
+ return output.Sync()
+}
diff --git a/storage/directory.go b/storage/directory.go
new file mode 100644
index 0000000..fe1ab8e
--- /dev/null
+++ b/storage/directory.go
@@ -0,0 +1,69 @@
+package storage
+
+import (
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "slices"
+)
+
+// PublishDir synchronizes a staged directory tree, atomically renames it to an
+// absent final path, and synchronizes both parents. Source and destination must
+// share a filesystem.
+func PublishDir(staged, final string) error {
+ if err := CheckPath(staged); err != nil {
+ return err
+ }
+ if err := CheckPath(final); err != nil {
+ return err
+ }
+ if _, err := os.Lstat(final); err == nil {
+ return fmt.Errorf("publish directory destination %s already exists", final)
+ } else if !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ if err := SyncTree(staged); err != nil {
+ return err
+ }
+ if err := os.Rename(staged, final); err != nil {
+ return fmt.Errorf("publish directory %s: %w", final, err)
+ }
+ return errors.Join(syncPath(filepath.Dir(final)), syncPath(filepath.Dir(staged)))
+}
+
+// SyncTree flushes regular files and directories from leaves to root. Symlinks
+// and special files are rejected because managed artifact trees must be closed.
+func SyncTree(root string) error {
+ var directories []string
+ err := filepath.WalkDir(root, func(path string, entry os.DirEntry, walkErr error) error {
+ if walkErr != nil {
+ return walkErr
+ }
+ info, err := entry.Info()
+ if err != nil {
+ return err
+ }
+ switch {
+ case info.IsDir():
+ directories = append(directories, path)
+ case info.Mode().IsRegular():
+ if err := syncPath(path); err != nil {
+ return err
+ }
+ default:
+ return fmt.Errorf("snapshot artifact %s is not a regular file or directory", path)
+ }
+ return nil
+ })
+ if err != nil {
+ return err
+ }
+ for _, directory := range slices.Backward(directories) {
+ if err := syncPath(directory); err != nil {
+ return err
+ }
+ }
+ return nil
+}
diff --git a/types/snapshot.go b/types/snapshot.go
new file mode 100644
index 0000000..e8f1d88
--- /dev/null
+++ b/types/snapshot.go
@@ -0,0 +1,85 @@
+package types
+
+import (
+ "errors"
+ "fmt"
+ "regexp"
+ "time"
+)
+
+var validSnapshotName = regexp.MustCompile(`^[a-zA-Z0-9][a-zA-Z0-9._:/-]{0,62}$`)
+
+// SnapshotID is the immutable UUIDv4 identity of one captured sandbox state.
+type SnapshotID string
+
+// NewSnapshotID generates a snapshot identity from the same cryptographic UUID
+// source used for sandboxes.
+func NewSnapshotID() (SnapshotID, error) {
+ id, err := NewSandboxID()
+ return SnapshotID(id), err
+}
+
+// ParseSnapshotID validates the canonical UUIDv4 representation.
+func ParseSnapshotID(value string) (SnapshotID, error) {
+ if _, err := ParseSandboxID(value); err != nil {
+ return "", fmt.Errorf("invalid snapshot ID %q", value)
+ }
+ return SnapshotID(value), nil
+}
+
+// String returns the canonical snapshot identifier.
+func (id SnapshotID) String() string { return string(id) }
+
+// Snapshot is the durable description of one complete VMM and writable-disk
+// capture. Immutable image layers remain pinned by ImageDigest.
+type Snapshot struct {
+ // ID is the immutable metadata and artifact directory identity.
+ ID SnapshotID
+ // Name is an optional human-readable lookup key.
+ Name string
+ // Description is optional operator context.
+ Description string
+ // SandboxID identifies the source lineage accepted by restore.
+ SandboxID SandboxID
+ // SourceGeneration is the Running generation captured by this snapshot.
+ SourceGeneration uint64
+ // ImageDigest pins the immutable image layers required by the sandbox.
+ ImageDigest Digest
+ // VMM selects the adapter capable of restoring the native snapshot.
+ VMM VMMType
+ // Config is the source sandbox resource and network request.
+ Config SandboxConfig
+ // Size is the allocated snapshot artifact size in bytes.
+ Size int64
+ // CreatedAt records when capture was requested.
+ CreatedAt time.Time
+}
+
+// Validate rejects snapshot facts that cannot safely drive lookup or restore.
+func (s Snapshot) Validate() error {
+ if _, err := ParseSnapshotID(s.ID.String()); err != nil {
+ return err
+ }
+ if s.Name != "" && !validSnapshotName.MatchString(s.Name) {
+ return fmt.Errorf("snapshot name %q must match %s", s.Name, validSnapshotName)
+ }
+ if _, err := ParseSandboxID(s.SandboxID.String()); err != nil {
+ return err
+ }
+ if s.SourceGeneration == 0 {
+ return errors.New("snapshot source generation must be positive")
+ }
+ if _, err := ParseDigest(s.ImageDigest.String()); err != nil {
+ return err
+ }
+ if err := s.VMM.Validate(); err != nil {
+ return err
+ }
+ if err := s.Config.Validate(); err != nil {
+ return err
+ }
+ if s.Size < 0 || s.CreatedAt.IsZero() {
+ return errors.New("snapshot size must be non-negative and creation time must be set")
+ }
+ return nil
+}
diff --git a/types/snapshot_test.go b/types/snapshot_test.go
new file mode 100644
index 0000000..264a9b1
--- /dev/null
+++ b/types/snapshot_test.go
@@ -0,0 +1,31 @@
+package types
+
+import (
+ "strings"
+ "testing"
+ "time"
+)
+
+func TestSnapshotValidation(t *testing.T) {
+ id, err := NewSnapshotID()
+ if err != nil {
+ t.Fatal(err)
+ }
+ digest, err := ParseDigest("sha256:" + strings.Repeat("a", 64))
+ if err != nil {
+ t.Fatal(err)
+ }
+ snapshot := Snapshot{
+ ID: id, Name: "release/one:ready", SandboxID: SandboxID("123e4567-e89b-42d3-a456-426614174000"),
+ SourceGeneration: 4, ImageDigest: digest, VMM: VMMCloudHypervisor,
+ Config: SandboxConfig{Name: "box", CPUs: 1, Memory: DefaultSandboxMemory, Storage: DefaultSandboxStorage},
+ CreatedAt: time.Now().UTC(),
+ }
+ if err := snapshot.Validate(); err != nil {
+ t.Fatal(err)
+ }
+ snapshot.Name = "bad name"
+ if err := snapshot.Validate(); err == nil {
+ t.Fatal("Snapshot.Validate accepted an invalid name")
+ }
+}
diff --git a/vmm/backend.go b/vmm/backend.go
index 42f613a..932e15a 100644
--- a/vmm/backend.go
+++ b/vmm/backend.go
@@ -30,6 +30,30 @@ type Backend interface {
RemoveLogs(context.Context, types.SandboxID) error
}
+// SnapshotFile describes one writable disk copied inside the VMM pause window.
+type SnapshotFile struct {
+ // Source is the current sandbox-owned writable disk.
+ Source string
+ // Destination is an absent path inside the private capture directory.
+ Destination string
+}
+
+// SnapshotPlan contains all inputs required for one consistent live capture.
+type SnapshotPlan struct {
+ // Process is the exact VMM generation being captured.
+ Process Process
+ // Destination receives native VMM memory and device-state files.
+ Destination string
+ // WritableFiles are copied while the guest remains paused.
+ WritableFiles []SnapshotFile
+}
+
+// Snapshotter is the optional live-capture capability implemented by VMMs that
+// can pause, save native state, copy writable disks, and resume safely.
+type Snapshotter interface {
+ Snapshot(context.Context, SnapshotPlan) error
+}
+
// Registry is an immutable routing table from durable VMM identities to their
// process adapters. Construction validates the complete backend set so runtime
// lookup cannot depend on package initialization or registration order.
diff --git a/vmm/cloudhypervisor/snapshot.go b/vmm/cloudhypervisor/snapshot.go
new file mode 100644
index 0000000..deaa39b
--- /dev/null
+++ b/vmm/cloudhypervisor/snapshot.go
@@ -0,0 +1,88 @@
+package cloudhypervisor
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "net/http"
+ "time"
+
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/storage"
+ "github.com/kumabox/kumabox/vmm"
+)
+
+const snapshotTimeout = 10 * time.Minute
+
+var _ vmm.Snapshotter = (*Driver)(nil)
+
+// Snapshot pauses the exact owned process, captures native VMM state and every
+// writable disk, then resumes the guest even when capture fails.
+//
+// verify -> pause -> native state -> writable disks -> resume
+// \----------- any error -----------/
+func (d *Driver) Snapshot(ctx context.Context, plan vmm.SnapshotPlan) (returnErr error) {
+ if err := plan.Validate(); err != nil {
+ return errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ observation, err := d.Observe(ctx, plan.Process.SandboxID, plan.Process.Generation)
+ if err != nil {
+ return err
+ }
+ if observation.State != vmm.ProcessRunning || observation.Process.PID != plan.Process.PID || observation.Process.StartTicks != plan.Process.StartTicks {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, errors.New("sandbox VMM changed before snapshot capture"))
+ }
+ if err := d.snapshotAction(ctx, plan.Process.APISocket, "vm.pause", nil, probeTimeout); err != nil {
+ return fmt.Errorf("pause cloud-hypervisor: %w", err)
+ }
+ defer func() {
+ resumeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), d.startupTimeout)
+ defer cancel()
+ returnErr = errors.Join(returnErr, d.snapshotAction(resumeCtx, plan.Process.APISocket, "vm.resume", nil, d.startupTimeout))
+ }()
+ payload, err := json.Marshal(map[string]string{"destination_url": "file://" + plan.Destination})
+ if err != nil {
+ return err
+ }
+ if err := d.snapshotAction(ctx, plan.Process.APISocket, "vm.snapshot", payload, snapshotTimeout); err != nil {
+ return fmt.Errorf("capture cloud-hypervisor state: %w", err)
+ }
+ for _, file := range plan.WritableFiles {
+ if err := storage.CopySparse(file.Destination, file.Source); err != nil {
+ return fmt.Errorf("capture writable disk: %w", err)
+ }
+ }
+ return nil
+}
+
+func (d *Driver) snapshotAction(ctx context.Context, socket, endpoint string, payload []byte, timeout time.Duration) error {
+ client, closeClient, err := unixAPIClient(socket)
+ if err != nil {
+ return err
+ }
+ defer closeClient()
+ client.Timeout = timeout
+ request, err := http.NewRequestWithContext(ctx, http.MethodPut, "http://localhost/api/v1/"+endpoint, bytes.NewReader(payload))
+ if err != nil {
+ return err
+ }
+ if len(payload) > 0 {
+ request.Header.Set("Content-Type", "application/json")
+ }
+ response, err := client.Do(request)
+ if err != nil {
+ return err
+ }
+ defer response.Body.Close() //nolint:errcheck // status and bounded body are authoritative
+ body, readErr := io.ReadAll(io.LimitReader(response.Body, maxAPIResponse))
+ if readErr != nil {
+ return readErr
+ }
+ if response.StatusCode != http.StatusOK && response.StatusCode != http.StatusNoContent {
+ return fmt.Errorf("cloud hypervisor %s returned HTTP %d: %s", endpoint, response.StatusCode, bytes.TrimSpace(body))
+ }
+ return nil
+}
diff --git a/vmm/vmm.go b/vmm/vmm.go
index 03e5a3f..6b15683 100644
--- a/vmm/vmm.go
+++ b/vmm/vmm.go
@@ -214,3 +214,24 @@ type Observation struct {
// Process is populated for starting and running observations.
Process Process
}
+
+// Validate rejects capture plans that could write outside their prepared
+// directory or alias a source and destination.
+func (p SnapshotPlan) Validate() error {
+ if err := p.Process.Validate(); err != nil {
+ return err
+ }
+ if !filepath.IsAbs(p.Destination) || len(p.WritableFiles) == 0 {
+ return errors.New("snapshot plan requires an absolute destination and writable files")
+ }
+ for _, file := range p.WritableFiles {
+ if !filepath.IsAbs(file.Source) || !filepath.IsAbs(file.Destination) || file.Source == file.Destination {
+ return errors.New("snapshot writable file paths must be distinct and absolute")
+ }
+ relative, err := filepath.Rel(p.Destination, file.Destination)
+ if err != nil || relative == "." || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
+ return errors.New("snapshot writable destination escapes capture directory")
+ }
+ }
+ return nil
+}
From bacc617a018fec9c6a372a80130db8d9a50fef29 Mon Sep 17 00:00:00 2001
From: YoungLH <974840768@qq.com>
Date: Tue, 22 Sep 2026 15:36:31 +0800
Subject: [PATCH 44/45] feat: restore sandboxes from snapshots
---
cli/root.go | 1 +
cli/root_test.go | 1 +
cli/sandbox/progress.go | 21 +++
cli/sandbox/restore.go | 46 ++++++
core/sandbox_test.go | 12 ++
core/snapshot.go | 247 ++++++++++++++++++++++++----
core/snapshot_test.go | 121 ++++++++++++++
snapshot/paths.go | 12 ++
vmm/backend.go | 27 +++
vmm/cloudhypervisor/restore.go | 168 +++++++++++++++++++
vmm/cloudhypervisor/restore_test.go | 29 ++++
vmm/vmm.go | 11 ++
vmm/vmm_test.go | 16 ++
13 files changed, 681 insertions(+), 31 deletions(-)
create mode 100644 cli/sandbox/restore.go
create mode 100644 vmm/cloudhypervisor/restore.go
create mode 100644 vmm/cloudhypervisor/restore_test.go
diff --git a/cli/root.go b/cli/root.go
index 76dd20e..cd07edd 100644
--- a/cli/root.go
+++ b/cli/root.go
@@ -148,6 +148,7 @@ func newRootCommand() (*cobra.Command, error) {
root.AddCommand(sandboxcmd.NewListCommand(provideConfig))
root.AddCommand(sandboxcmd.NewLogsCommand(provideConfig))
root.AddCommand(sandboxcmd.NewRemoveCommand(provideConfig))
+ root.AddCommand(sandboxcmd.NewRestoreCommand(provideConfig))
root.AddCommand(sandboxcmd.NewRunCommand(provideConfig))
root.AddCommand(sandboxcmd.NewStartCommand(provideConfig))
root.AddCommand(sandboxcmd.NewStopCommand(provideConfig))
diff --git a/cli/root_test.go b/cli/root_test.go
index 7375f72..50a967a 100644
--- a/cli/root_test.go
+++ b/cli/root_test.go
@@ -142,6 +142,7 @@ func TestImageAndUsageExitCodes(t *testing.T) {
{"missing inspect sandbox", []string{"inspect"}, 2},
{"missing logs sandbox", []string{"logs"}, 2},
{"missing remove sandbox", []string{"rm"}, 2},
+ {"missing restore references", []string{"restore", "box"}, 2},
{"missing run image", []string{"run", "--name", "box"}, 2},
{"missing start sandbox", []string{"start"}, 2},
{"missing stop sandbox", []string{"stop"}, 2},
diff --git a/cli/sandbox/progress.go b/cli/sandbox/progress.go
index 7e96391..d3a0504 100644
--- a/cli/sandbox/progress.go
+++ b/cli/sandbox/progress.go
@@ -61,6 +61,27 @@ func startStopProgress(command *cobra.Command, reference string) (*sandboxProgre
return startProgress(command, "stop sandbox", fmt.Sprintf("Stop %q", reference), "preparing stop", "retry the stop or inspect the sandbox runtime")
}
+// snapshotStatusProgress adapts snapshot-service status callbacks while using
+// the sandbox renderer for restore output and failure semantics.
+type snapshotStatusProgress struct{ *sandboxProgress }
+
+// Committed records a saved snapshot if a shared snapshot workflow emits one.
+func (p *snapshotStatusProgress) Committed(types.Snapshot) error {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ p.committed = true
+ return p.renderer.Err()
+}
+
+// startRestoreProgress starts progress for one native snapshot restore.
+func startRestoreProgress(command *cobra.Command, reference string) (*snapshotStatusProgress, error) {
+ progress, err := startProgress(command, "restore sandbox", fmt.Sprintf("Restore %q", reference), "preparing restore", "inspect the sandbox state and VMM log")
+ if err != nil {
+ return nil, err
+ }
+ return &snapshotStatusProgress{sandboxProgress: progress}, nil
+}
+
func startProgress(command *cobra.Command, operation, label, status, recovery string) (*sandboxProgress, error) {
return newSandboxProgress(command.Context(), command.ErrOrStderr(), operation, label, status, recovery)
}
diff --git a/cli/sandbox/restore.go b/cli/sandbox/restore.go
new file mode 100644
index 0000000..03431fc
--- /dev/null
+++ b/cli/sandbox/restore.go
@@ -0,0 +1,46 @@
+package sandbox
+
+import (
+ "errors"
+
+ "github.com/spf13/cobra"
+
+ "github.com/kumabox/kumabox/core"
+ "github.com/kumabox/kumabox/errdefs"
+)
+
+// NewRestoreCommand builds the top-level native snapshot restore command.
+func NewRestoreCommand(configuration configProvider) *cobra.Command {
+ var asJSON bool
+ command := &cobra.Command{
+ Use: "restore SANDBOX SNAPSHOT",
+ Short: "restore a sandbox to a saved snapshot",
+ Args: cobra.ExactArgs(2),
+ RunE: func(command *cobra.Command, args []string) (returnErr error) {
+ progress, err := startRestoreProgress(command, args[0])
+ if err != nil {
+ return err
+ }
+ defer func() { returnErr = errors.Join(returnErr, progress.Finish(returnErr)) }()
+ service, err := core.OpenSnapshots(command.Context(), configuration(), progress)
+ if err != nil {
+ return err
+ }
+ committed := false
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(service.Close(), "restore sandbox", args[0], "close metadata", "inspect the sandbox before retrying", committed))
+ }()
+ record, err := service.Restore(command.Context(), args[0], args[1])
+ if err != nil {
+ return err
+ }
+ committed = true
+ if err := writeSandboxResult(progress.Output(command.OutOrStdout()), record, asJSON); err != nil {
+ return errdefs.Context(err, "restore sandbox", args[0], "output", "sandbox is running; inspect it before retrying", true)
+ }
+ return nil
+ },
+ }
+ command.Flags().BoolVar(&asJSON, "json", false, "print the restored sandbox as indented JSON")
+ return command
+}
diff --git a/core/sandbox_test.go b/core/sandbox_test.go
index 7d10062..c5b761d 100644
--- a/core/sandbox_test.go
+++ b/core/sandbox_test.go
@@ -294,6 +294,8 @@ type fakeRuntime struct {
logOptions vmm.LogOptions
snapshotPlan vmm.SnapshotPlan
snapshotErr error
+ restorePlan vmm.RestorePlan
+ restoreErr error
}
func (f *fakeRuntime) Snapshot(_ context.Context, plan vmm.SnapshotPlan) error {
@@ -313,6 +315,16 @@ func (f *fakeRuntime) Snapshot(_ context.Context, plan vmm.SnapshotPlan) error {
return nil
}
+func (f *fakeRuntime) Restore(_ context.Context, plan vmm.RestorePlan) (vmm.Process, error) {
+ *f.steps = append(*f.steps, "restore")
+ f.restorePlan = plan
+ process := vmm.Process{
+ PID: 43, StartTicks: 11, BootID: "boot", SandboxID: plan.SandboxID,
+ Generation: plan.Generation, Binary: "cloud-hypervisor", APISocket: "/run/kumabox/restore.sock",
+ }
+ return process, f.restoreErr
+}
+
func (f *fakeRuntime) Type() types.VMMType {
if f.typ == "" {
return types.VMMCloudHypervisor
diff --git a/core/snapshot.go b/core/snapshot.go
index 2b22e6e..6584e8c 100644
--- a/core/snapshot.go
+++ b/core/snapshot.go
@@ -4,19 +4,18 @@ import (
"context"
"errors"
"fmt"
+ "os"
+ "reflect"
"time"
"github.com/kumabox/kumabox/config"
"github.com/kumabox/kumabox/errdefs"
- "github.com/kumabox/kumabox/images"
- imagecatalog "github.com/kumabox/kumabox/images/catalog"
filelock "github.com/kumabox/kumabox/lock/flock"
"github.com/kumabox/kumabox/metadata"
- "github.com/kumabox/kumabox/metadata/sqlite"
sandboxfs "github.com/kumabox/kumabox/sandbox"
- sandboxcatalog "github.com/kumabox/kumabox/sandbox/catalog"
"github.com/kumabox/kumabox/snapshot"
snapshotcatalog "github.com/kumabox/kumabox/snapshot/catalog"
+ "github.com/kumabox/kumabox/storage"
"github.com/kumabox/kumabox/types"
"github.com/kumabox/kumabox/vmm"
)
@@ -59,52 +58,42 @@ type SnapshotService struct {
newID func() (types.SnapshotID, error)
now func() time.Time
store metadata.Store
+ lifecycle *SandboxService
}
// OpenSnapshots assembles the local snapshot service. The caller must close it.
func OpenSnapshots(ctx context.Context, configuration config.Config, reporter SnapshotReporter) (*SnapshotService, error) {
- if err := configuration.Validate(); err != nil {
- return nil, err
- }
- imagePaths, err := images.NewPaths(configuration.Paths)
- if err != nil {
- return nil, err
- }
- sandboxPaths, err := sandboxfs.NewPaths(configuration.Paths)
+ lifecycle, err := OpenSandbox(ctx, configuration, nil)
if err != nil {
return nil, err
}
snapshotPaths, err := snapshot.NewPaths(configuration.Paths)
if err != nil {
- return nil, err
- }
- if err := errors.Join(imagePaths.Ensure(), sandboxPaths.Ensure(), snapshotPaths.Ensure()); err != nil {
- return nil, err
- }
- store, err := sqlite.Open(ctx, imagePaths.MetadataDB(), metadataCollections(), sqlite.Options{
- BusyTimeout: configuration.Metadata.BusyTimeout,
- RetryLimit: configuration.Metadata.RetryLimit,
- })
- if err != nil {
- return nil, err
+ return nil, errors.Join(err, lifecycle.Close())
}
- runtimes, err := openVMMRegistry(configuration)
- if err != nil {
- return nil, errors.Join(err, store.Close())
+ if err := snapshotPaths.Ensure(); err != nil {
+ return nil, errors.Join(err, lifecycle.Close())
}
if reporter == nil {
reporter = discardSnapshotReporter{}
}
return &SnapshotService{
- paths: snapshotPaths, sandboxPaths: sandboxPaths,
- sandboxes: sandboxcatalog.New(store, imagecatalog.Reader{}), snapshots: snapshotcatalog.New(store),
- runtimes: runtimes, reporter: reporter, newID: types.NewSnapshotID, now: time.Now, store: store,
+ paths: snapshotPaths, sandboxPaths: lifecycle.dependencies.paths,
+ sandboxes: lifecycle.dependencies.catalog, snapshots: snapshotcatalog.New(lifecycle.dependencies.store),
+ runtimes: lifecycle.dependencies.runtimes, reporter: reporter,
+ newID: types.NewSnapshotID, now: time.Now, store: lifecycle.dependencies.store, lifecycle: lifecycle,
}, nil
}
// Close releases the shared metadata engine.
func (s *SnapshotService) Close() error {
- if s == nil || s.store == nil {
+ if s == nil {
+ return nil
+ }
+ if s.lifecycle != nil {
+ return s.lifecycle.Close()
+ }
+ if s.store == nil {
return nil
}
return s.store.Close()
@@ -222,7 +211,12 @@ func (s *SnapshotService) Save(ctx context.Context, request SaveSnapshotRequest)
return types.Snapshot{}, err
}
if err := s.paths.Publish(id); err != nil {
- return types.Snapshot{}, errdefs.Context(err, "save snapshot", request.SandboxReference, "publish", "inspect snapshot storage before retrying", false)
+ final, pathErr := s.paths.Dir(id)
+ _, statErr := os.Stat(final)
+ if pathErr == nil && statErr == nil {
+ published = true
+ }
+ return types.Snapshot{}, errdefs.Context(errors.Join(err, pathErr), "save snapshot", request.SandboxReference, "publish", "inspect snapshot storage before retrying", published)
}
published = true
size, err := s.paths.Size(id)
@@ -288,6 +282,197 @@ func (s *SnapshotService) Remove(ctx context.Context, reference string) (result
return record, nil
}
+// Restore replaces a stopped sandbox's writable disk and launches its native
+// VMM snapshot. A live or retained-error source is cleaned through the normal
+// stop lifecycle before replacement.
+//
+// snapshot lock -> validate + stage disk -> stop -> sandbox lock -> Starting
+// -> disk replace
+// -> VMM restore -> Running
+func (s *SnapshotService) Restore(ctx context.Context, sandboxReference, snapshotReference string) (result types.Sandbox, returnErr error) {
+ if s == nil || s.lifecycle == nil || s.snapshots == nil || s.runtimes == nil || s.reporter == nil || s.now == nil {
+ return types.Sandbox{}, errors.New("snapshot restore service is not configured")
+ }
+ if sandboxReference == "" || snapshotReference == "" {
+ return types.Sandbox{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, errors.New("SANDBOX and SNAPSHOT must not be empty"))
+ }
+ if err := s.reporter.Status("resolving snapshot and sandbox"); err != nil {
+ return types.Sandbox{}, err
+ }
+ capture, err := s.snapshots.Resolve(ctx, snapshotReference)
+ if err != nil {
+ return types.Sandbox{}, err
+ }
+ snapshotLockPath, err := s.paths.Lock(capture.ID)
+ if err != nil {
+ return types.Sandbox{}, err
+ }
+ snapshotLock := filelock.New(snapshotLockPath)
+ if err := snapshotLock.Lock(ctx); err != nil {
+ return types.Sandbox{}, errdefs.Context(err, "restore sandbox", sandboxReference, "lock snapshot", "retry the restore", false)
+ }
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(snapshotLock.Unlock(context.WithoutCancel(ctx)), "restore sandbox", sandboxReference, "unlock snapshot", "inspect the sandbox before retrying", result.Generation > 0))
+ }()
+ record, err := s.sandboxes.Resolve(ctx, sandboxReference)
+ if err != nil {
+ return types.Sandbox{}, err
+ }
+ if err := validateRestoreLineage(record, capture); err != nil {
+ return types.Sandbox{}, err
+ }
+ backend, err := s.runtimes.Backend(record.VMM)
+ if err != nil {
+ return record, err
+ }
+ restorer, ok := backend.(vmm.Restorer)
+ if !ok {
+ return record, errdefs.New(errdefs.ClassInvalid, errdefs.CodeHostIncompatible, fmt.Errorf("VMM backend %q does not support restore", record.VMM))
+ }
+ if err := s.reporter.Status("validating snapshot artifacts"); err != nil {
+ return record, err
+ }
+ snapshotDir, err := s.paths.Dir(capture.ID)
+ if err != nil {
+ return record, err
+ }
+ snapshotCOW, err := s.paths.COW(capture.ID)
+ if err != nil {
+ return record, err
+ }
+ if info, err := os.Lstat(snapshotCOW); err != nil {
+ return record, errdefs.New(errdefs.ClassUnavailable, errdefs.CodeArtifactUnavailable, err)
+ } else if !info.Mode().IsRegular() || info.Size() == 0 {
+ return record, errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, errors.New("snapshot COW is not a nonempty regular file"))
+ }
+ if validator, ok := backend.(vmm.RestoreValidator); ok {
+ if err := validator.ValidateRestore(ctx, snapshotDir); err != nil {
+ return record, errdefs.New(errdefs.ClassCorrupt, errdefs.CodeArtifactCorrupt, err)
+ }
+ }
+ if err := s.reporter.Status("checking host runtime"); err != nil {
+ return record, err
+ }
+ if err := backend.Preflight(); err != nil {
+ return record, err
+ }
+ stagedCOW, err := s.paths.RestoreCOW(capture.ID, record.ID)
+ if err != nil {
+ return record, err
+ }
+ if err := ignoreNotExist(os.Remove(stagedCOW)); err != nil {
+ return record, errdefs.Context(err, "restore sandbox", sandboxReference, "clean staging disk", "inspect snapshot staging storage before retrying", false)
+ }
+ defer func() { returnErr = errors.Join(returnErr, ignoreNotExist(os.Remove(stagedCOW))) }()
+ if err := s.reporter.Status("staging snapshot writable disk"); err != nil {
+ return record, err
+ }
+ if err := storage.CopySparse(stagedCOW, snapshotCOW); err != nil {
+ return record, errdefs.Context(err, "restore sandbox", sandboxReference, "stage disk", "verify the snapshot and retry", false)
+ }
+ stoppedForRestore := false
+ defer func() {
+ if stoppedForRestore && returnErr != nil {
+ returnErr = errdefs.Context(returnErr, "restore sandbox", sandboxReference, "after stop", "inspect the stopped or retained-error sandbox before retrying", true)
+ }
+ }()
+ switch record.State {
+ case types.SandboxStateRunning, types.SandboxStateStarting, types.SandboxStateStopping, types.SandboxStateError:
+ if err := s.reporter.Status("stopping current sandbox runtime"); err != nil {
+ return types.Sandbox{}, err
+ }
+ if _, err := s.lifecycle.Stop(ctx, record.ID.String()); err != nil {
+ return types.Sandbox{}, errdefs.Context(err, "restore sandbox", sandboxReference, "stop", "inspect the sandbox before retrying", true)
+ }
+ stoppedForRestore = true
+ case types.SandboxStateStopped:
+ default:
+ return types.Sandbox{}, errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, fmt.Errorf("sandbox %s in state %s cannot be restored", record.ID, record.State))
+ }
+ sandboxLockPath, err := s.sandboxPaths.Lock(record.ID)
+ if err != nil {
+ return types.Sandbox{}, err
+ }
+ if err := s.reporter.Status("waiting for sandbox operation lock"); err != nil {
+ return types.Sandbox{}, err
+ }
+ sandboxLock := filelock.New(sandboxLockPath)
+ if err := sandboxLock.Lock(ctx); err != nil {
+ return types.Sandbox{}, errdefs.Context(err, "restore sandbox", sandboxReference, "lock sandbox", "retry the restore", false)
+ }
+ committed := false
+ defer func() {
+ returnErr = errors.Join(returnErr, errdefs.Context(sandboxLock.Unlock(context.WithoutCancel(ctx)), "restore sandbox", sandboxReference, "unlock sandbox", "inspect the sandbox before retrying", committed))
+ }()
+ record, err = s.sandboxes.Resolve(ctx, record.ID.String())
+ if err != nil {
+ return types.Sandbox{}, err
+ }
+ if record.State != types.SandboxStateStopped && record.State != types.SandboxStateError {
+ return record, errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, fmt.Errorf("sandbox %s changed to state %s before restore", record.ID, record.State))
+ }
+ if err := validateRestoreLineage(record, capture); err != nil {
+ return record, err
+ }
+ if err := s.reporter.Status("committing starting state"); err != nil {
+ return record, err
+ }
+ starting, err := s.sandboxes.BeginStart(ctx, record.ID, record.Generation, s.now().UTC())
+ if err != nil {
+ return record, err
+ }
+ committed = true
+ result = starting
+ if err := s.lifecycle.recoverNetwork(ctx, starting); err != nil {
+ return starting, s.lifecycle.failStart(ctx, backend, starting, "recover network", err, vmm.Process{})
+ }
+ liveCOW, err := s.sandboxPaths.COW(record.ID)
+ if err != nil {
+ return starting, s.lifecycle.failStart(ctx, backend, starting, "resolve disk", err, vmm.Process{})
+ }
+ if err := s.reporter.Status("replacing writable disk"); err != nil {
+ return starting, s.lifecycle.failStart(ctx, backend, starting, "report", err, vmm.Process{})
+ }
+ if err := storage.Publish(stagedCOW, liveCOW); err != nil {
+ return starting, s.lifecycle.failStart(ctx, backend, starting, "replace disk", err, vmm.Process{})
+ }
+ if err := s.reporter.Status("restoring VMM state"); err != nil {
+ return starting, s.lifecycle.failStart(ctx, backend, starting, "report", err, vmm.Process{})
+ }
+ process, err := restorer.Restore(ctx, vmm.RestorePlan{
+ SandboxID: starting.ID, Generation: starting.Generation, CPUs: starting.Config.CPUs,
+ SnapshotDir: snapshotDir, Network: starting.Network,
+ })
+ if err != nil {
+ return starting, s.lifecycle.failStart(ctx, backend, starting, "restore VMM", err, process)
+ }
+ if err := s.reporter.Status("committing running state"); err != nil {
+ return starting, s.lifecycle.failStart(ctx, backend, starting, "report", err, process)
+ }
+ running, err := s.sandboxes.MarkRunning(ctx, starting.ID, starting.Generation, s.now().UTC())
+ if err != nil {
+ return starting, s.lifecycle.failStart(ctx, backend, starting, "commit running", err, process)
+ }
+ return running, nil
+}
+
+func validateRestoreLineage(sandbox types.Sandbox, capture types.Snapshot) error {
+ if capture.SandboxID != sandbox.ID {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, errors.New("snapshot belongs to another sandbox"))
+ }
+ if capture.VMM != sandbox.VMM || capture.ImageDigest != sandbox.ImageDigest || !reflect.DeepEqual(capture.Config, sandbox.Config) {
+ return errdefs.New(errdefs.ClassConflict, errdefs.CodeStateConflict, errors.New("snapshot runtime configuration differs from the target sandbox"))
+ }
+ return nil
+}
+
+func ignoreNotExist(err error) error {
+ if errors.Is(err, os.ErrNotExist) {
+ return nil
+ }
+ return err
+}
+
type discardSnapshotReporter struct{}
func (discardSnapshotReporter) Status(string) error { return nil }
diff --git a/core/snapshot_test.go b/core/snapshot_test.go
index 4e486e8..5a0b01c 100644
--- a/core/snapshot_test.go
+++ b/core/snapshot_test.go
@@ -51,6 +51,20 @@ func newTestSnapshotService(t *testing.T) (*SnapshotService, *SandboxService, *[
Generation: 3, Binary: "cloud-hypervisor", APISocket: "/run/kumabox/api.sock",
},
}
+ sandboxDir, err := sandboxService.dependencies.paths.Dir(fixedID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := storage.EnsureDir(sandboxDir); err != nil {
+ t.Fatal(err)
+ }
+ cow, err := sandboxService.dependencies.paths.COW(fixedID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(cow, []byte("live-cow"), 0o600); err != nil {
+ t.Fatal(err)
+ }
roots := storage.Roots{
Data: filepath.Join(t.TempDir(), "data"), Run: filepath.Join(t.TempDir(), "run"), Log: filepath.Join(t.TempDir(), "log"),
}
@@ -70,10 +84,117 @@ func newTestSnapshotService(t *testing.T) (*SnapshotService, *SandboxService, *[
sandboxes: catalog, snapshots: snapshotcatalog.New(memory), runtimes: sandboxService.dependencies.runtimes,
reporter: fakeSnapshotReporter{steps: steps}, newID: func() (types.SnapshotID, error) { return fixedSnapshotID, nil },
now: func() time.Time { return time.Date(2026, 9, 22, 12, 0, 0, 0, time.UTC) }, store: memory,
+ lifecycle: sandboxService,
}
return service, sandboxService, steps
}
+func TestRestoreStopsRunningSandboxAndResumesSnapshot(t *testing.T) {
+ service, sandboxService, steps := newTestSnapshotService(t)
+ capture, err := service.Save(t.Context(), SaveSnapshotRequest{SandboxReference: "box", Name: "checkpoint"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ *steps = nil
+ record, err := service.Restore(t.Context(), "box", capture.ID.String())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record.State != types.SandboxStateRunning || record.Generation != 8 {
+ t.Fatalf("restored sandbox = %+v", record)
+ }
+ plan := testRuntime(t, sandboxService).restorePlan
+ if plan.SandboxID != fixedID || plan.Generation != 7 || plan.SnapshotDir == "" {
+ t.Fatalf("restore plan = %+v", plan)
+ }
+ wantSequence := []string{"stopping", "stop", "stopped", "starting", "restore", "running"}
+ position := 0
+ for _, step := range *steps {
+ if position < len(wantSequence) && step == wantSequence[position] {
+ position++
+ }
+ }
+ if position != len(wantSequence) {
+ t.Fatalf("restore steps = %v, missing sequence %v", *steps, wantSequence)
+ }
+}
+
+func TestRestoreFailureRetainsErrorSandbox(t *testing.T) {
+ service, sandboxService, _ := newTestSnapshotService(t)
+ capture, err := service.Save(t.Context(), SaveSnapshotRequest{SandboxReference: "box"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ failure := errors.New("restore failed")
+ testRuntime(t, sandboxService).restoreErr = failure
+ if _, err := service.Restore(t.Context(), "box", capture.ID.String()); !errors.Is(err, failure) {
+ t.Fatalf("Restore error = %v", err)
+ }
+ record := sandboxService.dependencies.catalog.(*fakeCatalog).record
+ if record.State != types.SandboxStateError || record.Failure == nil || record.Failure.Phase != "restore VMM" {
+ t.Fatalf("retained sandbox = %+v", record)
+ }
+}
+
+func TestRestoreRejectsMissingCOWBeforeStoppingSandbox(t *testing.T) {
+ service, sandboxService, steps := newTestSnapshotService(t)
+ capture, err := service.Save(t.Context(), SaveSnapshotRequest{SandboxReference: "box"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ snapshotCOW, err := service.paths.COW(capture.ID)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Remove(snapshotCOW); err != nil {
+ t.Fatal(err)
+ }
+ *steps = nil
+ if _, err := service.Restore(t.Context(), "box", capture.ID.String()); err == nil {
+ t.Fatal("Restore accepted a snapshot without its writable disk")
+ }
+ record := sandboxService.dependencies.catalog.(*fakeCatalog).record
+ if record.State != types.SandboxStateRunning || record.Generation != 4 {
+ t.Fatalf("sandbox changed before snapshot validation: %+v", record)
+ }
+ for _, step := range *steps {
+ if step == "stopping" || step == "stop" {
+ t.Fatalf("restore stopped the sandbox before validation: %v", *steps)
+ }
+ }
+}
+
+func TestRestoreRecoversRetainedErrorSandbox(t *testing.T) {
+ service, sandboxService, steps := newTestSnapshotService(t)
+ capture, err := service.Save(t.Context(), SaveSnapshotRequest{SandboxReference: "box"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ catalog := sandboxService.dependencies.catalog.(*fakeCatalog)
+ catalog.record.State = types.SandboxStateError
+ catalog.record.Generation = 5
+ catalog.record.Failure = &types.SandboxFailure{Phase: "previous start", Message: "failed"}
+ testRuntime(t, sandboxService).observation = vmm.Observation{State: vmm.ProcessAbsent}
+ *steps = nil
+ record, err := service.Restore(t.Context(), "box", capture.ID.String())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if record.State != types.SandboxStateRunning || record.Generation != 7 || record.Failure != nil {
+ t.Fatalf("restored sandbox = %+v", record)
+ }
+ wantSequence := []string{"cleanup", "starting", "restore", "running"}
+ position := 0
+ for _, step := range *steps {
+ if position < len(wantSequence) && step == wantSequence[position] {
+ position++
+ }
+ }
+ if position != len(wantSequence) {
+ t.Fatalf("restore steps = %v, missing sequence %v", *steps, wantSequence)
+ }
+}
+
func TestSaveSnapshotPublishesCompleteCapture(t *testing.T) {
service, sandboxService, _ := newTestSnapshotService(t)
record, err := service.Save(t.Context(), SaveSnapshotRequest{
diff --git a/snapshot/paths.go b/snapshot/paths.go
index 2fef025..fbe2d10 100644
--- a/snapshot/paths.go
+++ b/snapshot/paths.go
@@ -81,6 +81,18 @@ func (p Paths) StageCOW(id types.SnapshotID) (string, error) {
return storage.Join(dir, cowName)
}
+// RestoreCOW returns a private scratch file used to prepare one sandbox's
+// writable disk while its current VMM can continue running.
+func (p Paths) RestoreCOW(snapshotID types.SnapshotID, sandboxID types.SandboxID) (string, error) {
+ if _, err := types.ParseSnapshotID(snapshotID.String()); err != nil {
+ return "", err
+ }
+ if _, err := types.ParseSandboxID(sandboxID.String()); err != nil {
+ return "", err
+ }
+ return storage.Join(p.StagingDir(), snapshotID.String()+"-restore-"+sandboxID.String()+".raw")
+}
+
// PrepareStage creates an empty private capture directory.
func (p Paths) PrepareStage(id types.SnapshotID) error {
dir, err := p.Stage(id)
diff --git a/vmm/backend.go b/vmm/backend.go
index 932e15a..987cf8c 100644
--- a/vmm/backend.go
+++ b/vmm/backend.go
@@ -54,6 +54,33 @@ type Snapshotter interface {
Snapshot(context.Context, SnapshotPlan) error
}
+// RestorePlan contains the immutable ownership and native capture inputs for a
+// VMM restore launch.
+type RestorePlan struct {
+ // SandboxID owns the restored process and runtime files.
+ SandboxID types.SandboxID
+ // Generation is the durable Starting generation for this launch.
+ Generation uint64
+ // CPUs sizes the process cgroup consistently with a normal launch.
+ CPUs uint32
+ // SnapshotDir contains native VMM state with already restored writable disks.
+ SnapshotDir string
+ // Network supplies the recovered namespace and stable TAP identities.
+ Network types.NetworkSetup
+}
+
+// Restorer is the optional native-state restore capability implemented by VMMs
+// whose snapshot format can resume a stopped process.
+type Restorer interface {
+ Restore(context.Context, RestorePlan) (Process, error)
+}
+
+// RestoreValidator optionally validates native snapshot files before a running
+// sandbox is stopped for restore.
+type RestoreValidator interface {
+ ValidateRestore(context.Context, string) error
+}
+
// Registry is an immutable routing table from durable VMM identities to their
// process adapters. Construction validates the complete backend set so runtime
// lookup cannot depend on package initialization or registration order.
diff --git a/vmm/cloudhypervisor/restore.go b/vmm/cloudhypervisor/restore.go
new file mode 100644
index 0000000..b3a7160
--- /dev/null
+++ b/vmm/cloudhypervisor/restore.go
@@ -0,0 +1,168 @@
+package cloudhypervisor
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "net"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "strings"
+ "time"
+
+ "github.com/kumabox/kumabox/errdefs"
+ "github.com/kumabox/kumabox/vmm"
+)
+
+var (
+ _ vmm.Restorer = (*Driver)(nil)
+ _ vmm.RestoreValidator = (*Driver)(nil)
+)
+
+// ValidateRestore checks the native files Cloud Hypervisor requires before a
+// caller stops the current sandbox process.
+func (*Driver) ValidateRestore(_ context.Context, directory string) error {
+ for _, name := range []string{"config.json", "state.json"} {
+ path := filepath.Join(directory, name)
+ info, err := os.Lstat(path)
+ if err != nil {
+ return fmt.Errorf("inspect %s: %w", name, err)
+ }
+ if !info.Mode().IsRegular() || info.Size() == 0 {
+ return fmt.Errorf("snapshot %s is not a nonempty regular file", name)
+ }
+ }
+ raw, err := os.ReadFile(filepath.Join(directory, "config.json")) //nolint:gosec // managed snapshot path
+ if err != nil {
+ return err
+ }
+ var config map[string]json.RawMessage
+ if err := json.Unmarshal(raw, &config); err != nil || len(config) == 0 {
+ return errors.Join(err, errors.New("snapshot config.json is empty or invalid"))
+ }
+ entries, err := os.ReadDir(directory)
+ if err != nil {
+ return err
+ }
+ for _, entry := range entries {
+ if strings.HasPrefix(entry.Name(), "memory-range") {
+ info, err := entry.Info()
+ if err != nil {
+ return err
+ }
+ if info.Mode().IsRegular() && info.Size() > 0 {
+ return nil
+ }
+ }
+ }
+ return errors.New("snapshot has no nonempty memory-range file")
+}
+
+// Restore launches an API-only process in the target sandbox's cgroup and
+// namespace, loads native state, resumes the VM, and proves readiness.
+//
+// runtime dirs -> API-only process -> vm.restore -> vm.resume -> Running
+func (d *Driver) Restore(ctx context.Context, plan vmm.RestorePlan) (result vmm.Process, returnErr error) {
+ if err := plan.Validate(); err != nil {
+ return vmm.Process{}, errdefs.New(errdefs.ClassInvalid, errdefs.CodeInvalidArgument, err)
+ }
+ if err := d.Preflight(); err != nil {
+ return vmm.Process{}, err
+ }
+ if err := d.paths.Prepare(plan.SandboxID); err != nil {
+ return vmm.Process{}, err
+ }
+ var command *exec.Cmd
+ defer func() {
+ if returnErr == nil {
+ return
+ }
+ cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), d.abortGrace+time.Second)
+ defer cancel()
+ switch {
+ case result.PID > 0:
+ returnErr = errors.Join(returnErr, d.Abort(cleanupCtx, result))
+ case command != nil && command.Process != nil:
+ returnErr = errors.Join(returnErr, command.Process.Kill(), command.Wait(), d.scopes.Remove(cleanupCtx, plan.SandboxID), d.paths.Clear(plan.SandboxID))
+ default:
+ returnErr = errors.Join(returnErr, d.scopes.Remove(cleanupCtx, plan.SandboxID), d.paths.Clear(plan.SandboxID))
+ }
+ }()
+ apiSocket, _ := d.paths.APISocket(plan.SandboxID)
+ args := []string{"--api-socket", apiSocket}
+ if err := d.paths.WriteCmdline(plan.SandboxID, diagnosticCommand(d.binary, args)); err != nil {
+ return vmm.Process{}, err
+ }
+ scope, err := d.scopes.Prepare(ctx, plan.SandboxID, plan.CPUs)
+ if err != nil {
+ return vmm.Process{}, err
+ }
+ defer func() { returnErr = errors.Join(returnErr, scope.Close()) }()
+ logPath, _ := d.paths.LogFile(plan.SandboxID)
+ logFile, err := os.OpenFile(logPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o600) //nolint:gosec // managed path
+ if err != nil {
+ return vmm.Process{}, fmt.Errorf("open VMM log: %w", err)
+ }
+ defer func() { returnErr = errors.Join(returnErr, logFile.Close()) }()
+ command = exec.Command(d.binary, args...) //nolint:gosec // configured executable, no shell
+ command.Stdout, command.Stderr = logFile, logFile
+ configureProcess(command, scope)
+ if err := startProcess(command, plan.Network.Namespace); err != nil {
+ return vmm.Process{}, fmt.Errorf("exec cloud-hypervisor restore process: %w", err)
+ }
+ result, err = captureProcess(command.Process.Pid, plan.SandboxID, plan.Generation, filepath.Base(d.binary), apiSocket)
+ if err != nil {
+ return result, fmt.Errorf("capture restore process identity: %w", err)
+ }
+ if err := d.paths.WriteProcess(result); err != nil {
+ return result, fmt.Errorf("persist restore process identity: %w", err)
+ }
+ go func() { _ = command.Wait() }()
+ if err := d.waitAPISocket(ctx, result); err != nil {
+ return result, err
+ }
+ payload, err := json.Marshal(map[string]string{"source_url": "file://" + plan.SnapshotDir})
+ if err != nil {
+ return result, err
+ }
+ if err := d.snapshotAction(ctx, apiSocket, "vm.restore", payload, snapshotTimeout); err != nil {
+ return result, fmt.Errorf("restore cloud-hypervisor state: %w", err)
+ }
+ if err := d.snapshotAction(ctx, apiSocket, "vm.resume", nil, d.startupTimeout); err != nil {
+ return result, fmt.Errorf("resume restored cloud-hypervisor: %w", err)
+ }
+ if err := d.WaitReady(ctx, result); err != nil {
+ return result, err
+ }
+ return result, nil
+}
+
+func (d *Driver) waitAPISocket(ctx context.Context, process vmm.Process) error {
+ deadline := time.NewTimer(d.startupTimeout)
+ defer deadline.Stop()
+ ticker := time.NewTicker(probeInterval)
+ defer ticker.Stop()
+ for {
+ connection, err := net.DialTimeout("unix", process.APISocket, probeInterval)
+ if err == nil {
+ _ = connection.Close()
+ return nil
+ }
+ located, exists, locateErr := d.Locate(ctx, process.SandboxID, process.Generation)
+ if locateErr != nil {
+ return locateErr
+ }
+ if !exists || located.PID != process.PID || located.StartTicks != process.StartTicks {
+ return errors.New("cloud-hypervisor restore process exited before its API socket became ready")
+ }
+ select {
+ case <-ctx.Done():
+ return ctx.Err()
+ case <-deadline.C:
+ return errors.New("timed out waiting for cloud-hypervisor restore API socket")
+ case <-ticker.C:
+ }
+ }
+}
diff --git a/vmm/cloudhypervisor/restore_test.go b/vmm/cloudhypervisor/restore_test.go
new file mode 100644
index 0000000..4773db8
--- /dev/null
+++ b/vmm/cloudhypervisor/restore_test.go
@@ -0,0 +1,29 @@
+package cloudhypervisor
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+)
+
+func TestValidateRestoreRequiresCompleteNativeSnapshot(t *testing.T) {
+ directory := t.TempDir()
+ for name, content := range map[string]string{
+ "config.json": `{"cpus":{"boot_vcpus":2}}`,
+ "state.json": `{"version":1}`,
+ "memory-range-0": "memory",
+ } {
+ if err := os.WriteFile(filepath.Join(directory, name), []byte(content), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if err := (*Driver)(nil).ValidateRestore(t.Context(), directory); err != nil {
+ t.Fatalf("ValidateRestore() = %v", err)
+ }
+ if err := os.Remove(filepath.Join(directory, "memory-range-0")); err != nil {
+ t.Fatal(err)
+ }
+ if err := (*Driver)(nil).ValidateRestore(t.Context(), directory); err == nil {
+ t.Fatal("ValidateRestore accepted native state without memory")
+ }
+}
diff --git a/vmm/vmm.go b/vmm/vmm.go
index 6b15683..239d2b2 100644
--- a/vmm/vmm.go
+++ b/vmm/vmm.go
@@ -235,3 +235,14 @@ func (p SnapshotPlan) Validate() error {
}
return nil
}
+
+// Validate rejects incomplete restore ownership before a process is launched.
+func (p RestorePlan) Validate() error {
+ if _, err := types.ParseSandboxID(p.SandboxID.String()); err != nil {
+ return err
+ }
+ if p.Generation == 0 || p.CPUs == 0 || !filepath.IsAbs(p.SnapshotDir) {
+ return errors.New("restore plan requires generation, CPUs, and an absolute snapshot directory")
+ }
+ return p.Network.Validate()
+}
diff --git a/vmm/vmm_test.go b/vmm/vmm_test.go
index f998fbf..539fb08 100644
--- a/vmm/vmm_test.go
+++ b/vmm/vmm_test.go
@@ -88,3 +88,19 @@ func TestProcessValidationRequiresCompleteIdentity(t *testing.T) {
})
}
}
+
+func TestRestorePlanRequiresOwnedAbsoluteSnapshot(t *testing.T) {
+ plan := RestorePlan{
+ SandboxID: "123e4567-e89b-42d3-a456-426614174000",
+ Generation: 7,
+ CPUs: 2,
+ SnapshotDir: "/var/lib/kumabox/snapshots/example",
+ }
+ if err := plan.Validate(); err != nil {
+ t.Fatal(err)
+ }
+ plan.SnapshotDir = "relative/snapshot"
+ if err := plan.Validate(); err == nil {
+ t.Fatal("RestorePlan accepted a relative snapshot directory")
+ }
+}
From 3259ee144624745fa4e812b1d608351810953880 Mon Sep 17 00:00:00 2001
From: YoungLH <974840768@qq.com>
Date: Wed, 23 Sep 2026 11:10:35 +0800
Subject: [PATCH 45/45] docs: update README.md
---
README.md | 286 +++++++++++++++++++++++--------
README.zh-CN.md | 246 ++++++++++++++++++++++++++
assets/readme/architecture.svg | 115 +++++++++++++
assets/readme/comparison.svg | 85 +++++++++
assets/readme/hero.svg | 90 ++++++++++
assets/readme/lifecycle.svg | 86 ++++++++++
assets/readme/product.svg | 78 +++++++++
assets/readme/src/build.py | 6 +
assets/readme/src/gen_arch.py | 90 ++++++++++
assets/readme/src/gen_common.py | 122 +++++++++++++
assets/readme/src/gen_compare.py | 45 +++++
assets/readme/src/gen_hero.py | 42 +++++
assets/readme/src/gen_life.py | 50 ++++++
assets/readme/src/gen_product.py | 48 ++++++
14 files changed, 1318 insertions(+), 71 deletions(-)
create mode 100644 README.zh-CN.md
create mode 100644 assets/readme/architecture.svg
create mode 100644 assets/readme/comparison.svg
create mode 100644 assets/readme/hero.svg
create mode 100644 assets/readme/lifecycle.svg
create mode 100644 assets/readme/product.svg
create mode 100644 assets/readme/src/build.py
create mode 100644 assets/readme/src/gen_arch.py
create mode 100644 assets/readme/src/gen_common.py
create mode 100644 assets/readme/src/gen_compare.py
create mode 100644 assets/readme/src/gen_hero.py
create mode 100644 assets/readme/src/gen_life.py
create mode 100644 assets/readme/src/gen_product.py
diff --git a/README.md b/README.md
index 08b7420..28f5082 100644
--- a/README.md
+++ b/README.md
@@ -1,112 +1,256 @@
+ English · 简体中文
+
+
-# KumaBox
+
+ Quick start ·
+ Architecture ·
+ Comparison ·
+ Roadmap
+
-KumaBox is a daemonless microVM sandbox runtime for AI agents. The current implementation imports OCI and Docker images, creates persistent sandboxes, boots them with Cloud Hypervisor, and provides console and guest command access.
+AI agents write code, install packages, open network connections and touch
+files nobody reviewed. Running that on a shared kernel is a bet. KumaBox gives
+every task its own **KVM microVM** with its own kernel, disk and network
+namespace, and gets you from an OCI image to a running sandbox in one command.
-KumaBox uses Cocoon commit `27ae1e0b2a65c9082c7a1b33c5245bfe43a4854d` as its behavior and capability baseline. Compatibility is evaluated feature by feature. KumaBox keeps its own package structure, error model, metadata schema, and `kumabox.*` guest boot protocol.
+> [!WARNING]
+> KumaBox is under active development. The CLI, metadata schema and snapshot
+> format are not yet covered by a stability guarantee. Use disposable Linux/KVM
+> hosts until the first stable release.
-## Current status
+## What KumaBox is
-Available commands:
+
-```text
-kumabox doctor
-kumabox image pull|import|list|inspect|verify|remove
-kumabox create|start|stop|ps|inspect|logs|console|exec|rm
-kumabox version
-```
+KumaBox is a **microVM sandbox runtime for AI agents and untrusted
+workloads**. It handles images, VM lifecycle, networking, snapshots, devices and
+guest execution end to end, so you work with sandboxes rather than raw VMMs.
+Anything that can run a command can drive it today: a coding agent, an agent framework's tool call,
+an RL or eval harness fanning out thousands of attempts, a CI job, or you at a
+terminal.
-Image import and sandbox lifecycle are implemented locally. Real Cloud Hypervisor, cgroup, vsock, ext4, and EROFS behavior requires Linux acceptance testing. Networking, `run`, snapshots, clone, and Firecracker remain planned work.
+Each sandbox is a real machine:
-## Build and test
+- **Hardware isolation.** A dedicated guest kernel behind KVM, with one Cloud Hypervisor process per VM.
+- **OCI in, microVM out.** Digest-pinned OCI images become shared, read-only EROFS layers plus a private copy-on-write disk per VM.
+- **Real networking.** A network namespace per VM, multiqueue TAP and tc redirect through CNI, multiple NICs, live NIC resize.
+- **Guest execution without SSH.** `exec` over vsock with streamed stdout and stderr, stdin, env, workdir, TTY and real exit codes.
+- **Snapshots as first-class artifacts.** Stopped or running snapshots that you can verify, export, import, restore, hibernate, or clone with a fresh identity.
+- **Real devices when you need them.** Hotplug data disks, virtio-fs shares and VFIO PCI passthrough, for example a GPU.
+- **Built to be scripted.** `--json` output, versioned dry-run launch plans (`kumabox debug launch`) and per-VM usage intervals (`kumabox usage`).
-KumaBox requires Go 1.24 or newer.
+## Architecture
-```bash
-make build
-make verify
-make lint
-```
+
-The binaries are written to `bin/`. `make verify` checks formatting, documentation links, Linux and Darwin vet, shell syntax, race-enabled tests, and the build.
+**Lightweight control plane.** Every `kumabox` call opens durable state, takes
+resource locks, performs the operation and records the result. Each running VM
+is backed by its own Cloud Hypervisor process, so one sandbox can never take
+down another.
-Install locally with:
+**Crash-consistent, by design.** Multi-step changes are recorded in one
+operation journal covering VM lifecycle, network, devices, snapshots, clone,
+restore and hibernate. If a command is killed halfway, the next command
+reconciles the records against the real VMM and host-network state. Named
+fault-injection points across metadata, network, snapshot, clone, delete and GC
+boundaries are exercised in tests.
-```bash
-sudo make install
-kumabox doctor
-```
+**Switchable metadata.** JSON by default. SQLite when you need heavier
+concurrency, with `metadata status`, `metadata verify` and verified backups.
-The host checker reports Linux, KVM, cgroup v2, Cloud Hypervisor, `mkfs.erofs`, `mkfs.ext4`, and other runtime prerequisites.
+| Path | Purpose |
+| --- | --- |
+| `/var/lib/kumabox` | Images, VM records, snapshots, network leases, content |
+| `/var/lib/kumabox/run` | PID files, API sockets, native restore staging |
+| `/var/log/kumabox` | VM and runtime logs |
-## Image workflow
+## Warm once, fork many
-A bootable image must contain a kernel and initramfs and declare the OCI label `io.kumabox.boot.profile=overlay-v1`.
+
-```bash
-kumabox image pull ghcr.io/example/image:tag --platform linux/amd64
-kumabox image import demo ./docker-save.tar --format docker --platform linux/amd64
-kumabox image import demo ./oci-layout --format oci --platform linux/amd64
-kumabox image list
-kumabox image inspect demo
-kumabox image verify demo
-```
+Agents retry, branch and explore. Pay the setup cost once: boot, install
+dependencies, warm caches. Capture a **running snapshot** of memory and disks,
+then `clone` it for every attempt. Each clone gets a new network identity and a
+reseeded guest identity and entropy pool, so clones do not accidentally share
+secrets. Memory restore is selectable with `--restore-mode copy|ondemand|mmap`.
-Local import auto-detects OCI layouts, OCI archives, and `docker save` archives. `docker export` filesystem archives are unsupported. Source layers are verified, converted to EROFS, and published by digest. Boot candidates follow layer overwrite, whiteout, and opaque-directory semantics.
+## Quick start
-## Sandbox workflow
+You need Linux amd64 or arm64 with `/dev/kvm`, and root.
```bash
-kumabox create demo --name box --cpus 2 --memory 1GiB --storage 10GiB
-kumabox start box
-kumabox exec box -- uname -a
-kumabox logs --tail 50 box
-kumabox logs -f box
-kumabox console box
-kumabox stop box
-kumabox rm box
+# 1. Install and verify the release
+curl -fsSLO https://github.com/kgpp34/KumaBox/releases/latest/download/kumabox-install.sh
+curl -fsSLO https://github.com/kgpp34/KumaBox/releases/latest/download/kumabox-install.sh.sha256
+sha256sum --check kumabox-install.sh.sha256
+sudo sh kumabox-install.sh
+
+# 2. Prepare the host once: Cloud Hypervisor, firmware, CNI plugins, EROFS tools
+sudo kumabox-check --upgrade
+sudo kumabox doctor
+
+# 3. Build the published guest image
+sudo kumabox image build ghcr.io/kgpp34/kumabox/ubuntu:24.04 --name ubuntu
+
+# 4. Run a sandbox and talk to it
+sudo kumabox run ubuntu --name my-vm --cpus 2 --memory 1G --storage 4G
+sudo kumabox exec my-vm -- uname -a
+sudo kumabox exec -it my-vm -- sh
+
+# 5. Warm once, fork many
+sudo kumabox snapshot create my-vm --name base --type running
+sudo kumabox clone base --name fresh
+sudo kumabox exec fresh -- hostname
+
+# 6. Clean up
+sudo kumabox delete fresh my-vm --force
+sudo kumabox snapshot rm base
+sudo kumabox image rm ubuntu
+sudo kumabox gc
```
-`create` prepares a sparse ext4 COW disk but does not start the VMM. `start` uses direct kernel boot, records a PID-reuse-safe process identity, and commits `running` only after the Cloud Hypervisor API reports readiness. `stop` requests shutdown, then uses an identity-checked TERM-to-KILL fallback. `exec` uses the guest agent over private hybrid-vsock transport. `logs` reads persistent backend output with tail and follow support, including after stop.
+Host and guest artifacts are a matched release pair. Pin a versioned guest tag
+such as `24.04-v0.1.0`, or an OCI digest, when reproducibility matters.
+`sudo kumabox-check` alone performs a read-only host audit.
-`ps` prints a table with headers. `inspect` and every `--json` mode emit indented JSON. Progress goes to stderr; command results go to stdout.
+### Drive it from an agent
-## Configuration
+`exec --json` prints `ok`, `exitCode`, and base64-encoded `stdout` and `stderr`.
+The process exit code mirrors the guest command's exit code.
-Configuration precedence is:
+```python
+import base64, json, subprocess
-```text
-explicit flag > environment > explicit --config file > default
+def run_in_sandbox(vm: str, script: str, timeout: str = "120s") -> dict:
+ proc = subprocess.run(
+ ["sudo", "kumabox", "exec", "--json", "--timeout", timeout,
+ vm, "--", "sh", "-c", script],
+ capture_output=True, text=True,
+ )
+ result = json.loads(proc.stdout)
+ for key in ("stdout", "stderr"):
+ result[key] = base64.b64decode(result.get(key) or "").decode(errors="replace")
+ return result
+
+print(run_in_sandbox("fresh", "echo hello from $(hostname)"))
```
-KumaBox never searches for an implicit configuration file. Root paths can be set with `--root-dir`, `--run-dir`, and `--log-dir`.
+Fan out parallel attempts from one warm snapshot:
-## Architecture
+```bash
+for i in $(seq 1 8); do
+ sudo kumabox clone base --name try-$i &
+done
+wait
+sudo kumabox ps
+```
+
+The published Ubuntu guest is intentionally minimal. To bake in your own
+toolchain (Python, Node, browsers), extend
+[`oci-images/ubuntu/24.04/Dockerfile`](oci-images/ubuntu/24.04/Dockerfile),
+which already installs the matching `kumabox-agent`, kernel and initramfs.
+
+## Core commands
+
+| Area | Commands |
+| --- | --- |
+| VM lifecycle | `run`, `create`, `start`, `stop`, `pause`, `resume`, `delete`, `ps`, `inspect` |
+| Guest access | `exec`, `console`, `logs`, `agent status`, `agent ping`, `agent reseed` |
+| Images | `image build`, `image add`, `image pull-oci`, `image pull`, `image import`, `image inspect`, `image ls`, `image rm` |
+| Snapshots | `snapshot create`, `snapshot verify`, `snapshot export`, `snapshot import`, `restore`, `clone`, `hibernate` |
+| Networking | `network inspect`, `network setup`, `network teardown`, `network resize` |
+| Devices | `disk attach/detach/list`, `fs attach/detach/list`, `device attach/detach/list/state` |
+| Operations | `doctor`, `metadata`, `usage`, `gc`, `debug launch` |
+
+`kumabox --help` is the authoritative reference.
+
+## How KumaBox compares
+
+
+
+[E2B](https://github.com/e2b-dev/infra) and
+[CubeSandbox](https://github.com/TencentCloud/CubeSandbox) are excellent
+projects that share KumaBox's goal of giving every agent task its own kernel.
+KumaBox takes a different path in a few places:
+
+- **VM-native, not container-shaped.** Sandboxes are real VMs with the full device model of Cloud Hypervisor: hotplug disks, virtio-fs shares, live NIC resize and VFIO PCI passthrough for GPUs and other accelerators.
+- **Snapshots you can hold.** A running snapshot is a verifiable, portable package. Export it, move it to another host, import it and clone from it.
+- **Layered images, shared on disk.** OCI layers become read-only EROFS images shared by every VM on the host; each VM only pays for its own copy-on-write writes.
+- **Minimal to install.** One Go binary plus Cloud Hypervisor and CNI plugins. Metadata lives in JSON or embedded SQLite, with no external database, cache or object store to operate.
+- **Correctness you can audit.** A single operation journal and named fault-injection points cover lifecycle, network, snapshot, clone and GC paths.
+- **MIT licensed**, on amd64 and arm64.
+
+Related projects: [Kata Containers](https://katacontainers.io/),
+[gVisor](https://gvisor.dev/),
+[Firecracker](https://firecracker-microvm.github.io/),
+[Cloud Hypervisor](https://www.cloudhypervisor.org/) and
+[Cocoon](https://github.com/cocoonstack/cocoon).
+
+## Vision
+
+Every agent action should get a disposable computer that is as cheap to fork as
+a git branch and as safe as a separate machine. KumaBox builds that from the
+bottom up: first a correct, crash-consistent runtime on every host, then a
+long-running service and a multi-node control plane on top of the same
+journal and metadata, so a sandbox behaves the same on a laptop-sized server
+and across a fleet.
+
+## Roadmap
+
+> Proposed direction. Open an issue to weigh in.
+
+- [x] OCI to EROFS images, CNI networking, guest exec over vsock
+- [x] Running snapshots, clone, restore, hibernate, export and import
+- [x] Hotplug disks, virtio-fs, VFIO PCI; JSON and SQLite metadata
+- [ ] Daemon mode with an HTTP API
+- [ ] Multi-node control plane and scheduling
+- [ ] Go, Python and TypeScript SDKs
+- [ ] E2B-compatible API, so existing E2B code can point at KumaBox
+- [ ] MCP server, so agents can create and drive sandboxes as tools
+- [ ] Warm pools and published clone-latency benchmarks
+- [ ] Per-sandbox egress policy
+
+## Build and test
-The repository uses root-level modules instead of `internal` or a generic `pkg` tree:
+```bash
+git clone https://github.com/kgpp34/KumaBox.git && cd KumaBox
+make build
+make test
+go vet ./...
+./bin/kumabox version --json
+```
+
+The E2E suite needs a Linux/KVM host and exercises OCI image creation, cold
+boot, guest exec and TTY, CNI allocation and cleanup, stopped and native
+snapshots, clone and restore, disk hotplug and metadata backup:
+
+```bash
+GO_BIN="$(go env GOROOT)/bin/go"
+sudo test/e2e/e2e.sh --go-bin "$GO_BIN" --metadata-backend sqlite
+sudo test/e2e/e2e.sh --go-bin "$GO_BIN" --metadata-backend json
+```
-| Package | Responsibility |
-|---|---|
-| `cli` | Cobra command tree, argument validation, and presentation |
-| `core` | Application services and concrete adapter assembly |
-| `types` | Shared image and sandbox values; no capability interfaces |
-| `images` | Source resolution, conversion, verification, and removal |
-| `sandbox`, `disk` | Sandbox paths, locks, and writable COW disks |
-| `vmm` | VMM contracts, launch plans, process identity, and backend registry |
-| `vmm/cloudhypervisor` | Cloud Hypervisor process and API adapter |
-| `agent` | Guest exec protocol and host/guest transports |
-| `metadata` | Transaction contracts and SQLite implementation |
-| `cgroup`, `storage`, `lock/flock` | Host resource adapters |
+README graphics are generated from code. Edit the scripts in
+`assets/readme/src/` and run `python3 assets/readme/src/build.py`.
-Local design notes and Linux runbooks live under the ignored `docs/` directory. They are working material and are never committed.
+## Security model
-## Contributing
+- KumaBox adds a VM boundary, but the VMM, KVM, guest kernel, firmware, images and agent remain in the trusted computing base.
+- Host setup changes privileged networking and system configuration. Review `scripts/check.sh` before running `--fix` or `--upgrade`.
+- VFIO hands a physical device to a guest and requires correct IOMMU grouping; misuse can affect host stability and isolation.
+- Snapshot compatibility depends on host architecture, Cloud Hypervisor version, VM configuration and capture mode.
-See [CONTRIBUTING.md](CONTRIBUTING.md).
+Report reproducible bugs and security concerns through the issue tracker. Do
+not attach secrets, private images or production snapshots.
## License
-[MIT](LICENSE)
+KumaBox is available under the [MIT License](LICENSE).
diff --git a/README.zh-CN.md b/README.zh-CN.md
new file mode 100644
index 0000000..7e8a43b
--- /dev/null
+++ b/README.zh-CN.md
@@ -0,0 +1,246 @@
+
+
+
+
+ English · 简体中文
+
+
+
+
+
+
+ 快速开始 ·
+ 架构 ·
+ 对比 ·
+ 路线图
+
+
+AI Agent 会写代码、装依赖、发起网络连接，还会读写没人审过的文件。把这些放在共享内核上跑，
+本质上是在赌运气。KumaBox 为每个任务分配一台独立的 **KVM microVM**：独立的内核、独立的磁盘、
+独立的网络命名空间。从 OCI 镜像到一个可用的沙箱，只需要一条命令。
+
+> [!WARNING]
+> KumaBox 仍在快速迭代中。CLI、元数据格式和快照格式暂不承诺向后兼容。
+> 在首个稳定版发布之前，请在可随时重建的 Linux/KVM 主机上使用。
+
+## KumaBox 是什么
+
+
+
+KumaBox 是一个**面向 AI Agent 和不可信工作负载的 microVM 沙箱运行时**。镜像、虚拟机生命周期、
+网络、快照、设备和 guest 内命令执行，都由它一站式管理。你面对的是"沙箱"，而不是裸的 VMM。
+
+任何能执行命令的程序现在都能驱动它，例如：
+编程 Agent、Agent 框架里的工具调用、一次拉起成千上万次尝试的 RL / 评测框架、CI 任务，
+或者坐在终端前的你。
+
+每个沙箱都是一台真正的机器：
+
+- **硬件级隔离。** 每个沙箱运行在 KVM 之上的独立 guest 内核里，每台 VM 对应一个独立的 Cloud Hypervisor 进程。
+- **OCI 进，microVM 出。** 按 digest 固定的 OCI 镜像被转换成共享、只读的 EROFS 层，再为每台 VM 叠加一块私有的写时复制磁盘。
+- **真实的网络。** 每台 VM 拥有独立的网络命名空间，通过 CNI 使用多队列 TAP 和 tc redirect。支持多网卡，也支持在线调整网卡。
+- **无需 SSH 即可执行命令。** `exec` 走 vsock 通道，支持流式 stdout / stderr、stdin、环境变量、工作目录、TTY，并返回真实的退出码。
+- **快照是一等公民。** 支持停机快照和运行态快照，可以校验、导出、导入、原地恢复、休眠，或者以全新身份克隆。
+- **按需使用真实设备。** 支持热插拔数据盘、virtio-fs 共享目录，以及 VFIO PCI 直通（例如 GPU）。
+- **为脚本化而生。** 提供 `--json` 输出、带版本号的启动计划预演（`kumabox debug launch`），以及按 VM 统计的用量区间（`kumabox usage`）。
+
+## 架构
+
+
+
+**轻量的控制面。** 每次调用 `kumabox`，都会打开持久化状态、获取资源锁、执行操作并记录结果。
+每台运行中的 VM 由各自独立的 Cloud Hypervisor 进程承载，一个沙箱出问题不会拖垮其他沙箱。
+
+**崩溃一致性是设计目标。** 所有多步骤变更都记录在同一套操作日志（operation journal）里，
+覆盖 VM 生命周期、网络、设备、快照、克隆、恢复和休眠。如果某条命令执行到一半被中断，
+下一次调用会把记录与真实的 VMM 进程和主机网络状态重新对齐。
+元数据、网络、快照、克隆、删除和 GC 的关键边界上都埋了具名的故障注入点，并有测试覆盖。
+
+**可切换的元数据后端。** 默认使用 JSON。需要更高并发时可以切换到 SQLite，
+并配合 `metadata status`、`metadata verify` 和经过校验的备份使用。
+
+| 路径 | 用途 |
+| --- | --- |
+| `/var/lib/kumabox` | 镜像、VM 记录、快照、网络租约和内容存储 |
+| `/var/lib/kumabox/run` | PID 文件、API socket、运行态恢复的暂存目录 |
+| `/var/log/kumabox` | VM 与运行时日志 |
+
+## 一次预热，无限分叉
+
+
+
+Agent 天生就会重试、分支和探索。准备环境的成本只需要付一次：启动、安装依赖、预热缓存。
+然后对内存和磁盘打一个**运行态快照**，每次尝试都从它 `clone` 出一台新沙箱。
+每个克隆都会分配新的网络身份，并重新注入 guest 身份标识和熵，避免克隆之间意外共享密钥。
+内存恢复方式可以通过 `--restore-mode copy|ondemand|mmap` 选择。
+
+## 快速开始
+
+需要一台 amd64 或 arm64 的 Linux 主机，能访问 `/dev/kvm`，并具备 root 权限。
+
+```bash
+# 1. 安装并校验发布包
+curl -fsSLO https://github.com/kgpp34/KumaBox/releases/latest/download/kumabox-install.sh
+curl -fsSLO https://github.com/kgpp34/KumaBox/releases/latest/download/kumabox-install.sh.sha256
+sha256sum --check kumabox-install.sh.sha256
+sudo sh kumabox-install.sh
+
+# 2. 一次性准备主机：Cloud Hypervisor、固件、CNI 插件、EROFS 工具
+sudo kumabox-check --upgrade
+sudo kumabox doctor
+
+# 3. 构建官方 guest 镜像
+sudo kumabox image build ghcr.io/kgpp34/kumabox/ubuntu:24.04 --name ubuntu
+
+# 4. 启动一个沙箱并与之交互
+sudo kumabox run ubuntu --name my-vm --cpus 2 --memory 1G --storage 4G
+sudo kumabox exec my-vm -- uname -a
+sudo kumabox exec -it my-vm -- sh
+
+# 5. 一次预热，无限分叉
+sudo kumabox snapshot create my-vm --name base --type running
+sudo kumabox clone base --name fresh
+sudo kumabox exec fresh -- hostname
+
+# 6. 清理
+sudo kumabox delete fresh my-vm --force
+sudo kumabox snapshot rm base
+sudo kumabox image rm ubuntu
+sudo kumabox gc
+```
+
+主机端和 guest 端的产物需要配套使用。对可复现性有要求时，请固定带版本号的 guest 标签
+（例如 `24.04-v0.1.0`）或 OCI digest。单独执行 `sudo kumabox-check` 会做一次只读的主机检查。
+
+### 在 Agent 中调用
+
+`exec --json` 会输出 `ok`、`exitCode`，以及经过 base64 编码的 `stdout` 和 `stderr`。
+进程的退出码与 guest 内命令的退出码一致。
+
+```python
+import base64, json, subprocess
+
+def run_in_sandbox(vm: str, script: str, timeout: str = "120s") -> dict:
+ proc = subprocess.run(
+ ["sudo", "kumabox", "exec", "--json", "--timeout", timeout,
+ vm, "--", "sh", "-c", script],
+ capture_output=True, text=True,
+ )
+ result = json.loads(proc.stdout)
+ for key in ("stdout", "stderr"):
+ result[key] = base64.b64decode(result.get(key) or "").decode(errors="replace")
+ return result
+
+print(run_in_sandbox("fresh", "echo hello from $(hostname)"))
+```
+
+从同一个预热好的快照并行分叉出多个尝试：
+
+```bash
+for i in $(seq 1 8); do
+ sudo kumabox clone base --name try-$i &
+done
+wait
+sudo kumabox ps
+```
+
+官方的 Ubuntu guest 镜像刻意保持精简。如果需要预装自己的工具链（Python、Node、浏览器等），
+可以在 [`oci-images/ubuntu/24.04/Dockerfile`](oci-images/ubuntu/24.04/Dockerfile) 的基础上扩展。
+这个 Dockerfile 已经内置了配套的 `kumabox-agent`、内核和 initramfs。
+
+## 常用命令
+
+| 领域 | 命令 |
+| --- | --- |
+| VM 生命周期 | `run`、`create`、`start`、`stop`、`pause`、`resume`、`delete`、`ps`、`inspect` |
+| Guest 访问 | `exec`、`console`、`logs`、`agent status`、`agent ping`、`agent reseed` |
+| 镜像 | `image build`、`image add`、`image pull-oci`、`image pull`、`image import`、`image inspect`、`image ls`、`image rm` |
+| 快照 | `snapshot create`、`snapshot verify`、`snapshot export`、`snapshot import`、`restore`、`clone`、`hibernate` |
+| 网络 | `network inspect`、`network setup`、`network teardown`、`network resize` |
+| 设备 | `disk attach/detach/list`、`fs attach/detach/list`、`device attach/detach/list/state` |
+| 运维 | `doctor`、`metadata`、`usage`、`gc`、`debug launch` |
+
+完整参数以 `kumabox --help` 为准。
+
+## 与同类项目的对比
+
+
+
+[E2B](https://github.com/e2b-dev/infra) 和
+[CubeSandbox](https://github.com/TencentCloud/CubeSandbox) 都是非常优秀的项目。
+它们和 KumaBox 目标一致：为每个 Agent 任务提供独立的内核。KumaBox 在以下几个方面走了不同的路线：
+
+- **VM 原生，而不是"套了 VM 的容器"。** 沙箱是真正的虚拟机，完整继承 Cloud Hypervisor 的设备模型：热插拔磁盘、virtio-fs 共享目录、在线调整网卡，以及面向 GPU 等加速卡的 VFIO PCI 直通。
+- **快照可以带走。** 运行态快照是一个可校验、可移植的包：导出、拷贝到另一台主机、导入，再从它克隆。
+- **分层镜像，磁盘共享。** OCI 层被转换成只读 EROFS 镜像，由同一主机上的所有 VM 共享。每台 VM 只为自己的写时复制数据付出存储成本。
+- **安装极简。** 只需一个 Go 二进制，加上 Cloud Hypervisor 和 CNI 插件。元数据存放在 JSON 或内嵌的 SQLite 中，不需要额外运维数据库、缓存或对象存储。
+- **正确性可审计。** 统一的操作日志和具名故障注入点，覆盖生命周期、网络、快照、克隆和 GC 等路径。
+- **MIT 许可**，同时支持 amd64 和 arm64。
+
+相关项目：[Kata Containers](https://katacontainers.io/)、
+[gVisor](https://gvisor.dev/)、
+[Firecracker](https://firecracker-microvm.github.io/)、
+[Cloud Hypervisor](https://www.cloudhypervisor.org/)、
+[Cocoon](https://github.com/cocoonstack/cocoon)。
+
+## 愿景
+
+每一次 Agent 行动都应该拥有一台用完即弃的计算机：像 git 分支一样便宜地分叉，像独立机器一样安全。
+KumaBox 自底向上构建这一目标。先在每台主机上做好一个正确、崩溃一致的运行时，
+再基于同一套操作日志和元数据，往上构建常驻服务和多节点控制面。
+这样，同一个沙箱在单台服务器上和整个集群中的行为是一致的。
+
+## 路线图
+
+> 规划方向，欢迎在 Issue 中参与讨论。
+
+- [x] OCI 转 EROFS 镜像、CNI 网络、基于 vsock 的 guest 命令执行
+- [x] 运行态快照、克隆、恢复、休眠、导出与导入
+- [x] 热插拔磁盘、virtio-fs、VFIO PCI；JSON 与 SQLite 元数据后端
+- [ ] 带 HTTP API 的 daemon 模式
+- [ ] 多节点控制面与调度
+- [ ] Go、Python、TypeScript SDK
+- [ ] E2B 兼容 API，让现有 E2B 代码可以直接切换到 KumaBox
+- [ ] MCP server，让 Agent 以工具的形式创建和操作沙箱
+- [ ] 预热池，以及公开的克隆延迟基准测试
+- [ ] 按沙箱粒度的出网策略
+
+## 构建与测试
+
+```bash
+git clone https://github.com/kgpp34/KumaBox.git && cd KumaBox
+make build
+make test
+go vet ./...
+./bin/kumabox version --json
+```
+
+E2E 测试需要 Linux/KVM 主机。它覆盖的流程包括：OCI 镜像构建、冷启动、guest 命令执行与 TTY、
+CNI 地址分配与清理、停机快照与运行态快照、克隆与恢复、磁盘热插拔，以及元数据备份。
+
+```bash
+GO_BIN="$(go env GOROOT)/bin/go"
+sudo test/e2e/e2e.sh --go-bin "$GO_BIN" --metadata-backend sqlite
+sudo test/e2e/e2e.sh --go-bin "$GO_BIN" --metadata-backend json
+```
+
+README 中的配图由代码生成。修改 `assets/readme/src/` 下的脚本后，
+运行 `python3 assets/readme/src/build.py` 即可重新生成。
+
+## 安全模型
+
+- KumaBox 在工作负载外增加了一层虚拟机边界，但 VMM、KVM、guest 内核、固件、镜像和 agent 仍然属于可信计算基。
+- 主机初始化会修改特权网络和系统配置。运行 `--fix` 或 `--upgrade` 之前，请先审阅 `scripts/check.sh`。
+- VFIO 会把物理设备直接交给 guest，需要正确的 IOMMU 分组；使用不当可能影响主机的稳定性和隔离性。
+- 快照的兼容性取决于主机架构、Cloud Hypervisor 版本、VM 配置和快照类型。
+
+可复现的 Bug 和安全问题请通过 Issue 反馈。请不要在公开报告中附带密钥、私有镜像或生产环境快照。
+
+## 许可证
+
+KumaBox 基于 [MIT 许可证](LICENSE) 开源。
diff --git a/assets/readme/architecture.svg b/assets/readme/architecture.svg
new file mode 100644
index 0000000..96d6303
--- /dev/null
+++ b/assets/readme/architecture.svg
@@ -0,0 +1,115 @@
+
+KumaBox architecture: a short-lived CLI control plane driving Cloud Hypervisor microVMs
+
+
+
+
+
+
+
+
+
+
+
+
+
+How KumaBox works
+Each command is a short-lived process. Only the microVMs keep running.
+Your agent, script or CI job
+kumabox run | exec | clone --json
+kumabox
+one process per command
+Resource locks
+safe concurrent commands
+Operation journal
+resumes interrupted work
+Metadata
+JSON or SQLite backend
+Reconcile and GC
+records match reality
+Exits when the work is done. Nothing idles on the host.
+Subsystems each command drives
+Images
+OCI to shared EROFS layers
++ private copy-on-write disk
+Network
+CNI netns per VM
+multiqueue TAP, tc redirect
+Snapshots
+running or stopped, verified
+clone, hibernate, restore
+Devices
+hotplug disks, virtio-fs
+VFIO PCI passthrough
+Linux host with KVM
+amd64 or arm64
+cloud-hypervisor
+VMM process for vm-1
+microVM guest
+hardware boundary
+kumabox-agent on vsock :1024
+your workload: shell, code, tools
+rootfs: shared EROFS + private COW disk
+dedicated Linux guest kernel
+vm-2 ... vm-N, one VMM process each
+vm-2
+vm-3
+vm-4
+vm-5
+vm-6
+...
+Shared EROFS layer store: read-only, deduplicated across VMs
+CNI network: per-VM netns and TAP, NAT to the outside
+vsock
+VMM API
+
diff --git a/assets/readme/comparison.svg b/assets/readme/comparison.svg
new file mode 100644
index 0000000..d49a01c
--- /dev/null
+++ b/assets/readme/comparison.svg
@@ -0,0 +1,85 @@
+
+What you operate to get hardware-isolated sandboxes: KumaBox vs CubeSandbox vs self-hosted E2B
+
+
+
+
+
+
+
+
+
+
+
+
+
+What you run on one host to get one kernel per task
+Taller stacks buy multi-tenant APIs, scheduling and dashboards. KumaBox keeps only what a single host needs.
+KumaBox
+MIT, Go
+Linux + KVM, amd64 or arm64
+cloud-hypervisor + CNI plugins
+kumabox, one binary
+That's the whole stack.
+No database, no cluster manager,
+no resident daemon.
+CubeSandbox, one-click node
+Apache-2.0
+Linux + KVM, x86_64 or ARM64
+CubeHypervisor + CubeShim
+Cubelet + CubeVS eBPF network
+CubeMaster + lifecycle manager
+CubeAPI, CubeProxy, CubeEgress
+MySQL, Redis, MinIO
+Web UI + CubeOps
+Full sandbox service with E2B API.
+Terraform or Kubernetes for clusters.
+E2B Embed, one machine
+Apache-2.0, Go
+Linux + KVM + Docker Compose
+Firecracker VMM
+Orchestrator + template builder
+API server + client proxy
+PostgreSQL, Redis, ClickHouse
+Dashboard + log pipeline
+Same runtime as E2B Cloud.
+Terraform or Kubernetes for more nodes.
+Single-host install footprint, from each project's repository and deploy files, September 2026.
+
diff --git a/assets/readme/hero.svg b/assets/readme/hero.svg
new file mode 100644
index 0000000..9f236ca
--- /dev/null
+++ b/assets/readme/hero.svg
@@ -0,0 +1,90 @@
+
+KumaBox: a disposable computer for every agent task
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Kuma Box A disposable computer
+for every agent task.
+Hardware-isolated microVMs on KVM,
+driven by one daemonless CLI.
+snapshot: base
+fresh-1
+fresh-2
+fresh-3
+fresh-4
+$ kumabox clone base --name fresh-N
+Own kernel per sandbox
+Zero resident daemons
+OCI images
+Running snapshots
+
diff --git a/assets/readme/lifecycle.svg b/assets/readme/lifecycle.svg
new file mode 100644
index 0000000..b3130a9
--- /dev/null
+++ b/assets/readme/lifecycle.svg
@@ -0,0 +1,86 @@
+
+Sandbox lifecycle: build once, warm once, fork many
+
+
+
+
+
+
+
+
+
+
+
+
+
+Warm once, fork many
+Pay the setup cost one time, then hand every agent attempt its own copy of a ready machine.
+1
+OCI image
+any digest-pinned ref
+image build
+2
+EROFS layers
+shared, read-only
+run
+3
+Running microVM
+agent ready on vsock
+exec
+4
+Warmed sandbox
+deps installed, caches hot
+snapshot
+5
+Running snapshot
+memory + disks captured
+clone x N
+new IP, new identity
+The same snapshot also lets you
+restore in place to roll back
+export and import to another host
+hibernate: free the VMM, keep the state
+
diff --git a/assets/readme/product.svg b/assets/readme/product.svg
new file mode 100644
index 0000000..80b14b8
--- /dev/null
+++ b/assets/readme/product.svg
@@ -0,0 +1,78 @@
+
+Product shape: callers, interfaces, and what one KumaBox sandbox gives you
+
+
+
+
+
+
+
+
+
+
+
+
+
+What KumaBox is
+A sandbox runtime you install on one Linux box. Anything that can run a command can drive it.
+Who drives it
+Coding agents
+Agent frameworks
+RL and eval harnesses
+CI and batch jobs
+You, at a terminal
+How it is driven
+kumabox CLI
+available now
+Human output, or --json for machines
+Streams stdout, stderr and exit codes
+Dry-run launch plans via kumabox debug
+Planned
+Go SDK
+HTTP API, E2B-compatible
+MCP server for tool use
+What each sandbox gets
+one microVM
+Its own guest kernel behind KVM
+OCI rootfs + private writable disk
+Own network namespace and IP
+exec with env, workdir, stdin, TTY
+Snapshot, clone, hibernate, restore
+Opt-in data disks, virtio-fs, VFIO
+
diff --git a/assets/readme/src/build.py b/assets/readme/src/build.py
new file mode 100644
index 0000000..a03ea22
--- /dev/null
+++ b/assets/readme/src/build.py
@@ -0,0 +1,6 @@
+"""Regenerate every README graphic: python3 assets/readme/src/build.py"""
+import os, runpy, sys
+here = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, here)
+for name in ["gen_hero", "gen_product", "gen_arch", "gen_life", "gen_compare"]:
+ runpy.run_path(os.path.join(here, name + ".py"))
diff --git a/assets/readme/src/gen_arch.py b/assets/readme/src/gen_arch.py
new file mode 100644
index 0000000..1c49a16
--- /dev/null
+++ b/assets/readme/src/gen_arch.py
@@ -0,0 +1,90 @@
+from gen_common import *
+W, H = 1200, 780
+s = svg_open(W, H, "KumaBox architecture: a short-lived CLI control plane driving Cloud Hypervisor microVMs")
+s += t(48, 58, "How KumaBox works", 26, FROST, 700)
+s += t(48, 88, "Each command is a short-lived process. Only the microVMs keep running.", 16, MUTED)
+
+# ---- left column: control plane
+L, LW = 48, 472
+s += box(L, 116, LW, 56, fill=PANEL2)
+s += t(L+LW/2, 150, "Your agent, script or CI job", 17, FROST, 600, "middle")
+s += line(L+LW/2, 172, L+LW/2, 214)
+s += t(L+LW/2+14, 199, "kumabox run | exec | clone --json", 13, MUTED, cls="mono")
+
+s += box(L, 220, LW, 236, fill=PANEL, stroke=SLATE, sw=1.6)
+s += t(L+24, 256, "kumabox", 20, FROST, 700, cls="mono")
+s += t(L+130, 256, "one process per command", 14, MUTED)
+chips = [("Resource locks", "safe concurrent commands"),
+ ("Operation journal", "resumes interrupted work"),
+ ("Metadata", "JSON or SQLite backend"),
+ ("Reconcile and GC", "records match reality")]
+cw = (LW - 48 - 14) / 2
+for i, (a, b) in enumerate(chips):
+ x = L + 24 + (i % 2) * (cw + 14); y = 276 + (i // 2) * 66
+ s += box(x, y, cw, 54, fill=PANEL2, rx=10)
+ s += t(x+14, y+23, a, 15, FROST, 600)
+ s += t(x+14, y+43, b, 12.5, MUTED)
+s += f'
+{esc(title)}
+
+
+
+
+
+
+
+
+
+
+
+
+
+{esc(s)} \n'
+
+def cube(cx, cy, s, top=SLATE_L, left=PANEL, right=SLATE, stroke="#0F1826", sw=2, extra=""):
+ """Isometric cube; (cx,cy) is the centre of the top face, s = half-diagonal."""
+ k = 0.866 * s
+ h = s * 1.15
+ topf = f"{cx},{cy-s/2} {cx+k},{cy} {cx},{cy+s/2} {cx-k},{cy}"
+ leftf = f"{cx-k},{cy} {cx},{cy+s/2} {cx},{cy+s/2+h} {cx-k},{cy+h}"
+ rightf= f"{cx+k},{cy} {cx},{cy+s/2} {cx},{cy+s/2+h} {cx+k},{cy+h}"
+ return (f'']
+ k = 86.6; hh = 50
+ # back rim of box (behind bear)
+ g.append(f''
+ ' ')
+ g.append(' ')
+ return "\n".join(g) + "\n"
diff --git a/assets/readme/src/gen_compare.py b/assets/readme/src/gen_compare.py
new file mode 100644
index 0000000..1dcebcb
--- /dev/null
+++ b/assets/readme/src/gen_compare.py
@@ -0,0 +1,45 @@
+from gen_common import *
+W, H = 1200, 664
+s = svg_open(W, H, "What you operate to get hardware-isolated sandboxes: KumaBox vs CubeSandbox vs self-hosted E2B")
+s += t(48, 58, "What you run on one host to get one kernel per task", 26, FROST, 700)
+s += t(48, 88, "Taller stacks buy multi-tenant APIs, scheduling and dashboards. KumaBox keeps only what a single host needs.", 16, MUTED)
+
+cols = [
+ ("KumaBox", "MIT, Go", HONEY,
+ ["Linux + KVM, amd64 or arm64", "cloud-hypervisor + CNI plugins", "kumabox, one binary"],
+ ["No database, no cluster manager,", "no resident daemon."]),
+ ("CubeSandbox, one-click node", "Apache-2.0", SLATE_L,
+ ["Linux + KVM, x86_64 or ARM64", "CubeHypervisor + CubeShim", "Cubelet + CubeVS eBPF network", "CubeMaster + lifecycle manager",
+ "CubeAPI, CubeProxy, CubeEgress", "MySQL, Redis, MinIO", "Web UI + CubeOps"],
+ ["Full sandbox service with E2B API.", "Terraform or Kubernetes for clusters."]),
+ ("E2B Embed, one machine", "Apache-2.0, Go", SLATE_L,
+ ["Linux + KVM + Docker Compose", "Firecracker VMM", "Orchestrator + template builder", "API server + client proxy",
+ "PostgreSQL, Redis, ClickHouse", "Dashboard + log pipeline"],
+ ["Same runtime as E2B Cloud.", "Terraform or Kubernetes for more nodes."]),
+]
+base = 548; bh = 44; gap = 8; cw = 344
+for ci, (name, lic, accent, blocks, foot) in enumerate(cols):
+ x = 48 + ci * (cw + 36)
+ s += t(x, 146, name, 19, HONEY if ci == 0 else FROST, 700)
+ s += t(x + cw, 146, lic, 13, MUTED, anchor="end")
+ s += f' \n'
+open(os.path.join(OUT, "comparison.svg"), "w").write(s)
+print("cmp ok")
diff --git a/assets/readme/src/gen_hero.py b/assets/readme/src/gen_hero.py
new file mode 100644
index 0000000..f0ed1d5
--- /dev/null
+++ b/assets/readme/src/gen_hero.py
@@ -0,0 +1,42 @@
+from gen_common import *
+W, H = 1200, 460
+style = """
+ .clone { opacity: 0; animation: pop 6s ease-out infinite; }
+ .c1 { animation-delay: 0.3s; } .c2 { animation-delay: 0.7s; } .c3 { animation-delay: 1.1s; } .c4 { animation-delay: 1.5s; }
+ @keyframes pop { 0% {opacity:0; transform: translateX(-18px);} 12% {opacity:1; transform: translateX(0);} 86% {opacity:1;} 100% {opacity:0;} }
+ .pulse { animation: pulse 3s ease-in-out infinite; transform-origin: 820px 205px; }
+ @keyframes pulse { 0%,100% { opacity: .55; } 50% { opacity: 1; } }
+ @media (prefers-reduced-motion: reduce) { .clone { opacity: 1; } }
+"""
+s = svg_open(W, H, "KumaBox: a disposable computer for every agent task", style)
+s += bear_box(165, 188, 1.05)
+# text column
+x0 = 330
+s += f'Kuma Box ' + cube(tx, ty, 24, top=SLATE_L) + t(tx+34, ty+18, f"fresh-{i}", 14, FROST, 500, cls="mono") + ' \n'
+s += t(940, 410, "$ kumabox clone base --name fresh-N", 14, MUTED, 400, "middle", "mono")
+
+# chips
+cx = 60; cy = 384
+for label in ["Own kernel per sandbox", "Zero resident daemons", "OCI images", "Running snapshots", "amd64 + arm64", "MIT"]:
+ p, w = pill(cx, cy, label, size=13, h=28, pad=12, cw=7.2)
+ if cx + w > 790:
+ break
+ s += p; cx += w + 10
+s += '\n'
+open(os.path.join(OUT, "hero.svg"), "w").write(s)
+print("hero ok")
diff --git a/assets/readme/src/gen_life.py b/assets/readme/src/gen_life.py
new file mode 100644
index 0000000..61b46e8
--- /dev/null
+++ b/assets/readme/src/gen_life.py
@@ -0,0 +1,50 @@
+from gen_common import *
+W, H = 1200, 430
+style = """
+ .fan { opacity: 0; animation: fan 5s ease-out infinite; }
+ .f1 { animation-delay: .2s; } .f2 { animation-delay: .5s; } .f3 { animation-delay: .8s; }
+ @keyframes fan { 0% {opacity:0;} 14% {opacity:1;} 88% {opacity:1;} 100% {opacity:0;} }
+ @media (prefers-reduced-motion: reduce) { .fan { opacity: 1; } }
+"""
+s = svg_open(W, H, "Sandbox lifecycle: build once, warm once, fork many", style)
+s += t(48, 58, "Warm once, fork many", 26, FROST, 700)
+s += t(48, 88, "Pay the setup cost one time, then hand every agent attempt its own copy of a ready machine.", 16, MUTED)
+
+steps = [(120, "OCI image", "any digest-pinned ref", SLATE_L, "image build"),
+ (330, "EROFS layers", "shared, read-only", SLATE_L, "run"),
+ (540, "Running microVM", "agent ready on vsock", SLATE_L, "exec"),
+ (750, "Warmed sandbox", "deps installed, caches hot", SLATE_L, "snapshot"),
+ (960, "Running snapshot", "memory + disks captured", HONEY, "clone")]
+cy = 190
+for i, (x, a, b, top, verb) in enumerate(steps):
+ s += cube(x, cy, 34, top=top)
+ s += f''
+ s += path(f"M1000 {cy+20} C 1040 {cy+20}, 1050 {cy+20+dy}, {fx-30} {cy+20+dy}", color=HONEY, sw=1.6, marker=None, dash="4 6")
+ s += cube(fx, cy+6+dy, 18, top=SLATE_L)
+ s += ' \n'
+s += t(fx, cy+132, "clone x N", 16, FROST, 700, "middle")
+s += t(fx, cy+154, "new IP, new identity", 13, MUTED, anchor="middle")
+
+# side branches from snapshot
+by = 368
+opts = ["restore in place to roll back", "export and import to another host", "hibernate: free the VMM, keep the state"]
+x = 48
+s += t(48, by-16, "The same snapshot also lets you", 14, MUTED, 600)
+for o in opts:
+ p, w = pill(x, by, o, size=13.5, h=32, pad=16, cw=7.3)
+ s += p; x += w + 12
+s += '\n'
+open(os.path.join(OUT, "lifecycle.svg"), "w").write(s)
+print("life ok")
diff --git a/assets/readme/src/gen_product.py b/assets/readme/src/gen_product.py
new file mode 100644
index 0000000..3f3d226
--- /dev/null
+++ b/assets/readme/src/gen_product.py
@@ -0,0 +1,48 @@
+from gen_common import *
+W, H = 1200, 520
+s = svg_open(W, H, "Product shape: callers, interfaces, and what one KumaBox sandbox gives you")
+s += t(48, 58, "What KumaBox is", 26, FROST, 700)
+s += t(48, 88, "A sandbox runtime you install on one Linux box. Anything that can run a command can drive it.", 16, MUTED)
+
+# left: callers
+s += t(48, 140, "Who drives it", 14, MUTED, 600)
+callers = ["Coding agents", "Agent frameworks", "RL and eval harnesses", "CI and batch jobs", "You, at a terminal"]
+for i, c in enumerate(callers):
+ y = 156 + i * 58
+ s += box(48, y, 268, 44, fill=PANEL2, rx=22)
+ s += t(182, y+28, c, 15, FROST, 500, "middle")
+
+# middle: interfaces
+MX, MW = 404, 336
+s += t(MX, 140, "How it is driven", 14, MUTED, 600)
+s += box(MX, 156, MW, 132, fill=PANEL, stroke=HONEY, sw=1.8)
+s += t(MX+22, 192, "kumabox CLI", 20, FROST, 700, cls="mono")
+s += t(MX+MW-22, 190, "available now", 12.5, HONEY, 600, "end")
+s += t(MX+22, 222, "Human output, or --json for machines", 13.5, MUTED)
+s += t(MX+22, 244, "Streams stdout, stderr and exit codes", 13.5, MUTED)
+s += t(MX+22, 266, "Dry-run launch plans via kumabox debug", 13.5, MUTED)
+planned = ["Go SDK", "HTTP API, E2B-compatible", "MCP server for tool use"]
+s += t(MX, 318, "Planned", 13, MUTED, 600)
+for i, p in enumerate(planned):
+ y = 330 + i * 52
+ s += box(MX, y, MW, 42, fill="none", stroke=SLATE, rx=10, dash="5 5")
+ s += t(MX+22, y+27, p, 14.5, MUTED, 500)
+
+# right: sandbox
+RX, RW = 820, 332
+s += t(RX, 140, "What each sandbox gets", 14, MUTED, 600)
+s += box(RX, 156, RW, 322, fill=PANEL, stroke=SLATE_L, sw=1.6)
+s += cube(RX+46, 190, 22, top=SLATE_L)
+s += t(RX+84, 204, "one microVM", 18, FROST, 700)
+feats = ["Its own guest kernel behind KVM", "OCI rootfs + private writable disk", "Own network namespace and IP",
+ "exec with env, workdir, stdin, TTY", "Snapshot, clone, hibernate, restore", "Opt-in data disks, virtio-fs, VFIO"]
+for i, f in enumerate(feats):
+ y = 262 + i * 36
+ s += f' 
+ 
+
