Epic: first-class libkrun backend with virtio-fs --mount support

[aniketmaurya]

Context

Filed after the #279 → #280 → #282 chain that fixed smolvm create --mount ./ on QEMU. During that work it became clear that --mount will stay QEMU-only as long as Firecracker is the alternative backend (Firecracker rejects host-share devices by design — discussed in #282 thread). The realistic path to mount support on a minimalist VMM is finishing the libkrun integration that's currently a stub, not forking Firecracker.

This issue captures the full scoping report from a research agent so we have a durable record before deciding whether/when to commit to the work.

Bottom line up front: ~3.25–5.5 engineer-weeks (realistically 4–6) to make smolvm create --mount --backend libkrun ./ work end-to-end. Significantly cheaper than a Firecracker fork, and unblocks future libkrun features (snapshots, pause/resume) that are stubbed out today.

---

SmolVM --mount on libkrun: Scoping Report

1. Current libkrun integration state in this repo

Wired (process-level only):

• src/smolvm/runtime/libkrun.py — LibkrunRuntimeAdapter. Spawns/stops processes; pause, resume, create_snapshot, restore_snapshot all raise SmolVMError("...does not support...yet").
• src/smolvm/vm.py:1773-1825 (_find_krunvm_binary, _start_libkrun) — shells out to the krunvm binary with run --cpus --memory --kernel --rootfs --kernel-params [--initrd --vsock-cid]. No -v/--volume flag is ever passed.
• src/smolvm/runtime/backends.py — BACKEND_LIBKRUN = "libkrun" registered.
• src/smolvm/cli/main.py — libkrun accepted in --backend choices on four subcommands.
• src/smolvm/host/doctor.py:609-619 — checks krunvm binary on PATH; warns "currently tuned for macOS in SmolVM."

Stub / explicit reject:

• src/smolvm/facade.py:2224 and src/smolvm/vm.py:811-815 — hard-rejects workspace_mounts on any non-QEMU backend with "Workspace mounts (virtio-9p) are only supported with the QEMU backend".
• src/smolvm/images/published.py:48-50 — comment: "libkrun is reserved here for a future spike — manifest accepts the type but the CLI never resolves a host to it until libkrun support is wired."

Rust bindings (smolvm-core/): Do not mention libkrun. The Rust crate is purely Linux netlink + sysctl + TAP helpers (network plumbing for Firecracker). Cargo.toml has no libkrun-sys or FFI to the C library.

No references anywhere to virtiofs, virtio-fs, virtiofsd, krun_add_virtiofs, or KRUN_FS_ROOT_TAG outside the kernel README's "future enhancement" note.

Recent libkrun commits: b0b6e49 (initial krunvm-shell-out backend), cd810ec (default to QEMU on macOS, away from libkrun), 88ac028 ("libkrun stubs"). Direction has been "stubs that compile" rather than first-class.

2. libkrun upstream feature state for virtio-fs

• virtio-fs is in the public C API. include/libkrun.h exposes krun_add_virtiofs(ctx, tag, host_path), krun_add_virtiofs2(..., shm_size), and krun_add_virtiofs3(..., shm_size, read_only). Multiple shares allowed. Root-fs sharing via krun_set_root or virtiofs3 with KRUN_FS_ROOT_TAG.
• No external virtiofsd daemon required. libkrun's src/devices/src/virtio/fs/ ships an in-process server (server.rs, worker.rs, filesystem.rs, fuse.rs, plus linux/+macos/ backends) — i.e., libkrun is itself the virtio-fs server (a "passthrough" FUSE-over-virtio backend baked into the library).
• macOS works. README requires "macOS 14 or newer," uses Hypervisor.framework on arm64. The macOS path of krunvm calls krun_add_virtiofs directly. Apple Silicon is the original target.
• Critical gotcha — krunvm on Linux does not use virtio-fs. In krunvm/src/commands/start.rs, the Linux map_volumes() calls libc::mount() (a host-side bind mount into the container chroot before VM launch); only the macOS map_volumes() calls krun_add_virtiofs. So if SmolVM keeps shelling out to krunvm, mounts on Linux work via a different mechanism than on macOS — and they're tied to krunvm's OCI-image workflow, not arbitrary kernel+rootfs invocations.

libkrunfw kernel config (config-libkrunfw_x86_64): CONFIG_VIRTIO_FS=y, CONFIG_FUSE_FS=y, CONFIG_OVERLAY_FS=y, CONFIG_NET_9P not set. Confirmation that virtio-fs + FUSE are the libkrun-blessed stack. Irrelevant for us in practice — SmolVM ships its own kernel via kernel/microvm/build.sh and points --kernel at it; libkrunfw is bypassed.

3. virtiofsd packaging

Mostly moot — libkrun has its own embedded server, so SmolVM does not need to ship virtiofsd if it talks to libkrun via the C API. But for context: Ubuntu (24.04 noble and 26.04 resolute) has virtiofsd 1.13.2 in universe. macOS has no official Homebrew formula; only the third-party slp/virtiofs tap. If we were forced down a virtiofsd path (e.g., to drive raw libkrun without C bindings), macOS packaging would be a real problem — another argument for the embedded-server route.

4. Guest kernel implications

• virtio-fs guest support requires both CONFIG_VIRTIO_FS=y and CONFIG_FUSE_FS=y (virtio-fs is a FUSE transport — FUSE is the dependency). CONFIG_OVERLAY_FS=y is already set. Adding both to kernel/microvm/config.fragment is independent of and non-conflicting with the existing 9p stanza (CONFIG_NET_9P*, CONFIG_9P_FS, CONFIG_OVERLAY_FS); both can coexist.
• vmlinux size: FUSE+virtio-fs adds roughly 60–120 KB to a stripped vmlinux (low six-figure bytes). Firecracker's ELF loader has no fixed cap; this is a non-issue.
• Cheaper-path question — can we run 9p over libkrun's virtio? No. libkrun does not implement a virtio-9p device — its only host-share device is virtio-fs. The 9p kernel bits already in config.fragment are unusable on libkrun. There is no path that avoids virtio-fs.

5. Concrete punch list

(A) Decision: drive libkrun via C FFI, not krunvm. krunvm's Linux mount path is a host bind mount that requires its OCI/buildah workflow — it doesn't fit SmolVM's "explicit kernel+rootfs+config" model. So SmolVM should call libkrun's C API directly (a small Rust wrapper in smolvm-core/), bypassing krunvm for --mount. This also unblocks future pause/resume/snapshot work since krunvm doesn't expose those either.

• smolvm-core/Cargo.toml: add [target.'cfg(any(target_os = "linux", target_os = "macos"))'.dependencies] libkrun-sys (or hand-roll bindgen against libkrun.h). ~10 lines.
• smolvm-core/src/krun.rs (new): wrap krun_create_ctx, krun_set_vm_config (vCPU, mem), krun_set_root, krun_add_virtiofs3 (multi-call for N mounts), krun_set_kernel, krun_set_exec/krun_set_env, krun_start_enter. PyO3 functions: krun_launch(spec: PyDict) -> int (pid). ~250–350 LoC.
• smolvm-core/src/lib.rs: register the new module, gate non-Linux/macOS to Err. ~30 LoC.

(B) Python backend rewiring.

• src/smolvm/vm.py _start_libkrun / _async_start_libkrun (≈100 lines): replace the krunvm subprocess.Popen with a call into _smolvm_core.krun_launch(...) that passes vm_info.config.workspace_mounts straight through. Keep a KRUNVM_FALLBACK flag for one release. ~150 LoC delta.
• src/smolvm/facade.py:2224 and vm.py:811-815: drop the backend != BACKEND_QEMU reject; gate on backend not in {BACKEND_QEMU, BACKEND_LIBKRUN}.
• src/smolvm/facade.py:1899 _ensure_9p_workspace_support: rename to _ensure_workspace_fs_support; on libkrun, probe \tvirtiofs\n and \tfuse\n in /proc/filesystems; on QEMU keep the 9p probe. The Ubuntu linux-modules-extra apt-fallback is irrelevant for our built-in kernel and can stay 9p-only since libkrun never runs the Ubuntu cloud image without our kernel.
• src/smolvm/facade.py _mount_workspaces (~vm.py:1826): branch on backend. Guest mount command becomes mount -t virtiofs <tag> <guest_path> (no trans=virtio, no version=9p2000.L). Overlay-stack logic is unchanged.

(C) Kernel config.fragment. Add CONFIG_VIRTIO_FS=y and CONFIG_FUSE_FS=y next to the 9p block; both archs, common file. No conflicts. Bump kernel/microvm/linux.sha256 artifact hashes after rebake (existing CI flow — see 1fd6478).

(D) Doctor. host/doctor.py: drop the macOS-only warning when --mount is in play; instead check _smolvm_core exposes the krun functions. Replace the krunvm binary check with library-presence (libkrun.so.1 / libkrun.dylib found via ctypes.util.find_library or maven-style probe). ~40 LoC.

(E) CLI routing. src/smolvm/cli/main.py: --backend libkrun --mount already passes through arg parsing. The auto-select (8a5c33d, cb20d4d — auto-pick QEMU when --mount and no --backend) should be revisited: keep QEMU as auto-default on Linux for parity, but allow libkrun explicitly. ~5–10 LoC.

(F) Tests.

• Unit: tests/test_backends.py/test_vm.py/test_workspace.py — drop the pytest.mark.parametrize("backend", ["firecracker", "libkrun"]) assertion that mounts raise; add libkrun positive case with mocked _smolvm_core.krun_launch.
• Integration: a new tests/integration/test_libkrun_mount.py that, when _smolvm_core reports libkrun-capable, boots a tiny rootfs with --mount /tmp/foo:/workspace, sshes in, verifies a host-written file is visible. Linux-only in CI (macOS CI separately).
• Rust: smolvm-core/src/krun.rs unit tests behind #[cfg(feature = "libkrun-tests")].

6. Risks and open questions

• libkrun-sys Rust crate maturity. No widely-maintained libkrun-sys exists on crates.io; we'll likely hand-roll bindings via bindgen against the system libkrun.h. Adds a build-time dep on libclang for SmolVM contributors building from source.
• libkrun availability in distro packages. libkrun itself is in Fedora COPR but not in Ubuntu apt; users would need the slp/krun Homebrew tap on macOS or a Copr/PPA on Linux. SmolVM has to either (a) document install steps, (b) detect-and-error-clearly, or (c) bundle the .so/.dylib. Bundling pulls licensing scrutiny (libkrun is LGPL-2.1).
• macOS Apple Silicon path is the supported one — but our CI runs on Linux. virtio-fs on libkrun's Linux KVM backend is less battle-tested than the macOS HVF backend (per upstream issue traffic). We may hit perf or hang regressions that don't repro in macOS smoke tests.
• Existing _ensure_9p_workspace_support Ubuntu apt fallback is conceptually wrong for libkrun (we never run Ubuntu cloud image under libkrun in practice), but the rename to _ensure_workspace_fs_support must keep the QEMU+Ubuntu path working for users still on the QEMU backend. Don't collapse them into one branch-free helper.
• krunvm as fallback. If we keep it as a one-release fallback, the Linux/macOS divergence in volume semantics (host-bind vs virtio-fs) leaks into user-visible behavior — e.g., file-uid mapping differs. Probably better to cut over cleanly.
• Snapshot/pause still unimplemented in libkrun adapter — adding --mount doesn't fix that, and the QEMU-only error in _check_workspace_mounts already says "Snapshotting is not supported for VMs with workspace mounts," which now applies to libkrun too. Fine for now, but flag for users.
• Vsock CID collisions. The current _start_libkrun passes --vsock-cid to krunvm; the C API path requires us to allocate CIDs ourselves (libkrun has krun_set_vsock_port/related calls). One more spec field to wire.
• virtio-fs DAX window sizing. krun_add_virtiofs3 takes shm_size; default of 0 means no DAX (slower, but simpler). Picking too large a window on memory-constrained guests can OOM. Recommend defaulting shm_size = 0 initially.

7. Effort sizing (engineer-weeks, honest)

Area	Weeks
Rust FFI: bindgen against libkrun.h, krun.rs PyO3 wrapper, error mapping, tests	1.0–1.5
Python rewiring: _start_libkrun C-FFI path, mount probe split, mount command branch, dropping the QEMU-only reject	0.5–0.75
Kernel: add VIRTIO_FS/FUSE_FS, rebake artifacts, sha resync, CI green	0.25
Doctor + install docs (Homebrew tap, COPR/apt), packaging story for libkrun.{so,dylib} discovery	0.25–0.5
Tests: unit + Linux integration; macOS smoke if a runner exists	0.5–1.0
Bugs surfaced during real-world dogfooding (uid mapping, perf, hangs on KVM Linux path), debug + fix	0.75–1.5
Total	3.25–5.5 engineer-weeks

Honest read: a well-scoped intern-plus-staff-review effort lands in ~4 weeks if the libkrun KVM Linux path Just Works; budget closer to 6 if it doesn't and we end up filing upstream issues. The "fork Firecracker for 9p" alternative would be at least 2x this with ongoing maintenance tail — finishing libkrun is the right call.

Suggested execution order

If/when this is scheduled:

1. Land virtio-fs + FUSE in the kernel fragment — 5-min change, can ride the next kernel rebake. Cheap insurance even if we don't ship libkrun mounts immediately.
2. Replace krunvm shell-out with FFI in smolvm-core — the 1–1.5 week chunk that unlocks everything else (mounts + future pause/resume + snapshot).
3. Workspace-mount routing + tests — the application of (2) to --mount.

Sources

• containers/libkrun (README and headers)
• containers/libkrun virtio-fs source dir
• containers/libkrunfw (kernel blob library)
• containers/krunvm
• virtiofsd on GitLab
• Linux kernel virtio-fs documentation
• Sergio López — Running microVMs on M1/M2