Research Firecracker jailer integration and VMM privilege reduction

[hbrodin]

Observation

At commit df5909c, setup extracts the Firecracker jailer, but src/vm.rs::FirecrackerVm::start launches the configured Firecracker binary directly through a PID-recording shell under sudo. No jailer invocation or equivalent namespace/privilege-dropping setup was found in that launch path.

docs/getting-started.md says Firecracker uses the jailer, while docs/backends.md describes the direct root launch. These descriptions need to agree with the implementation.

Firecracker's production host guidance recommends the jailer or equivalent process restrictions. The current launch does not disable Firecracker's default seccomp filters; lack of jailer integration should not be described as lack of all runtime sandboxing. No guest-to-host exploit was reproduced for this finding.

Research scope

• Confirm the effective VMM privileges and restrictions in supported Linux deployments, including configured binary overrides and seccomp behavior.
• Evaluate jailer integration versus equivalent restrictions, including dedicated identities, privilege dropping, filesystem isolation, device access, and resource limits.
• Design ownership and path rules for the executable, jail root, kernel, rootfs, API socket, logs, and PID state. Account for the jailer's trusted-input requirements and user-owned coop data directories.
• Determine migration and lifecycle behavior for existing instances, concurrent VMs, snapshots, restore/reprovision, partial startup failures, timeouts, and cleanup.
• Identify startup checks that fail clearly when required confinement cannot be established, without silently falling back to a less restricted launch.
• Correct the documentation discrepancy independently of whether jailer integration is selected.

Expected outcome

Produce a bounded implementation proposal with explicit guarantees, residual risks, migration requirements, and Linux integration coverage. Validate effective process identity and isolation in addition to successful VM boot and shutdown. Preserve seccomp and verify resource cleanup on failed starts.

Related: #5. This issue isolates the VMM launch-confinement investigation from the broader networking and resource-hardening work.

Implementation research update (2026-09-15)

Source inspection at coop 6441210ed14fa123cf3e53401b266f3966cc8e03 confirms the original launch observation still holds. Upstream reference: Firecracker v1.17.0, commit 95f868c8e345b1cc8faccd1a3c910b4989dc3f58. This is a research reference, not a version coop currently pins. No KVM boot or effective runtime confinement was validated: the research environment has no /dev/kvm. Root VMM identity is inferred from the launch and ordinary sudo behavior; inherited container/LSM restrictions and binary overrides can change the effective environment.

Findings that change the implementation

1. Installation must become part of the confinement design. install_firecracker skips installation when Firecracker already exists, and jailer extraction/rename/chmod errors are nonfatal. We cannot assume a matching executable jailer exists. Upstream requires the matching static musl Firecracker/jailer pair and trusted paths/ancestors. The current user-owned data directory is unsuitable as the privileged installation/jail root without redesign. Use an approved root-owned installation, verify artifact identity, and define an explicit migration or rejection policy for firecracker_bin overrides. Running an arbitrary override as root to inspect --version is not validation. Current setup, upstream jailer contract

2. Seccomp guarantees depend on the executable, not just launch flags. coop does not pass --no-seccomp, but upstream debug/GNU builds can lack default filters. Preserve the approved release policy, check relevant API/VMM/vCPU threads, and include negative controls. Seccomp: 2 alone is insufficient because an inherited host/container filter can produce that result. Upstream seccomp

3. Use a fresh jail generation per launch and keep authoritative state outside it. The jailer makes its jail root and copied executable VMM-owned. Treat a previously used jail as potentially attacker-modified; do not feed it back into privileged startup. Its inner PID file is also under that writable root and must not independently authorize host signals or deletion. Generate IDs independently of coop names: coop permits underscores, while jailer IDs do not. Inspected implementation

4. Filesystem integration affects every disk lifecycle operation. Current VM JSON contains host paths for kernel, rootfs, and vsock. Stage only required resources, generate jail-relative paths, and maintain explicit host-side socket/log access. Separate the persistent disk and trusted registry from disposable jail contents. A same-filesystem hard link can expose the instance's authoritative writable disk; never link a writable instance to an image template or peer. Cross-filesystem placement needs an explicit copy/bind-mount strategy. Restore must replace the disk reference rather than keep a link to the previous inode. Audit create/start, disk_path/rootfs_path, resize, commit, restore/reprovision, image deletion, and uninstall. Privileged path operations must resist symlink/rename/ACL races; root ownership of a file does not protect writable ancestors.

5. Privilege dropping requires TAP ownership changes, not a network redesign. Today's setup_tap creates the persistent TAP without assigning the future VMM identity. Prepare it for a dedicated per-instance UID/GID before dropping privileges, while preserving existing bridge isolation checks. Do not grant the VMM CAP_NET_ADMIN as a shortcut. KVM/tun access also depends on host device-cgroup and LSM policy. Stock jailer may expose urandom and userfaultfd as well as KVM/tun, so an exact device allowlist must be deliberately implemented and tested. Current networking, kernel TAP permission model

6. PID tracking and failure cleanup are prerequisites. --daemonize and --new-pid-ns change the relevant process PID; the current shell trampoline cannot simply wrap those modes. Current liveness uses a firecracker command-line substring. start_existing propagates boot failures without the new-instance allocation path's cleanup, and stop removes PID/socket state even after its final exit wait fails. Add per-instance lifecycle locking, supervised process identity, bounded startup/API/termination deadlines, and explicit uncertain/cleanup-required states. Confirm all processes are gone before deleting resources or recycling an identity. Preserve an existing disk on failed restart. CLI SIGKILL, VMM crash, and host reboot require recovery beyond an in-process drop guard. VM lifecycle, backend boot paths

7. Resource configuration needs explicit policy and readback. With cgroup v2 and no --cgroup arguments, jailer can proceed without joining a nonexistent --parent-cgroup; successful launch does not prove membership. The inspected source defaults RLIMIT_NOFILE to 2048, despite production guidance mentioning 4096. Set/test our intended limit. Memory limits need headroom beyond guest RAM. Bound and drain logs/stderr; a small global RLIMIT_FSIZE is not a log-only limit and can interfere with rootfs writes. CPU/task/memory overhead and collector behavior still need measurements. Jailer implementation, resource limits

Proposed direction — decisions still open

• Prefer upstream jailer plus systemd system-service/cgroup-v2 supervision, with root-owned installation/registry and dedicated per-instance identities. A fixed privileged helper should accept typed instance operations, not arbitrary commands, paths, unit properties, or jailer flags.
• This would introduce a new supported-host requirement. Decide whether to adopt that initial profile or implement a portable supervisor. Neither systemd nor cgroup v2 is an approved new requirement yet. Unsupported hosts must not silently fall back to direct root launch.
• A foreground jailer without --daemonize or --new-pid-ns simplifies initial service supervision. It provides a private mount/root but does not provide a private PID namespace. If PID namespaces are required in the first slice, include host-PID tracking/reaping in that slice. Network namespaces and guest routing changes remain separate scope.
• An equivalent systemd sandbox is an alternative, but coop would own more filesystem/device confinement logic. UID dropping alone, or the proxy's Landlock policy alone, does not satisfy the proposed contract.

Migration and acceptance criteria

• Migrate stopped instances only, with versioned state and transactional disk import. Preserve the original disk until import commits; retain shared sidecars, image lineage, and persisted memory/vCPU settings. Identify already-running legacy VMs as legacy until stopped/restarted. Define downgrade/export behavior; failed migration must not trigger a legacy root boot.
• coop commit/restore currently copy disks, not Firecracker memory/CPU snapshots. Preserve this behavior. Native memory snapshots are separate work and must recreate confinement before resume.
• Require an actual KVM-capable test gate. Current CI runs unit and host integration checks, but not the full VM boot suite. Test effective UIDs/GIDs/groups/capabilities, mount/root isolation, inherited descriptors, per-thread seccomp/no-new-privileges, cgroup membership/limits, and host/peer access denials—not merely SSH readiness.
• Inject failures at staging, TAP creation, launch, API readiness, and boot; test timeout/OOM, collector failure, CLI termination, stale/reused PIDs, hostile old jail contents, concurrent lifecycle operations, migration interruption, restore, and uninstall. A failed start must leave no live VMM or active confinement/network resources; failed restart must preserve the disk. Deliberately remove checks to prove the tests detect missing confinement. Run macOS/Lima integration for shared lifecycle changes too.
• Correct the documentation independently. Local documentation corrections were prepared during research, but are not merged.

Suggested delivery sequence: documentation correction; separate lifecycle/storage prerequisite refactors; trusted installation and migration; confined launch and cleanup; mandatory confinement integration gate. Broader network hardening, storage/I/O quotas, kernel accounting gaps, and host-side handling of guest-mutated disks remain residual work under #5 or separate issues. No guest-to-host exploit was reproduced.