ARCHITECTURE.md

Architecture

This service exposes two endpoints:

1. POST /execute: takes a namespace_id and a bash command, runs the command inside that namespace's sandbox, and returns stdout, stderr, and the exit code. Each namespace has its own persistent filesystem, namespaces are isolated from one another, and requests for the same namespace run in the order they were received.
2. GET /health: Health check endpoint

This document explains the major decisions and their trade-offs. The code runs the full flow locally with mock backends; the production path (Firecracker, EBS, Redis, DynamoDB) is described here, stubbed behind the same traits, and declared as Terraform under terraform/.

Assumptions

• The caller is an AI agent in a distributed harness, so latency matters. This is the main reason the production target is a microVM (fast to boot) rather than a container task (slow to start).
• There is no "create namespace" call, so a namespace is created implicitly on first use.
• Authentication is out of scope.

Threat model

In scope: one namespace must never read another namespace's data, files, temporary artifacts, or cached files; and one namespace must not be able to starve others of host resources (RAM, disk, processes, CPU).

Out of scope: a malicious tenant exploiting the kernel to break out of the sandbox. The production target still uses a hardware-virtualized microVM (Firecracker), which eliminates the kernel-escape class of attacks as a side effect even though the brief did not require defending against it.

Overall shape: control plane vs data plane

The API never runs bash itself. It validates the request, orders it per namespace, and forwards it to a separate execution layer that can fail independently. Co-locating arbitrary bash with the API would let one namespace starve the others through resource exhaustion, degrading the endpoint for everyone, which is both a security and a scalability failure.

The expected production request flow is:

• An AI agent calls POST /execute through a load balancer fronting the stateless API tier.
• The API resolves which execution host currently owns the namespace and forwards the command to it.
• On that host, an agent attaches the namespace's storage, runs the command inside a microVM, and returns the output.

Modules

A few small traits keep the service backend-agnostic. Each has a runnable local implementation and a documented production one:

• Sandbox (src/sandbox) is the data-plane backend: given a namespace and a command, it makes the namespace's environment ready (it owns its provisioning) and runs the command. LocalSandbox runs a child bash process on the host over a mock provisioner. In production the API tier's sandbox is RemoteSandbox (a placeholder), which resolves the execution host that owns the namespace and forwards the command to it; that host's FirecrackerSandbox (also a placeholder) attaches the volume and runs the command inside a microVM. The API tier never attaches storage or runs bash — that is the execution fleet's job, which is why only the execution IAM role holds EBS/KVM rights.
• Coordinator (src/coordinator) provides per-namespace FIFO ordering and mutual exclusion. InMemoryCoordinator backs it with in-process structures; RedisCoordinator backs it with a Redis list and key shared by every server. See "Execution ordering".
• Provisioner (src/provisioner) makes a namespace's volume + microVM ready. It is owned by a Sandbox backend, not the executor, because provisioning is part of running the command and lives on the machine that runs it. MockProvisioner (owned by LocalSandbox) simulates the steps; AwsProvisioner (a placeholder, owned by the execution host's FirecrackerSandbox) attaches the EBS volume and boots/reuses the microVM.

NamespaceExecutor (src/executor) is the thin orchestrator over these boundaries (order, run, release) and is the only thing the API layer depends on; it holds just a coordinator and a sandbox. Because these are traits, moving from the local path to the production path is a wiring change in src/main.rs, not a rewrite.

Isolation

Runtime boundary. In production each namespace runs in its own Firecracker microVM, so memory and processes are isolated by hardware virtualization. The local backend runs the command as an ordinary host process (dropping to a non-root uid when the service itself runs as root); it shares the host kernel, so it is not a strong boundary, but it applies best-effort limits without root (src/sandbox/limits.rs).

Resource limits (noisy-neighbor problems). Aggregate limits need cgroups v2, not ulimits, because ulimits are per-process and cannot cap a whole process tree. In production the microVM's cgroups enforce:

• memory.max caps aggregate memory, including the RAM-backed tmpfs at /tmp; exceeding it triggers an OOM kill contained to that namespace. Swap is disabled (memory.swap.max = 0) so the cap fails deterministically instead of spilling to disk.
• pids.max caps the number of processes, the real defense against fork bombs.
• cpu.max bounds CPU so one namespace cannot peg the host.

Locally the same intent is best-effort: memory and process count via a per-execution cgroup v2 when a delegated cgroup is available (otherwise RLIMIT_AS for memory), and CPU time via RLIMIT_CPU, since the cpu controller is usually not delegated to user sessions.

Filesystem. The only writable locations are the namespace's volume and a size-capped tmpfs at /tmp. In production the root filesystem is mounted read-only, so a write to any other path fails with EROFS, and the read-only root image is shared across all namespaces. Local provides ordering, write-confinement (to the namespace dir and /tmp), and resource caps only. Cross-namespace read isolation and private /tmp are properties of the production microVM, not the local backend.

Data. Each namespace gets a dedicated EBS volume, and only that namespace's volume is attached to its microVM, so another namespace's data is not even present to be read. Each volume is encrypted with a per-namespace KMS key, so isolation holds at rest. /tmp is private per namespace so scratch files do not leak.

Persistent filesystem

Each namespace's filesystem is a dedicated EBS volume whose lifecycle follows the namespace, not the microVM: the microVM is ephemeral and the volume persists.

• On the first request for a new namespace, the control plane creates and formats a volume and records the namespace-to-volume mapping in DynamoDB.
• On every request, the volume is attached to the host running the namespace's microVM and mounted at the single writable path.
• After an idle period the microVM is torn down and the volume detached, but never deleted; the next request re-attaches it with its data intact.
• The volume is deleted only when the namespace itself is deleted, after snapshotting to S3.

Two constraints matter. EBS volumes are scoped to one Availability Zone, so a namespace is pinned to its volume's AZ when it is placed. And a gp3 volume with ext4 is single-attach and single-writer, so a namespace's filesystem can be mounted in only one place at a time, which matters for ordering below.

Alternatives considered: EFS access points mount faster on a cold start but rely on a shared filesystem for isolation, which is weaker than a dedicated block device; S3 hydrate/dehydrate keeps hosts stateless but its download cost is bad for large filesystems. EBS was chosen because strong isolation is non-negotiable, and cold-attach latency is amortized by keeping a volume attached while the namespace is warm.

Inside the microVM

A Firecracker microVM is a minimal Linux guest. For a namespace activation it holds:

• A small guest kernel, shared across all namespaces.
• A read-only root filesystem: a single shared, immutable image fanned out to every VM, holding bash, coreutils, the harness toolchain, and shared libraries. Read-only is the point — there is nothing per-namespace in it to tamper with, and a write outside the volume or /tmp fails with EROFS.
• The namespace's EBS volume, attached by the host as a virtio-block device and mounted at the single writable path (the working directory commands run in).
• A size-capped tmpfs at /tmp (RAM-backed, counted against the VM's memory.max).
• A vsock device for host-to-guest communication. There is deliberately no network interface in the guest.

What accepts and runs commands is a tiny in-guest agent running as PID 1, baked into the read-only rootfs. It is trusted, host-provided code — not user code. Its whole job: on boot, mount the volume and listen on a vsock port; on a request, run bash -c "<cmd>" with the working directory set to the mounted volume and a scrubbed environment; capture stdout, stderr, and the exit code; and send them back over vsock. A single request's command may fan out into many processes — they all run inside this one VM and are capped in aggregate by its cgroups.

So a production request passes through three distinct agents, worth not conflating:

• API tier (RemoteSandbox) — resolves the owning execution host from the registry and forwards the command. Holds no EBS/KVM rights.
• Host agent (FirecrackerSandbox + AwsProvisioner, on the .metal host, outside any VM) — attaches the EBS volume, boots or reuses the namespace's microVM, and bridges the command over vsock. The only component with EBS/KVM rights.
• In-guest agent (PID 1 inside the VM) — runs the bash and returns the output. Has no network and no AWS access at all.

Why vsock and no guest network: the guest has no NIC, so its only channel in or out is the vsock socket to its own host — no guest IPs, no SSH, no credentials to manage, and a minimal attack surface for untrusted bash. Combined with the hardware-virtualized kernel boundary, a breakout attempt lands inside an isolated VM with nothing reachable rather than on the shared host.

Execution ordering

Requests for the same namespace must run in arrival order, while different namespaces are independent. Multiple bash commands fanned out by a single request are fine; they run as concurrent processes inside that one execution.

Ordering is enforced by the Coordinator with three layers per namespace, keyed by a per-request uuid:

1. FIFO queue (a Redis list, or an in-memory VecDeque locally): a request joins the back on arrival and may run only when its uuid is at the front. This fixes arrival order.
2. Namespace lock (a Redis key with a TTL): the front request claims it before executing. This is the mutual exclusion, and the TTL gives crash recovery, so a dead holder's lock is reclaimed rather than wedging the namespace.
3. The uuid fences both: the lock is claimable only by the front-of-queue uuid, in one atomic step, and each request releases only what is its own.

Taking the turn is atomic; the front-check and the lock-claim happen together, so the front request is the only one that can hold the lock. That is what makes ordering strict rather than a race among waiters. The same code runs locally (InMemoryCoordinator) and across the fleet (RedisCoordinator); distribution is a wiring change, not a different design. Unit tests in src/coordinator/in_memory.rs cover the guarantee.

Why Redis rather than SQS FIFO. The endpoint is synchronous: the caller blocks and gets the output back on the same request. SQS FIFO is a producer/consumer queue, so the API would enqueue a message and then wait for the result to return out of band over a response queue or stream, turning a simple request/response into an async executor for no benefit the brief asks for. A Redis list and lock stay synchronous: a server polls the shared list and the front request runs in place, returning its output directly.

Defense in depth: EBS single-attach. A gp3/ext4 volume is single-attach and single-writer, so a namespace's filesystem can be mounted on only one host at a time. This is an independent, storage-level guarantee beneath the coordinator: even if the distributed lock had a bug, the volume itself refuses a second mount, so two VMs can never corrupt a namespace's data. The lock provides ordering and liveness; EBS provides the hard mutual-exclusion fence.

Limitation: a hard process crash cannot run the release path, so the dead holder's uuid lingers at the front of the queue. The lock's TTL frees the lock, but evicting a stuck queue head would need per-waiter heartbeats, deliberately omitted here for simplicity.

Streaming evolution: the endpoint is blocking today. The natural next step is streaming output (chunked response, SSE, or WebSocket) so an agent receives output incrementally. That is the one place a producer/consumer carrier such as SQS or Redis streams would re-enter, for moving output back across a process boundary, and it stays hidden behind the executor boundary.

Scalability of namespaces

The design targets a very large number of namespaces, most of them idle at any moment. Idle EBS volumes only cost storage, so the total namespace count is cheap. What costs money is the number of concurrently active namespaces, and the number of volumes that can attach to one host is limited (a few dozen on Nitro), so the execution fleet is sized by peak concurrency, not total namespaces. The API tier holds no per-namespace state and scales horizontally behind the load balancer; all per-namespace state lives in DynamoDB.

AWS resources and why

• EC2 .metal (Nitro) instances: provide /dev/kvm so Firecracker can run.
• EBS gp3, one volume per namespace: dedicated, encrypted, snapshot-able persistent filesystem.
• DynamoDB: the namespace registry (namespace to its volume + AZ and the execution host that currently owns it); keeps the API tier stateless and lets it route a request to the owning host.
• ElastiCache (Redis): the coordinator, a per-namespace FIFO queue and lock shared across the fleet.
• Application Load Balancer: fronts the stateless API tier.
• KMS: per-namespace encryption key, so data isolation holds at rest.
• S3: volume snapshots and backups.

A managed alternative (ECS/Fargate) was considered as a simpler start: it gives microVM isolation without running your own hosts, but task start-up is slow and per-namespace volume control is weaker. Lambda was rejected because it cannot bind a per-namespace persistent volume without leaking across reused execution environments.

Deployment

There are three artifacts, built independently and shipped to two tiers:

• Guest image — a pinned guest kernel plus a read-only rootfs with the in-guest agent baked in as PID 1, alongside bash, coreutils, and the harness toolchain. Built once in CI, versioned, and published to an artifact store (e.g. S3); every microVM boots a copy. Updating the toolchain or the agent means building a new image version and rolling the fleet — nothing per-namespace changes, and the running namespace volumes are untouched.
• Host agent — the execution-host program (FirecrackerSandbox + AwsProvisioner in code terms). A long-running service on each .metal node that receives forwarded commands, manages the per-namespace microVM pool (boot / reuse / idle-teardown), attaches and detaches EBS volumes, and bridges vsock. Runs under the execution IAM role (EBS, KVM, registry, KMS, S3).
• API service — the Rust binary in this repo, run with RUST_ENV=prod so it wires RemoteSandbox. Stateless, behind the ALB, under the API IAM role (registry-read only).

Both tiers are Auto Scaling Groups (see terraform/compute.tf): the API tier scales on request load, the execution fleet on concurrent active namespaces (bounded by how many EBS volumes attach to one Nitro host). Each ASG's launch-template user_data brings a node up — installing the service binary on the API tier, and Firecracker + the host agent + the current guest image on the execution fleet. The host agent is the same kind of boundary switch as local-vs-prod: it could be this same binary run in an "execution host" role (wiring FirecrackerSandbox/AwsProvisioner and listening for forwarded commands on :8081) rather than a separate program. The Terraform here declares this topology but is not applied.

What is intentionally not built

This is a small exercise that values simplicity, so the code is small and the ambition lives in this document.

• Authentication (out of scope).
• FirecrackerSandbox, RedisCoordinator, and AwsProvisioner are documented placeholders; InMemoryCoordinator, MockProvisioner, and LocalSandbox make the full flow run locally.
• The distributed control plane (the DynamoDB registry) is described, not coded. The Terraform under terraform/ declares the AWS topology but is not applied.

Running locally

See README.md. In short: set RUST_ENV=local, run with cargo run, and POST to /execute. Each namespace gets a directory under SANDBOX_FS_ROOT (default ./namespaces) that persists across requests, the local stand-in for an attached volume.