microVMs and Firecracker

I’m building a sandbox IDE — a place where an AI agent can run code remotely, on my infrastructure instead of someone’s laptop. This is the first post in a series about how that system is put together, and it starts where these decisions should start: with what the thing actually has to do, and then the one choice everything else hangs off, which is what you run the untrusted code inside.

Functional requirements

The workload I’m designing for is general-purpose code, Python-first but polyglot, and it’s interactive rather than long-running batch. People are running cells and watching for output, not submitting jobs and walking away, p99 for starttime of the sandbox should be under 500ms. The scale I’m aiming at is ten thousand concurrently active sessions against a hundred thousand registered users, and the latency that matters most is the one a person feels: a cold session should be ready in under three seconds of perceived wait, and resuming a warm one should be under a second.

The trust model is the part that shapes everything downstream. All user code is untrusted, full stop. Users share the underlying hardware, but no one can have any visibility into anyone else’s processes, filesystem, or network. That single sentence is what rules out the easy answers later.

It helps to separate two things that often get conflated. A workspace owns files and is the durable unit; a session is ephemeral compute that gets attached to a workspace for as long as someone is working. Collaboration — several people writing into one workspace at once — is out of scope for v1, where I’m assuming a single writer per workspace; CRDTs can be layered on later without the rest of the design having to change. For networking, outbound internet is allowed so that pip install and friends work, but it goes through NAT with rate limiting, while inbound traffic and any lateral traffic between sessions is blocked outright. The whole thing is AWS-native, but I’m keeping the cloud-specific pieces behind abstractions so it stays portable.

Compute substrate: Firecracker microVMs

The trust requirement is what forces the substrate decision. I need strong isolation between untrusted tenants and I need a cold start fast enough to hide inside a three-second budget, and most options give you one of those at the expense of the other.

Option	Cold start	Isolation	Density	Verdict
EC2 VMs	30–60s	Strong (hardware)	Low	Too slow
Docker / k8s	< 1s	Weak (shared kernel)	High	Insufficient isolation for untrusted code
Firecracker microVMs	~150ms	Strong (KVM)	High	✓ Chosen
gVisor (runsc)	~300ms	Strong (syscall interception)	High	Good fallback

Full EC2 VMs give you the hardware boundary but take half a minute or more to come up, which blows the latency budget on its own. Containers come up in under a second and pack densely, but they share the host kernel, and a shared kernel is not a boundary I’m willing to put between untrusted tenants. Firecracker sits exactly where I need it: a cold start around a hundred and fifty milliseconds, isolation enforced by KVM at the hardware-virtualization layer, and density high enough to make ten thousand concurrent sessions economical. gVisor, which intercepts syscalls in userspace, is the close runner-up and a good fallback, with a slightly higher cold start.

So the rule is simple: each session gets exactly one Firecracker microVM. The hypervisor layer enforces hardware-level isolation between tenants, while the startup time rivals a container’s. That’s the whole reason Firecracker exists, and it’s why the rest of this series is built on it.

High-level design

With the substrate chosen, the rest of the system is the machinery that gets a request from a browser down to a running microVM and back.

Browser ──► API gateway ──► Scheduler ──► Host agent ──► Firecracker microVM
                                │              │              │
                                ▼              ▼              ▼
                              Redis      TAP + iptables    vm-agent (vsock)
                            (hostpool)   (per-VM NAT)

A request comes in from the browser and lands at the API gateway, which hands it to the scheduler. The scheduler’s job is placement: it looks at Redis, which holds the pool of available hosts, and decides which machine should run the new session. From there a host agent running on the chosen machine does the local work of standing up the session — it boots the Firecracker microVM, and it wires up that VM’s networking by creating a TAP device and the per-VM iptables rules that give it NAT’d outbound access while keeping it isolated from everything else. Inside the guest, a small vm-agent is what the host actually talks to, over a vsock channel rather than the network.

That last hop — how the host and the guest talk to each other — is its own subject, and it’s where the next post goes. Before any of the scheduling or networking matters, you have to be able to send a command into a microVM and get its output back out, and that turns out to be a small lesson in socket programming.