Socket Programming

The previous post left off at the hop that everything else depends on: the host has to be able to reach into a microVM, tell it to do something, and read back the result. That communication runs over sockets, so it’s worth a short detour into socket programming. Sockets are one way to let two processes talk to each other — not the only way, shared memory and other IPC mechanisms exist — but they’re how the host talks to its sandbox microVMs, and Firecracker’s control plane is a clean example to learn from.

Starting Firecracker and creating a microVM

The Firecracker getting-started docs show how to start the API server and then create a microVM. You give Firecracker a socket path, run it, and then talk to it with curl:

API_SOCKET="/tmp/firecracker.socket"

# Remove any stale API unix socket
sudo rm -f $API_SOCKET

# Run firecracker
sudo ./firecracker --api-sock "${API_SOCKET}" --enable-pci

# To start the microVM, call the /actions endpoint
curl --unix-socket ${API_SOCKET} -i \
    -X PUT "http://localhost/actions" \
    -d '{ "action_type": "InstanceStart" }'

The first thing that should jump out is that the address of the server is a file path, /tmp/firecracker.socket, and the path itself is arbitrary — you could put it anywhere. That raises the obvious question: is a socket really just any file path? In Unix, more or less, yes. A socket is just a file.

A socket is just a file

What Firecracker is doing under the hood when it starts up is creating that socket file on the filesystem. The sequence is the standard one any server makes:

int fd = socket(AF_UNIX, SOCK_STREAM, 0); // create a socket object in the kernel
bind(fd, "/tmp/firecracker.socket");      // this step creates the file on disk
listen(fd, ...);                          // start accepting connections

The detail that matters is AF_UNIX. That argument tells the kernel to create a Unix domain socket rather than a network socket, which would be AF_INET. A Unix domain socket is a socket whose endpoint is a path on the local filesystem, and it exists specifically for inter-process communication between processes on the same machine. So /tmp/firecracker.socket is, in effect, Firecracker’s address. Firecracker listens on that file, and that file is its only listening endpoint. There is no TCP port anywhere in this exchange. When a client wants to talk to Firecracker, it sends to that path.

Where’s the port?

Now look at the curl command again, because it seems to contradict everything I just said. It says http://localhost/actions. That’s an HTTP URL with a hostname, and HTTP usually means TCP and a port. So what is localhost/actions, and where is the port?

The trick is that the port is never used. The --unix-socket /tmp/firecracker.socket flag tells curl: ignore the hostname in the URL, don’t do DNS, don’t open a TCP connection — instead, open this Unix socket file and write the HTTP bytes into it. So when curl connects, it’s writing HTTP bytes into a file on disk, not sending packets to any port.

Port 80 does flicker into existence for a moment, just not where you’d think. When curl parses the URL, the URL grammar says that for the http scheme with no explicit port, the default is 80, so internally curl decides “host is localhost, port is 80.” But before it ever acts on that and opens a TCP connection to localhost:80, the --unix-socket flag intercepts and says: forget the host and port, open this socket file instead. curl then writes the HTTP request bytes into the socket. Those bytes still carry a Host: localhost header, because HTTP requires one, but to Firecracker that header is just metadata it ignores. Firecracker reads the HTTP request off its end of the socket and processes it.

So the “port 80” only ever exists inside curl’s URL parser. It’s never put on a wire, never sent in a packet, never seen by Firecracker. The Unix socket bypasses TCP entirely — no IP, no port, no network stack.

Firecracker’s one-process-per-VM model

Once you know how to start Firecracker and create a microVM, the natural next question for a system that wants ten thousand of them is: do you run one Firecracker process on the host and spawn many microVMs from it? The answer is no. Each Firecracker process encapsulates one and only one microVM. One Firecracker process per VM, always. There’s no shared mode and no “Firecracker daemon” that hosts a fleet the way QEMU can be wrapped by libvirt. If you want a hundred VMs, you run a hundred Firecracker processes. Each process manages exactly one microVM, has exactly one API socket, runs that one guest kernel and its vCPUs as threads, and exits when the VM shuts down.

This isn’t a limitation to work around — it’s the security and simplicity story that makes Firecracker worth using for exactly this kind of multi-tenant, untrusted-code workload, and it’s worth understanding why.

The first reason is that isolation is the entire point. Firecracker was built by AWS to power Lambda and Fargate, where untrusted user code runs on shared hardware, and the threat model assumes the guest kernel can be compromised. The question that follows is: what’s the blast radius? With one process per VM, an exploit that breaks out of the guest only lands you in a process that has almost no privileges — Firecracker jails itself with seccomp, drops capabilities, and runs in a chroot — and it can’t see other guests’ memory, file descriptors, or state, because those live in entirely separate processes. A single shared process hosting many VMs would mean a guest escape immediately has access to every other tenant’s VM state in the same address space, which is a non-starter for a multi-tenant cloud.

The second reason is that a smaller attack surface buys simpler code. Because each process only has to manage one VM, it can stay minimal — Firecracker is roughly fifty thousand lines of Rust, against QEMU’s millions of lines of C, and less code means fewer bugs and an easier audit. A multi-VM Firecracker would have to grow tenant-separation logic to track which vCPU thread belongs to which guest, per-VM resource accounting for CPU and memory, failure isolation so one panicking VM can’t take down its neighbors, and locking around all the newly shared state. All of that marches you straight back toward QEMU’s complexity. Firecracker declined, and pushed that complexity up into the orchestrator — the host agent — where it’s easier to reason about and to fix.

The third reason is that resource accounting becomes the kernel’s job for free. When each VM is its own process, you get all the ordinary Linux primitives without writing any of them: cgroups limit CPU and memory per process and therefore per VM, the OOM killer can take out one VM and leave the rest alone, a plain kill -9 cleanly shuts down a single VM, ps and top and htop show per-VM resource use with no special tooling, and systemd can supervise VMs as services if you want it to. A shared process couldn’t lean on any of this; it would have to grow its own internal scheduler and accountant.

The fourth reason is crash isolation. If Firecracker itself hits a bug and crashes, the per-VM model means one VM dies while the others keep running, and the orchestrator notices the dead PID and restarts it. In a shared model, that same crash takes down every VM at once. For a cloud IDE serving many users, that’s the difference between one user’s notebook hiccupping and everyone getting logged out.

The cost of all this is simply that you end up running many processes of the same type. For a system whose whole job is to keep untrusted tenants apart, that’s a price well worth paying.