Open-sourcing our Confidential Virtual Machine builder

Attestable Virtual Machines, from firmware, to kernel, to OS

Since we launched Kettle, our attested build tool, we've been working to offer Confidential Virtual Machines that can be verified at every level. Today, we're launching ConfidentialOS Builder, an open-source tool that builds images for Confidential Virtual Machines.

Knowing the provenance of your own software is important, but you should be able to trace every piece of the system you are using — if your binary is trusted but the OS you run it inside could be compromised, it's pointless. What, exactly, is running inside your VM? ConfidentialOS Builder exists to make sure you know.

Our CVM builder brings together TEEs (Trusted Execution Environments), a custom Linux kernel, and a custom OS build, to provide a hardware-signed proof of exactly what ran inside your VM. This proof doesn't require trust in the cloud host machine, or trust in anyone with root access to the host machine. Instead, it uses AMD's SEV-SNP or Intel's TDX to cryptographically sign a boot measurement.

The signed boot measurement includes a checksum of the firmware and the initrc that will bootstrap the VM. The initrc includes a merkle-tree checksum of the entire VM disk, including every package and binary you included. By using overlayfs and a read-only disk, we can prove that every boot matches what was originally built. By building your own software into the image, you have hardware measurements that trace all the way to the exact binaries that the CVM runs.

The chain of trust

Like Kettle, our security promises start with the TEE: an encrypted virtual machine whose memory cannot be read by the host, thanks to an encryption key provided directly to the VM by the host's Trusted Platform Module. Since the encryption key is never exposed to the host node, even root access on the host does not allow any visibility into what is running in the TEE VM.

An attestation from a TEE includes boot measurements of the VM, the firmware and software used to bring the machine up, signed by a private key that can be verified belongs to the hardware manufacturer (in this case, AMD or Intel). In our images, the firmware and boot package include a cryptographic digest of the contents of the VM's entire boot disk. Once the firmware has loaded the kernel and started running the init process, the init process reads the disk checksum and uses dm-verity to ensure the entire disk exactly matches the build-time checksum.

That disk was built read-only and compressed using the erofs filesystem. It can't be changed by any action the VM takes, because that would alter the dm-verity checksum and cause the Kernel to terminate access to the disk. An overlayfs allows ephemeral writes, ensuring work can be done but changes will not persist to the next boot of the CVM.

Using our images

The easiest way to use our attested images is with our Confidential Cloud. If you want to replicate our VMs yourself, though, we've done our best to make it straightforward. If you have qemu and oras, you can run one of our images in just a few seconds. We're going to use the dev image, since that includes console access to the VM after it boots.

oras pull ghcr.io/confidential-dot-ai/confidential-os-builder:dev
qemu-system-x86_64 \
  -machine q35 \
  -kernel uki.efi \
  -drive if=pflash,format=raw,readonly=on,file=OVMF.fd \
  -drive file=disk.raw,format=raw,if=virtio \
  -smp 1 -m 4G -nographic

That's it! You've got a running VM. For a Confidential VM running inside a TEE, you'll also need the host machine to have SEV-SNP or TDX. For SEV-SNP, you'll need qemu 10.1 or higher, compiled with the IGVM extensions enabled as well. To make sure you're getting all the CLI flags right, you can use confos to help you, like this:

confos pull ghcr.io/confidential-dot-ai/confidential-os-builder:dev
confos run output/dev --scratch 20g

That will build the full qemu command line to enable KVM and TEE mode if your host supports it, as well as create and attach a 20GB disk image the VM can use for encrypted, ephemeral storage.

Building custom images

Unlike dev, the base profile leaves it up to you to layer on your own environment into your Confidential VM images. We've stripped things down to a locked-down kernel and just 29 packages, with a primary goal of ensuring the host doesn't have any ways to observe or interact with the CVM it's hosting. That means removing the SSH server, disabling the serial console, and counting on users to add their own packages.

You can customize the software that will run inside the CVM by creating a mkosi profile or by supplying a cloud-init configuration file that will run when your VM boots. You'll want to think about the tradeoff that's ideal for your usecase, where putting everything in the image gives you VMs that become ready faster and sign more of their code, but are bigger files to manage and less flexible if you need dynamic behavior.

From inside the ConfidentialOS Builder repo, you can see some example mkosi profiles in mkosi/base/mkosi.profiles. You can build a profile by running bin/confos build dev --profile dev, which will create files in output/dev.

There's also an example cloud-init config that installs the Caddy web server and hosts a static webpage in examples/. To include the example in your built image, run bin/confos build --cloud-init examples/caddy.yaml web to create an image in output/web.

After you've built an image, ConfidentialOS Builder wraps the oras tool to push the disk image and associated files to any OCI registry via confos push.

Share and enjoy gaps: the GPU's own attestation (SPDM) is not yet surfaced to

the relying party (a non-CC GPU fails closed at guest boot, but there is no positive GPU evidence in the attestation result); the GPU guest boots kata's GPU kernel (measured, but not our hardened configuration); and on SEV-SNP each GPU pod is pinned to a single vCPU to keep the launch measurement stable. Host GPU provisioning (vfio-pci binding, GPU CC mode, BAR resize) is assumed done

Thanks to ConfidentialOS Builder, we were able to add a detailed verification mode to Kettle that attests not just the source code that went into your binary, but the entire build VM, from firmware, to kernel, to OS, with a signature anyone can verify after downloading the public VM image used to create the build.

We're looking forward to seeing how this kind of tool enables new uses for TEEs and confidential VM workloads. If you're interested in building your own CVM images, check out the documentation on GitHub. We'd love to hear how ConfidentialOS Builder works for you!