Attestation-gated Key Release for Confidential Computing Workloads
In Confidential Computing, all the data in your machine is protected while in use; the CPU uses hardware isolation to protect the registers and caches, and the DRAM is protected from a nosy host machine using an ephemeral encryption key. This protects your applications while they're in operation, but at scale machines are spinning up and down all the time and it's essential to maintain state across reboots. We need a way to manage a long-lived secret that can be shared between different machines.
In normal Kubernetes key release, pods present ServiceAccount objects representing their identity to a secrets broker, and the secrets broker returns any secrets that the policy allows for that identity. It trusts that the Kubernetes Control Plane issues the correct identity to each pod; otherwise misidentified pods could fetch secrets not meant for them.
With attestation-gated key release, pods present a hardware-rooted attestation of the code they are running, and the key broker releases specific secrets knowing they can only be used with that code.
At Confidential AI, we run c8s - Confidential Kubernetes - a variant of Kubernetes that leverages Confidential Virtual Machines (CVMs) using trusted execution environments (TEEs) to ensure that your workloads run in a verifiably confidential way. In c8s, the control plane is treated as adversarial: it could issue incorrect identities to pods or worse, impersonate them.
c8s now supports attestation-gated key release: a workload retrieves a secret only if it can prove which code it is running. It reuses the same attestation path as the rest of c8s, but three problems made it harder than it looks: an attestation covers a whole node, the Container Runtime Interface (CRI) has no concept of a pod, and the containers that fetch secrets start before the container you want to gate on.
The problem
Imagine you've spent millions of dollars training a specialized model for a given task. You'd like to license it to a customer for use in their infrastructure, but if you hand them the model weights directly they can copy them and cut you out of the loop!
With attestation-gated key release, the credentials that decrypt your model go only to a workload that can prove two things: it runs inside a TEE and it runs a specific version of specific code. A rogue developer cannot configure their own app to decrypt and extract your weights.
┌───────────────────────────────────────────────────────────────┐
│ model provider │
│ encrypts the weights volume │
└─────────────┬───────────────────────────────────┬─────────────┘
│ 1 encrypted volume │ 2 key + policy
▼ ▼
┌───────────────────────────┐ ╔═══════════════════════════╗
│ customer storage │ ║ key broker (CVM) ║
└─────────────┬─────────────┘ ╚═══════╤═════╤═════════════╝
▼ 3 encrypted volume │ │
│ 4 attest ▲ │
│ │ ▼ 5 key
╔═════════════╧═════════════════════════════╧═════╧═════════════╗
║ │ serving pod (CVM) ──┘ │ ║
║ │ │ ║
║ └─────────────────┬─────────────────┘ ║
║ ▼ 6 decrypt volume ║
║ ┌─────────────────────────────────────┐ ║
║ │ /weights · plaintext in memory │ ║
║ └──────────────────┬──────────────────┘ ║
║ ▼ 7 reads ║
║ ┌─────────────────────────────────────┐ ║
║ │ serve@sha256:ab… --model=/weights │ ║
║ └─────────────────────────────────────┘ ║
╚═══════════════════════════════════════════════════════════════╝
The provider ships an encrypted volume and keeps the policy. Ciphertext and key only meet inside the CVM - the plaintext at /weights never leaves memory, and a pod running a different image, or the same image with different args, gets the volume and no key.
With a traditional Key Management System (KMS):
- the infra team might configure a network or namespace policy in Kubernetes to lock down a team's services
- an application or database team might write a KMS policy to manage which services can access which of their secrets
- and the security team might write or integrate software for audit logging which services fetch which secrets for later analysis
In c8s, each workload can prove what software it's running and its environment, so we can collapse these into a single policy.
"Stick it in c8s and you're done" - it sounds easy, but there were a few quirks that made designing the system harder than it appears.
What is an attestation?
For the most common case, an attestation is a signature over the launch measurements of a VM. An attestation therefore covers only what is measured at launch: software that is not in the VM's launch filesystem or kernel cannot appear in the report.
Our main mode of operation for c8s draws a trust boundary around an entire Kubernetes node. Workloads operate as normal containers through the traditional Kubernetes machinery. The attestation report contains kubelet, but not the individual containers because we can't possibly know which containers are going to be started at launch!
To be concrete, two Kubernetes pods running on the same c8s node will have the same attestation despite running different containers. This posed a problem for attestation-gated key release: if we rely solely on the launch measurements, any pod running on a given node image could fetch a secret for any other pod in the cluster.
In principle, we can bind a piece of runtime data into the attestation's REPORT_DATA field, but where does it come from?
We needed to create a notion of workload identity.
Workload identity
Workload identity proved harder than you might expect:
You can't read the pod from the control plane. The control plane is outside the trust boundary, so we can't believe anything it says.
CRI has no concept of a pod. In Kubernetes, kubelet starts containers using the CRI. When kubelet receives a pod from the control plane, it reads the containers in the manifest and calls CreateContainer on the CRI implementation for each one. The CRI has no concept of a pod: it does not know all the container digests up front, it does not know whether more containers are still to be admitted, and it has no knowledge of the control plane, which it could not trust anyway.
Pods are more than a container digest. Two pods with the same set of container digests aren't necessarily the same workloads. Pods are also dependent on their environment, their commands, and their arguments. In a big org, dozens of teams might be using busybox for dozens of different use cases: they definitely shouldn't all have the same access to secrets.
c8s certificate and secret sidecars run before other containers in a pod. Users typically want to gate secret release on the main container: only user-service@sha256:abcdef... should be able to fetch the user database password. c8s injects get-cert and get-secret as the first and second containers of a confidential pod. Because the CRI has no concept of a pod, the pod identity must bind to user-service@sha256:abcdef... before that container is admitted.
Binding the attestation to an identifier
While the CRI has no concept of a pod, it does have a concept of a sandbox. A sandbox holds a small set of metadata for each pod: its namespace, its mounts, and the containers that have been created, started, stopped, or paused. Each sandbox has a unique identifier, the sandbox ID.
When a trust boundary is drawn around the node, the CRI implementation sits inside it. It manages admissions against the allowlist, and its own attestation proves it runs inside a CVM. When a pod creates an attestation, the CRI signs a binding between that attestation and the sandbox ID. The pod presents that binding for secret release once all its containers have been admitted.
The sandbox ID names a pod whose containers are not yet admitted, and a component inside the trust boundary issues it. Attestation therefore happens once, before the container set is known, instead of once per admitted container.
initContainers cannot request secrets. All containers in a phase (init, main) are admitted at roughly the same time, so this restriction is the trade-off we accepted for now. A future release will remove it.
The digest alone isn't the full story, as described earlier. The policy engine now also pins the command and arguments for each allowed pod, so multiple busybox instances with different commands do not receive the same secrets.
We chose not to pin the environment variables in the policy. Workloads are generally separated by the command and arguments passed to them, not by their environment entries. The more a policy has to specify, the harder the system is to use. We will reverse this decision if it turns out to be a source of error.
From secrets to volumes
Fetching application secrets is useful for connecting to databases and third-party services, but the use case we most want to serve is encrypting and decrypting storage.
TEEs only protect data while it is inside the CPU and RAM. Once data touches a real disk, the host can mount the disk and read or write it, which breaks both the confidentiality and integrity guarantees.
c8s now has first-class support for encrypted disks, using dm-crypt for confidentiality and dm-verity for integrity. Encrypted volumes use the same attestation-gated key release as the rest of c8s, and c8s decrypts a volume before the main container starts.
What's next?
First, we're adding support for binding secrets to Kubernetes initContainers. This will allow you to set up the environment for the main container, e.g. by downloading a model or unlocking a volume, without providing credentials to a less trusted main container.
After that, we're adding support for key release for distributed storage. This will enable training at massive scale without compromising on security.