How c8s Secures the Network

A confidential VM (CVM) runs a workload in memory that the hardware encrypts, and attestation lets a remote party verify exactly what booted into it. That guarantee ends at the edge of one CVM, and a CVM tops out at one physical machine. Frontier inference runs across a cluster of machines. c8s (Confidential Kubernetes) is our solution to confidential computing at cluster scale. It runs each Kubernetes node as a CVM and draws one trust boundary around all of them. A client can verify that boundary from outside before it sends any data. c8s treats the host, the hypervisor, the network and the Kubernetes control plane as untrusted.

One of the biggest problems in making that work is the network inside the cluster. The parts of a service talk to each other across CVMs, so every connection between them has to be encrypted, and each end has to prove through attestation that it runs admitted code inside a CVM. A conventional service mesh already encrypts these connections, but its certificate authority issues certificates on the Kubernetes control plane's word, so a certificate says nothing about the code on either end.

An inference service, for example, runs its gateway, frontend, router and workers as separate pods on many CVMs. One CVM can host many pods, so pods and CVMs scale independently. c8s protects the traffic between them with attested identity: a pod gets a certificate only after hardware attestation shows it is running admitted code, and the per-pod mesh lets pods talk to each other only over mutual TLS with those certificates.

A mesh sidecar in every pod

In the per-pod design, which c8s is moving to, every confidential pod runs a c8s sidecar next to its application containers. The sidecar generates the pod's private key inside the CVM and keeps it; the application never sees the key. It intercepts the pod's TCP traffic inside the pod's own network namespace and carries each connection over mutual TLS to the sidecar of the destination pod, using certificates that the Certificate Distribution Service (CDS) issues only after attestation.

Applications need no changes. They keep sending plaintext to the names they already use, and the sidecars add identity and encryption underneath. Pods on the same node also talk sidecar to sidecar, so plaintext never leaves a pod.

The per-pod design: each pod's sidecar holds that pod's key and certificate and terminates TLS for it. Everything outside the CVMs carries only ciphertext and cannot obtain a certificate.

The per-pod design: each pod's sidecar holds that pod's key and certificate and terminates TLS for it. Everything outside the CVMs carries only ciphertext and cannot obtain a certificate.

Attestation before a certificate

When a pod starts, its sidecar asks the CDS for a certificate, which it will use for mutual TLS. Because the CDS issues that certificate only after attestation, presenting it in a handshake shows the peer that the pod passed attestation. The CDS runs in its own CVM, and the sidecar checks the CDS's attestation before trusting it.

Each node CVM also runs an admission component as part of its measured image. It checks every container against the allowlist before the container starts, and it keeps track of which pod each running container belongs to.

The CDS then challenges the pod:

  1. The CDS sends a fresh, single-use challenge.
  2. The sidecar obtains an Intel TDX quote (or an SEV-SNP report on AMD hosts) whose report data binds its public key to that challenge, and a signed token from the node's admission component that names the requesting pod.
  3. The CDS verifies the quote or report against the pinned node image (on TDX, MRTD plus the RTMRs covering the kernel and the root filesystem), the key binding and the challenge. It then checks the pod's container images, arguments and environment against a signed allowlist. Allowlist changes must be signed with one of the operator keys pinned at install, and a deployment that pins no operator key gets an immutable allowlist.
  4. If the pod matches an admitted workload, the CDS signs a leaf certificate for the pod's key. The certificate names the workload and the allowlist version it matched.

The private key never leaves the pod. Hardware attestation covers the node image and the key; the workload name comes from the node's admission component, which is part of that measured image.

Issuance is a challenge and response between the pod's sidecar and the CDS. Renewal repeats it before the current certificate expires.

Issuance is a challenge and response between the pod's sidecar and the CDS. Renewal repeats it before the current certificate expires.

Only attested pods talk to attested pods

In the per-pod design, sidecars connect with TLS 1.3 and mutual authentication. Each side presents its CDS certificate and checks the peer's: it must chain to the CDS CA, be inside its validity window, and name a workload that this pod's policy accepts. An endpoint without a CDS certificate cannot complete the handshake, so it can neither send to nor receive from an attested pod.

Because the check is on the peer's identity, a connection that DNS or a Kubernetes Service points somewhere else fails before any application data leaves the pod. Traffic to destinations outside the mesh is refused unless the workload's policy allows it. The host, network and control plane can delay or drop traffic, but they only ever see ciphertext.

Certificates expire by default

Leaf certificates are short-lived: at most 6 hours for a named workload, a ceiling the CDS enforces. The sidecar renews about every 2 hours, and each renewal repeats the full attestation with a fresh challenge. Expiry is the revocation mechanism: if an image is removed from the allowlist or a node falls below the Trusted Computing Base (TCB) floor, renewal fails and the pod's identity lapses within hours. The admission component also refuses to start new instances of a removed image; stopping instances that are already running is planned. In the per-pod design, sidecars also close connections when the peer's certificate expires, so expiry ends existing sessions as well as new ones.

Following a request

A client first verifies the CDS and the attested front door, then sends its request over TLS bound to that attestation. In the per-pod design, every hop inside the cluster that carries the prompt (front door to gateway, gateway to frontend, frontend to workers) is a sidecar-to-sidecar connection in which both ends present CDS certificates naming their workloads. Inside a node, traffic between GPUs uses NVLink, which GPU confidential computing encrypts. The sidecars carry requests, streamed tokens and control traffic; tensor and KV-cache traffic moves over NVLink and RDMA, so the mesh stays off the GPU data path.

Scope

  • GPU evidence. The CDS verifies the CPU TEE (Trusted Execution Environment) today. Requiring GPU attestation at issuance is on the roadmap; until then, a node CVM refuses to become ready unless every GPU is in CC mode and passes attestation.
  • Tenancy. Pods on one node share the node CVM's kernel, so each node CVM runs a single tenant.

What's next

We first ran NVIDIA Dynamo on c8s inside a single TDX CVM with eight B300 GPUs and GPU confidential computing on. Two things are next on the c8s roadmap:

  1. GPU evidence in every workload certificate. Fold B300 GPU attestation into CDS issuance, through NRAS or a local verifier, so a certificate for a GPU pod proves its GPUs as well as its CPU TEE.
  2. Multi-node results. Run that deployment end to end across several B300 CVMs, measure the mesh's cost on the request path, and publish a reproducible result.

Further reading