Kata Containers

When you install c8s with --cvm-mode=pod, every workload pod becomes its own confidential VM (CVM) — AMD SEV-SNP by default, Intel TDX with --hardware-platform=tdx. This is the Pod-as-CVM shape: Kata Containers provides the per-pod micro-VM, and c8s provides the attestation, identity, and policy that make it confidential and verifiable.

What gets installed

c8s install --cvm-mode=pod deploys the Kata runtime stack and registers four RuntimeClasses — the two non-confidential classes plus the declared platform's confidential (CPU, GPU) pair:

• kata-deploy (DaemonSet, digest-pinned) installs the Kata runtime and containerd-shim-kata-v2 on every Linux node. The kata-static payload it copies bundles the TEE-capable QEMU builds and Cloud Hypervisor, the guest kernel, guest images, and OVMF firmware — Kata never uses a distro QEMU. It then restarts containerd (or RKE2) so the runtimes register; running pods survive, because containerd shims outlive the daemon.

• c8s-kata-deploy-image-puller (DaemonSet) oras pulls the kata-guest-base artifact onto each node and writes a config.d drop-in so the confidential shim boots from it. A sibling DaemonSet, c8s-kata-deploy-image-puller-nvidia, does the same for the GPU guest.

• RuntimeClasses:

  RuntimeClass	Hypervisor	Confidential?
  kata-qemu	QEMU micro-VM	No — VM isolation from the host only
  kata-clh	Cloud Hypervisor	No — VM isolation only
  kata-qemu-snp / kata-qemu-tdx	QEMU + SEV-SNP / Intel TDX	Yes — pod memory is encrypted against the host; schedules only on platform-labeled nodes
  kata-qemu-snp-nvidia / kata-qemu-tdx-nvidia	as above + VFIO GPU passthrough	Yes — a confidential VM with a dedicated NVIDIA GPU

A cluster runs one platform's confidential classes: only the pair matching the install's --hardware-platform is rendered, and the webhook promotes confidential.ai/cw pods to its CPU class — mixed-TEE clusters are not supported. Confidential pods schedule only to nodes carrying the matching label (confidential.ai/sev-snp=true or confidential.ai/tdx=true); c8s install --cvm-mode=pod applies it to every kata-targeted node from --hardware-platform, and refuses to proceed if a node still carries the other platform's label. The label is a scheduling aid, not a security boundary — attestation is. The rest of this page describes the SEV-SNP path.

Installing --cvm-mode=pod also disables the host-side ratls-mesh, attestation-api, and nri-image-policy: under Kata their function moves inside the guest image, where the host cannot tamper with it. The chart refuses to render if any of them is left enabled alongside kata.enabled.

How pods are kept inside the boundary

--cvm-mode=pod is enforcing — there is no install-Kata-without-enforcing shape, because a workload that can dodge the CVM boundary makes the stack decorative. Two pieces cooperate.

The pod webhook injects a RuntimeClass into every in-scope workload pod that does not already request one:

A ValidatingAdmissionPolicy (c8s-kata-enforcement, failurePolicy: Fail) then rejects any workload pod naming a runtimeClassName outside the installed set. It is built-in CEL — no webhook server, no TLS — and needs Kubernetes 1.30+, which is the chart's floor anyway.

What enforcement does not touch:

• System namespaces — kube-system, kube-public, kube-node-lease, the c8s release namespace, and anything in webhook.extraExcluded. Injection scope and rejection scope read the same list, so they cannot drift apart.
• Pods that set their own runtimeClassName — honored, and still validated as a Kata class. This path also skips get-cert injection: a pod pinned to kata-qemu-snp without confidential.ai/cw runs as a confidential VM with no c8s workload identity. That is the supported bring-your-own-attestation path. To get both, annotate the pod and let the webhook choose the class.
• Already-running pods — the webhook fires on CREATE only.
• Host-namespace pods — a Kata pod is a VM and cannot join the host's network, PID, or IPC namespace, so a pod setting hostNetwork, hostPID, or hostIPC is exempt from both halves and runs as an ordinary container.

Reserved containers and volumes

For a pod annotated confidential.ai/cw, the webhook injects the get-cert sidecar and a memory-backed emptyDir to hold the leaf key, so the private key never leaves TEE memory. The webhook enforces that at admission:

• The container names c8s-cert, c8s-cert-wait, c8s-secret, and c8s-volume are reserved — a pod may not declare its own container or ephemeral container under them.
• If the pod pre-declares the reserved cert volume (c8s-certs, or whatever confidential.ai/c8s-cert-volume names), it must be an emptyDir with medium: Memory. A hostPath, PVC, or disk-backed emptyDir is rejected — otherwise the sidecar would write keys to persistent, host-visible storage. Omitting it is fine; the webhook injects the right one.
• A container that declares its own mount of a reserved volume keeps its mount path but the mount is forced read-only.
• An ephemeral container (kubectl debug) that mounts a reserved volume is rejected outright. The reserved set is read from spec.initContainers, which is immutable after CREATE, so rewriting the annotation on a live pod does not move the guard.

Application-secret and encrypted-volume injection annotations are rejected under Kata: both fetchers need a node-side socket that does not exist inside a guest.

Measured direct-kernel boot

Kata's confidential path does not use IGVM or a UKI. It uses measured direct-kernel boot with dm-verity:

The dm-verity root hash rides on the kernel cmdline, and Kata folds the resulting hash into the launch measurement — SNP kernel-hashes, or TDX RTMR[1]. So the SNP launch digest commits to OVMF + vmlinuz + the exact rootfs. On SNP the puller also pins default_vcpus and default_maxvcpus to 1, because the boot-time VMSA count is the one genuinely per-VM input to the digest; TDX carries no such pin, since the register c8s verifies (MRTD) does not measure vCPU init.

The digest is not published with the artifact. Predict it with sev-snp-measure from the exact OVMF, kernel, cmdline, and VM shape, then supply it to the policies that consume it — cds.measurements and ratlsMesh.measurements in a values file, and client-side --measurements on verification. All of those default to empty, which means "accept any attested peer".

kata.guestImage.tag is not a measurement pin: it selects which artifact the puller fetches and carries no cryptographic guarantee, and the puller has no digest: field yet. Pin a specific <short-sha> tag rather than main or latest, and pin the predicted launch digest separately. c8s install --cvm-mode=pod refuses --measurements — that flag pins a node CVM's digest, which is a different value from the per-pod guest's.

Any change to the guest rootfs moves the digest. The removal of the in-guest registry-auth mechanism (below) changed those bytes, so reference values predicted against an older kata-guest-base no longer match and must be re-predicted.

The sealed guest image

The kata-guest-base image is the dm-verity-sealed guest rootfs. Because it is part of the launch measurement, everything baked into it is transitively attested:

A peer that verifies an RA-TLS handshake therefore transitively verifies that the attestation service, mesh proxy, and policy monitor are the genuine, sealed versions — the host cannot swap them out.

The baked allowlist is a seed, not the whole story: policy-monitor loads it at boot so the guest enforces from t=0 with no network, then polls CDS's /allowlist over RA-TLS and merges what it serves on top. The merge only ever grows the set, so an unreachable or compromised CDS degrades to "stale but no smaller" — never to "open". Adding an image is therefore an allowlist write, not a guest rebuild.

Three layers of attestation

   LAYER 0                  LAYER 1                  LAYER 2
 ┌───────────────┐        ┌───────────────┐        ┌───────────────┐
 │  Guest image  │        │   Container   │        │   Workload    │
 │               │──────► │     image     │──────► │   identity    │
 │  SNP launch   │        │               │        │               │
 │    digest     │        │  OCI digest · │        │    RA-TLS     │
 │               │        │   allowlist   │        │  certificate  │
 └───────────────┘        └───────────────┘        └───────────────┘

Each layer is verified independently; only the first is baked into the hardware measurement.

1. Guest image — the SNP launch digest over OVMF + vmlinuz + cmdline (embedding the dm-verity root hash). The operator predicts it and pins it.
2. Container image — the OCI image digest, checked against the allowlist. The pod webhook injects get-cert, which fetches a leaf cert from the CDS — the CDS verifies the container measurement and signs in one step.
3. Workload identity — an RA-TLS cert carrying attestation evidence; peers verify it on the mTLS handshake.

The guest-pull model

With shared_fs = "none" and experimental_force_guest_pull = true, the kata-agent's confidential-data-hub pulls the OCI image inside the guest over virtio-net — not from a host bind-mount. The host never sees the unpacked workload rootfs, only encrypted VM memory.

The in-guest pull is anonymous, and there is no supported way to hand the guest a registry credential. The mechanism that used to bake one into the rootfs — ghcr-auth.json, surfaced as kata.guestImage.registryAuth and kata.gpu.guestImage.registryAuth — was removed, because it put pull credentials inside the dm-verity-measured root, where rotating them meant rebuilding and re-attesting the image. Both values are gone from the chart. What that means in practice:

• Workload images must be pullable without authentication from inside the guest. Private in-guest workload pulls are not supported. A private image fails during pod start, inside the VM.
• Host-side pull credentials are still required. The host's CRI does its own image-exists pull, so c8s install --image-pull-secret <NAME> (a kubernetes.io/dockerconfigjson Secret in the release namespace) still applies to every component image.
• kata.guestImage.pullerAuthSecret is a different credential. It authenticates the host-side oras pull of the kata-guest-base artifact, which reads /root/.docker/config.json and ignores kubelet pull secrets. It defaults to the chart-wide imagePullSecret, is not in the TCB or the launch measurement, and rotating it is a Secret update plus a DaemonSet restart. Set it only when the artifact needs a credential different from the c8s images.
• The host still observes the transport. It brokers the guest's outbound network, so it sees which image reference and layers are fetched. That is a metadata leak, not a content-confidentiality break — the bytes are public.
• The image unpacks into guest RAM. Guest-pull writes to a tmpfs, so a workload image has to fit in the pod's guest memory unless the node is provisioned to attach an encrypted scratch disk, which is a host-provisioning step and not wired by the chart.

Also raise kubelet's runtime-request-timeout on Kata clusters. The default is 2 minutes and it caps pod creation; a multi-GB model image guest-pulled inside the VM will hit that wall with the cause hidden. On RKE2, set kubelet-arg: runtime-request-timeout=20m in /etc/rancher/rke2/config.yaml.

Constraints

• x86-64, SEV-SNP or Intel TDX. The confidential pod path needs an AMD SEV-SNP host (Genoa / Milan / Turin class) or an Intel TDX host, declared at install time via --hardware-platform. kata-deploy installs the runtime; it does not enable the TEE — that is a kernel cmdline, firmware, and BIOS matter. Node kernels must be new enough for the TEE: 6.11 or newer for SEV-SNP, 6.16 or newer for TDX. On a host with neither TEE you can run kata-qemu (isolation, not confidentiality) but no confidential class. The chart renders no AArch64 shims, so Kata installs nothing on ARM nodes.
• Kata needs /dev/kvm, /dev/vhost-vsock, and /dev/vhost-net. These auto-load on standard systemd distros; if pods fail with open /dev/vhost-vsock: no such device, run modprobe vhost_vsock vhost_net and persist it via /etc/modules-load.d/.
• A webhook outage blocks all workload pod creation. The pod webhook is failurePolicy: Fail, and the chart refuses to render kata.enabled=true with anything else. With enforcement on, that widens an operator outage from "no get-cert injection" to "no workload pods".
• Installing Kata restarts containerd / RKE2 on every node, and kata-deploy takes 1–2 minutes per node. Pods created in that window are mutated to a Kata RuntimeClass and stay Pending until the runtime is in place — delayed, not lost.
• Confidential GPU is passthrough-based and comes with gaps. GPU pods run as confidential VMs with the NVIDIA GPU passed through over VFIO; there is no separate flag, the GPU stack ships with every --cvm-mode=pod install. The host setup (vfio-pci binding, NVIDIA Confidential Computing (CC mode), BAR resize) is provisioning's job, not the NVIDIA GPU Operator's — the operator assumes host-visible GPUs with a host driver and must not be installed on these clusters. A locked guest refuses to boot on a non-CC GPU, but the GPU's own attestation (SPDM) is not yet surfaced to the relying party, and SEV-SNP GPU pods inherit the single-vCPU pin. See Limitations.
• Per-pod, not per-node. Pod-as-CVM attests each pod; it does not by itself attest the node. If you need a single confidential node instead, see Pod-as-CVM vs Node-as-CVM.
• Not available on Azure. Per-pod CVMs need nested virtualization, which Azure does not expose; use Node-as-CVM there — see Provisioning on Azure.