Encrypted volumes

An encrypted volume keeps persistent data, such as model weights, as ciphertext on storage controlled by the host. C8s releases its key through the attested secrets flow. The node's volumed daemon opens and mounts the volume inside the confidential node's TEE (Trusted Execution Environment). An immutable volume verifies its contents; a mutable volume supports writes without authenticating them.

The operator's provisioning machine and key escrow are also trusted: the key originates there, outside the node and CDS TEEs. This page explains those boundaries and the volume lifecycle. For image creation, attachment, grants, and pod configuration, see Create an encrypted volume.

How a volume differs from a secret

A volume's ciphertext survives its pod and can be copied indefinitely by the host. Its key is stored as an application secret, but losing the live secret store does not erase the encrypted device. Opening that device again requires the original key.

c8s volume create generates the key on the operator's machine, builds the image, writes a key escrow file, and then stores the key blob in the CDS (Certificate Distribution Service). Escrow is a recoverable copy of the complete key blob. The build also handles the plaintext source and intermediate filesystem images. Protect the provisioning machine, its working storage, and the escrow independently of the cluster.

A leaked volume key decrypts retained copies of that volume, including copies made before the leak. Removing a grant or deleting the pod cannot revoke a key already copied. Replacing the secret-store value with a new random key does not re-encrypt the existing data.

Two kinds of volume

c8s volume create chooses the volume's format at build time:

Choose immutable for data the workload must trust. Its hash tree authenticates the filesystem against a root hash delivered with the key. A mutable volume provides confidentiality, but host tampering and rollback can affect the data the application consumes. Use it only where the application can tolerate or independently validate those changes.

Both kinds use a 64-byte XTS key, comprising two AES-256 keys, and 512-byte encryption sectors. The verity blocks and ext4 filesystem blocks are 4096 bytes. Mutable images are aligned to 4096 bytes and are at least 16 MiB.

Both mounts carry nosuid, nodev, and noexec. These restrict filesystem operations; an application can still read a file and interpret its contents as code.

The immutable stack

Inside the node TEE, the workload reads an erofs filesystem through dm-verity, which reads through dm-crypt. The host-backed block device beneath that stack holds ciphertext. Both the filesystem and its hash tree are encrypted; the root hash in the key blob commits to the plaintext contents.

Verification happens as blocks are read. A successful mount is not a full scan of the image: corruption in an unread block can surface later as an I/O error. The root hash authenticates content, not its age. Restoring bytes that match the same root hash is accepted.

The mutable stack

A mutable volume mounts ext4 directly over dm-crypt. Writes reach the encrypted device, and there is no dm-verity layer. The filesystem is built with an error policy of remount-ro, which can turn detected filesystem errors into a read-only mount. Filesystem consistency checks do not authenticate data or detect every host alteration.

Every sector is encrypted at creation, including free space. A large requested size therefore requires writing a correspondingly large ciphertext image; it is not a sparse allocation of only the initial files.

The key blob

The c8s.volume/v1 blob contains the base64-encoded key and the volume mode. An immutable blob also carries the verity root hash, salt, data-block count, and hash-tree offset. A mutable blob carries mutable: true and no verity geometry. The daemon rejects a blob that combines mutable mode with verity geometry, or supplies neither mode's required information.

The encryption parameters are fixed by the implementation, and verity geometry comes from the blob. The device has no LUKS header, and the verity opener does not read an on-device verity superblock. The filesystem still parses its own metadata. Encryption also does not hide the volume's size, access patterns, or every indication of its mode.

CDS stores the blob at an ordinary secret path. A read grant on that path authorizes release of the whole blob, including a mutable volume's write capability. A secret-store write grant governs creation of secret values; it is not required to write files on a mutable volume.

What happens at mount time

Trusted operator provisioning machine
  plaintext source + generated key + retained escrow
             |
             | operator-authorized write over verified encrypted transport
             v
+---------------- CDS VM TEE ----------------+
| Measured CDS configuration + secret store  |
+-------------------------------------------+
             |
             | attested, encrypted key release
             v
+------------- confidential node TEE ----------------+
| c8s-volume sidecar -- Unix socket --> volumed       |
|                                      |             |
|                              decrypted mount       |
|                                      v             |
|                               workload container   |
+--------------------------------------|-------------+
                                       |
                              encrypted block I/O
                                       |
                         host-controlled block device

The CDS runs in an external confidential VM, separate from the workload node. The sidecar verifies the CDS's image and measured configuration through its supplied attestation pins. See Trust root for those trust inputs.

Mounting involves distinct checks:

1. CDS authorizes release. The volume sidecar presents its pod's identity certificate and signed node evidence tying this fresh request to that pod. CDS verifies the node runtime's recorded container launch details against exactly one current allowlist entry with an eligible read grant covering the key's store path.
2. volumed resolves the destination. The sidecar supplies a volume name and key blob over a Unix socket. Kernel peer credentials and the caller's cgroup identify its pod. The daemon resolves a mount target beneath that pod's kubelet directory, rejecting symlinks and path escapes; the request cannot name another pod.
3. The daemon opens the stack. It selects the device by its c8s-vol-<name> serial, validates the blob, opens dm-crypt and any verity layer, and mounts the filesystem. The serial selects a device; it does not authenticate its contents. Duplicate matching serials are refused.

A repeat request for an already open volume under the same pod must present the same key, mode, and immutable root hash. A volume name alone cannot obtain an existing mapping. The default application path is /mnt/c8s-data/c8s-volumes/<name>.

Grant eligibility requires pinned commands, arguments, and mounts, as well as matching image digests. Each volume destination is covered by a data mount rule. Arbitrary host or GPU socket mounts are ineligible; a read-only nodeRootFile rule requires measured-root provenance. Private platform credentials are mounted only into platform containers. See Release is gated on a grant for the shared authorization contract.

One writer per device

volumed checks its tracked mounts and pending opens for the selected device. It rejects a conflict when either the requested open or the tracked use is mutable. The sidecar receives HTTP 409 and retries. Multiple immutable readers are allowed.

These conflict checks use one node's daemon state. They do not fence the same backing storage presented on another node, under another device identity, or through another opener. Storage presentation and scheduling must prevent concurrent writers across those boundaries.

A rolling update can leave the new pod waiting while the old pod holds the mutable mount. A single replica with Recreate avoids that overlap within the workload; it does not establish storage fencing. See Run one writer at a time.

Two timing rules

Consumers must wait for volume availability. CDS releases the key only after verifying that recorded launches admitted by the node runtime match exactly one current allowlist entry with an eligible read grant. Those records retain launches after containers stop; they do not report application readiness. The volume fetcher retries while the release condition is incomplete. A consumer must wait for each required volume before reading or writing it. Directory existence is insufficient: until the mount arrives, the application sees the underlying emptyDir. Writes there do not become writes to the encrypted device. Multiple requested volumes can become available separately.

Retries are unlimited by default, starting five seconds apart and doubling to a thirty-second cap. A policy denial can therefore leave the fetcher waiting indefinitely. Terminal errors stop a run, and the native sidecar restart policy permits another run. After opening all volumes successfully, the sidecar idles until termination; it does not watch the store for key replacements or grant revocation.

The original key must be in the store. get-volume only reads; it cannot generate a replacement for a missing key. A path-level read denial and a missing value both return 404. Earlier identity or workload authorization failures can return 403; invalid challenges return 400. Restoring an existing volume means restoring its original escrowed blob through the operator secrets write path. See Application secrets for store recovery and release diagnosis.

Where volumed runs

volumed runs inside the confidential node's trust boundary and operates on that node's kernel, block devices, and kubelet directories. On the prebuilt C8s node image, called the baked node image, it runs as volumed.service within the measured image; the chart refuses a second DaemonSet there. Other supported installations also require the node to be a confidential VM and can enable the DaemonSet with c8s install --volumes.

The DaemonSet is privileged, uses the host PID namespace, and mounts the kubelet directory with bidirectional mount propagation. These privileges make the node kernel and daemon part of the trusted computing base. Running the daemon on an ordinary untrusted host does not put the decrypted volume inside a pod TEE. It uses node-local interfaces rather than the Kubernetes API to resolve callers and manage mounts.

c8s volumed and c8s get-volume require Linux. Image creation also requires its external filesystem tools: mkfs.erofs and veritysetup for immutable images, or mkfs.ext4 for mutable ones. The c8s volume attach helper requires Linux LIO target support and is unsupported on the baked C8s node image. Attach ciphertext as a raw block device using a method supported by the node's VM environment; the creation guide owns the procedures.

Cleanup follows the pod

On termination, the volume sidecar asks the daemon to close its pod's volumes before kubelet cleans up the underlying directories. The daemon unmounts the filesystem and closes verity, when present, then dm-crypt. Closing the crypt mapping releases the kernel mapping holding the key and backing device.

A reaper provides fallback cleanup when the sidecar cannot complete that request. It checks whether the pod's cgroup is missing or has no processes, requiring two confirming sweeps by default. An unknown liveness result leaves the mounts open. Cleanup follows cgroup state, because a mounted volume can itself prevent kubelet from removing the pod directory. If teardown of a tracked mount cannot close dm-crypt, the daemon retains its record for later retries.

At startup the daemon attempts to close leftover C8s mappings. Mappings still held by live mounts can remain busy, and this sweep does not reconstruct the previous daemon's mount records. The chart's uninstall hook also attempts teardown and reports failure when mappings remain open. Neither a process restart nor an uninstall request proves that every key-bearing mapping has closed.

Teardown preserves the backing ciphertext and escrow. CDS state has a separate lifecycle: a singleton restart loses its volatile store, while an HA replica can recover from surviving replicas. Loss of all volatile CDS state requires restoration of the original escrow for future key releases. Existing mounts are not automatically revoked by CDS state loss.

Possession of the blob is the authorization

CDS checks entitlement when releasing a blob. volumed checks the caller's pod destination, blob format, and local mount constraints; it does not repeat the CDS grant decision or verify where the blob came from. A reachable caller with a usable blob and valid target can request its volume, including writable access for a mutable blob. Escrow is another source of that capability.

The node deployment assumes a single tenant. Sharing a node between tenants would require a daemon-side entitlement boundary beyond these checks. Even on a single-tenant node, releasing a key to one workload does not mean every pod is entitled to it: protection depends on keeping the blob within its authorized provisioning and workload paths. Removing a grant prevents subsequent authorized release; it does not close an existing mount or invalidate copied keys.

What this defends

The following assumes trusted provisioning and escrow, correct CDS image/configuration pins, and a protected node runtime:

Plain dm-crypt does not authenticate a key. An unrelated wrong key normally produces an unmountable filesystem, but a mutable filesystem mount is not a cryptographic validity check. An immutable volume's security depends on the trusted root hash, including verification of blocks read after mounting.

What it does not

• Freshness or mutable integrity. A root hash identifies immutable contents, not the latest authorized version. Mutable data needs application-level validation if its authenticity or freshness matters.
• Protection from authorized plaintext use. A workload can copy, transmit, or interpret data it can read. Runtime and administrative access also matter: locked baked nodes refuse arbitrary runtime exec, while debug images and managed installations have different controls. Kubernetes permissions alone do not describe the complete boundary.
• Protection from trusted authority compromise. Operator authority can change grants and stored blobs. CDS verifies its required launch configuration against its attestation report, and clients must pin both the expected image and configuration. Supported signed HA changes can update live policy; a launch pin does not freeze that policy forever.
• Availability or automatic storage evolution. The host can withhold or destroy storage. Resize, rekey, and conversion between formats require building a new volume and moving data through a trusted consumer with the necessary keys.

See Limitations for the platform-wide boundaries.