Encrypted volumes

An encrypted volume is data too large to be a secret. It sits as ciphertext on storage the untrusted host reads and writes freely, and it decrypts only inside a Trusted Execution Environment (TEE), only for a workload the allowlist names. Model weights are the case it is built for; a writable volume gives the same workload durable scratch space.

This page covers the two kinds of volume, what happens at mount time, and what each kind does and does not defend against. To build one, see Create an encrypted volume.

How a volume differs from a secret

Every other value c8s protects is RAM-resident and dies with the pod: a leaf private key, a released application secret, a session key. A volume does not. Its ciphertext lives on a block device the host attaches, and the host keeps that device — and any copy of it — for as long as it likes.

The operational consequence: a leaked volume key is retroactive and permanent. A leaked session key forges future connections; a leaked volume key decrypts a copy the adversary already has, including copies taken months ago. Handle volume keys — and the escrow files that hold them — on that basis.

Two kinds of volume

c8s volume create builds one image file. Which of two artifacts it builds is the one choice that matters, and it cannot be changed afterwards:

Choose immutable unless the workload must write. An immutable volume is verified block by block on every read, so a host that flips a bit gets an I/O error rather than a wrong answer. A mutable volume is confidential but not integrity-protected: the host can flip bits or roll the device back to an earlier state, and c8s cannot tell. That is the whole of the trade, and it is the reason the mode is a build-time decision rather than a runtime one.

Both kinds mount nosuid, nodev, and noexec. A workload reads data from a volume; it does not execute binaries out of one.

The immutable stack

 ┌───────────────────────────────────────────────────────────────┐
 │ workload container   read-only files at /run/c8s/volumes/…    │
 ├───────────────────────────────────────────────────────────────┤
 │ erofs                read-only filesystem                     │
 ├───────────────────────────────────────────────────────────────┤
 │ dm-verity            hash tree; root hash comes from the blob │
 ├───────────────────────────────────────────────────────────────┤
 │ dm-crypt             aes-xts-plain64, 512-bit key             │
 ╞═══════════════════════════════════════════════════════════════╡
 │ block device         ciphertext, attached by the host         │
 └───────────────────────────────────────────────────────────────┘

The host sees only the bottom layer, and it is ciphertext it can read, copy, and keep.

The mutable stack

 ┌───────────────────────────────────────────────────────────────┐
 │ workload container   read-write files at /run/c8s/volumes/…   │
 ├───────────────────────────────────────────────────────────────┤
 │ ext4                 read-write filesystem                    │
 ├───────────────────────────────────────────────────────────────┤
 │ dm-crypt             aes-xts-plain64, 512-bit key             │
 ╞═══════════════════════════════════════════════════════════════╡
 │ block device         ciphertext, attached by the host         │
 └───────────────────────────────────────────────────────────────┘

Same encryption, one layer fewer: writes reach the device, and nothing checks what comes back.

The filesystem is built with -e remount-ro, so a detected inconsistency degrades the mount to read-only instead of letting corruption spread. On a volume with no integrity layer, the likely cause of that inconsistency is the host.

There is no LUKS header in either kind. Nothing on the device is parsed as metadata; every parameter needed to open it comes from the key blob. A volume's key is fixed when it is built, so a new key means a new volume.

The hash tree is inside the encryption, for an immutable volume. The host cannot fingerprint the contents from the tree, and the root hash commits to the plaintext rather than to one encryption of it.

Sector size is fixed at 512 bytes and the block size at 4096 — the verity block for an immutable volume, the ext4 block for a mutable one. A mutable image is a whole number of 4096-byte blocks and is at least 16 MiB.

The key blob

The value stored at the secret path. It holds everything needed to open the volume and nothing taken from anywhere else. An immutable volume carries its verity geometry:

{
  "type": "c8s.volume/v1",
  "key": "<base64, 64 bytes>",
  "verity": {
    "root_hash": "<hex, 32 bytes>",
    "salt": "<hex>",
    "data_blocks": 26214400,
    "hash_offset": 107374182400
  }
}

A mutable volume carries the key and the mode:

{
  "type": "c8s.volume/v1",
  "key": "<base64, 64 bytes>",
  "mutable": true
}

key is the XTS key — two AES-256 keys, matching dm-crypt's --key-size 512. The hash algorithm is not a field: it is fixed at SHA-256. hash_offset must equal data_blocks × 4096. The two shapes are exclusive: a blob carrying both mutable and verity is refused, and so is one carrying neither.

The mode lives here, not in a pod annotation and not in the allowlist entry — and the blob travels only over the attested channel, so the host never sees which kind of volume a device holds. The verity root hash rides here for the same reason: it is the integrity anchor, and an anchor the host can edit anchors nothing.

A key blob is stored as an ordinary secret value at an ordinary secret path, and it is released to a pod by exactly the machinery described in Application secrets — RA-TLS to the CDS, a single-use challenge, an inventory-signed sandbox token, and a whole-container-set match against one allowlist entry.

What happens at mount time

 ╔═ TEE BOUNDARY · Node-as-CVM ══════════════════════════════════════════╗
 ║                                                                       ║
 ║   ┌─ pod ──────────────────────────────────────────────────────────┐  ║
 ║   │   c8s-volume sidecar                    workload container     │  ║
 ║   └──────┬──────────────┬────────────────────────────▲─────────────┘  ║
 ║          │ (1) RA-TLS   │ (2) unix socket            │                ║
 ║          │  GET blob    │  POST {name, blob}         │ (4) mount:     ║
 ║          ▼              ▼                            │  ro, or rw     ║
 ║   ┌────────────┐  ┌─────────────────────────────┐    │                ║
 ║   │    CDS     │  │ volumed (node DaemonSet)    │────┘                ║
 ║   │secret store│  │ (3) dm-crypt, +dm-verity    │                     ║
 ║   └────────────┘  └──────────────┬──────────────┘                     ║
 ╚══════════════════════════════════│════════════════════════════════════╝
                                    ▼
                    ┌────────────────────────────────┐
                    │ block device, serial           │  ciphertext at rest;
                    │ c8s-vol-<name>                 │  the host keeps a copy
                    └────────────────────────────────┘

Only the block device sits outside the boundary, and only ciphertext ever reaches it. Under See Where volumed runs.

What decides whether a mount happens, in order:

1. CDS releases the blob to the pod's sandbox — verified mesh leaf, single-use challenge, inventory-signed sandbox token, whole-container-set match against one workload entry, and a grant covering the path.
2. volumed mounts into the calling pod's directory and no other. On a node the pod comes from the caller's cgroup via kernel peer credentials; in a guest it is the one pod that guest holds. Either way the request body carries no field naming it, and the mount target is built from the resolved pod UID.
3. The device opens only if the key is right — and, for an immutable volume, only if the verity root hash matches.

A request naming a volume already open under that pod must present the same key and the same mode; otherwise it is refused. Without that, the volume name — a label in a host-written annotation — would be the credential.

One writer per device

Two read-write mounts of one filesystem corrupt it, and a read-write mount under read-only readers corrupts what those readers see. So volumed refuses a mutable open of a device that already carries any mount, and refuses every open of a device a mutable mount holds. The sidecar gets HTTP 409 and retries.

The case that bites is an ordinary rolling update: the new pod starts while the old one still holds the device, so it takes 409s until the old pod goes away. Run a single replica with a Recreate strategy — see Run one writer at a time.

Many pods may open one immutable device at once. Nothing writes, so there is nothing to serialize.

Two timing rules

The volume appears after the workload starts. Release is gated on the whole container set having been admitted, so get-volume is refused until every main container is running. It retries — 60 attempts, 5 seconds apart, by default — and the mount lands shortly after startup. A consumer must wait for the directory to fill rather than read it at main(). A workload that writes to a mutable volume must wait for the same reason: until the mount lands, the directory is the empty emptyDir underneath it.

The key must already be in the store. Unlike an application secret, where the first pod to ask may define the value, get-volume only ever reads. A pod scheduled before c8s volume create has run retries and then fails.

Where volumed runs

volumed opens devices and serves the endpoint the c8s-volume sidecar posts to. Enable the node DaemonSet with c8s install --volumes. The sidecar reaches it over a Unix socket, and the daemon identifies the caller by kernel peer credentials and its pod cgroup. The block device is opened on that node; a reaper closes mappings when the pod exits.

The node DaemonSet runs privileged, with hostPID and a bidirectional bind of the kubelet directory. That is inherent to opening a device and mounting into another pod's directory, and it makes the node volumed a host-side operator component sitting inside the node TEE boundary. It reaches no API server; everything it touches is node-local. Teardown follows the pod's cgroup, not its kubelet directory — kubelet cannot remove that directory while a volume is mounted under it. Its image is debian-slim rather than distroless, because it needs cryptsetup and veritysetup.

c8s volumed and c8s get-volume are Linux-only subcommands and are absent from a macOS build of the CLI. c8s volume create builds everywhere; c8s volume attach and c8s volume detach build everywhere and run on the node, where the kernel target stack is.

Possession of the blob is the authorization

volumed does not repeat the CDS release decision. It resolves who is calling only to decide where to mount, and checks nothing about what that caller is entitled to. Any caller that presents a well-formed blob has that volume opened into its own directory — and for a mutable volume, "opened" means writable.

This rests on the single-tenant node: every pod on it belongs to the same tenant, so a blob one can obtain is one they are all entitled to.

The blob still only comes from CDS, and only to a pod whose containers match an allowlist entry carrying the grant. A node shared between tenants would need a daemon-side entitlement check that does not exist today.

What this defends

An immutable volume:

Tamper detection is lazy. veritysetup open checks the top of the tree; a modified data block surfaces as an I/O error when that block is read, not at open time.

A mutable volume — the tampering and rollback rows are where it differs:

Treat a mutable volume as confidential storage, not as trusted input. Data that has to be trusted — model weights, a policy file, anything the workload's security depends on — belongs on an immutable volume, where every read is verified.

What it does not

• A mutable volume has no integrity protection. A host bit-flip returns wrong data, a failed mount, or a read-only remount; a rollback is indistinguishable from the current state. Confidentiality is the guarantee it carries.
• Any caller that holds the blob can open a volume. volumed authorizes on possession, not entitlement — see Possession of the blob is the authorization.
• Anyone with pod-create or exec RBAC in the workload's namespace can read a mounted volume. This is a Kubernetes RBAC boundary, not an attested one — the API server decides who holds it, and the control plane is untrusted in every shape.
• Volume integrity is rooted in the operator keys CDS pins, and CDS's arguments are host-supplied. A host that restarts CDS under its own operator key can write a matching grant and blob. This is detection, not prevention: the detection is c8s cds verify --operator-keys, and running it continuously is a precondition for trusting a volume.
• Access patterns are visible. Which sectors are read, and when, leaks structure. AES-XTS is deterministic, so on a mutable volume the host also sees when a sector returns to a value it has held before.
• A volume's shape is fixed at build time. There is no resize, no rekey, and no conversion between the two kinds. Build a new volume and copy the data through a workload that holds both keys.
• Availability. The host can withhold, corrupt, or destroy the device at any time.
• Whatever the workload does with the plaintext once it has it.

See Limitations for the platform-wide gap list.