Encrypted volumes

An encrypted volume is data too large to be a secret. It sits as ciphertext on storage the untrusted host reads and writes freely, and it decrypts only inside a Trusted Execution Environment (TEE), only for a workload the allowlist names. Model weights are the case it is built for.

This page covers the artifact, building one with c8s volume create, attaching it to a node, granting a workload access, mounting it into a pod — and what the design does and does not defend against.

How a volume differs from a secret

Every other value c8s protects is RAM-resident and dies with the pod: a leaf private key, a released application secret, a session key. A volume does not. Its ciphertext lives on a block device the host attaches, and the host keeps that device — and any copy of it — for as long as it likes.

The operational consequence: a leaked volume key is retroactive and permanent. A leaked session key forges future connections; a leaked volume key decrypts a copy the adversary already has, including copies taken months ago. Handle volume keys — and the escrow files that hold them — on that basis.

The artifact

c8s volume create packages a directory into a single image file, in three layers:

 ┌──────────────────────────────────────────────────────────────┐
 │ workload container   read-only files at /run/c8s/volumes/…    │
 ├──────────────────────────────────────────────────────────────┤
 │ erofs                read-only filesystem                     │
 ├──────────────────────────────────────────────────────────────┤
 │ dm-verity            hash tree; root hash comes from the blob │
 ├──────────────────────────────────────────────────────────────┤
 │ dm-crypt             aes-xts-plain64, 512-bit key             │
 ╞══════════════════════════════════════════════════════════════╡
 │ block device         ciphertext, attached by the host         │
 └──────────────────────────────────────────────────────────────┘

The host sees only the bottom layer, and it is ciphertext it can read, copy, and keep.

There is no LUKS header. Nothing on the device is parsed as metadata; every parameter needed to open it comes from the key blob. There is also no keyslot, so changing a volume's key means building a new volume, not rekeying this one.

The hash tree is inside the encryption. The host cannot fingerprint a volume's contents from the tree, and the root hash commits to the plaintext rather than to one encryption of it.

Sector size is fixed at 512 bytes and the verity block size at 4096. Neither is configurable.

The key blob

The value stored at the secret path. It holds everything needed to open the volume and nothing taken from anywhere else:

{
  "type": "c8s.volume/v1",
  "key": "<base64, 64 bytes>",
  "verity": {
    "root_hash": "<hex, 32 bytes>",
    "salt": "<hex>",
    "data_blocks": 26214400,
    "hash_offset": 107374182400
  }
}

key is the XTS key — two AES-256 keys, matching dm-crypt's --key-size 512. The hash algorithm is not a field: it is fixed at SHA-256. hash_offset must equal data_blocks × 4096, and a document carrying any field not listed above is rejected rather than parsed with the extra dropped.

The verity root hash rides in the blob, not in a pod annotation and not in the allowlist entry. It is the integrity anchor, and it only ever travels over the attested channel.

A key blob is stored as an ordinary secret value at an ordinary secret path, and it is released to a pod by exactly the machinery described in Application secrets — RA-TLS to the CDS, a single-use challenge, an inventory-signed sandbox token, and a whole-container-set match against one allowlist entry.

Building and mounting a volume

c8s volume create \
  --name weights \
  --source ./llama-3.1-8b \
  --out ./weights.img \
  --path /tenant-a/volumes/weights \
  --escrow-out ./weights.escrow.json \
  --node node-1 \
  --url https://cds.example \
  --measurements-file ./measurements.txt \
  --operator-key ./operator.key

+ ./weights.img (26214400 data blocks)
+ key stored at /tenant-a/volumes/weights
+ key escrowed to ./weights.escrow.json — keep it; a CDS restart needs it

Attach ./weights.img to the node as a raw block device with serial c8s-vol-weights.

Pod annotations:
  confidential.ai/cw: <workload-id>
  confidential.ai/c8s-volumes: "weights=/tenant-a/volumes/weights"

Pod nodeSelector (the device is on one node):
  kubernetes.io/hostname: node-1

Allowlist grant for the workload entry (read-only, exact path):
  "secrets": {"policy": "allow", "read": ["/tenant-a/volumes/weights"]}

A subtree grant would cover every volume beneath it, so this names one path.

c8s secrets put /tenant-a/volumes/weights \
  --from-file ./weights.escrow.json \
  --url https://cds.example \
  --operator-key ./operator.key

"secrets": { "policy": "allow", "read": ["/tenant-a/volumes/weights"] }

metadata:
  annotations:
    confidential.ai/cw: llama-infer
    confidential.ai/c8s-volumes: "weights=/tenant-a/volumes/weights"
    confidential.ai/c8s-volume-dir: "/models"    # optional

kubectl exec -n <NAMESPACE> <POD> -- ls /models/weights

What happens at mount time

 ╔═ TEE BOUNDARY · node-as-CVM ══════════════════════════════════════════╗
 ║                                                                       ║
 ║   ┌─ pod ──────────────────────────────────────────────────────────┐  ║
 ║   │   c8s-volume sidecar                    workload container     │  ║
 ║   └──────┬──────────────┬────────────────────────────▲─────────────┘  ║
 ║          │ (1) RA-TLS   │ (2) unix socket            │                ║
 ║          │  GET blob    │  POST {name, blob}         │ (4) read-only  ║
 ║          ▼              ▼                            │     mount      ║
 ║   ┌────────────┐  ┌─────────────────────────────┐    │                ║
 ║   │    CDS     │  │ volumed (node DaemonSet)    │────┘                ║
 ║   │secret store│  │ (3) dm-crypt + dm-verity    │                     ║
 ║   └────────────┘  └──────────────┬──────────────┘                     ║
 ╚══════════════════════════════════│════════════════════════════════════╝
                                    ▼
                    ┌────────────────────────────────┐
                    │ block device, serial           │  ciphertext at rest;
                    │ c8s-vol-<name>                 │  the host keeps a copy
                    └────────────────────────────────┘

Only the block device sits outside the boundary, and only ciphertext ever reaches it.

What decides whether a mount happens, in order:

1. CDS releases the blob to the pod's sandbox — verified mesh leaf, single-use challenge, inventory-signed sandbox token, whole-container-set match against one workload entry, and a grant covering the path.
2. volumed mounts into the calling pod's directory and no other. The pod comes from the caller's cgroup via kernel peer credentials; the request body carries no field naming it, and the mount target is built from the resolved pod UID.
3. The device opens only if the key is right and the verity root hash matches.

A request naming a volume already open under that pod must present the same key and root hash; otherwise it is refused. Without that, the volume name — a label in a host-written annotation — would be the credential.

Two timing rules

The volume appears after the workload starts. Release is gated on the whole container set having been admitted, so get-volume is refused until every main container is running. It retries — 60 attempts, 5 seconds apart, by default — and the mount lands shortly after startup. A consumer must wait for the directory to fill rather than read it at main().

The key must already be in the store. Unlike an application secret, where the first pod to ask may define the value, get-volume only ever reads. A pod scheduled before c8s volume create has run retries and then fails.

volumed, the node agent

volumed is the privileged node DaemonSet that opens devices and serves the socket the sidecar posts to. It is listed with the rest of the platform in Components.

volumed:
  enabled: true
  maxMounts: 64        # live volumes per node; each costs two dm devices and a mount
  reapInterval: 15s    # how often teardown checks which pods have gone
  nodeSelector: {}     # confine it to the nodes that carry volume devices

Its image is ghcr.io/confidential-dot-ai/volumed — debian-slim rather than distroless, because it needs cryptsetup and veritysetup.

It runs privileged, with hostPID and a bidirectional bind of the kubelet directory. That is inherent to opening a device and mounting into another pod's directory, and it makes volumed a host-side operator component sitting outside the guest TEE boundary. It reaches no API server; everything it touches is node-local. Teardown follows the pod's cgroup, not its kubelet directory — kubelet cannot remove that directory while a volume is mounted under it.

c8s volumed and c8s get-volume are Linux-only subcommands and are absent from a macOS build of the CLI. c8s volume create — the only one an operator runs by hand — builds everywhere.

Volumes require Node-as-CVM

The webhook rejects confidential.ai/c8s-volumes at admission when the operator has no --workload-claims-host-dir — that is, under Pod-as-CVM, or with nri-image-policy disabled. The fetcher hands the key to a node agent over the inventory's socket directory and the agent mounts into the pod's kubelet directory; neither exists inside a per-pod guest. The pod is refused rather than left waiting on a mount that can never land. See Kata containers.

Possession of the blob is the authorization

volumed does not repeat the CDS release decision. It resolves who is calling only to decide where to mount, and checks nothing about what that caller is entitled to. Any pod on the node that presents a well-formed blob has that volume opened into its own directory.

This rests on Node-as-CVM being single-tenant: every pod on the node belongs to the same tenant, so a blob one of them can obtain is one they are all entitled to. Under Pod-as-CVM, volumes are refused at admission, so the case does not arise there.

The blob still only comes from CDS, and only to a pod whose containers match an allowlist entry carrying the grant. But a node shared between tenants, or a Pod-as-CVM path for volumes, would need a daemon-side entitlement check that does not exist today.

What this defends

Tamper detection is lazy. veritysetup open checks the top of the tree; a modified data block surfaces as an I/O error when that block is read, not at open time.

What it does not

• Any pod on the node can open a volume whose blob it holds. volumed authorizes on possession, not entitlement — see Possession of the blob is the authorization.
• Anyone with pod-create or exec RBAC in the workload's namespace can read a mounted volume. Under --cvm-mode=node the control plane runs inside the node CVM, so this is not a capability the host has — but it is a Kubernetes RBAC boundary, not an attested one.
• Volume integrity is rooted in the operator keys CDS pins, and CDS's arguments are host-supplied. A host that restarts CDS under its own operator key can write a matching grant and blob. This is detection, not prevention: the detection is c8s cds verify --operator-keys, and running it continuously is a precondition for trusting a volume.
• Access patterns are visible. Which sectors are read, and when, leaks structure.
• Availability. The host can withhold, corrupt, or destroy the device at any time.
• Whatever the workload does with the plaintext once it has it.

See Limitations for the platform-wide gap list.