Application secrets

c8s releases application secrets under attestation. A confidential pod asks the CDS (Certificate Distribution Service) for a value — an API token, a database password, a session key — and receives it only if the images actually running in that pod's sandbox match an allowlist entry that grants the path. The value arrives as a file on a memory-backed volume inside the pod. It is never a Kubernetes Secret, never reaches etcd, and never leaves the memory of the TEE (Trusted Execution Environment).

One thing dominates how you write the consumer: the file appears after your container starts. Read The file appears after your container starts before anything else on this page.

Release is gated on a grant

The gate is a secrets grant on the workload entry that describes the pod. The grant is written with the operator key, served by the CDS alongside the rest of the allowlist, and changeable without restarting anything.

An entry without a grant releases nothing. Writing the grant is what turns release on for a workload, and the endpoint is inert until one exists. The grant belongs to the whole entry rather than to a container, because the value is delivered on a volume every container in the pod can read — a per-container grant would not describe what is actually released.

For the grant's shape and how to write it, see The Allowlist.

Request the secret in the pod

Annotate the pod alongside confidential.ai/cw:

apiVersion: v1
kind: Pod
metadata:
  name: api
  annotations:
    confidential.ai/cw: api
    confidential.ai/c8s-secrets: "DB=/tenant-a/db,HF=/tenant-a/hf-token"
spec:
  containers:
    - name: api
      image: example.com/api@sha256:<DIGEST>

confidential.ai/cw is required — the secret annotations without it are rejected at admission, since there is no workload identity to release against.

Each NAME becomes a filename, so it may not contain a path separator and may not be . or .., and two secrets may not share one. Each store path must be absolute, clean, free of a trailing slash, free of percent-encoding, and free of wildcards. A path that is not already canonical is rejected rather than repaired, so the bytes matched against the grant and the bytes used as the store key are the bytes you wrote.

The file appears after your container starts

until [ -f /run/c8s/secrets/DB ]; do sleep 1; done

  t0   c8s-cert        writes the pod's mesh leaf
  t1   c8s-cert-wait   gate clears; main containers may start
  t2   c8s-secret      asks the CDS  ──►  refused, the container set is incomplete
       api             running — /run/c8s/secrets/DB does not exist yet
  t3   (every main container is now running)
       c8s-secret      asks again    ──►  released; DB written atomically
       api             its wait loop returns

The consumer is running before its secret exists. The gap is structural, not a race to be tuned away.

Each value is written to a temporary file and renamed into place, so a poll never reads a torn one, and the whole set is written at once — a consumer never gets some of its secrets and waits forever for the rest.

Two consequences follow, and neither has a workaround:

• An init container cannot close the gap. It would be asking before its siblings exist, and would deadlock the pod it gates. The fetcher is a native sidecar for exactly this reason.
• There is no fail-closed delivery gate. A terminal fetch failure leaves a Running pod with no secret and an Init:CrashLoopBackOff sub-status — the fetcher lives in initContainers, so that is where a crash surfaces. It retries 60 times at 5-second intervals before it exits, and the kubelet restarts it into backoff, so a pod that is only waiting on a slow start recovers on its own. A pod refused for a policy reason retries forever without progress; see Diagnosing a refusal.

What the webhook injects

Setting confidential.ai/c8s-secrets makes the pod webhook add two things:

• c8s-secret, the fetcher sidecar. It runs the c8s image, authenticates to the CDS with the pod's mesh leaf, and is ordered after the c8s-cert-wait gate so that leaf is already on disk.
• c8s-secrets, a memory-backed emptyDir mounted read-only into every container in the pod at the secret directory. The fetcher's own mount is the only writable one, so a compromised workload container cannot replace a value another container has yet to read.

The container name and the volume name are both reserved: a pod that declares its own c8s-secret container, or declares c8s-secrets as anything but a memory-backed emptyDir, is rejected at admission — a hostPath there would write a released secret to host-visible storage. The full set of reserved-name rules, including what happens to a pre-declared mount and to kubectl debug, is on Kata Containers.

Files are written mode 0640 onto a volume group-owned by the pod's fsGroup — 65532 unless the pod sets its own — and every container in the pod carries that group, so a consumer running as any UID can read them.

The fetcher pins the CDS to the launch measurements the install configured (cds.measurements, which the operator forwards to every injected sidecar). With no measurements pinned, an impostor CDS can answer with a value of its choosing — the same warning that applies to the rest of the mesh.

The in-pod destination is not an authorization boundary. The workload owns its own filesystem once the value is inside it; only the store path is policy.

Where the value comes from

A workload that finds its path empty creates it. The fetcher reads with GET, and on a path the store does not hold yet it POSTs, which makes the CDS mint 32 random bytes and store them. The first pod of a workload to ask is the one that defines the value. A replica that loses the race gets a 409 with no body — returning the value there would turn a write grant into a read grant — and re-reads with GET.

A POST never carries a value, so no caller chooses what another caller will later read. That covers a session key. It does not cover an API token, a database password, or a wrapped volume key: those come from an operator.

Operator-supplied values

c8s secrets put /tenant-a/hf-token \
  --url "https://<CDS_HOST>:<PORT>" \
  --measurements <SHA384_LAUNCH_DIGEST> \
  --operator-key operator.key < token.txt

<CDS_HOST>:<PORT> is the CDS or the CDS-issued-TLS tls-lb endpoint; <SHA384_LAUNCH_DIGEST> is the launch measurement that endpoint must present.

The value is read from stdin or --from-file and stored exactly as read — a trailing newline is part of the value — and the byte count is printed so you can confirm which bytes were sent. --dry-run prints the intended change without calling the CDS.

Writes are authorized by the operator key the CDS already pins for allowlist mutations (--operator-keys), which is the same key the grants themselves are rooted in. The signed token binds the method, the path, and the body, so a captured one cannot be replayed against a different path or a different value.

Replacing a value

A path that already holds a value is refused, and the CLI names what put it there — a workload-generated value or an earlier operator write. --overwrite replaces it and prints what it is replacing before it does:

~ /tenant-a/db (replaces a workload-generated value)
wrote 24 bytes to /tenant-a/db

The store has no versioning and no delete, so a displaced value is gone.

A workload reads its secret into a file once, at startup. A replacement therefore reaches a pod only when that pod next restarts: a Deployment holding the old value keeps it until you roll it. Replacing a path a workload created is worth pausing over for that reason — the pods that generated the value go on using it.

What the CDS checks before it releases

Every request carries a single-use challenge and a fresh sandbox token, so a release is bound to one caller and cannot be replayed. The CDS then decides in this order:

1. The client certificate chains to the mesh CA. It is the pod's mesh leaf, verified by TLS itself; the CDS-stamped sandbox ID on it is the caller's identity.
2. The sandbox token verifies against the node's admission inventory, carries this request's challenge, and names the same sandbox as the leaf.
3. The inventory asked is the one the CDS bound to that sandbox at certificate issuance — first-write-wins — not the one the request names. A conflicting binding leaves the sandbox unusable for secrets without invalidating its certificate.
4. That inventory reports what the sandbox has run, minus the containers c8s injects.
5. The remaining set matches exactly one workload entry.
6. That entry's grant covers the requested path.

Any failure refuses. So does an unreachable inventory, an unknown sandbox, an empty container set, an entry with no grant, and an ambiguous match — two entries that no running set can tell apart refuse both, which is why c8s allowlist errors on them at write time.

Denials are opaque on the wire. An ungranted path answers 404, indistinguishable from a path that does not exist, so the API cannot be used to enumerate the store; the reason goes to the CDS log. The workload routes are rate-limited per sandbox, keyed on the ID in the verified client certificate, because every pod on a node reaches the CDS from the same address — bounded on the address instead, one pod could spend a budget its co-tenants share.

What the inventory reports is a high-water mark: every container ever admitted in the sandbox, not those running right now. A pod that ever ran an image outside its entry never receives a secret, even after that image is gone. Attaching a kubectl debug container to a pod joins that record permanently, so a pod that has not yet received its secret will not receive it afterwards.

Diagnosing a refusal

A refused pod is told only that it was refused, and the input that decides the matter — what the sandbox is running — is visible only to the CDS. c8s secrets explain is where that is read:

c8s secrets explain --sandbox <SANDBOX_ID> \
  --url "https://<CDS_HOST>:<PORT>" \
  --measurements <SHA384_LAUNCH_DIGEST> \
  --operator-key operator.key

sandbox    0123456789abcdef…
inventory  10.0.0.7
reported   3 container(s)
dropped    1 injected by c8s
candidates 2
    sha256:1111…  [/serve]
  - sha256:9999…  [get-secret]
    sha256:8888…  [sh -c sleep 1]

vllm-llama  NEAR MISS
  foreign  sha256:8888…  [sh -c sleep 1]
           no container in this entry declares it

nothing is released: no entry describes the candidate set

The report is laid out in the order the CDS decides, so the first thing that goes wrong is the first thing you read. It uses the same inventory, binding, and allowlist the release path uses, and measures entries with the same matcher — it reports the decision rather than a reconstruction of it.

<SANDBOX_ID> is on the pod's certificate; c8s verify prints it (see Verify a workload). --json emits the report as it arrives. It answers to the operator key, since it describes a pod the caller may not own. The report carries grant paths; a value never appears in it.

When the CDS will not serve /secrets at all

Answering at all needs what sandbox identity already needs. Miss any of these and the CDS logs a warning naming the one it is missing, and does not serve /secrets:

It also refuses to serve when CA handoff is configured (--handoff-peer-url or --handoff-measurements). A handoff roll puts two CDS pods behind the Service at once and the surge replica serves an empty store, so a workload landing on it would mint a value diverging from the one its siblings already hold, with no error anywhere.

On a cluster in either state, a pod annotated confidential.ai/c8s-secrets starts, never gets its file, and blocks. Check the CDS log before assuming the grant is wrong.

Limits

All three are fail-closed: past the bound a write is refused, nothing is evicted to make room. The CDS is a single in-memory process holding the mesh CA, so a workload able to grow either map without limit could take it down and every certificate in the cluster with it. The chart renders no overrides for these, so a Helm install runs at the defaults; see the CLI reference.

A CDS restart clears the store

So after a CDS restart, roll every secret-consuming Deployment rather than letting pods recover piecemeal, and re-put every operator-supplied value. Treat a released value as ephemeral for the lifetime of the CDS process: nothing durable may be keyed on one.

The sandbox ledger is process memory too, so a leaf that outlives a restart has no inventory binding until its next renewal and is refused meanwhile.

Not available under Pod-as-CVM

Secrets are out of scope for Pod-as-CVM, and the webhook rejects confidential.ai/c8s-secrets at admission there rather than admitting a pod that would hang. c8s install --cvm-mode=pod refuses --measurements — it pins the node CVM's launch measurement, which the per-pod guests do not have — so the measurement requirement above is unmeetable there, and the fetcher's node-side inventory socket does not exist inside a guest either. See Kata Containers.