Application secrets

C8s releases application secrets under attestation. A confidential pod receives an API token, database password, or session key only when its attested container inventory matches exactly one current allowlist entry that grants the requested store path. The CDS (Certificate Distribution Service) holds the value in memory and delivers it over an authenticated, encrypted channel to a memory-backed volume in the pod. This delivery path does not put the value in a Kubernetes Secret or etcd.

CDS stores secret values in TEE (Trusted Execution Environment) memory, and the fetcher publishes them to memory-backed storage inside the confidential pod. The channel encrypts values in transit between CDS and the pod, including while traffic crosses host-controlled infrastructure. An operator-supplied value can originate in an external file, and an authorized application can copy or transmit a value after reading it. Secret release does not confine those copies.

The file may appear after your container starts. A consumer must wait for every file it needs; see the timing constraint. For manifests, consumer examples, operator writes, and diagnosis, see Use application secrets.

Release is gated on a grant

A secrets grant authorizes read or write operations on store paths. It belongs to the whole workload entry because the released files are shared across the pod's application containers. An entry without a grant releases nothing. Operator-authorized allowlist updates change what subsequent release requests may obtain; they do not erase values already delivered.

An image digest alone is insufficient. Every init and main container in a granted entry must pin its command and arguments with exact or deny, and its mounts with deny or exact. Arbitrary host mounts, including host or GPU sockets, are ineligible. A read-only nodeRootFile rule is eligible only when the node proves the mounted file comes from its measured root. This provenance is evidence about the mount's origin, not a claim supplied by the pod. Environment pinning is not a prerequisite for a grant; any environment constraints in the entry still participate in matching.

The mount policy describes the application's mounts, including its shared secret directory. The private credential volume at /etc/c8s/certs belongs to platform containers and must not be added as an application mount. The CDS checks grant eligibility both when accepting policy and when releasing a value.

See the allowlist data model for the grant shape and Grant secret paths for the procedure.

The file may appear after your container starts

The injected c8s-cert sidecar first publishes the pod's credentials. The c8s-cert-wait gate then lets the remaining containers start. The c8s-secret fetcher asks for secrets and retries while the observed set fails to match. An init container waiting for a secret can block the launches needed to satisfy the entry. Matching checks observed launch properties; it does not prove that every Kubernetes main-container slot has started. An init launch can satisfy a main policy with the same constraints.

The fetcher obtains all requested values before writing any files. It then writes a temporary file and renames it into place for each value. Each file is atomically replaced, so readers do not see a partially written value. Publication of the whole set is not atomic: files become visible separately, and a filesystem failure can leave only part of the set published.

Fetch retries are unlimited by default. The delay starts at five seconds, doubles after each failed attempt, and caps at thirty seconds. The secret and encrypted-volume fetchers share these defaults. A policy denial can therefore keep a fetcher waiting indefinitely. Terminal errors, such as secret_store_full, and file-publication failures make it exit; its native sidecar restart policy then allows Kubernetes to restart it. Application execution is not gated on successful secret delivery.

After successful publication, the fetcher stays idle until shutdown. It does not watch for operator replacements. A new fetcher run rechecks authorization and reads the current values; applications must also account for any values they have already cached. See Diagnosing a refusal.

What the webhook injects

The confidential.ai/c8s-secrets annotation requests a mapping from store paths to local filenames. The webhook adds:

• c8s-secret, a native sidecar that authenticates to the CDS with the pod's private mesh credentials after c8s-cert-wait has completed.
• c8s-secrets, a memory-backed emptyDir shared read-only with the application's init and main containers. Its default mount is /run/c8s/secrets; the fetcher is its only writer.

The fetcher writes to /run/c8s/secrets. The optional confidential.ai/c8s-secret-dir annotation accepts only that path or an empty value. An application's pre-declared mount of the same secret volume can use a different path, which is retained and forced read-only. Reserved names in regular or ephemeral containers are rejected; the webhook rebuilds reserved init containers. A reserved secret volume with a different storage type is rejected. Files use mode 0640, with access through the pod's fsGroup (normally 65532, unless configured otherwise). Workload annotations describes the admission rules.

The shared application-secret directory is distinct from the private platform credentials. The pod's mesh private key, certificates, and issuer record are mounted only into platform containers. Application credential mounts and nonempty credential path or volume overrides are refused. Applications use the injected mesh endpoint without reading its private key; see Private pod mesh credentials.

The fetcher uses the CDS endpoint and attestation pins supplied by the node. Those pins identify both the CDS image and its measured launch configuration. The destination filename inside the pod is not an authorization boundary: grants authorize the CDS store path.

Where the value comes from

The fetcher first reads with GET. A 404 can mean either that reading the path is denied or that no value exists. It then attempts POST, which can create a value only with a write grant. A read-only workload cannot create a missing secret. An authorized creation generates 32 random bytes inside the CDS; the caller supplies no value. If another caller has already created the path, POST returns 409 without the existing value. The fetcher then reads again, which requires a read grant.

Operator-supplied API tokens, passwords, and encrypted-volume key blobs enter through c8s secrets put. The operator's signed authorization binds the write to its method, path, and body. The CLI verifies the CDS against its attestation pins and, by default, checks the served mesh CA against the operator's --mesh-ca bundle before sending the value. The mesh CA identifies the live cluster in addition to the build and configuration identified by the attestation pins. These requests go to the verified external CDS VM endpoint, whose default client port is 8443.

An operator can overwrite an existing value, but the store provides neither version history nor a consumer delete operation. Existing files and application caches keep their old bytes until refetched or reloaded. Removing a grant prevents later authorized fetches; it cannot revoke a copy the application already holds. For a lost encrypted-volume key, recovery needs the original escrowed bytes; generating a new key cannot decrypt the existing volume. See Encrypted volumes.

What the CDS checks before it releases

Each request uses a single-use challenge and a fresh sandbox token. A sandbox is the pod's runtime isolation unit; its identifier ties the request to the inventory recorded by its node. The CDS checks:

1. TLS verified the caller's certificate against the mesh CA, and that certificate carries a sandbox ID.
2. The sandbox token binds this challenge, the caller's certificate key, and the same sandbox.
3. The token is verified using the key of the node's admission inventory service, whose host address was bound to that sandbox during certificate issuance. The request cannot choose a different service. The service's host address must be inside the configured node network ranges.
4. The attested inventory reports the sandbox's admitted containers. Only proven platform mesh and credential roles are excluded from workload matching.
5. The remaining container set matches exactly one entry in the current allowlist, including its launch constraints, and that entry is eligible to hold a grant.
6. The grant permits the requested operation on the store path.

The excluded platform roles come from the node's measured policy matching a finalized launch and its owned injected mounts. A lookalike container name does not establish that provenance. An unreachable inventory, unknown or conflicting sandbox binding, empty workload set, ambiguous match, or missing grant refuses release. Workload requests are rate-limited per sandbox.

Read denials and missing values both return 404, preventing path enumeration through that response. Other authorization failures are also opaque to the workload; the CDS log and the operator's c8s secrets explain report carry the diagnostic context.

The inventory is a high-water mark: it includes every container admitted in the sandbox, including ones that have stopped. Removing a foreign container does not remove its record. An extra kubectl debug container therefore remains part of subsequent release decisions; a sandbox that no longer matches a granted entry is refused. Already delivered values remain in the pod.

When the CDS will not serve /secrets at all

CDS startup verifies its required c8s-cds/v1 configuration byte-for-byte against its own attestation report. That configuration supplies operator public keys, initial workloads, node image policy, and node network ranges. Missing operator authority or invalid network ranges fail startup; default-route ranges are rejected. The /secrets handlers require a node identity pin and a bounded inventory address set. Without those prerequisites, CDS does not register the handlers and logs the reason.

The launch file contains public trust inputs, not private keys or secret values. Its config pin binds the starting authority; supported signed changes can update live policy in replicated mode. See Measured authority.

Limits

CDS bounds the number of stored paths, paths charged to a workload, bytes in a value, and sandbox-to-inventory bindings. These are CDS server options, validated by the server; the CLI reference lists their defaults and constraints.

Capacity failures refuse the operation rather than evicting secret values to make room. Path quotas return 507 with secret_holder_quota or secret_store_full. Replicated writes can also return 503 when unavailable or 507 with state_capacity when the bounded replicated state cannot accept the change. Replicated state holds more than secrets, so its total budget is not a promise of usable secret capacity.

CDS restarts and volatile state

The launch configuration selects a singleton CDS or replicated high-availability (HA) mode. Both keep secrets in volatile memory, but their restart behavior differs.

A singleton restart loses its secrets and sandbox bindings and generates a new mesh CA key. Surviving pod files can still contain old values even though the new CDS store is empty. If a workload with a write grant creates the same path again, it receives a new random value.

In replicated mode, CDS replicas share the mesh CA private key, secrets, sandbox bindings, and policy state over protected peer channels. A restarting replica can recover from surviving cluster state. It may serve only after it has applied the state required by a valid leader-confirmed read lease. Losing quorum eventually stops serving as leases expire; it does not itself erase surviving memory.

Recovery from total state loss must account for operator-supplied values, original volume-key escrow, renewed pod identity, and consistent values across consumers. The operator procedures belong in Use application secrets. A generated secret has no durable recovery guarantee; data encrypted under it needs an independently retained key if it must survive total CDS state loss.