Troubleshoot an install

Find the symptom, not the subsystem. Each entry names what you would see, what causes it, and what to do.

kubectl get pods -A -w

The install

c8s install sits for minutes with no output

Expected. helm --wait blocks until every component reports Ready. The wait is bounded: 5 minutes, or 10 minutes for --cvm-mode=pod. Watch progress from a second shell with kubectl get pods -A -w — kata-deploy and the CNI live outside c8s-system.

c8s install fails once the timeout elapses

The resources it already applied stay in place. Fix the cause and re-run it — the install is idempotent (helm upgrade --install) and completes once every pod is Ready.

On a node with no prior Kata install, kata-deploy downloads a multi-GB payload inside the wait window. A slow registry can still blow the 10 minutes: the release lands as failed while the cluster converges, and a second run flips it to deployed.

crane digest ...: UNAUTHORIZED: authentication required

Digest resolution runs crane locally during the install, reading your ~/.docker/config.json. Log in on the box:

crane auth login <REGISTRY> -u <USER> --password-stdin

This is a separate credential from --image-pull-secret, which is the cluster-side kubelet credential. An authenticated registry needs both. Or skip resolution with --resolve-digests=false and supply digests via -f.

The install refuses to proceed without --operator-keys

Deliberate: installing without it disables allowlist writes, so nobody can allowlist a workload image. Create the credential (see Manage the allowlist) or acknowledge with --force. Supplying any -f values file also clears the guard — those files own cds.operatorKeys.

A TDX install fails fast at the node check

TDX RuntimeClasses schedule on nodes labeled confidential.ai/tdx=true. The default --cvm-mode=pod path applies the label itself; a -f install or a non-pod shape does not.

kubectl label node <NODE> confidential.ai/tdx=true

c8s uninstall sweeps these labels along with the Kata artifacts, so a reinstall after an uninstall can hit this even though the first install worked. Relabel and rerun.

Pods that will not start

A pod stays Pending with a node-affinity message

Either CDS cannot find its node, or a confidential pod cannot find a TEE-labeled one.

• role=cds — CDS pins to a labeled node by default. Label it, pass --single-node, or set a real selector. Note that cds.node.selector: null in a values file does not survive Helm's multi-file merge and silently reverts to role: cds; set {} or label the node.
• confidential.ai/sev-snp=true / confidential.ai/tdx=true — the confidential RuntimeClasses carry a node selector on these.

tls-lb stays Pending — "didn't have free ports for the requested pod ports"

tlsLb.hostPort.enabled is true by default and grabs the node's 443. On a cluster whose ingress already owns it — notably RKE2, which bundles rke2-ingress-nginx on 80/443 — set tlsLb.hostPort.enabled: false and reach tls-lb through its Service, or bind a free port.

A confidential pod loops in ContainerCreating

The effective ceiling is min(kubelet runtime-request-timeout, kata timeout). kubelet's default is 2 minutes; it cancels the CRI call, containerd tears the sandbox down, and the cause is hidden. Any slow path — cold registry, huge image, big-memory guest — hits that wall first. On RKE2, raise it in /etc/rancher/rke2/config.yaml:

kubelet-arg: runtime-request-timeout=20m

QEMU crash-loops with register dumps every ~90 seconds

kata-qemu-snp on a non-SNP host does not fail cleanly — it launches a guest with the SNP-shaped config that cannot boot, and kubelet retries forever. The node label is declared, not probed, so --hardware-platform set to the wrong hardware reproduces this on every node at once. Check what the host actually is before relabeling.

A brief ImageInspectError, or one ratls-mesh restart, early in the rollout

Normal. nri-image-policy restarts containerd as it installs; the mesh pod retries through that and self-heals.

ImagePullBackOff or CrashLoopBackOff past the timeout

Usually a missing or wrong image pull secret. --image-pull-secret names an existing kubernetes.io/dockerconfigjson Secret in the release namespace — the install never creates it.

Under --cvm-mode=pod, a workload image's layers are fetched inside the guest and that pull is anonymous. A pod-as-CVM workload image must be pullable with no credentials at all.

Workloads

A pod hangs in Init — "Startup probe errored … ExecProcessRequest is blocked by policy"

The locked guest denies ExecProcessRequest, and kubelet cannot distinguish an exec probe from a host exec. Any exec probe on a container inside a locked guest never passes. Use a run-once init container, or httpGet / tcpSocket / grpc.

The container is refused at creation

Its image digest is not on the allowlist. Add it:

c8s allowlist add "$(crane digest <IMAGE>)" <IMAGE> \
  --url $URL --measurements $DIGEST --operator-key operator.key

Two things that make a correct-looking digest still fail:

• Multi-arch images. crane digest returns the index digest, but the node resolves the tag to its platform-specific manifest and the enforcer matches that. Pin crane digest --platform linux/amd64 …, or read it back from a running pod with kubectl get pod <POD> -o jsonpath='{.status.containerStatuses[*].imageID}'.
• argv policy. A workload entry pins the command line too. An absent policy normalizes to deny, and command: deny requires an empty argv, so it can never start. Run c8s allowlist lint --strict — it flags exactly this.

Under Pod-as-CVM, an allowlist write does not reach running guests

Expected today. The in-guest refresh is gated on a pinned CDS measurement, and no shipping path delivers that pin — so policy-monitor keeps enforcing the measured seed baked into the guest image. Budget a guest-image rebuild for Pod-as-CVM allowlist changes. See Limitations.

kubectl logs returns nothing on a confidential pod

By design. The locked guest policy denies ReadStreamRequest, so the untrusted host cannot read container stdout — even for a healthy pod. Judge by exit code, or install --debug for the debug guest variant. It has a different launch measurement and exposes container I/O to the host: development only.

Secrets and volumes

The secret file never appears and the pod blocks

Work through these in order.

1. Is it just timing? The CDS releases only once every main container is running, so the file appears after your container starts. A consumer must wait for it, not read it at startup.
2. Is there a grant? An entry with no secrets grant releases nothing. Note that workload edit reports no changes for a grant-only edit — its field diff does not cover the grant. Use workload apply.
3. Is the CDS serving /secrets at all? It declines outright when --ratls-platform, --measurements, or --sandbox-inventory-cidr is missing, or when CA handoff is configured. It logs a warning naming the one it is missing. Check the CDS log before assuming the grant is wrong.
4. Ask the CDS what it saw. c8s secrets explain --sandbox <SANDBOX_ID> reports the decision in the order the CDS makes it. See Diagnosing a refusal.

A pod refused for a policy reason retries forever without progress. The fetcher lives in initContainers, so a terminal failure surfaces as Init:CrashLoopBackOff on an otherwise Running pod.

After a CDS restart, replicas hold different values for one path

The store is process memory with no persistence. A pod recreated after the restart finds its path empty, calls POST, and is given a new value while its siblings hold the old one. Nothing reports it. Roll every secret-consuming Deployment rather than letting pods recover piecemeal, and re-put every operator-supplied value.

The volume directory is empty

Check the c8s-volume sidecar's logs first. Then:

• volumed.enabled is false by default. Nothing works without it.
• The key must already be in the store. Unlike a secret, get-volume only ever reads — a pod scheduled before c8s volume create has run retries, then fails.
• The device must be on the pod's node, attached as a raw block device with serial c8s-vol-<NAME>. create prints the matching nodeSelector.
• Volumes require Node-as-CVM. Under Pod-as-CVM the webhook rejects the annotation at admission.

Verification

c8s verify hangs or returns exit 3

Exit 3 is evidence unavailable — unreachable, unparseable, or collateral it could not fetch. It is deliberately distinct from exit 2, which means the evidence arrived and failed the check. A bare SEV-SNP report needs outbound HTTPS to AMD KDS (kdsintf.amd.com) to fetch the VCEK; evidence that ships the VCEK inline verifies offline.

tls: certificate required

You dialed the in-cluster ClusterIP, which goes through the RA-TLS mesh and demands an attested client certificate. Reach the component on its public / host address instead. The CDS RA-TLS endpoint and the tls-lb serving port both answer unattested clients there.

A PKI or SAN mismatch when dialing a pod IP

Expected and fine. verify trusts the attestation embedded in the serving certificate, not the hostname on it. Pass --server-name to set SNI when forwarding to localhost.

verified: true but nothing was actually proven

Read it together with measurement_pinned. false there means any genuine TEE would pass identically. Pin the digest — see Obtain launch measurements.

The allowlist CLI refuses the front door

tlsLb.publicTLS.secretName makes tls-lb serve your own public certificate, which is not cryptographically bound to the discovery attestation — so the CLI rejects the endpoint rather than trust it. Use a direct CDS RA-TLS URL and pin the CDS launch digest.

Things not to debug

• Empty kubectl logs on a locked guest — by design, above.
• debug= and smt= in c8s verify text output — never populated; they always print false. The debug policy is still enforced inside the verifier.
• A failed Helm release on a converging cluster — check the pods, then re-run the install.

See also

• Install c8s — prerequisites and the flags that decide the shape.
• Limitations — the gaps that are known and deliberate.
• Verify a deployment — the exit-code contract in full.