Troubleshoot C8s

Start with the failing component's events and logs, then compare its configuration with the approved deployment inputs. Logs help locate a failure; use verification to establish identity and trust.

Collect diagnostics

These Kubernetes reads require a reachable API server and a kubeconfig authorized to list pods, events and Nodes, describe the affected pod, and read its logs. Use credentials scoped to the affected namespaces where cluster-wide reads are unavailable: replace -A with -n <NAMESPACE> in the pod/event commands. Node reads remain cluster-scoped. Node journals require operator-provided access. Baked-node operator credentials do not authorize a full platform installation or teardown.

kubectl get pods -A
kubectl get events -A --sort-by=.lastTimestamp | tail -40
kubectl describe pod -n <NAMESPACE> <POD>
kubectl logs -n <NAMESPACE> <POD> -c <CONTAINER> --tail=100

c8s render-values --cvm-mode bare-metal --hardware-platform sev-snp \
  --cds-address <CDS_HOST>:8443 --cds-pin cds-pin.json \
  --operator-keys operator.pub --resolve-digests=false

Install failures

Check --cvm-mode (bare-metal, gke, or aks) and --hardware-platform (sev-snp or tdx) against the deployment's evidence source. Installation needs helm, kubectl, and API permissions for the resources it manages. Default digest resolution also needs crane and registry access.

If installation rejects the named CDS, regenerate the component seed with c8s install --allowlist-seed-out component-seed.json. Keep the failed installation's release, namespace, image, platform, router and other component options, plus its ordered values files, against the same Kubernetes version. Omit --cds-address and --cds-pin for this seed command. The generator supplies placeholder CDS values. This writes the canonical seed and stops before installation; render-values produces a different artifact. Compare each seed workload entry with the corresponding entry in the JSON allowlist stored in the retained cds-config.yaml workloads field, whose configuration bytes are bound into cds-pin.json. Every rendered component entry must match. The component-seed handoff shows how that artifact becomes a CDS configuration input. Changing router TLS mode or enabling a component can change the required policy. Diagnose the mismatch before planning a configuration change; replacing CDS can lose volatile state.

Inspect kubectl get nodes -o yaml for each Node's spec.podCIDR or spec.podCIDRs. Final installation requires these fields even when --pod-cidr was supplied while preparing CDS. CDS node-inventory subnets must exclude pod ranges. Successful seed generation does not check this live-cluster prerequisite.

On a baked C8s node, c8s-system carries confidential.ai/baked=true and c8s install refuses to run. The measured image owns core services and admission policies; its signed launch configuration names the external CDS. Follow that deployment's launch procedure.

For a Helm wait timeout, use the pod's events to distinguish scheduling, image retrieval and admission failures. A pending router may have a host-port conflict; inspect its rendered ports and existing listeners before changing deployment inputs. For persistent credential-release readiness failure on a baked server, have the operator collect the cred-release service journal, including psa-ready output, and API readiness and admission-policy diagnostics. Credential release depends on these guards passing; a timeout does not establish that another wait will succeed.

Workload failures

For an admission denial, compare the pod's image digests, argv, environment and mounts with its allowlist entry. Inspect pod events and the node's NRI image-policy logs. On baked nodes, the plugin writes to syslog with tag nri-image-policy, available in the node journal. Allowlist pulls stalled for at least five minutes report the last failure. Environment-denial messages identify fields and names without printing their values.

For an Init or unready pod, inspect c8s-cert for CDS reachability, attestation, admission matching and renewal failures; inspect c8s-mesh for rejected or withdrawn credential generations and local-delivery failures. CDS matches the cumulative sandbox admission record, including exited containers; required main-container policies must be present in that record. A matching entry is not a proof of current container liveness. For operator-assisted inspection, follow optional sandbox diagnosis: secrets explain reports the admitted candidates, near misses and missing main-container policies. It requires an operator-supplied runtime sandbox ID and signing authority. If that input is unavailable, continue with the certificate/delivery logs and reviewed allowlist entry.

The mesh keeps its private credentials separate from application containers and authenticates incoming connections before forwarding plaintext to the application's ordinary socket. Check that socket and its configured port through application logs and supported health probes. kubectl port-forward is not a supported path through the pod's mesh boundary. Same-key credential renewal is supported. Replacing the mesh CA key requires recreating pods and updating independently trusted client anchors; a plugin restart alone does not imply that all pods need recreation. Credential expiry or withdrawal makes the endpoint unready and refuses new forwarding; existing streams are not guaranteed to close immediately.

For rejected probes, use an HTTP probe with an empty httpGet.host, an absolute clean application path using unreserved characters, a numeric application port or a named port declared by that same container, empty headers and a timeout of at most five seconds. Choose a harmless path. The injector renders the health target; its response has an empty body and does not follow redirects. TCP/gRPC probes and HTTP lifecycle hooks are refused. Exec probes depend on the deployment's exec policy. Health status, timing and the effects of its GET remain visible on the plaintext health listener.

For secret or volume delivery, inspect the delivery sidecar and the named workload's grants. A secret 404 can mean either a denied read or an absent value. Follow diagnosing a refusal for grant readback and the optional secrets explain procedure. Its sandbox ID comes from operator runtime/inventory records or captured workload evidence, not the Kubernetes pod UID. If the grant is correct but the value is absent, an authorized operator must supply the value through secrets put or restore it from retained escrow. Check volumed diagnostics for encrypted-volume failures; its placement depends on the deployment.

Delivery attempts default to unlimited retries, with a five-second initial delay doubling to 30 seconds. Repeated retries do not prove eventual release. Secret files are replaced atomically one file at a time; consumers cannot assume whole-set atomic publication.

CDS failures

Check the independent CDS VM through its actual client endpoint:

c8s cds verify https://<CDS_HOST>:8443 --image-policy-file cds-pin.json

CDS verification requires the image and launch-file pin. Its admitted node policy is a separate check: use --served-policy-file with the independently intended current node set, as described in verify a deployment. c8s cds ca, c8s secrets and c8s volume create need the external CDS URL and this complete pin; they refuse router discovery endpoints. Changing chart cds.operatorKeys does not change the external VM's launch-bound operator authority.

For HA CDS, inspect the selected replicas with the exact measured configuration and CDS pin:

c8s cds ha status --config cds-config.yaml --cds-pin cds-pin.json --json

A nonserving replica returns 503 for ordinary APIs while /ha/status, /ha/pins and /ha/decrees remain available. /ha/pins returns 404 until a pin decree is applied. Status fails for unreachable, nonserving or wrong-cluster replicas and version/CA disagreement; inspect stuck, roll_safe and divergence in the JSON reports too, because not every reported problem changes its exit status. Omitting --json prints text with ALARM lines.

A singleton restart loses its in-memory secrets. HA replicates secrets and CA state in memory; one replica restart does not erase a surviving cluster's state. Loss of all replica memory loses that state, so retain escrow. On permanent quorum loss, preserve surviving VMs and collect status, then hand the incident to the deployment operator. A decree reported held is accepted, not applied. c8s cds ha decree recover delivers a signed Recover intent, but the current runtime has no supported workflow for a surviving minority to resume service after permanent quorum loss. See the HA recovery limits.

Front door failures

Use the deployment's reachable router hostname and ingress. Inspect the cds-attest log for route manifest applied or route manifest poll failed, and compare the stored manifest with c8s routes show against the external CDS using its pin. A successful /readyz response does not prove that a route manifest has been applied.

For a backend identity failure, inspect the error field in the router's cds-attest log. Workload-mismatch messages name the backend leaf's workload and the route's expected name; digest-mismatch messages include both backend and router allowlist digests. Independent credential renewals can temporarily leave those digests different. If the operator supplies captured workload evidence, follow verify a workload, add -o json, and inspect workload and workload_allowlist_digest; that procedure explains the evidence prerequisite.

Route or router identity changes refuse later bound requests and end the affected PQ session; they do not establish that already running streams have stopped. The browser client reports outer non-success responses as channel_error and does not reconnect or replay automatically. Handle recovery explicitly, accounting for whether retrying the application request is safe. CLI router discovery can re-attest a renewed leaf while retaining its first issuer-key anchor; that behavior is separate from browser session recovery.

Verification failures

Read the full verdict and preserve its exit status:

Keep the CDS identity pin, the intended admitted node set, and the router/workload node pin separate. For CA-issued evidence, supply the independently trusted --mesh-ca anchor. Generic verification without image pins does not establish your expected image; CDS requires its complete launch-file policy. attest-pq and attest-lb additionally require the intended --route-digest or trusted --operator-keys. attest-lb needs a cds or acme front door; attest-pq remains available with webpki. Follow verify a deployment for the appropriate complete check.