Fetch a kubeconfig from a running CVM

Attest a c8s node CVM you launched yourself, prove it booted the image you pinned and was launched to trust your operator key, and write a 24-hour cluster-admin kubeconfig over RA-TLS.

c8s get-kubeconfig runs on your machine and gives you kubectl access to a c8s node confidential VM (CVM) that no cluster you can reach knows about yet. It attests the guest through the measured-image + operator-key gate, then exchanges a locally generated certificate signing request (CSR) for a system:masters client certificate over Remote-Attestation TLS (RA-TLS) and writes a kubeconfig. Its counterpart inside the guest is cred-release, a service in the c8s node image that signs that certificate with the cluster's client CA.

Nothing is unsealed from storage. The credential is minted on demand, the private half of the client key never leaves your machine, and the certificate is valid for 24 hours — run the command again for a new one.

This is for node CVMs you launch yourself.

On AKS and GKE the provider issues the kubeconfig and this command does not apply. See Why this does not apply on AKS or GKE.

Before you start

  • A booted c8s node CVM you can reach on 8400 (attestation-api), 8443 (cred-release), and 6443 (kube-apiserver). See Boot a node CVM on bare metal.
  • That CVM must have been launched for your operator key. On Intel TDX the public key reaches the guest as a disk labeled opkeydata holding one file, pubkey, and the measured initrd extends its digest into RTMR[3] before switch_root. On AMD SEV-SNP the launcher commits HOSTDATA = SHA-256(pubkey bytes) at launch. Either way the binding is set at launch and cannot be added to a running guest.
  • The operator EC private key (operator.key).
  • The build-artifact manifest of the exact image the CVM booted (step 1 gets it).
  • On SEV-SNP, outbound HTTPS from your machine to AMD's Key Distribution Service: bare-metal SNP evidence carries no inline VCEK, so the verifier fetches one to check the quote's signature.
  • The c8s CLI — Install c8s has the build steps.

One keypair, two gates

One EC keypair does double duty across the whole bootstrap. Generate it once:

umask 077
openssl ecparam -name prime256v1 -genkey -noout -out operator.key
openssl ec -in operator.key -pubout -out operator.pub
HalfGoes toAuthorizes
operator.pubthe CVM's launch identity (TDX RTMR[3], SNP HOSTDATA) and c8s install --operator-keyswhich key the guest releases a credential to, and which keys the CDS accepts operator writes from
operator.keyc8s get-kubeconfig --operator-key and c8s allowlist / c8s secrets writessigning the release request, and signing allowlist and secrets writes

These are two independent gates that in practice hold one keypair: the launch binding is verified by the guest and by this command, while --operator-keys is a Helm value the install pins into the CDS, the cluster's Certificate Distribution Service. c8s get-kubeconfig derives the public half from --operator-key itself, so the bytes it compares against the quote are the bytes openssl ec -pubout wrote.

That forces the order. The key must exist before the CVM boots, and the CVM has no kubeconfig until this command runs, so the bootstrap is:

  1. Create the keypair (above).
  2. Launch the node CVM with operator.pub bound into its launch identity.
  3. Run c8s get-kubeconfig — this is the first kubectl access anyone has to the guest.
  4. Run c8s install --operator-keys operator.pub against that kubeconfig.

Fetch the kubeconfig

The commands below assume a bare-metal Intel TDX guest running under KubeVirt, addressed by its guest IP. Substitute <GUEST_IP>.

Get the image manifest for the build the CVM booted

The manifest ships as a manifest.json layer of the oras artifact published beside the CDI image the CVM boots — the same repository and the same tag with the -cdi suffix removed. Pull the layer:

crane manifest <REGISTRY>/<REPO>:<TAG> \
  | jq -r '.layers[] | select(.annotations["org.opencontainers.image.title"] == "manifest.json") | .digest'

crane blob <REGISTRY>/<REPO>@<MANIFEST_LAYER_DIGEST> > image-manifest.json

<TAG> is the boot image's tag without -cdi; <MANIFEST_LAYER_DIGEST> is the digest the first command printed. Confirm it describes the build your CVM booted by matching the whole tuple, not just mrtd — two builds that share a firmware and memory layout report the same MRTD while their kernels, and so their RTMR[1], differ:

jq '.tdx' image-manifest.json           # TDX: mrtd, rtmr1, rtmr2
jq '.snp_variants' image-manifest.json  # SNP: one launch digest per vCPU count

The manifest's shape selects the platform: a TDX build publishes mrtd/rtmr1/rtmr2, an SNP build publishes snp_variants. A generic artifact-hash manifest.json carries neither and is rejected.

Confirm the guest is answering

curl -s http://<GUEST_IP>:8400/health                                    # "platform":"tdx" (or "snp")
curl -sk -o /dev/null -w '%{http_code}\n' https://<GUEST_IP>:6443/livez  # 200, 401, or 403
curl -sk -o /dev/null -w '%{http_code}\n' https://<GUEST_IP>:8443/       # any code but 000

The attestation-api answers as soon as the TEE device is up. cred-release answers much later: it starts after RKE2 has initialized and written the cluster client CA, which on a first boot takes minutes.

Attest the CVM and fetch the kubeconfig

c8s get-kubeconfig \
  --node <GUEST_IP> \
  --operator-key operator.key \
  --image-manifest image-manifest.json \
  --out guest.kubeconfig

--node fills in the three endpoints with their standard ports. On success the command writes guest.kubeconfig with mode 0600 and prints:

wrote guest.kubeconfig (context "c8s") — attested: image tuple + operator-key chain verified

While cred-release is still coming up you will see cred-release not listening yet; retrying on stderr. The command retries a refused connection every 5 seconds for --release-wait (2m by default); raise it on a slow first boot.

Add --workload-image <REF>@sha256:<HEX> once per digest-pinned image the node's measurer extended into RTMR[3], in first-extend order. Omit it when the node runs no measured workloads — the register must then equal the bare operator-key seed. The flag is TDX-only.

Confirm what you got

export KUBECONFIG=$PWD/guest.kubeconfig
kubectl auth whoami -o jsonpath='{.status.userInfo.username}'   # operator
kubectl get nodes

The identity is operator in the group system:masters, and the kubeconfig anchors to the cluster's serving CA, which the guest returned with the certificate. It expires 24 hours after issue; re-run c8s get-kubeconfig for a fresh one. Next stop is Install c8s, with --operator-keys operator.pub.

Address a KubeVirt guest by name

If the guest runs as a KubeVirt VirtualMachineInstance, name it instead of chasing its address:

c8s get-kubeconfig \
  --vmi <NAMESPACE>/<VMI_NAME> \
  --operator-key operator.key \
  --image-manifest image-manifest.json \
  --out guest.kubeconfig

--vmi reads the VMI's first reported status.interfaces[].ipAddress through your current kubeconfig — the outer cluster that hosts the VM — and then talks to the guest directly. The namespace defaults to your current context's. A VMI that is still booting reports no address yet, and the lookup fails with has no reported address. --vmi and --node are mutually exclusive.

SEV-SNP under direct QEMU

A guest launched by hand under QEMU usually reaches you on host-forwarded ports rather than 8400/8443/6443, so give all three URLs explicitly instead of --node:

c8s get-kubeconfig \
  --attest-url    http://127.0.0.1:<ATTEST_PORT>/attest \
  --release-url   https://127.0.0.1:<RELEASE_PORT> \
  --apiserver-url https://127.0.0.1:<APISERVER_PORT> \
  --operator-key operator.key \
  --image-manifest manifest.json \
  --out guest.kubeconfig

Use the SNP build's manifest.json, the one carrying snp_variants. --workload-image is rejected on SNP: there is no runtime-extend register, so a workload chain cannot be verified.

The kubeconfig's tls-server-name is c8s-cvm by default, which is the SAN the node image bakes into RKE2's tls-san. That is what makes the apiserver certificate verify against a stable name rather than the per-launch address you dialed.

What the gate checked

The manifest names the platform, and the node must be that platform: evidence tagged az-snp, az-tdx, or gcp-* is refused before any claim is read. On top of that:

PlatformThe verified quote must carry
Intel TDXMRTD, RTMR[1] (guest kernel), and RTMR[2] (guest rootfs) equal to the manifest's tuple, and RTMR[3] equal to the operator-key seed extended, in order, by each --workload-image digest
AMD SEV-SNPa launch digest that is one of the manifest's per-vCPU-count variants, and a 32-byte HOSTDATA equal to SHA-256 of the operator public key

The image tuple and RTMR[3] are both load-bearing. RTMR[3] alone is not an identity: the untrusted host stages the operator key, so it could boot any image at all and still reproduce the bare operator-key register. The manifest anchors the image; the register binds the key.

The same policy is enforced twice. Once on the attest gate, against a fresh 32-byte nonce returned in report_data. Then again on the RA-TLS dial to cred-release, where the serving certificate must be self-signed, carry a quote bound to its own public key, and satisfy the identical pins — so a host that cannot forge the first gate cannot sit in the middle of the channel that carries the CSR either. For how this verification model works across the rest of the platform, see The verification model.

When it fails

Every check fails closed, and the message names which one.

MessageCauseFix
MRTD mismatch / RTMR[1] mismatch (guest kernel) / RTMR[2] mismatch (guest rootfs)the manifest is from a different build than the one that bootedfetch the manifest published beside the image the CVM actually booted, and match the whole tuple
RTMR[3] mismatch (operator-key + workload chain)the guest was launched for a different key, or --workload-image is missing, extra, or out of orderconfirm which operator.pub went on the opkeydata disk; list each measured image once, in first-extend order
launch digest mismatch: … (a different guest image booted, or a vCPU count the manifest has no variant for)SEV-SNP: the image or the vCPU count is outside the pinned setlaunch at a vCPU count the manifest publishes a variant for
HOSTDATA mismatch … (the VM was not launched for this key) or quote carries no HOSTDATASEV-SNP: the launcher committed another key's digest, or nonerelaunch with host-data set to SHA-256(operator.pub)
node platform is "az-snp" but --image-manifest pins "snp"the target is a cloud CVM, not one you launcheduse the provider's kubeconfig — see below
image manifest …: missing "mrtd" or no "snp_variants"a generic artifact-hash manifest.json, or the other platform's manifestpull the manifest.json layer of the build's oras artifact
cred-release not listening yet; retrying (stderr, not fatal)boot ordering: RKE2 has not written the client CA yetwait; raise --release-wait on a slow first boot
credential release: release request: … EOFthe guest accepted the connection before its TLS listener was serving; only a refused connection is retriedrun the command again
release HTTP 401: unauthorized: …the request was signed by a key the guest has not measured, or the two clocks differ by more than 60 secondscheck which key you passed; sync NTP on both ends
every release dial refused for the full --release-waitcred-release is skipped, not failed, on a guest with no opkeydata disk or on an agent-role noderelaunch the guest with the operator key, on a server-role node

Why this does not apply on AKS or GKE

Each of these is enough on its own:

  • cred-release ships only in the c8s node image. An AKS or GKE node does not run that image, so nothing serves :8443.
  • The gate refuses the evidence. A manifest pins tdx or snp; Azure and GCP confidential nodes report az-tdx, az-snp, or gcp-*, and the platform compare rejects them.
  • HOSTDATA is owned by the cloud stack on those platforms, so it cannot carry the operator-key binding at all.

Get the kubeconfig from the provider instead — az aks get-credentials on AKS. See Provision a confidential cluster on Azure. The operator keypair still matters there: c8s install --operator-keys operator.pub is what authorizes allowlist and secrets writes.

See also