Start a worker

Boot a worker with the worker.yaml admitted during Start policy enforcement. The worker runs kubelet and containerd inside its confidential VM. Your existing Kubernetes API, scheduler, and controllers run outside it.

Before you start

Continue in the deployment directory and shell from Prepare launch inputs. You need:

• A healthy CDS quorum.
• worker.yaml, worker-image.json, and artifacts/worker/.
• The component digest variables, admin.kubeconfig, and the private bootstrap token.

Keep the approved worker.yaml bytes unchanged.

The commands below use a Linux systemd control-plane host, sudo, Docker for extracting the Cilium operator binary, and kubectl compatible with your API server.

Provision each API-server host with:

• The matching bin/c8s and the two controller kubeconfigs generated below.
• A deployment directory that persists across reboot, with no spaces in its absolute path.
• The recorded OPERATOR_DIGEST and MESH_DIGEST values.

Keep operator.key on the trusted operator machine. Protect controller kubeconfigs as credentials.

Use your ordinary administrator kubeconfig on the operator machine:

export KUBECONFIG="$PWD/admin.kubeconfig"
kubectl get --raw=/readyz

Expect ok. Check the external API's bootstrap-token authenticator, kubelet CSR signer/approver, certificate rotation, and pod-CIDR allocation before continuing. Workers supply Cilium, kube-proxy, and CoreDNS; the control plane must have the bundled agents disabled as described in preparation.

Install the cluster objects

Render with the same component digests and platform as the worker image:

c8s worker control-plane render \
  --document worker.yaml --webhook-host 127.0.0.1 \
  --hardware-platform tdx --kube-version v1.36.4 \
  --image-digest "$OPERATOR_DIGEST" \
  --armtls-mesh-image-digest "$MESH_DIGEST" \
  --output-dir control-plane
kubectl apply -f control-plane/cluster.yaml
c8s worker control-plane credentials \
  --kubeconfig admin.kubeconfig --output-dir controllers

Use sev-snp for an AMD worker. The render installs cluster RBAC, the workload CRD, webhook configurations, router, and DNS Service. The credential command creates and approves its own certificate requests using your admin credentials. It writes controllers/c8s-operator.kubeconfig and controllers/c8s-cilium-operator.kubeconfig with mode 0600.

Supervise the external controllers

Run both controllers beside each API-server replica. With the loopback webhook, the c8s operator must share that API server's network namespace: 127.0.0.1:9443 must reach its webhook from the API-server process. For a containerized API server, arrange that namespace explicitly in your substrate. A host-loopback listener alone does not establish this connectivity.

On each control-plane host, obtain the matching Cilium operator without running its image. Resolve and retain the digest before extracting the binary:

CILIUM_DIGEST=$(oras resolve quay.io/cilium/operator-generic:v1.19.6)
printf '%s\n' "$CILIUM_DIGEST" > cilium-operator-digest.txt
docker create --name c8s-cilium-extract \
  "quay.io/cilium/operator-generic@$CILIUM_DIGEST"
docker cp c8s-cilium-extract:/usr/bin/cilium-operator-generic bin/cilium-operator-generic
docker rm c8s-cilium-extract
chmod 0755 bin/cilium-operator-generic

Approve that image under your image acquisition process. The version matches Cilium baked into this worker.

Save the following two unit files in the deployment directory on each host. Before installing them, replace every placeholder with that host's recorded value:

• <DEPLOYMENT_DIRECTORY>: the persistent, absolute, space-free deployment path, equal to that host's deployment-directory pwd output.
• <OPERATOR_DIGEST>: the complete OPERATOR_DIGEST value from preparation, including sha256:.
• <MESH_DIGEST>: the complete MESH_DIGEST value from preparation, including sha256:.

Keep the matching binaries and protected controller kubeconfigs at those paths. The units contain the substituted values literally; systemd does not expand shell variables in these files.

Save c8s-operator.service:

[Unit]
Description=c8s external operator
After=network-online.target
Wants=network-online.target
[Service]
WorkingDirectory=<DEPLOYMENT_DIRECTORY>
ExecStart=<DEPLOYMENT_DIRECTORY>/bin/c8s operator --kubeconfig <DEPLOYMENT_DIRECTORY>/controllers/c8s-operator.kubeconfig --webhook-host 127.0.0.1 --get-cert-image ghcr.io/confidential-dot-ai/c8s-operator@<OPERATOR_DIGEST> --mesh-image ghcr.io/confidential-dot-ai/armtls-mesh@<MESH_DIGEST>
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

Save c8s-cilium-operator.service:

[Unit]
Description=Cilium external operator
After=network-online.target
Wants=network-online.target
[Service]
WorkingDirectory=<DEPLOYMENT_DIRECTORY>
ExecStart=<DEPLOYMENT_DIRECTORY>/bin/cilium-operator-generic --k8s-kubeconfig-path <DEPLOYMENT_DIRECTORY>/controllers/c8s-cilium-operator.kubeconfig --k8s-namespace kube-system --ipam kubernetes --identity-allocation-mode crd --enable-l7-proxy=false
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target

After making the substitutions in both files, install and start the services:

sudo install -m 0644 c8s-operator.service c8s-cilium-operator.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now c8s-operator c8s-cilium-operator
sudo systemctl is-active c8s-operator c8s-cilium-operator

Expect both services active.

• The Cilium operator creates its CRDs; its leader lease lives in kube-system.
• The c8s operator serves and configures the admission webhook.

Inspect controller startup failures with journalctl -u on the control-plane host.

Inspect each controller certificate's actual expiry:

for config in controllers/*.kubeconfig; do
  printf '%s: ' "$config"
  kubectl config view --raw --kubeconfig "$config" \
    -o jsonpath='{.users[0].user.client-certificate-data}' | \
    base64 -d | openssl x509 -noout -enddate
done

Controller credentials request 30 days of validity, capped by the cluster signer's lifetime. Record each issued certificate's actual expiry.

Renew controller credentials before that expiry:

1. Rerun worker control-plane credentials with the administrator kubeconfig.
2. Securely replace each host's files.
3. Restart both services.

These controller kubeconfigs require operator-managed renewal. The worker's kubelet handles its own client-certificate rotation.

Return to the original trusted operator machine, deployment directory and preparation shell before continuing.

Prepare the worker launch disk

Create an ISO containing the approved document and raw bootstrap token. The prompt accepts only the token after any k3s/RKE2 K10<hash>:: prefix:

umask 077
mkdir worker-opkeydata
cp worker.yaml worker-opkeydata/pubkey
read -r -s -p 'Kubernetes bootstrap token: ' WORKER_BOOTSTRAP_TOKEN
printf '\n'
[[ "$WORKER_BOOTSTRAP_TOKEN" =~ ^[a-z0-9]{6}\.[a-z0-9]{16}$ ]]
printf '%s' "$WORKER_BOOTSTRAP_TOKEN" > worker-opkeydata/bootstrap-token
unset WORKER_BOOTSTRAP_TOKEN
cmp worker.yaml worker-opkeydata/pubkey
genisoimage -quiet -V opkeydata -J -r \
  -o worker-opkeydata.iso worker-opkeydata
sha256sum worker-opkeydata/pubkey > worker-launch-sha256.txt

Protect the ISO and staging directory: the bootstrap token is plaintext and outside the measured document. The worker verifies the launch document, then uses the token to request a kubelet certificate from the authenticated API. Apply your control plane's bootstrap-token expiry and rotation procedure.

Launch and check the worker

Use the TEE launcher prepared for your substrate with these inputs:

• The approved boot disk and matching firmware/IGVM files under artifacts/worker/; choose the manifest's matching TDX or SEV-SNP variant and vCPU count.
• worker-opkeydata.iso attached read-only, with ISO label opkeydata and both files at its root.
• A dedicated scratch disk of at least 64 GiB, attached as virtio-blk with serial confai-scratch. The measured guest initializes its encryption; runtime state disappears on reboot.
• For SEV-SNP, HOST_DATA equal to SHA-256 of the exact worker-opkeydata/pubkey bytes, recorded in worker-launch-sha256.txt. TDX binds those bytes through the measured initrd's RTMR[3].
• An exclusive primary IPv4 inside NODE_CIDR. Stop or fence any old VM before reusing its address.

Allow these network paths:

• Workers to the external API, CDS TCP 8443, registries, DNS, and CPU endorsement-certificate services.
• UDP 8473 between workers for Cilium VXLAN.
• CDS to worker TCP 1019 for admission inventory.
• Control plane to authenticated kubelet TCP 10250.
• Client TCP 443 to workers hosting the router.
• CDS peer TCP 8444 between CDS replicas.

Keep the CDS client path as TLS passthrough.

The substrate launches the VM; c8s supplies measured guest artifacts and launch documents. Its boot gates verify the document, scratch encryption, runtime restrictions, and GPU CC readiness when GPUs are present before starting pods.

Set WORKER_IP to the primary IPv4 assigned by the launcher:

WORKER_IP='<WORKER_PRIMARY_IPV4>'
WORKER_NODE="n-${WORKER_IP//./-}"
kubectl wait --for=create "node/$WORKER_NODE" --timeout=5m
kubectl wait --for=condition=Ready "node/$WORKER_NODE" --timeout=5m
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl -n c8s-router rollout status deployment/c8s-router --timeout=5m

Require:

• The node is Ready.
• Its InternalIP equals WORKER_IP.
• The router is ready.

If registration or readiness times out, check kubectl get csr and control-plane events. Check DNS, routing, bootstrap credentials, signer availability, and VM boot-gate failures before proceeding.

The measured kubelet disables exec, attach, port-forward, debug, and logs. Use object status/events and application diagnostics. Node readiness is a Kubernetes operating signal; client attestation and policy verification establish what handles private data. Continue to Run a workload.