Confidential AI ･ Deploy NVIDIA Dynamo as a confidential workload

Run NVIDIA Dynamo's serving stack — frontend, router, worker, and discovery — on a c8s cluster, with every image digest-allowlisted, both serving components holding CDS-issued TEE identities, and the OpenAI endpoint behind the attested front door.

This tutorial deploys NVIDIA Dynamo — the distributed LLM inference framework — onto a c8s cluster as a set of confidential workloads.

Dynamo splits serving into components: an OpenAI-compatible frontend that routes requests, workers that run the engine, and a discovery plane that connects them. That makes it a realistic test of running a multi-component serving system inside the Trusted Execution Environment (TEE).

You'll allowlist every image, give the frontend and worker identities issued by the CDS (Certificate Distribution Service, the cluster's attested trust root), and chat with the deployment from your laptop over a verified, sealed channel. Every image is public — there is nothing to build or push.

CPU cluster, mock engine.

Dynamo's real engine backends (vLLM, SGLang, TensorRT-LLM) ship as CUDA images and need GPUs — and this tutorial's AKS cluster is CPU-only. So it runs Dynamo's own GPU-free mocker engine (dynamo.mocker, the upstream-supported way to exercise the frontend, router, and discovery without GPUs). The mocker downloads only the model's tokenizer — a few MB — and never loads weights, so it runs comfortably on the smallest confidential node sizes. Every c8s property you set up — digest enforcement, workload identity, the attested front door — transfers unchanged to a GPU-backed deployment: swap the worker's image and command, keep the annotations.

Prerequisites

This tutorial continues from Deploy a confidential AKS cluster and run c8s and assumes that cluster: single-node AKS in node-as-CVM mode, with operator.pub pinned at install. On a different c8s cluster everything here transfers — adjust the install flags in Step 5 and how you expose the front door.

You'll also need:

The c8s CLI, kubectl, and crane on your laptop (all already there if you followed that tutorial).
The operator private key (operator.key) — allowlist writes are signed with it.
For the final step: Node ≥ 20, plus the LB launch measurement and your cluster's mesh CA PEM, supplied out of band — see Consumer verification.

Step by step

1. Deploy the discovery and event planes

Plain-Deployment Dynamo (no Dynamo operator) uses etcd for endpoint discovery, and its KV-event plane defaults to NATS under etcd discovery — so deploy both, single-replica. This etcd is Dynamo's own registry, deployed like any app — it is not the Kubernetes control plane's etcd, which stays untouched (and on AKS isn't even reachable).

These are infrastructure, not confidential workloads: they carry routing metadata, and under node-as-CVM they already run inside the node's confidential boundary. Don't annotate them — c8s identities belong on the components that terminate model traffic.

# dynamo-infra.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: etcd
  namespace: workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      containers:
        - name: etcd
          image: quay.io/coreos/etcd:v3.6.13
          command:
            - /usr/local/bin/etcd
            - --listen-client-urls=http://0.0.0.0:2379
            - --advertise-client-urls=http://etcd.workloads.svc:2379
          ports:
            - containerPort: 2379
---
apiVersion: v1
kind: Service
metadata:
  name: etcd
  namespace: workloads
spec:
  selector:
    app: etcd
  ports:
    - port: 2379
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nats
  namespace: workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nats
  template:
    metadata:
      labels:
        app: nats
    spec:
      containers:
        - name: nats
          image: nats:2.14.3-alpine
          ports:
            - containerPort: 4222
---
apiVersion: v1
kind: Service
metadata:
  name: nats
  namespace: workloads
spec:
  selector:
    app: nats
  ports:
    - port: 4222

kubectl create namespace workloads   # skip if it exists from another tutorial
kubectl apply -f dynamo-infra.yaml

The pods won't start yet — their images aren't on the allowlist. That's enforcement working; Step 3 fixes it.

Scaling past one node changes the picture.

The RA-TLS mesh wraps pod-IP traffic — which covers Dynamo's frontend→worker request plane, since workers register their pod IPs in etcd. But etcd and NATS are dialed here through Service VIPs, which the mesh cannot intercept: on a single node that traffic never leaves the CVM; across nodes it would cross the wire unmeshed. Keep the graph on one node, or move inter-node infra hops onto meshed pod-IP addressing before scaling out.

2. Deploy the frontend and worker as confidential workloads

Both components run from one public image: dynamo-frontend is NVIDIA's non-CUDA Dynamo image, and it carries the whole ai-dynamo Python package — dynamo.frontend and dynamo.mocker — so the two Deployments differ only in command: (which overrides the image's entrypoint). It pulls anonymously from nvcr.io.

The confidential.ai/cw annotation opts each Deployment into c8s: the webhook injects the c8s-cert identity sidecar (a CDS-issued, TEE-bound certificate), and the operator mints a headless Service per workload id — c8s-dynamo is what the front door will dial in Step 5. The components find etcd and NATS via the env vars Dynamo reads (ETCD_ENDPOINTS, NATS_SERVER); requests then flow frontend → worker over Dynamo's TCP request plane. The --model-path names the tokenizer the mocker fetches — no weights are loaded.

# dynamo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dynamo-frontend
  namespace: workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dynamo-frontend
  template:
    metadata:
      labels:
        app: dynamo-frontend
      annotations:
        # workload id → headless Service c8s-dynamo
        confidential.ai/cw: dynamo
    spec:
      containers:
        - name: frontend
          image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1
          command:
            - python3
            - -m
            - dynamo.frontend
            - --http-port
            - "8000"
          ports:
            - containerPort: 8000
          env:
            - name: ETCD_ENDPOINTS
              value: http://etcd.workloads.svc:2379
            - name: NATS_SERVER
              value: nats://nats.workloads.svc:4222
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dynamo-worker
  namespace: workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dynamo-worker
  template:
    metadata:
      labels:
        app: dynamo-worker
      annotations:
        confidential.ai/cw: dynamo-worker
    spec:
      containers:
        - name: worker
          image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1
          command:
            - python3
            - -m
            - dynamo.mocker
            - --model-path
            - Qwen/Qwen3-0.6B
          env:
            - name: ETCD_ENDPOINTS
              value: http://etcd.workloads.svc:2379
            - name: NATS_SERVER
              value: nats://nats.workloads.svc:4222

kubectl apply -f dynamo.yaml

On a GPU cluster the worker's command would run dynamo.vllm --model Qwen/Qwen3-0.6B on a CUDA runtime image — the c8s wiring around it stays identical. (And if you'd rather serve from a minimal image you control, pip install ai-dynamo==1.2.1 on python:3.12-slim provides the same entrypoints — build it, push it, and allowlist that digest instead.)

3. Allow the images

Everything is deployed and nothing runs — check the events and you'll see c8s enforcement denying container creation at the node:

kubectl get events -n workloads | grep allowlist
# Warning  Failed  ...  image not in allowlist: quay.io/coreos/etcd:v3.6.13

Add the three digests, signed with your operator key. The CDS has no public ingress, so port-forward it first — along with the attestation-api, which the CLI uses to verify the CDS's own attestation before talking to it:

kubectl port-forward -n c8s-system svc/c8s-cds 8443:8443 &
kubectl port-forward -n c8s-system svc/c8s-attestation-api 8400:8400 &

DYNAMO_IMAGE=nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1
c8s allowlist add "$(crane digest "$DYNAMO_IMAGE")" "$DYNAMO_IMAGE" \
  --url https://localhost:8443 \
  --attestation-api-url http://localhost:8400 \
  --operator-key operator.key

ETCD_IMAGE=quay.io/coreos/etcd:v3.6.13
c8s allowlist add "$(crane digest "$ETCD_IMAGE")" "$ETCD_IMAGE" \
  --url https://localhost:8443 \
  --attestation-api-url http://localhost:8400 \
  --operator-key operator.key

NATS_IMAGE=nats:2.14.3-alpine
c8s allowlist add "$(crane digest "$NATS_IMAGE")" "$NATS_IMAGE" \
  --url https://localhost:8443 \
  --attestation-api-url http://localhost:8400 \
  --operator-key operator.key

The CLI warns that no --measurements are pinned — it's accepting any attested CDS, fine for a tutorial cluster; in production pin your CDS launch digest so a write can never land on a rogue CDS. The enforcement plugins poll the CDS-served list, so the kubelet's next retry succeeds — the Dynamo image is large, so its first pull takes a few minutes:

kubectl get pods -n workloads -w
# etcd-…             1/1   Running
# nats-…             1/1   Running
# dynamo-frontend-…  2/2   Running   ← app + injected c8s-cert sidecar
# dynamo-worker-…    2/2   Running

Both serving components now hold TEE-bound identities. Mirror that into kubectl with two optional ConfidentialWorkload CRs (a status view — injection works without them):

# dynamo-cwl.yaml
apiVersion: confidential.ai/v1alpha2
kind: ConfidentialWorkload
metadata:
  name: dynamo
  namespace: workloads
spec:
  workloadRef:
    kind: Deployment
    name: dynamo-frontend
---
apiVersion: confidential.ai/v1alpha2
kind: ConfidentialWorkload
metadata:
  name: dynamo-worker
  namespace: workloads
spec:
  workloadRef:
    kind: Deployment
    name: dynamo-worker

kubectl apply -f dynamo-cwl.yaml
kubectl get cwl -n workloads
# NAME            WORKLOAD          ATTESTED   TOTAL
# dynamo          dynamo-frontend   1          1
# dynamo-worker   dynamo-worker     1          1

4. Smoke-test the serving graph

Before wiring the front door, confirm discovery worked end to end: the worker registered in etcd, and the frontend found it. kubectl port-forward reaches the pod over the kubelet's own path, so it works against confidential pods:

kubectl port-forward -n workloads deploy/dynamo-frontend 8000:8000 &

curl -s localhost:8000/v1/models
# {"object":"list","data":[{"id":"Qwen/Qwen3-0.6B", ...}]}

curl -s localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model": "Qwen/Qwen3-0.6B",
       "messages": [{"role": "user", "content": "hello"}],
       "max_tokens": 32}'
# {"id":"...","choices":[{"message":{"role":"assistant", ...

kill %%   # stop this port-forward (the most recent background job)

The completion text is mock output — the mocker simulates engine scheduling and token timing, not language. What it proves is the part this tutorial is about: requests traverse frontend → router → worker across attested, identity-bearing components.

5. Point the front door at the frontend

External clients come in through the chart's front door — tls-lb (public TLS + over-encryption) → your frontend, dialed over the node mesh's attested mTLS. Point it there with --workload-ref + --upstream: the ref's :8000 (the frontend's --http-port) derives the upstream c8s-dynamo.workloads.svc.cluster.local:8000 — the minted headless Service, whose DNS returns pod IPs the node mesh wraps in attested mTLS; a Service VIP it cannot. service.type: LoadBalancer gives the front door a public IP on AKS:

# values.yaml
tlsLb:
  attest:
    enabled: true    # serve the c8s-verify/v1 attestation endpoints
  service:
    type: LoadBalancer

Apply it by re-running the c8s install from the Azure tutorial with the two flags and -f values.yaml added (the front door has one catch-all upstream — this repoints it if another tutorial's workload was wired first). --workload-ref also stamps confidential.ai/cw: dynamo onto the Deployment's pod template — a no-op here, since Step 2's manifest already carries it:

c8s install \
  --single-node \
  --cvm-mode aks \
  --image-pull-secret ghcr-secret \
  --operator-keys operator.pub \
  --workload-ref dynamo=workloads/deployment/dynamo-frontend:8000 \
  --upstream dynamo \
  -f values.yaml

Then grab the public address Azure assigns — it's <pending> for a minute or so:

kubectl get svc -n c8s-system c8s-tls-lb -w
# NAME         TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)
# c8s-tls-lb   LoadBalancer   10.0.…       20.…          443:…/TCP

LB_ADDRESS=<EXTERNAL-IP>

Not on the Azure tutorial's cluster?

Use the flags you originally installed with, and expose c8s-tls-lb however your environment does it. With no cloud load balancer, set tlsLb.service.type: NodePort in values.yaml and reach the front door on the node:

NODE_PORT=$(kubectl get svc -n c8s-system c8s-tls-lb -o jsonpath='{.spec.ports[0].nodePort}')
LB_ADDRESS=<node-ip>:$NODE_PORT

On RKE2, tls-lb also binds the node's :443 directly (hostPort); if RKE2's bundled rke2-ingress-nginx already owns :443, set tlsLb.hostPort.enabled: false and reach it through the NodePort. Then set LB_ADDRESS to whatever address terminates at tls-lb.

6. Verify the enclave and chat

From your laptop, prove the front door is a genuine TEE in your cluster before the first prompt leaves your machine. The CLI check pins the LB launch measurement:

c8s verify "https://$LB_ADDRESS" --kind lb --measurements <LB_LAUNCH_DIGEST>
# exit code 0: evidence verified against the pinned measurement

Then chat over the sealed channel with c8s-verify — a zero-build ES module you vendor (not an npm package). Follow its installation — npm install mlkem-wasm, then map the c8s-verify specifier to the vendored src/index.js — before node chat.mjs. It re-verifies the attestation, checks the cert chains to your pinned mesh CA, and opens the post-quantum over-encrypted tunnel; keep requests non-streaming (the tunnel seals whole request/response envelopes):

// chat.mjs
import { C8sClient } from "c8s-verify";
import { readFileSync } from "node:fs";

const client = new C8sClient({
  baseUrl: `https://${process.env.LB_ADDRESS}`,
  measurements: [process.env.LB_MEASUREMENT],     // pinned LB launch digest
  meshCaPem: readFileSync("mesh-ca.pem", "utf8"), // pinned cluster anchor
});

const session = await client.connect(); // throws C8sVerifyError on any failure

const res = await session.fetch("/v1/chat/completions", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({
    model: "Qwen/Qwen3-0.6B",
    messages: [{ role: "user", content: "hello from outside the enclave" }],
    max_tokens: 32,
  }),
});
console.log(JSON.parse(res.text()).choices[0].message.content);

LB_ADDRESS=$LB_ADDRESS \
LB_MEASUREMENT=<sha384-launch-digest-of-the-lb> \
NODE_TLS_REJECT_UNAUTHORIZED=0 \
node chat.mjs

Why NODE_TLS_REJECT_UNAUTHORIZED=0 is fine here: you dialed a bare IP, and the LB's outer TLS cert is its CDS-issued one, which web PKI doesn't know. In this design the outer TLS layer is untrusted transport by definition — trust comes from the attestation evidence, the pinned measurement, and the pinned mesh CA, and every request is sealed inside the over-encrypted channel, so a hostile TLS terminator sees only ciphertext. The env var relaxes only the outer web-PKI check, for this process. (c8s verify above does the equivalent internally.) For a real deployment you'd put a public certificate on the LB via tlsLb.publicTLS and drop the env var.

For a full interactive client — REPL loop, failure drills, what each check means — see Verified chat over confidential vLLM; the session API is identical.

What you've proven

A multi-component serving system — not just a single pod — running with the threat model intact:

Every image was admitted by digest — Dynamo components and infrastructure alike; the node refused each container until its digest was allowlisted.
Each serving component holds a hardware-rooted identity — the frontend and worker carry CDS-issued certificates bound to TEE attestation, visible in kubectl get cwl.
The request path stays inside the trust boundary — client → LB enclave over the over-encrypted channel, LB → frontend → worker over attested pod-IP hops; on this single-node cluster the whole graph shares one confidential boundary.
The client verified before it trusted — pinned launch measurement, pinned mesh CA, fail-closed.

Where to go next

The Dynamo Kubernetes Platform — running the Dynamo operator instead of plain Deployments? The same two ingredients apply: allowlist the platform images, and put confidential.ai/cw on the pods via spec.services.<Name>.extraPodMetadata.annotations (v1alpha1) or spec.components[].podTemplate.metadata.annotations (v1beta1) in a DynamoGraphDeployment.
Real engines need GPUs — on a GPU-equipped c8s cluster they just run: node-as-CVM nodes schedule GPUs as normal Kubernetes resources, and under Kata, GPU pods become confidential VMs with the GPU passed through — see Kata containers and Limitations for the current gaps. The swap is the worker's image, command, and a GPU resource limit — not the c8s wiring.
Classical model serving — Serving a confidential model with KServe runs the same pattern with KServe's InferenceService.
Automate the digests — wire c8s allowlist add into CI: Automating the allowlist.