Deploy NVIDIA Dynamo

1. Deploy the discovery and event planes

Plain-Deployment Dynamo (no Dynamo operator) uses etcd for endpoint discovery, and its KV-event plane defaults to NATS under etcd discovery — so deploy both, single-replica. This etcd is Dynamo's own registry, deployed like any app — it is not the Kubernetes control plane's etcd, which stays untouched (and on AKS isn't even reachable).

These are infrastructure, not confidential workloads: they carry routing metadata, and under node-as-CVM they already run inside the node's confidential boundary. Don't annotate them — c8s identities belong on the components that terminate model traffic.

# dynamo-infra.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: etcd
  namespace: workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: etcd
  template:
    metadata:
      labels:
        app: etcd
    spec:
      containers:
        - name: etcd
          image: quay.io/coreos/etcd:v3.6.13
          command:
            - /usr/local/bin/etcd
            - --listen-client-urls=http://0.0.0.0:2379
            - --advertise-client-urls=http://etcd.workloads.svc:2379
          ports:
            - containerPort: 2379
---
apiVersion: v1
kind: Service
metadata:
  name: etcd
  namespace: workloads
spec:
  selector:
    app: etcd
  ports:
    - port: 2379
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nats
  namespace: workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nats
  template:
    metadata:
      labels:
        app: nats
    spec:
      containers:
        - name: nats
          image: nats:2.14.3-alpine
          ports:
            - containerPort: 4222
---
apiVersion: v1
kind: Service
metadata:
  name: nats
  namespace: workloads
spec:
  selector:
    app: nats
  ports:
    - port: 4222

kubectl create namespace workloads   # skip if it exists from another tutorial
kubectl apply -f dynamo-infra.yaml

The pods won't start yet — their images aren't on the allowlist. That's enforcement working; Step 3 fixes it.

2. Deploy the frontend and worker as confidential workloads

Both components run from one public image: dynamo-frontend is NVIDIA's non-CUDA Dynamo image, and it carries the whole ai-dynamo Python package — dynamo.frontend and dynamo.mocker — so the two Deployments differ only in command: (which overrides the image's entrypoint). It pulls anonymously from nvcr.io.

The confidential.ai/cw annotation opts each Deployment into c8s: the webhook injects the c8s-cert identity sidecar (a CDS-issued, TEE-bound certificate) plus a c8s-cert-wait init container that holds the app until that first certificate exists, and the operator mints a headless Service per workload id — c8s-dynamo is what the front door will dial in Step 5. The components find etcd and NATS via the env vars Dynamo reads (ETCD_ENDPOINTS, NATS_SERVER); requests then flow frontend → worker over Dynamo's TCP request plane. The --model-path names the tokenizer the mocker fetches — no weights are loaded.

# dynamo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dynamo-frontend
  namespace: workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dynamo-frontend
  template:
    metadata:
      labels:
        app: dynamo-frontend
      annotations:
        # workload id → headless Service c8s-dynamo
        confidential.ai/cw: dynamo
    spec:
      containers:
        - name: frontend
          image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1
          command:
            - python3
            - -m
            - dynamo.frontend
            - --http-port
            - "8000"
          ports:
            - containerPort: 8000
          env:
            - name: ETCD_ENDPOINTS
              value: http://etcd.workloads.svc:2379
            - name: NATS_SERVER
              value: nats://nats.workloads.svc:4222
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dynamo-worker
  namespace: workloads
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dynamo-worker
  template:
    metadata:
      labels:
        app: dynamo-worker
      annotations:
        confidential.ai/cw: dynamo-worker
    spec:
      containers:
        - name: worker
          image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1
          command:
            - python3
            - -m
            - dynamo.mocker
            - --model-path
            - Qwen/Qwen3-0.6B
          env:
            - name: ETCD_ENDPOINTS
              value: http://etcd.workloads.svc:2379
            - name: NATS_SERVER
              value: nats://nats.workloads.svc:4222

kubectl apply -f dynamo.yaml

On a GPU cluster the worker's command would run dynamo.vllm --model Qwen/Qwen3-0.6B on a CUDA runtime image — the c8s wiring around it stays identical. (And if you'd rather serve from a minimal image you control, pip install ai-dynamo==1.2.1 on python:3.12-slim provides the same entrypoints — build it, push it, and allowlist that digest instead.)

3. Allow the images

Everything is deployed and nothing runs — check the events and you'll see c8s enforcement denying container creation at the node:

kubectl get events -n workloads | grep allowlist
# Warning  Failed  ...  image not in allowlist: quay.io/coreos/etcd:v3.6.13

Add the three digests, signed with your operator key. The chart publishes CDS's full /allowlist API through tls-lb, but the front door has no address yet on this cluster — you point it at Dynamo in Step 5 — so reach the CDS directly for now. It has no public ingress, so port-forward it:

kubectl port-forward -n c8s-system svc/c8s-cds 8443:8443 &

CDS_URL=https://localhost:8443
CDS_MEASUREMENT=<CDS_LAUNCH_DIGEST>

DYNAMO_IMAGE=nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1
c8s allowlist add "$(crane digest "$DYNAMO_IMAGE")" "$DYNAMO_IMAGE" \
  --url "$CDS_URL" \
  --measurements "$CDS_MEASUREMENT" \
  --operator-key operator.key

ETCD_IMAGE=quay.io/coreos/etcd:v3.6.13
c8s allowlist add "$(crane digest "$ETCD_IMAGE")" "$ETCD_IMAGE" \
  --url "$CDS_URL" \
  --measurements "$CDS_MEASUREMENT" \
  --operator-key operator.key

NATS_IMAGE=nats:2.14.3-alpine
c8s allowlist add "$(crane digest "$NATS_IMAGE")" "$NATS_IMAGE" \
  --url "$CDS_URL" \
  --measurements "$CDS_MEASUREMENT" \
  --operator-key operator.key

One port-forward covers it: the CLI verifies the CDS's RA-TLS attestation in-process before any write, so nothing else has to be reachable. --measurements is the CDS launch digest — on a node-as-CVM cluster, the node's own measurement. Leave it out and the CLI warns no --measurements set; accepting any attested endpoint build (UNSAFE), which is survivable on a tutorial cluster and not in production: an unpinned write can land on a rogue CDS.

Once tls-lb has a public address (Step 5) you can drop the port-forward entirely and pass --url "https://$LB_ADDRESS" --measurements <LB_LAUNCH_DIGEST> instead: the CLI verifies the front door's discovery attestation and the write is proxied on to CDS. The operator private key stays on your laptop either way — it only ever signs a short-lived token bound to that one request body.

The enforcement plugins poll the CDS-served list, so the kubelet's next retry succeeds — the Dynamo image is large, so its first pull takes a few minutes:

kubectl get pods -n workloads -w
# etcd-…             1/1   Running
# nats-…             1/1   Running
# dynamo-frontend-…  2/2   Running   ← app + injected c8s-cert sidecar
# dynamo-worker-…    2/2   Running

Both serving components now hold TEE-bound identities. Mirror that into kubectl with two optional ConfidentialWorkload CRs (a status view — injection works without them):

# dynamo-cwl.yaml
apiVersion: confidential.ai/v1alpha2
kind: ConfidentialWorkload
metadata:
  name: dynamo
  namespace: workloads
spec:
  workloadRef:
    kind: Deployment
    name: dynamo-frontend
---
apiVersion: confidential.ai/v1alpha2
kind: ConfidentialWorkload
metadata:
  name: dynamo-worker
  namespace: workloads
spec:
  workloadRef:
    kind: Deployment
    name: dynamo-worker

kubectl apply -f dynamo-cwl.yaml
kubectl get cwl -n workloads
# NAME            WORKLOAD          ATTESTED   TOTAL
# dynamo          dynamo-frontend   1          1
# dynamo-worker   dynamo-worker     1          1

4. Smoke-test the serving graph

Before wiring the front door, confirm discovery worked end to end: the worker registered in etcd, and the frontend found it. kubectl port-forward reaches the pod over the kubelet's own path, so it works against confidential pods:

kubectl port-forward -n workloads deploy/dynamo-frontend 8000:8000 &

curl -s localhost:8000/v1/models
# {"object":"list","data":[{"id":"Qwen/Qwen3-0.6B", ...}]}

curl -s localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model": "Qwen/Qwen3-0.6B",
       "messages": [{"role": "user", "content": "hello"}],
       "max_tokens": 32}'
# {"id":"...","choices":[{"message":{"role":"assistant", ...

kill %%   # stop this port-forward (the most recent background job)

The completion text is mock output — the mocker simulates engine scheduling and token timing, not language. What it proves is the part this guide is about: requests traverse frontend → router → worker across attested, identity-bearing components.

5. Point the front door at the frontend

External clients come in through the chart's front door — tls-lb (public TLS + over-encryption) → your frontend, dialed over the node mesh's attested mTLS. Point it there with --workload-ref + --upstream: the ref's :8000 (the frontend's --http-port) derives the upstream c8s-dynamo.workloads.svc.cluster.local:8000 — the minted headless Service, whose DNS returns pod IPs the node mesh wraps in attested mTLS; a Service VIP it cannot. service.type: LoadBalancer gives the front door a public IP on AKS:

# values.yaml
tlsLb:
  service:
    type: LoadBalancer

Apply it by re-running the c8s install from the Azure tutorial with the two flags and -f values.yaml added (the front door has one catch-all upstream — this repoints it if another tutorial's workload was wired first). --workload-ref also stamps confidential.ai/cw: dynamo onto the Deployment's pod template — a no-op here, since Step 2's manifest already carries it:

c8s install \
  --single-node \
  --cvm-mode aks \
  --operator-keys operator.pub \
  --workload-ref dynamo=workloads/deployment/dynamo-frontend:8000 \
  --upstream dynamo \
  -f values.yaml

Repeat the flags from the original install: --cvm-mode is required on every run, and the values the flags derive take precedence over anything a -f file sets on the same keys.

Then grab the public address Azure assigns — it's <pending> for a minute or so:

kubectl get svc -n c8s-system c8s-tls-lb -w
# NAME         TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)
# c8s-tls-lb   LoadBalancer   10.0.…       20.…          443:…/TCP

LB_ADDRESS=<EXTERNAL-IP>

NODE_PORT=$(kubectl get svc -n c8s-system c8s-tls-lb -o jsonpath='{.spec.ports[0].nodePort}')
LB_ADDRESS=<NODE_IP>:$NODE_PORT

6. Verify the enclave and chat

From your laptop, prove the front door is a genuine TEE in your cluster before the first prompt leaves your machine. The CLI check pins the LB launch measurement:

c8s verify "https://$LB_ADDRESS" --kind lb --measurements <LB_LAUNCH_DIGEST>

✓ VERIFIED  (attestation-go backend)
  source:       discovery document https://20.…/v1/discovery
  platform:     az-snp
  measurement:  9f2c…
  …

Exit code 0 means verified; 2 is a verification or policy failure and 3 means no evidence could be fetched, so the same command works unchanged as a CI gate. See the verification CLI for the rest of its checks.

Then chat over the sealed channel with the c8s-verify JavaScript client. Set it up first — see its installation — then run node chat.mjs. It re-verifies the attestation, checks the cert chains to your pinned mesh CA, and opens the post-quantum over-encrypted tunnel. Keep these requests non-streaming: the tunnel seals one whole request envelope and one whole response envelope, so tokens can't arrive incrementally through it. (Streaming does work over tls-lb's ordinary route, which forwards Server-Sent Events straight through instead of buffering them — but that gives you TLS to the front door, not a channel sealed to the enclave.)

// chat.mjs
import { C8sClient } from "c8s-verify";
import { readFileSync } from "node:fs";

const client = new C8sClient({
  baseUrl: `https://${process.env.LB_ADDRESS}`,
  measurements: [process.env.LB_MEASUREMENT],     // pinned LB launch digest
  meshCaPem: readFileSync("mesh-ca.pem", "utf8"), // pinned cluster anchor
});

const session = await client.connect(); // throws C8sVerifyError on any failure

const res = await session.fetch("/v1/chat/completions", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({
    model: "Qwen/Qwen3-0.6B",
    messages: [{ role: "user", content: "hello from outside the enclave" }],
    max_tokens: 32,
  }),
});
console.log(JSON.parse(res.text()).choices[0].message.content);

LB_ADDRESS=$LB_ADDRESS \
LB_MEASUREMENT=<LB_LAUNCH_DIGEST> \
NODE_TLS_REJECT_UNAUTHORIZED=0 \
node chat.mjs

Why NODE_TLS_REJECT_UNAUTHORIZED=0 is fine here: you dialed a bare IP, and the LB's outer TLS cert is its CDS-issued one, which web PKI doesn't know. In this design the outer TLS layer is untrusted transport by definition — trust comes from the attestation evidence, the pinned measurement, and the pinned mesh CA, and every request is sealed inside the over-encrypted channel, so a hostile TLS terminator sees only ciphertext. The env var relaxes only the outer web-PKI check, for this process. (c8s verify above does the equivalent internally.) For a real deployment you'd put a public certificate on the LB via tlsLb.publicTLS and drop the env var.

For a full interactive client — REPL loop, failure drills, what each check means — see Verified chat over confidential vLLM; the session API is identical.