The verification model

Who verifies what in a c8s cluster — services verifying each other automatically over RA-TLS, an operator verifying a component by hand, and an end consumer verifying the front door from a browser it cannot extend — plus the over-encryption design that makes the last one possible.

Attestation is only as good as somebody's ability to check it. In a c8s cluster three parties do, and they reach the same enclave by different routes:

WhoWhenHow
Services, on each othercontinuously, on every connectionRA-TLS — evidence in the peer's serving certificate, checked mid-handshake
The operator, on a componentafter install, then on a schedule from CIc8s verify — dial it, pull its evidence, check it against a pinned measurement
The end consumer, on the front doorper session, from a browserover-encryption — a second, attested channel inside the ordinary HTTPS one

The first two both pull evidence out of an X.509 extension mid-handshake. Browsers expose no API to do that, so the consumer path cannot work the same way — and the consumer is the party whose independent check matters most. The design that works anyway is the bulk of this page.

Service-to-service verification (RA-TLS)

Inside the cluster you rarely verify by hand. Backend components authenticate each other with RA-TLS: a workload's get-cert and the ratls-mesh proxies pull the peer's attestation evidence out of its serving certificate during the handshake, check it against the mesh CA, and only then complete the connection. That verification is automatic and continuous — there is no operator step.

Two things a CDS-issued leaf now carries, which change what a relying party can re-check:

  • The evidence CDS accepted at issuance, stamped onto the leaf. A peer doing RA-TLS-mode verification sees exactly what CDS verified instead of falling back to the CA chain alone.
  • The pod sandbox ID, when the requester presented an inventory-signed sandbox token. It sits in the leaf's signed area, not in REPORTDATA — the mesh CA signature is what vouches for it, which is why --sandbox-id below requires --mesh-ca.

The mesh's RA-TLS verifier accepts the snp, gcp-snp, az-snp, tdx, and az-tdx evidence shapes. az-tdx is an Azure confidential VM's TD quote wrapped in the vTPM HCL report — the Azure TDX path has no /dev/tdx-guest — and pins MRTD as the launch measurement.

What you verify by hand is the trust root: confirm the CDS itself is a genuine TEE at your expected measurement with c8s cds verify. Everything else chains to it.

The verifier engine

c8s verify does not embed a verifier of its own. It verifies in-process with attestation-go — the Go port of the same attestation-rs engine the cluster runs — so there's no container to launch and no service to reach. Using the cluster's own engine means the product line never has to be supplied by hand: it auto-detects the platform and AMD product, including Zen4c (Siena / Bergamo), which stock go-sev-guest cannot classify. Any AMD collateral that isn't shipped inline (e.g. the VCEK for a bare report) is fetched from AMD KDS. Intel TDX quotes verify against an Intel SGX Root CA pinned inside the verifier and fingerprint-checked when it loads — no Intel endpoint is contacted.

The vendor signature chain, the REPORTDATA binding, the debug-guest rejection, and the minimum-TCB floor are enforced inside the verifier. The launch-measurement allowlist (--measurements) has no verifier-side input, so it is enforced client-side and fails closed: if the report's launch digest isn't in your allowlist, verification fails.

What the front door exposes

The consumer-facing surface is the c8s Load Balancer (tls-lb): an nginx front end plus a cds-attest sidecar, both inside the TEE (its own CVM under Pod-as-CVM, the node's under Node-as-CVM). The sidecar ships enabled by default (tlsLb.attest.enabled), so a stock install already answers browser verifiers; install with --attest=false to omit it.

PathServed byCarries
/.well-known/c8s/attestationcds-attestnonce-bound evidence, session keys, mesh leaf + CA, identity proof
/.well-known/c8s/handshakecds-attestthe hybrid key-agreement exchange
/.well-known/c8s/tunnelcds-attestover-encrypted application records
/v1/discoverynginxthe LB's issued cert + evidence with the VCEK inline
/allowlist, /allowlist/nginx → in-pod proxyCDS's image allowlist API
/.well-known/cds-cert.pem, /.well-known/mesh-ca.pemnginxstatic PEM files for in-cluster discovery

All of those locations answer any origin by default. tlsLb.cors.protocolEndpoints (default true) puts Access-Control-Allow-Origin: * on them, with no Access-Control-Allow-Credentials, and short-circuits preflight OPTIONS at nginx with a 204 advertising GET, POST, OPTIONS, the headers Authorization, Content-Type, X-C8s-Session, and a 600-second cache. That is deliberate: these endpoints exist so any browser anywhere can verify the cluster, every response is either self-authenticating (hardware evidence, CDS-signed certificates, sealed tunnel records) or public by design, and no request on them relies on ambient browser credentials — allowlist writes are authorized by an operator-signed token over method, path, and body, not by a cookie.

Two boundaries to keep straight:

  • Workload traffic is not covered. tlsLb.routes and the catch-all upstream get no CORS headers from this default. Setting the operator's own tlsLb.cors.enabled: true block replaces the built-in policy everywhere, protocol endpoints included; tlsLb.cors.protocolEndpoints: false removes CORS from the protocol endpoints entirely.
  • Reading from this front door is not verification. /.well-known/mesh-ca.pem and a plain GET /allowlist both arrive over ordinary TLS through whatever terminator sits in front of the LB. They are discovery material, not trust anchors — pin the mesh CA out of band (below), and see Image allowlist for the attested way to read the allowlist.

The idea: over-encryption, not certificate extensions

Instead of hiding attestation inside the TLS certificate, the LB serves it over a normal HTTPS endpoint, via a challenge/response, and the client establishes a second, attested channel inside the existing connection:

 ┌─────────────────┐   ┌─────────────────┐   ╔══════════════════════╗
 │     Browser     │   │  TLS terminator │   ║    LB inside a TEE   ║
 │    c8s-verify   │   │    untrusted    │   ║  nginx + cds-attest  ║
 └────────┬────────┘   └────────┬────────┘   ╚══════════┬═══════════╝
          │                     │                       │
          │ GET .../attestation?nonce=N   (N = 32 bytes)│
          ├─────────────────────┼──────────────────────►│
          │                     │      ┌────────────────┴───────────┐
          │                     │      │ mint per-session X25519 +  │
          │                     │      │ ML-KEM-768 key             │
          │                     │      │ transcript = H(version,    │
          │                     │      │   mesh CA, mesh leaf,      │
          │                     │      │   both pubkeys, N)         │
          │                     │      │ HW report_data = transcript│
          │                     │      │ mesh leaf signs transcript │
          │                     │      └────────────────┬───────────┘
          │ evidence + pubkeys + leaf/CA PEM + proof    │
          │◄────────────────────┼───────────────────────┤
 ┌────────┴─────────────────┐   │                       │
 │ verify in WASM:          │   │                       │
 │  hardware chain          │   │                       │
 │  measurement ∈ allowlist │   │                       │
 │  report_data = transcript│   │                       │
 │  leaf → pinned mesh CA   │   │                       │
 │  proof signature         │   │                       │
 └────────┬─────────────────┘   │                       │
          │ POST .../handshake  (client X25519 + ML-KEM ciphertext)
          ├─────────────────────┼──────────────────────►│
          │◄────────────────────┼───────────────────────┤ session_id
          │  AES-256-GCM = HKDF(secrets, salt=transcript)
          │ POST .../tunnel  + X-C8s-Session   (sealed) │
          ├─────────────────────┼──────────────────────►│
          │◄────────────────────┼───────────────────────┤ sealed response
          ▼                     ▼                       ▼

The plain HTTPS hop is never trusted; every check runs on the returned payload, and the channel terminates inside the LB's enclave.

A malicious TLS-terminating proxy may sit in front of the real LB. Verification is performed entirely on the returned payload, and only then does the client derive an end-to-end encrypted channel to the attested per-session key. Everything after that is confidential to the LB's enclave regardless of the outer TLS terminator.

The post-quantum over-encryption channel

The channel uses a hybrid KEM — X25519 (classical, WebCrypto) + ML-KEM-768 (post-quantum, mlkem-wasm) — following the TLS X25519MLKEM768 convention:

  1. Encapsulate against the attested ML-KEM key → (mlkem_ct, mlkem_ss).
  2. ECDH an ephemeral X25519 key with the attested X25519 key → x25519_ss.
  3. ikm = mlkem_ss ‖ x25519_ss.
  4. key = HKDF-SHA256(ikm, salt = transcript, info = "c8s-verify/v1/over-encryption", L = 32)AES-256-GCM.

The salt is the identity transcript the hardware report committed to, not the bare nonce, so the derived key is worthless to anyone who could not produce that report.

Application traffic then rides a single POST /.well-known/c8s/tunnel endpoint as CBOR, one AES-256-GCM record per message with a fresh 12-byte IV, keyed to a session the client names with the X-C8s-Session header. The entire request is sealed — method, path, headers, and body — so a proxy in front of the LB sees only ciphertext, not even the path or Authorization header. The LB enclave opens the record, forwards the plaintext request to the backend over the cluster's RA-TLS mesh, and seals the response back.

Three limits worth knowing:

  • Replay is rejected. Each channel remembers the IV of every record it has successfully opened and refuses a repeat, so the untrusted terminator cannot resubmit a captured request for a second authenticated backend action. The set is bounded (4096 records); past that the channel fails closed and must be re-established.
  • Sessions expire. --session-ttl (default 5 minutes) bounds both the pending handshake between the attestation fetch and the POST /handshake, and the idle life of an established session.
  • Records are size-capped. A sealed request is read up to 8 MiB; an upstream response larger than 32 MiB is refused rather than sealed.

Streaming responses

The tunnel is request/response: cds-attest reads the whole backend response, seals it as one record, and returns it. A Server-Sent Events body from a streaming inference backend therefore arrives in a single chunk at the end, and is bounded by the 32 MiB response cap.

Streaming works on the plain front-door path — the nginx catch-all upstream, without over-encryption. That location sets proxy_buffering off and proxy_http_version 1.1, so tokens are forwarded as the backend emits them instead of being accumulated and released in clumps. The two are a real trade: over-encryption to the enclave, or token-by-token streaming.

Why cluster identity is pinned

The client pins two things out of band: the LB measurement allowlist and the mesh CA certificate. Pinning a measurement alone is not enough, and the reason is cluster identity:

The CDS and LB images are open source and reproducible — which is what makes them auditable — but it also means a valid measurement only proves "a genuine instance of the audited code on real silicon," not "my cluster." An attacker could stand up their own genuine LB enclave (same image, valid measurement, real vendor chain) and proxy you to it. The one value that is unique per cluster is the mesh CA key, generated inside the CDS TEE — so the client must pin something cluster-unique, and today that is the mesh CA certificate.

The binding closes the copy attack: report_data commits the exact mesh leaf and its issuing CA, and the LB signs the same transcript with the leaf's private key. Copying a victim cluster's public certificate chain gets an attacker nothing without that private key, which never leaves the CDS-issued pod's TEE. There is no legacy or downgrade binding — the attestation endpoint rejects a binding parameter outright, and rejects any nonce that is not exactly 32 bytes.

The proof signature is ECDSA, so cluster authentication is classical. X25519 + ML-KEM-768 gives the session key hybrid confidentiality; this path does not claim post-quantum authentication.

Transitivity of trust

The consumer only verifies the LB. Because the LB and CDS are open-source c8s components whose code anyone can audit, and because the LB's leaf chains to the CDS mesh CA — as do all other pods in the cluster — relying on the LB as a single point of trust transitively vouches for the backend pods it talks to over RA-TLS. The c8s mesh carries the rest.

See also