# Confidential — Full Documentation > The AI confidential compute platform. Complete content from all pages. ## TLDR Confidential runs AI workloads (inference, training, agents) inside hardware-encrypted Trusted Execution Environments (TEEs). Data and code stay private during processing, can't be tampered with, and both claims are cryptographically verifiable without trusting Confidential. Deploy code unchanged — no performance penalty (~1.5% overhead on CPUs, max 2.08% observed). **Use Confidential two ways:** - **Confidential Cloud** — hosted TEE-backed infrastructure: pay-per-token inference, GPU and CPU VMs, and a Confidential Agents API for spinning up private agent environments. - **Enterprise / Licensed** — deploy the modular component stack on your own infrastructure (on-prem, bare metal, all major clouds). **Components:** Attestation service, CDS (attestation-gated secrets), attested build pipeline (SLSA L3), TEE networking (mTLS + multi-recipient hybrid encryption), OHTTP gateway, control plane (with optional ZK proofs), hardened VM image, client/server SDKs. All modular — use individually or together. **Enterprise model:** Discovery (free) → Pilot (1-4 months, deploy on your infra) → Production. On-prem, bare metal, all major clouds. **Key benchmarks (AMD EPYC, SEV-SNP, cryptographic proving workloads):** | Metric | TEE | Regular | Difference | | ---------------- | ------- | ------- | ---------- | | Execution Speed | 5.16 kHz | 5.26 kHz | -1.90% | | Runtime Duration | 89.46s | 87.78s | +1.91% | | CPU Utilization | 26.61% | 26.23% | +1.45% | | Memory Usage | 19.02 GB | 18.87 GB | +0.80% | **Contact:** [hello@confidential.ai](mailto:hello@confidential.ai) --- # Confidential AI The confidential computing stack for AI at scale. We run your AI workloads (inference, agents, training) in hardware-protected Trusted Execution Environments (TEEs). Data and weights stay private and untamperable while being processed. ## Use Cases * **Private inference.** Guarantee data privacy during inference. Customer prompts, responses, and model interactions are never visible. * **Confidential Agents.** Agents run inside TEEs with hardware-enforced credential isolation. Tokens and API keys never exist in plaintext outside a TEE. * **Model weight protection.** Protect proprietary model weights during inference, training, and fine-tuning. Weights never leave hardware-isolated enclaves. * **Private training.** Train on sensitive data and cryptographically prove exactly what data was used. ## Get Started Run AI workloads on our infra, see [Cloud](/cloud). For enterprise and on-prem deployments, or anything else, [contact us](mailto:hello@confidential.ai). ## Member of * [Confidential Computing Consortium](https://confidentialcomputing.io/) * [Linux Foundation](https://www.linuxfoundation.org/) * [Nvidia Inception](https://www.nvidia.com/en-us/startups/) --- # Confidential Cloud Run AI workloads on TEE-backed infrastructure with end-to-end attestations of hardware-enforced integrity, privacy and verifiability. No code changes. No TEE expertise required. For detailed rates, see [Pricing](/pricing.md). **Enterprise & on-prem.** Prefer to deploy on your own infrastructure? We license Confidential's modular stack to AI labs and infrastructure providers, on-prem, bare metal, all major clouds. [Contact sales](mailto:hello@confidential.ai). ### Table of Contents - [Confidential Inference](/confidential-inference) - [Confidential VMs](/confidential-vms) - [Confidential Agents](/confidential-agents) --- # Confidential Agents Spin up a private, isolated environment for your AI agent in under 15 seconds. Each instance comes preloaded with an agent runtime and inference, ready over SSH, and is completely invisible to everyone except you, including us. [Sign up for the beta here](https://forms.gle/QkfCfAjvDcujZLzB6). ## Up and running in under 15 seconds Call the API and you're SSH'd into a ready environment in under fifteen seconds. No waiting, no manual provisioning. Treat agent environments as disposable, spin one up for every task, run it, throw it away. ## Completely private, even from us Each instance runs inside a Confidential VM with hardware-enforced memory encryption. The code, data, and agent state inside are invisible to us, to other customers, and to the infrastructure operators we run on. You hold the SSH key; we have no way to look inside. ## Agents, preloaded Every instance launches with the OpenClaw agent runtime installed and ready to go. Additional agents available soon. ## Confidential inference, included Every instance gets attested inference out of the box, open-weight models served from the TEE pool, with no separate setup. See [Confidential Inference](/confidential-inference) for the model lineup. ## How it works 1. `POST /v1/instances` with your SSH public key. 2. Poll `GET /v1/instances/{name}` until `status` is `ready`. 3. `ssh ..confidential.ai`, run your workload inside the TEE. 4. `DELETE /v1/instances/{name}` when done. ```bash curl -sS -X POST https://api.confidential.ai/v1/instances \ -H "Authorization: Bearer $CA_API_KEY" \ -H "Content-Type: application/json" \ -d "{\"public_key\": \"$(cat ~/.ssh/id_ed25519.pub)\", \"agent\": \"openclaw\"}" ``` See the [Agents API reference](/docs/agents-api/reference) for the full request and response surface. ## Get access API keys are issued per organization. [Contact us](mailto:hello@confidential.ai) to request access, and see the [Agents API reference](/docs/agents-api/reference) for the full endpoint surface. --- # Confidential Inference Private inference as an API. Pay per token. Send requests to open-weight models running inside TEEs on our cloud. Your prompts, responses, and model interactions are never visible to us or our infrastructure. Every response includes an attestation proof. OpenAI-compatible API. Drop-in replacement for existing inference providers. Switch your base URL and get hardware-enforced privacy with no other code changes. | Model | Best for | |---|---| | Kimi K3 | Strongest open model. Limited capacity: [join the waitlist](/kimi-k3) | | GLM 5.2 | Reasoning, multilingual | | DeepSeek V4-Pro | Reasoning, coding, long context | | Qwen3.6 27B | General purpose | | DeepSeek V4-Flash | General purpose, coding, long context | ## Using the API Point any OpenAI client at our base URL. Switch the base URL, keep the rest of your code. Every response carries an attestation you can verify. ```bash curl https://api.confidential.ai/v1/chat/completions \ -H "Authorization: Bearer $CONFIDENTIAL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-v4-pro", "messages": [{"role": "user", "content": "Explain remote attestation in one sentence."}] }' ``` ```python from openai import OpenAI client = OpenAI( base_url="https://api.confidential.ai/v1", api_key="$CONFIDENTIAL_API_KEY", ) resp = client.chat.completions.create( model="deepseek-v4-pro", # any model id from the table above messages=[{"role": "user", "content": "Explain remote attestation in one sentence."}], ) print(resp.choices[0].message.content) ``` See [inference pricing](/pricing.md#confidential-inference) for per-token rates. Model requests: [hello@confidential.ai](mailto:hello@confidential.ai). Confidential inference vs non-confidential inference: 4% lower token throughput, negligible impact on Time to First Token (TTFT). --- # Confidential VMs Dedicated VMs on our cloud, running inside TEEs. You rent the VM, we run the infrastructure. **GPU VMs.** Single-GPU and multi-GPU configurations for inference, training, fine-tuning, and containers. | GPU | VRAM | Host CPU TEE | Best for | |---|---|---|---| | RTX PRO 6000 | 96 GB GDDR7 | AMD SEV-SNP | Low cost, high performance for single GPU models | | H100 | 80 GB HBM3 | AMD SEV-SNP or Intel TDX | Training, fine-tuning, latency-sensitive inference | | B200 | 192 GB HBM3e | AMD SEV-SNP or Intel TDX | Frontier training, high performance | | B300 | 288 GB HBM3e | AMD SEV-SNP or Intel TDX | Frontier training, maximum performance | Configurations: Three confidential computing deployment modes are available, depending on GPU and workload. Single GPU pass-through attaches one GPU to one Confidential VM and is supported on all listed GPUs. Protected PCIe lets multiple GPUs share one confidential domain over PCIe, though GPU-GPU traffic over NVLink and NVSwitch is not encrypted. This mode is supported on H100 only. Multi-GPU pass-through attaches multiple independently attested GPUs to one VM with encrypted NVLink between them. This mode is supported on B200 and B300. **CPU VMs.** TEE-backed vCPUs for general-purpose confidential workloads. AMD SEV-SNP and Intel TDX available. See [Confidential VM pricing](/pricing.md#confidential-vms) for per-GPU-hour and per-core-hour rates. --- # Pricing All prices include hardware-enforced integrity, privacy and verifiability by TEEs. Two ways to run Confidential: - **Cloud.** Our infrastructure. Usage-based. Pay for what you use. - **Licensed.** Your infrastructure. Per hardware unit, per year. Run on our infra: Cloud. Run on yours: Licensed. --- ## Cloud On-demand usage. No commitments. Custom pricing is available for high-volume deals, [get in touch](mailto:hello@confidential.ai). ### Confidential Inference | Model | Input (per 1M tokens) | Input Cached (per 1M tokens) | Output (per 1M tokens) | | ----------------- | --------------------- | ---------------------------- | ---------------------- | | Kimi K3 | $3.00 | $0.30 | $15.00 | | GLM 5.2 | $1.25 | $0.26 | $4.00 | | DeepSeek V4-Pro | $0.60 | $0.06 | $1.25 | | Qwen3.6 27B | $0.30 | $0.025 | $2.00 | | DeepSeek V4-Flash | $0.20 | $0.018 | $0.40 | **Kimi K3**: private inference on dedicated hardware. Capacity is limited, so access goes through the waitlist: [join it here](/kimi-k3) for access and pricing. Additional models available on request: [hello@confidential.ai](mailto:hello@confidential.ai). ### Confidential VMs **GPU VMs**: billed per GPU-hour. | GPU | VRAM | Host CPU TEE | Per GPU-Hour | | ------------ | ------------ | ------------------------ | ------------ | | RTX PRO 6000 | 96 GB GDDR7 | AMD SEV-SNP | $1.90 | | H100 | 80 GB HBM3 | AMD SEV-SNP or Intel TDX | $3.25 | | B200 | 192 GB HBM3e | AMD SEV-SNP or Intel TDX | $6.50 | | B300 | 288 GB HBM3e | AMD SEV-SNP or Intel TDX | $7.50 | **CPU VMs**: billed per vCPU core-hour, plus per GB-hour of RAM. | TEE Backend | Per Core-Hour | Per GB-Hour (RAM) | | ----------- | ------------- | ----------------- | | AMD SEV-SNP | $0.05 | $0.012 | | Intel TDX | $0.05 | $0.012 | --- ## Licensed Deploy the Confidential stack on your own infrastructure. On-prem, bare metal, all major clouds. One stack. Licensed per enabled hardware unit, per year. ### What you get The full confidential-computing stack, one bundle: - **Confidential Metal**: attestable, verifiable confidential VMs on bare metal. - **C8s**: Confidential Kubernetes. Scale AI workloads to datacenter scale. - **AI Workload Services**: Confidential-optimized inference, training, fine-tuning, agents. - **Confidential OS**: hardened VM guest OS for development and production. - **Client Libraries & SDKs**: clients verify confidentiality claims themselves. No per-component line items. Software updates included. Standard support included. ### How pricing works License per GPU per year, or per CPU-core per year. CPU-cores in machines with GPUs are covered by the GPU license. Pricing is annual. Same prices in every region. ### GPU licenses Per GPU, per year: | GPU class | Per GPU-Year | | ------------ | ------------ | | RTX PRO 6000 | $2,789 | | H100 | $3,876 | | H200 | $4,974 | | B200 | $7,625 | | B300 | $9,884 | Each license covers one GPU of that class. Licenses are not tied to unique GPUs; they transfer freely across GPUs of the same class. GPU prices include Nvidia's requisite Confidential Computing (CC) SKU license. We handle CC licensing with Nvidia on your behalf. No action required. ### CPU-core licenses Per CPU-core, per year: | | Per CPU-core-Year | | ------------- | ----------------- | | CPU-core only | $48.85 | Covers Intel and AMD cores. Physical cores, not vCPUs or hyperthreads. Licenses transfer freely across cores. Not required for cores in GPU machines. ### Support | | Standard | Enterprise | | ---------------------------- | --------------- | ----------------------------------- | | Price | Included | 25% of total license price | | Coverage | Business hours | 24/7 for production outages | | Response, production outage | 1 business day | 1 hour | | Contact | Shared queue | Dedicated technical account manager | ### Discounts Multi-year and volume commitments are negotiable. [Contact sales](mailto:hello@confidential.ai). ### Example 500 B200s: 500 x $7,625 = $3,812,500/year, invoiced monthly at about $317,708. That is $635 per GPU per month, with the full stack, Nvidia CC licensing, updates, and support included. Order of operations at larger scale: license total, any negotiated discounts, then any enterprise support uplift. Multiplicative. --- ## Getting Started Let us know what you need: [hello@confidential.ai](mailto:hello@confidential.ai). --- # Products Confidential AI is a stack of licensed software for running private AI on your infrastructure. Prompts, data, and agent state stay encrypted in memory while they run. Model weights sit in the GPU's hardware-protected region, unreadable to the host or operator. Every result carries a cryptographic attestation of what ran and where. The stack is built in layers. Confidential Metal and Confidential Kubernetes are the foundation; Confidential Inference and Confidential Agents are private AI workloads you run on top of them. License the pieces you need and run them on your own hardware or any major cloud. Confidential AI also runs all of these layers on the Confidential Cloud. ``` ┌────────────────────────────────────────────────────────┐ │ Confidential Inference / Confidential Agents │ │ private AI you run on the stack │ ╞════════════════════════════════════════════════════════╡ │ Confidential Kubernetes (C8s) │ │ one CVM becomes a platform you can scale │ ╞════════════════════════════════════════════════════════╡ │ Confidential Metal │ │ hardware becomes a CVM you can trust │ ╞════════════════════════════════════════════════════════╡ │ TEE hardware: AMD SEV-SNP / Intel TDX / NVIDIA CC │ └────────────────────────────────────────────────────────┘ ``` ## Confidential Metal The foundation for bare metal. Confidential Metal turns on confidential computing on CC-capable hardware: software that sets the BIOS and a hardened host image carrying the right TEE firmware and kernel parameters. On top of that it provides the machinery to launch CVMs you can fully measure and verify, a hardened guest image measured from firmware all the way to user space, with that measurement bound to an attested build and the attestation library baked in. Managed clouds like GCP and Azure already hand you measurable CVMs; Confidential Metal brings the same guarantees to your own bare metal. ``` ┌────────────────────────────────────────────────┐ │ Host: CC enabled │ │ BIOS + TEE firmware, host image │ └────────────────────────────────────────────────┘ │ launches ▼ ╔════════════════════════════════════════════════╗ ║ Confidential VM ║ ║ ║ ║ hardened guest image ║ ║ measured: firmware to user space ║ ║ attestation library built in ║ ╚════════════════════════════════════════════════╝ │ ▼ attestation ┌────────────────────────────────────────────────┐ │ measured, attested, trusted │ └────────────────────────────────────────────────┘ ``` ## Confidential Kubernetes One CVM is not a service. Confidential Kubernetes (C8s) turns it into a platform you can host and scale on, with confidentiality spanning the whole cluster. Every workload gets an attested identity, all traffic between components is encrypted, and the control plane stays outside the boundary, so an operator can run your workloads without ever seeing them. C8s builds on the measurable CVMs and CC-enabled hardware that Confidential Metal provides on bare metal (and that GCP and Azure provide in the cloud). ``` ┌──────────────────────────────────────────────────────┐ │ Control plane (untrusted, outside boundary) │ └──────────────────────────────────────────────────────┘ │ schedules ╔══════════════════════════════════════════════════════╗ ║ ║ ║ [ Pod CVM ] [ Pod CVM ] [ Pod CVM ] ║ ║ attested identity, encrypted mesh ║ ║ ║ ║ CDS - root of trust, issues certs ║ ║ ║ ╚══════════════════════════════════════════════════════╝ hardware-enforced trust boundary ``` ## Confidential Inference Private inference you run on top of your own confidential stack. It serves open-weight models behind an OpenAI-compatible API as a drop-in replacement: switch one base URL and your prompts and responses stay encrypted in CVM memory, the model weights stay isolated in the GPU's hardware-protected region, and every response carries an attestation. Deploy it on Confidential Kubernetes and it inherits the guarantees of the layers beneath it. ``` ┌──────────────────────────────────────────────────┐ │ Your app │ │ swap one base URL │ └──────────────────────────────────────────────────┘ │ OpenAI-compatible API ▼ ╔══════════════════════════════════════════════════╗ ║ TEE model pool ║ ║ prompts encrypted, weights hardware-isolated ║ ╚══════════════════════════════════════════════════╝ │ ▼ response + attestation ┌──────────────────────────────────────────────────┐ │ Your app │ └──────────────────────────────────────────────────┘ ``` ## Confidential Agents Private, isolated environments for your AI agents, running on top of your confidential stack. Each agent runs in its own Confidential VM with hardware-encrypted memory, ready over SSH in seconds and preloaded with an agent runtime and attested inference. The code, data, and keys inside stay invisible to the infrastructure they run on. Treat each one as disposable: spin it up for a task, run it, throw it away. ``` ┌────────────────────────────────────────────────┐ │ You hold the SSH key │ └────────────────────────────────────────────────┘ │ ssh ▼ ╔════════════════════════════════════════════════╗ ║ Agent CVM ║ ║ ║ ║ agent runtime + attested inference ║ ║ code, data, keys invisible to the host ║ ╚════════════════════════════════════════════════╝ ``` ## How they fit together Confidential Metal produces trustworthy CVMs. Confidential Kubernetes turns them into a platform that can host and scale a real service. Confidential Inference and Confidential Agents are the private AI workloads that run on top. License the whole stack and run it yourself, from the metal up, on your own hardware or any major cloud. ## Or let us run it for you Confidential Inference and Confidential Agents are also available as a managed service on our own cloud: we operate the entire stack so you get private AI behind an API without running any of it yourself. See [Cloud](/cloud), or [contact sales](mailto:hello@confidential.ai) to scope a deployment. --- # c8s-docs Style Guide How we write the c8s documentation. The goal: docs an experienced infra/Kubernetes engineer can navigate fast, trust completely, and copy-paste from — even if they've never touched confidential computing before. For *how the site is wired* and the source-of-truth rules, see [`CLAUDE.md`](./CLAUDE.md). Two standing principles behind everything below: 1. **The source code is authoritative.** Never document a flag, default, name, or behavior you haven't verified against `../c8s` (or the relevant Lunal repo). Prose that drifts from the code is worse than no prose. 2. **Be concrete.** Lead with the command, the manifest, the exact term. Prefer a runnable example over a paragraph describing one. --- ## 1. Audience Write for an **experienced Kubernetes / infrastructure engineer who is new to confidential computing.** Assume fluency with: pods, nodes, DaemonSets, CRDs, Helm, `kubectl`, ingress, TLS, container registries and digests. Do **not** assume familiarity with: TEEs, attestation, measured boot, SEV-SNP/TDX, RA-TLS, CVMs, IGVM. Introduce those the first time they appear. The reader's recurring question is *"is my data actually protected, and how do I prove it?"* Answer it. The threat model (host is adversarial) is the spine of the product — keep it visible. --- ## 2. Voice and tone **Default voice: direct, second-person, confident, security-first — and a little opinionated.** This is the established voice; preserve it. It's the voice of an engineer who has thought hard about the threat model and will tell you the truth. - **Second person.** Address the reader as "you". Use imperative mood for steps ("Run", "Set", "Verify"). - **Active, present tense.** "The CDS verifies the report and issues a leaf." Not "The report will be verified by the CDS." - **Opinionated where it earns it.** State the strong position and why. The site already does this well — keep that spine: > "The design assumption is simple and uncompromising: **the host is adversarial.**" > "This page is the honest list of what the current milestone does **not** do." - **Honest about gaps.** Never oversell. If something isn't enforced, say so plainly and link to the workaround or the limitation. Trust is the product. - **Concise.** Cut throat-clearing ("It is important to note that…", "Simply…", "In order to…"). Short sentences. One idea per sentence. ### Tone carve-outs - **Reference (`c8s/reference/**`): neutral, reference register.** No editorializing, no "you", no opinion. Just flag, type, default, effect — terse and uniform. This section is looked up, not read. - **Tutorials (`c8s/tutorials/**`): warmer and more relaxed.** A guiding, slightly conversational hand is welcome here — you're walking someone through an end-to-end run. You may say "we'll", set expectations ("this takes a few minutes"), and reassure. Keep the security framing, but lighten the formality. ### Avoid - Marketing fluff and superlatives ("blazing-fast", "seamless", "revolutionary"). - Hedging that undercuts authority ("might", "should probably") when you actually know. - Whitepaper-speak in user docs — the paper's tone is academic; the docs are operational. - Apologizing for the product, or burying a real limitation in qualifiers. --- ## 3. Page structure - **Frontmatter is required:** `title` and `description` only (both used by Fumadocs). - `title`: sentence case, concise, no trailing period. It *is* the page H1. - `description`: one or two real sentences (shown in sidebar cards and ``). Make it say what the page delivers, not "This page describes…". - **No `# H1` in the body.** Start at `##`. Don't skip levels (`##` → `###`, not `## ` → `####`). - **Headings: sentence case**, no terminal punctuation, descriptive enough to scan. Phrase them as the question the reader has or the task they're doing ("Why pod-as-CVM is not available on Azure", "Install in base mode"), not bare nouns where a task is meant. - **Lead with the point.** First paragraph states what this page is for and the one thing to take away. Then details. Don't make readers scroll for the command. - **One page, one mode** (below). This is the rule the tree is organized around. ### The four page types The docs follow [Diátaxis](https://diataxis.fr/): a page serves exactly one reader need, and the directory it lives in names that need. Decide the type *before* you write — it determines the shape, the voice, and where the page goes. | Type | Directory | The reader is… | Shape | | --- | --- | --- | --- | | **Tutorial** | `c8s/tutorials/**` | learning by doing, for the first time | One end-to-end path, followed top to bottom. Every choice made for the reader. Warmer tone, ``, expected output shown, "what you've built" at the foot. | | **How-to guide** | `c8s/how-to/**` | competent, with a goal | Prereqs → `` with copy-paste commands → how you know it worked. No background; link to the concept. | | **Reference** | `c8s/reference/**` | at work, looking something up | Neutral register, exhaustive, table-driven. No "you", no opinion. Gated against source where possible. | | **Concept** | `c8s/concepts/**` | trying to understand | Prose + ASCII diagram + comparison tables. The *why* and the trust implications. No procedures. | Three tests when a page feels wrong: - **Writing "why" inside a how-to?** It belongs on the concept page — link to it. A `` that explains rationale rather than warning about a footgun is the usual tell. - **Writing a command inside a concept page?** It belongs in a how-to. A concept page may name a command; it does not walk you through running one. - **Writing a flag table outside `reference/`?** Cut it and link to the CLI reference, which CI keeps in sync. A duplicate table drifts silently. **Tutorials may duplicate how-to guides on purpose.** The tutorial owns one path end to end, with every choice already made; the how-to owns the general case. Don't make a tutorial apologize for the overlap or send the reader away mid-flow. --- ## 4. Terminology and capitalization Use these exact forms. Source spelling beats the whitepaper every time. When an acronym first appears on a page, expand it once: "Trusted Execution Environment (TEE)". After that, the acronym alone. | Term | Use exactly | Notes / not | | --- | --- | --- | | Product name | **c8s** | Always lowercase, even at sentence start. Bold on first mention per page (`**c8s**`). Not "C8s", "C8S", "c8S". | | Confidential Kubernetes | "Confidential Kubernetes" | Capitalized product term. Not "confidential Kubernetes". | | CDS | **CDS** (Certificate Distribution Service) | The trust root. Not "cds" in prose. | | RA-TLS | **RA-TLS** (Remote-Attestation TLS) | Not "raTLS", "RATLS", "ra-tls". | | EAR | **EAR** (Entity Attestation Result) | An ES256 JWT the CDS issues. Consumed by `/handoff` and `/sign-csr` only — `/attest` returns the leaf directly, so the EAR is not part of ordinary workload cert issuance. | | Sandbox identity | **sandbox identity**, **sandbox token**, **sandbox ID** | The per-pod identity the admission inventory vouches for and the CDS stamps into the leaf. Not "workload identity". | | Inventory | **admission inventory** (first use), **inventory** after | The `nri-image-policy` / `policy-monitor` pair acting as the sandbox-identity arbiter. | | Mesh identity | **mesh identity** | The mesh leaf + issuing CA that a PQ session transcript commits to. | | TEE | **TEE** (Trusted Execution Environment) | | | CVM | **CVM** (confidential VM) | | | SEV-SNP | **AMD SEV-SNP** (first use), **SEV-SNP** after | Not "SEV/SNP", "sev-snp". | | TDX | **Intel TDX** (first use), **TDX** after | | | GPU CC | "NVIDIA Confidential Computing (CC mode)" | Not shipped — see roadmap rules. | | Deployment shapes | **Pod-as-CVM**, **Node-as-CVM** | Hyphenated, capitalized as shown. Not "pod-level CVM" (that's whitepaper). | | Modes | **Pod-as-CVM**, **Node-as-CVM** | Name the shape, not the mechanism — these are the same terms as the Deployment shapes row, and `--cvm-mode` is what selects them. Don't write "Kata mode" or "base mode"; mention Kata only where the mechanism is the actual subject. | | Allowlist | **allowlist** | One word. Not "allow-list" or "allow list". | | Measurement | "launch measurement" / "measurement" | The SHA-384 launch digest. | | Components (code names) | `c8s operator`, `attestation-api`, `ratls-mesh`, `nri-image-policy`, `policy-monitor`, `get-cert`, `cds-attest`, `tls-lb`, `volumed` | Lowercase, code-formatted as identifiers; match the source exactly. | | CRD | `ConfidentialWorkload` (short name `cwl`), group `confidential.ai`, version `v1alpha2` | | | Runtime classes | `kata-qemu`, `kata-qemu-snp`, `kata-qemu-tdx`, `kata-clh` | Code-formatted; exact. | | Annotations / labels | `confidential.ai/cw`, `confidential.ai/c8s-injected`, … | Always code-formatted and exact. | | Namespace | `c8s-system` | Default install namespace. | - **Kubernetes nouns** stay lowercase as the project does them: pod, node, namespace, container, control plane, kubelet, containerd. Capitalize only proper nouns (Kubernetes, Kata, Helm, Azure, Vercel, NVIDIA, AMD, Intel). - **Don't invent synonyms.** One concept, one name, everywhere. If the code calls it `get-cert`, the docs call it `get-cert` — not "the cert helper". --- ## 5. Mechanics - **Spelling: US English** ("behavior", "authorize", "canceled"). Matches code (`allowlist`, `authorization`). - **Oxford comma:** yes. - **Em dashes** for asides — like this — no spaces around a true em dash in prose; the existing pages use spaced em dashes (` — `), so match that for consistency. - **Numbers:** spell out zero–nine in prose; numerals for 10+ and for anything with a unit, flag, or version (`3 nodes`, `8443`, `v1alpha2`, `6h`). Always numerals in commands/tables. - **Bold** for first-use key terms and UI/identity names you want to anchor (`**mesh CA**`). Don't bold whole sentences. *Italics* sparingly, for genuine emphasis. - **Identifiers are always code-formatted:** flags (`--single-node`), files (`reference/cli.mdx`), env vars (`C8S_DIR`), values (`role=cds`), ports (`8443`), digests, paths. Never leave a flag or filename in plain prose. --- ## 6. Code blocks and shell examples Code examples are the heart of these docs — make them correct and copy-paste-ready. - **Always tag the language:** ` ```bash `, ` ```yaml `, ` ```json `, ` ```go `, ` ```text `. - **Commands are copy-pasteable.** No leading `$` prompt (it breaks copy). Show output separately if needed, in its own block or with a comment. ````md ```bash c8s install --single-node --cvm-mode=node ``` ```` - **Placeholders:** use `` for values the user must replace, and say what they are right after the block. Be consistent (``, ``). Don't mix `$VAR` and `` styles for user-supplied values. - **Multi-step procedures** use ``/``, one logical action per step, with the command in the step and a one-line "what this does / how you know it worked" after it. - **Show verification.** After an install/run step, show how the user confirms success (`kubectl get pods -n c8s-system`, expected status). This is a security product — "it should work" isn't enough; show the check. - **YAML manifests:** complete enough to apply, minimal enough to read. Annotate the c8s-specific lines (e.g. the `confidential.ai/cw` annotation) in prose, not with inline noise. - **Keep flags/defaults exact.** If you write a default (`--leader-elect` is `true`, `--port` is `8443`), it must match source. When in doubt, check `../c8s`. --- ## 7. Callouts Use Fumadocs `` deliberately — they're signposts, not decoration. One or two per page at most; if everything is a callout, nothing is. When a callout opens with a bold title (`**Why Standard mode?**`, `**Verify the node-pool shape.**`), put a **blank line after the title** so it renders as its own line, with the body as its own paragraph below. | Type | Use for | Example | | --- | --- | --- | | `type="info"` | Helpful context, orientation, "new here?" pointers. | The site's intro "this walks you end-to-end" note. | | `type="warn"` | Gotchas, footguns, and **production requirements**. | "The default chart pins **no** measurements — fine for demos, mandatory to set for production." | | `type="error"` | Genuinely dangerous / breaking actions only. | Flipping `kata.enabled` on a live cluster; running a second service mesh alongside c8s. | **Roadmap / not-yet-shipped (decided policy: shipped-first, roadmap quarantined).** Don't put unimplemented features in the happy path. When you must mention one at its point of relevance, use a short, clearly-labeled callout and link to [Limitations](/docs/c8s/concepts/limitations): ```md **Planned, not yet shipped.** GPU confidential computing (NVIDIA CC mode) is out of scope for the current milestone. See [Limitations](/docs/c8s/concepts/limitations). ``` The full gap list lives on the Limitations page and tracks `../c8s/docs/THREAT_MODEL.md`. Keep it honest and current. --- ## 8. Diagrams (ASCII-first) - **Default to ASCII art in a fenced block** (` ```text `) for new diagrams — trust boundaries, component layouts, flows, sequences. It diffs cleanly, needs no tooling, and matches the whitepaper's house style. Study `../whitepapers/c8s-whitepaper/whitepaper-md.md` and the `ascii-diagrams` skill for the conventions (double walls `╔═╗` for the CVM/trust boundary, single boxes for ordinary components, `─►`/`┈►` for solid/optional flows). - Keep ASCII diagrams **narrow enough to read** in the content column (~80 cols); align boxes on a monospace grid; add a one-line italic caption under it explaining the takeaway. - **Committed SVGs** (`public/diagrams/*.svg`) are reserved for the existing polished "hero" diagrams. Reference them with descriptive alt text: `![Base-mode certificate flow: the workload requests a challenge, generates evidence bound to its CSR, and the CDS verifies it before signing a leaf.](/diagrams/cert-flow-base.svg)`. Don't generate new SVGs for routine diagrams — reach for ASCII. - There is **no `` component** (the README is wrong). Don't author Mermaid. --- ## 9. Links and cross-references - **Internal links use absolute site paths:** `[threat model](/docs/c8s/concepts/threat-model)` and `#anchor` for sections. Not relative `../` paths, not `.mdx` extensions. - **There is no link checker** in CI — broken internal links pass the build. **Verify every link by hand**: confirm the target page exists (`content/docs/...`) and the anchor matches a real heading. Re-check anchors when you rename a heading. - **Cross-link generously but purposefully.** This is a journey-structured site; when you name a concept defined elsewhere (CDS, allowlist, RA-TLS), link its home page on first use. - **External links:** whitepaper at `/papers/c8s.pdf`; upstream projects (Kata, Fumadocs) linked to their canonical sites. Don't link to the GitHub web UI for repo operations — that's a `gh` concern, not a docs one. - When prose depends on a specific source behavior, it's fine (encouraged) to be precise about the component/flag so a reader can grep the code — but link to the *docs* page for the concept, not to source files (those move). --- ## 10. Before you call it done - [ ] Every flag, default, name, and behavior **verified against `../c8s`** (or relevant repo). - [ ] Terminology matches §4 exactly (casing, hyphenation, one name per concept). - [ ] The page serves **one** reader need and sits in the matching directory (§3). - [ ] New page added to its folder's `meta.json` `pages`; frontmatter `title` + `description` present; no `# H1` in body. - [ ] Code blocks are language-tagged and copy-paste-ready; verification steps shown. - [ ] `npm run check:links` passes (dead `/docs/...` links, dead anchors, and pages missing from a `meta.json` all fail the build). - [ ] If a `c8s` CLI flag changed: `reference/cli.mdx` updated and `C8S_DIR=/workspace/c8s npm run check:flags` passes. - [ ] Not-yet-shipped features kept out of the happy path; roadmap items labeled and linked to Limitations. - [ ] `npm run typecheck && npm run lint && npm run build` all pass. --- # Team - **Ansgar Grunseid**, Cofounder & CEO: [GitHub](https://github.com/gruns) · [LinkedIn](https://www.linkedin.com/in/ansgargrunseid/) - **Amean Asad**, Cofounder & CTO: [GitHub](https://github.com/AmeanAsad) · [LinkedIn](https://www.linkedin.com/in/ameanasad/) - **Patrick Woodhead**, Head of Product: [GitHub](https://github.com/geomatrick) · [LinkedIn](https://www.linkedin.com/in/patrick-woodhead/) - **Yolan Romailler**, Head of Security: [GitHub](https://github.com/AnomalRoil) · [LinkedIn](https://www.linkedin.com/in/anomalroil/) - **Aamir Rasheed**, Member of Technical Staff: [GitHub](https://github.com/aamirrasheed) · [LinkedIn](https://www.linkedin.com/in/aamir1rasheed/) - **João Sousa Andrade**, Member of Technical Staff: [GitHub](https://github.com/joaosa) · [LinkedIn](https://www.linkedin.com/in/thejoaosa/) - **Patrick McClurg**, Member of Technical Staff: [GitHub](https://github.com/CluEleSsUK) · [LinkedIn](https://www.linkedin.com/in/patrickmcclurg/) - **Mahmoud Shehata**, Member of Technical Staff: [GitHub](https://github.com/b0xtch) · [LinkedIn](https://www.linkedin.com/in/botch/) --- # Documentation Confidential AI runs AI workloads — inference, agents, training, and more — inside hardware-backed Trusted Execution Environments, so prompts, responses, and credentials stay encrypted end to end and model weights stay hardware-isolated on the GPU, invisible even to the infrastructure operator. ## Start here ## Do a specific thing ## Look something up ## Understand how it works --- # Get started This walkthrough creates an instance, waits until it is ready, reads its instance record, and deletes it. The examples use [curl](https://curl.se/) and [jq](https://jqlang.org/). Every endpoint, field, and error code is in the [API reference](/docs/agents-api/reference). This walkthrough creates an instance, waits until it is ready, reads its instance record, and deletes it. The examples use [curl](https://curl.se/) and [jq](https://jqlang.org/). ### 1. Obtain credentials Organizations and API keys are provisioned by Confidential. To request access, [contact us](mailto:hello@confidential.ai). When your organization is created, you receive: - An organization slug, used in instance hostnames such as `{instance-name}.{organization-slug}.confidential.ai`. - An API key, used as a Bearer token for API requests. Export both values before running the examples: ```bash export API_BASE="https://api.confidential.ai" export ORGANIZATION_SLUG="acme" export CA_API_KEY="confai_live_replace_with_your_key" ``` ### 2. Test your API key Call the usage endpoint. A successful response returns a `data.pricing` object and a `data.usage` object for the current billing cycle. ```bash curl -sS "$API_BASE/v1/usage" \ -H "Authorization: Bearer $CA_API_KEY" \ | jq . ``` ### 3. Create an instance Use any valid OpenSSH public key (`ssh-ed25519`, `ssh-rsa`, `ecdsa-sha2-*`, etc.). We recommend `ssh-ed25519` for new keys; `ssh-rsa` is supported for backwards compatibility but should be at least 2048 bits. ```bash export SSH_PUBLIC_KEY="$(cat ~/.ssh/id_ed25519.pub)" CREATE_RESPONSE="$( curl -sS -X POST "$API_BASE/v1/instances" \ -H "Authorization: Bearer $CA_API_KEY" \ -H "Content-Type: application/json" \ --data "$(jq -n --arg public_key "$SSH_PUBLIC_KEY" '{ public_key: $public_key, agent: "openclaw", inference_mode: "default_gateway" }')" )" echo "$CREATE_RESPONSE" | jq . export INSTANCE_NAME="$(echo "$CREATE_RESPONSE" | jq -r '.data.name')" export INSTANCE_HOSTNAME="$(echo "$CREATE_RESPONSE" | jq -r '.data.hostname')" ``` The create response is `202 Accepted`. The instance starts in `provisioning`. ### 4. Get instance info ```bash curl -sS "$API_BASE/v1/instances/$INSTANCE_NAME" \ -H "Authorization: Bearer $CA_API_KEY" \ | jq . ``` The response includes fields such as `name`, `status`, `agent`, `hostname`, `inference_mode`, `created_at`, and `ready_at`. ### 5. Poll until the instance is ready There are no webhooks for instance state changes. Poll `GET /v1/instances/{name}` until `status` becomes `ready`. ```bash while true; do INSTANCE_RESPONSE="$( curl -sS "$API_BASE/v1/instances/$INSTANCE_NAME" \ -H "Authorization: Bearer $CA_API_KEY" )" STATUS="$(echo "$INSTANCE_RESPONSE" | jq -r '.data.status')" echo "status=$STATUS" if [ "$STATUS" = "ready" ]; then export INSTANCE_HOSTNAME="$(echo "$INSTANCE_RESPONSE" | jq -r '.data.hostname')" break fi if [ "$STATUS" = "failed" ]; then echo "$INSTANCE_RESPONSE" | jq . exit 1 fi sleep 15 done echo "Instance is ready: $INSTANCE_HOSTNAME" ``` Once the instance is ready, connect over SSH with the private key that matches the public key from the create request: ```bash ssh -i ~/.ssh/id_ed25519 "$INSTANCE_HOSTNAME" ``` ### 6. Verify the CVM attestation `ccvm` is a CLI that runs inside the CVM and validates the hardware attestation, TPM measurements, host-key fingerprints, and inference gateway attestation chain — confirming the CVM is what it claims to be and is connected to the expected gateway. Run it from within the SSH session immediately after claiming the instance. ```bash ccvm verify ``` Example output: ``` [1/5] SEV-SNP Hardware PASS [2/5] TPM Attestation PASS [3/5] Host Key Binding PASS [4/5] Inference Provider PASS [5/5] External Access Lockout PASS (FAIL on staging — debug SSH access intentional) ``` The tool is open source: [github.com/confidential-dot-ai/confidential-cvm-cli](https://github.com/confidential-dot-ai/confidential-cvm-cli). ### 7. Delete the instance When you are done, delete the instance. The response is 202 Accepted with status: `terminating`. The instance reaches `terminated` asynchronously once Azure resources are cleaned up. ```bash curl -sS -X DELETE "$API_BASE/v1/instances/$INSTANCE_NAME" \ -H "Authorization: Bearer $CA_API_KEY" \ | jq . ``` ## Next steps - [API reference](/docs/agents-api/reference) — every endpoint, request field, and response shape. - [Introduction to TEEs](/docs/concepts/intro-to-tees) — what the hardware isolation actually guarantees. --- # API reference The Confidential Agents API provisions and manages per-organization confidential VM (CVM) instances that run packaged agents such as OpenClaw. Each instance is a hardware-isolated workload running inside a Trusted Execution Environment, addressable over SSH and uniquely named under your organization subdomain. New here? [Get started](/docs/agents-api/get-started) creates an instance end to end. ## Endpoints All endpoints are served from `https://api.confidential.ai` and versioned under `/v1`. ### Create Instance ```http POST /v1/instances ``` Provisions a new confidential VM under your organization. Returns `202 Accepted` with the generated instance `name`. Provisioning typically completes within a few minutes; poll the Retrieve Instance endpoint until `status` flips from `provisioning` to `ready`. #### Headers | Header | Required | Description | | --- | --- | --- | | `Authorization` | yes | `Bearer `. | | `Idempotency-Key` | recommended | Prevents duplicate provisioning on retry. See below. | | `X-Correlation-ID` | optional | Client-supplied tag echoed back on the response. | #### Idempotency-Key `Idempotency-Key` prevents duplicate side effects when retrying `POST /v1/instances`. It is **not** a request ID and **not** a correlation ID — each retry still receives a fresh `request_id`. - A duplicate request within 24 hours returns the original result and does **not** provision a second CVM. - The response body is the original API result, but `request_id` and `correlation_id` reflect the current attempt. - Reusing the same `Idempotency-Key` with a different request body returns `409 conflict`. #### Request body ```json { "public_key": "ssh-ed25519 AAAAC3... user@example", "agent": "openclaw", "inference_mode": "default_gateway", "inference_model": "" } ``` | Field | Type | Required | Description | | --- | --- | --- | --- | | `public_key` | string | yes | OpenSSH-formatted public key (`ssh-rsa`, `ssh-ed25519`, `ecdsa-sha2-*`, etc.). We recommend `ssh-ed25519` for new keys; `ssh-rsa` is accepted for backwards compatibility. | | `agent` | string | no | Packaged agent to install. Defaults to `openclaw`. Currently `openclaw` is the only supported value. | | `inference_mode` | string | no | Inference routing mode. If omitted, the platform selects one for you. Currently `default_gateway` is the only supported value. | | `inference_model` | string | no | Model identifier to use through the default inference gateway. **If omitted, the platform selects a model for you and returns the chosen identifier in every subsequent response for this instance.** Specify this field only if you need a particular model. | #### Response — `202 Accepted` ```json { "request_id": "4f1f6b6ab27d49bdb1a6a7a21c9f3b42", "correlation_id": null, "data": { "name": "4k9p2xq7", "status": "provisioning", "agent": "openclaw", "hostname": "4k9p2xq7.acme.confidential.ai", "public_key": "ssh-ed25519 AAAAC3... user@example", "inference_mode": "default_gateway", "inference_model": "", "egress_limit_bytes": 5368709120, "created_at": "2026-05-01T20:14:22Z" } } ``` Once `status` reaches `ready`, connect over SSH: ```bash ssh ``` If your SSH client requires an explicit user or key path: ```bash ssh -i ``` #### Errors | Code | HTTP | When | | --- | --- | --- | | `invalid_request` | `400` | `public_key` is missing or not a valid OpenSSH public key, or an unsupported `agent` or `inference_mode` was supplied. | | `unauthenticated` | `401` | Missing or invalid Bearer token. | | `conflict` | `409` | The same `Idempotency-Key` was reused with a different request body. | ### List Instances ```http GET /v1/instances ``` Returns all instances belonging to your organization, including terminated instances retained for the 30-day name-reservation window. #### Response — `200 OK` ```json { "request_id": "4f1f6b6ab27d49bdb1a6a7a21c9f3b42", "correlation_id": null, "data": { "instances": [ { "name": "4k9p2xq7", "status": "ready", "agent": "openclaw", "hostname": "4k9p2xq7.acme.confidential.ai", "public_key": "ssh-ed25519 AAAAC3... user@example", "inference_mode": "default_gateway", "inference_model": "", "egress_limit_bytes": 5368709120, "failure_code": null, "failure_message": null, "terminated_reason": null, "created_at": "2026-05-01T20:14:22Z", "ready_at": "2026-05-01T20:18:33Z", "terminated_at": null } ], "next_cursor": null } } ``` See the Status enum under Retrieve Instance below for the full list of `status` values and field semantics. #### Pagination The response includes `next_cursor`, currently always `null`. Cursor pagination will be activated in a future release; the response shape is stable. ### Retrieve Instance ```http GET /v1/instances/{name} ``` Returns the full record for one instance. Use this endpoint to poll provisioning status and to read current state. There are no webhooks for instance state changes — poll this endpoint instead. #### Path parameters | Parameter | Description | | --- | --- | | `name` | The 8-character generated instance name, e.g. `4k9p2xq7`. | #### Response — `200 OK` ```json { "request_id": "4f1f6b6ab27d49bdb1a6a7a21c9f3b42", "correlation_id": "venice-job-2026-05-03-001", "data": { "name": "4k9p2xq7", "status": "ready", "agent": "openclaw", "hostname": "4k9p2xq7.acme.confidential.ai", "public_key": "ssh-ed25519 AAAAC3... user@example", "inference_mode": "default_gateway", "inference_model": "", "egress_limit_bytes": 5368709120, "failure_code": null, "failure_message": null, "terminated_reason": null, "created_at": "2026-05-01T20:14:22Z", "ready_at": "2026-05-01T20:18:33Z", "terminated_at": null } } ``` #### Status enum | Value | Meaning | | --- | --- | | `provisioning` | Instance is being claimed or cold-started. | | `ready` | SSH is reachable and claim-time setup has completed. Run in-CVM verification to confirm attestation before trusting the workload. | | `failed` | Provisioning failed terminally. `failure_code` and `failure_message` may be populated. | | `terminating` | `DELETE /v1/instances/{name}` has been accepted and resource teardown is in progress. The instance transitions to `terminated` once Azure resources are fully released. | | `terminated` | Resources have been released. The record and name reservation are retained for at least 30 days. | #### Egress Each instance has a hard 5 GB egress limit per its lifetime. The platform monitors per-VM network egress and enforces the limit at the instance firewall layer once the threshold is observed. `egress_limit_bytes` is reported on every instance record. #### Errors | Code | HTTP | When | | --- | --- | --- | | `not_found` | `404` | Instance does not exist or belongs to another organization. | | `unauthenticated` | `401` | Missing or invalid Bearer token. | ### Delete Instance ```http DELETE /v1/instances/{name} ``` Tears down the CVM and removes its DNS record. The database row is retained with a terminal status, and the name remains reserved for at least 30 days. Returns `202 Accepted` with status: `terminating`. Azure resource teardown completes asynchronously; the instance transitions to `terminated` once cleanup finishes. Poll `GET /v1/instances/{name}` until status is `terminated` if you need confirmation that resources have been released. #### Path parameters | Parameter | Description | | --- | --- | | `name` | The 8-character generated instance name, e.g. `4k9p2xq7`. | #### Response — `202 Accepted` ```json { "request_id": "4f1f6b6ab27d49bdb1a6a7a21c9f3b42", "correlation_id": null, "data": { "name": "4k9p2xq7", "status": "terminating", "agent": "openclaw", "hostname": "4k9p2xq7.acme.confidential.ai", "public_key": "ssh-ed25519 AAAAC3... user@example", "inference_mode": "default_gateway", "inference_model": "", "egress_limit_bytes": 5368709120, "failure_code": null, "failure_message": null, "terminated_reason": "deleted_via_api", "created_at": "2026-05-01T20:14:22Z", "ready_at": "2026-05-01T20:18:33Z", "terminated_at": null } } ``` #### Errors | Code | HTTP | When | | --- | --- | --- | | `not_found` | `404` | Instance does not exist or belongs to another organization. | | `unauthenticated` | `401` | Missing or invalid Bearer token. | ### Get Usage ```http GET /v1/usage ``` Returns the current billing-cycle consumption summary for your organization — the same numbers that drive your invoice. #### Response — `200 OK` ```json { "request_id": "4f1f6b6ab27d49bdb1a6a7a21c9f3b42", "correlation_id": null, "data": { "pricing": { "period_start": "2026-04-15T14:32:11Z", "period_end": "2026-05-15T14:32:11Z", "subscription_cost_usd": "200.00", "instance_hours_included": 200, "overage_per_hour_usd": "0.45" }, "usage": { "instance_hours_used": 142.7, "inference_cost_usd": "28.23", "egress_bytes": 1048576, "egress_limit_bytes": 5368709120, "egress_observed_at": "2026-05-01T20:14:22Z", "egress_blocked": false, "egress_block_applied_at": null } } } ``` #### `pricing` fields | Field | Description | | --- | --- | | `period_start`, `period_end` | Start and end of the current billing window (ISO 8601). The window is anchored to your organization's signup time, not to calendar months. | | `subscription_cost_usd` | Base subscription cost for the period. | | `instance_hours_included` | Instance-hours included in the base subscription. | | `overage_per_hour_usd` | Cost per instance-hour beyond the included amount. | #### `usage` fields | Field | Description | | --- | --- | | `instance_hours_used` | Total instance-hours consumed this period across all instances. | | `inference_cost_usd` | Inference spend this period through the default gateway. | | `egress_bytes` | Most recent observed egress total for the active instance(s). | | `egress_limit_bytes` | Per-instance hard egress limit. | | `egress_observed_at` | When the egress total was last sampled. | | `egress_blocked` | `true` if egress is currently blocked at the firewall. | | `egress_block_applied_at` | When the block was applied, or `null`. | Per-day and per-instance usage breakdowns are not currently exposed through the API. ## Authentication All endpoints require a Bearer token in the `Authorization` header: ```http Authorization: Bearer ca_<8 char lowercase alphanumeric>_<18 char lowercase alphanumeric> ``` #### Obtaining credentials Organizations and API keys are provisioned by Confidential. To request access, [contact us](mailto:hello@confidential.ai). Self-service signup through the public API is not currently available. When your organization is created, you receive: - An **organization slug** (e.g. `acme`), which is used in your per-organization subdomain (`{instance-name}.acme.confidential.ai`). - One or more **API keys** of the form `ca__`, where `prefix` and `secret` are lowercase alphanumeric strings (8 and 18 characters respectively). Treat these as secrets — anyone with the key can provision and delete instances against your organization. #### Authentication errors A missing, malformed, or unknown token returns: ```json { "request_id": "4f1f6b6ab27d49bdb1a6a7a21c9f3b42", "correlation_id": null, "error": { "code": "unauthenticated", "message": "missing or invalid Bearer token" } } ``` A valid token used against an organization or resource it is not authorized for returns `forbidden` (`403`). See the error code table under Conventions below. ## Conventions #### Response envelope Every JSON response is wrapped in a consistent envelope. On success: ```json { "request_id": "4f1f6b6ab27d49bdb1a6a7a21c9f3b42", "correlation_id": "venice-job-2026-05-03-001", "data": { } } ``` On error: ```json { "request_id": "4f1f6b6ab27d49bdb1a6a7a21c9f3b42", "correlation_id": null, "error": { "code": "invalid_request", "message": "public_key is not a valid OpenSSH public key: ...", "param": "public_key" } } ``` - `request_id` — server-generated identifier for this HTTP request. Always present. Quote it when contacting support. - `correlation_id` — the value of the `X-Correlation-ID` request header if you sent one, otherwise `null`. - `error.param` — present when the error maps cleanly to a single request field. - `error.message` — safe to surface to end users. The same `request_id` is mirrored in the `X-Request-ID` response header. If you supplied an `X-Correlation-ID`, it is echoed back in `X-Correlation-ID` on the response. #### Request IDs and Correlation IDs The API generates the canonical `request_id` for every request. Clients cannot choose it. If you send `X-Request-ID`, it is ignored and a fresh server-side ID is generated. To tag requests for your own tracking, send `X-Correlation-ID`: ```http X-Correlation-ID: venice-job-2026-05-03-001 ``` Correlation IDs are not required to be unique and the server does not enforce a format. #### Error codes | Code | HTTP | Meaning | | --- | --- | --- | | `invalid_request` | `400` | Malformed or invalid input. | | `unauthenticated` | `401` | Missing or invalid Bearer token. | | `forbidden` | `403` | Token is valid but not allowed for this organization or resource. | | `not_found` | `404` | Resource does not exist, or belongs to another organization. | | `conflict` | `409` | State or idempotency conflict. | | `internal_error` | `500` | Unexpected server error. Quote `request_id` when reporting. | #### Per-organization subdomain Each organization is assigned a slug at onboarding and gets a dedicated subdomain: ``` {customer-slug}.confidential.ai ``` All instances are addressable as: ``` {instance-name}.{customer-slug}.confidential.ai ``` Example: ``` 4k9p2xq7.acme.confidential.ai ``` #### Instance names Instances are identified by an auto-generated `name` scoped to your organization. - **Format:** 8-character lowercase alphanumeric string (e.g. `4k9p2xq7`). - **Uniqueness:** Names are unique within an organization. - **Reuse:** Names are not reused for at least 30 days after termination, so a stored reference cannot silently point at a different instance. - **Custom names:** Customer-supplied instance names are not currently supported. ## Need help? [Contact us](mailto:hello@confidential.ai) for access, support, or feature requests. --- # Attested Builds An attested build runs inside a Trusted Execution Environment and emits a signed record of exactly what went in and what came out, so a consumer can check that an artifact was built from the source it claims — without trusting the builder. We assume familiarity with git, package managers, and the concept of cryptographic hashes. ## Understand it Read in order. Each one depends on the last. ### [01. What Are Attested Builds?](/docs/attested-builds/what-are-attested-builds) What attested builds are, why they matter, and how they solve the software verification problem. Covers the core insight, why reproducible builds remain elusive in practice, and why Trusted Execution Environments make a different approach possible now. ### [02. How It Works](/docs/attested-builds/how-it-works) The architecture and mechanics end to end. Each phase of the build, how cryptographic binding works at every step, and how verification closes the loop from source to running code — using Kettle, Confidential's implementation, as the reference. ### [03. Provenance & Standards](/docs/attested-builds/provenance-standards) How the claims an attested build produces fit into the supply-chain ecosystem: what SLSA and in-toto are, why standards matter for interoperability, and how to read a provenance document. ### [04. Threat Model and Security Boundaries](/docs/attested-builds/threat-model) The line between what attested builds protect against and what they don't — where the trust boundaries lie, which attacks are prevented, and which assumptions you are still making. ## Look it up --- # How Attested Builds Work This document explains the architecture and mechanics of attested builds. After reading it, you'll understand each phase of the build process, how cryptographic binding works at every step, and how verification closes the loop from source to running code. We assume you've read [What Are Attested Builds?](/docs/attested-builds/what-are-attested-builds) and understand the core concept: attested builds prove that specific inputs were used to produce specific outputs in a verified environment. This document shows exactly how that works. ## Kettle: Confidential's Implementation Kettle is Confidential's implementation of attested builds. It handles the entire pipeline: input verification, manifest generation, TEE orchestration, build execution, and provenance signing. When we describe "how attested builds work" in this document, we're describing how Kettle works. Kettle will be open source and audited. ## The Build Flow An attested build has three phases, split across two environments. The first phase happens locally on the developer's machine. The second and third phases happen inside a TEE. ``` ┌───────────────────────────────────────────────────────────────────────────────┐ │ Attested Build Flow │ │ │ │ DEVELOPER MACHINE │ │ ───────────────── │ │ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ Phase 1: Manifest Creation │ │ │ │ │ │ │ │ Source ───┐ │ │ │ │ │ │ │ │ │ Deps ─────┼──▶ Kettle ──▶ Manifest ──▶ Merkle Root │ │ │ │ │ │ │ │ │ Toolchain─┘ │ │ │ │ │ │ │ └──────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ manifest + source archive │ │ ▼ │ │ TRUSTED EXECUTION ENVIRONMENT │ │ ───────────────────────────── │ │ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ Phase 2: TEE Setup │ │ │ │ │ │ │ │ Hardened VM ──▶ Kettle (measured) ──▶ Inputs Loaded ──▶ Isolated │ │ │ │ │ │ │ │ Hardware Attestation: "This environment contains Kettle with │ │ │ │ measurement X, loaded with manifest Y" │ │ │ │ │ │ │ └──────────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ Phase 3: Build and Signing │ │ │ │ │ │ │ │ Verify Manifest ──▶ Execute Build ──▶ Hash Outputs ──▶ Sign │ │ │ │ │ │ │ │ Output: Artifacts + Provenance + TEE Attestation │ │ │ │ │ │ │ └──────────────────────────────────────────────────────────────────────┘ │ │ │ └───────────────────────────────────────────────────────────────────────────────┘ ``` The split between local and TEE environments is intentional. Manifest creation requires access to your development environment: your source tree, your lockfiles, your toolchain. This happens on a machine you control. The actual build happens in a TEE where the hardware provides isolation and attestation. The manifest bridges these environments, carrying cryptographic commitments from one to the other. ## Phase 1: Manifest Creation Before any build starts, the developer locks in their dependencies and creates a manifest. This happens locally, outside the TEE. The manifest becomes the source of truth for what should be included in the build. **Locking dependencies.** The developer ensures their project has a lockfile that pins every dependency to a specific version with a cryptographic checksum. This is standard practice in modern package managers. The lockfile captures the exact dependency graph at a point in time. **Input verification.** Kettle walks through every build input and computes its cryptographic hash: - Source code: git commit hash, tree hash (content-addressed hash of the file tree), repository signature - Dependencies: each package identified by name, version, and checksum from the lockfile - Toolchain: hashes of compiler and build tool binaries The verification ensures that cached artifacts match their expected checksums. Any mismatch fails immediately. You cannot create a manifest with unverified inputs. **Merkle tree construction.** All input hashes become leaves in a Merkle tree. The tree is constructed in deterministic order: git information first, then lockfile hash, then dependencies sorted alphabetically, then toolchain hashes. This ordering is fixed by convention so that anyone can reconstruct the same tree from the same inputs. ``` [input_merkle_root] ────────┬──────── ┌─────────┴─────────┐ [subtree] [subtree] ────┬──── ────┬──── ┌──────┴──────┐ ┌──────┴──────┐ [git_commit] [tree_hash] [lockfile] [deps_root] ────┬──── ┌────────┼────────┐ [dep_a] [dep_b] [dep_c] ... ... ... ``` The root of this tree, the `input_merkle_root`, is a single 32-byte hash that uniquely identifies the complete set of build inputs. If any input changes by a single byte, the root changes. **Manifest generation.** Kettle produces a manifest containing the Merkle root, the full list of inputs with their hashes, and metadata about the build configuration. This manifest is the commitment: it says "a valid build of this project must use exactly these inputs." **Developer signature.** The developer signs the manifest. This signature attests that the developer reviewed and approved this set of inputs. The signature is optional but recommended. It creates accountability: the signed manifest traces back to a specific person who vouched for these inputs. The manifest can be stored alongside your source code, published to a registry, or transmitted directly to a TEE build service. It carries everything needed to verify inputs without access to your development machine. The Merkle structure enables selective disclosure. If someone asks "prove you're building against dependency X version Y," you can provide a Merkle inclusion proof: just the path from that dependency's leaf to the root. The verifier confirms the proof against the published root without seeing your other dependencies. ## Phase 2: TEE Environment Setup The build environment boots inside a TEE with measured boot. This is where hardware takes over and starts generating evidence that software cannot forge. ### The Measured Boot Chain When the TEE initializes, the hardware measures every component that loads. "Measured" means the TEE's security processor computes a hash of the component and extends it into a cumulative measurement register. The extension operation is: `new_measurement = hash(old_measurement || component_hash)`. Because extension is one-way (you cannot "un-extend"), the final measurement depends on exactly what was loaded, in exactly what order. The measurement chain includes: | Component | What Gets Measured | | --------- | --------------------------------------- | | Firmware | UEFI/boot code that initializes the TEE | | VM Image | The hardened operating system image | | Kettle | The attested build orchestrator | Kettle is measured as part of the TEE's initial state. This is critical: the attestation report proves not just "this is a TEE" but "this is a TEE running Kettle with measurement X." A verifier can confirm they're talking to a genuine Kettle instance, not arbitrary code claiming to be Kettle. ### Hardening the Build Environment The TEE provides hardware isolation, but defense in depth requires additional hardening layers. The build environment is configured to minimize attack surface: **Hardened VM image.** The base image is minimal: only the components needed to run Kettle and execute builds. No unnecessary services, no debugging tools, no package managers. The image is built reproducibly so its measurement is predictable. **Mandatory access control.** The VM runs with MAC (Mandatory Access Control) enforced through SELinux or equivalent. MAC policies restrict what processes can access, even if they're running as root. The build process cannot access files or resources outside its designated scope. **System call filtering.** Kettle enforces seccomp (secure computing mode) filters on the build process. The filter allows only the system calls needed for compilation and file I/O. Dangerous calls (like those that would enable network access or privilege escalation) are blocked. **Network isolation.** After the manifest and inputs are loaded, networking is disabled. The build cannot fetch additional resources, phone home, or exfiltrate data. This isolation is enforced at multiple layers. For example, seccomp blocks network-related syscalls and iptables rules drop all traffic. ### Loading Inputs With the environment hardened, Kettle loads the build inputs. There are two approaches: **Network-enabled loading.** Kettle fetches dependencies from their registries based on the manifest. Each fetched artifact is verified against the checksum in the manifest. After all inputs are loaded and verified, networking is disabled. This approach is simpler but requires trusting the network path during loading. **Pre-loaded inputs.** All dependencies are bundled with the source archive and loaded into the TEE before it boots. The inputs are measured as part of the initial TEE state. This approach eliminates network trust during the build entirely, at the cost of larger input bundles. The measurement includes not just Kettle but the specific inputs, providing stronger binding. Either way, Kettle verifies that the loaded inputs match the manifest's Merkle root. If anything doesn't match, the build fails. ### TEE Attestation Once the environment is set up and inputs are loaded, the TEE generates an attestation report. This report contains: - The cumulative measurement (what was loaded) - Platform information (CPU model, firmware version, security features) - A hash of the manifest in the report's custom data field - A signature from a key that only the TEE can access The signature chains to the CPU vendor's root certificate. A verifier can confirm: "This report was generated by genuine hardware running a TEE with this measurement, and it was loaded with inputs matching this manifest." This is the bridge between software claims and hardware proof. The manifest says "these inputs should be used." The attestation proves "these inputs were loaded into this measured environment." ## Phase 3: Build Execution and Signing The build runs inside the isolated TEE using the verified inputs. ### Executing the Build Kettle invokes the build toolchain according to the build configuration. The compilation, linking, and packaging happen normally. Kettle doesn't modify the build process. It wraps it in a verified, isolated environment. The build runs under the hardening constraints described above: MAC policies restrict file access, seccomp filters block dangerous syscalls, and networking is disabled. If the build attempts anything outside its permitted scope, the operation fails. ### Hashing Outputs When the build completes, Kettle hashes each output artifact. These hashes become part of the provenance. Because the hashing happens inside the TEE, the output hashes benefit from the same isolation guarantees as the build itself. For artifacts that will run in confidential VMs, Kettle also computes the expected launch measurement. This is the hash that will appear in runtime attestation when the artifact boots. Computing it now, at build time, enables the critical verification: does the runtime measurement match what the build produced? ### Signing Provenance The TEE generates a signing key derived from its hardware root of trust. This key is bound to the specific TEE instance and its measured state: it can only be produced by a TEE with exactly this configuration. A different TEE, or the same TEE with different loaded software, would produce a different key. Kettle constructs a provenance document containing: - The manifest (including Merkle root and all input hashes) - The TEE attestation report - Output artifact hashes - Expected launch measurement for runtime verification - Timestamps and build metadata The TEE signs this provenance document. The signature cryptographically binds the outputs to the attested build environment and verified inputs. Forging this signature requires compromising the TEE hardware. ### What Kettle Produces A complete build generates three files: **manifest.json**: The input commitment created in Phase 1. Contains the Merkle root, full input list, and (if signed) the developer's signature. **provenance.json**: The build record created in Phase 3. Contains the manifest, output hashes, and build metadata in SLSA v1.2 format for interoperability with other supply chain tools. **evidence**: The TEE attestation report. The first 32 bytes of the custom data field contain the SHA256 hash of the provenance, binding the attestation to the provenance content. These three files together provide the complete evidence chain. The manifest commits to inputs. The provenance binds inputs to outputs. The attestation proves the provenance was generated in a genuine TEE. ## The Complete Chain The result is a cryptographic chain from source to running code: ``` ┌─────────────────────────────────────┐ │ BUILD INPUTS │ │ ┌─────────┐ ┌──────┐ ┌──────────┐ │ │ │ Source │ │ Deps │ │ Toolchain│ │ │ └────┬────┘ └──┬───┘ └────┬─────┘ │ │ └─────────┼──────────┘ │ └─────────────────┼────────────────────┘ │ ▼ ┌──────────────┐ │ Merkle Root │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ Manifest │ └──────┬───────┘ │ ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄│┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄ TEE boundary │ ▼ ┌─────────────────────────────────────┐ │ TRUSTED EXECUTION │ │ │ │ Kettle ──▶ Verify ──▶ Build │ │ (measured) Inputs │ │ │ │ │ ▼ │ │ Hash Outputs │ │ │ │ │ ▼ │ │ Sign with TEE Key │ │ │ │ └──────────────────────────┼───────────┘ │ ▼ ┌─────────────────────────────────────┐ │ OUTPUTS │ │ ┌────────────┐ ┌──────────────┐ │ │ │ Provenance │ │ Attestation │ │ │ └─────┬──────┘ └──────┬───────┘ │ │ └────────┬───────┘ │ └─────────────────┼────────────────────┘ │ ▼ ┌─────────────────────────────────────┐ │ RUNTIME VERIFICATION │ │ │ │ Live Attestation ══ Expected │ │ from Service Measurement │ │ │ └─────────────────────────────────────┘ ``` Let's trace why an attack at each stage would fail: **Attacker modifies source after commit.** The tree hash changes. The Merkle root changes. The manifest no longer matches. Verification fails. **Attacker substitutes a dependency.** The checksum doesn't match the lockfile. Input verification fails before the build starts. **Attacker compromises the build machine.** The build runs inside the TEE, isolated from the host. The attacker cannot read memory or inject code. If they load different inputs, the measurement changes and attestation verification fails. **Attacker provides a malicious Kettle version.** The TEE measurement would differ from the expected Kettle measurement. Verifiers reject attestations from unrecognized builds. **Attacker swaps the output artifact.** The artifact hash doesn't match the provenance. Verification fails. **Attacker forges provenance.** They cannot produce a valid TEE attestation signature without the hardware key. Signature verification fails. ## Verification Flow A client verifying your service checks the complete chain from source to runtime. | Step | What the client checks | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1. Verify TEE attestation | Attestation signature chains to hardware vendor root of trust. Kettle measurement matches expected value. Attestation is fresh. | | 2. Verify provenance binding | Provenance hash in attestation matches actual provenance document. | | 3. Verify manifest signature | Developer signature on manifest is valid (if present). | | 4. Verify input hashes | Compare Merkle root hash. Optionally recompute Merkle root from inputs. Check against manifest. Optionally verify specific dependencies against registries. | | 5. Verify artifact binding | Output artifact hashes match provenance. | If all checks pass, the client has cryptographic proof that: - The running code has measurement X - Measurement X was produced by a build inside an attested TEE running verified Kettle - That build used exactly the inputs specified in a signed manifest - Every link in the chain is independently verifiable Verification takes OOM ~100s of milliseconds. Each step is a cryptographic check against hardware-rooted signatures or known public keys. --- # Provenance format Kettle emits three artifacts alongside a build's outputs. This page is their shape. For what they are *for* and how they map onto SLSA and in-toto, see [Provenance & Standards](/docs/attested-builds/provenance-standards). ## What Kettle Produces A complete Kettle build generates three files. Each serves a different purpose in the verification chain. ### manifest.json The manifest is a human-readable summary of the build. It contains: ```json { "git_commit": "a1b2c3d4e5f67890abcdef1234567890abcdef12", "git_tree": "7890abcdef1234567890abcdef1234567890abcde", "lockfile_hash": "23b2e23aa04c93c350cac09ac73636e4ecedf564...", "input_merkle_root": "72a97c73d0c59905c89dc7da145a5ecc3d809be5...", "toolchain": { "rustc_hash": "e6abf55ab1859e7c990be77fd593f5166...", "cargo_hash": "51de284e8bb0d03dcee595a0fb1cb3a952..." }, "artifacts": [ { "name": "my-app", "hash": "1d1ea25c371d4f6de8d6e3c26fdad2238..." } ] } ``` This is for humans to inspect and debug. When something goes wrong, you look here to understand what inputs were used and what outputs were produced. The git commit and tree hash let you identify the exact source. The lockfile hash lets you verify dependency pinning. The input Merkle root is the cryptographic commitment to all inputs combined. ### provenance.json The provenance is a SLSA v1.2 statement in in-toto format. This is the machine-readable, interoperable record. It follows the SLSA specification exactly, which means other tools in the ecosystem can consume it without custom parsing. The structure has two main sections: **buildDefinition** describes the inputs to the build: ```json { "buildDefinition": { "buildType": "https://example.com/attested-build/v1", "externalParameters": { "repository": "https://github.com/org/repo", "ref": "refs/heads/main" }, "internalParameters": {}, "resolvedDependencies": [ { "uri": "git+https://github.com/org/repo@refs/heads/main", "digest": { "gitCommit": "a1b2c3d4..." } }, { "uri": "pkg:cargo/serde@1.0.228", "digest": { "sha256": "9a8e94ea..." } } ] } } ``` The `buildType` identifies how to interpret the parameters. It's a URI that should resolve to documentation explaining the build process. The `externalParameters` are the top-level inputs controlled by the user: which repository to build, which ref to check out, which entry point to use. These are untrusted from SLSA's perspective and must be verified downstream. The `internalParameters` are set by the build platform itself. In attested builds, this might include TEE configuration or platform version information. The `resolvedDependencies` capture what was actually fetched during the build. Notice the distinction: `externalParameters` might say "build from refs/heads/main", while `resolvedDependencies` records that this resolved to commit `a1b2c3d4...`. The dependencies use Package URLs (PURLs) for standardized identification. A Cargo dependency looks like `pkg:cargo/serde@1.0.228?checksum=sha256:9a8e94ea...`. **runDetails** describes the build execution: ```json { "runDetails": { "builder": { "id": "https://example.com/tee-builder/v1" }, "metadata": { "invocationId": "build-12345", "startedOn": "2024-01-15T10:30:00Z", "finishedOn": "2024-01-15T10:35:00Z" } } } ``` The `builder.id` is the critical field. It identifies the build platform and represents the transitive closure of everything you're trusting to faithfully run the build and record provenance. For attested builds, this ID represents the TEE-based build system with its specific security properties. The `metadata` provides operational information: when the build ran, how long it took, and an identifier for this specific invocation. ### Evidence The `evidence` file contains the TEE attestation report, base64-encoded. This is what roots everything in hardware. The attestation report is signed by the TEE using keys that chain back to the hardware vendor's root of trust. A verifier can check this signature against the vendor's certificate chain to confirm the report came from genuine hardware. Critically, the first 32 bytes of the report's custom data field contain the SHA256 hash of the provenance document. This cryptographically binds the attestation to the provenance. You can't take an attestation from one build and attach it to provenance from a different build. The hash must match. The verification chain works like this: 1. Verify attestation signature against hardware vendor's certificate chain 2. Extract provenance hash from attestation report 3. Verify hash matches actual provenance document 4. Verify provenance contents (inputs, outputs, builder ID) 5. Verify artifact hashes match provenance subjects If any step fails, the verification fails. This chain ensures that the provenance is exactly what was generated inside the attested TEE, not something fabricated afterward. ## The Provenance Structure in Detail Understanding the provenance structure helps when debugging builds or writing verification policies. Let's look at each field more carefully. ### buildType The build type is a URI that identifies how to interpret the build definition. It encapsulates the build process independent of what platform ran it. ```json "buildType": "https://kettle.confidential.ai/cargo-build/v1" ``` The URI should resolve to documentation explaining: what the build process does, what externalParameters and internalParameters mean for this build type, and how to initiate a build given this definition. Different build types exist for different toolchains (Cargo vs Nix) or different build configurations. ### externalParameters vs internalParameters The distinction matters for verification. External parameters are untrusted. They come from outside the build platform: a user requesting a build, a CI trigger, a webhook. Verifiers must check these against expectations. Internal parameters are set by the platform itself. They're trusted because the platform is trusted. A verifier doesn't need to check them individually, though they might be useful for debugging or reproducibility. In practice, external parameters should be minimal. The more you put in external parameters, the more a verifier needs to check. Good build type design pushes configuration into the source repository (where it's covered by the source commit hash) rather than into external parameters. ### resolvedDependencies This field captures what was actually used during the build, not just what was requested. The distinction matters because many inputs resolve dynamically: ```json { "resolvedDependencies": [ { "uri": "git+https://github.com/org/repo@refs/heads/main", "digest": { "gitCommit": "a1b2c3d4e5f67890abcdef1234567890abcdef12" } }, { "uri": "pkg:cargo/serde@1.0.228", "digest": { "sha256": "9a8e94ea..." }, "name": "serde" }, { "uri": "pkg:cargo/tokio@1.35.0", "digest": { "sha256": "7b4c89..." }, "name": "tokio" } ] } ``` A request to build "refs/heads/main" resolves to a specific commit. A dependency on "serde ^1.0" resolves to a specific version. The resolved dependencies record these resolutions. Each dependency is a ResourceDescriptor with several optional fields: `uri` identifies the dependency, `digest` provides cryptographic verification, `name` is human-readable, `downloadLocation` says where it was fetched from if different from the URI. Completeness is best-effort at L3. Ideally, every artifact fetched during the build would be recorded. In practice, some builds fetch things that are hard to track. The goal is to capture enough that a security team can investigate if something goes wrong. ### builder.id The builder ID identifies the transitive closure of everything trusted to faithfully run the build. For attested builds: ```json { "builder": { "id": "https://kettle.confidential.ai/tee-builder/v1" } } ``` This ID should resolve to documentation explaining: the scope of what the ID represents, the claimed SLSA Build level, the accuracy and completeness guarantees of the provenance fields, and any fields that are generated by tenant-controlled processes rather than the trusted control plane. A consumer's verification policy specifies which builder IDs they trust. The policy might say "I trust builds from https://kettle.confidential.ai/tee-builder/v1 at SLSA L3" while rejecting builds from other platforms. ## See also - [Provenance & Standards](/docs/attested-builds/provenance-standards) — what these documents claim and which standards they satisfy. - [How attested builds work](/docs/attested-builds/how-it-works) — the phases that produce them. --- # Provenance & Standards This document explains how attested builds produce verifiable claims about software and how those claims fit into the broader supply chain security ecosystem. After reading it, you'll understand what SLSA and in-toto are, why standards matter for interoperability, what artifacts Kettle produces, and how to interpret provenance documents. We assume familiarity with the attested builds concept from [What Are Attested Builds?](/docs/attested-builds/what-are-attested-builds) and the build flow from [How It Works](/docs/attested-builds/how-it-works). No prior knowledge of supply chain security standards is required. ## Why Standards Matter Attested builds produce claims about how software was built. Those claims are only useful if others can understand and verify them. A proprietary format creates a silo: your tools can read it, but nothing else in the ecosystem can. Standards create ecosystems where provenance is usable by other tools, verifiable by third parties without special knowledge, and auditable against known criteria. Two prominent supply chain security standards are: SLSA (Supply-chain Levels for Software Artifacts) for defining security requirements and levels, and in-toto for the attestation format itself. Using these standards means that security scanners, package registries, deployment systems, and audit tools can all consume attested build output without custom integration work. ## SLSA: Supply-chain Levels for Software Artifacts SLSA (pronounced "salsa") is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. It provides two things: a common vocabulary for talking about supply chain security, and a framework for evaluating the trustworthiness of software artifacts. The framework emerged from Google's internal Binary Authorization for Borg system, which they've used for years to ensure that production binaries are built from reviewed source through authorized build systems. SLSA generalizes these practices into levels that any organization can adopt incrementally. ### The Build Track SLSA organizes requirements into "tracks" that focus on different aspects of the supply chain. The Build track, which is most relevant to attested builds, focuses on build integrity: ensuring that packages are built from the correct, unmodified sources and dependencies according to the build recipe defined by the software producer. The Build track defines three levels: | Level | Summary | Key Requirements | | ----- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | L1 | Provenance exists | Package has provenance showing how it was built. Can be unsigned. Prevents mistakes but trivial to forge. | | L2 | Hosted build platform | Provenance is signed by a hosted build platform. Forging requires an explicit attack, not just a mistake. | | L3 | Hardened builds | Build platform has strong tamper protection. Builds are isolated from each other. Signing keys are inaccessible to user-defined build steps. | Each level provides stronger guarantees but requires more investment. L1 catches mistakes and aids debugging. L2 deters unsophisticated adversaries and those who face legal or financial risk from evading controls. L3 prevents tampering during the build itself, even from insider threats or compromised credentials. ### Attested Builds Achieve L3 Attested builds using TEEs achieve SLSA Build L3 through hardware enforcement: | L3 Requirement | How Attested Builds Meet It | | -------------------------------------------------------------- | -------------------------------------------------------------------------------- | | Provenance generated by build platform's trusted control plane | TEE generates provenance in its trusted control plane, not in user-defined steps | | Provenance signed by build platform | Hardware-rooted key signs provenance. Key is derived from TEE attestation. | | Builds isolated from one another | TEE hardware isolation prevents cross-build interference | | Signing keys inaccessible to user-defined build steps | Keys derived from TEE hardware, never exposed to build scripts | The TEE provides stronger guarantees than typical L3 implementations because the isolation and key protection are hardware-enforced, not software-enforced. A compromised build script cannot escape the TEE's memory encryption. A malicious insider cannot extract signing keys because they're derived from hardware that only the CPU's security processor can access. ## The in-toto Attestation Framework While SLSA defines what security properties you need, in-toto defines how to express claims about software. It's an attestation framework that provides a standard structure for making signed statements about artifacts. The core concept is simple: an attestation is a signed statement that says something about one or more software artifacts. The statement has a subject (what artifacts it's about), a predicate type (what kind of claim this is), and a predicate (the actual claim data). ```json { "_type": "https://in-toto.io/Statement/v1", "subject": [{ "name": "my-app", "digest": { "sha256": "abc123..." } }], "predicateType": "https://slsa.dev/provenance/v1", "predicate": { // claim-specific data } } ``` The subject identifies artifacts by cryptographic digest, not by name alone. This means the statement is bound to specific bytes, not to a filename that could be reused for different content. The predicate type is a URI that identifies the schema for interpreting the predicate. Different predicate types exist for different purposes: SLSA Provenance for build information, SCAI for security attributes, test results, code review attestations, and SBOMs. The framework is extensible. New predicate types can be defined without changing the core format. This modularity matters for real-world use. A complete picture of an artifact's security posture might include multiple attestations: provenance showing how it was built, a code review attestation showing the source was reviewed, test results showing it passed CI, and a vulnerability scan showing no known CVEs. All of these use the same outer structure, can be bundled together, and can be verified with the same tooling. ### SLSA Provenance as a Predicate Type SLSA Provenance is one predicate type within the in-toto framework. It's specifically designed to record build information: what inputs went into a build, what process was used, and what outputs were produced. The relationship between SLSA and in-toto is complementary. SLSA defines the requirements ("provenance must be signed by the build platform"). in-toto defines the format ("here's how to structure a signed provenance statement"). You can think of in-toto as the unopinionated layer for expressing supply chain information, and SLSA as the opinionated layer specifying exactly what information must be captured to achieve specific security guarantees. ## What a build produces A build emits a manifest, a signed provenance document, and the TEE attestation evidence that binds them to the environment that ran. Their field-by-field shape is in [Provenance format](/docs/attested-builds/provenance-format). ## How Provenance Relates to SBOMs Software Bill of Materials (SBOMs) and SLSA Provenance are related but distinct. An SBOM describes what components are present in software. It's focused on understanding software for vulnerability assessment and license compliance. SBOMs need fine-grained, timely data: exact versions, licenses, known vulnerabilities. SLSA Provenance describes how software was built. It's focused on trustworthiness of the build process. Provenance is generated in the build platform's trusted control plane, which in practice makes it coarser-grained than an SBOM. The two are complementary. An SBOM tells you what's in the artifact. Provenance tells you that the SBOM (and the artifact) came from a verified build process. You might use provenance to trust the SBOM's accuracy: "I believe this SBOM is correct because it was generated by a build system I trust." Kettle's provenance includes dependency information in resolvedDependencies, but this isn't a full SBOM. It captures build-time dependencies at a coarse level (what was fetched during the build), not the fine-grained component analysis that SBOM tools provide. ## Summary Attested builds produce SLSA Build L3 provenance in standard in-toto format. The three output files (manifest.json, provenance.json, evidence) together provide a complete, verifiable record of the build. The provenance structure follows SLSA v1.2: buildDefinition captures inputs, runDetails captures execution context, and the builder.id identifies the trust anchor. The TEE attestation binds the provenance to hardware, preventing forgery. Using industry standards means the output integrates with existing supply chain security tools. Verifiers don't need custom code to consume attested build provenance. They check the attestation against the hardware vendor's certificates, verify the provenance hash binding, and evaluate the provenance contents against their policy. The next section, [Threat Model](/docs/attested-builds/threat-model), covers what exactly attested builds protect against, what they don't, and how to reason about the security boundaries. --- # Threat Model & Security Boundaries This document explains what attested builds protect against and what they don't. After reading it, you'll understand where the trust boundaries lie, what specific attacks are prevented, and which assumptions you're still making. We assume familiarity with the concepts from the previous sections: [What Are Attested Builds?](/docs/attested-builds/what-are-attested-builds), [How It Works](/docs/attested-builds/how-it-works), and [Provenance & Standards](/docs/attested-builds/provenance-standards). Security claims without clear boundaries are marketing. This document draws the boundaries. ## The Trust Model Every security system has a root of trust: something you assume is correct because you can't verify it further down. The question isn't whether you trust something, but what you trust and whether that trust is well-placed. Attested builds shift where the root of trust sits. Instead of trusting the entire infrastructure stack (cloud provider's software, employees, other tenants, build systems), you trust a much smaller set of components. ### What You Trust **TEE hardware and firmware.** You trust that the CPU vendor's manufacturing process didn't embed backdoors in the silicon. You trust that their key management practices keep root keys secure. You trust that the security processor firmware doesn't have exploitable vulnerabilities. **The physical host of the hardware.** Whoever physically hosts the hardware can perform classes of physical attacks that compromise TEE protections. This includes cold boot attacks, hardware interposers, and other physical tampering. If you don't control the physical infrastructure, you're trusting whoever does. **Cryptographic primitives.** AES for memory encryption, ECDSA for attestation signatures, SHA for measurements. If these break, the guarantees break. **Source repository integrity.** You trust that the commit you're building represents legitimate code. If your repository is compromised and contains a backdoor, the attested build faithfully builds the backdoor. Attestation proves what code was used, not that the code is trustworthy. **Package registries and toolchain distribution (in upstream mode).** When Kettle fetches dependencies or toolchains directly from external sources, you trust those sources to serve legitimate packages. In cached mode, these artifacts are verified once when cached and can be pinned into the TEE at initialization, removing external sources from the runtime trust path. ### What You Don't Trust **Host software.** The hypervisor, host OS, management plane, monitoring agents, and other tenants are all outside the trust boundary. They can be compromised without affecting the confidentiality or integrity of your build. They cannot read your build environment's memory or tamper with it undetected. **Host employees.** Administrators with root access, datacenter technicians, operators, and anyone else in the operational chain cannot access your build environment's data or tamper with it undetected. This includes cloud provider employees and your own infrastructure team. **Build infrastructure outside the TEE.** Your CI system, build servers, artifact storage, and distribution channels are untrusted. They can trigger builds and receive outputs, but they cannot observe or tamper with the build process itself. ## What Attested Builds Protect Against The protection comes from two sources: TEE isolation during the build, and cryptographic binding between inputs, the build process, and outputs. ### Tampering During the Build An attacker who compromises the build platform and injects malicious behavior during builds would normally be undetectable. The source code is clean, but the resulting binaries are backdoored. Attested builds address this by running the build inside a TEE. The build environment is hardware-isolated from the host. The attestation proves what code was loaded and that the environment wasn't tampered with. If someone modifies the build process, the measurement changes, and verification fails. ### Tampering After the Build An attacker who gains access to artifact storage could replace legitimate builds with malicious ones. Users download what they think is the official binary. Attested builds address this by cryptographically binding outputs to the attested build. The provenance contains hashes of all output artifacts, and the provenance itself is bound to the TEE attestation. If someone replaces an artifact, the hash won't match. If someone forges new provenance, it won't have a valid TEE attestation. ### Forged Provenance Without TEEs, provenance is just a claim. Anyone with access to signing keys can create a JSON file saying "this binary came from this source." Attested builds change this. The provenance is bound to a hardware-signed attestation. The attestation is signed by a key derived from secrets fused into the CPU at manufacturing, keys that no software can access. Forging provenance requires compromising the TEE hardware itself. ### Dependency Substitution If an attacker tries to swap in a different dependency during the build (different from what's pinned in your lockfile), the checksum won't match, and verification fails. The provenance records exact versions and checksums for every dependency, making substitution attacks detectable. ## What Attested Builds Do NOT Protect Against These aren't weaknesses to be fixed. They're architectural constraints. Being clear about them is essential for understanding the actual security posture. ### Malicious Source Code Attested builds verify that specific source was used, not that the source is safe. If the upstream repository contains a backdoor, the attested build faithfully builds the backdoor. The provenance accurately records that the backdoored commit was used. Everything works as designed, and you still have malware. Attestation proves identity, not intent. ### Compromised Upstream Dependencies This depends on how Kettle handles dependencies. **In cached mode:** Dependencies are fetched and verified once when the cache is populated. After that, builds use the cached artifacts. The registry is only trusted at cache population time. If a malicious version enters the cache, subsequent builds will use it, but the window of exposure is limited to when the cache was built. **In upstream mode:** Dependencies are fetched from registries for each build. If a registry serves a malicious package (with a valid checksum because the malicious version is what's actually published), attested builds will verify that you built with that malicious package. The checksums match because you're building exactly what's in the registry. In both cases, attested builds verify the integrity of the build process given specific inputs. They don't verify that the inputs themselves are trustworthy. ### TEE Hardware or Firmware Compromise The trust anchor is the CPU vendor's hardware and key management. If these are compromised, the guarantees break. Firmware vulnerabilities have been found in TEE implementations before; they will be found again. This is a much smaller attack surface than trusting the entire host. But it's not zero. You're trading a large, complex trust surface for a small, well-defined one. ### Side-Channel Attacks TEEs share microarchitectural state with the host. Cache timing, branch prediction, and other side channels can leak information. TEE vendors have added mitigations that raise the bar significantly, but side channels remain an active research area. Attested builds reduce the attack surface compared to unprotected builds. They don't eliminate all information leakage. ### Availability The host controls whether your build runs. It can terminate the TEE at any time. Attested builds protect confidentiality and integrity, not availability. ### Bugs in Your Own Code The TEE protects the execution environment, not the code running in it. If your application has a vulnerability, an attacker can exploit it. Attestation proves what code loaded, not that the code is correct. ## The Security Delta ### Before Attested Builds You trust your build infrastructure, its operators, and the distribution channel. Verification is policy-based: you trust that people followed the right procedures, that access controls were configured correctly, that credentials weren't compromised. ### After Attested Builds You trust the TEE hardware vendor, the physical host, and source infrastructure. The build infrastructure and distribution channel become untrusted. Verification is cryptographic: you check signatures against hardware-rooted keys, verify that provenance hashes match attestation reports, confirm that artifact hashes match provenance claims. The trust model shifts from trusting infrastructure to trusting hardware vendors. This is a much smaller trust surface. --- # What Are Attested Builds? This document explains what attested builds are, why they matter, and how they solve the software verification problem. After reading it, you'll understand the core insight behind attested builds and why Trusted Execution Environments make them possible now. We assume familiarity with git, package managers, and the concept of cryptographic hashes. ## The Verification Problem One party wants assurances about another party's software. The specific assurances vary by context: - A customer deploying your service wants to know it's running the code you claim, built from the source they audited. - A compliance team wants evidence that a binary was built with specific dependency versions, not newer ones with unknown changes. - A security auditor wants to verify that the toolchain used to compile a release matches the one specified in your security documentation. - A regulated enterprise wants proof that sensitive data will be processed only by code that passed their review, not by a modified version. - A package consumer wants to ensure that the binary they downloaded corresponds to the source code and dependencies they reviewed, not a tampered version. The common thread: someone who didn't build the software needs to verify claims about how it was built. At small scale, this is might be manageable through process. You can walk an auditor through your build system, show them your CI logs, explain your release procedures. They decide whether to trust your operational controls. At scale, process-based trust breaks down. A package registry serves millions of artifacts to millions of consumers. A cloud platform runs workloads for hundreds of thousands of customers. No one can personally audit every build. Consumers rely on claims: signed manifests, published hashes, attestations of policy compliance. But claims are not evidence. The signature proves who made the claim, not that the claim is true. ## Reproducible Builds: The Holy Grail The theoretical ideal for software verification is reproducible builds: given the same source code, build environment, and build instructions, any party can recreate bit-for-bit identical artifacts. If compilation is fully deterministic, verification becomes straightforward. Compute the expected hash from the source, compare it to the artifact's hash, done. This is the holy grail because it eliminates trust entirely. You don't need to trust the builder, the build infrastructure, or any claims about the build process. You trust math. If the hashes match, the artifact is correct. Reference measurements become meaningful because anyone can reproduce them. The problem is that most toolchains are non-deterministic. Compilers embed timestamps. Parallel builds produce outputs in varying order. Linkers record file paths. Archive tools don't preserve consistent ordering. Achieving bit-for-bit identical output requires controlling all of this across your entire dependency tree. A single non-reproducible component anywhere in the chain breaks the guarantee. Projects like Reproducible Builds have made significant progress, but full reproducibility across an entire software ecosystem remains elusive. ## But What About Nix? Nix is often cited as a solution to the reproducibility problem, but this usually a confusion between two different concepts: reproducible build environments and reproducible build outputs. Nix provides reproducible build environments. Given a flake.lock or derivation, Nix guarantees you will build with the exact same inputs: the same source, the same dependencies, the same toolchain. This is a significant improvement over traditional package managers where dependency resolution can vary between machines or over time. But reproducible inputs don't guarantee reproducible outputs. The same source built with the same compiler can still produce different binaries due to non-determinism in the compilation process itself. Nix makes the build environment deterministic, not the build output. Of course, Nix can produce bit by bit reproducible builds as well, but that is depdendent on the underlying toolchain being deterministic, which is not the case for a lot of software out there. There's also an issue of how do you trust Nix itself? A compromised Nix installation can produce bogus derivations. A malicious actor with access to your build machine can modify the Nix toolchain to inject code regardless of what your flake.lock specifies. The derivation hashes prove consistency with a particular Nix evaluation, but they don't prove that evaluation was honest. ## Attested Builds: A Different Approach Here is the core insight: if you can cryptographically verify how something was built, and what went into that build, then bind it to the outcome, then that gives you a lot of the guarantees you'd want out of reproducible builds. It is a shift from asking "does this binary have hash X?" to asking "was this binary with hash X produced by process Y from sources Z in environment W?" With **reproducible builds**, you build the same source twice and check whether both outputs are bit-for-bit identical. If they are, you know the artifact matches the source. The problem is that this fails whenever timestamps, file ordering, or compiler optimizations introduce non-determinism. They almost always do. With **attested builds**, you take a different approach. You feed your source, dependencies, and toolchain into a hardware-isolated build environment, and you get back both the binary and a provenance record. Verification then becomes three questions: 1. Was the build environment tamper-proof? 2. Were the inputs the expected ones? 3. Is the output cryptographically bound to that environment and those inputs? This is a mouthful to state, but it turns out these verifications are much easier to achieve than bit-for-bit reproducible builds. You don't need to eliminate non-determinism from every compiler, linker, and archive tool in your dependency tree. You need to verify the process and bind the result to it. This reframes the problem entirely. You're not trying to achieve deterministic compilation. You're trying to create a cryptographic chain of evidence that binds your output to auditable inputs through a process that can be independently verified. A verifier checks cryptographic signatures and attestation reports. The verification is independent: anyone can verify without trusting the builder's claims. ## Why Now: TEEs as Root of Trust Why is the attested builds approach possible now? How do we actually verify the statement "was this binary with hash X produced by process Y from sources Z in environment W?" One answer is Trusted Execution Environments. TEEs provide a root of trust that is cryptographically attested. Let's unpack what that means. A TEE is a hardware-isolated execution environment where the CPU itself enforces protections that software cannot override. You can think of it as a hardened and encrypted version of a VM. Even the operating system, hypervisor, and cloud operators cannot read or tamper with code running inside a TEE. Three primitives matter for attested builds. **Isolation.** Code runs in a hardware-protected environment. The host system (hypervisor, cloud provider, operators) cannot access memory or tamper with execution. Memory is encrypted with a per-VM key that only the CPU's security processor knows. When the hypervisor reads a guest's memory region, it gets ciphertext. The encryption key is generated by hardware, stored in hardware, and never exposed to any software. **Integrity.** The hardware detects if the host tampers with guest memory. Substituting pages, replaying old data, or remapping addresses all trigger faults. The host cannot silently corrupt guest state. This is enforced by hardware checks on every memory access. **Attestation.** The guest can prove to a remote party what code it's running. The CPU's security processor measures the initial guest image and signs a report with a key rooted in the silicon. A verifier can check this signature against the hardware vendor's certificate chain to confirm the report came from real hardware, not from software pretending to be a confidential VM. This does not guarantee zero trust. You trust the CPU vendor's silicon and firmware. You trust that their manufacturing process didn't embed backdoors. You trust that their key management practices keep root keys secure. You trust that the security processor firmware doesn't have exploitable vulnerabilities. You don't trust the cloud provider's software stack. The hypervisor, host OS, management plane, orchestration systems, and monitoring agents are all outside the trust boundary. They can be compromised, malicious, or buggy without affecting the confidentiality or integrity of your workload. You also don't trust the cloud provider's employees. Administrators with root access to the hypervisor, and anyone in the operational chain cannot read your memory or tamper with it undetected. This is a much smaller trust surface than trusting the entire infrastructure. The TEE provides a hardware root of trust that allows you to verify that a specific process ran in a specific environment with specific inputs, and that the output is cryptographically bound to that process. This is the foundation that makes attested builds possible now. This creates the root of trust needed to make process verification meaningful. Without TEEs, "we verified the inputs" is just a claim. With TEEs, it's a hardware-signed assertion. Yes it is not zero trust, but it is still a huge upgrade security wise. ## What Attested Builds Give You Attested builds provide three properties that together close the verification gap. **Supply chain verification.** The artifact was built from a specific source commit, with specific dependency versions, using a specific toolchain. Each input is cryptographically identified, not just named. The source is pinned to a git commit hash and tree hash. Dependencies are pinned via lockfiles with each package identified by cryptographic checksum. The toolchain (compiler, linker, build tools) is hashed. All of these become leaves in a Merkle tree, producing a single root hash that uniquely identifies the complete set of build inputs. If any input changes by a single byte, the root changes. **Build process integrity.** The build executed in a hardware-isolated environment that can be remotely verified. The TEE attestation proves the build environment wasn't tampered with and that only the specified process ran. Networking is disabled after inputs are loaded. The build environment is isolated: nothing can be injected or exfiltrated during compilation. The TEE generates an attestation report proving the integrity of the loaded environment, signed by hardware and rooted in keys that only the CPU can access. **Verifiable chain of custody.** Every link from source commit to running code is cryptographically bound. Verification is fast. Instead of rebuilding, a client checks cryptographic signatures and attestation reports. This is feasible at scale. The chain looks like this: 1. Source commit (signed by developers) 2. Dependencies (pinned by hash) 3. Toolchain (hashed) 4. Merkle root of all inputs 5. TEE attestation (hardware-signed proof of build environment) 6. Output artifacts (signed by TEE-derived key) 7. Runtime measurement (referenced in both build and runtime attestation) Every link is verifiable. The chain connects auditable source to running code with no gaps. This doesn't require your builds to be reproducible. It requires them to be consistent enough that you can verify the inputs and trust the process. The TEE attestation substitutes for bit-for-bit determinism: instead of proving "anyone would get the same output," you prove "this specific process ran in a verified environment with these specific inputs." --- # Overview **c8s** (Confidential Kubernetes) makes any Kubernetes cluster confidential. You have a cluster, you want it confidential — c8s is the switch. Your workloads' data and artifacts — model weights, prompts, responses, datasets, and credentials — stay inside hardware-backed Trusted Execution Environments (TEEs) from the moment they enter the cluster until they leave it: encrypted in CVM memory, hardware-isolated on the GPU, and that protection is **cryptographically provable** to any independent third party. Private inference, training and fine-tuning, agents, or any existing Kubernetes workload can inherit that guarantee with minimal changes — c8s keeps the Kubernetes surface you already use. The design assumption is simple and uncompromising: **the host is adversarial.** The cloud provider, the bare-metal operator, and anything that can read raw host memory or inspect a QEMU command line are all outside the trust boundary. "It works on a normal cluster" is not the bar. It must work on a confidential one, with the threat model intact. ## Why Confidential Kubernetes Running workloads on someone else's infrastructure creates a **three-sided trust problem**, and contractual promises do not solve any side of it: - **Artifact owners** — model weights, datasets, proprietary code — risk exfiltration the moment their artifacts touch third-party infrastructure. - **Compute providers** want to host valuable workloads without being able (or liable) to observe them. - **End users** in healthcare, legal, financial, and national-security settings need a hard guarantee that their inputs and outputs stay opaque to the operator. Hardware TEEs (AMD SEV-SNP, Intel TDX, and NVIDIA Confidential Computing) resolve all three at once: they **encrypt the runtime** in hardware, **measure the code** that booted, and let that measurement be **attested** to a remote verifier. c8s turns those primitives into a working Kubernetes platform — even on managed services like GKE and AKS where the control plane itself cannot be attested. > Read the full design in the [c8s whitepaper](https://arxiv.org/abs/2604.26974) *(arXiv)*. ## The five principles c8s applies the same five principles end-to-end: 1. **Encrypt the runtime** — workloads run in hardware-encrypted memory the host cannot read. 2. **Measure the code** — the hardware computes a launch digest over exactly what booted. 3. **Bind identity to measurement** — credentials are issued *only* after the measurement is verified, so a certificate is proof of what is running. 4. **Verify before connecting** — peers require an attestation-rooted identity before traffic flows. 5. **Secure the egress** — all traffic is encrypted to verified destinations. ## Where to go next The one decision to make before anything else is the **deployment shape** — each pod as its own confidential VM, or the whole node as one. It sets the trust boundary and it is not something you flip later. See [trust boundaries](/docs/c8s/concepts/trust-boundaries). --- # Application secrets **c8s** releases application secrets under attestation. A confidential pod asks the **CDS** (Certificate Distribution Service) for a value — an API token, a database password, a session key — and receives it only if the images actually running in that pod's sandbox match an [allowlist](/docs/c8s/concepts/image-allowlist) entry that grants the path. The value arrives as a file on a memory-backed volume inside the pod. It is never a Kubernetes Secret, never reaches etcd, and never leaves the memory of the **TEE** (Trusted Execution Environment). One thing dominates how you write the consumer: **the file appears after your container starts.** Read [The file appears after your container starts](#the-file-appears-after-your-container-starts) before anything else on this page. To give a workload a secret, see [Use application secrets](/docs/c8s/how-to/use-application-secrets). ## Release is gated on a grant The gate is a `secrets` grant on the workload entry that describes the pod. The grant is written with the operator key, served by the CDS alongside the rest of the allowlist, and changeable without restarting anything. **An entry without a grant releases nothing.** Writing the grant is what turns release on for a workload, and the endpoint is inert until one exists. The grant belongs to the whole entry rather than to a container, because the value is delivered on a volume every container in the pod can read — a per-container grant would not describe what is actually released. The grant's shape is in [The image allowlist](/docs/c8s/concepts/image-allowlist#data-model); writing one is [Grant secret paths](/docs/c8s/how-to/manage-the-allowlist#grant-secret-paths). ## The file appears after your container starts **Wait for the file; do not read it at startup.** The CDS releases only once **every** main container in the pod is running — that is the moment the sandbox's container set matches a whole workload entry. The fetcher starts alongside your workload, is refused while the set is incomplete, and writes when it completes. A consumer that reads its secret path at startup finds nothing there. ```bash until [ -f /run/c8s/secrets/DB ]; do sleep 1; done ``` ```text t0 c8s-cert writes the pod's mesh leaf t1 c8s-cert-wait gate clears; main containers may start t2 c8s-secret asks the CDS ──► refused, the container set is incomplete api running — /run/c8s/secrets/DB does not exist yet t3 (every main container is now running) c8s-secret asks again ──► released; DB written atomically api its wait loop returns ``` *The consumer is running before its secret exists. The gap is structural, not a race to be tuned away.* Each value is written to a temporary file and renamed into place, so a poll never reads a torn one, and the whole set is written at once — a consumer never gets some of its secrets and waits forever for the rest. Two consequences follow, and neither has a workaround: - **An init container cannot close the gap.** It would be asking before its siblings exist, and would deadlock the pod it gates. The fetcher is a native sidecar for exactly this reason. - **There is no fail-closed delivery gate.** A terminal fetch failure leaves a `Running` pod with no secret and an `Init:CrashLoopBackOff` sub-status — the fetcher lives in `initContainers`, so that is where a crash surfaces. It retries 60 times at 5-second intervals before it exits, and the kubelet restarts it into backoff, so a pod that is only waiting on a slow start recovers on its own. A pod refused for a policy reason retries forever without progress; see [Diagnosing a refusal](/docs/c8s/how-to/use-application-secrets#diagnosing-a-refusal). ## What the webhook injects Setting `confidential.ai/c8s-secrets` makes the pod webhook add two things: - **`c8s-secret`**, the fetcher sidecar. It runs the c8s image, authenticates to the CDS with the pod's mesh leaf, and is ordered after the `c8s-cert-wait` gate so that leaf is already on disk. - **`c8s-secrets`**, a memory-backed `emptyDir` mounted **read-only into every container in the pod** at the secret directory. The fetcher's own mount is the only writable one, so a compromised workload container cannot replace a value another container has yet to read. The container name and the volume name are both reserved: a pod that declares its own `c8s-secret` container, or declares `c8s-secrets` as anything but a memory-backed `emptyDir`, is rejected at admission — a `hostPath` there would write a released secret to host-visible storage. The full set of reserved-name rules, including what happens to a pre-declared mount and to `kubectl debug`, is on [Kata Containers](/docs/c8s/concepts/kata-runtime#reserved-containers-and-volumes). Files are written mode `0640` onto a volume group-owned by the pod's `fsGroup` — `65532` unless the pod sets its own — and every container in the pod carries that group, so a consumer running as any UID can read them. The fetcher pins the CDS to the launch measurements the install configured (`cds.measurements`, which the operator forwards to every injected sidecar). With no measurements pinned, an impostor CDS can answer with a value of its choosing — the same warning that applies to the rest of the mesh. The in-pod destination is **not** an authorization boundary. The workload owns its own filesystem once the value is inside it; only the store path is policy. ## Where the value comes from **A workload that finds its path empty creates it.** The fetcher reads with `GET`, and on a path the store does not hold yet it `POST`s, which makes the CDS mint 32 random bytes and store them. The first pod of a workload to ask is the one that defines the value. A replica that loses the race gets a `409` with no body — returning the value there would turn a write grant into a read grant — and re-reads with `GET`. A `POST` never carries a value, so no caller chooses what another caller will later read. That covers a session key. It does not cover an API token, a database password, or a wrapped [volume key](/docs/c8s/concepts/encrypted-volumes): those come from an operator. ## What the CDS checks before it releases Every request carries a single-use challenge and a fresh sandbox token, so a release is bound to one caller and cannot be replayed. The CDS then decides in this order: 1. The client certificate chains to the mesh CA. It is the pod's mesh leaf, verified by TLS itself; the CDS-stamped [sandbox ID](/docs/c8s/concepts/trust-root#sandbox-identity) on it is the caller's identity. 2. The sandbox token verifies against the node's admission inventory, carries this request's challenge, and names the same sandbox as the leaf. 3. The inventory asked is the one the CDS bound to that sandbox at certificate issuance — first-write-wins — not the one the request names. A conflicting binding leaves the sandbox unusable for secrets without invalidating its certificate. 4. That inventory reports what the sandbox has run, minus the containers c8s injects. 5. The remaining set matches **exactly one** workload entry. 6. That entry's grant covers the requested path. Any failure refuses. So does an unreachable inventory, an unknown sandbox, an empty container set, an entry with no grant, and an ambiguous match — two entries that no running set can tell apart refuse both, which is why `c8s allowlist` errors on them at write time. Denials are opaque on the wire. An ungranted path answers `404`, indistinguishable from a path that does not exist, so the API cannot be used to enumerate the store; the reason goes to the CDS log. The workload routes are rate-limited **per sandbox**, keyed on the ID in the verified client certificate, because every pod on a node reaches the CDS from the same address — bounded on the address instead, one pod could spend a budget its co-tenants share. What the inventory reports is a **high-water mark**: every container ever admitted in the sandbox, not those running right now. A pod that ever ran an image outside its entry never receives a secret, even after that image is gone. Attaching a `kubectl debug` container to a pod joins that record permanently, so a pod that has not yet received its secret will not receive it afterwards. ## When the CDS will not serve `/secrets` at all Answering at all needs what sandbox identity already needs. Miss any of these and the CDS logs a warning naming the one it is missing, and does not serve `/secrets`: | Flag | Why it is required | | --- | --- | | `--ratls-platform` | Without it the CDS has no attested channel to an inventory. | | `--measurements` | Without it any TEE could answer as a sandbox's inventory. | | `--sandbox-inventory-cidr` | Bounds which addresses the inventory callback may dial. | It also refuses to serve when [CA handoff](/docs/c8s/concepts/trust-root#ca-handoff) is configured (`--handoff-peer-url` or `--handoff-measurements`). A handoff roll puts two CDS pods behind the Service at once and the surge replica serves an empty store, so a workload landing on it would mint a value diverging from the one its siblings already hold, with no error anywhere. On a cluster in either state, a pod annotated `confidential.ai/c8s-secrets` starts, never gets its file, and blocks. Check the CDS log before assuming the grant is wrong. ## Limits | Flag | Default | Bounds | | --- | --- | --- | | `--secrets-max-paths` | `1024` | Distinct secret paths held in memory. | | `--secrets-max-value-bytes` | `4096` | Bytes in one secret value. | | `--sandbox-ledger-max-entries` | `10000` | Sandbox-to-inventory bindings held in memory. | All three are fail-closed: past the bound a write is refused, nothing is evicted to make room. The CDS is a single in-memory process holding the mesh CA, so a workload able to grow either map without limit could take it down and every certificate in the cluster with it. The chart renders no overrides for these, so a Helm install runs at the defaults; see the [CLI reference](/docs/c8s/reference/cli#c8s-cds). ## A CDS restart clears the store **A CDS restart destroys every secret, and requires rolling every workload that holds one.** The store is process memory and there is no persistence. Worse than losing the values: a pod recreated after the restart calls `POST`, finds its path empty, and is given a **new** value while its siblings still hold the old one. Nothing reports this, and a partially rolled Deployment ends up with two different values for one path. So after a CDS restart, roll every secret-consuming Deployment rather than letting pods recover piecemeal, and re-`put` every operator-supplied value. Treat a released value as ephemeral for the lifetime of the CDS process: nothing durable may be keyed on one. The sandbox ledger is process memory too, so a leaf that outlives a restart has no inventory binding until its next renewal and is refused meanwhile. ## Not available under Pod-as-CVM Secrets are out of scope for **Pod-as-CVM**, and the webhook rejects `confidential.ai/c8s-secrets` at admission there rather than admitting a pod that would hang. `c8s install --cvm-mode=pod` refuses `--measurements` — it pins the node CVM's launch measurement, which the per-pod guests do not have — so the measurement requirement above is unmeetable there, and the fetcher's node-side inventory socket does not exist inside a guest either. See [Kata Containers](/docs/c8s/concepts/kata-runtime#reserved-containers-and-volumes). --- # Components & Data Flow **c8s** adds five capabilities to a Kubernetes cluster: **TEE attestation verification**, **certificate management** (RA-TLS + a mesh CA), **RA-TLS mesh networking**, **container-image policy enforcement**, and an **attested public front door**. This page describes the components that provide them and how requests flow between them. ## The components | Component | Image | Node-as-CVM (`--cvm-mode=node`) | Pod-as-CVM (`--cvm-mode=pod`) | | --- | --- | --- | --- | | **CDS** — Certificate Distribution Service | `cds` | runc container | its own Kata CVM | | **c8s operator** + admission webhook | `c8s-operator` | runc container (untrusted host) | runc container (untrusted host, exempt) | | **attestation-api** | `attestation-api` | baked into the node image, host loopback `:8400` | baked into the guest image, `127.0.0.1:8400` | | **ratls-mesh** | `ratls-mesh` | host DaemonSet (L4 proxy) | in-guest, baked into the guest image | | **nri-image-policy** | `nri-image-policy` | baked into the node image (NRI plugin) | in-guest `policy-monitor`, baked into the guest image | | **get-cert** — the `c8s-cert` sidecar | `c8s-operator` | webhook-injected native sidecar | webhook-injected native sidecar | | **get-secret** — the `c8s-secret` sidecar | `c8s-operator` | webhook-injected native sidecar, on `confidential.ai/c8s-secrets` | rejected at admission | | **get-volume** — the `c8s-volume` sidecar | `c8s-operator` | webhook-injected native sidecar, on `confidential.ai/c8s-volumes` | rejected at admission | | **tls-lb** — the public front door | `c8s-operator` + upstream nginx | Deployment: nginx, `cds-attest`, allowlist proxy | same, pinned to a Kata CVM | | **volumed** — encrypted-volume node agent | `volumed` | node DaemonSet, **off by default** | node DaemonSet, **off by default**; no pod can request a volume here | Every c8s image ships the *same* multi-mode `c8s` binary and differs only in its entrypoint subcommand and base layer, so a role is a command, not a separate program — the chart runs `c8s-operator` for the controller, the injected `c8s-cert`, [`c8s-secret`](/docs/c8s/concepts/application-secrets), and [`c8s-volume`](/docs/c8s/concepts/encrypted-volumes) sidecars, and the tls-lb sidecars alike. Most images are distroless; `ratls-mesh` is debian-slim because it needs `iptables`, and `volumed` is debian-slim because it needs `cryptsetup`/`veritysetup`. All are published under `ghcr.io/confidential-dot-ai/`, alongside a standalone `get-cert` image for use outside the chart. Two images come from elsewhere: `attestation-api` is built from the Rust `attestation-rs` tree, and tls-lb's web server is upstream `nginxinc/nginx-unprivileged`. Where those services live depends on the shape. Under `--cvm-mode=node` the confidential node image (`c8s-base`, built and published by c8s CI on every merge to `main`) bakes the attestation-api and a fail-closed nri-image-policy, so the chart installs neither as a workload; ratls-mesh is not baked and stays a DaemonSet. Under `--cvm-mode=pod` the same services are baked into the measured Kata guest image instead. On the managed shapes (`--cvm-mode=gke`, `--cvm-mode=aks`) the node image is the cloud's, not ours, so the chart installs attestation-api and nri-image-policy as ordinary host workloads — and there the chart **requires** nri-image-policy: `nriImagePolicy.enabled=false` fails the render, because it is the only image-admission gate that would be left running. The CDS is the trust root. It performs attestation verification, mesh-CA signing, and optional CA handoff **all in one process**, so there is no internal RA-TLS hop to an external signer and the CA private key never leaves CDS memory. The **c8s operator and its admission webhook are untrusted.** Even under pod-as-CVM (Kata) they stay in a plain `runc` container on the host — outside the per-pod TEE boundary — and they are *exempt* from the workload enforcement they configure. They are control-plane orchestration, **not** part of the trust boundary: a compromised operator can disrupt scheduling (e.g. refuse to inject `get-cert`, which fails closed under the webhook's `Fail` policy) but it **cannot** forge attestation, mint certificates, or read TEE memory — those happen inside the CDS, in the TEE. The cluster's confidentiality never depends on trusting the operator. ### The admission inventory The image-policy component does double duty, and the second job is easy to miss. Whichever enforcer admitted a pod's containers — `nri-image-policy` on the node, `policy-monitor` inside the Kata guest — is also the **admission inventory**: the authority on which pod sandbox a process belongs to, and on what is actually running in that sandbox. It serves two disjoint surfaces. A local token surface mints an inventory-signed **sandbox token** for the calling process, where the caller is identified by kernel peer credentials (node-as-CVM) or by the guest boundary itself (pod-as-CVM, one pod per guest) — never by anything the caller sends. A separate network surface, on a privileged port reachable only from the node's own network namespace and over mutually-attested RA-TLS, answers CDS's queries for the inventory's signing key and for a sandbox's current image digests. CDS uses both when issuing a leaf: it verifies the token, gates issuance on every running image being allowlisted, and stamps the sandbox ID into the certificate's signed area. That is what lets a relying party ask "*which* workload is behind this key?" rather than only "is this a genuine TEE?". The mesh CA signature — not hardware evidence — is what vouches for that ID; see [Sandbox identity](/docs/c8s/concepts/trust-root#sandbox-identity) for the full treatment. ## Node-as-CVM and Pod-as-CVM The trust boundary is a deployment-time choice — see [Pod-as-CVM vs Node-as-CVM](/docs/c8s/concepts/trust-boundaries) for the full tradeoff. `c8s install` **requires** you to state it with `--cvm-mode` (`pod`, `node`, `gke`, or `aks`; there is no default), and **Kata is the mechanism c8s uses to give each pod its own CVM** — not the name of a mode: - **Node-as-CVM** (`--cvm-mode=node`, and the managed `gke` / `aks` variants) — the c8s services run as ordinary node containers and the mesh and attestation operate at node level. The node itself is a confidential VM (an Azure AKS or GKE confidential node, or a bare-metal AMD SEV-SNP / Intel TDX host), so the whole node is the trust boundary and everything on it inherits it. One launch digest covers the lot, which is why `c8s install --measurements ` can pin the whole mesh in a single step here. - **Pod-as-CVM** (`--cvm-mode=pod`) — each workload pod runs in its own confidential VM, launched via **Kata Containers**. The security services (`attestation-api`, `ratls-mesh`, `policy-monitor`) are **baked into the measured guest image**, so they are covered by the launch measurement and the host — adversarial here, since it sits outside every pod CVM — cannot MITM them. Both AMD SEV-SNP (`kata-qemu-snp`) and Intel TDX (`kata-qemu-tdx`) are wired, selected per install by `--hardware-platform`; a cluster runs one CPU TEE, not both. Pod-as-CVM enforcement is two cooperating pieces — both, or nothing: 1. A **mutating webhook** injects a `runtimeClassName` into workload pods that lack one — the confidential class (`kata-qemu-snp` or `kata-qemu-tdx`, and their `-nvidia` variants for GPU pods) for pods annotated `confidential.ai/cw`, or `kata-qemu` otherwise. 2. A **ValidatingAdmissionPolicy** (`c8s-kata-enforcement`) rejects workload pods that request a non-Kata `runtimeClassName`. The webhook's failure policy is **Fail**: if the webhook is unreachable, pod creation is rejected, so an unmutated `runc` pod can never slip through. Two further policies back it up in both shapes — `c8s-cw-label-integrity` denies a `confidential.ai/cw` pod that has shed its injected `c8s-cert` sidecar, and `c8s-deny-host-namespaces` denies tenant pods `hostNetwork`, `hostPID`, `hostIPC`, and `hostPath` volumes, which is what keeps the admission inventory's privileged port out of a tenant's reach. ## Trust boundaries Under **node-as-CVM**, the boundary is the node. The whole node — c8s services and workload pods alike — sits inside one TEE the host cannot read: ```text ╔═ TEE BOUNDARY · the whole node is one CVM, host cannot read ═══════════════╗ ║ ║ ║ ┌──────────────── Kubernetes node = one confidential VM ─────────────┐ ║ ║ │ attestation-api · nri-image-policy │ ║ ║ │ (baked into the node image on --cvm-mode=node; │ ║ ║ │ chart DaemonSets on gke / aks) │ ║ ║ │ chart-installed: │ ║ ║ │ CDS · ratls-mesh · operator + webhook · tls-lb │ ║ ║ │ workload pods (runc), each with a get-cert sidecar │ ║ ║ └────────────────────────────────────────────────────────────────────┘ ║ ╚════════════════════════════════════════════════════════════════════════════╝ host side (adversarial): hypervisor · cloud or bare-metal operator ``` Where those two components live depends on whether you control the node image. On `--cvm-mode=node` you build and measure it yourself, so they are baked in and covered by the node's launch digest. On the managed shapes (`gke`, `aks`) the provider owns the image and c8s cannot add to it, so the chart runs them as DaemonSets inside the same CVM boundary instead. *One launch digest covers everything on the node, including the kubelet.* The corollary is that pods inside the node are separated only by the kernel. That is fine — node-as-CVM is a single-tenant shape and the adversary is outside the node — but see the [threat model](/docs/c8s/concepts/threat-model) for what c8s does enforce between workloads on one node. Under **pod-as-CVM**, the boundary is each pod's own CVM; the host is outside every one of them: ```text ╔═ TEE BOUNDARY · per-pod confidential CVM, host cannot read ════════════════╗ ║ ║ ║ ┌──── kata-qemu-snp/tdx CVM ────┐ ┌──── kata-qemu-snp/tdx CVM ────┐ ║ ║ │ CDS │ │ workload │ ║ ║ │ RA-TLS serving cert │ │ + c8s-cert sidecar │ ║ ║ │ (SNP / TDX evidence) │ │ (leaf cert from CDS) │ ║ ║ │ baked in: attestation-api · │ │ baked in: attestation-api · │ ║ ║ │ ratls-mesh · policy-monitor │ │ ratls-mesh · policy-monitor │ ║ ║ └───────────────────────────────┘ └───────────────────────────────┘ ║ ╚════════════════════════════════════════════════════════════════════════════╝ host side (adversarial): containerd · kata-runtime · kata-shim · operator ``` *Every pod carries its own launch digest, so each attests independently.* ## Certificate flow — node-as-CVM A workload obtains a TEE-bound certificate from the CDS in one issuance. The nonce CDS mints for the challenge is reused as the sandbox token's nonce, so one single-use value binds both the hardware evidence and the identity claim — there is no clock and no replay window: ```text Workload CDS attestation-api admission [c8s-cert] [same TCB] inventory │ │ │ │ │ POST /authenticate │ │ │ │───────────────────►│ │ │ │ single-use nonce │ │ │ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ │ │ │ │ POST /sandbox — CSR public key + that nonce │ │───────────────────────────────────────────────────────────►│ │ inventory-signed sandbox token │ │ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ │ │ │ report_data = SHA-384(CSR public key ‖ nonce) │ │ │ │ │ │ POST /attest — evidence + CSR + token │ │ │───────────────────►│ │ │ │ │ verify(evidence, report_data) │ │ │────────────────────►│ │ │ │ valid · launch_digest │ │ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│ │ │ │ GET /identity · GET /digests/{sandbox}│ │ │──────────────────────────────────────►│ │ │ │ │ │ check measurement · every running image allowlisted │ │ validate CSR · sign with the mesh CA │ │ │ │ │ │ │ leaf cert + CA chain, sandbox ID in the signed area │ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ │ │ │ ``` *get-cert never reports its pod's images — CDS reads them live from the component that admitted them.* ## Certificate flow — pod-as-CVM (Kata) Under pod-as-CVM the same flow happens **inside the pod's TEE boundary**. The CDS mints its own RA-TLS serving cert from its hardware evidence, the workload reaches it over RA-TLS, and the attestation-api is the in-guest service at `127.0.0.1:8400`. The inventory is `policy-monitor` in the same guest, reached on guest loopback rather than a mounted socket, so there is no caller to disambiguate — one pod per guest. The trust anchor is the **launch measurement**: CDS, attestation-api, ratls-mesh, and policy-monitor are all baked into the guest image and therefore covered by it. ```text ╔═ SEV-SNP / TDX BOUNDARY · HOST CANNOT READ ══════════════════════════════╗ ║ ║ ║ Workload CVM CDS CVM in-guest services ║ ║ [c8s-cert] [mesh CA] [127.0.0.1] ║ ║ │ │ │ ║ ║ │ RA-TLS connect, verify CDS measurement │ ║ ║ │────────────────────►│ │ ║ ║ │ │ │ ║ ║ │ sandbox token, over guest loopback │ ║ ║ │◄────────────────────────────────────────────► ║ ║ │ │ │ ║ ║ │ POST /attest — evidence + CSR + token │ ║ ║ │────────────────────►│ │ ║ ║ │ │ verify in-process │ ║ ║ │ │──────────────────────►│ ║ ║ │ │ verdict + launch_digest ║ ║ │ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│ ║ ║ │ │ │ ║ ║ │ leaf cert from the mesh CA │ ║ ║ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│ │ ║ ╚══════════════════════════════════════════════════════════════════════════╝ ``` *Nothing in this exchange crosses the host — the whole issuance is inside measured memory.* ## The public front door External traffic enters through **tls-lb**, a Deployment whose pod holds nginx alongside three c8s containers: a `c8s-cert` sidecar that keeps nginx's serving certificate fresh, and the two below. Under Kata the whole pod is pinned to a confidential RuntimeClass, because it terminates client TLS with mesh-issued keys and that plaintext must stay inside the TEE. - **`cds-attest`** (`tlsLb.attest.enabled`, **on by default**) serves the browser-facing `c8s-verify/v1` protocol on loopback: the per-session attestation challenge and the post-quantum over-encryption handshake. nginx reverse-proxies `/.well-known/c8s/` to it. Exposing it does not weaken the server — it publishes a signed quote of its own TEE and nothing else. Trust is established on the *client*, which must pin expected measurements. - **The allowlist proxy** (`tlsLb.allowlist.enabled`, **on by default**) publishes CDS's complete `/allowlist` API through the front door. It verifies CDS's RA-TLS evidence before forwarding, so the public hop and the in-cluster hop are both attested. Reads stay unauthenticated at CDS and are rate-limited per client; writes still require the short-lived, body-bound operator token, and the operator private key never enters the cluster. See [Allowlist](/docs/c8s/concepts/image-allowlist). **The protocol endpoints serve wide-open CORS by default.** `tlsLb.cors.protocolEndpoints` defaults to `true`, which sends `Access-Control-Allow-Origin: *` (with no credentials) on `/.well-known/c8s/`, the discovery document, the certificate endpoints, and the built-in `/allowlist` route. Those responses are self-authenticating — hardware evidence, CDS-signed certificates, sealed tunnel records — and no request on them relies on ambient browser credentials, so an origin allowlist would block third-party verifiers without protecting anything. Your **workload** routes and the catch-all upstream are never covered by this flag; set `tlsLb.cors.enabled` if you want an origin policy of your own, and it then governs every location. ## Install & admission ordering The install sequence is load-bearing, and the operator — not a Helm hook — closes the loop: 1. Helm creates the `MutatingWebhookConfiguration` along with the rest of the release. Its `caBundle` is empty at this point, so the API server cannot yet reach the webhook. 2. The operator starts, mints a long-lived webhook CA and a short-lived serving leaf, and patches the `caBundle` onto the webhook configuration. The patch is idempotent, and an in-process rotator re-mints the leaf under the stable CA so the bundle never needs re-patching. 3. Once the operator's cache is up it runs a one-shot **reinject sweep**: any pod annotated `confidential.ai/cw` that was admitted during the gap — and so never got its `c8s-cert` sidecar — is deleted so its controller recreates it through the webhook. Pods with no controller owner are logged and left running, because deleting them would destroy the workload with no recreate path. 4. On uninstall, where the chart installed the host NRI plugin, a pre-delete hook runs a privileged DaemonSet that removes it from every node's containerd configuration before the release goes away. Next: the [threat model](/docs/c8s/concepts/threat-model) makes the trust assumptions explicit and lists what is — and isn't — enforced today. --- # Encrypted volumes An **encrypted volume** is data too large to be a secret. It sits as ciphertext on storage the untrusted host reads and writes freely, and it decrypts only inside a Trusted Execution Environment (TEE), only for a workload the [allowlist](/docs/c8s/concepts/image-allowlist) names. Model weights are the case it is built for. This page covers the artifact, what happens at mount time, and what the design does and does not defend against. To build one, see [Create an encrypted volume](/docs/c8s/how-to/create-an-encrypted-volume). ## How a volume differs from a secret Every other value **c8s** protects is RAM-resident and dies with the pod: a leaf private key, a released [application secret](/docs/c8s/concepts/application-secrets), a session key. A volume does not. Its ciphertext lives on a block device the host attaches, and the host keeps that device — and any copy of it — for as long as it likes. The operational consequence: **a leaked volume key is retroactive and permanent.** A leaked session key forges future connections; a leaked volume key decrypts a copy the adversary already has, including copies taken months ago. Handle volume keys — and the escrow files that hold them — on that basis. ## The artifact `c8s volume create` packages a directory into a single image file, in three layers: | Layer | What it is | | --- | --- | | filesystem | **erofs** — read-only by construction | | integrity | **dm-verity** — SHA-256 hash tree, 4096-byte blocks, appended to the filesystem | | confidentiality | **plain dm-crypt** — `aes-xts-plain64`, 512-bit key, 512-byte sectors | ```text ┌───────────────────────────────────────────────────────────────┐ │ workload container read-only files at /run/c8s/volumes/… │ ├───────────────────────────────────────────────────────────────┤ │ erofs read-only filesystem │ ├───────────────────────────────────────────────────────────────┤ │ dm-verity hash tree; root hash comes from the blob │ ├───────────────────────────────────────────────────────────────┤ │ dm-crypt aes-xts-plain64, 512-bit key │ ╞═══════════════════════════════════════════════════════════════╡ │ block device ciphertext, attached by the host │ └───────────────────────────────────────────────────────────────┘ ``` *The host sees only the bottom layer, and it is ciphertext it can read, copy, and keep.* **There is no LUKS header.** Nothing on the device is parsed as metadata; every parameter needed to open it comes from the key blob. There is also no keyslot, so changing a volume's key means building a new volume, not rekeying this one. **The hash tree is inside the encryption.** The host cannot fingerprint a volume's contents from the tree, and the root hash commits to the plaintext rather than to one encryption of it. Sector size is fixed at 512 bytes and the verity block size at 4096. Neither is configurable. ### The key blob The value stored at the secret path. It holds everything needed to open the volume and nothing taken from anywhere else: ```json { "type": "c8s.volume/v1", "key": "", "verity": { "root_hash": "", "salt": "", "data_blocks": 26214400, "hash_offset": 107374182400 } } ``` `key` is the XTS key — two AES-256 keys, matching dm-crypt's `--key-size 512`. The hash algorithm is not a field: it is fixed at SHA-256. `hash_offset` must equal `data_blocks × 4096`, and a document carrying any field not listed above is rejected rather than parsed with the extra dropped. The verity root hash rides **in the blob**, not in a pod annotation and not in the allowlist entry. It is the integrity anchor, and it only ever travels over the attested channel. A key blob is stored as an ordinary secret value at an ordinary secret path, and it is released to a pod by exactly the machinery described in [Application secrets](/docs/c8s/concepts/application-secrets) — RA-TLS to the [CDS](/docs/c8s/concepts/trust-root), a single-use challenge, an inventory-signed sandbox token, and a whole-container-set match against one allowlist entry. ## What happens at mount time ```text ╔═ TEE BOUNDARY · node-as-CVM ══════════════════════════════════════════╗ ║ ║ ║ ┌─ pod ──────────────────────────────────────────────────────────┐ ║ ║ │ c8s-volume sidecar workload container │ ║ ║ └──────┬──────────────┬────────────────────────────▲─────────────┘ ║ ║ │ (1) RA-TLS │ (2) unix socket │ ║ ║ │ GET blob │ POST {name, blob} │ (4) read-only ║ ║ ▼ ▼ │ mount ║ ║ ┌────────────┐ ┌─────────────────────────────┐ │ ║ ║ │ CDS │ │ volumed (node DaemonSet) │────┘ ║ ║ │secret store│ │ (3) dm-crypt + dm-verity │ ║ ║ └────────────┘ └──────────────┬──────────────┘ ║ ╚══════════════════════════════════│════════════════════════════════════╝ ▼ ┌────────────────────────────────┐ │ block device, serial │ ciphertext at rest; │ c8s-vol- │ the host keeps a copy └────────────────────────────────┘ ``` *Only the block device sits outside the boundary, and only ciphertext ever reaches it.* What decides whether a mount happens, in order: 1. **CDS releases the blob to the pod's sandbox** — verified mesh leaf, single-use challenge, inventory-signed sandbox token, whole-container-set match against one workload entry, and a grant covering the path. 2. **`volumed` mounts into the calling pod's directory and no other.** The pod comes from the caller's cgroup via kernel peer credentials; the request body carries no field naming it, and the mount target is built from the resolved pod UID. 3. **The device opens only if the key is right and the verity root hash matches.** A request naming a volume already open under that pod must present the same key and root hash; otherwise it is refused. Without that, the volume *name* — a label in a host-written annotation — would be the credential. ### Two timing rules **The volume appears after the workload starts.** Release is gated on the whole container set having been admitted, so `get-volume` is refused until every main container is running. It retries — 60 attempts, 5 seconds apart, by default — and the mount lands shortly after startup. A consumer must **wait for the directory to fill** rather than read it at `main()`. **The key must already be in the store.** Unlike an application secret, where the first pod to ask may define the value, `get-volume` only ever reads. A pod scheduled before `c8s volume create` has run retries and then fails. ## volumed, the node agent `volumed` is the privileged node DaemonSet that opens devices and serves the socket the sidecar posts to. It is listed with the rest of the platform in [Components](/docs/c8s/concepts/architecture#the-components). **`volumed` is off by default.** `volumed.enabled` is `false` in the chart, and nothing about encrypted volumes works without it. Turn it on at install with a values file (`c8s install -f values.yaml`). `c8s install` resolves and pins the image digest for every enabled component and derives it into the allowlist floor, so the daemon's own image is admitted. ```yaml volumed: enabled: true maxMounts: 64 # live volumes per node; each costs two dm devices and a mount reapInterval: 15s # how often teardown checks which pods have gone nodeSelector: {} # confine it to the nodes that carry volume devices ``` Its image is `ghcr.io/confidential-dot-ai/volumed` — debian-slim rather than distroless, because it needs `cryptsetup` and `veritysetup`. It runs **privileged**, with `hostPID` and a bidirectional bind of the kubelet directory. That is inherent to opening a device and mounting into another pod's directory, and it makes `volumed` a host-side operator component sitting outside the guest TEE boundary. It reaches no API server; everything it touches is node-local. Teardown follows the pod's **cgroup**, not its kubelet directory — kubelet cannot remove that directory while a volume is mounted under it. `c8s volumed` and `c8s get-volume` are **Linux-only** subcommands and are absent from a macOS build of the CLI. `c8s volume create` — the only one an operator runs by hand — builds everywhere. ### Volumes require Node-as-CVM The webhook **rejects** `confidential.ai/c8s-volumes` at admission when the operator has no `--workload-claims-host-dir` — that is, under Pod-as-CVM, or with `nri-image-policy` disabled. The fetcher hands the key to a node agent over the inventory's socket directory and the agent mounts into the pod's kubelet directory; neither exists inside a per-pod guest. The pod is refused rather than left waiting on a mount that can never land. See [Kata containers](/docs/c8s/concepts/kata-runtime#reserved-containers-and-volumes). ## Possession of the blob is the authorization `volumed` does not repeat the CDS release decision. It resolves who is calling only to decide **where** to mount, and checks nothing about what that caller is entitled to. Any pod on the node that presents a well-formed blob has that volume opened into its own directory. This rests on **Node-as-CVM being single-tenant**: every pod on the node belongs to the same tenant, so a blob one of them can obtain is one they are all entitled to. Under Pod-as-CVM, volumes are refused at admission, so the case does not arise there. The blob still only comes from CDS, and only to a pod whose containers match an allowlist entry carrying the grant. But a node shared between tenants, or a Pod-as-CVM path for volumes, would need a daemon-side entitlement check that does not exist today. ## What this defends | Threat | Outcome | | --- | --- | | Host reads the volume at rest | **prevented** — AES-XTS; the key never leaves the TEE | | Host tampers with the ciphertext | **detected** — dm-verity fails the affected read | | Host rolls the volume back | **detected** — the root hash covers the whole plaintext | | Host swaps in a different device | **fails closed** — wrong key, or wrong root hash | | A pod outside the grant reads it | **refused** — no grant, no key | | An allowlisted but different workload reads it | **refused** — whole-entry match | Tamper detection is **lazy**. `veritysetup open` checks the top of the tree; a modified data block surfaces as an I/O error when that block is read, not at open time. ### What it does not - **Any pod on the node can open a volume whose blob it holds.** `volumed` authorizes on possession, not entitlement — see [Possession of the blob is the authorization](#possession-of-the-blob-is-the-authorization). - **Anyone with pod-create or exec RBAC in the workload's namespace can read a mounted volume.** Under `--cvm-mode=node` the control plane runs inside the node CVM, so this is not a capability the host has — but it is a Kubernetes RBAC boundary, not an attested one. - **Volume integrity is rooted in the operator keys CDS pins, and CDS's arguments are host-supplied.** A host that restarts CDS under its own operator key can write a matching grant and blob. This is detection, not prevention: the detection is [`c8s cds verify --operator-keys`](/docs/c8s/how-to/verify-a-deployment#check-the-operator-keys-cds-pins), and running it continuously is a precondition for trusting a volume. - **Access patterns are visible.** Which sectors are read, and when, leaks structure. - **Availability.** The host can withhold, corrupt, or destroy the device at any time. - **Whatever the workload does with the plaintext** once it has it. See [Limitations](/docs/c8s/concepts/limitations) for the platform-wide gap list. --- # The image allowlist The **allowlist** decides what may run. Nothing on it, nothing starts. It has two layers: a **floor** of image digests admitted by digest alone, and named **workload entries** that additionally pin the command line each image may run with. Every decision keys on the **image digest**, never the image reference. The reference a pod presents (`docker.io/vllm/vllm-openai:v0.6.3`) is chosen by the untrusted host; the digest is bound to the bytes that execute. References in the allowlist are labels for humans. The allowlist is **owned and served by the [CDS](/docs/c8s/concepts/trust-root)**, enforced at container creation by the `nri-image-policy` plugin (node-as-CVM) and the in-guest `policy-monitor` (pod-as-CVM), and managed with the [`c8s allowlist` CLI](/docs/c8s/how-to/manage-the-allowlist) against an operator-authorized API. This page is the model; the CLI page is the task. The wire surface is in the [CDS HTTP API](/docs/c8s/reference/cds-api#allowlist). ## Data model ```json { "schema": "c8s.allowlist/v1", "digests": { "sha256:": "ghcr.io/confidential-dot-ai/cds", "sha256:": "ghcr.io/confidential-dot-ai/get-cert" }, "workloads": { "vllm-llama": { "label": "docker.io/vllm/vllm-openai:v0.6.3", "initContainers": [], "containers": [ { "digest": "sha256:", "image": "docker.io/vllm/vllm-openai:v0.6.3", "command": { "policy": "exact", "argv": ["python3"] }, "args": { "policy": "exact", "argv": ["-m", "vllm.entrypoints.openai.api_server"] } } ], "secrets": { "policy": "allow", "read": ["/tenant-a/**"] } } } } ``` - **`schema`** must be `c8s.allowlist/v1`. It is the first field of the canonical form, so a foreign or malformed document fails loudly instead of parsing as an empty (deny-everything) allowlist. - **`digests`** is the floor: digest → image reference. A floor image runs **by digest alone**, whatever its command line. The platform's own component images and the measured guest seed live here, because their argv is per-pod and must not be argv-policed. - **`workloads`** maps an operator-chosen name to one entry. The name must match `[A-Za-z0-9][A-Za-z0-9._-]*` — it is a URL path segment. Each entry lists `initContainers` and `containers`; each container binds a `digest` to its `command` and `args` policy. The entry `label` and each container's `image` are informational. - **`secrets`** is an optional grant carried by the **whole entry**, not by a container: the secret-store paths the workload it names may read and write. `policy` is `allow` or `deny` — there is no `any`. Paths are absolute and clean, and the only wildcard is a trailing `/**`, matching strictly *beneath* its base, so `/tenant-a/**` does not grant `/tenant-a` itself. A grant carrying `write` paths must carry at least one `read` path. An entry with no grant releases nothing, and a `deny` grant normalizes away entirely, so it never reaches the canonical document. See [Secrets](/docs/c8s/concepts/application-secrets). Digests are validated as `sha256:<64 hex chars>` (the OCI image-digest format). The store is a SQLite database (`--allowlist-db`) — a floor table, a table of canonical workload entries, and a digest index over them. Both layers share one **version counter**, bumped on every mutation and served as the ETag. The serialization is canonical: fixed field order, sorted map keys, sorted container lists. Any holder of an equivalent document reproduces the same bytes, so what `c8s allowlist export` writes round-trips as an `upload` or as CDS's `--allowlist-seed`. ### Process policy: command and args A digest already pins the image's baked `ENTRYPOINT`/`CMD` — they are in the OCI config the digest covers. A workload policy constrains what a pod may *run* for those bytes. Without it, an image with an overridable entrypoint can be pointed at an arbitrary command — credential extraction, a reverse shell — while still presenting an allowlisted digest. The two fields mirror the Kubernetes container fields: `command` overrides the image `ENTRYPOINT`, `args` overrides `CMD`. The enforcers do not see the override as an override — they see the container's **effective argv** (the OCI `process.args`, already merged from the image config and any pod-spec override) and match policy against that: - **`command`** is matched as an exact **prefix** of the argv. It may be several tokens (`/bin/sh -c`, `/docker-entrypoint.sh nginx`, `python3`). - **`args`** governs the **remainder** of the argv after that prefix. | `policy` | `command` (a prefix) | `args` (the remainder) | | --- | --- | --- | | `exact` | argv must **start with** its `argv` | the remainder must **equal** its `argv` | | `any` | no prefix constraint | the remainder is unconstrained | | `deny` | the whole argv must be empty | there must be **no** remainder | `exact` requires a non-empty `argv`; `any` and `deny` take none. Every combination is well-defined: `command exact` + `args any` pins the executable and lets flags vary, `exact` + `exact` pins the whole argv, `args deny` means "no arguments beyond the command". An absent policy normalizes to `deny`, so a minimally specified container is maximally restrictive. That also means `command: deny` requires an empty argv and can therefore never start — write `command: any` if you mean "any argv". `c8s allowlist lint` flags it. ### A digest may run several ways One digest can appear under several containers, within an entry or across entries. At the per-container gate, admission is the **union**: the container runs if its effective argv satisfies *some* listing container's policy. This is deliberate — a shared base image (busybox, a distroless runtime) is legitimately invoked with different command lines by different workloads. Two consequences to know, both of which `lint` reports: - **One entry widening a shared digest to `any` becomes that digest's effective container-level policy everywhere**, because the host chooses which pod pairs a digest with which argv. - **A floor digest short-circuits every workload policy written for it.** The floor admits by digest alone, so the argv policy is silently not enforced. Remove it from the floor to make the policy bite. ## Authorizing mutations The allowlist is the source of truth for what may run, so writes are authorized by an **operator key**: an EC keypair whose **public** half you pin into the CDS at install time (`c8s install --operator-keys` → `cds.operatorKeys`) and whose **private** half never leaves your side. There is no server-side session and no static bearer secret to hand out — for every write, the [`c8s allowlist` CLI](/docs/c8s/how-to/manage-the-allowlist) signs a fresh, single-purpose token locally with the operator private key. ```text ┌─────────────────────────────┐ ╔═════════════════════════════╗ │ Operator │ ║ CDS ║ │ holds the EC private key │ ║ pins operator public keys ║ │ (--operator-key) │ ║ (cds.operatorKeys) ║ └──────────────┬──────────────┘ ╚══════════════╤══════════════╝ │ │ │ attested dial · verify the endpoint │ │ against --measurements │ │────────────────────────────────────────► │ │ │ mint token: pbh = SHA-256(body) · │ │ htm/htu = method/path · 60s TTL │ │ │ │ POST /allowlist/digests │ │ {digest, image} + Bearer token │ │────────────────────────────────────────► │ │ │ ┌─────────────┴─────────────┐ │ │ signature ∈ pinned keys · │ │ │ exp−iat ≤ 5m · htm/htu │ │ │ match request · pbh = │ │ │ SHA-256(received body) │ │ └─────────────┬─────────────┘ │ │ │ 204 · version bumped │ ◄────────────────────────────────────────│ │ │ ▼ ▼ ``` *A write is bound to the operator key, the HTTP method and path, and the exact body bytes — a captured token cannot be replayed against a different change.* The token is a **JSON Web Token (JWT)** signed with the operator's ECDSA key (ES256, ES384, or ES512, matching the key's curve), minted fresh by the CLI for each write with a 60-second lifetime. Three claims bind it to that one write: - `pbh` — the SHA-256 hash of the exact request body, - `htm` — the HTTP method, - `htu` — the URL path. The CDS accepts a mutation only when all of it checks out: the signature verifies against a pinned key; the token carries issue and expiry times no more than **five minutes** apart (a server-side cap, so no client tooling can mint a long-lived token); the method and path match the request it is actually handling; and the body hash matches a re-hash of the body the CDS actually received, compared in constant time. Verification happens at the application layer — the listener stays plain RA-TLS, so mesh clients are unaffected. Writes **fail closed**: with no pinned keys, every mutation is rejected while reads keep serving. The same keys authorize floor and workload writes alike. Know the boundaries of this design before relying on it: - **A pinned operator key is the image-integrity control.** Anyone holding the private key can rewrite what may run. Keep it in a vault, HSM, or hardware token, and supply it to the CLI per invocation. - **Revocation is coarse.** Operator keys are long-lived, and revoking one means removing its public key from `cds.operatorKeys` and re-installing — there is no CRL/OCSP-style revocation to lean on. - **A captured token is briefly replayable — against the same change.** The body/method/path binding stops cross-payload replay, but the token carries no cluster (`aud`) binding: two clusters pinning the *same* operator key would accept each other's tokens within the validity window. Pin **distinct keys per cluster**. - **The pinned-key list is host-supplied config, detected rather than prevented.** The CDS serving certificate commits its key and launch measurement — not the operator-key set, and not the applied seed. A control plane can restart the CDS with a different bundle. What closes the loop is [`c8s cds verify --operator-keys`](/docs/c8s/how-to/verify-a-deployment#check-the-operator-keys-cds-pins), which fetches the served set over the attested serving certificate and **fails closed** on a mismatch. It protects only the verifier that runs it, so run it continuously in CI rather than once at bootstrap. The key set *is* covered by attestation on the CDS `/handoff` and `/attest-key` paths, so a replacement replica cannot adopt the mesh CA under a different write policy. **Planned, not yet shipped.** A CA issuing short-lived operator certificates — giving delegated issuance and real revocation instead of editing a pinned-key list. See [Limitations](/docs/c8s/concepts/limitations). ## Seeding and bootstrap - **At CDS startup**, `--allowlist-seed ` loads a full document (floor and workloads) into the store before the server serves its first request. Seeding is additive: it inserts only what is missing and leaves existing entries untouched. Any seed error halts startup (fail-closed) — the CDS must not serve a partial allowlist. The chart renders the seed from `nriImagePolicy.bootstrapAllowlist` (its `digests` and `workloads` maps). With the default `--resolve-digests=true`, `c8s install` resolves each c8s component image to its digest and derives it into the floor, so the platform's own images are covered from first boot. - **Under pod-as-CVM**, the guest image bakes `/etc/c8s/bootstrap-allowlist.json` — a flat `sha256_digests` list that is part of the guest's launch measurement. `policy-monitor` enforces against it from `t=0`, before the first container starts and with no network at all — so there is no boot-path fetch a host could stall to open a window. The in-guest CDS refresh that would layer operator writes on top of that baked seed is **gated on a pinned CDS measurement** (`C8S_CDS_MEASUREMENTS` alongside `C8S_CDS_URL`): with no pin, `policy-monitor` disables the refresh rather than accept any attested CDS, because the host can boot its own CVM from the same guest image. No shipping path delivers that pin today, so on a default Kata install the refresh is **off** — a `c8s allowlist` write reaches the CDS and the host-side enforcer, but not running guests, which keep enforcing the measured seed alone. Budget a guest-image rebuild for pod-as-CVM allowlist changes. See [Limitations](/docs/c8s/concepts/limitations). ## Enforcement Three independent points enforce, at different strengths: - **Host NRI plugin** (`nri-image-policy`), on the `CreateContainer` hook, per container. It resolves the image digest and checks it plus the effective argv against the allowlist index. It is fail-closed before the allowlist first loads — with no list available, nothing runs. `nriImagePolicy.policy.mode` is `fail-closed` by default; `audit` logs the would-be denial and admits, which is a bring-up setting, not a production one. For kata pods the plugin sits on the untrusted side of the TEE boundary, so it is defense-in-depth there and the primary gate for base-mode pods. - **In-guest `policy-monitor`** (pod-as-CVM), watching each new container bundle's `config.json` for the digest and `process.args`. This is the load-bearing gate for confidential pods: the host is untrusted, guest-pull is forced, and a violation kills the container's whole cgroup as a unit through the kernel's `cgroup.kill` interface — never a PID picked out of `cgroup.procs`, which the kernel does not order and which a recycled PID can turn into the wrong kill. A kill that cannot be delivered is reported as a failure, never as a success. Because the monitor is baked into the launch measurement, the host cannot disable or bypass it. - **CDS at certificate issuance.** Before signing a leaf for a pod, the CDS asks that pod's own admission inventory which images its sandbox is running — a report of `(digest, argv)` pairs for everything ever admitted there. Every reported digest must be allowlisted, in the floor or in some workload entry. This is a membership check: issuance lands mid-lifecycle, when the running set is a strict subset of the declared one, so requiring a whole entry would refuse ordinary pods. See [CDS](/docs/c8s/concepts/trust-root). The honest guarantee is **per-container digest plus argv, everywhere — and no combination gating**. Nothing today enforces "only this set of images may run *together*": NRI and `policy-monitor` see containers one at a time and cannot detect a *missing* one, and the CDS sees the whole reported set only at issuance, where it can check membership but not composition. Env, mounts, capabilities, and the rest of the pod spec are likewise outside the allowlist. See [Limitations](/docs/c8s/concepts/limitations). The two layers also refresh differently. The **floor is additive** — a digest, once served, is never dropped by a consumer, so a CDS outage or a stale read degrades to "the same set or larger", never to "open". The **workload overlay swaps wholesale**, gated on the version counter: a consumer applies a pulled overlay only if its version is greater than the last one it applied. Workload policy can *tighten*, and a plain additive merge would let a host that withholds an update keep a laxer policy live forever. The high-water mark is process-local, so this rejects rollback only within a consumer's lifetime; after a restart the first version seen is trusted. ## See also - [Manage the allowlist](/docs/c8s/how-to/manage-the-allowlist) — the `c8s allowlist` CLI: read it, change the floor, edit workload policy, grant secret paths. - [CDS HTTP API → Allowlist](/docs/c8s/reference/cds-api#allowlist) — the wire surface every one of those commands drives. - [CLI reference → `c8s allowlist`](/docs/c8s/reference/cli#c8s-allowlist) — every flag. - [Application secrets](/docs/c8s/concepts/application-secrets) — what an entry's `secrets` grant releases. --- # Kata Containers When you install **c8s** with `--cvm-mode=pod`, every workload pod becomes its own **confidential VM (CVM)** — **AMD SEV-SNP** by default, **Intel TDX** with `--hardware-platform=tdx`. This is the **Pod-as-CVM** shape: Kata Containers provides the per-pod micro-VM, and c8s provides the attestation, identity, and policy that make it confidential and verifiable. ## What gets installed `c8s install --cvm-mode=pod` deploys the Kata runtime stack and registers four RuntimeClasses — the two non-confidential classes plus the declared platform's confidential (CPU, GPU) pair: - **`kata-deploy`** (DaemonSet, digest-pinned) installs the Kata runtime and `containerd-shim-kata-v2` on every Linux node. The kata-static payload it copies bundles the TEE-capable QEMU builds and Cloud Hypervisor, the guest kernel, guest images, and OVMF firmware — Kata never uses a distro QEMU. It then restarts containerd (or RKE2) so the runtimes register; running pods survive, because containerd shims outlive the daemon. - **`c8s-kata-deploy-image-puller`** (DaemonSet) `oras pull`s the `kata-guest-base` artifact onto each node and writes a `config.d` drop-in so the confidential shim boots from it. A sibling DaemonSet, `c8s-kata-deploy-image-puller-nvidia`, does the same for the GPU guest. - **RuntimeClasses:** | RuntimeClass | Hypervisor | Confidential? | | --- | --- | --- | | `kata-qemu` | QEMU micro-VM | No — VM isolation from the host only | | `kata-clh` | Cloud Hypervisor | No — VM isolation only | | `kata-qemu-snp` / `kata-qemu-tdx` | QEMU + SEV-SNP / Intel TDX | **Yes** — pod memory is encrypted against the host; schedules only on platform-labeled nodes | | `kata-qemu-snp-nvidia` / `kata-qemu-tdx-nvidia` | as above + VFIO GPU passthrough | **Yes** — a confidential VM with a dedicated NVIDIA GPU | A cluster runs **one** platform's confidential classes: only the pair matching the install's `--hardware-platform` is rendered, and the webhook promotes `confidential.ai/cw` pods to its CPU class — mixed-TEE clusters are not supported. Confidential pods schedule only to nodes carrying the matching label (`confidential.ai/sev-snp=true` or `confidential.ai/tdx=true`); `c8s install --cvm-mode=pod` applies it to every kata-targeted node from `--hardware-platform`, and refuses to proceed if a node still carries the *other* platform's label. The label is a scheduling aid, not a security boundary — attestation is. The rest of this page describes the SEV-SNP path. Installing `--cvm-mode=pod` also **disables** the host-side `ratls-mesh`, `attestation-api`, and `nri-image-policy`: under Kata their function moves *inside* the guest image, where the host cannot tamper with it. The chart refuses to render if any of them is left enabled alongside `kata.enabled`. ## How pods are kept inside the boundary `--cvm-mode=pod` is enforcing — there is no install-Kata-without-enforcing shape, because a workload that can dodge the CVM boundary makes the stack decorative. Two pieces cooperate. **The pod webhook injects a RuntimeClass** into every in-scope workload pod that does not already request one: | Pod | RuntimeClass injected | | --- | --- | | plain workload | `kata-qemu` (not confidential) | | annotated `confidential.ai/cw` | platform CPU class (`kata-qemu-snp` / `kata-qemu-tdx`) | | requests an `nvidia.com/*` resource | platform GPU class (`kata-qemu-snp-nvidia` / `kata-qemu-tdx-nvidia`) | | requests a GPU **and** annotated | platform GPU class — GPU implies confidential | **A `ValidatingAdmissionPolicy`** (`c8s-kata-enforcement`, `failurePolicy: Fail`) then rejects any workload pod naming a `runtimeClassName` outside the installed set. It is built-in CEL — no webhook server, no TLS — and needs Kubernetes 1.30+, which is the chart's floor anyway. What enforcement does **not** touch: - **System namespaces** — `kube-system`, `kube-public`, `kube-node-lease`, the c8s release namespace, and anything in `webhook.extraExcluded`. Injection scope and rejection scope read the same list, so they cannot drift apart. - **Pods that set their own `runtimeClassName`** — honored, and still validated as a Kata class. This path also skips `get-cert` injection: a pod pinned to `kata-qemu-snp` without `confidential.ai/cw` runs as a confidential VM with no c8s workload identity. That is the supported bring-your-own-attestation path. To get both, annotate the pod and let the webhook choose the class. - **Already-running pods** — the webhook fires on `CREATE` only. - **Host-namespace pods** — a Kata pod is a VM and cannot join the host's network, PID, or IPC namespace, so a pod setting `hostNetwork`, `hostPID`, or `hostIPC` is exempt from both halves and runs as an ordinary container. **Pair `--cvm-mode=pod` with a PodSecurity floor.** The host-namespace exemption is what makes Kata usable for CNI agents and CSI drivers, but without a cluster-wide PodSecurityAdmission floor of `baseline` or `restricted` on workload namespaces, any user with create-pod RBAC can opt out of the CVM boundary by setting `hostNetwork: true`. The chart labels only its own namespace. Audit non-system namespaces before enabling enforcement — infrastructure that mounts host paths cannot run under Kata and must be excluded via `webhook.extraExcluded`. ### Reserved containers and volumes For a pod annotated `confidential.ai/cw`, the webhook injects the `get-cert` sidecar and a memory-backed `emptyDir` to hold the leaf key, so the private key never leaves TEE memory. The webhook enforces that at admission: - The container names `c8s-cert`, `c8s-cert-wait`, `c8s-secret`, and `c8s-volume` are reserved — a pod may not declare its own container or ephemeral container under them. - If the pod pre-declares the reserved cert volume (`c8s-certs`, or whatever `confidential.ai/c8s-cert-volume` names), it must be an `emptyDir` with `medium: Memory`. A `hostPath`, PVC, or disk-backed `emptyDir` is rejected — otherwise the sidecar would write keys to persistent, host-visible storage. Omitting it is fine; the webhook injects the right one. - A container that declares its own mount of a reserved volume keeps its mount path but the mount is **forced read-only**. - An ephemeral container (`kubectl debug`) that mounts a reserved volume is rejected outright. The reserved set is read from `spec.initContainers`, which is immutable after `CREATE`, so rewriting the annotation on a live pod does not move the guard. Application-secret and encrypted-volume injection annotations are **rejected under Kata**: both fetchers need a node-side socket that does not exist inside a guest. ## Measured direct-kernel boot Kata's confidential path does **not** use IGVM or a UKI. It uses **measured direct-kernel boot with dm-verity**: | Component | What's supplied | | --- | --- | | kernel | a bare `vmlinuz` (a hardened kernel) | | image | `kata-rootfs.img` — an ext4 rootfs plus a dm-verity hash tree | | verity params | `root_hash=…,salt=…,data_blocks=…` on the kernel cmdline | The dm-verity **root hash rides on the kernel cmdline**, and Kata folds the resulting hash into the launch measurement — SNP kernel-hashes, or TDX RTMR[1]. So the SNP launch digest commits to OVMF + `vmlinuz` + the exact rootfs. On SNP the puller also pins `default_vcpus` and `default_maxvcpus` to `1`, because the boot-time VMSA count is the one genuinely per-VM input to the digest; TDX carries no such pin, since the register c8s verifies (MRTD) does not measure vCPU init. The digest is not published with the artifact. Predict it with `sev-snp-measure` from the exact OVMF, kernel, cmdline, and VM shape, then supply it to the policies that consume it — `cds.measurements` and `ratlsMesh.measurements` in a values file, and client-side `--measurements` on verification. All of those default to empty, which means "accept any attested peer". `kata.guestImage.tag` is **not** a measurement pin: it selects which artifact the puller fetches and carries no cryptographic guarantee, and the puller has no `digest:` field yet. Pin a specific `` tag rather than `main` or `latest`, and pin the predicted launch digest separately. `c8s install --cvm-mode=pod` refuses `--measurements` — that flag pins a *node* CVM's digest, which is a different value from the per-pod guest's. Any change to the guest rootfs moves the digest. The removal of the in-guest registry-auth mechanism (below) changed those bytes, so reference values predicted against an older `kata-guest-base` no longer match and must be re-predicted. ## The sealed guest image The **`kata-guest-base`** image is the dm-verity-sealed guest rootfs. Because it is part of the launch measurement, everything baked into it is transitively attested: | Baked-in | Role | | --- | --- | | `attestation-service` | localhost-only attester at `127.0.0.1:8400` (the `attestation-api` binary, staged under this role name) | | `ratls-mesh` | in-guest mesh proxy (`ratls-mesh in-guest`) | | `policy-monitor` | watches container creation and SIGKILLs non-allowlisted images | | `rtmr3-measurer` | on TDX, extends RTMR[3] with each deployed workload's image digest | | `/etc/c8s/bootstrap-allowlist.json` | the image-digest allowlist seed (part of the measurement) | | `/etc/kata-opa/default-policy.rego` | OPA policy denying host `exec`/`log` RPCs and `SetPolicyRequest` | A peer that verifies an RA-TLS handshake therefore transitively verifies that the attestation service, mesh proxy, and policy monitor are the genuine, sealed versions — the host cannot swap them out. The baked allowlist is a **seed, not the whole story**: `policy-monitor` loads it at boot so the guest enforces from t=0 with no network, then polls CDS's `/allowlist` over RA-TLS and merges what it serves on top. The merge only ever *grows* the set, so an unreachable or compromised CDS degrades to "stale but no smaller" — never to "open". Adding an image is therefore an [allowlist](/docs/c8s/concepts/image-allowlist) write, not a guest rebuild. In debug builds (`--cvm-mode=pod --debug`), the guest policy allows host log/exec streams so `kubectl logs` and `kubectl exec` work — but container I/O becomes readable by the untrusted host, and the launch measurement differs from the locked image, so attestation pinned to the locked value rejects a debug guest. Development only. ## Three layers of attestation ```text LAYER 0 LAYER 1 LAYER 2 ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ Guest image │ │ Container │ │ Workload │ │ │──────► │ image │──────► │ identity │ │ SNP launch │ │ │ │ │ │ digest │ │ OCI digest · │ │ RA-TLS │ │ │ │ allowlist │ │ certificate │ └───────────────┘ └───────────────┘ └───────────────┘ ``` *Each layer is verified independently; only the first is baked into the hardware measurement.* 1. **Guest image** — the SNP launch digest over OVMF + `vmlinuz` + cmdline (embedding the dm-verity root hash). The operator predicts it and pins it. 2. **Container image** — the OCI image digest, checked against the [allowlist](/docs/c8s/concepts/image-allowlist). The pod webhook injects `get-cert`, which fetches a leaf cert from the CDS — the CDS verifies the container measurement and signs in one step. 3. **Workload identity** — an RA-TLS cert carrying attestation evidence; peers verify it on the mTLS handshake. ## The guest-pull model With `shared_fs = "none"` and `experimental_force_guest_pull = true`, the kata-agent's confidential-data-hub **pulls the OCI image inside the guest over virtio-net** — not from a host bind-mount. The host never sees the unpacked workload rootfs, only encrypted VM memory. The in-guest pull is **anonymous**, and there is no supported way to hand the guest a registry credential. The mechanism that used to bake one into the rootfs — `ghcr-auth.json`, surfaced as `kata.guestImage.registryAuth` and `kata.gpu.guestImage.registryAuth` — was removed, because it put pull credentials inside the dm-verity-measured root, where rotating them meant rebuilding and re-attesting the image. Both values are gone from the chart. What that means in practice: - **Workload images must be pullable without authentication from inside the guest.** Private in-guest workload pulls are not supported. A private image fails during pod start, inside the VM. - **Host-side pull credentials are still required.** The host's CRI does its own image-exists pull, so `c8s install --image-pull-secret ` (a `kubernetes.io/dockerconfigjson` Secret in the release namespace) still applies to every component image. - **`kata.guestImage.pullerAuthSecret` is a different credential.** It authenticates the host-side `oras pull` of the `kata-guest-base` artifact, which reads `/root/.docker/config.json` and ignores kubelet pull secrets. It defaults to the chart-wide `imagePullSecret`, is not in the TCB or the launch measurement, and rotating it is a Secret update plus a DaemonSet restart. Set it only when the artifact needs a credential different from the c8s images. - **The host still observes the transport.** It brokers the guest's outbound network, so it sees which image reference and layers are fetched. That is a metadata leak, not a content-confidentiality break — the bytes are public. - **The image unpacks into guest RAM.** Guest-pull writes to a tmpfs, so a workload image has to fit in the pod's guest memory unless the node is provisioned to attach an encrypted scratch disk, which is a host-provisioning step and not wired by the chart. Also raise kubelet's `runtime-request-timeout` on Kata clusters. The default is 2 minutes and it caps pod creation; a multi-GB model image guest-pulled inside the VM will hit that wall with the cause hidden. On RKE2, set `kubelet-arg: runtime-request-timeout=20m` in `/etc/rancher/rke2/config.yaml`. ## Constraints - **x86-64, SEV-SNP or Intel TDX.** The confidential *pod* path needs an AMD SEV-SNP host (Genoa / Milan / Turin class) or an Intel TDX host, declared at install time via `--hardware-platform`. `kata-deploy` installs the runtime; it does **not** enable the TEE — that is a kernel cmdline, firmware, and BIOS matter. Node kernels must be new enough for the TEE: 6.11 or newer for SEV-SNP, 6.16 or newer for TDX. On a host with neither TEE you can run `kata-qemu` (isolation, not confidentiality) but no confidential class. The chart renders no AArch64 shims, so Kata installs nothing on ARM nodes. - **Kata needs `/dev/kvm`, `/dev/vhost-vsock`, and `/dev/vhost-net`.** These auto-load on standard systemd distros; if pods fail with `open /dev/vhost-vsock: no such device`, run `modprobe vhost_vsock vhost_net` and persist it via `/etc/modules-load.d/`. - **A webhook outage blocks all workload pod creation.** The pod webhook is `failurePolicy: Fail`, and the chart refuses to render `kata.enabled=true` with anything else. With enforcement on, that widens an operator outage from "no `get-cert` injection" to "no workload pods". - **Installing Kata restarts containerd / RKE2 on every node**, and `kata-deploy` takes 1–2 minutes per node. Pods created in that window are mutated to a Kata RuntimeClass and stay `Pending` until the runtime is in place — delayed, not lost. - **Confidential GPU is passthrough-based and comes with gaps.** GPU pods run as confidential VMs with the NVIDIA GPU passed through over VFIO; there is no separate flag, the GPU stack ships with every `--cvm-mode=pod` install. The host setup (vfio-pci binding, NVIDIA Confidential Computing (CC mode), BAR resize) is provisioning's job, **not** the NVIDIA GPU Operator's — the operator assumes host-visible GPUs with a host driver and must not be installed on these clusters. A locked guest refuses to boot on a non-CC GPU, but the GPU's own attestation (SPDM) is not yet surfaced to the relying party, and SEV-SNP GPU pods inherit the single-vCPU pin. See [Limitations](/docs/c8s/concepts/limitations). - **Per-pod, not per-node.** Pod-as-CVM attests each pod; it does not by itself attest the node. If you need a single confidential node instead, see [Pod-as-CVM vs Node-as-CVM](/docs/c8s/concepts/trust-boundaries). - **Not available on Azure.** Per-pod CVMs need nested virtualization, which Azure does not expose; use Node-as-CVM there — see [Provisioning on Azure](/docs/c8s/how-to/provision-azure-aks#why-pod-as-cvm-is-not-available-on-azure). --- # Limitations **c8s** is built around a strong threat model, but not every gap is closed yet. This page is the honest list of what the current milestone does **not** do, so you can plan around it. Several items are deliberate non-goals for now; others are tracked work. ## Trust and enforcement gaps - **Measurements are not pinned by default.** The chart ships `cds.measurements` and `ratlsMesh.measurements` empty, which means "accept any attested peer". Both the CDS and `ratls-mesh` log warnings when their allowlists are empty, and `ratls-mesh` publishes `ratls_mesh_measurement_pinning=0`, but the accept-any policy still stands until you set the values. On a node-shaped install (`--cvm-mode=node`, `gke`, `aks`), `c8s install --measurements ` fans one digest into both; under `--cvm-mode=pod` the flag is rejected, because each Kata guest is measured separately — set the values directly. See the [threat model](/docs/c8s/concepts/threat-model#measurement-pinning-is-mandatory-for-production). - **Mesh peer verification doesn't pin the peer measurement.** A mesh peer is accepted if its certificate chains to the mesh CA. Issued leaves do now carry the evidence the **CDS** (Certificate Distribution Service) verified at issuance, and the **RA-TLS** (Remote-Attestation TLS) verify policy has a `RequireCAEvidence` mode that re-checks it — measurement included — on every connection, but no shipped profile enables it. - **A workload's sandbox identity is CA-vouched, not hardware-bound.** `c8s verify --sandbox-id` checks the CRI sandbox ID stamped into a leaf, but that ID is signed in by the mesh CA on the word of an on-node admission inventory; it is not folded into the Trusted Execution Environment (TEE) report. Any process that can bind the node's privileged inventory port — including a privileged node DaemonSet — can vouch for a sandbox it does not run, and a compromised mesh CA can mint any ID. Treat it as CA-vouched identity, not attested identity. - **The allowlist gates each container's digest and command line — not the rest of the pod spec.** Floor entries admit an image by digest alone. Workload entries additionally pin each container's `command` prefix and `args` remainder against the effective argv, and all three enforcement points — `nri-image-policy`, in-guest `policy-monitor`, and the CDS at certificate issuance — apply it. Env, mounts, capabilities, and the remaining pod-spec fields are not checked. - **Nothing enforces which images run *together*.** Every running image must be allowlisted, but no gate requires the set that runs in one pod to match a single workload entry. Container-start gates see one container at a time and cannot detect a missing one; the CDS sees the whole reported set but only at issuance, mid-lifecycle, so it checks membership rather than composition. The one decision that does require a whole entry to match is [secret release](/docs/c8s/concepts/application-secrets), which lands after every main container is up — but it withholds a secret, it does not stop the wrong set from running. - **Allowlist write authorization is pinned-key, not PKI.** Writes are authorized by long-lived operator public keys pinned in `cds.operatorKeys`. Revocation is coarse — remove the key and re-install, with no CRL/OCSP. Write tokens are bound to body, method, and path with a five-minute server cap, but carry no cluster binding, so clusters pinning the *same* operator key accept each other's captured tokens within that window — pin distinct keys per cluster. The pinned list itself is host-supplied config that the CDS serving certificate does not commit: `c8s cds verify --operator-keys` fetches the served set over the attested serving cert and fails closed against your own bundle, but that protects only the verifier that runs it — in-cluster enforcers run none, so run it continuously, not just at bootstrap. The applied allowlist seed is not covered by any attestation. A CA with short-lived operator certificates is the planned replacement. See [Authorizing mutations](/docs/c8s/concepts/image-allowlist#authorizing-mutations). - **Certificate revocation is not checked.** The `attestation-api` ships with CRL checking off (`require_crl=false`, and the chart exposes no value to turn it on), and the in-process Go verification path for **AMD SEV-SNP** performs no revocation check at all. An adversary who blackholes the AMD CRL endpoint gets a revoked AMD attestation key (VCEK) accepted. - **No per-workload peer allowlists in the mesh.** Any attested mesh peer can talk to any other; there is no per-workload restriction on who may connect to whom. - **No SPIFFE-style URI SAN identities.** Workload identity is carried by the CDS-issued cert's SANs/CN, not a SPIFFE URI scheme. - **Exempt namespaces bypass injection and image policy.** The release namespace, `kube-system`, `kube-public`, and `kube-node-lease` skip webhook injection and the pod-as-CVM RuntimeClass policy, and `kube-system` also skips image policy. Host-namespace pods are exempt from that policy too, so anyone with create-pod RBAC in an exempt namespace opts out with `hostNetwork: true` — and so does anyone in any namespace if you set `hostNamespacePolicy.enabled=false` (the `deny-host-namespaces` validating policy is on by default). RuntimeClass enforcement is a guardrail; the boundary is per-pod attestation. ## Confidential pods under Pod-as-CVM - **`c8s allowlist add` does not reach running Kata guests.** In-guest `policy-monitor` refuses to refresh from the CDS unless `C8S_CDS_MEASUREMENTS` pins the CDS launch digest, and no shipping path delivers that pin: baking it is self-referential (under Kata the CDS runs from the same guest image, so the value would change the measurement it pins), and per-pod cloud-init is host-controlled. The refresh is therefore disabled on every default install and each guest enforces only the seed baked into its measured image — the `cds`, `get-cert`, and `c8s-operator` digests resolved at guest-image build time. Operator additions reach host-side enforcement and the CDS, but not a running guest, so admitting a new workload image inside a confidential pod means rebuilding the guest image (which moves its launch measurement). This is deliberately fail-closed: "any attested TEE" is not good enough for the refresh, because the host can boot its own confidential VM (CVM) from the same guest image and serve an attacker-chosen allowlist. - **Secrets and encrypted volumes are unavailable under pod-as-CVM.** Both injected fetchers redeem their sandbox token over the node's mounted admission-inventory socket — unlike `get-cert`, neither has a guest-loopback path — and the volume agent mounts into the pod's kubelet directory, which does not exist for a Kata guest. The webhook therefore rejects `confidential.ai/c8s-secrets` and `confidential.ai/c8s-volumes` at admission wherever the operator has no node inventory socket (`kata.enabled=true`, or `nri-image-policy` disabled), so the pod is refused outright rather than left blocking on a file or a mount that can never land. See [Secrets](/docs/c8s/concepts/application-secrets) and [Encrypted volumes](/docs/c8s/concepts/encrypted-volumes). - **A small post-start kill window exists.** `policy-monitor` SIGKILLs a non-allowlisted container's cgroup, but the init fork happens single-digit milliseconds before the kill lands. The container has no network and cannot `execve` a user binary in that window. - **Every Kata guest accepts inbound TCP `8443` without mesh mTLS.** The guest image bakes `C8S_MESH_INBOUND_PASSTHROUGH=tcp:8443` so the front doors (the tls-lb nginx, the CDS) can terminate their own attestation-anchored TLS for external clients. Because that value is a single baked default, any workload listening on `8443` in a Kata pod is reachable without a mesh client certificate. Don't serve mesh-trusting endpoints on `8443`. - **Root workloads bypass the in-guest mesh.** The in-guest mesh exempts all UID-0 egress so the attestation service can reach the AMD KDS, so a container running as root egresses in plaintext. Run workloads non-root. - **Workload images must be pullable anonymously.** Under guest-pull the kata-agent fetches image layers *inside* the guest, and there is no in-guest registry-auth path. A private image 401s the in-guest layer fetch; `imagePullSecrets` only covers the host-side manifest resolution. Publish the workload image, or host it on a registry the guest reaches without credentials. - **One vCPU per confidential pod on SEV-SNP.** Both SEV-SNP shims — CPU and GPU — pin `default_vcpus = default_maxvcpus = 1`, because the SEV-SNP launch digest measures one VMSA per boot-time vCPU and a floating count would fragment the reference measurement per pod shape. CPU hotplug cannot raise it at runtime. This is a policy pin, not a hard limit: any fixed count works if you re-predict and re-pin the launch digest. **Intel TDX** installs need no pin. ## Availability and operations - **The CDS is a singleton by default.** The mesh CA key lives only in CDS process memory, so a restart generates a new CA and invalidates existing leaves. Restart continuity via attested **CA handoff** is opt-in (`cds.handoff.enabled=true`) — see [CDS handoff](/docs/c8s/concepts/trust-root#ca-handoff). True active/active is still blocked by per-pod **EAR** (Entity Attestation Result) signing keys; replicas stay fixed at one. - **Operator-added allowlist entries don't survive a CDS restart.** `cds.persistence.enabled` is `false` by default, so the store is an `emptyDir` and a restart drops back to the install seed — workloads pulling those images are denied roughly one poll interval (~30s) later. Set `cds.persistence.enabled=true` for an RWO PVC, or re-run `c8s allowlist add` after a restart. A *planned* handoff roll is the exception: it carries the complete allowlist inside the encrypted snapshot, so freeze allowlist writes for the duration of the rollout. - **Flipping `kata.enabled` on a live cluster is disruptive.** Node-as-CVM and pod-as-CVM move components between host and guest; switching in place is not a seamless toggle. ## Platform constraints - **Pod-as-CVM is unavailable on Azure.** Azure does not expose nested virtualization, so AKS runs node-as-CVM only — all pods on a node share one confidential boundary. See [Provisioning on Azure](/docs/c8s/how-to/provision-azure-aks#why-pod-as-cvm-is-not-available-on-azure). - **One CPU TEE per cluster for confidential pods.** The per-pod runtime (`--cvm-mode=pod`) supports SEV-SNP and TDX, but an install picks exactly one via `--hardware-platform` (rendering `kata-qemu-snp` *or* `kata-qemu-tdx`), and the webhook promotes every confidential pod to that one class — mixed SNP + TDX clusters are not supported. On a host with neither TEE you can still run `kata-qemu` (isolation without confidentiality). - **Confidential GPU ships, with gaps.** Kata clusters run GPU pods as confidential VMs with a Blackwell-class NVIDIA GPU passed through over VFIO in **NVIDIA Confidential Computing (CC mode)** — request an `nvidia.com/*` resource and the webhook injects the confidential GPU RuntimeClass. Current gaps: the GPU's own attestation (SPDM) is not collected in the guest or required at certificate issuance, so no positive GPU evidence reaches the relying party — a non-CC GPU fails closed at guest boot, which is a guard, not a proof. The GPU guest boots kata's GPU kernel with the NVIDIA modules grafted from kata's rootfs, so kernel and driver provenance is the kata release rather than the c8s build (everything grafted is inside the measured verity root). GPU pods take no memory limits, and one pod targets one GPU model. Host GPU provisioning (vfio-pci binding, GPU CC mode, BAR resize) is assumed done before install; the NVIDIA GPU Operator is **not** used on this path. - **Don't run an external service mesh alongside c8s.** c8s installs its own pod-to-pod **RA-TLS mesh** — transparent iptables redirection plus attestation-gated mTLS. A second service mesh (Istio, Linkerd, or any sidecar / mTLS plumbing) double-intercepts the same traffic, collides with the RA-TLS handshake, and injects **un-attested** proxies into the confidential path — breaking the trust model rather than adding to it. Run c8s without one; it already provides the mTLS layer, rooted in hardware attestation. ## Browser and out-of-cluster verification - **Browser verification is SEV-SNP only.** The `c8s cds-attest` sidecar can serve TDX evidence, but the [c8s-verify](/docs/c8s/concepts/verification-model) WASM verifier implements the `snp` and `az-snp` paths only; `tdx` is reserved in the wire protocol and not implemented. - **The WASM verifier enforces fewer checks than the server verifiers.** Its bare-`snp` path omits the VMPL-0 check, debug-policy rejection, minimum-TCB enforcement, VEK validity, and CRL. A browser client would accept a debug-enabled or non-VMPL-0 guest — one whose memory the host can read — if its measurement is allowlisted. - **The over-encrypted channel is not streaming.** The sidecar buffers each sealed request and each upstream response into a single tunnel envelope, so chunked transfer from the upstream does not bypass the buffering. Uploads are bounded by the sidecar's request-record cap, and upstream responses over 32 MiB fail rather than stream. Large transfers need application-level range or chunk APIs. ## Out of scope for this milestone - Pod-spec integrity checking beyond the per-container digest and argv policy. - Active/active CDS high availability. - Multi-tenant isolation between workloads on the same cluster beyond the CVM boundary, and federated multi-cluster control planes. --- # Threat Model **c8s** is designed around a single, demanding assumption: **the infrastructure operator is adversarial.** That includes the cloud provider, the bare-metal operator, and anything that can read raw host memory or inspect a QEMU command line. The trust boundary is the TEE — the node (an AMD SEV-SNP or Intel TDX confidential VM) under node-as-CVM, or each pod's own confidential VM under pod-as-CVM (Kata). The Kubernetes control plane is on the far side of that boundary too. etcd sees only ciphertext and public material for the keys c8s holds in the TEE — the mesh CA, the **EAR** (Entity Attestation Result) issuer, RA-TLS leaf keys — because none of them is ever written to a Kubernetes Secret. Ordinary Kubernetes Secrets are *not* covered: image-pull credentials, the webhook `caBundle`, and any tenant Secret are plaintext to whoever reads etcd. ## What the adversary can and cannot do The left column is the baseline you are coming from: an ordinary Kubernetes cluster on non-confidential hardware. **Node-as-CVM** gives the same protections as the right column, but at node granularity rather than per pod. | | Ordinary Kubernetes | Pod-as-CVM (Kata) | | --- | --- | --- | | Read workload memory | **Yes** (host-trusted) | **No** — the TEE encrypts pod memory | | Tamper with the runtime undetected | n/a | **No** — tampering changes the launch measurement; attestation fails | | See the unpacked workload rootfs | Yes | **No** — images are pulled and unpacked inside the guest | | See which image a pod runs | Yes | Yes — the host brokers the pull, so it observes the reference and layers (metadata, not content) | | Observe inter-node traffic | Mitigated by RA-TLS mesh | Mitigated by RA-TLS mesh | | Read a running container's stdout | Yes | **No** — the locked guest policy denies `ReadStreamRequest` and `ExecProcessRequest`, so `kubectl logs` is empty by design | Under pod-as-CVM, if the host tampers with the Kata runtime, QEMU, or the shim, it changes the pod's launch measurement, and attestation fails. The host can *break* a pod, but it cannot break it *silently* — detection is by attestation, not by trust. ### A co-tenant pod is not the confidentiality adversary under node-as-CVM Node-as-CVM is a **single-tenant** shape. The boundary is the node, the adversary (the cloud or bare-metal operator) sits outside it, and pods inside the node are separated only by the kernel — as they would be on any Kubernetes cluster. What c8s does enforce inside the node is **least privilege between workloads**: a compromised pod must not be able to obtain another pod's identity. When CDS issues a certificate it asks the node's [admission inventory](/docs/c8s/concepts/architecture#the-admission-inventory) which pod is behind the requesting key, and the inventory answers on a privileged port that only the node's own network namespace can bind. The `c8s-deny-host-namespaces` ValidatingAdmissionPolicy (`hostNamespacePolicy.enabled`, on by default) denies tenant pods `hostNetwork`, `hostPID`, `hostIPC`, and `hostPath` volumes, which is what keeps a tenant pod out of that namespace and stops it from answering as the inventory. Privileged node DaemonSets — a CNI, a CSI driver, the NVIDIA GPU operator — *can* bind that port. They are already root inside the node CVM and can read another pod's memory directly, so c8s **assumes** them to be part of the node's trusted computing base rather than checking them. The policy exempts the release namespace, `kube-system`, and anything you add to `hostNamespacePolicy.exemptNamespaces`, so admitting one of those DaemonSets is a visible decision in your values file. Under pod-as-CVM each pod is its own CVM and the separation is hardware-backed instead. ## Where the boundary sits ```text OUTSIDE the TEE, and still deciding what the TEE will accept: CI · ghcr.io ────────────► the image digests baked into the measurement your operator key ───────► who may rewrite the served allowlist your measurement pins ───► which builds CDS and the mesh talk to ╔═ TEE BOUNDARY · HOST CANNOT READ ══════════════════════════════════════════╗ ║ ║ ║ attestation-api CDS workload ║ ║ verifies hardware mesh CA + EAR issuer, prompts · weights ║ ║ evidence one process KV cache ║ ║ │ │ │ ║ ║ └───── verdict ──────────►│──── leaf cert ───────────►│ ║ ║ │ ║ ║ │ CA bundle ║ ║ ▼ ║ ║ ratls-mesh ◄── attested mTLS ──► peers ║ ╚════════════════════════════════════════════════════════════════════════════╝ ALSO outside, and assumed hostile: hypervisor · host OS · kubelet · containerd · Kubernetes API · etcd · CNI ``` *Two different kinds of "outside". The supply chain never touches your data but decides what the platform will accept, so it is load-bearing; the host decides nothing and is assumed hostile.* ## What is enforced today | Gate | Enforced by | Source of truth | | --- | --- | --- | | TEE evidence is valid | attestation-api, co-located with CDS in the same TCB | hardware evidence verification | | A CSR can be signed | CDS | verified evidence, plus `cds.measurements` when set | | An image — and the argv it runs with — is allowed | nri-image-policy on the host, or in-guest `policy-monitor` under Kata | CDS-served allowlist, plus the seed baked into the measured image | | Mesh peer chains to the mesh CA | ratls-mesh | mesh CA bundle (chain only — the peer's measurement is not pinned) | | Pod is an injection candidate | mutating admission webhook, `failurePolicy: Fail` | pod annotation `confidential.ai/cw` | | A `confidential.ai/cw` pod cannot shed its injected `c8s-cert` sidecar | `c8s-cw-label-integrity` ValidatingAdmissionPolicy | API server | | A tenant pod cannot take the node's network, PID, or IPC namespace, or a `hostPath` | `c8s-deny-host-namespaces` ValidatingAdmissionPolicy | API server | | The front door's attestation is bound to *this* cluster's identity | `c8s cds-attest` sidecar | a TEE report whose `report_data` transcript commits the session keys, the client nonce, the exact mesh leaf, and its issuing CA | Two of these carry a caveat worth stating. The attestation-api's verdict is **unsigned**, so the gate holds because the verifier runs inside the same TCB as CDS — not because the response is authenticated. And the image gate covers the digest and the effective argv (see [Allowlist](/docs/c8s/concepts/image-allowlist)); it does not cover env, mounts, capabilities, or the rest of the pod spec. ## Measurement pinning is mandatory for production The default chart pins **no** launch measurements. Both `cds.measurements` and `ratlsMesh.measurements` default to *empty*, which means "accept any attested peer". That is fine for a demo or a dev loop, but it is **not the production trust model**: With empty measurements, an attacker on the pod network can stand up their own genuine TEE enclave (same open-source image, valid measurement, real silicon) and MITM the bootstrap — every signature checks out, but you are talking to an attacker-operated enclave. Pin both `cds.measurements` and `ratlsMesh.measurements` to your known-good launch digests in production — see [Obtaining launch measurements](/docs/c8s/how-to/obtain-launch-measurements) for how to get those values. Under node-as-CVM, `c8s install --measurements ` pins both values in one step: `M` is the node image's launch measurement, published in its `manifest.json`, and it is known before the cluster runs. The flag is rejected with `--cvm-mode=pod`, where each Kata guest is measured separately and the pins are per-guest. Empty is not a bug — it is "pin nothing yet", and only you know your measurements. CDS and ratls-mesh both log loudly when their allowlist is empty, and ratls-mesh publishes `ratls_mesh_measurement_pinning=0` for alerting. Neither signal makes accept-any safe. ## Proving it is *your* cluster A launch measurement proves "genuine audited code on real silicon". It does not prove "*my* cluster" — the open-source images are deliberately fungible across deployments, so anyone can boot the same build and produce a valid measurement. An outside verifier therefore has to pin a **cluster-unique** anchor as well: today, the mesh CA certificate. Those two pins now authenticate a cluster. The front door's attestation binds the exact mesh leaf and its issuing CA into the `report_data` transcript, and signs that transcript with the leaf's private key, so copying a target cluster's public certificate chain is not enough to impersonate it. The client verifies the hardware signature, the measurement against its allowlist, the transcript, the chain to the pinned mesh CA, and the possession proof before any application traffic flows. The proof is ECDSA, so cluster authentication is classical; the session keys are X25519 + ML-KEM-768 hybrid, which covers confidentiality but is not a post-quantum authentication claim. See [consumer verification](/docs/c8s/concepts/verification-model) for the client side of this. ## The production direction The chart-managed default keeps the CA key in CDS process memory: any restart generates a new CA and invalidates old leaves, and there is a single CDS. The production direction holds the CA key only inside attested CVM memory and lets replicas join via **attested key handoff** (`cds.handoff.enabled=true`), so Kubernetes only ever sees ciphertext and public material. A joining replica must present a launch digest in `cds.measurements` *and* a REPORTDATA-bound hash of the operator-key set that exactly matches the serving CDS, so a hostile control plane cannot hand the mesh CA to a replica running a substituted allowlist-write policy. The chart refuses to render if either value is empty while handoff is enabled. See [CDS bootstrapping](/docs/c8s/concepts/trust-root#ca-handoff). ## Out of scope (today) These are known gaps, tracked but not yet enforced — see [Limitations](/docs/c8s/concepts/limitations) for the full list: - The image gate covers the digest and argv, but not env, mounts, capabilities, or the rest of the pod spec. - Mesh peer verification chains to the mesh CA but does **not** pin the peer's measurement. - The [sandbox ID](/docs/c8s/concepts/architecture#the-admission-inventory) stamped into a leaf is vouched for by the mesh CA signature and the node's admission inventory, not by hardware evidence. - No per-workload peer allowlists in the mesh; no multi-tenant isolation between workloads on the same node CVM beyond what admission policy provides. And these are permanent non-goals: side-channel attacks (micro-architectural, timing, power), denial of service and availability, application-layer vulnerabilities in your own workload, model distillation or dataset reconstruction from legitimate query access, physical attacks on the host, and a compromised hardware manufacturer. --- # Pod-as-CVM vs Node-as-CVM This is the most consequential architectural decision in **c8s**: **what is the unit of trust and attestation?** There are two answers, and they correspond to two different ways of placing the Trusted Execution Environment (TEE) boundary. The layered picture (L0 = host/hypervisor, L1 = the node, L2 = the pods) makes the difference precise. You pick the shape at install time with `--cvm-mode`, which is required and has no default. Switching in place is disruptive — the two shapes move security components between host and guest — so choose before you install. ## Node-as-CVM (node-as-guest) ```text L1 (the node) = confidential VM (SEV-SNP or TDX) of L0 ← the attested unit L2 (the pods) = ordinary containers in L1 ← protected by L1's encryption boundary ``` The **entire Kubernetes node is one confidential VM (CVM)** — an AMD SEV-SNP or Intel TDX enclave. L0 (the cloud or bare-metal operator) sees only ciphertext of the node. Workloads run as ordinary containers inside that node; from a pod's point of view it is a normal node that happens to be a confidential VM. A verifier checks the **node's** launch digest — if it matches the published image, every container on the node is running in a verified state by definition. The control plane (kubelet, etcd, secrets) lives **inside** the verified node VM. Three install shapes land here: `--cvm-mode=node` for confidential VMs you run yourself (the evidence comes from `/dev/sev-guest` or `/dev/tdx-guest`), `--cvm-mode=gke` for Google's managed confidential VMs, and `--cvm-mode=aks` for Azure, where the evidence rides the vTPM at `/dev/tpm0` instead of a guest device. Pass `--measurements ` — the node image's launch digest, which you know before the cluster runs — and the install pins it into both internal trust boundaries at once. ## Pod-as-CVM (node-as-host) ```text L1 (the node) = not trusted (a plain launchpad) L2 (each pod) = its own per-pod confidential VM ← the attested unit ``` Each **pod is its own confidential enclave**, via [Kata](/docs/c8s/concepts/kata-runtime) and `--cvm-mode=pod`. The node is just the launchpad that asks KVM to start each pod's VM. A verifier checks the **pod's** launch digest, not the node's. Even a malicious node cannot tamper with what a pod runs — it can break the pod, but the break is *detected by attestation*, not silently exploited. Both CPU TEEs ship: `--hardware-platform=sev-snp` (the default) renders `kata-qemu-snp` and `--hardware-platform=tdx` renders `kata-qemu-tdx`. A cluster picks exactly one — mixed SNP + TDX clusters are not supported. ## Side-by-side | | Node-as-CVM | Pod-as-CVM | | --- | --- | --- | | Install flag | `--cvm-mode=node` / `gke` / `aks` | `--cvm-mode=pod` | | Attested unit | the whole node | each pod | | Trust required in the node operator | implicit — the in-node control plane is inside the boundary | **none** — the host is adversarial | | Isolation between tenants on a node | container-level only | per-pod hardware memory encryption | | Per-pod attestation to an external verifier | no (all pods share the node digest) | **yes** | | Measurement pinning | `--measurements` at install pins the node digest into `cds.measurements` and `ratlsMesh.measurements` | `--measurements` is refused; predict the guest digest and set those values in a values file | | Image admission | baked into the node image (`node`), or the host `nri-image-policy` DaemonSet (`gke` / `aks`) | in-guest `policy-monitor` | | Density / overhead | high density, one VM per node | one VM per pod; boot cost per pod | | Startup latency | normal pod start | includes VM boot | | Control plane in the boundary | yes (kubelet, etcd, secrets) | no — per-pod | | GPU | drivers baked into the node image, measured into the node digest | NVIDIA GPU passed through per pod over VFIO | | Image / build | one node image — `ghcr.io/confidential-dot-ai/c8s-base` (`:rke2`), rebuilt by c8s CI on every merge | sealed guest rootfs (`kata-guest-base`) plus a hypervisor-capable node OS (`rke2-kata`) | | Works on Azure AKS | **yes** — SEV-SNP or Intel TDX, both via the vTPM | **no** (needs nesting AKS lacks) | ### Image admission moves with the shape Every shape enforces the container-image [allowlist](/docs/c8s/concepts/image-allowlist), but the enforcement point differs — and the chart will not let you end up with none: - **Pod-as-CVM** (`--cvm-mode=pod`) enforces in-guest with `policy-monitor`, fed by an allowlist seed baked into the launch measurement plus CDS's served `/allowlist`. The chart *fails the render* if the host-side `nri-image-policy`, `ratls-mesh`, or `attestation-api` are left enabled alongside it: the host copies would be a second, unattested enforcement path. - **`--cvm-mode=node`** bakes a fail-closed `nri-image-policy` into the node image, so the chart's own copy is disabled to avoid duplicating it. - **Every other cluster** — `gke`, `aks`, or any non-Kata install — has the host `nri-image-policy` DaemonSet as its *only* image-admission gate. The chart therefore **refuses to render** with `nriImagePolicy.enabled=false` there; disabling it would run confidential workloads with no attested allowlist at all. ## When to choose which In short: - **Node-as-CVM** is the all-or-nothing model: verify the node once, and everything on it is trusted. Simplest, densest, and the only option on managed services without nested virtualization. The tradeoff is that you must trust the in-node control plane and you get no per-pod attestation. - **Pod-as-CVM** is the mutual-distrust model: the platform operator and the tenant workloads do not trust each other, and each pod carries its own attested identity. The tradeoff is one VM per pod (more overhead, slower starts) and a more complex install (Kata runtime, per-pod measurement pinning, admission enforcement). **Pod-as-CVM is not available on Azure.** Azure's hypervisor does not expose nested virtualization, so a per-pod VM cannot be launched inside an AKS node. On Azure you use Node-as-CVM, where the AKS node is a single confidential CVM and all its pods share that one boundary. Both CPU TEEs are supported there — `--hardware-platform=sev-snp` and `--hardware-platform=tdx` both attest through the Azure vTPM, since AKS exposes no guest TEE device. See [Provisioning on Azure](/docs/c8s/how-to/provision-azure-aks) for the details and what you still get. ## Implementation note The two shapes place different demands on the node image. **Node-as-CVM** works on any confidential VM image. If you supply your own, c8s CI publishes one built for it — `ghcr.io/confidential-dot-ai/c8s-base` (hardened kernel plus kubelet/containerd/CNI, dm-verity with IGVM/UKI), which also bakes in the attestation and image-policy components so they fall under the node's launch digest. On a managed platform that does not let you change the node image — AKS and GKE confidential node pools — you run the provider's image and the chart deploys those components as DaemonSets instead. Either way the node carries a launch digest; what differs is who builds the thing being measured, and therefore how much of the stack the digest covers. **Pod-as-CVM** does constrain the node OS: it needs a hypervisor-capable one (`rke2-kata`, a full kernel with KVM and VFIO) whose job is to launch the sealed `kata-guest-base` per-pod VMs. Here the per-pod VM is the confidential unit and carries the digest that matters, so the node OS itself is outside the boundary. The host requirements differ too. Booting a node image via IGVM needs QEMU 10.1 or newer built with `--enable-igvm`, which most distributions do not ship — check with `qemu-system-x86_64 -object igvm-cfg,help`. Pod-as-CVM needs no host QEMU at all: `kata-deploy` brings its own TEE-capable build. Once you've chosen a shape, [Kata Containers](/docs/c8s/concepts/kata-runtime) covers how the Pod-as-CVM path is actually built — the measured boot, the sealed guest image, the admission rules, and the guest-pull model. --- # The trust root The **Certificate Distribution Service (CDS)** is the trust root of a c8s cluster. In one process it verifies TEE attestation evidence, issues short-lived **EARs** (Entity Attestation Results), manages an in-memory **mesh CA**, signs leaf certificates, gates issuance on what the requesting pod is actually running, and — optionally — hands its CA off to peer replicas. There is no internal RA-TLS hop to an external signer, so the CA private key never leaves CDS memory. ## Mesh CA bootstrap At startup the CDS obtains its **ECDSA P-384** mesh CA entirely in memory: - The private key is **in-process only** and is never written to disk or to a Kubernetes Secret. - Validity defaults to **8760h (one year)**, configurable with `--ca-cert-validity`. The CN defaults to `c8s Mesh CA` (`--ca-common-name`). - With no `--handoff-peer-url` the CDS generates a fresh self-signed CA. With one set it adopts a surviving peer's CA over the attested [handoff](#ca-handoff) flow instead, and **fails startup** if that peer is unreachable, refuses, or attests a different operator-key policy — so a partition can never mint a divergent trust root. - The public CA bundle is published so peers and clients can chain to it; the CA fingerprint and its source (`self-generated` or `adopted-from-peer`) are logged at startup. Because the key lives only in memory, a CDS restart *without* a handoff peer generates a *new* CA and invalidates previously issued leaves. That is acceptable for the chart-managed default; the production direction keeps the CA in attested CVM memory and replicates it via [handoff](#ca-handoff). ## EAR signing and JWKS rotation The **EAR** is a JWT, signed with **ES256** (ECDSA P-256), that the CDS mints via `/attest-key` after a successful attestation. It is a TEE-bound credential for flows that attest a *key* rather than obtain a certificate: CDS's own handoff machinery uses it to authorize [CA handoff](#ca-handoff), and a caller already holding one can have a CSR signed via `POST /sign-csr`. Workload certificate issuance does **not** use it — `/attest` verifies evidence and returns the signed leaf in one step. (Allowlist writes use an [operator token](/docs/c8s/concepts/image-allowlist#authorizing-mutations), not an EAR.) Key claims: | Claim | Meaning | | --- | --- | | `iss` | issuer (`--ear-issuer`, default `cds`) | | `iat` / `exp` | issued-at and expiry. the lifetime follows `--cert-ttl` (default 24h) | | `submods.attester` | the attestation result, including `ear_status`, the trustworthiness vector, raw evidence, and the normalized `launch_digest` | | `tee_public_key` | base64url PKIX DER of the ECDSA key the TEE attested — used for key binding | | `operator_keys_hash` *(optional)* | the canonical hash of the CDS operator public-key set, folded into REPORTDATA so two handoff replicas can require the same write policy | | `pbh` *(optional)* | payload-body hash, binding the token to a specific request body | The signing key rotates with overlap so verification never breaks mid-rotation: - `--token-signer-rotation-interval` (default `720h`) — how often a new P-256 key becomes active. - `--token-signer-overlap` (default `25h`) — how long a retired key stays valid and published. - `--token-signer-rotation-jitter` (default `0.1`) — jitter on the first tick. Each key's `kid` is its RFC 7638 JWK thumbprint. The current JWKS is served at `GET /.well-known/jwks.json`. A retiring key is rejected the moment it passes its overlap deadline, and drops out of the JWKS at the same point — not at the next rotation, which with the defaults is weeks later. ## The attestation flow The CDS uses a nonce-challenge model so attestation evidence is cryptographically bound to a CSR's public key, preventing evidence replay or substitution. One challenge covers the whole issuance: it binds the evidence *and* the [sandbox token](#sandbox-identity), so neither can be replayed into a different request. ```text get-cert CDS attestation-api [in the pod] [trust root] [same TCB as CDS] │ │ │ │ POST /authenticate │ │ │──────────────────────►│ │ │ │ │ │ challenge │ │ │ (single-use, 32 B) │ │ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│ │ │ │ │ ┌───┴─────────────────────────────────────┐ │ │ redeem a sandbox token from the local │ │ │ inventory (see Sandbox identity) │ │ │ report_data = │ │ │ SHA-384(CSR pubkey ‖ challenge) │ │ └───┬─────────────────────────────────────┘ │ │ │ │ │ POST /attest │ │ │ challenge + evidence + CSR + sandbox_token │ │──────────────────────►│ │ │ │ verify(evidence, │ │ │ report_data) │ │ │──────────────────────►│ │ │ │ │ │ valid · launch_digest│ │ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│ │ │ │ │ ┌────────┴─────────────────────┐ │ │ │ measurement pin · CSR policy │ │ │ │ sandbox images allowlisted │ │ │ │ sign (mesh CA) │ │ │ └────────┬─────────────────────┘ │ │ │ │ │ leaf certificate + CA chain │ │◄ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─│ │ │ │ │ ``` *The challenge is the freshness proof; the evidence proves the key lives in a measured TEE; the sandbox token proves which pod is asking.* Step by step, on `POST /attest` the CDS: 1. consumes the one-time challenge (decoded from base64; `--challenge-ttl`, default `60s`); 2. parses the CSR and verifies its self-signature; 3. verifies the sandbox token, if one was sent — see [Sandbox identity](#sandbox-identity); 4. computes the expected `report_data = SHA-384(CSR_public_key ‖ challenge)` (first 48 bytes); 5. calls the attestation-api `/verify` with the evidence and that expected report data; 6. requires `signature_valid == true` **and** `report_data_match == true`; 7. if `--measurements` is set, requires the evidence `launch_digest` to be in that allowlist; 8. validates the CSR's DNS/IP SANs and CN against policy (`--dns-san-pattern`, `--san-validation`, `--allowed-cn-pattern`); 9. asks the sandbox's inventory what it is running and requires every image to be allowlisted; 10. signs the CSR with the mesh CA and returns the PEM chain. `--measurements` defaults to empty, which means **no pinning** — any TEE that produces valid evidence is accepted. The CDS logs a warning at startup in this mode. Pin it in production — see [Obtain launch measurements](/docs/c8s/how-to/obtain-launch-measurements) for how to get the values to put here. ### What an issued leaf carries Leaves are short-lived: `/attest` issues at `--cert-ttl`, hard-capped at 24h, and `/sign-csr` honors the caller's requested TTL up to `--max-ttl` (default 24h). Every leaf carries, in its signed area: | OID | Extension | Contents | | --- | --- | --- | | `1.3.6.1.4.1.59888.1.1` | RA-TLS attestation | the requester's own hardware evidence, copied verbatim from the CSR | | `1.3.6.1.4.1.59888.1.2` | audit digest | SHA-256 of the evidence the CDS accepted at issuance | | `1.3.6.1.4.1.59888.1.4` | pod sandbox ID | present only when the requester presented a sandbox token | The `.1.1` extension is what makes an issued cert **re-verifiable**: get-cert binds evidence to its bare CSR key with no nonce, and the CDS copies that extension onto the leaf, so a relying party can re-check the hardware evidence itself rather than trusting the CA chain alone. The challenge-bound evidence verified at issuance is deliberately *not* embedded — its REPORTDATA includes the consumed challenge, so it could never re-verify against the bare key. ## Sandbox identity Hardware attestation binds a *key* to a *launch measurement*. Under Node-as-CVM every pod on a node shares that one measurement, so it says nothing about **which pod** stands behind a mesh key. Sandbox identity narrows that: a leaf names the CRI **pod sandbox** the CDS issued it to, and the CDS issues only after asking that sandbox's own admission record what it is running. The **inventory** is the component that admitted the pod's containers — `nri-image-policy` on Node-as-CVM, `policy-monitor` inside the Kata guest. It is the arbiter of both which sandbox a process belongs to and what runs in that sandbox, and it serves two deliberately disjoint surfaces: a **local token route** that only the pod's own `get-cert` reaches, and a **network identity/digests route** on port `1019` that only the CDS reaches. The token route cannot enumerate other sandboxes; the network route cannot mint identity. ```text ╔═ one TEE: the node under Node-as-CVM, the guest under Pod-as-CVM ═╗ ║ ║ ║ ┌────────────┐ 1 CSR pubkey + ┌────────────────────────┐ ║ ║ │ get-cert │──── CDS challenge ─►│ inventory: │ ║ ║ │ [c8s-cert] │◄─── signed sandbox ─│ nri-image-policy on │ ║ ║ └─────┬──────┘ 2 token │ Node-as-CVM, or │ ║ ║ │ │ policy-monitor in the │ ║ ║ │ │ Kata guest │ ║ ║ │ └───────────▲────────────┘ ║ ╚════════║════════════════════════════════════════║═════════════════╝ │ │ 4 GET /identity │ 3 sandbox_token, forwarded │ GET /digests/ │ verbatim on POST /attest │ on :1019, mutually ▼ │ attested RA-TLS ┌──────────────────────────────────────────┐ │ │ CDS — verifies the token under the key ├──────┘ │ it fetched, requires every reported │ │ image to be allowlisted, then stamps │ │ the sandbox ID onto the leaf │ └──────────────────────────────────────────┘ ``` *The requester never names its pod and never reports its images; both answers come from the component that made the admission decision.* 1. **get-cert asks anonymously.** It `POST`s `/sandbox` carrying only its CSR public key and the CDS challenge — no PID, pod name, or container ID. On Node-as-CVM this is a node-local Unix socket, where the kernel stamps the caller's credentials (`SO_PEERCRED` plus `SO_PEERPIDFD`) and the inventory resolves them PID → cgroup → container → sandbox from its own admission record. Inside a Kata guest it is the guest's loopback `127.0.0.1:8401`, where the one-pod- per-guest boundary leaves nothing to disambiguate. The webhook injects `--workload-claims` into the `c8s-cert` sidecar (plus `--workload-claims-guest` under Kata); both addresses are compiled into get-cert, so that flag selects the *shape*, never an address, and a wrong setting fails closed against a port nothing serves. 2. **The inventory signs a token** over the sandbox ID, SHA-256 of the requester's public key, the CDS challenge, and the IP of the node or guest serving its own identity endpoint. The envelope carries **no credential** for the signing key. 3. **get-cert forwards it opaquely** in the `/attest` body as `sandbox_token`. 4. **The CDS resolves the key itself.** It reads the inventory host out of the *unverified* token — the only thing that value ever does is pick a dial target — requires it to be a routable unicast IP literal (never a name, never loopback, link-local, or multicast) inside a CIDR the operator configured with `--sandbox-inventory-cidr`, then fetches the signing key from `GET /identity` at `:1019` over mutually-attested RA-TLS. Only then does it verify the signature, require the token's nonce to be the challenge it is consuming, and require the key digest to name the CSR key. 5. **The CDS asks what the sandbox runs.** `GET /digests/` at the same endpoint returns the sorted, deduplicated image digests the inventory currently tracks in that sandbox. Every one must be in the [allowlist](/docs/c8s/concepts/image-allowlist). An empty answer is refused, not treated as "nothing to check" — a sandbox always runs at least the sidecar that is asking. Port `1019` is the whole identity argument. It is a compiled constant, not a deployment value, and it is privileged: binding it requires the node's own network namespace, which the chart's `deny-host-namespaces` ValidatingAdmissionPolicy (`hostNamespacePolicy.enabled`, on by default) withholds from tenant pods. Measurement cannot make this distinction on Node-as-CVM — every pod there shares the node's launch digest — but "answers on `:1019` in the node's netns, at an address inside the operator's node range" can. **`--sandbox-inventory-cidr` is required for any of this.** Unset, the CDS refuses every request carrying a sandbox token and logs a warning at startup; pods still get mesh certificates, but with no sandbox ID and no issuance-time image gate. `c8s install --node-cidr` sets the chart value `cds.sandboxInventoryCIDRs`, and with the flag omitted `c8s install` fills it in from the cluster as one host route per node — a point-in-time snapshot, so a node added later is not covered until the value is refreshed. A CDS with no `--ratls-platform` has no RA-TLS identity to present to an inventory, so it also refuses every token. ### What vouches for a sandbox ID — and what does not The sandbox ID rides the leaf's signed area; it is **not** folded into REPORTDATA. The mesh CA signature, not the hardware evidence, is what authenticates it. Everything downstream encodes that: - A relying party pins it with `c8s verify --sandbox-id --mesh-ca `. `--mesh-ca` is **mandatory** with `--sandbox-id`. Without it the verdict still reports the ID, but alongside a `sandbox_id_note` saying it is not verified — an unqualified ID never reads as attested. - In-mesh, a `SandboxID` pin is enforced only on the CA-verified branch of the dual verifier. Pure RA-TLS verification of a self-signed peer **fails closed** on a pin, because a self-signed leaf's extension is whatever its holder chose. The gate is **membership, not composition**: every running image must be allowlisted, but the running set is not required to match a whole workload entry. Issuance lands at arbitrary points in a pod's lifecycle — an init container running, main containers coming up one at a time, one restarting, completed init containers reaped — and in each of those the running set is a strict subset of what the pod declares. Requiring the whole set would deny certificates for ordinary states, permanently so once init containers are reaped. So a leaf's sandbox ID means *this key belongs to pod X*, not *pod X runs exactly workload Y*. Per-container digest and argv policy is enforced continuously at admission by `nri-image-policy` / `policy-monitor` instead — see [The Allowlist](/docs/c8s/concepts/image-allowlist). The residual trust is the inventory itself: the key's provenance narrows to a *node*, not to a process, so anything able to bind `:1019` on a node — the inventory, or a privileged node DaemonSet — can sign for any sandbox that node admitted. Under Kata each guest holds one pod and the token's host selects which guest the CDS asks, which is tighter. See [Limitations](/docs/c8s/concepts/limitations). ## The image allowlist The CDS also **owns and serves the image allowlist** — the digests of the container images permitted to run (a SQLite store at `--allowlist-db`, served at `/allowlist`). Updating it always goes *through the CDS*; there is no side channel. It is the same store the sandbox gate above checks at issuance. Who may update it is decided by the operator **at CDS bootstrap**: the install-time value `cds.operatorKeys` (`--operator-keys`, set by `c8s install --operator-keys`) pins the operator EC **public** key(s) authorized to write; empty rejects every write while reads keep serving. For each write, the [`c8s allowlist` CLI](/docs/c8s/how-to/manage-the-allowlist) signs a short-lived token with the operator's private key, bound to the exact method, path, and request body; the CDS verifies the signature against its pinned keys and re-checks each binding before committing. The pinned bundle is public material (served back at `GET /operator-keys`), but it is host-supplied config — it is committed to REPORTDATA only on the `/attest-key` and `/handoff` paths, not by the serving certificate. Cross-check it with `c8s cds verify --operator-keys `, which fails closed on a swapped key set. See [The Allowlist](/docs/c8s/concepts/image-allowlist) for the data model, the full write flow, seeding, and enforcement. ## CA handoff The in-memory-only CA is exactly what makes high availability tricky. Because the private key never touches disk (see [Mesh CA bootstrap](#mesh-ca-bootstrap)), the usual HA tactics don't apply: a cold-standby replica or a simple pod restart each comes up generating a *fresh* CA, which signs a different chain and **invalidates every leaf already issued**. There is no shared key on a PersistentVolume to fail over to — that's the whole point of keeping it in memory. Handoff resolves this by moving the *live* CA from one attested CDS to another, in memory, over an attested channel — never through disk or a third party: 1. A successor replica starts, generates an ephemeral **X25519** keypair, attests itself, and calls `POST /handoff` on the active CDS, presenting its EAR and that ephemeral public key. 2. The active CDS verifies the EAR against its own JWKS, checks the recipient's `launch_digest` against `--handoff-measurements`, and requires the recipient's REPORTDATA-bound `operator_keys_hash` to equal its own — so the CA only ever leaves for another CDS whose exact launch measurement you have pinned *and* whose allowlist-write policy matches. 3. It returns the payload — the CA cert and private key, plus the current allowlist snapshot (digest floor and workload entries) — **AES-256-GCM-encrypted to the recipient's X25519 key** (X25519-ECDH → HKDF-SHA256 → AES-256-GCM). The key crosses the wire only as ciphertext that one attested enclave can open, and it is written to disk on neither side. The successor now holds the **same** CA and the same runtime policy state, so every certificate the cluster has already issued stays valid across the changeover — a rolling upgrade or a failover happens with no trust-on-first-use gap and no mass re-issuance. Handoff is opt-in (`cds.handoff.enabled=true`, which passes `cds.measurements` through as `--handoff-measurements`) and **requires `--operator-keys`**, since the operator policy is what both sides attest. With no measurements listed, `POST /handoff` is not mounted at all. Pair it with the CA's validity window (`--ca-cert-validity`) and the `/readyz` gate — which starts failing once remaining validity drops below `--min-ca-validity` (default `1h`) — so a successor is brought up and handed the CA well before the active one's certificate expires. ## Verifying the CDS itself (RA-TLS) Clients verify the CDS *before* trusting anything it returns. The CDS serves its API over **RA-TLS**: its serving certificate carries hardware evidence (SEV-SNP or TDX) in an X.509 extension, minted for the platform named by `--ratls-platform` (default `sev-snp`). A client (for example `get-cert --cds-measurements `) verifies that evidence through its local attestation-api and checks the CDS launch digest against the supplied allowlist before completing the handshake. With no measurements supplied the client accepts **any** RA-TLS-attested CDS and logs a warning — pin `--cds-measurements` in production (the CDS's own launch digest; see [Obtain launch measurements](/docs/c8s/how-to/obtain-launch-measurements)). To check the CDS on demand rather than as part of a client handshake — after install, or on a schedule from CI — run [`c8s cds verify`](/docs/c8s/how-to/verify-a-deployment#verify-the-cds). Browsers cannot do RA-TLS at all — they cannot inspect a certificate mid-handshake — so external clients get a different proof, and it terminates in the same place: the `cds-attest` sidecar beside tls-lb signs its post-quantum session transcript with the **CDS-issued mesh leaf** and commits the mesh CA alongside it, so the identity a browser ends up trusting is the mesh identity this CA issued. See [consumer & browser verification](/docs/c8s/concepts/verification-model). ## See also - [CDS HTTP API](/docs/c8s/reference/cds-api) — every endpoint, with auth and response codes. - [CLI reference → `c8s cds`](/docs/c8s/reference/cli#c8s-cds) — every CDS flag. - [Obtain launch measurements](/docs/c8s/how-to/obtain-launch-measurements) — where the values the CDS pins come from. - [Verify a deployment](/docs/c8s/how-to/verify-a-deployment) — checking the CDS, the load balancer, and workloads by hand or in CI. --- # The verification model Attestation is only as good as somebody's ability to check it. In a c8s cluster three parties do, and they reach the same enclave by different routes: | Who | When | How | | --- | --- | --- | | **Services**, on each other | continuously, on every connection | [RA-TLS](#service-to-service-verification-ra-tls) — evidence in the peer's serving certificate, checked mid-handshake | | **The operator**, on a component | after install, then on a schedule from CI | [`c8s verify`](/docs/c8s/how-to/verify-a-deployment) — dial it, pull its evidence, check it against a pinned measurement | | **The end consumer**, on the front door | per session, from a browser | [over-encryption](#the-idea-over-encryption-not-certificate-extensions) — a second, attested channel inside the ordinary HTTPS one | The first two both pull evidence out of an X.509 extension mid-handshake. **Browsers expose no API to do that**, so the consumer path cannot work the same way — and the consumer is the party whose independent check matters most. The design that works anyway is the bulk of this page. ## Service-to-service verification (RA-TLS) Inside the cluster you rarely verify by hand. Backend components authenticate each other with **[RA-TLS](/docs/c8s/concepts/trust-root#verifying-the-cds-itself-ra-tls)**: a workload's `get-cert` and the `ratls-mesh` proxies pull the peer's attestation evidence out of its serving certificate during the handshake, check it against the mesh CA, and only then complete the connection. That verification is automatic and continuous — there is no operator step. Two things a CDS-issued leaf now carries, which change what a relying party can re-check: - **The evidence CDS accepted at issuance**, stamped onto the leaf. A peer doing RA-TLS-mode verification sees exactly what CDS verified instead of falling back to the CA chain alone. - **The pod sandbox ID**, when the requester presented an inventory-signed sandbox token. It sits in the leaf's *signed* area, not in REPORTDATA — the mesh CA signature is what vouches for it, which is why `--sandbox-id` below requires `--mesh-ca`. The mesh's RA-TLS verifier accepts the `snp`, `gcp-snp`, `az-snp`, `tdx`, and `az-tdx` evidence shapes. `az-tdx` is an Azure confidential VM's TD quote wrapped in the vTPM HCL report — the Azure TDX path has no `/dev/tdx-guest` — and pins MRTD as the launch measurement. What you verify by hand is the trust root: confirm the **CDS** itself is a genuine TEE at your expected measurement with [`c8s cds verify`](/docs/c8s/how-to/verify-a-deployment#verify-the-cds). Everything else chains to it. ## The verifier engine `c8s verify` does not embed a verifier of its own. It verifies **in-process** with `attestation-go` — the Go port of the same `attestation-rs` engine the cluster runs — so there's **no container to launch and no service to reach**. Using the cluster's own engine means the product line never has to be supplied by hand: it auto-detects the platform and AMD product, including **Zen4c (Siena / Bergamo)**, which stock `go-sev-guest` cannot classify. Any AMD collateral that isn't shipped inline (e.g. the VCEK for a bare report) is fetched from **AMD KDS**. Intel TDX quotes verify against an **Intel SGX Root CA pinned inside the verifier** and fingerprint-checked when it loads — no Intel endpoint is contacted. The vendor signature chain, the REPORTDATA binding, the debug-guest rejection, and the minimum-TCB floor are enforced inside the verifier. The **launch-measurement allowlist** (`--measurements`) has no verifier-side input, so it is enforced client-side and **fails closed**: if the report's launch digest isn't in your allowlist, verification fails. ## What the front door exposes The consumer-facing surface is the **c8s** Load Balancer (`tls-lb`): an nginx front end plus a `cds-attest` sidecar, both inside the TEE (its own CVM under Pod-as-CVM, the node's under Node-as-CVM). The sidecar ships **enabled by default** (`tlsLb.attest.enabled`), so a stock install already answers browser verifiers; install with `--attest=false` to omit it. | Path | Served by | Carries | | --- | --- | --- | | `/.well-known/c8s/attestation` | `cds-attest` | nonce-bound evidence, session keys, mesh leaf + CA, identity proof | | `/.well-known/c8s/handshake` | `cds-attest` | the hybrid key-agreement exchange | | `/.well-known/c8s/tunnel` | `cds-attest` | over-encrypted application records | | `/v1/discovery` | nginx | the LB's issued cert + evidence with the VCEK inline | | `/allowlist`, `/allowlist/` | nginx → in-pod proxy | CDS's [image allowlist](/docs/c8s/concepts/image-allowlist) API | | `/.well-known/cds-cert.pem`, `/.well-known/mesh-ca.pem` | nginx | static PEM files for in-cluster discovery | All of those locations answer **any origin** by default. `tlsLb.cors.protocolEndpoints` (default `true`) puts `Access-Control-Allow-Origin: *` on them, with **no** `Access-Control-Allow-Credentials`, and short-circuits preflight `OPTIONS` at nginx with a `204` advertising `GET, POST, OPTIONS`, the headers `Authorization, Content-Type, X-C8s-Session`, and a 600-second cache. That is deliberate: these endpoints exist so any browser anywhere can verify the cluster, every response is either self-authenticating (hardware evidence, CDS-signed certificates, sealed tunnel records) or public by design, and no request on them relies on ambient browser credentials — allowlist writes are authorized by an operator-signed token over method, path, and body, not by a cookie. Two boundaries to keep straight: - **Workload traffic is not covered.** `tlsLb.routes` and the catch-all upstream get no CORS headers from this default. Setting the operator's own `tlsLb.cors.enabled: true` block replaces the built-in policy everywhere, protocol endpoints included; `tlsLb.cors.protocolEndpoints: false` removes CORS from the protocol endpoints entirely. - **Reading from this front door is not verification.** `/.well-known/mesh-ca.pem` and a plain `GET /allowlist` both arrive over ordinary TLS through whatever terminator sits in front of the LB. They are discovery material, not trust anchors — pin the mesh CA out of band (below), and see [Image allowlist](/docs/c8s/concepts/image-allowlist) for the attested way to read the allowlist. ## The idea: over-encryption, not certificate extensions Instead of hiding attestation inside the TLS certificate, the LB serves it over a normal HTTPS endpoint, via a challenge/response, and the client establishes a second, attested channel *inside* the existing connection: ```text ┌─────────────────┐ ┌─────────────────┐ ╔══════════════════════╗ │ Browser │ │ TLS terminator │ ║ LB inside a TEE ║ │ c8s-verify │ │ untrusted │ ║ nginx + cds-attest ║ └────────┬────────┘ └────────┬────────┘ ╚══════════┬═══════════╝ │ │ │ │ GET .../attestation?nonce=N (N = 32 bytes)│ ├─────────────────────┼──────────────────────►│ │ │ ┌────────────────┴───────────┐ │ │ │ mint per-session X25519 + │ │ │ │ ML-KEM-768 key │ │ │ │ transcript = H(version, │ │ │ │ mesh CA, mesh leaf, │ │ │ │ both pubkeys, N) │ │ │ │ HW report_data = transcript│ │ │ │ mesh leaf signs transcript │ │ │ └────────────────┬───────────┘ │ evidence + pubkeys + leaf/CA PEM + proof │ │◄────────────────────┼───────────────────────┤ ┌────────┴─────────────────┐ │ │ │ verify in WASM: │ │ │ │ hardware chain │ │ │ │ measurement ∈ allowlist │ │ │ │ report_data = transcript│ │ │ │ leaf → pinned mesh CA │ │ │ │ proof signature │ │ │ └────────┬─────────────────┘ │ │ │ POST .../handshake (client X25519 + ML-KEM ciphertext) ├─────────────────────┼──────────────────────►│ │◄────────────────────┼───────────────────────┤ session_id │ AES-256-GCM = HKDF(secrets, salt=transcript) │ POST .../tunnel + X-C8s-Session (sealed) │ ├─────────────────────┼──────────────────────►│ │◄────────────────────┼───────────────────────┤ sealed response ▼ ▼ ▼ ``` *The plain HTTPS hop is never trusted; every check runs on the returned payload, and the channel terminates inside the LB's enclave.* A malicious TLS-terminating proxy may sit in front of the real LB. Verification is performed entirely on the returned payload, and only then does the client derive an end-to-end encrypted channel to the attested per-session key. Everything after that is confidential to the LB's enclave regardless of the outer TLS terminator. ## The post-quantum over-encryption channel The channel uses a hybrid KEM — **X25519** (classical, WebCrypto) **+ ML-KEM-768** (post-quantum, `mlkem-wasm`) — following the TLS `X25519MLKEM768` convention: 1. Encapsulate against the attested ML-KEM key → `(mlkem_ct, mlkem_ss)`. 2. ECDH an ephemeral X25519 key with the attested X25519 key → `x25519_ss`. 3. `ikm = mlkem_ss ‖ x25519_ss`. 4. `key = HKDF-SHA256(ikm, salt = transcript, info = "c8s-verify/v1/over-encryption", L = 32)` → **AES-256-GCM**. The salt is the **identity transcript** the hardware report committed to, not the bare nonce, so the derived key is worthless to anyone who could not produce that report. Application traffic then rides a single `POST /.well-known/c8s/tunnel` endpoint as **CBOR**, one AES-256-GCM record per message with a fresh 12-byte IV, keyed to a session the client names with the `X-C8s-Session` header. The **entire** request is sealed — method, path, headers, and body — so a proxy in front of the LB sees only ciphertext, not even the path or `Authorization` header. The LB enclave opens the record, forwards the plaintext request to the backend over the cluster's RA-TLS mesh, and seals the response back. Three limits worth knowing: - **Replay is rejected.** Each channel remembers the IV of every record it has successfully opened and refuses a repeat, so the untrusted terminator cannot resubmit a captured request for a second authenticated backend action. The set is bounded (4096 records); past that the channel fails closed and must be re-established. - **Sessions expire.** `--session-ttl` (default 5 minutes) bounds both the pending handshake between the attestation fetch and the `POST /handshake`, and the idle life of an established session. - **Records are size-capped.** A sealed request is read up to 8 MiB; an upstream response larger than 32 MiB is refused rather than sealed. ## Streaming responses The tunnel is request/response: `cds-attest` reads the whole backend response, seals it as one record, and returns it. A Server-Sent Events body from a streaming inference backend therefore arrives in a single chunk at the end, and is bounded by the 32 MiB response cap. Streaming works on the **plain front-door path** — the nginx catch-all upstream, without over-encryption. That location sets `proxy_buffering off` and `proxy_http_version 1.1`, so tokens are forwarded as the backend emits them instead of being accumulated and released in clumps. The two are a real trade: over-encryption to the enclave, or token-by-token streaming. ## Why cluster identity is pinned The client pins **two** things out of band: the LB **measurement** allowlist and the **mesh CA certificate**. Pinning a measurement alone is not enough, and the reason is **cluster identity**: The CDS and LB images are open source and reproducible — which is what makes them auditable — but it also means a valid measurement only proves *"a genuine instance of the audited code on real silicon,"* not *"my cluster."* An attacker could stand up their own genuine LB enclave (same image, valid measurement, real vendor chain) and proxy you to it. The one value that is unique per cluster is the **mesh CA key**, generated inside the CDS TEE — so the client must pin something cluster-unique, and today that is the mesh CA certificate. The binding closes the copy attack: `report_data` commits the **exact mesh leaf and its issuing CA**, and the LB signs the same transcript with the leaf's private key. Copying a victim cluster's public certificate chain gets an attacker nothing without that private key, which never leaves the CDS-issued pod's TEE. There is no legacy or downgrade binding — the attestation endpoint rejects a `binding` parameter outright, and rejects any nonce that is not exactly 32 bytes. The proof signature is ECDSA, so **cluster authentication is classical**. X25519 + ML-KEM-768 gives the session key hybrid confidentiality; this path does not claim post-quantum authentication. ## Transitivity of trust The consumer only verifies the **LB**. Because the LB and CDS are open-source c8s components whose code anyone can audit, and because the LB's leaf chains to the CDS mesh CA — as do all other pods in the cluster — relying on the LB as a single point of trust transitively vouches for the backend pods it talks to over RA-TLS. The c8s mesh carries the rest. ## See also - [c8s-verify (JavaScript)](/docs/c8s/reference/c8s-verify-js) — the library that implements the consumer path. - [Verify a deployment](/docs/c8s/how-to/verify-a-deployment) — the operator path, command by command. - [The trust root](/docs/c8s/concepts/trust-root) — what the CDS vouches for, and how it is itself verified. --- # Create an encrypted volume An **encrypted volume** is data too large to be a secret — model weights are the case it is built for. It sits as ciphertext on storage the untrusted host reads freely and decrypts only inside a TEE, for a workload the [allowlist](/docs/c8s/concepts/image-allowlist) names. This page is the procedure. For the artifact's layers, what happens at mount time, and what the design does and does not defend against, see [Encrypted volumes](/docs/c8s/concepts/encrypted-volumes). **`volumed` is off by default** (`volumed.enabled: false`) and nothing here works without it. Turn it on at install with a values file — see [volumed, the node agent](/docs/c8s/concepts/encrypted-volumes#volumed-the-node-agent). Volumes also require **Node-as-CVM**; the webhook rejects `confidential.ai/c8s-volumes` at admission under Pod-as-CVM. ### Build the image and store the key `c8s volume create` formats the filesystem, builds the hash tree, generates a key, encrypts, and `PUT`s the blob to the CDS secret store. It modifies no workload: ```bash c8s volume create \ --name weights \ --source ./llama-3.1-8b \ --out ./weights.img \ --path /tenant-a/volumes/weights \ --escrow-out ./weights.escrow.json \ --node node-1 \ --url https://cds.example \ --measurements-file ./measurements.txt \ --operator-key ./operator.key ``` | Flag | Required | Effect | | --- | --- | --- | | `--name` | yes | Volume name. A DNS-1123 label of at most **12 characters** — it forms the device serial `c8s-vol-` and the directory the plaintext appears in. A longer name is rejected. | | `--source` | yes | Directory whose contents become the volume. | | `--out` | yes | Where to write the encrypted image. Must not already exist. | | `--path` | yes | Secret-store path for the key, e.g. `/tenant-a/volumes/weights`. Absolute, clean, no wildcards. | | `--escrow-out` | yes | Where to write the key blob you must keep. Written `0600`; refuses to overwrite. | | `--node` | no | Node holding the device. Emitted as a `nodeSelector` in the printed output. | | `--work-dir` | no | Directory for build intermediates. Default: a temp dir. The intermediates — the plaintext image and the tree — are removed either way, success or failure. | | `--dry-run` | no | Build the image and write escrow, but do not call CDS. | It also takes the shared CDS connection and credential flags: `--url`, `--measurements` / `--measurements-file`, `--operator-key` (or `C8S_OPERATOR_KEY`), `--timeout` (default `15s`), and `--insecure`. The write is authorized by an operator EC key whose public half CDS pins via `c8s install --operator-keys`. The build needs `mkfs.erofs` and `veritysetup` on the machine running it. It does **not** need root, a loop device, or `cryptsetup` — the encryption is done in process. The key is generated per volume and never taken from you. There is no flag to supply one. Output: ```text + ./weights.img (26214400 data blocks) + key stored at /tenant-a/volumes/weights + key escrowed to ./weights.escrow.json — keep it; a CDS restart needs it Attach ./weights.img to the node as a raw block device with serial c8s-vol-weights. Pod annotations: confidential.ai/cw: confidential.ai/c8s-volumes: "weights=/tenant-a/volumes/weights" Pod nodeSelector (the device is on one node): kubernetes.io/hostname: node-1 Allowlist grant for the workload entry (read-only, exact path): "secrets": {"policy": "allow", "read": ["/tenant-a/volumes/weights"]} A subtree grant would cover every volume beneath it, so this names one path. ``` The store write is **create-only**. A path that already holds a value is refused: a volume's key and its ciphertext are one unit, so replacing the key at a path some volume already uses strands that volume rather than rotating anything. Choose another path. ### Keep the escrow file CDS keeps secrets in process memory and nowhere else. **A CDS restart makes every volume in the cluster unopenable until its key is written back**, and the escrow file is what you write it back from: ```bash c8s secrets put /tenant-a/volumes/weights \ --from-file ./weights.escrow.json \ --url https://cds.example \ --operator-key ./operator.key ``` **The escrow file is the only copy of the key outside the CDS process.** Lose it and restart CDS, and the ciphertext is unrecoverable — there is no other copy, no versioning, and no recovery path. Its compromise is equivalent to handing over the plaintext, permanently. Store escrow files somewhere durable and access-controlled. ### Attach the image to a node The image is ciphertext. Copy it to the node by any means, including through the untrusted host — that the host holds the bytes is the premise, not a compromise of it. Attach it as a **raw block device** whose virtio serial is `c8s-vol-`. `volumed` finds it by reading `serial` under `/sys/block`, so no udev rules are needed. A confidential node has no persistent writable storage — the root overlay is reformatted on every boot — so a volume has to be its own device rather than a file on the node's filesystem. The serial is a **selector, not a trust input**. The host chooses it and answers the query per read. Pointing a pod at the wrong device fails closed: the wrong key produces noise, and verity refuses it. Two devices claiming the same serial are refused outright rather than resolved by scan order. Because the device lives on one node, the pod must be scheduled there. `create` emits the matching `nodeSelector`. ### Grant the workload the key path Release is gated on the workload entry's `secrets` grant in the [allowlist](/docs/c8s/concepts/image-allowlist): ```json "secrets": { "policy": "allow", "read": ["/tenant-a/volumes/weights"] } ``` **Name the exact path, not a subtree.** `/tenant-a/volumes/**` grants every volume beneath it, and the annotation naming which volume to open is host-written. `create` prints an exact-path grant for this reason. `read` only. A volume is mounted read-only, so a write grant says nothing about whether a workload may see the plaintext. ### Request the volume from the pod A pod names its volumes in an annotation: ```yaml metadata: annotations: confidential.ai/cw: llama-infer confidential.ai/c8s-volumes: "weights=/tenant-a/volumes/weights" confidential.ai/c8s-volume-dir: "/models" # optional ``` Each entry is `NAME=/store/path`, comma-separated. `NAME` selects the node's device by its `c8s-vol-` serial and names the directory the plaintext appears in under the volume dir — above, `/models/weights`. Without `confidential.ai/c8s-volume-dir` the default is `/run/c8s/volumes`. For a pod carrying `confidential.ai/cw`, the webhook then injects: - a **`c8s-volume` native sidecar**, ordered after `c8s-cert-wait` — it authenticates with the leaf that sidecar writes; - per volume, a default-medium `emptyDir` named `c8s-volume-`, mounted into every container **read-only** with `mountPropagation: HostToContainer`. Both names are reserved. A pod may not declare its own container called `c8s-volume`, and a volume it pre-declares under the `c8s-volume-` prefix must be a default-medium `emptyDir` or be omitted entirely — a `hostPath`, a PVC, or a memory-backed `emptyDir` is rejected at admission. See [Reserved containers and volumes](/docs/c8s/concepts/kata-runtime#reserved-containers-and-volumes). Verify the mount landed: ```bash kubectl exec -n -- ls /models/weights ``` Expect the volume's files. An empty directory means the mount has not landed yet — check the `c8s-volume` sidecar's logs. ## See also - [Encrypted volumes](/docs/c8s/concepts/encrypted-volumes) — the artifact, the mount decision, and the threat table. - [CLI reference → `c8s volume`](/docs/c8s/reference/cli#c8s-volume) — every flag. - [Workload annotations](/docs/c8s/reference/workload-annotations) — `confidential.ai/c8s-volumes` and the reserved names. --- # Deploy NVIDIA Dynamo This guide deploys **NVIDIA Dynamo** — the distributed LLM inference framework — onto an existing **c8s** cluster as a set of confidential workloads. Dynamo splits serving into components: an OpenAI-compatible **frontend** that routes requests, **workers** that run the engine, and a discovery plane that connects them. That makes it a realistic test of running a *multi-component* serving system inside the Trusted Execution Environment (TEE). You'll allowlist every image, give the frontend and worker identities issued by the **CDS** (Certificate Distribution Service, the cluster's attested trust root), and chat with the deployment from your laptop over a verified, sealed channel. Every image is public — there is nothing to build or push. **CPU cluster, mock engine.** Dynamo's real engine backends (vLLM, SGLang, TensorRT-LLM) ship as CUDA images and need GPUs — and that AKS cluster is CPU-only. So it runs Dynamo's own GPU-free **mocker** engine (`dynamo.mocker`, the upstream-supported way to exercise the frontend, router, and discovery without GPUs). The mocker downloads only the model's *tokenizer* — a few MB — and never loads weights, so it runs comfortably on the smallest confidential node sizes. Every c8s property you set up — digest enforcement, workload identity, the attested front door — transfers unchanged to a GPU-backed deployment: swap the worker's image and command, keep the annotations. ## Prerequisites This guide assumes the cluster from [Your first confidential cluster](/docs/c8s/tutorials/first-confidential-cluster): single-node AKS in **node-as-CVM** mode, with `operator.pub` pinned at install. On a different c8s cluster everything here transfers — adjust the install flags in Step 5 and how you expose the front door. You'll also need: - The `c8s` CLI, `kubectl`, and `crane` on your laptop (all already there if you followed that tutorial). - The **operator private key** (`operator.key`) — [allowlist writes](/docs/c8s/concepts/image-allowlist#authorizing-mutations) are signed with it. - For the final step: **Node ≥ 20**, plus the LB **launch measurement** and your cluster's **mesh CA** PEM, supplied out of band — see [Consumer verification](/docs/c8s/concepts/verification-model). ## Step by step ### 1. Deploy the discovery and event planes Plain-Deployment Dynamo (no Dynamo operator) uses **etcd** for endpoint discovery, and its KV-event plane defaults to **NATS** under etcd discovery — so deploy both, single-replica. This etcd is Dynamo's own registry, deployed like any app — it is **not** the Kubernetes control plane's etcd, which stays untouched (and on AKS isn't even reachable). These are infrastructure, not confidential workloads: they carry routing metadata, and under node-as-CVM they already run inside the node's confidential boundary. Don't annotate them — c8s identities belong on the components that terminate model traffic. ```yaml # dynamo-infra.yaml apiVersion: apps/v1 kind: Deployment metadata: name: etcd namespace: workloads spec: replicas: 1 selector: matchLabels: app: etcd template: metadata: labels: app: etcd spec: containers: - name: etcd image: quay.io/coreos/etcd:v3.6.13 command: - /usr/local/bin/etcd - --listen-client-urls=http://0.0.0.0:2379 - --advertise-client-urls=http://etcd.workloads.svc:2379 ports: - containerPort: 2379 --- apiVersion: v1 kind: Service metadata: name: etcd namespace: workloads spec: selector: app: etcd ports: - port: 2379 --- apiVersion: apps/v1 kind: Deployment metadata: name: nats namespace: workloads spec: replicas: 1 selector: matchLabels: app: nats template: metadata: labels: app: nats spec: containers: - name: nats image: nats:2.14.3-alpine ports: - containerPort: 4222 --- apiVersion: v1 kind: Service metadata: name: nats namespace: workloads spec: selector: app: nats ports: - port: 4222 ``` ```bash kubectl create namespace workloads # skip if it exists from another tutorial kubectl apply -f dynamo-infra.yaml ``` The pods won't start yet — their images aren't on the allowlist. That's enforcement working; Step 3 fixes it. **Scaling past one node changes the picture.** The RA-TLS mesh wraps **pod-IP** traffic — which covers Dynamo's frontend→worker request plane, since workers register their pod IPs in etcd. But etcd and NATS are dialed here through Service VIPs, which the mesh cannot intercept: on a single node that traffic never leaves the CVM; across nodes it would cross the wire unmeshed. Keep the graph on one node, or move inter-node infra hops onto meshed pod-IP addressing before scaling out. ### 2. Deploy the frontend and worker as confidential workloads Both components run from one **public** image: `dynamo-frontend` is NVIDIA's non-CUDA Dynamo image, and it carries the whole `ai-dynamo` Python package — `dynamo.frontend` *and* `dynamo.mocker` — so the two Deployments differ only in `command:` (which overrides the image's entrypoint). It pulls anonymously from `nvcr.io`. The `confidential.ai/cw` annotation opts each Deployment into c8s: the webhook injects the `c8s-cert` identity sidecar (a CDS-issued, TEE-bound certificate) plus a `c8s-cert-wait` init container that holds the app until that first certificate exists, and the operator mints a headless Service per workload id — `c8s-dynamo` is what the front door will dial in Step 5. The components find etcd and NATS via the env vars Dynamo reads (`ETCD_ENDPOINTS`, `NATS_SERVER`); requests then flow frontend → worker over Dynamo's TCP request plane. The `--model-path` names the tokenizer the mocker fetches — no weights are loaded. ```yaml # dynamo.yaml apiVersion: apps/v1 kind: Deployment metadata: name: dynamo-frontend namespace: workloads spec: replicas: 1 selector: matchLabels: app: dynamo-frontend template: metadata: labels: app: dynamo-frontend annotations: # workload id → headless Service c8s-dynamo confidential.ai/cw: dynamo spec: containers: - name: frontend image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1 command: - python3 - -m - dynamo.frontend - --http-port - "8000" ports: - containerPort: 8000 env: - name: ETCD_ENDPOINTS value: http://etcd.workloads.svc:2379 - name: NATS_SERVER value: nats://nats.workloads.svc:4222 --- apiVersion: apps/v1 kind: Deployment metadata: name: dynamo-worker namespace: workloads spec: replicas: 1 selector: matchLabels: app: dynamo-worker template: metadata: labels: app: dynamo-worker annotations: confidential.ai/cw: dynamo-worker spec: containers: - name: worker image: nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1 command: - python3 - -m - dynamo.mocker - --model-path - Qwen/Qwen3-0.6B env: - name: ETCD_ENDPOINTS value: http://etcd.workloads.svc:2379 - name: NATS_SERVER value: nats://nats.workloads.svc:4222 ``` ```bash kubectl apply -f dynamo.yaml ``` On a GPU cluster the worker's command would run `dynamo.vllm --model Qwen/Qwen3-0.6B` on a CUDA runtime image — the c8s wiring around it stays identical. (And if you'd rather serve from a minimal image you control, `pip install ai-dynamo==1.2.1` on `python:3.12-slim` provides the same entrypoints — build it, push it, and allowlist *that* digest instead.) ### 3. Allow the images Everything is deployed and nothing runs — check the events and you'll see c8s enforcement denying container creation at the node: ```bash kubectl get events -n workloads | grep allowlist # Warning Failed ... image not in allowlist: quay.io/coreos/etcd:v3.6.13 ``` Add the three digests, signed with your operator key. The chart publishes CDS's full `/allowlist` API through tls-lb, but the front door has no address yet on this cluster — you point it at Dynamo in Step 5 — so reach the CDS directly for now. It has no public ingress, so port-forward it: ```bash kubectl port-forward -n c8s-system svc/c8s-cds 8443:8443 & CDS_URL=https://localhost:8443 CDS_MEASUREMENT= DYNAMO_IMAGE=nvcr.io/nvidia/ai-dynamo/dynamo-frontend:1.2.1 c8s allowlist add "$(crane digest "$DYNAMO_IMAGE")" "$DYNAMO_IMAGE" \ --url "$CDS_URL" \ --measurements "$CDS_MEASUREMENT" \ --operator-key operator.key ETCD_IMAGE=quay.io/coreos/etcd:v3.6.13 c8s allowlist add "$(crane digest "$ETCD_IMAGE")" "$ETCD_IMAGE" \ --url "$CDS_URL" \ --measurements "$CDS_MEASUREMENT" \ --operator-key operator.key NATS_IMAGE=nats:2.14.3-alpine c8s allowlist add "$(crane digest "$NATS_IMAGE")" "$NATS_IMAGE" \ --url "$CDS_URL" \ --measurements "$CDS_MEASUREMENT" \ --operator-key operator.key ``` One port-forward covers it: the CLI verifies the CDS's RA-TLS attestation **in-process** before any write, so nothing else has to be reachable. `--measurements` is the CDS launch digest — on a node-as-CVM cluster, the node's own measurement. Leave it out and the CLI warns `no --measurements set; accepting any attested endpoint build (UNSAFE)`, which is survivable on a tutorial cluster and not in production: an unpinned write can land on a rogue CDS. Once tls-lb has a public address (Step 5) you can drop the port-forward entirely and pass `--url "https://$LB_ADDRESS" --measurements ` instead: the CLI verifies the front door's discovery attestation and the write is proxied on to CDS. The operator private key stays on your laptop either way — it only ever signs a short-lived token bound to that one request body. The enforcement plugins poll the CDS-served list, so the kubelet's next retry succeeds — the Dynamo image is large, so its first pull takes a few minutes: ```bash kubectl get pods -n workloads -w # etcd-… 1/1 Running # nats-… 1/1 Running # dynamo-frontend-… 2/2 Running ← app + injected c8s-cert sidecar # dynamo-worker-… 2/2 Running ``` Both serving components now hold TEE-bound identities. Mirror that into `kubectl` with two optional `ConfidentialWorkload` CRs (a status view — injection works without them): ```yaml # dynamo-cwl.yaml apiVersion: confidential.ai/v1alpha2 kind: ConfidentialWorkload metadata: name: dynamo namespace: workloads spec: workloadRef: kind: Deployment name: dynamo-frontend --- apiVersion: confidential.ai/v1alpha2 kind: ConfidentialWorkload metadata: name: dynamo-worker namespace: workloads spec: workloadRef: kind: Deployment name: dynamo-worker ``` ```bash kubectl apply -f dynamo-cwl.yaml kubectl get cwl -n workloads # NAME WORKLOAD ATTESTED TOTAL # dynamo dynamo-frontend 1 1 # dynamo-worker dynamo-worker 1 1 ``` ### 4. Smoke-test the serving graph Before wiring the front door, confirm discovery worked end to end: the worker registered in etcd, and the frontend found it. `kubectl port-forward` reaches the pod over the kubelet's own path, so it works against confidential pods: ```bash kubectl port-forward -n workloads deploy/dynamo-frontend 8000:8000 & curl -s localhost:8000/v1/models # {"object":"list","data":[{"id":"Qwen/Qwen3-0.6B", ...}]} curl -s localhost:8000/v1/chat/completions \ -H 'content-type: application/json' \ -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "hello"}], "max_tokens": 32}' # {"id":"...","choices":[{"message":{"role":"assistant", ... kill %% # stop this port-forward (the most recent background job) ``` The completion text is **mock output** — the mocker simulates engine scheduling and token timing, not language. What it proves is the part this guide is about: requests traverse frontend → router → worker across attested, identity-bearing components. ### 5. Point the front door at the frontend External clients come in through the chart's front door — **tls-lb (public TLS + over-encryption) → your frontend**, dialed over the node mesh's attested mTLS. Point it there with `--workload-ref` + `--upstream`: the ref's `:8000` (the frontend's `--http-port`) derives the upstream `c8s-dynamo.workloads.svc.cluster.local:8000` — the minted headless Service, whose DNS returns pod IPs the node mesh wraps in attested mTLS; a Service VIP it cannot. `service.type: LoadBalancer` gives the front door a public IP on AKS: ```yaml # values.yaml tlsLb: service: type: LoadBalancer ``` Apply it by re-running the `c8s install` from the Azure tutorial with the two flags and `-f values.yaml` added (the front door has one catch-all upstream — this repoints it if another tutorial's workload was wired first). `--workload-ref` also stamps `confidential.ai/cw: dynamo` onto the Deployment's pod template — a no-op here, since Step 2's manifest already carries it: ```bash c8s install \ --single-node \ --cvm-mode aks \ --operator-keys operator.pub \ --workload-ref dynamo=workloads/deployment/dynamo-frontend:8000 \ --upstream dynamo \ -f values.yaml ``` Repeat the flags from the original install: `--cvm-mode` is required on every run, and the values the flags derive take precedence over anything a `-f` file sets on the same keys. Then grab the public address Azure assigns — it's `` for a minute or so: ```bash kubectl get svc -n c8s-system c8s-tls-lb -w # NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) # c8s-tls-lb LoadBalancer 10.0.… 20.… 443:…/TCP LB_ADDRESS= ``` **Not on the Azure tutorial's cluster?** Use the flags you originally installed with, and expose `c8s-tls-lb` however your environment does it. With no cloud load balancer, set `tlsLb.service.type: NodePort` in `values.yaml` and reach the front door on the node: ```bash NODE_PORT=$(kubectl get svc -n c8s-system c8s-tls-lb -o jsonpath='{.spec.ports[0].nodePort}') LB_ADDRESS=:$NODE_PORT ``` On RKE2, tls-lb also binds the node's `:443` directly (`tlsLb.hostPort.enabled` defaults to `true`); if RKE2's bundled `rke2-ingress-nginx` already owns `:443`, set `tlsLb.hostPort.enabled: false` and reach it through the NodePort. `c8s install` catches that collision up front only on a default install — once you pass `-f`, the tls-lb pod just sits `Pending`. Then set `LB_ADDRESS` to whatever address terminates at tls-lb. ### 6. Verify the enclave and chat From your laptop, prove the front door is a genuine TEE in *your* cluster before the first prompt leaves your machine. The CLI check pins the LB launch measurement: ```bash c8s verify "https://$LB_ADDRESS" --kind lb --measurements ``` ```text ✓ VERIFIED (attestation-go backend) source: discovery document https://20.…/v1/discovery platform: az-snp measurement: 9f2c… … ``` Exit code 0 means verified; 2 is a verification or policy failure and 3 means no evidence could be fetched, so the same command works unchanged as a CI gate. See [the verification CLI](/docs/c8s/how-to/verify-a-deployment) for the rest of its checks. Then chat over the sealed channel with the [c8s-verify](/docs/c8s/concepts/verification-model) JavaScript client. Set it up first — see [its installation](/docs/c8s/reference/c8s-verify-js#installation) — then run `node chat.mjs`. It re-verifies the attestation, checks the cert chains to your pinned mesh CA, and opens the post-quantum over-encrypted tunnel. Keep these requests non-streaming: the tunnel seals one whole request envelope and one whole response envelope, so tokens can't arrive incrementally through it. (Streaming does work over tls-lb's ordinary route, which forwards Server-Sent Events straight through instead of buffering them — but that gives you TLS to the front door, not a channel sealed to the enclave.) ```js // chat.mjs import { C8sClient } from "c8s-verify"; import { readFileSync } from "node:fs"; const client = new C8sClient({ baseUrl: `https://${process.env.LB_ADDRESS}`, measurements: [process.env.LB_MEASUREMENT], // pinned LB launch digest meshCaPem: readFileSync("mesh-ca.pem", "utf8"), // pinned cluster anchor }); const session = await client.connect(); // throws C8sVerifyError on any failure const res = await session.fetch("/v1/chat/completions", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "Qwen/Qwen3-0.6B", messages: [{ role: "user", content: "hello from outside the enclave" }], max_tokens: 32, }), }); console.log(JSON.parse(res.text()).choices[0].message.content); ``` ```bash LB_ADDRESS=$LB_ADDRESS \ LB_MEASUREMENT= \ NODE_TLS_REJECT_UNAUTHORIZED=0 \ node chat.mjs ``` Why `NODE_TLS_REJECT_UNAUTHORIZED=0` is fine *here*: you dialed a bare IP, and the LB's outer TLS cert is its CDS-issued one, which web PKI doesn't know. In this design the outer TLS layer is **untrusted transport by definition** — trust comes from the attestation evidence, the pinned measurement, and the pinned mesh CA, and every request is sealed inside the over-encrypted channel, so a hostile TLS terminator sees only ciphertext. The env var relaxes only the outer web-PKI check, for this process. (`c8s verify` above does the equivalent internally.) For a real deployment you'd put a public certificate on the LB via `tlsLb.publicTLS` and drop the env var. For a full interactive client — REPL loop, failure drills, what each check means — see [Verified chat over confidential vLLM](/docs/c8s/tutorials/verified-chat); the session API is identical. ## What you've proven A multi-component serving system — not just a single pod — running with the threat model intact: 1. **Every image was admitted by digest** — Dynamo components and infrastructure alike; the node refused each container until its digest was allowlisted. 2. **Each serving component holds a hardware-rooted identity** — the frontend and worker carry CDS-issued certificates bound to TEE attestation, visible in `kubectl get cwl`. 3. **The request path stays inside the trust boundary** — client → LB enclave over the over-encrypted channel, LB → frontend → worker over attested pod-IP hops; on this single-node cluster the whole graph shares one confidential boundary. 4. **The client verified before it trusted** — pinned launch measurement, pinned mesh CA, fail-closed. ## Where to go next - **The Dynamo Kubernetes Platform** — running the Dynamo operator instead of plain Deployments? The same two ingredients apply: allowlist the platform images, and put `confidential.ai/cw` on the pods via `spec.services..extraPodMetadata.annotations` (`v1alpha1`) or `spec.components[].podTemplate.metadata.annotations` (`v1beta1`) in a `DynamoGraphDeployment`. - **Real engines need GPUs** — on a GPU-equipped c8s cluster they just run: node-as-CVM nodes schedule GPUs as normal Kubernetes resources, and under Kata, GPU pods become confidential VMs with the GPU passed through — see [Kata containers](/docs/c8s/concepts/kata-runtime) and [Limitations](/docs/c8s/concepts/limitations) for the current gaps. The swap is the worker's image, command, and a GPU resource limit — not the c8s wiring. - **Classical model serving** — [Serving a confidential model with KServe](/docs/c8s/how-to/deploy-kserve) runs the same pattern with KServe's `InferenceService`. - **Automate the digests** — wire `c8s allowlist add` into CI: [Automating the allowlist](/docs/c8s/how-to/manage-the-allowlist#automating-the-allowlist). --- # Serve a model with KServe This guide puts **KServe** — the standard Kubernetes model-serving layer — on top of an existing **c8s** cluster, so a deployed model and every request to it stay inside the Trusted Execution Environment (TEE). You'll install the KServe control plane with every image on the [allowlist](/docs/c8s/concepts/image-allowlist), deploy a scikit-learn `InferenceService` as a confidential workload, and call it from your laptop only after cryptographically verifying the enclave — the host, the cloud operator, and any TLS-terminating proxy in between see only ciphertext. KServe keeps its whole API surface — `InferenceService`, model formats, autoscaling — and c8s adds the confidential substrate underneath: digest enforcement at container creation, an identity issued by the **CDS** (Certificate Distribution Service, the cluster's attested trust root) bound to the predictor, and an attested front door. **Why Standard mode?** KServe's Knative (serverless) mode requires Knative and Istio. c8s already runs its own pod-to-pod mesh — **RA-TLS** (Remote-Attestation TLS) rooted in hardware attestation — and a second service mesh double-intercepts the same traffic and injects un-attested proxies into the confidential path (see [Limitations](/docs/c8s/concepts/limitations)). Standard mode runs each predictor as a plain Deployment: no Knative, no Istio, nothing between your pods and the attested mesh. ## Prerequisites This guide assumes the cluster from [Your first confidential cluster](/docs/c8s/tutorials/first-confidential-cluster): single-node AKS in **node-as-CVM** mode, with `operator.pub` pinned at install. On a different c8s cluster everything here transfers — adjust the install flags in Step 4 and how you expose the front door. You'll also need: - The `c8s` CLI, `kubectl`, `helm`, and `crane` on your laptop (all already there if you followed that tutorial). - The **operator private key** (`operator.key`) — [allowlist writes](/docs/c8s/concepts/image-allowlist#authorizing-mutations) are signed with it. - For the final step: **Node ≥ 20**, plus the LB **launch measurement** and your cluster's **mesh CA** PEM, supplied out of band — see [Consumer verification](/docs/c8s/concepts/verification-model). ## Step by step ### 1. Allowlist the KServe platform images On a c8s cluster, **nothing runs unless its image digest is on the allowlist** — and that includes control planes, not just your models. This is the point, not a hurdle: the same enforcement that will gate your model server also gates cert-manager and the KServe controller, so a compromised upstream image can't quietly land next to your weights. Reach the CDS, then add the digests of everything this guide runs: cert-manager's three components, the KServe controller pod's two containers, and the two images every sklearn predictor pod uses — `storage-initializer` (the init container that fetches the model) and `sklearnserver` (the model server). The chart publishes CDS's full `/allowlist` API through tls-lb by default, but that front door has no address yet on this cluster — you point it at the predictor in Step 4 — so talk to the CDS directly. It has no public ingress, so port-forward it: ```bash kubectl port-forward -n c8s-system svc/c8s-cds 8443:8443 & CDS_URL=https://localhost:8443 CDS_MEASUREMENT= for IMAGE in \ "quay.io/jetstack/cert-manager-controller:v1.20.3" \ "quay.io/jetstack/cert-manager-webhook:v1.20.3" \ "quay.io/jetstack/cert-manager-cainjector:v1.20.3" \ "kserve/kserve-controller:v0.19.0" \ "quay.io/brancz/kube-rbac-proxy:v0.18.0" \ "kserve/storage-initializer:v0.19.0" \ "kserve/sklearnserver:v0.19.0"; do c8s allowlist add "$(crane digest "$IMAGE")" "$IMAGE" \ --url "$CDS_URL" \ --measurements "$CDS_MEASUREMENT" \ --operator-key operator.key done ``` Each `add` prints `added sha256:…`. That single port-forward is all the CLI needs — it verifies the CDS's RA-TLS attestation **in-process** before every write. `--measurements` is the CDS launch digest, which on a node-as-CVM cluster is the node's own measurement; omit it and the CLI warns `no --measurements set; accepting any attested endpoint build (UNSAFE)`. Fine for a test cluster cluster, not for production — an unpinned write can land on a rogue CDS. After Step 4 gives tls-lb a public address, the port-forward becomes optional: point `--url` at `https://$LB_ADDRESS` with `--measurements ` and the CLI verifies the front door's discovery attestation and proxies the write through to CDS. Either way the operator private key stays on your laptop — it signs a short-lived token bound to that one request body. `crane digest` reads the digest a tag currently points to — for these multi-arch images that's the index digest, the one containerd records when a node pulls the tag. If a pod later still reports `image not in allowlist`, compare against the digest the node actually resolved: `kubectl get pod -o jsonpath='{.status.containerStatuses[*].imageID}'` — see [Getting an image's digest](/docs/c8s/how-to/manage-the-allowlist#getting-an-images-digest). ### 2. Install cert-manager and KServe cert-manager issues the serving certificates for KServe's own admission webhooks — control-plane plumbing, separate from the CDS-issued workload identities: ```bash kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.20.3/cert-manager.yaml kubectl wait --for=condition=Available -n cert-manager deploy --all --timeout=180s ``` Then KServe itself: the CRDs, the controller in **Standard** deployment mode, and the `ClusterServingRuntime` catalog (runtime images pull only when an `InferenceService` uses them, so allowlisting `sklearnserver` alone was enough). `disableIngressCreation` stops the controller from minting Ingress objects — the attested front door replaces that role in Step 4: ```bash helm install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd --version v0.19.0 helm install kserve oci://ghcr.io/kserve/charts/kserve-resources --version v0.19.0 \ --namespace kserve \ --create-namespace \ --set kserve.controller.deploymentMode=Standard \ --set kserve.controller.gateway.disableIngressCreation=true # Wait for the controller before applying the runtimes: the ClusterServingRuntimes go through # KServe's validating webhook, which returns "no endpoints available" until its pod is Ready. kubectl wait --for=condition=Available -n kserve deploy/kserve-controller-manager --timeout=180s kubectl apply --server-side -f https://github.com/kserve/kserve/releases/download/v0.19.0/kserve-cluster-resources.yaml # This manifest also contains LLMInferenceServiceConfig objects (a KServe LLM CRD the v0.19.0 # kserve-crd chart doesn't install); the "no matches for kind" lines printed for those are # harmless — the sklearn runtime this guide serves is created. ``` Confirm both control planes are up before moving on: ```bash kubectl get pods -n cert-manager # three pods Running — allowlisted before the kubelet ever asked kubectl get pods -n kserve # kserve-controller-manager-… 2/2 Running ``` ### 3. Deploy a confidential InferenceService From here it looks like any KServe cluster — one `InferenceService`, serving the classic scikit-learn iris classifier from a public bucket. The only c8s-specific line is the `confidential.ai/cw: iris` annotation: predictor-level annotations land on the pods KServe creates, so the c8s webhook injects the `c8s-cert` identity sidecar (and the `c8s-cert-wait` gate in front of it), and the value `iris` is the **workload id** from which the operator mints the headless Service `c8s-iris`: ```yaml # iris.yaml apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: iris namespace: workloads spec: predictor: annotations: confidential.ai/cw: iris # workload id → headless Service c8s-iris model: modelFormat: name: sklearn storageUri: gs://kfserving-examples/models/sklearn/1.0/model ``` ```bash kubectl create namespace workloads # skip if it exists from another tutorial kubectl apply -f iris.yaml kubectl get inferenceservice -n workloads iris -w # NAME URL READY # iris … True ``` The model downloads *inside* the TEE: `storage-initializer` runs as an init container in the predictor pod, so the artifact goes encrypted-in-memory from its first byte on the cluster — the host never sees it unwrapped. While it starts, look at what c8s injected: ```bash kubectl get pod -n workloads -l serving.kserve.io/inferenceservice=iris \ -o jsonpath='{range .items[0].spec.initContainers[*]}{.name}{"\n"}{end}' # c8s-cert ← injected identity sidecar (holds the CDS-issued cert) # c8s-cert-wait ← injected gate: blocks until that cert exists # storage-initializer ← KServe's model fetcher kubectl get svc -n workloads c8s-iris # NAME TYPE CLUSTER-IP … # c8s-iris ClusterIP None ← headless: resolves to pod IPs ``` Headless matters: DNS returns pod IPs, which the RA-TLS mesh intercepts and wraps in attested mTLS — a Service VIP it cannot. Optionally, mirror attestation status into a `ConfidentialWorkload` CR so `kubectl` can show it (a status view — injection works with or without it): ```yaml # iris-cwl.yaml apiVersion: confidential.ai/v1alpha2 kind: ConfidentialWorkload metadata: name: iris # must equal the confidential.ai/cw value namespace: workloads spec: workloadRef: kind: Deployment name: iris-predictor ``` ```bash kubectl apply -f iris-cwl.yaml kubectl get cwl -n workloads # NAME WORKLOAD ATTESTED TOTAL # iris iris-predictor 1 1 ``` **In-cluster clients: dial `c8s-iris`, not the KServe Service.** KServe also creates an ordinary ClusterIP Service (`iris-predictor`), but the mesh cannot intercept Service-VIP traffic — so c8s **drops** VIP flows to confidential pods rather than let them arrive in plaintext. The headless `c8s-iris` resolves to pod IPs, which the mesh wraps in attested mTLS. (Kubelet probes and the front-door path below are unaffected.) ### 4. Point the front door at the predictor External clients reach the model through the chart's front door — **tls-lb (public TLS + over-encryption) → your predictor**, dialed over the node mesh's attested mTLS. KServe's predictor serves on `8080`, so point the front door there with `--workload-ref` + `--upstream`: the ref's `:8080` derives the upstream `c8s-iris.workloads.svc.cluster.local:8080` — the mesh-wrapped headless Service, so the hop is attested, not a Service VIP the mesh can't intercept: ```yaml # values.yaml tlsLb: service: type: LoadBalancer ``` `service.type: LoadBalancer` gives the front door a public IP on AKS, and it's the only value you need — the tls-lb attestation sidecar that serves the `c8s-verify/v1` endpoints is on by default. Apply it all by re-running the `c8s install` from the Azure tutorial with the two flags and `-f values.yaml` added (the front door has one catch-all upstream — this repoints it if another tutorial's workload was wired first). The ref names the Deployment KServe manages; the `confidential.ai/cw: iris` stamp it applies is a no-op here, since Step 3's predictor annotation already landed it on the pods: ```bash c8s install \ --single-node \ --cvm-mode aks \ --operator-keys operator.pub \ --workload-ref iris=workloads/deployment/iris-predictor:8080 \ --upstream iris \ -f values.yaml ``` Repeat the flags from the original install: `--cvm-mode` is required on every run, the install sets `cds.operatorKeys` only while `--operator-keys` is passed, and the values these flags derive win over anything a `-f` file sets on the same keys. tls-lb now dials the predictor's headless Service over the node mesh's attested mTLS and fronts it with public TLS. A request from outside is decrypted **inside the LB enclave** and forwarded over the mesh to KServe's v1 REST endpoint. Grab the public address Azure assigns — it's `` for a minute or so: ```bash kubectl get svc -n c8s-system c8s-tls-lb -w # NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) # c8s-tls-lb LoadBalancer 10.0.… 20.… 443:…/TCP LB_ADDRESS= ``` **Not on the Azure tutorial's cluster?** Use the flags you originally installed with, and expose `c8s-tls-lb` however your environment does it. With no cloud load balancer, set `tlsLb.service.type: NodePort` in `values.yaml` and reach the front door on the node: ```bash NODE_PORT=$(kubectl get svc -n c8s-system c8s-tls-lb -o jsonpath='{.spec.ports[0].nodePort}') LB_ADDRESS=:$NODE_PORT ``` On RKE2, tls-lb also binds the node's `:443` directly (`tlsLb.hostPort.enabled` defaults to `true`); if RKE2's bundled `rke2-ingress-nginx` already owns `:443`, set `tlsLb.hostPort.enabled: false` and reach it through the NodePort. `c8s install` catches that collision up front only on a default install — once you pass `-f`, the tls-lb pod just sits `Pending`. Then set `LB_ADDRESS` to whatever address terminates at tls-lb. ### 5. Verify the enclave and predict First, prove the front door is a genuine TEE running the image you expect — from your laptop, before any feature vector leaves it. The CLI check pins the LB launch measurement: ```bash c8s verify "https://$LB_ADDRESS" --kind lb --measurements ``` ```text ✓ VERIFIED (attestation-go backend) source: discovery document https://20.…/v1/discovery platform: az-snp measurement: 9f2c… … ``` Exit code 0 means verified; 2 is a verification or policy failure and 3 means the evidence couldn't be fetched — so the same command drops straight into CI. See [the verification CLI](/docs/c8s/how-to/verify-a-deployment) for its other checks. Then make a verified prediction with the [c8s-verify](/docs/c8s/concepts/verification-model) JavaScript client. Set it up first — see [its installation](/docs/c8s/reference/c8s-verify-js#installation) — before running `node predict.mjs` below. The client re-checks the attestation, confirms the LB belongs to *your* cluster (its cert chains to your pinned mesh CA), and opens the post-quantum over-encrypted channel; only then does the request go out — sealed end-to-end into the enclave: ```js // predict.mjs import { C8sClient } from "c8s-verify"; import { readFileSync } from "node:fs"; const client = new C8sClient({ baseUrl: `https://${process.env.LB_ADDRESS}`, measurements: [process.env.LB_MEASUREMENT], // pinned LB launch digest meshCaPem: readFileSync("mesh-ca.pem", "utf8"), // pinned cluster anchor }); const session = await client.connect(); // throws C8sVerifyError on any failure const res = await session.fetch("/v1/models/iris:predict", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ instances: [ [6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6], ], }), }); console.log(res.text()); ``` ```bash LB_ADDRESS=$LB_ADDRESS \ LB_MEASUREMENT= \ NODE_TLS_REJECT_UNAUTHORIZED=0 \ node predict.mjs # {"predictions":[1,1]} ``` `NODE_TLS_REJECT_UNAUTHORIZED=0` is fine *here*: you dialed a bare IP whose outer TLS cert is the LB's CDS-issued one, which web PKI doesn't know — and in this design the outer TLS layer is untrusted transport by definition. Trust comes from the attestation checks and the sealed channel, so the env var relaxes only the outer web-PKI check. For a real deployment you'd put a public certificate on the LB via `tlsLb.publicTLS` and drop it. Both flowers classify as class `1` (versicolor). The feature vectors traveled sealed to the LB enclave, crossed the cluster only over attested mTLS, and were scored by a model the host never saw. The full client walkthrough — including a failure drill where a wrong measurement refuses to connect — is in [Verified chat over confidential vLLM](/docs/c8s/tutorials/verified-chat). ## What you've proven By the time `{"predictions":[1,1]}` comes back, you've established — from an untrusted client, over an untrusted network — that: 1. every image in the serving path, control plane included, was **admitted by digest** before it could run; 2. the front door is a **genuine TEE** running the exact measured image you pinned, in **your** cluster (cert chained to your pinned mesh CA); 3. the model was fetched, loaded, and scored **inside the confidential boundary** — by [transitivity](/docs/c8s/concepts/verification-model#transitivity-of-trust), the attested LB only forwards over the RA-TLS mesh to the predictor behind it; 4. request and response crossed the infrastructure **sealed end-to-end** — the TLS terminator, the host, and the operator saw ciphertext. ## Where to go next - **Serve a different model format** — any KServe runtime works the same way: allowlist the runtime image's digest, keep the `confidential.ai/cw` annotation on the predictor. See [the allowlist](/docs/c8s/how-to/manage-the-allowlist). - **LLM serving** — [Verified chat over confidential vLLM](/docs/c8s/tutorials/verified-chat) runs the same pattern with an OpenAI-compatible engine and a chat client, and [NVIDIA Dynamo](/docs/c8s/how-to/deploy-dynamo) shows a multi-component serving graph. - **Automate the digests** — wire `c8s allowlist add` into the pipeline that builds your model images: [Automating the allowlist](/docs/c8s/how-to/manage-the-allowlist#automating-the-allowlist). - **Pin for production** — set `cds.measurements` and per-cluster operator keys before real traffic: [threat model](/docs/c8s/concepts/threat-model). --- # Install c8s This guide installs **c8s** onto an **existing** Kubernetes cluster already running on confidential hardware. No cluster yet? Start with [Provision a confidential cluster on Azure](/docs/c8s/how-to/provision-azure-aks), then come back here. Never done any of this before? The [first confidential cluster tutorial](/docs/c8s/tutorials/first-confidential-cluster) walks the whole path in one sitting. ## Prerequisites - A Kubernetes cluster (k8s or RKE2) whose nodes run on confidential hardware — **AMD SEV-SNP** (Genoa / Milan / Turin) or **Intel TDX**. For the confidential pod runtime you additionally need a bare-metal-class host that allows nested VMs — see [pod-as-CVM vs node-as-CVM](/docs/c8s/concepts/trust-boundaries). - **Node kernels new enough for the TEE**: Linux ≥ 6.11 for SEV-SNP, ≥ 6.16 for TDX. - `kubectl` configured for the target cluster. - `helm` on `PATH` (the chart is embedded in the binary and installed via Helm). - `crane` on `PATH` — needed for the default `--resolve-digests=true`, which pins each component image to its registry digest. - The `c8s` CLI, built from source with `make install` (see below).
Building the c8s CLI (make, Go, GOPATH) The source repo is public and the build needs **Go 1.26+**. `make install` builds for your host platform — Linux and macOS both work: ```bash git clone https://github.com/confidential-dot-ai/c8s cd c8s make install ``` `make install` runs `go install`, which drops the binary in `$(go env GOBIN)`, or `$(go env GOPATH)/bin` (default `~/go/bin`) when `GOBIN` is unset — make sure that directory is on your `PATH`. You'll also need `make` itself; it's not preinstalled on stock Azure images (`sudo apt install make` on Debian/Ubuntu).
Pulling from a private mirror (crane login, --image-pull-secret) The stock c8s component images are public, so the default install needs no registry credentials. You need both of the following only when you re-host the images on a registry that requires authentication, or when your own workload images do: - **`crane` runs locally** during `c8s install` and authenticates from your `~/.docker/config.json`, so log in on the box first. Missing it surfaces as `crane digest ...: UNAUTHORIZED: authentication required`. ```bash crane auth login -u --password-stdin ``` Alternatively skip digest resolution with `--resolve-digests=false` and supply digests via `-f`. - **`--image-pull-secret `** is the separate cluster-side credential: the name of an existing `kubernetes.io/dockerconfigjson` Secret in the release namespace, which the chart appends to every component's `imagePullSecrets`. The install never creates it, and fails fast if it is missing or has the wrong type. Neither credential reaches a Kata guest: under `--cvm-mode=pod` a workload image's *layers* are fetched **inside** the guest, and that pull is anonymous. A pod-as-CVM workload image must be pullable without credentials.
**Which TEEs?** The platform — CDS, attestation verification, the RA-TLS mesh, the allowlist, and the **node-as-CVM** shape — runs on both **AMD SEV-SNP** and **Intel TDX**. `--cvm-mode` (deployment shape) and `--hardware-platform` (CPU TEE) are **orthogonal**, and every combination is accepted — including `--cvm-mode=aks --hardware-platform=tdx`, the Azure vTPM TDX shape. A cluster picks **one** CPU TEE at install time (`sev-snp` by default; under `--cvm-mode=pod`, `tdx` renders the `kata-qemu-tdx` RuntimeClass instead of `kata-qemu-snp`) — there are no mixed-TEE clusters. ## Quickstart: a single-node cluster For a single-node cluster, pass `--single-node`. This clears the dedicated-CDS node selector and taint toleration so **every node is CDS-eligible** — there is no separate `role=cds` node to label or schedule. (It's about node count, not the number of pod CVMs: a multi-node cluster running a single Kata pod still wants a dedicated CDS node, so leave it off there.) ### Create the operator credential Writes to the [image allowlist](/docs/c8s/concepts/image-allowlist) are authorized by an operator EC key whose **public** half the install pins into the CDS. Generate the pair once; the private key stays with you (vault/HSM): ```bash openssl ecparam -name prime256v1 -genkey -noout -out operator.key openssl ec -in operator.key -pubout -out operator.pub ``` Installing without `--operator-keys` disables allowlist writes — nobody can allowlist a workload image — so the installer refuses the default path unless you acknowledge with `--force`. (Supplying any `-f` values file also clears the guard: those files own `cds.operatorKeys`.) ### Install the default shape (node-as-CVM) Start with the node-as-CVM shape to confirm the control plane is healthy. Here the nodes are themselves confidential VMs and pods run as ordinary processes; there is no per-pod confidentiality until you select the pod runtime with `--cvm-mode=pod`. `--cvm-mode` is required — there is no CLI default, so state the shape explicitly: ```bash c8s install --single-node --cvm-mode=node \ --operator-keys operator.pub \ --measurements ``` `` is the hex launch measurement of the node image you booted, published in its `manifest.json`. It pins **both** internal trust boundaries in this one install (`cds.measurements` and `ratlsMesh.measurements`), so the mesh is pinned from first boot. Leave it off only for a throwaway cluster — see the pinning callout below. `node` is the cloud-agnostic node-as-CVM shape, for confidential VMs that expose the TEE device directly. Two managed platforms need a different device shape instead: `--cvm-mode=gke` for GKE's managed confidential VMs, and `--cvm-mode=aks` for any Azure confidential VM, where the report comes from the vTPM at `/dev/tpm0` rather than a guest device: ```bash c8s install --single-node --cvm-mode=aks --operator-keys operator.pub --measurements ``` ### Or install pod-as-CVM (`--cvm-mode=pod`) To make each workload pod its own confidential VM, use `--cvm-mode=pod`. This installs the Kata runtime stack, injects kata RuntimeClasses into workload pods, rejects non-kata pods, and moves the security services into the guest image. ```bash c8s install --single-node --cvm-mode=pod --operator-keys operator.pub ``` On Intel TDX hosts, add `--hardware-platform=tdx` (the default is `sev-snp`): ```bash c8s install --single-node --cvm-mode=pod --hardware-platform=tdx --operator-keys operator.pub ``` `--measurements` is **rejected** under `--cvm-mode=pod`: the peers CDS talks to here are per-pod Kata guests measured against the `kata-guest-base` image, not the node image, so a node measurement would reject every real peer. Pin the guest measurement through `-f` instead (`cds.measurements`). ### Wait for readiness The installer passes `helm --wait` by default, so the command returns only once the release is ready. Confirm the components are running: ```bash kubectl get pods -n c8s-system ``` `helm --wait` makes `c8s install` **block** until every component reports Ready, so the command can sit there for a few minutes — that's expected, not a hang. To watch progress while it waits, open a **second shell** on the box and tail the pods across *all* namespaces (`kata-deploy` and the CNI live outside `c8s-system`): ```bash kubectl get pods -A -w ``` The wait is bounded: `helm --wait` runs with a **5-minute timeout** (**10 minutes** for a `--cvm-mode=pod` install). If a pod stays wedged in `ImagePullBackOff` or `CrashLoopBackOff` past it (a missing or wrong image pull secret is the usual culprit), `c8s install` **fails** when the timeout elapses rather than waiting forever. The resources it already applied remain, so fix the cause and **re-run** `c8s install` — it's idempotent (`helm upgrade --install`) and completes once every pod is Ready. The last thing the installer prints is the pinning state it left behind. With `--measurements` it confirms `mesh pinned to --measurements`; without it, read this and act on it before production: ```text + tls-lb attestation sidecar enabled, but the mesh is UNPINNED (accepts any attested TEE). Pin it with the node image's launch measurement M (its manifest.json): reinstall with --measurements . Clients verify with the same M: c8s verify https:// --measurements . ``` ## Choosing the install shape Four flags decide the trust model and topology; the rest tune webhook-injected certificate defaults. Every flag, with its type and default, is in the [CLI reference](/docs/c8s/reference/cli#c8s-install). - **`--cvm-mode`** — the deployment shape, and the one decision you cannot defer. See [trust boundaries](/docs/c8s/concepts/trust-boundaries) before you pick. - **`--hardware-platform`** — the CPU TEE, `sev-snp` or `tdx`. Orthogonal to `--cvm-mode`. - **`--measurements`** — what pins the mesh. Read the callout below before leaving it off. - **`--operator-keys`** — who may write the [image allowlist](/docs/c8s/concepts/image-allowlist). Omitting it disables writes entirely. **Pin the mesh, or it accepts any attested peer.** `cds.measurements` is what the mesh, `nri-image-policy`, the operator, and tls-lb pin when they dial CDS, and what CDS admits to `/attest` and `/sign-csr`; `ratlsMesh.measurements` is what mesh peers pin on each other. With neither set, every one of those checks confirms the peer is a genuine TEE and stops there — any attested CVM is admitted. `--measurements ` sets both on the install itself, so there is no separate pinning step. [Obtain launch measurements](/docs/c8s/how-to/obtain-launch-measurements) explains how to get ``; the [threat model](/docs/c8s/concepts/threat-model#measurement-pinning-is-mandatory-for-production) explains why it matters.
How -f values files interact with the flags `c8s install` collects everything its flags imply into one computed values file and passes it to Helm as the **last** `-f`. Precedence is therefore: chart defaults, then your `-f` files in the order you list them, then the installer's computed values. A flag always wins on the keys it sets; your files supply everything else. Passing any `-f` also hands you ownership of the default-path checks, which the installer then skips: the `role=cds` node-label preflight, the tls-lb host-port check, the `--resolve-digests=false` operator-image existence check, the `--operator-keys` guard, and — under `--cvm-mode=pod` — the automatic TEE node labeling. Host distro autodetection is suppressed only if a values file actually sets `kata.distro` or `nriImagePolicy.distro`. One value the chart will not let you turn off: `nriImagePolicy.enabled=false` renders only under `--cvm-mode=pod` (the in-guest `policy-monitor` admits instead) or `--cvm-mode=node` (the node image bakes the plugin). Anywhere else the host NRI plugin is the only image-admission gate, and the render fails rather than leave confidential workloads with no allowlist enforcement.
TDX host prerequisites (--hardware-platform=tdx) TDX RuntimeClasses and the TDX attestation path schedule on nodes labeled `confidential.ai/tdx=true`. On the default `--cvm-mode=pod` path the installer applies that label itself to every kata-targeted node, trusting `--hardware-platform` — and aborts if any node still carries the other platform's label, so a platform switch has to be your explicit act. On a `-f` install or a non-pod shape the label is yours to apply, and the install fails fast without it: ```bash kubectl label node confidential.ai/tdx=true ``` The label is a scheduling aid, not a hardware probe. Apply it only once the host is actually TDX-ready: - the `tdx_guest` module is available, - `qgsd` (Intel DCAP Quote Generation Service) is running, with a `socat` unix→vsock bridge so kata's quote-generation socket reaches it, - an Intel PCS API key is configured in `/etc/sgx_default_qcnl.conf` (DCAP fetches TCB collateral during verification). `--cvm-mode=aks` is exempt from all of this: the TD quote comes from the Azure vTPM, so there is no TDX node label and no guest device.
## Verify the install A few quick checks confirm everything came up. **Components are healthy** — CDS, the operator, and (under pod-as-CVM) `kata-deploy` should be `Running`: ```bash kubectl get pods -n c8s-system ``` **The admission webhook is wired up** — the operator patches its `caBundle` once at startup, so it should be non-empty: ```bash kubectl get mutatingwebhookconfiguration | grep c8s # caBundle populated? (any non-zero count means it's patched) # the config is named -pod-injector (release defaults to c8s): kubectl get mutatingwebhookconfiguration c8s-pod-injector \ -o jsonpath='{.webhooks[*].clientConfig.caBundle}' | wc -c ``` **A workload got its confidential identity** — inspect a pod you annotated `confidential.ai/cw`. Under pod-as-CVM it should run under the confidential RuntimeClass for your CPU TEE and carry both injected init containers: ```bash # confidential RuntimeClass (pod-as-CVM) — describe shows it directly: kubectl describe pod | grep "Runtime Class" # Runtime Class Name: kata-qemu-snp (kata-qemu-tdx on TDX) # the webhook-injected containers are present, in this order: kubectl get pod -o jsonpath='{range .spec.initContainers[*]}{.name}{"\n"}{end}' # c8s-cert the get-cert sidecar # c8s-cert-wait blocks the workload until the first cert is written ``` **End-to-end confidentiality** — verify it from *outside* the cluster with [c8s-verify](/docs/c8s/concepts/verification-model). An external client pins the same `` you installed with: ```bash c8s verify https:// --measurements ``` Run with no `--measurements`, it prints the measurement it saw plus `WARNING: no --measurements pinned — any genuine TEE is accepted (UNSAFE for production)`. That is how you read `` off a booted cluster; it is never the production check. ## Next steps - [Run a confidential workload](/docs/c8s/how-to/run-a-confidential-workload) — mark a Deployment for confidential treatment. - [Manage the allowlist](/docs/c8s/how-to/manage-the-allowlist) — nothing starts until its digest is allowlisted. - [Verify a deployment](/docs/c8s/how-to/verify-a-deployment) — prove the cluster is what you think it is. - [Troubleshoot an install](/docs/c8s/how-to/troubleshoot) — when a component will not come up. - [Uninstall c8s](/docs/c8s/how-to/uninstall) — including the host-side kata sweep. --- # Manage the allowlist `c8s allowlist` is the operator tool for the image allowlist: unauthenticated reads (`list`, `export`, `diff`, `workload list`, `workload get`, `lint`, `inspect-image`) and operator-signed writes (`add`, `remove`, `upload`, `workload apply`, `workload edit`, `workload delete`). Add every image a confidential workload needs — the workload image and any init containers — *before* you deploy, or the enforcer will refuse the container at start. For what the two layers mean and what an operator key does and does not protect, see [The image allowlist](/docs/c8s/concepts/image-allowlist). Every flag is in the [CLI reference](/docs/c8s/reference/cli#c8s-allowlist). `c8s allowlist` is the operator tool for the API above — unauthenticated reads (`list`, `export`, `diff`, `workload list`, `workload get`, `lint`, `inspect-image`) and operator-signed writes (`add`, `remove`, `upload`, `workload apply`, `workload edit`, `workload delete`). Add every image a confidential workload needs (the workload image and any init containers) *before* you deploy, or the enforcer will refuse the container at start. Every flag is in the [CLI Reference](/docs/c8s/reference/cli#c8s-allowlist). ## Create the operator credential Generate an EC keypair (P-256, P-384, and P-521 are accepted) and pin the **public** half at install time — the private key stays with you: ```bash # the operator private key — keep it in a vault/HSM; it never leaves your machine openssl ecparam -name prime256v1 -genkey -noout -out operator.key # the public half — this is what the CDS pins openssl ec -in operator.key -pubout -out operator.pub c8s install --operator-keys operator.pub # plus your other install flags ``` To authorize several operators, concatenate their `.pub` files into one bundle — and pin **distinct keys per cluster**. Installing *without* `--operator-keys` leaves allowlist writes disabled (reads still work), so `c8s install` refuses the default path unless you acknowledge with `--force`. `cds.operatorKeys` is the PEM **content**, never a file path. If you set the chart value yourself — a values file, a Flux `HelmRelease`, or helm `--set-file` — paste the PEM block; a path from the machine that rendered the values is meaningless in-cluster, and the chart fails the render if the value doesn't look like PEM. `c8s install --operator-keys` and `c8s render-values --operator-keys` both read the file and embed its content for you. ## Reach the allowlist API By default the chart publishes the complete `/allowlist` API through the **tls-lb front door** — the same host your application traffic uses. Point `--url` at it and pin the **tls-lb** launch digest: ```bash URL=https:// DIGEST= ``` ```text ┌───────────────────────────────┐ │ c8s allowlist │ │ --url https:// │ │ --measurements │ └───────────────┬───────────────┘ │ 1. GET /v1/discovery over one TLS connection; │ verify the evidence against --measurements; │ require the attested cert to be this handshake's leaf ▼ ╔═══════════════════════════════╗ ║ tls-lb ║ nginx + an in-pod RA-TLS-verifying proxy ║ /allowlist ║ (and /allowlist/) ╚═══════════════╤═══════════════╝ │ 2. verify the CDS's RA-TLS evidence against │ cds.measurements, then forward the method, │ URI, body, and Authorization header unchanged ▼ ╔═══════════════════════════════╗ ║ CDS ║ owns the store; checks the operator token ╚═══════════════════════════════╝ ``` *Two attested hops. Because the front door forwards the request bytes untouched, the operator token's method, path, and body bindings still match what the CDS receives.* The CLI picks its verification path by probing the target: if it serves a discovery document it is treated as a front door and verified through it; if it serves none — a direct CDS URL — the CLI falls back to verifying the CDS's RA-TLS serving certificate. A discovery document that is present but fails verification is a hard error, never a fallback. **The CLI refuses a WebPKI front door.** When `tlsLb.publicTLS.secretName` is set, tls-lb serves your own public certificate and reports `public_tls.mode=webpki` in its discovery document. That certificate is not cryptographically bound to the discovery attestation, so the CLI rejects the endpoint rather than trust it. Use a direct CDS RA-TLS URL and pin the **CDS** launch digest instead. The same applies when the public route is gone: `tlsLb.allowlist.enabled=false`, or a `tlsLb.routes` entry of your own at `/allowlist` or `/allowlist/`, removes the built-in route. For the direct path, pin the **CDS** launch digest, and port-forward the CDS if it is not otherwise routable — the CLI verifies the CDS's attestation, so the localhost hop is fine: ```bash kubectl port-forward -n c8s-system svc/c8s-cds 8443:8443 & URL=https://localhost:8443 DIGEST= ``` This works under Pod-as-CVM too: the guest image exempts port 8443 from the in-guest mesh's inbound mTLS redirect, so a port-forward reaches the CDS's RA-TLS listener. What the locked guest policy denies is `kubectl exec` and `kubectl logs`, not port-forward. `--measurements` (or `--measurements-file`, one hex digest per line) always names the trusted build of the endpoint you connected to — never a third party's. Get the value from [Obtaining launch measurements](/docs/c8s/how-to/obtain-launch-measurements), and check the front door on its own with [`c8s verify --kind lb`](/docs/c8s/how-to/verify-a-deployment#verify-the-load-balancer). Empty measurements accept *any* attested endpoint and warn `UNSAFE`; a plaintext `http://` URL is refused outright unless you pass `--insecure` (dev/test only, skips attestation entirely). Two operational notes for the front-door path. The session is bound to the single TLS connection whose handshake was attested and is never redialed, because a second handshake could reach a different tls-lb replica holding a different certificate — if the connection drops, the command fails closed and asks you to re-run it. And the route is rate-limited before nginx collapses callers onto the in-pod proxy: mutations at 1 r/s with burst 5 per client and 8 r/s with burst 15 across all clients, reads at 20 r/s with burst 40 per client, over-limit requests answered `429`. A direct CDS URL is accounted under your own source address instead. ## Read it ```bash # the floor and the workload entries (text; -o json for machine-readable) c8s allowlist list --url $URL --measurements $DIGEST # back it up — canonical JSON that round-trips as an upload or a seed c8s allowlist export allowlist.json --url $URL --measurements $DIGEST # preview what a changed file would do; --exit-code makes any difference non-zero c8s allowlist diff allowlist.json --url $URL --measurements $DIGEST ``` `diff` reports the floor as `+` (added), `-` (removed), and `~` (image reference changed), and workload entries the same way, with a field-level breakdown of every changed entry. Run it before any `upload`. ## Change the floor Writes additionally take the operator private key, by flag or the `C8S_OPERATOR_KEY` environment variable (the flag wins). `add`, `remove`, and `upload` all support `--dry-run`: ```bash # add one digest (see "Getting an image's digest" below) c8s allowlist add \ --url $URL \ --measurements $DIGEST \ --operator-key operator.key # remove one or more digests c8s allowlist remove \ --url $URL \ --measurements $DIGEST \ --operator-key operator.key # replace the whole document from a file — lints it, prints the diff, then writes c8s allowlist upload allowlist.json \ --url $URL \ --measurements $DIGEST \ --operator-key operator.key ``` `` is the image's `sha256:` manifest digest and `` the image reference it maps to. `remove` warns when the digest you dropped is a c8s component image: removing it from the CDS does not lock it out, because the plugin's `always_allow` set and the in-guest baked seed still admit it — replace a compromised component digest by rolling the chart instead. `upload` refuses a file whose image labels name none of the core c8s components (`cds`, `ratls-mesh`, `nri-image-policy`, `attestation-api`, `nginx`) — a cluster missing them cannot pull its own control plane. Override with `--force`, or change the required set with `--require`. It also runs the offline lint first: a lint **error** blocks the write regardless of `--force`, and `--strict` makes warnings block it too. ## Change workload policy Workload entries are managed as whole entries. Nothing field-merges, so no command can silently clobber a sibling field, and every mutating path shows the field diff first: ```bash # what is there c8s allowlist workload list --url $URL --measurements $DIGEST c8s allowlist workload get vllm-llama --url $URL --measurements $DIGEST # upsert entries from a file (or '-' for stdin); --dry-run stops before the write c8s allowlist workload apply entries.json \ --url $URL \ --measurements $DIGEST \ --operator-key operator.key # fetch → $EDITOR → lint → diff → confirm → write c8s allowlist workload edit vllm-llama \ --url $URL \ --measurements $DIGEST \ --operator-key operator.key c8s allowlist workload delete vllm-llama \ --url $URL \ --measurements $DIGEST \ --operator-key operator.key ``` `apply` accepts either a full allowlist document or a bare name-keyed map of entries; floor digests in the file are ignored (use `add` or `upload` for those). It lints the file *and* cross-checks it against the served allowlist, because the entry a new one collides with is usually one already there. Check a file before you write it with `lint`, which reads a file or stdin and never contacts the CDS: ```bash c8s allowlist lint entries.json --strict ``` It reports entries with no containers, a `command: deny` container that can never start, a shared digest whose union has been widened to `any`, a digest that is floor-listed while also carrying a workload policy, tag-form image labels (mutable, so TOCTOU), an entry granting the root secret subtree `/**` (every secret in the store), and a count of the `any` policies the document carries. Two entries declaring the same containers with the same argv policy are an **error**, not a warning: nothing a running pod can do tells them apart, so both are refused forever. Errors exit non-zero on their own; `--strict` makes warnings do the same, which is what you want in CI. `--online` additionally checks each digest resolves in its registry via `crane`. ## Grant secret paths A grant belongs to the entry, so it takes the same path as everything else in it: a whole-entry, operator-signed write. There is no separate command and no second credential. ```json { "vllm-llama": { "initContainers": [], "containers": [ { "digest": "sha256:", "command": { "policy": "exact", "argv": ["python3"] }, "args": { "policy": "any" } } ], "secrets": { "policy": "allow", "read": ["/tenant-a/**"], "write": ["/tenant-a/session"] } } } ``` ```bash c8s allowlist workload apply entries.json \ --url $URL \ --measurements $DIGEST \ --operator-key operator.key ``` Writing the grant is what turns release on: an entry carrying none releases nothing, and the CDS refuses a workload that resolves to it. Revoking is the same write in reverse — set `policy: deny`, or drop the field, and apply. Because the entry is replaced whole, the grant and the container policy it rides on always move together. A grant is only as reachable as its entry. Release resolves a pod's running containers to **exactly one** workload entry — not to membership, as certificate issuance does — so an entry nothing resolves to hands out nothing, and neither does either half of a pair no running set can tell apart (`lint` reports that pair as an error). See [Secrets](/docs/c8s/concepts/application-secrets) for the release decision and how to debug one. Two tooling notes. The field diff that `apply` and `edit` print covers the label and the container policies but **not** the grant: `workload edit` therefore reports `no changes` and writes nothing when the grant is your only edit, while `workload apply` prints `= (unchanged)` and writes it anyway — so use `apply` for a grant-only change. And the workload table printed by `list` and `workload list` summarizes each entry's grant in its last column (still headed `PATHS`): `deny`, or `allow(r=…,w=…)` with the number of read and write paths it carries. ## Verify who can write Which operator keys the CDS actually pins is a *verification* task, and `c8s cds verify --operator-keys` is where it lives — it fetches `GET /operator-keys` over a connection pinned to the attested serving certificate and fails closed on a key set that does not match your bundle. See [Check the operator keys CDS pins](/docs/c8s/how-to/verify-a-deployment#check-the-operator-keys-cds-pins). Run it on a schedule in CI rather than once at bootstrap — the pinned-key list is host-supplied config, so it is [detected, not prevented](/docs/c8s/concepts/image-allowlist#authorizing-mutations). ## Getting an image's digest Every allowlist entry is keyed by an image's **`sha256:` manifest digest** — the immutable content address, not a tag. A few ways to get one, no full pull required: **With `c8s allowlist inspect-image`** — resolves the digest *and* prints the image's baked `Entrypoint` and `Cmd`, which is what you need to write an `exact` argv policy. It reads the registry only and never contacts the CDS: ```bash c8s allowlist inspect-image ghcr.io/example/inference:v3 ``` ```text ref: ghcr.io/example/inference:v3 digest: sha256:9f2c… entrypoint: /usr/bin/inference cmd: --port 8080 ``` **With `crane`** (`c8s install` requires it on PATH unless you pass `--resolve-digests=false`, and `inspect-image` and `lint --online` shell out to it) — works against any OCI registry: ghcr.io, Docker Hub, an internal mirror: ```bash crane digest ghcr.io/example/inference:v3 # sha256:9f2c… ``` **With Docker** — `buildx imagetools` reads it straight from the registry without pulling: ```bash docker buildx imagetools inspect ghcr.io/example/inference:v3 --format '{{.Manifest.Digest}}' ``` …or, if you've already pulled the image, read the digest it resolved to: ```bash docker inspect --format '{{index .RepoDigests 0}}' ghcr.io/example/inference:v3 # ghcr.io/example/inference@sha256:9f2c… ``` **With `skopeo`** — handy in CI where Docker isn't running: ```bash skopeo inspect docker://ghcr.io/example/inference:v3 | jq -r .Digest ``` Registry UIs show it too — ghcr.io lists the digest on a package's version page, and Docker Hub shows it per tag. **Multi-arch images.** A tag like `…:v3` can point to a manifest *list* with one entry per platform. `crane digest` returns the **index** digest by default, but a node resolves that tag to its **platform-specific** manifest when it pulls — and the enforcer matches the `…@sha256:…` digest the runtime actually resolved. Pin *that* one, e.g. `crane digest --platform linux/amd64 ghcr.io/example/inference:v3`, or read it back from a running pod: ```bash kubectl get pod -o jsonpath='{.status.containerStatuses[*].imageID}' ``` Pin the index digest of a multi-arch image and the check will reject the very image you meant to allow. ## Automating the allowlist A write is a single CLI call authorized by a key file, so it scripts cleanly — but automation here holds a live credential for the image-integrity control. Treat the operator key like a release-signing key, not a CI variable: vault-managed, one key per cluster and per pipeline, and removed from `cds.operatorKeys` when retired. **In CI.** A pipeline that builds a workload image knows the resulting digest at build time. Give the release job its own operator key and make the final step: ```bash c8s allowlist add "$(crane digest "$IMAGE")" "$IMAGE" \ --url "$URL" \ --measurements "$DIGEST" \ --operator-key /run/secrets/operator.key ``` Pair it with two gates in the same pipeline: `c8s allowlist lint --strict` on any policy file you keep in source control, and `c8s cds verify --operator-keys` so a swapped write policy fails the build rather than going unnoticed. The token minted for that call expires in 60 seconds and can authorize only that exact change, so what transits the pipeline is narrow; the key file is the thing to protect. The CDS logs every write it accepts — the floor digest and image, or the workload entry name — and every one it rejects, but never which pinned key signed it. **With kettle (planned).** [kettle](/docs/papers/kettle.pdf), Confidential AI's attested-builds system, produces images whose digest is tied to a cryptographic attestation of *how they were built* — reproducible source → measured artifact. The goal is to let a kettle-attested build feed the allowlist directly: a digest admitted because the *build* proved itself, not because a key holder vouched for it. That closes the loop — **source → reproducible build → measured image → allowed to run in the TEE** — with nobody hand-curating digests. None of this is wired up yet. It needs an attested-writer path — a launch measurement, rather than a pinned key, authorizing the write. See [Limitations](/docs/c8s/concepts/limitations). ## See also - [The image allowlist](/docs/c8s/concepts/image-allowlist) — the data model, argv policy, and enforcement. - [CDS HTTP API → Allowlist](/docs/c8s/reference/cds-api#allowlist) — the endpoints these commands drive. - [Verify a deployment](/docs/c8s/how-to/verify-a-deployment#check-the-operator-keys-cds-pins) — cross-check which keys the CDS actually pins. --- # Obtain launch measurements A measurement is the **SHA-384 launch digest** of a confidential VM — a hash the TEE hardware (AMD SEV-SNP or Intel TDX) computes over the exact launch state (firmware, guest kernel, and the kernel command line, which embeds the dm-verity root hash of the rootfs). Every `*measurements` flag in **c8s** takes one or more of these hex strings. Because a reproducible image always yields the same digest, pinning it means *"accept only this exact, audited image — nothing else."* Three ways to get the value, best first. ### Use the digest published with the image (recommended) c8s guest images are reproducible and ship their **predicted** launch digest. The Kata guest image's digest is published in its `manifest.json` and pinned via `kata.guestImage.tag`; component CVM images publish theirs the same way. Pin the published digest for the exact image **version (tag)** you deploy — since the build is reproducible, that value is "exactly this audited code." This is the production path. ### Recompute it with `sev-snp-measure` To independently verify the published value (or for a custom image), recompute the expected digest yourself with AMD's [`sev-snp-measure`](https://github.com/virtee/sev-snp-measure) from the launch components — the OVMF firmware, the guest `vmlinuz`, and the kernel cmdline (which carries the dm-verity `root_hash`) — at the **same vCPU count used at launch** (c8s measures at 1 vCPU for a stable digest): ```bash # illustrative — see the sev-snp-measure docs for the exact flags for your image sev-snp-measure --mode snp --vcpus 1 \ --ovmf OVMF.fd \ --kernel vmlinuz \ --append "" \ --output-format hex ``` A reproducible build yields the same digest that's in the manifest; a mismatch means the image isn't what you think it is. (`sev-snp-measure` is AMD-specific; on **Intel TDX** the equivalent launch values — the `MRTD` and `RTMR`s — are predicted with TDX's own tooling, but the pin-the-published-digest workflow above is the same.) To use this as a *post-deploy* cross-check — recompute, pin, then verify a live node reports the same digest — see [Independently recompute the measurement](/docs/c8s/how-to/verify-a-deployment#independently-recompute-the-measurement). ### Read it from a live attestation report (inspect, then pin) To see what a running component actually measures, read the `launch_digest` from its evidence: `get-cert --discovery-out discovery.json …` writes the attestation evidence and measurement for the issued certificate, and the attestation-api's verify result returns `claims.launch_digest`. Use this to *inspect* what's running — but in production pin the **published or recomputed** value above, not merely whatever a node reports, or you've only achieved trust-on-first-use. ## Which flag wants which digest Each flag wants the digest of a different party: | Flag | Pin the launch digest of… | | --- | --- | | `cds.measurements` (`--measurements`) | the workloads and peers allowed to attest to the CDS — and, reused, the inventories the CDS dials for sandbox digests | | `ratlsMesh.measurements` | the mesh peers allowed to connect | | `--cds-measurements` (get-cert, ratls-mesh `cds` mode, policy-monitor) and `allowlist.pull.cds_measurements` (nri-image-policy) | the CDS itself | | `--measurements` (`c8s allowlist`) | the endpoint the CLI connects to — tls-lb on the default public route, or the CDS on a direct URL | | `--handoff-measurements` (the chart reuses `cds.measurements` when `cds.handoff.enabled`) | peer CDS replicas allowed to receive the CA and allowlist | | `measurements` in [c8s-verify](/docs/c8s/reference/c8s-verify-js) | the user-facing LB enclave | On a Node-as-CVM install, `c8s install --measurements ` fills the first two rows in one pass — the node image's digest from its `manifest.json` — so the mesh is pinned from first boot rather than accept-any-then-tighten. It is rejected with `--cvm-mode=pod`, where the components that speak to the CDS are per-pod Kata guests measured separately. A `--measurements` entry that is not valid hex fails CDS startup rather than silently unpinning the sandbox callback. Measurements identify **confidential VMs** — per-pod CVMs (Pod-as-CVM) or a confidential node (Node-as-CVM). On a non-confidential host, components run as ordinary containers with no launch digest, so pinning only becomes meaningful once the node or pods are actually CVMs. ## See also - [Verify a deployment](/docs/c8s/how-to/verify-a-deployment) — pin a digest and check a live component against it. - [The trust root](/docs/c8s/concepts/trust-root) — why the CDS refuses to pin by default, and what an unpinned mesh accepts. - [Threat model](/docs/c8s/concepts/threat-model#measurement-pinning-is-mandatory-for-production) — why this is mandatory for production. --- # Provisioning a confidential cluster on Azure (AKS) The simplest way to get a Confidential Kubernetes cluster on Azure is a managed **AKS** cluster with a **confidential VM node pool**: Azure runs the control plane and gives you nodes that are already confidential VMs, so there is no VM to build and no Kubernetes to install by hand. You point `kubectl` at it from your laptop and install c8s. The AKS control plane is managed by Azure and — like any control plane — stays **outside** the trust boundary; the confidential node pool is your **node-as-CVM**. When the cluster is up, continue to **[Install c8s](/docs/c8s/how-to/install)**. New to c8s? Skim the [overview](/docs/c8s) and the [runtime model](/docs/c8s/concepts/trust-boundaries) first — on Azure you run the **node-as-CVM** shape. This page assumes only basic Azure and `kubectl` familiarity. ## Prerequisites On your **laptop**: - An **Azure subscription**, with the [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) installed and logged in (`az login`). - **`kubectl`** — if you don't have it, `az aks install-cli` installs a compatible build. ## Confidential VM sizes AKS confidential node pools use the same AMD SEV-SNP **DCas_v5** family as standalone Azure confidential VMs. Pick a node size by workload: | SKU | vCPU | Typical use | | --- | --- | --- | | `Standard_DC2as_v5` | 2 | smallest — quota/cost-friendly smoke tests | | `Standard_DC4as_v5` | 4 | a comfortable single-node dev cluster | | `Standard_DC48as_v5` | 48 | a large single node | Confidential VM sizes exist only in some regions (for example `northeurope`); check with `az vm list-skus --location --size Standard_DC` before you provision. Azure also offers **Intel TDX** confidential VMs (the `DCes_v5` / `ECes_v5` family), and c8s supports them on the same vTPM attestation path. The choice is a second install flag, `--hardware-platform` — see [Next: install c8s](#next-install-c8s). Everything else in this guide is identical; substitute the TDX size for the `DCas_v5` one. ## Step by step ### 1. Create the AKS cluster with a confidential node pool Choosing a `DCas_v5` node size makes the pool a **SEV-SNP confidential VM node pool** (a `DCes_v5` / `ECes_v5` size makes it an **Intel TDX** one) — each node boots as a CVM with a vTPM. `--node-count 1` keeps it single-node for a dev cluster. ```bash az group create --name c8s-rg --location northeurope az aks create \ --resource-group c8s-rg \ --name c8s-aks \ --node-count 1 \ --node-vm-size Standard_DC4as_v5 \ --os-sku Ubuntu \ --generate-ssh-keys ``` **Verify the node-pool shape.** Some subscriptions/regions only allow the confidential VM size on a **user** node pool, not the default system pool. If `az aks create` rejects the CVM size, create a minimal regular system pool and add the confidential pool separately: ```bash az aks nodepool add \ --resource-group c8s-rg \ --cluster-name c8s-aks \ --name conf \ --mode User \ --node-count 1 \ --node-vm-size Standard_DC4as_v5 \ --os-sku Ubuntu ``` With a separate system pool you no longer have a single node — drop `--single-node` at install time and let the CDS schedule onto the confidential pool (label/taint it so only confidential workloads land there). ### 2. Point kubectl at the cluster ```bash az aks get-credentials --resource-group c8s-rg --name c8s-aks kubectl get nodes -o wide ``` Everything from here runs **from your laptop** — there's no node to SSH into. When the node reports `Ready`, you're ready to install c8s. ## Next: install c8s Your cluster is now a managed AKS cluster whose node is a confidential CVM, with `kubectl` wired up on your laptop. Continue to **[Install c8s](/docs/c8s/how-to/install)**. Two Azure-specific flags to know up front. **`--cvm-mode aks`** selects the device shape. Attestation on Azure comes from the **vTPM at `/dev/tpm0`**, never a native `/dev/sev-guest` or `/dev/tdx-guest` device — those are not exposed to an Azure guest. Despite the name the mode means "Azure vTPM" and applies to any Azure confidential VM. **`--hardware-platform`** selects the CPU TEE inside that vTPM report, which wraps either an SNP report (az-snp) or a TD quote (az-tdx). The default `sev-snp` is right for the `DCas_v5` pool above; on a `DCes_v5` / `ECes_v5` Intel TDX pool, pass `tdx`. Nothing else differs — there is no TDX node label to apply and no guest device to expose: ```bash c8s install --single-node --cvm-mode=aks --hardware-platform=tdx --operator-keys operator.pub ``` `operator.pub` is the allowlist-write credential you generate in the first step of the install guide. Neither install pins a launch measurement on its own. Because the node image here is Azure's, not one you built, read the measurement off the running cluster once with an unpinned [`c8s verify`](/docs/c8s/how-to/verify-a-deployment), then re-run `c8s install` with `--measurements ` — [Obtaining launch measurements](/docs/c8s/how-to/obtain-launch-measurements) covers the procedure. For the full provision-through-verify path on one page, follow [Your first confidential cluster](/docs/c8s/tutorials/first-confidential-cluster). **Validate the host-side install on managed nodes.** c8s's node-as-CVM install runs privileged DaemonSets that touch the node host: `nri-image-policy` patches the node's containerd config and restarts it, `ratls-mesh` sets up `iptables`, and `attestation-api` mounts `/dev/tpm0`. This is routine on self-managed nodes; on a **managed AKS node pool**, confirm these complete and **survive node image upgrades / reconciliation** before relying on it in production — AKS may reconcile host changes on upgrade. ## Why pod-as-CVM is not available on Azure This is the key limitation to plan around: Azure's hypervisor does **not** expose nested virtualization. A [pod-as-CVM](/docs/c8s/concepts/trust-boundaries) design launches a *new* confidential VM **inside** the node for every pod — that requires nesting, which Azure does not provide. So each AKS confidential node is a single node-level CVM, and **all pods on it share that one confidential boundary.** Don't use `--cvm-mode=pod` on Azure. Concretely, on AKS: - ✅ **Node-as-CVM works.** Each node runs as one confidential CVM; its memory is opaque to the Azure operator. Attestation is at the node level — verify the node's launch state once and every pod on it inherits that verified boundary. - ❌ **Pod-as-CVM (Kata per-pod VMs) does not work.** Launching a VM inside the node VM would be nested virtualization, which the platform does not support. - ✅ **You still get strong guarantees.** Pod identities are issued by the CDS, the image allowlist is enforced at the node, inter-node traffic (once you scale past one node) is RA-TLS-protected, and the CDS issues certificates only to peers whose evidence verifies. The tradeoff is attestation granularity: on Azure you cannot distinguish two pods on the same node at the attestation level — they are inside the same TEE. ## Self-managed and bare-metal hosts If you need **pod-as-CVM** (per-pod isolation from the cluster operator), or you're running outside Azure, you need a **bare-metal-class confidential host that allows nested VMs**. Provisioning that host (and a self-managed Kubernetes on it) is out of scope for this AKS guide; once you have a cluster on it, [Install c8s](/docs/c8s/how-to/install) is the same flow — use `--cvm-mode=pod` for pod-as-CVM, or `--cvm-mode=node` (`--cvm-mode=gke` on GKE's native `/dev/sev-guest`) for node-as-CVM. See [pod-as-CVM vs node-as-CVM](/docs/c8s/concepts/trust-boundaries) for the full comparison. --- # Run a confidential workload Once installed, mark a workload for confidential treatment with the `confidential.ai/cw` annotation. The webhook injects a `get-cert` sidecar that fetches a TEE-bound certificate from the CDS. Under pod-as-CVM (`--cvm-mode=pod`) it also injects the confidential RuntimeClass for the cluster's CPU TEE — `kata-qemu-snp` on SEV-SNP, `kata-qemu-tdx` on TDX (the `-nvidia` variants when the pod requests a GPU): ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-inference spec: template: metadata: annotations: confidential.ai/cw: my-inference # the workload identity (NOT a boolean) spec: containers: - name: server image: ghcr.io/example/inference@sha256:... # must be on the allowlist ``` The annotation **value is the workload's identity**, not a `true`/`false` flag. The operator derives a managed headless Service named `c8s-` from it and the workload's certificate SAN is `c8s-..svc` (overridable with `confidential.ai/c8s-san`). So `confidential.ai/cw: my-inference` yields the Service `c8s-my-inference` — which is what the front door (tls-lb's upstream, derived via `--workload-ref my-inference=//:` plus `--upstream my-inference`) dials to reach the workload over the RA-TLS mesh. Use a stable, DNS-label-safe name; a value like `"true"` would only produce a Service named `c8s-true`. Make sure the image digest is on the [allowlist](/docs/c8s/concepts/image-allowlist), or image admission refuses to start the container — `nri-image-policy` on the host, or the in-guest `policy-monitor` under `--cvm-mode=pod`. ## What the webhook adds Every annotation the webhook reads, and every container and volume name it reserves, is in [Workload annotations](/docs/c8s/reference/workload-annotations). A workload can also ask for [application secrets](/docs/c8s/how-to/use-application-secrets) and [encrypted volumes](/docs/c8s/how-to/create-an-encrypted-volume) with further annotations on the same pod. ## Confirm it worked ```bash # under Pod-as-CVM the pod runs under the confidential RuntimeClass: kubectl describe pod | grep "Runtime Class" # Runtime Class Name: kata-qemu-snp (kata-qemu-tdx on TDX) # the webhook-injected containers are present, in this order: kubectl get pod -o jsonpath='{range .spec.initContainers[*]}{.name}{"\n"}{end}' # c8s-cert the get-cert sidecar # c8s-cert-wait blocks the workload until the first cert is written ``` A pod stuck before those appear is usually an allowlist miss — see [Troubleshoot an install](/docs/c8s/how-to/troubleshoot). --- # Troubleshoot an install Find the symptom, not the subsystem. Each entry names what you would see, what causes it, and what to do. **Start with the pods, not the Helm status.** A `--cvm-mode=pod` install can exceed Helm's wait window while the cluster converges fine underneath, so the release reads `failed` on a cluster that is coming up correctly. ```bash kubectl get pods -A -w ``` ## The install ### `c8s install` sits for minutes with no output Expected. `helm --wait` blocks until every component reports Ready. The wait is bounded: **5 minutes**, or **10 minutes** for `--cvm-mode=pod`. Watch progress from a second shell with `kubectl get pods -A -w` — `kata-deploy` and the CNI live outside `c8s-system`. ### `c8s install` fails once the timeout elapses The resources it already applied stay in place. Fix the cause and **re-run it** — the install is idempotent (`helm upgrade --install`) and completes once every pod is Ready. On a node with no prior Kata install, `kata-deploy` downloads a multi-GB payload inside the wait window. A slow registry can still blow the 10 minutes: the release lands as `failed` while the cluster converges, and a second run flips it to `deployed`. ### `crane digest ...: UNAUTHORIZED: authentication required` Digest resolution runs `crane` **locally** during the install, reading your `~/.docker/config.json`. Log in on the box: ```bash crane auth login -u --password-stdin ``` This is a separate credential from `--image-pull-secret`, which is the cluster-side kubelet credential. An authenticated registry needs both. Or skip resolution with `--resolve-digests=false` and supply digests via `-f`. ### The install refuses to proceed without `--operator-keys` Deliberate: installing without it disables allowlist writes, so nobody can allowlist a workload image. Create the credential (see [Manage the allowlist](/docs/c8s/how-to/manage-the-allowlist#create-the-operator-credential)) or acknowledge with `--force`. Supplying any `-f` values file also clears the guard — those files own `cds.operatorKeys`. ### A TDX install fails fast at the node check TDX RuntimeClasses schedule on nodes labeled `confidential.ai/tdx=true`. The default `--cvm-mode=pod` path applies the label itself; a `-f` install or a non-pod shape does not. ```bash kubectl label node confidential.ai/tdx=true ``` `c8s uninstall` **sweeps these labels** along with the Kata artifacts, so a reinstall after an uninstall can hit this even though the first install worked. Relabel and rerun. ## Pods that will not start ### A pod stays `Pending` with a node-affinity message Either CDS cannot find its node, or a confidential pod cannot find a TEE-labeled one. - `role=cds` — CDS pins to a labeled node by default. Label it, pass `--single-node`, or set a real selector. Note that `cds.node.selector: null` in a values file **does not survive** Helm's multi-file merge and silently reverts to `role: cds`; set `{}` or label the node. - `confidential.ai/sev-snp=true` / `confidential.ai/tdx=true` — the confidential RuntimeClasses carry a node selector on these. ### `tls-lb` stays `Pending` — "didn't have free ports for the requested pod ports" `tlsLb.hostPort.enabled` is `true` by default and grabs the node's `443`. On a cluster whose ingress already owns it — notably RKE2, which bundles `rke2-ingress-nginx` on 80/443 — set `tlsLb.hostPort.enabled: false` and reach `tls-lb` through its Service, or bind a free port. ### A confidential pod loops in `ContainerCreating` The effective ceiling is `min(kubelet runtime-request-timeout, kata timeout)`. kubelet's default is **2 minutes**; it cancels the CRI call, containerd tears the sandbox down, and the cause is hidden. Any slow path — cold registry, huge image, big-memory guest — hits that wall first. On RKE2, raise it in `/etc/rancher/rke2/config.yaml`: ```yaml kubelet-arg: runtime-request-timeout=20m ``` ### QEMU crash-loops with register dumps every ~90 seconds `kata-qemu-snp` on a non-SNP host does not fail cleanly — it launches a guest with the SNP-shaped config that cannot boot, and kubelet retries forever. The node label is **declared, not probed**, so `--hardware-platform` set to the wrong hardware reproduces this on every node at once. Check what the host actually is before relabeling. ### A brief `ImageInspectError`, or one `ratls-mesh` restart, early in the rollout Normal. `nri-image-policy` restarts containerd as it installs; the mesh pod retries through that and self-heals. ### `ImagePullBackOff` or `CrashLoopBackOff` past the timeout Usually a missing or wrong image pull secret. `--image-pull-secret` names an **existing** `kubernetes.io/dockerconfigjson` Secret in the release namespace — the install never creates it. Under `--cvm-mode=pod`, a workload image's layers are fetched **inside** the guest and that pull is **anonymous**. A pod-as-CVM workload image must be pullable with no credentials at all. ## Workloads ### A pod hangs in `Init` — "Startup probe errored … ExecProcessRequest is blocked by policy" The locked guest denies `ExecProcessRequest`, and kubelet cannot distinguish an exec *probe* from a host exec. **Any `exec` probe on a container inside a locked guest never passes.** Use a run-once init container, or `httpGet` / `tcpSocket` / `grpc`. ### The container is refused at creation Its image digest is not on the [allowlist](/docs/c8s/concepts/image-allowlist). Add it: ```bash c8s allowlist add "$(crane digest )" \ --url $URL --measurements $DIGEST --operator-key operator.key ``` Two things that make a correct-looking digest still fail: - **Multi-arch images.** `crane digest` returns the **index** digest, but the node resolves the tag to its **platform-specific** manifest and the enforcer matches *that*. Pin `crane digest --platform linux/amd64 …`, or read it back from a running pod with `kubectl get pod -o jsonpath='{.status.containerStatuses[*].imageID}'`. - **argv policy.** A workload entry pins the command line too. An absent policy normalizes to `deny`, and `command: deny` requires an empty argv, so it can never start. Run `c8s allowlist lint --strict` — it flags exactly this. ### Under Pod-as-CVM, an allowlist write does not reach running guests Expected today. The in-guest refresh is gated on a pinned CDS measurement, and no shipping path delivers that pin — so `policy-monitor` keeps enforcing the measured seed baked into the guest image. Budget a guest-image rebuild for Pod-as-CVM allowlist changes. See [Limitations](/docs/c8s/concepts/limitations). ### `kubectl logs` returns nothing on a confidential pod By design. The locked guest policy denies `ReadStreamRequest`, so the untrusted host cannot read container stdout — even for a healthy pod. Judge by exit code, or install `--debug` for the debug guest variant. It has a **different launch measurement** and exposes container I/O to the host: development only. ## Secrets and volumes ### The secret file never appears and the pod blocks Work through these in order. 1. **Is it just timing?** The CDS releases only once *every* main container is running, so the file appears *after* your container starts. A consumer must wait for it, not read it at startup. 2. **Is there a grant?** An entry with no `secrets` grant releases nothing. Note that `workload edit` reports `no changes` for a grant-only edit — its field diff does not cover the grant. Use `workload apply`. 3. **Is the CDS serving `/secrets` at all?** It declines outright when `--ratls-platform`, `--measurements`, or `--sandbox-inventory-cidr` is missing, or when CA handoff is configured. It logs a warning naming the one it is missing. **Check the CDS log before assuming the grant is wrong.** 4. **Ask the CDS what it saw.** `c8s secrets explain --sandbox ` reports the decision in the order the CDS makes it. See [Diagnosing a refusal](/docs/c8s/how-to/use-application-secrets#diagnosing-a-refusal). A pod refused for a policy reason retries forever without progress. The fetcher lives in `initContainers`, so a terminal failure surfaces as `Init:CrashLoopBackOff` on an otherwise `Running` pod. **A `kubectl debug` container joins the sandbox record permanently.** The inventory reports a high-water mark — every container ever admitted — so attaching one to a pod that has not yet received its secret means it never will. ### After a CDS restart, replicas hold different values for one path The store is process memory with no persistence. A pod recreated after the restart finds its path empty, calls `POST`, and is given a **new** value while its siblings hold the old one. Nothing reports it. Roll every secret-consuming Deployment rather than letting pods recover piecemeal, and re-`put` every operator-supplied value. ### The volume directory is empty Check the `c8s-volume` sidecar's logs first. Then: - **`volumed.enabled` is `false` by default.** Nothing works without it. - **The key must already be in the store.** Unlike a secret, `get-volume` only ever reads — a pod scheduled before `c8s volume create` has run retries, then fails. - **The device must be on the pod's node**, attached as a raw block device with serial `c8s-vol-`. `create` prints the matching `nodeSelector`. - **Volumes require Node-as-CVM.** Under Pod-as-CVM the webhook rejects the annotation at admission. ## Verification ### `c8s verify` hangs or returns exit `3` Exit `3` is *evidence unavailable* — unreachable, unparseable, or collateral it could not fetch. It is deliberately distinct from exit `2`, which means the evidence arrived and **failed** the check. A bare SEV-SNP report needs outbound HTTPS to AMD KDS (`kdsintf.amd.com`) to fetch the VCEK; evidence that ships the VCEK inline verifies offline. ### `tls: certificate required` You dialed the in-cluster ClusterIP, which goes through the RA-TLS mesh and demands an attested client certificate. Reach the component on its **public / host address** instead. The CDS RA-TLS endpoint and the `tls-lb` serving port both answer unattested clients there. ### A PKI or SAN mismatch when dialing a pod IP Expected and fine. `verify` trusts the attestation embedded in the serving certificate, not the hostname on it. Pass `--server-name` to set SNI when forwarding to `localhost`. ### `verified: true` but nothing was actually proven Read it together with `measurement_pinned`. `false` there means *any* genuine TEE would pass identically. Pin the digest — see [Obtain launch measurements](/docs/c8s/how-to/obtain-launch-measurements). ### The allowlist CLI refuses the front door `tlsLb.publicTLS.secretName` makes `tls-lb` serve your own public certificate, which is not cryptographically bound to the discovery attestation — so the CLI rejects the endpoint rather than trust it. Use a direct CDS RA-TLS URL and pin the **CDS** launch digest. ## Things not to debug - **Empty `kubectl logs` on a locked guest** — by design, above. - **`debug=` and `smt=` in `c8s verify` text output** — never populated; they always print `false`. The debug policy is still enforced inside the verifier. - **A `failed` Helm release on a converging cluster** — check the pods, then re-run the install. ## See also - [Install c8s](/docs/c8s/how-to/install) — prerequisites and the flags that decide the shape. - [Limitations](/docs/c8s/concepts/limitations) — the gaps that are known and deliberate. - [Verify a deployment](/docs/c8s/how-to/verify-a-deployment) — the exit-code contract in full. --- # Uninstall c8s `c8s uninstall` reverses the install — same `--namespace` / `--release` defaults as `c8s install`: ```bash c8s uninstall ``` It runs `helm uninstall` to remove the release (operator, CDS, attestation-api, ratls-mesh, tls-lb, webhook configuration, RuntimeClasses, enforcement policy), and for a `--cvm-mode=pod` install it then **sweeps the host-side kata artifacts** off every node — `/opt/kata`, the containerd runtime drop-in, the multi-GB `kata-guest-base` image, the RKE2 prep template, and the kata-runtime node labels — via a short-lived privileged DaemonSet. The sweep is read from the release's computed values before deletion, so install-time `-f` overrides are honored, and it's skipped automatically for a non-pod install. The sweep only recursively deletes guest-image directories strictly under `/var/lib/c8s/`; a `kata.guestImage.hostPath` you relocated elsewhere aborts the sweep with an error rather than being deleted by a privileged DaemonSet — remove that directory yourself. To remove kata leftovers from a cluster whose release is **already gone** (e.g. a previous bare `helm uninstall`), run just the sweep: ```bash c8s uninstall --host-sweep-only ``` Uninstall **refuses to proceed while pods with a kata RuntimeClass are still running** — pulling the runtime out from under a confidential workload kills it without cleanup. Delete those workloads first, or pass `--force` (the VMs keep running unmanaged but cannot restart). Two opt-ins are destructive and **off by default**: `--delete-crds` removes the `ConfidentialWorkload` CRD **and every `ConfidentialWorkload` object in the cluster with it**, and `--delete-namespace` deletes the release namespace and everything left in it. See the [CLI Reference](/docs/c8s/reference/cli#c8s-uninstall) for every flag. --- # Use application secrets **c8s** releases application secrets under attestation: a pod gets a value only if the images running in its sandbox match an [allowlist](/docs/c8s/concepts/image-allowlist) entry that grants the path. This page is the task. For what the CDS checks, when it will not serve `/secrets` at all, and why a CDS restart is disruptive, see [Application secrets](/docs/c8s/concepts/application-secrets). **The file appears *after* your container starts.** The CDS releases only once every main container in the pod is running, so a consumer that reads its secret path at startup finds nothing there. Wait for the file. This is structural — an init container cannot fix it. See [the timing constraint](/docs/c8s/concepts/application-secrets#the-file-appears-after-your-container-starts). Grant the path first: an entry with no `secrets` grant releases nothing. See [Grant secret paths](/docs/c8s/how-to/manage-the-allowlist#grant-secret-paths). ## Request the secret in the pod Annotate the pod alongside `confidential.ai/cw`: | Annotation | What it does | | --- | --- | | `confidential.ai/c8s-secrets` | Comma-separated `NAME=/store/path` pairs. `NAME` is the file each value is written to; `/store/path` is where the value lives in the CDS store. | | `confidential.ai/c8s-secret-dir` | Where the files land. Default `/run/c8s/secrets`. | ```yaml apiVersion: v1 kind: Pod metadata: name: api annotations: confidential.ai/cw: api confidential.ai/c8s-secrets: "DB=/tenant-a/db,HF=/tenant-a/hf-token" spec: containers: - name: api image: example.com/api@sha256: ``` `confidential.ai/cw` is required — the secret annotations without it are rejected at admission, since there is no workload identity to release against. Each `NAME` becomes a filename, so it may not contain a path separator and may not be `.` or `..`, and two secrets may not share one. Each store path must be **absolute, clean, free of a trailing slash, free of percent-encoding, and free of wildcards**. A path that is not already canonical is rejected rather than repaired, so the bytes matched against the grant and the bytes used as the store key are the bytes you wrote. ## Operator-supplied values ```bash c8s secrets put /tenant-a/hf-token \ --url "https://:" \ --measurements \ --operator-key operator.key < token.txt ``` `:` is the CDS or the CDS-issued-TLS tls-lb endpoint; `` is the launch measurement that endpoint must present. The value is read from stdin or `--from-file` and stored exactly as read — a trailing newline is part of the value — and the byte count is printed so you can confirm which bytes were sent. `--dry-run` prints the intended change without calling the CDS. Writes are authorized by the operator key the CDS already pins for allowlist mutations (`--operator-keys`), which is the same key the grants themselves are rooted in. The signed token binds the method, the path, and the body, so a captured one cannot be replayed against a different path or a different value. ### Replacing a value A path that already holds a value is refused, and the CLI names what put it there — a workload-generated value or an earlier operator write. `--overwrite` replaces it and prints what it is replacing before it does: ```text ~ /tenant-a/db (replaces a workload-generated value) wrote 24 bytes to /tenant-a/db ``` The store has no versioning and no delete, so a displaced value is gone. A workload reads its secret into a file once, at startup. A replacement therefore reaches a pod only when that pod next restarts: a Deployment holding the old value keeps it until you roll it. Replacing a path a workload created is worth pausing over for that reason — the pods that generated the value go on using it. ## Diagnosing a refusal A refused pod is told only that it was refused, and the input that decides the matter — what the sandbox is running — is visible only to the CDS. `c8s secrets explain` is where that is read: ```bash c8s secrets explain --sandbox \ --url "https://:" \ --measurements \ --operator-key operator.key ``` ```text sandbox 0123456789abcdef… inventory 10.0.0.7 reported 3 container(s) dropped 1 injected by c8s candidates 2 sha256:1111… [/serve] - sha256:9999… [get-secret] sha256:8888… [sh -c sleep 1] vllm-llama NEAR MISS foreign sha256:8888… [sh -c sleep 1] no container in this entry declares it nothing is released: no entry describes the candidate set ``` The report is laid out in the order the CDS decides, so the first thing that goes wrong is the first thing you read. It uses the same inventory, binding, and allowlist the release path uses, and measures entries with the same matcher — it reports the decision rather than a reconstruction of it. `` is on the pod's certificate; `c8s verify` prints it (see [Verify a workload](/docs/c8s/how-to/verify-a-deployment#verify-a-workload)). `--json` emits the report as it arrives. It answers to the operator key, since it describes a pod the caller may not own. The report carries grant paths; a value never appears in it. ## See also - [Application secrets](/docs/c8s/concepts/application-secrets) — the release decision, the limits, and what a CDS restart destroys. - [Workload annotations](/docs/c8s/reference/workload-annotations) — every annotation, and the reserved container and volume names. - [CLI reference → `c8s secrets`](/docs/c8s/reference/cli#c8s-secrets) — every flag. - [Create an encrypted volume](/docs/c8s/how-to/create-an-encrypted-volume) — for data too large to be a secret. --- # Verify a deployment `c8s verify` confirms that a deployed **c8s** component is a **genuine TEE running the exact code you expect** — after install, from your laptop or a CI runner. `c8s cds verify` is the same command with CDS defaults. Verification only means something once you [pin a launch measurement](#pin-the-launch-measurement); run it unpinned and it accepts any genuine TEE. For how the verifier reaches its verdict, and for the browser path this one does not cover, see [the verification model](/docs/c8s/concepts/verification-model). Every flag is in the [CLI reference](/docs/c8s/reference/cli#c8s-verify). ## Requirements On the machine running `c8s verify`: - **Outbound HTTPS to AMD KDS** (`kdsintf.amd.com`), which the verifier uses to fetch the VCEK when verifying a bare SEV-SNP report. Evidence that ships the VCEK inline verifies offline: the LB discovery document, and every `az-snp` envelope (Azure wraps the VCEK in its HCL report). TDX quotes carry their DCAP certification chain inline and verify offline too. - **Network reach to the component** you're verifying (see [Reachability](#reachability-under-kata)). No container runtime is required — verification is entirely in-process. ## What you can verify | `--kind` | Default evidence mode | What you point it at | | --- | --- | --- | | `cds` | `ratls-cert` | the CDS RA-TLS endpoint (its serving cert carries the evidence) | | `lb` | `discovery` | the load balancer's `/v1/discovery` document (cert + evidence, VCEK inline) | | `workload` | `ratls-cert` | a workload's RA-TLS serving endpoint | | `auto` | `auto` (discovery doc first, then RA-TLS cert) | when you don't pass `--kind` | An RA-TLS certificate carries either a raw SEV-SNP report (bare-metal SNP) or a self-describing evidence envelope (`az-snp`, `gcp-snp`, `tdx`, `az-tdx`). A raw TDX report with no envelope is rejected — read that component's discovery document instead. Override the chosen mode with `--mode` (`ratls-cert`, `discovery`, `attestation-endpoint`, or `auto`). You can also skip the network entirely and [verify saved evidence](#verify-saved-evidence) with `--from-file`. **Don't use `--mode attestation-endpoint` against the `tls-lb`.** That mode GETs `/.well-known/c8s/attestation` with a fresh nonce and expects REPORTDATA to commit `SHA-384(x25519 ‖ mlkem768 ‖ nonce)`. The `tls-lb` attestation sidecar commits an identity transcript instead (see [Consumer verification](/docs/c8s/concepts/verification-model)), which this mode does not reconstruct, so the check fails on a binding mismatch. Verify the LB with `--kind lb` (discovery mode). ## Verify the CDS `c8s cds verify` is a shorthand for `c8s verify --kind cds` (default port `8443`); `--mode` stays `auto`, which resolves to `ratls-cert` for a CDS target. The CDS serves [RA-TLS](/docs/c8s/concepts/trust-root#verifying-the-cds-itself-ra-tls): its serving certificate carries the TEE evidence in an X.509 extension, and `verify` trusts *that* attestation, not the certificate chain. ```bash # c8s-cds is headless under Pod-as-CVM, so read its endpoint IP and dial the pod directly # from somewhere with cluster-network reach (a node, or over a VPN): CDS_IP=$(kubectl get endpoints c8s-cds -n c8s-system -o jsonpath='{.subsets[0].addresses[0].ip}') c8s cds verify "https://$CDS_IP:8443" --measurements ``` `kubectl port-forward` also works: Kata guests bake `C8S_MESH_INBOUND_PASSTHROUGH=tcp:8443`, which is what keeps the CDS front door reachable by clients with no mesh certificate. Pass `--server-name` to set SNI when you forward to `localhost`. (`kubectl exec` and `kubectl logs` *are* denied by the locked guest policy — only the `--debug` guest image allows them, and it has a different launch measurement.) A PKI / SAN mismatch when you dial the raw IP is expected and fine — `verify` trusts the attestation embedded in the serving cert, not the hostname on the certificate. A successful run prints: ```text ✓ VERIFIED (attestation-go backend) source: RA-TLS serving certificate at 10.42.0.153:8443 verified at: 2026-07-31T09:14:22Z platform: snp measurement: f894e79d20ab98ba8fc878425739d1b5900999835fe3f94c90fa8603e0636325c131192eb0005f5d3b8134e363d434a3 TCB: bootloader=12 tee=0 snp=28 microcode=28 debug=false smt=false cert sha256: 0109d41e0907ab4ecd5823b5d08c3a6b5d69751322338e28c78f8378c56f92f9 binding: REPORTDATA binds the certificate public key (no per-request nonce — not a freshness proof) note: freshness NOT proven (no per-request nonce bound) ``` ## Check the operator keys CDS pins For a `cds` target the verdict also reports the **pinned operator keys** — SHA-256 fingerprints of the public keys authorized to write the [image allowlist](/docs/c8s/concepts/image-allowlist) — fetched from `GET /operator-keys` over a connection pinned to the attested serving certificate, so the list cannot be substituted in transit: ```text operator keys (allowlist writes; CDS-reported config, NOT covered by the measurement): sha256: ``` As the output says, the key list is CDS-reported config, **not** part of the launch measurement. Reporting it alone is visibility, not proof. Turn it into a check by passing your own bundle — the same public keys you pinned at `c8s install --operator-keys`: ```bash c8s cds verify "https://$CDS_IP:8443" \ --measurements \ --operator-keys operator.pub ``` Verification then **fails** unless the key set CDS serves matches the bundle exactly, and it fails closed: an endpoint that errors on `/operator-keys` cannot dodge the check. A `404` (allowlist writes disabled) is not an error — it compares as the empty set, so `--operator-keys` still catches a CDS that quietly dropped its keys. The fingerprints are over the PKIX/SPKI DER, so you can reproduce one locally: ```bash openssl pkey -pubin -in operator.pub -outform DER | sha256sum ``` **This check only protects the verifier that runs it.** CDS's arguments are host-supplied: a control plane can restart CDS with a different operator key set, and nothing in the cluster notices. Run `c8s cds verify --measurements --operator-keys` continuously from CI — not once at bootstrap — and gate public ingress on it passing. Without `--kind cds` the fetch is skipped rather than silently omitted, and the verdict says so: `operator-keys cross-check skipped: target kind is not cds (use --kind cds to enable)`. With writes disabled it reads `operator keys: endpoint reports no pinned operator keys (allowlist writes disabled)`. ## Verify the load balancer The LB exposes its evidence over a plain HTTPS **discovery document** (the same one the browser library reads), so no special network path is needed: ```bash c8s verify "https://lb.example.com:443" --kind lb --measurements ``` `--kind lb` selects `discovery` mode, which GETs `/v1/discovery` (override the path with `--discovery-path`). The discovery doc ships the VCEK inline, so this works even without KDS egress. ## Verify a workload A workload that serves RA-TLS is verified like the CDS — point at its serving endpoint and pin its launch digest: ```bash c8s verify "https://:" --kind workload --measurements ``` Under Node-as-CVM every pod on a node shares that node's launch digest, so the measurement alone does not tell you *which* pod answered. Two extra pins narrow it: ```bash c8s verify "https://:" --kind workload \ --measurements \ --mesh-ca mesh-ca.pem \ --sandbox-id ``` - `--mesh-ca` takes a PEM bundle; the target's leaf must chain to it. That chain is the only thing that authenticates the sandbox ID, because the ID rides the leaf's signed area and is never folded into REPORTDATA. - `--sandbox-id` pins the CRI pod sandbox the leaf names (containerd emits 64 hex characters). It **requires** `--mesh-ca` and errors out without it, rather than pinning a string the presenter chose. Both need a certificate to inspect, so they only work in a cert mode — not with `--mode attestation-endpoint`. Whenever the evidence verifies and the leaf carries an ID, the verdict reports it with a note naming what stands behind it, so an unqualified ID never reads as attested: ```text sandbox id: 4c9a...e17b verified: the leaf chains to the supplied mesh CA ``` Without `--mesh-ca` the same line reads `not verified: CDS's signature on the leaf vouches for this ID; pass --mesh-ca to check it`. ## Verify saved evidence To verify evidence you've already captured — a saved RA-TLS PEM certificate or an attestation-response JSON — pass `--from-file` and skip the dial: ```bash c8s verify --from-file cds-cert.pem --measurements ``` For a *bare* evidence file with no transport to bind REPORTDATA to, supply the expected value with `--expected-report-data ` (1–64 bytes). Pass the anchor **exactly as the producer bound it** — for c8s bindings that is the unpadded 48-byte SHA-384. Do not zero-pad it to 64 yourself: the hardware-report verifiers pad per platform, and the Azure vTPM verifiers compare the value raw, so a pre-padded anchor fails there. The flag does not apply to a certificate (its binding is the certificate key) and is rejected on one. ## Pin the launch measurement This is the part that makes verification mean anything. With **no** `--measurements`, the command still runs and reports the report's digest, but prints an **UNSAFE** warning — *any* genuine TEE is accepted: ```text WARNING: no --measurements pinned — any genuine TEE is accepted (UNSAFE for production) ``` Pin one or more allowed SHA-384 launch digests so only your exact, audited image passes: ```bash # repeatable / comma-separated c8s cds verify "https://$CDS_IP:8443" --measurements , # or from a file, one hex digest per line c8s cds verify "https://$CDS_IP:8443" --measurements-file digests.txt ``` These are the same digests described under [Obtaining launch measurements](/docs/c8s/how-to/obtain-launch-measurements) — for the CDS, pin the CDS's own launch digest. You can additionally require minimum TCB component versions (`--min-tcb-bootloader`, `--min-tcb-tee`, `--min-tcb-snp`, `--min-tcb-microcode`, each `0`–`255`) and reject debug-enabled guests by leaving `--allow-debug` off (the default). ## Use it in CI The exit codes are a **stable contract**, so a wrong measurement is distinguishable from an unreachable endpoint: | Code | Meaning | | --- | --- | | `0` | verified | | `1` | usage error | | `2` | evidence obtained, but verification / policy failed (e.g. wrong measurement) | | `3` | evidence unavailable (unreachable / unparseable / collateral not fetchable) | Pair that with `-o json` for a machine-readable verdict: ```bash c8s cds verify "https://$CDS_IP:8443" --measurements-file digests.txt \ --operator-keys operator.pub -o json ``` ```json { "verified": true, "verified_at": "2026-07-31T09:14:22Z", "backend": "attestation-go", "source": "RA-TLS serving certificate at 10.42.0.153:8443", "fresh": false, "binding": "REPORTDATA binds the certificate public key (no per-request nonce — not a freshness proof)", "platform": "snp", "measurement": "f894e79d20ab98ba8fc878425739d1b5900999835fe3f94c90fa8603e0636325c131192eb0005f5d3b8134e363d434a3", "current_tcb": "bootloader=12 tee=0 snp=28 microcode=28", "cert_sha256": "0109d41e0907ab4ecd5823b5d08c3a6b5d69751322338e28c78f8378c56f92f9", "measurement_pinned": true, "operator_keys": ["9d1f...c204"] } ``` `operator_keys` carries the hex SHA-256 fingerprints, or `operator_keys_note` explains why they were not fetched. A leaf with a sandbox ID adds `sandbox_id` and `sandbox_id_note`. On failure, `verified` is `false` and `error` names the check that failed. ### Reading the verdict field by field | Field | What it tells you | | --- | --- | | `verified` | The verdict — the vendor signature chain, the REPORTDATA binding, the debug-guest check, and any TCB floor all passed. **Read it together with `measurement_pinned`.** | | `backend` | Who produced the verdict: `attestation-go`, the in-process Go port of the `attestation-rs` engine the cluster itself runs. | | `source` | Where the evidence came from — an RA-TLS serving cert, an LB discovery document, or a `--from-file`. | | `platform` | The attested TEE platform: `snp`, `az-snp`, `gcp-snp`, `tdx`, or `az-tdx`. | | `measurement` | The **SHA-384 launch digest the hardware actually reported.** This is the value you pin — compare it to the component's published or [recomputed](#independently-recompute-the-measurement) digest. | | `current_tcb` | The platform's current TCB (security version numbers). Enforce minimums with `--min-tcb-bootloader` / `-tee` / `-snp` / `-microcode`. SNP only. | | `fresh` / `binding` | Freshness. REPORTDATA binds the **certificate's public key**, not a live nonce — so it proves *"this key was minted in a TEE with this measurement,"* not *"the enclave answered just now."* | | `cert_sha256` | SHA-256 of the serving cert the evidence was bound to — worth recording in an audit log. | | `measurement_pinned` | Whether you supplied `--measurements`. **`false` means nothing was pinned.** | **`verified: true` with `measurement_pinned: false` is the deceptive combination.** It means *"a genuine TEE,"* **not** *"the image you audited."* Any attested enclave running any code passes identically. In text mode the command prints an explicit `WARNING: no --measurements pinned — any genuine TEE is accepted (UNSAFE for production)`. ## Independently recompute the measurement The "spicier" cross-check: rather than trusting the digest you were handed, **recompute it yourself** from the launch components and confirm that what a live node reports matches what the code *should* produce. The worked example below is AMD SEV-SNP, using [`sev-snp-measure`](https://github.com/virtee/sev-snp-measure); Intel TDX is covered at the end of the section. ### Recompute the expected digest From the launch components — the OVMF firmware, the guest `vmlinuz`, and the kernel cmdline (which carries the dm-verity `root_hash`) — at the **same vCPU count used at launch** (c8s measures at 1 vCPU for a stable digest): ```bash # illustrative — see the sev-snp-measure docs for the exact flags for your image sev-snp-measure --mode snp --vcpus 1 \ --ovmf OVMF.fd \ --kernel vmlinuz \ --append "" \ --output-format hex ``` A reproducible build yields the same digest published in the image manifest. A mismatch means the image isn't what you think it is. ### Pin it and verify the live component Feed the recomputed digest straight into `c8s verify`. If the live node's report carries a different launch digest, verification fails with exit code `2`: ```bash c8s cds verify "https://$CDS_IP:8443" --measurements ``` This closes the loop end-to-end: *the code you audited → the digest you computed → the digest the hardware actually measured.* Running once without `--measurements` to read what a node reports is fine for discovery — but always pin the **recomputed or published** value, or you've only achieved trust-on-first-use. **On Intel TDX** the pinned value is the **`MRTD`** — the TD's build-time measurement, the same 48-byte SHA-384 shape as the SNP launch digest, and what `--measurements` compares for a TDX target. Predict it from the TD's virtual firmware with Intel's TDX measurement tooling, then pin the predicted value exactly as above. The runtime measurement registers (`RTMR`s), which record later boot components, are **not** replayed by the c8s RA-TLS verify path — the TDX event log is deliberately kept out of the serving certificate (it can be fetched out-of-band) — so on TDX this cross-check covers the `MRTD`. The `--min-tcb-*` floor is SNP-only and is not enforced on a TDX target. ## Caveats - **Freshness.** Verifying an RA-TLS serving cert binds REPORTDATA to the certificate key, not a per-request nonce — so it proves *"this key was born in a TEE with this measurement,"* not *"freshly, right now"* (`fresh: false` in the output). The discovery document is the same: its challenge is fixed at issuance time. The CDS's own challenge/attest flow uses a one-time nonce; `c8s verify` of a serving cert or discovery doc does not. - **Revocation.** The verifier checks the vendor signature chain and the TCB floor. It does not fetch Intel TCB-info or CRL collateral, so a TDX quote is not checked against Intel's revocation lists. - **`debug=` and `smt=` in the text output are not populated** — they always print `false`. The debug policy is still enforced inside the verifier (that is what leaving `--allow-debug` off does, and a debug guest fails the run); don't read that line as the proof. - **Reachability under Kata.** Reach each component on its **public / host address**, not the in-cluster ClusterIP. The ClusterIP path goes through the RA-TLS mesh and demands an attested client cert (`tls: certificate required`). The CDS RA-TLS endpoint and the `tls-lb`'s nginx serving port both answer unattested clients on their public address (the `tls-lb` serves `/v1/discovery` there with no client cert), so `c8s cds verify` and `c8s verify ` work with no mesh changes. ## See also - [CLI reference → `c8s verify`](/docs/c8s/reference/cli#c8s-verify) — every flag. - [Obtain launch measurements](/docs/c8s/how-to/obtain-launch-measurements) — where the value you pin comes from. - [The verification model](/docs/c8s/concepts/verification-model) — who verifies what, and the browser path this page does not cover. - [The trust root](/docs/c8s/concepts/trust-root) — how the CDS verifies *clients*. - [The image allowlist](/docs/c8s/concepts/image-allowlist) — what the operator keys authorize. --- # c8s-verify (JavaScript) [`c8s-verify`](https://github.com/confidential-dot-ai/c8s-verify-js) is the client-side library that implements everything above: it verifies a c8s Load Balancer's attestation from a browser or Node, then opens the post-quantum over-encrypted channel that terminates inside the LB's enclave. Evidence is verified **in your browser** by the [`attestation-rs`](https://github.com/confidential-dot-ai/attestation-rs) verifier compiled to WebAssembly, with the AMD roots bundled — no network calls during verification. Its only runtime dependency is `mlkem-wasm` for ML-KEM-768. **Use a build that speaks `c8s-verify/v1`.** The wire protocol is versioned and the LB serves exactly one binding: the identity-bound transcript above. A client that computes the older session-key-only `report_data` fails closed on a binding mismatch. Check the library's `PROTOCOL.md` against the `version` string the attestation bundle carries before pinning a release. ## Prerequisites - **Runtime:** Node ≥ 20, or a modern browser — the library uses WebCrypto and WebAssembly. - **Two values pinned out of band.** Without both you cannot prove you reached *your* cluster, and the library refuses to run: - the LB **measurement** allowlist — one or more SHA-384 launch digests (see [Obtaining launch measurements](/docs/c8s/how-to/obtain-launch-measurements)); - the **mesh CA certificate** (PEM) your cluster's CDS issued — the cluster-unique anchor (see [why cluster identity is pinned](/docs/c8s/concepts/verification-model#why-cluster-identity-is-pinned)). - **A reachable LB** serving the `c8s-verify/v1` endpoints under `/.well-known/c8s/`. The browser verifier covers the **SEV-SNP family only** — bare `snp` and Azure vTPM `az-snp`. Intel TDX (`tdx`, `az-tdx`) is carried by the protocol and served by the LB, but the WASM verifier does not implement it; verify TDX deployments from the [operator path](/docs/c8s/how-to/verify-a-deployment) instead. ## Installation The package name is `c8s-verify` and it is not on the public npm registry yet, so consume it from the [c8s-verify-js](https://github.com/confidential-dot-ai/c8s-verify-js) repo — vendor it into your app or add it as a git submodule — and install its one runtime dependency: ```bash npm install mlkem-wasm ``` The prebuilt WASM verifier ships in the repo, so verification itself makes no network calls. ## Usage The high-level `C8sClient` runs the entire flow — fetch the attestation, verify it, run the handshake, derive the channel — and returns a `Session` whose `fetch` is end-to-end encrypted to the enclave: ```js import { C8sClient } from "c8s-verify"; const client = new C8sClient({ baseUrl: "https://lb.example.com", measurements: [""], // pinned out of band; empty is refused meshCaPem: pinnedMeshCaPem, // pinned cluster anchor; required }); // Generates a 32-byte nonce, fetches the LB attestation, verifies the hardware // evidence, the measurement, the identity transcript, the leaf's chain to the // pinned mesh CA, and the proof signature — then runs the X25519 + ML-KEM-768 // handshake and derives the AES-256-GCM channel. const session = await client.connect(); console.log(session.attestation.measurement, session.attestation.cert.sha256); // Every request on session.fetch is sealed end-to-end to the LB enclave — // method, path, headers and body — underneath whatever TLS terminator is in front. const res = await session.fetch("/v1/chat", { method: "POST", body: prompt }); console.log(res.text()); ``` If any check fails, `connect()` throws a typed `C8sVerifyError` and **no channel is established** (fail-closed) — see [Errors](#errors). ## Configuration `new C8sClient(options)`: | Option | Type | Default | Description | | --- | --- | --- | --- | | `baseUrl` | string | — (**required**) | LB origin, e.g. `https://lb.example.com`. | | `measurements` | string[] | — (**required**) | Accepted launch digests (hex SHA-384). An empty list is rejected. | | `meshCaPem` | string | — (**required**) | Pinned mesh CA (PEM) the LB's leaf must chain to. | | `platform` | string | `"snp"` | Expected TEE platform; `"az-snp"` for Azure vTPM evidence. | | `requireFreshness` | bool | `true` | Require `report_data` to bind the identity transcript. `false` downgrades the check to a `warnings[]` entry the embedding app must inspect (**UNSAFE** for a live channel; for recorded-evidence demos). | | `fetch` | function | `globalThis.fetch` | Custom fetch implementation (e.g. for Node without a global fetch, or to inject headers). | | `wellKnownPrefix` | string | `/.well-known/c8s` | Base path for the protocol endpoints. | | `at` | Date | now | Validity reference time for certificate checks. | #### The session object `client.connect()` resolves to a `Session`: - **`session.attestation`** — the verification result: `measurement`, `platform`, `cert` info (incl. `cert.sha256`), and any `warnings`. - **`session.fetch(path, init?)`** — an over-encrypted request. The **entire** request — method, path, headers, and body — is sealed with AES-256-GCM and POSTed to the tunnel endpoint, so a proxy in front of the LB sees only ciphertext. Resolves to `{ status, headers, bytes, text() }`. ## What it verifies `connect()` (via `verifyAttestation`) performs these checks, failing closed on the first failure: 1. **nonce echo** — the response carries back the exact nonce the client sent. 2. **hardware signature + vendor chain** — verified in WASM against the bundled AMD roots. 3. **measurement ∈ allowlist** — the report's launch digest matches a pinned measurement (case-insensitive hex). 4. **identity transcript** — `report_data` equals `SHA-384` over the length-prefixed `(version, mesh-CA digest, mesh-leaf digest, X25519 pubkey, ML-KEM-768 pubkey, nonce)`. One check proves the evidence is fresh, that the session key was generated inside the attested TEE, *and* which cluster identity it belongs to. 5. **leaf chains to the pinned mesh CA** — cluster identity. 6. **proof of possession** — the ECDSA-SHA384 signature over the transcript verifies under the leaf's public key. The verified transcript is then the HKDF salt, so a channel cannot be derived from checks that did not pass. The browser's bare-`snp` WASM path enforces **fewer** checks than the Go and Rust verifiers the cluster runs: it omits the VMPL-0 requirement, debug-policy rejection, the minimum-TCB floor, and VEK validity/CRL. A browser client would accept a debug-enabled or non-VMPL-0 guest whose measurement is allowlisted. See [Limitations](/docs/c8s/concepts/limitations). ## Errors Every failure throws a typed `C8sVerifyError` carrying a `.code` (and `.details`). The codes mirror the c8s server error envelope where they overlap: | Code | Meaning | | --- | --- | | `invalid_request` | Bad client arguments (e.g. missing `baseUrl`, empty measurement allowlist). | | `nonce_mismatch` | The response did not echo the nonce that was sent. | | `verification_failed` | Hardware signature / vendor chain or platform check failed. | | `report_data_mismatch` | `report_data` did not match the expected transcript. | | `measurement_denied` | The launch digest is not in the allowlist. | | `invalid_cert` / `cert_chain` | The leaf is malformed, or did not chain to the mesh CA. | | `identity_binding` | `meshCaPem` is missing, or the proof of possession failed. | | `key_binding` | The session key is not bound to the attested report. | | `channel_error` | The handshake or an over-encrypted request failed. | | `unsupported` | Unsupported platform or protocol feature. | The server side answers in the same envelope, with `binding_unavailable` when the LB has no mesh identity credentials configured and `attestation_unavailable` when it cannot reach its attestation-api. ## Lower-level: verifying bare evidence If you obtain SNP evidence through your own transport (e.g. a [discovery document](/docs/c8s/concepts/trust-root)) rather than the `c8s-verify/v1` challenge/response bundle, use `verifyEvidence`. It runs the same hardware verification and measurement / platform checks, plus a `report_data` binding when you pass `expectedReportData` — with no bundle, nonce, session key, or certificate required (do any mesh-CA chaining yourself): ```js import { verifyEvidence } from "c8s-verify"; const r = await verifyEvidence(evidence /* { attestation_report, cert_chain:{ vcek } } */, { generation: "genoa", // "milan" | "genoa" | "turin" (required for bare snp) measurements: [""], expectedReportData, // optional Uint8Array, exactly as the producer bound it }); console.log(r.measurement, r.reportDataMatch, r.claims); ``` Pass `expectedReportData` **unpadded** — the anchor the producer bound (48 bytes for c8s bindings), not the zero-padded 64-byte hardware field. The raw WASM entrypoints `verifySnp` and `verifyAzSnp` are also exported for full control. ## Try it offline The repo ships a self-contained mock LB so you can run the whole flow with no TEE: ```bash npm install npm run gen-fixtures # openssl mesh CA + leaf, copies recorded SNP evidence npm run demo # serves the mock LB + browser demo on http://localhost:8799 ``` Open the page and click **Run verification**; each step shows green or red, and a *Tamper with evidence* toggle flips a byte of the signed report to demonstrate failing closed. The recorded evidence is real hardware-signed SNP evidence, so the signature, measurement, certificate chain, and post-quantum channel are all genuine — only the live `report_data` binding is simulated, which is the same downgrade `cds-attest --evidence-fixture` serves and why that flag is **dev only**. ## See also - [The verification model](/docs/c8s/concepts/verification-model) — the design this library implements, and why it pins two values. - [Verified chat over confidential vLLM](/docs/c8s/tutorials/verified-chat) — the library end to end against a real cluster. - [Verify a deployment](/docs/c8s/how-to/verify-a-deployment) — the operator-side path, for TDX and for CI. --- # CDS HTTP API The **CDS** (Certificate Distribution Service) serves one HTTP API over [RA-TLS](/docs/c8s/concepts/trust-root#verifying-the-cds-itself-ra-tls) on port `8443`. Every flag that configures it is in the [CLI reference](/docs/c8s/reference/cli#c8s-cds). ## Attestation and issuance | Method | Path | Auth | Purpose | | --- | --- | --- | --- | | `POST` | `/authenticate` | none | issue a one-time challenge nonce (`--challenge-ttl`, default `60s`) | | `POST` | `/attest` | challenge + evidence | verify evidence (and an optional sandbox token) and sign a CSR, returning the leaf PEM chain | | `POST` | `/attest-key` | challenge + evidence | verify evidence and issue an **EAR** bound to the caller's key | | `POST` | `/sign-csr` | EAR | sign a CSR for a caller that already holds an EAR bound to the CSR key | | `POST` | `/handoff` | EAR | hand the mesh CA and allowlist snapshot to an attested peer replica; mounted only with `--handoff-measurements` | ## Allowlist | Method | Path | Auth | Purpose | | --- | --- | --- | --- | | `GET` | `/allowlist` | none | the whole document as canonical JSON; returns a weak ETag `W/""` and honors `If-None-Match` (`304` when unchanged) | | `PUT` | `/allowlist` | operator token | **replace** floor and workloads atomically; the CDS assigns the new version; `204` | | `POST` | `/allowlist/digests` | operator token | add one floor entry `{ digest, image }`; `204` | | `DELETE` | `/allowlist/digests` | operator token | delete `{ digests: [...] }` **atomically** — all-or-nothing, `404` if any is missing | | `PUT` | `/allowlist/workloads/` | operator token | create or replace one workload entry, whole; `204` | | `DELETE` | `/allowlist/workloads/` | operator token | delete one workload entry; `404` if absent | A missing or invalid operator token is `401`; a body the CDS cannot parse or validate is `422`. Writes carry their own 1 MiB body cap, independent of the `--max-request-size` that bounds the attestation endpoints. Enforcers poll `GET /allowlist` and use the ETag to avoid redundant work — the version only changes when the set actually changes (`nriImagePolicy.refresh.interval`, default `30s`). ## Secrets | Method | Path | Auth | Purpose | | --- | --- | --- | --- | | `GET` | `/secrets/` | mesh leaf + sandbox token | read a released value; `404` when the path is ungranted *or* absent | | `POST` | `/secrets/` | mesh leaf + sandbox token | mint and store 32 random bytes at an empty path; `409` when another caller won the race | | `PUT` | `/secrets/` | operator token | store an operator-supplied value; create-only unless the caller asked to overwrite | Denials are deliberately opaque — an ungranted path is indistinguishable from a missing one, so the API cannot enumerate the store. The reason goes to the CDS log. Workload routes are rate-limited **per sandbox**, keyed on the ID in the verified client certificate. The CDS declines to serve `/secrets` at all when `--ratls-platform`, `--measurements`, or `--sandbox-inventory-cidr` is missing, or when CA handoff is configured. See [Application secrets](/docs/c8s/concepts/application-secrets#when-the-cds-will-not-serve-secrets-at-all). ## Discovery | Method | Path | Auth | Purpose | | --- | --- | --- | --- | | `GET` | `/operator-keys` | none | the pinned operator public-key PEM bundle; `404` when allowlist writes are disabled | | `GET` | `/.well-known/jwks.json` | none | the EAR-signing JWKS (ES256) | | `GET` | `/ca` | none | the mesh CA bundle | | `GET` | `/readyz` | none | readiness; fails if the attestation-api is unhealthy, or CA validity is below `--min-ca-validity` | ## Operator tokens Every allowlist and operator-supplied secret write carries a **JWT signed with the operator's ECDSA key** (ES256, ES384, or ES512, matching the key's curve), minted fresh per request with a 60-second lifetime. Three claims bind it to that one write: | Claim | Binds | | --- | --- | | `pbh` | SHA-256 of the exact request body | | `htm` | the HTTP method | | `htu` | the URL path | The CDS accepts a mutation only when the signature verifies against a pinned key, `exp − iat` is at most **five minutes** (a server-side cap), the method and path match the request being handled, and the body hash matches a re-hash of the body actually received, compared in constant time. See [Authorizing mutations](/docs/c8s/concepts/image-allowlist#authorizing-mutations) for the trust boundaries of this design, and [Manage the allowlist](/docs/c8s/how-to/manage-the-allowlist) for the CLI that mints them. --- # CLI Reference One section per command, mirroring the flag definitions in the c8s source. Each table lists **every flag** of the command above it. The `c8s` binary also responds to the symlink aliases `get-cert`, `ratls-mesh`, and `nri-image-policy` (each auto-prepends the matching subcommand). {/* flags:install */} ## `c8s install` Install the c8s operator, CRDs, attestation-api, and component charts via Helm. Flags are registered with cobra/pflag. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--namespace` | string | `c8s-system` | No | namespace to install into | | `--release` | string | `c8s` | No | Helm release name | | `-f, --values` | string slice | `nil` | No | values files (repeatable) | | `--wait` | bool | `true` | No | wait for the release to become ready (helm --wait) | | `--install-crds` | bool | `true` | No | install chart CRDs (false passes helm --skip-crds) | | `--webhook-cert-fs-group` | int64 | `65532` | No | fsGroup for injected certificate volume | | `--webhook-cert-key-mode` | string | `0640` | No | octal mode for injected tls.key | | `--webhook-get-cert-renew-interval` | duration | `6h` | No | renewal interval for injected workload certificates | | `--webhook-get-cert-run-as-user` | int64 | `65532` | No | runAsUser for injected get-cert containers | | `--webhook-get-cert-run-as-group` | int64 | `65532` | No | runAsGroup for injected get-cert containers | | `--webhook-get-cert-run-as-non-root` | bool | `true` | No | set runAsNonRoot for injected get-cert containers | | `--single-node` | bool | `false` | No | single-node / single-CVM cluster: clear the dedicated-CDS-node selector and taint toleration so every node is CDS-eligible (no role=cds label or dedicated node needed). Sets `cds.node.selector={}` and `cds.node.tolerations=[]` | | `--cvm-mode` | string | `""` | **Yes** | CVM deployment shape (orthogonal to `--hardware-platform`): `pod` (per-pod confidential VMs via the Kata runtime — every workload pod is a kata CVM; host-side attestation-api/nri/ratls-mesh served by the in-guest counterparts), `node` (generalized node-as-CVM: our own TDX/SNP nodes are themselves confidential VMs, pods run as ordinary processes; attestation-api + nri baked into the node image), `gke` (GKE managed confidential VMs), or `aks` (vTPM /dev/tpm0). All modes render a privileged attestation-api (a hostPath device mount alone does not grant device-cgroup access) | | `--hardware-platform` | string | `sev-snp` | No | CPU-level TEE hardware (orthogonal to `--cvm-mode`): `sev-snp` (/dev/sev-guest) or `tdx` (Intel TDX, /dev/tdx-guest). Under `--cvm-mode=aks` both ride the Azure vTPM (/dev/tpm0, no guest device): `sev-snp` selects the az-snp shape, `tdx` selects az-tdx | | `--debug` | bool | `false` | No | use the kata-guest-base DEBUG guest variant (`-debug`): kubectl logs/exec work on kata pods, but container I/O becomes readable by the untrusted host and the launch measurement differs from the locked image. Requires `--cvm-mode=pod`; development only | | `--image-tag` | string | `""` | No | component image tag to resolve digests at (default: the CLI build version, or `main` for an unstamped build); override to pin a specific branch/tag/release | | `--resolve-digests` | bool | `true` | No | resolve each component image tag to its registry digest (via crane), pin it, and add the resolved images to the NRI allowlist. Pass `--resolve-digests=false` when supplying digests via `-f` | | `--attest` | bool | `true` | No | deploy the tls-lb attestation sidecar that serves `/.well-known/c8s/` (browser and CLI verification via `c8s-verify`); `--attest=false` sets `tlsLb.attest.enabled=false` and omits it | | `--node-cidr` | string slice | `nil` | No | CIDR(s) holding this cluster's node addresses (repeatable / comma-separated); sets `cds.sandboxInventoryCIDRs`, the only addresses CDS dials for a node's admission inventory. Unset, install reads the cluster and emits one host route (`/32` or `/128`) per node, which does not cover nodes added later; pass a range when the node network is separate | | `--measurements` | string slice | `nil` | No | expected hex launch measurement(s) of this cluster's CVM (repeatable / comma-separated), taken from the node image's `manifest.json`. Sets `cds.measurements` and `ratlsMesh.measurements`, pinning the internal mesh from first boot; empty = no pinning (UNSAFE). Rejected with `--cvm-mode=pod`, whose per-pod kata guests carry a different measurement | | `--image-pull-secret` | string | `""` | No | name of an existing registry-credential Secret (kubernetes.io/dockerconfigjson) in the release namespace; the chart appends it to every component's imagePullSecrets, so all pods can pull the c8s images from an authenticated registry (e.g. a private mirror) from first start. The Secret itself is never created or managed by the install — the install fails fast if it is missing or has the wrong type | | `--workload-ref` | string slice | `nil` | No | existing workload to adopt as a confidential workload, as `=//[:]`; repeatable. Kind is any resource exposing a pod template at `spec.template` (deployment, statefulset, daemonset, or an operator CRD such as `.`); install patches the pod template with `confidential.ai/cw=` once c8s is ready. The optional `:` is the tls-lb upstream port, required on the ref `--upstream` selects | | `--upstream` | string | `""` | No | `confidential.ai/cw` id of the adopted `--workload-ref` workload tls-lb routes its catch-all to; derives the mesh-wrapped upstream `c8s-..svc.cluster.local:` from that ref's `:`. Without this or a verified-https `tlsLb.upstream`, tls-lb renders no catch-all route until one is attached | | `--operator-keys` | string | `""` | No | path to a PEM bundle of operator EC public keys that authorize [`c8s allowlist`](#c8s-allowlist) writes; sets `cds.operatorKeys`. Without it, allowlist writes are disabled (reads still served). See [creating the operator credential](/docs/c8s/how-to/manage-the-allowlist#create-the-operator-credential) | | `--force` | bool | `false` | No | proceed past guarded prompts — currently: install without `--operator-keys` (allowlist writes disabled). Not needed when values are supplied via `-f` | {/* flags:uninstall */} ## `c8s uninstall` Uninstall the c8s Helm release and, for a `--cvm-mode=pod` install, sweep the host-side kata artifacts off every node. `helm uninstall` unwinds the release resources; the host sweep then removes what that path cannot guarantee (`/opt/kata`, the containerd drop-in, the multi-GB `kata-guest-base` image, the RKE2 prep template, and the kata-runtime node labels). Requires the `helm` and `kubectl` CLIs on `PATH`. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--namespace` | string | `c8s-system` | No | namespace the release was installed into | | `--release` | string | `c8s` | No | Helm release name | | `--wait` | bool | `true` | No | wait for the release deletion to complete (helm --wait); the kata host sweep additionally waits for the kata pods to be gone either way | | `--kata-sweep` | bool | `true` | No | after the release is deleted, sweep the kata host artifacts off every kata node via a short-lived privileged DaemonSet. Skipped automatically when the release was installed without `--cvm-mode=pod` | | `--host-sweep-only` | bool | `false` | No | skip the helm uninstall and only run the kata host sweep — for a cluster whose release is already gone (e.g. a previous bare `helm uninstall`) but whose nodes still carry kata artifacts. Uses the chart defaults and the distro detected from the cluster when the release values are unavailable | | `--force` | bool | `false` | No | uninstall even while pods with a kata RuntimeClass are running (they lose their runtime: kata VMs keep running unmanaged but cannot restart) | | `--delete-crds` | bool | `false` | No | also delete the ConfidentialWorkload CRD — this deletes **every** ConfidentialWorkload object in the cluster with it | | `--delete-namespace` | bool | `false` | No | also delete the release namespace (and everything left in it, e.g. an operator-created image pull Secret) | {/* flags:render-values */} ## `c8s render-values` Print the resolved Helm values an install would apply, to stdout — **without contacting a cluster**. Useful for feeding a GitOps consumer (e.g. a Flux `HelmRelease`'s `valuesFrom`) instead of recomputing digests and device mappings. Unlike `install`, the host distro isn't autodetected — pass `--distro` to pin it. Still needs the registry reachable for the default digest resolution (`--resolve-digests=true`). | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--distro` | string | `""` | No | host Kubernetes distro (`k8s` \| `rke2`); install autodetects this, render-values can't, so pass it when you need it pinned (unset leaves the chart default) | | `--single-node` | bool | `false` | No | single-node / single-CVM cluster: clear the dedicated-CDS-node selector and toleration (`cds.node.selector={}`, `cds.node.tolerations=[]`) | | `--cvm-mode` | string | `""` | **Yes** | CVM deployment shape (orthogonal to `--hardware-platform`): `pod` (per-pod kata CVMs; disables host-side ratls-mesh/attestation-api/nri-image-policy), `node` (generalized node-as-CVM native TEE device), `gke` (GKE managed CVMs), or `aks` (vTPM /dev/tpm0) | | `--hardware-platform` | string | `sev-snp` | No | CPU-level TEE hardware (orthogonal to `--cvm-mode`): `sev-snp` (/dev/sev-guest) or `tdx` (Intel TDX, /dev/tdx-guest). Under `--cvm-mode=aks` both ride the Azure vTPM (/dev/tpm0): `sev-snp` selects az-snp, `tdx` selects az-tdx | | `--measurements` | string slice | `nil` | No | expected hex launch measurement(s) of this cluster's CVM (repeatable / comma-separated), from the node image's `manifest.json`; emits `cds.measurements` + `ratlsMesh.measurements`. Empty = no pinning (UNSAFE). Rejected with `--cvm-mode=pod` | | `--debug` | bool | `false` | No | use the kata-guest-base DEBUG image variant (requires `--cvm-mode=pod`) | | `--resolve-digests` | bool | `true` | No | resolve each component image tag to its registry digest (via crane), pin it, and enable the NRI allowlist derivation | | `--image-tag` | string | `""` | No | component image tag to resolve digests at (default: the CLI build version, or `main`) | | `--image-pull-secret` | string | `""` | No | name of an existing dockerconfigjson Secret the chart wires into every component's imagePullSecrets | | `--operator-keys` | string | `""` | No | path to a PEM bundle of operator EC public keys that authorize [`c8s allowlist`](#c8s-allowlist) writes; the file's **content** is embedded as `cds.operatorKeys` in the emitted values (the chart value is PEM content, never a path) | | `--install-crds` | bool | `true` | No | emit values for chart CRDs (false sets `statusMirror.enabled=false`, matching `install --install-crds=false`) | | `--workload-ref` | string slice | `nil` | No | adopted workload as `=//[:]`; repeatable. Used here only to derive `--upstream`'s address (render-values patches nothing) | | `--upstream` | string | `""` | No | `confidential.ai/cw` id of the adopted `--workload-ref` workload tls-lb routes its catch-all to; derives `tlsLb.upstream.address` `c8s-..svc.cluster.local:` from that ref's `:` | {/* flags:operator */} ## `c8s operator` Run the c8s controller-manager and admission webhook. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--metrics-bind-address` | string | `:8080` | No | address for Prometheus metrics | | `--health-probe-bind-address` | string | `:8081` | No | address for health/readyz probes | | `--leader-elect` | bool | `true` | No | enable leader election for HA | | `--leader-election-namespace` | string | `c8s-system` | No | namespace holding the leader-election Lease | | `--status-mirror-enabled` | bool | `true` | No | enable CRD-backed ConfidentialWorkload status mirror controller | | `--get-cert-image` | string | `""` | No | image reference the webhook injects for get-cert containers (empty = webhook disabled) | | `--cds-url` | string | `""` | No | CDS Service URL the injected get-cert containers POST to | | `--attestation-api-url` | string | `""` | No | attestation-api endpoint (empty = no verification) | | `--cds-measurements` | string slice | `nil` | No | SHA-384 hex launch measurement(s) the injected secret-fetcher container requires CDS to present (repeatable; empty pins none) | | `--exclude-namespaces` | string slice | `nil` | No | extra namespaces the startup reinject sweep skips (mirrors `webhook.extraExcluded`) | | `--webhook-config-name` | string | `""` | No | MutatingWebhookConfiguration to patch caBundle (empty = skip) | | `--webhook-service-name` | string | `""` | No | webhook Service name (defaults to c8s) | | `--webhook-service-namespace` | string | `""` | No | webhook Service namespace (defaults to --leader-election-namespace) | | `--cert-fs-group` | int64 | `65532` | No | fsGroup applied to injected pods when unset (-1 disables mutation) | | `--cert-key-mode` | string | `0640` | No | octal mode for injected tls.key | | `--get-cert-renew-interval` | duration | `6h` | No | renewal interval for injected workload certificates | | `--get-cert-run-as-user` | int64 | `65532` | No | runAsUser for injected get-cert containers | | `--get-cert-run-as-group` | int64 | `65532` | No | runAsGroup for injected get-cert containers | | `--get-cert-run-as-non-root` | bool | `true` | No | set runAsNonRoot for injected get-cert containers | | `--kata-enforce` | bool | `false` | No | inject a kata runtimeClassName into workload pods that don't request one and enforce kata RuntimeClasses (set by the chart under `kata.enabled`) | | `--hardware-platform` | string | `sev-snp` | No | CPU TEE the injected confidential kata classes target: `sev-snp` or `tdx` (set by the chart to match the RuntimeClasses it renders) | | `--workload-claims-host-dir` | string | `""` | No | host directory holding the `nri-image-policy` inventory socket (Node-as-CVM); when set, the webhook mounts it into the `c8s-cert` container and injects `--workload-claims` so `get-cert` redeems a sandbox token | | `--workload-claims-guest` | bool | `false` | No | Kata shape: the inventory is `policy-monitor` inside the guest, reached on guest loopback, so the webhook injects `--workload-claims` with no socket mount | {/* flags:cds */} ## `c8s cds` Run the Certificate Distribution Service (CDS). Requires `--attestation-api-url` and `--allowlist-db`. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--host` | string | `0.0.0.0` | No | listen host | | `-p, --port` | int | `8443` | No | listen port | | `--log-level` | string | `info` | No | log level: debug, info, warn, error | | `--attestation-api-url` | string | `""` | **Yes** | URL of the attestation-api service | | `--ca-common-name` | string | `c8s Mesh CA` | No | common name for the in-memory generated mesh CA | | `--ca-cert-validity` | duration | `8760h` | No | validity period of the in-memory mesh CA certificate | | `--measurements` | string slice | `nil` | No | SHA-384 hex launch measurements allowed to call /attest (empty = no pinning, UNSAFE) | | `--ear-issuer` | string | `cds` | No | EAR (Entity Attestation Result) JWT issuer claim | | `--expected-issuer` | string | `""` | No | EAR JWT issuer claim required on /sign-csr (empty disables) | | `--jwt-clock-skew` | int64 | `30` | No | EAR JWT exp/nbf/iat clock skew tolerance in seconds | | `--max-ttl` | duration | `24h` | No | upper bound on /sign-csr leaf TTL | | `--cert-ttl` | duration | `24h` | No | certificate TTL | | `--challenge-ttl` | duration | `60s` | No | challenge TTL | | `--request-timeout` | duration | `5s` | No | per-request /attest timeout (0 disables) | | `--max-request-size` | int64 | `65536` | No | max request body bytes on write endpoints | | `--read-timeout` | duration | `10s` | No | HTTP server read timeout | | `--read-header-timeout` | duration | `5s` | No | HTTP server read-header timeout | | `--write-timeout` | duration | `10s` | No | HTTP server write timeout | | `--idle-timeout` | duration | `20s` | No | HTTP server idle timeout | | `--max-header-bytes` | int | `1048576` | No | maximum HTTP request header bytes | | `--san-validation` | bool | `true` | No | require CSR IP SANs to equal the request source IP (false rejects CSRs carrying IP SANs) | | `--dns-san-pattern` | string slice | `nil` | No | regex a CSR's DNS SANs may match in full; repeatable, and a SAN passes if it matches any one. The chart always supplies the in-cluster Service DNS pattern and appends a public hostname when tls-lb fronts a routed domain. A CSR carrying DNS SANs is rejected when none are set | | `--allowed-cn-pattern` | string | `""` | No | regex the CSR Subject CN must match in full (empty disables) | | `--readiness-interval` | duration | `10s` | No | readiness check interval | | `--min-ca-validity` | duration | `1h` | No | /readyz fails when the loaded mesh CA has less than this remaining lifetime | | `--allowlist-db` | string | `""` | **Yes** | path to the allowlist SQLite database | | `--allowlist-persistent` | bool | `false` | No | whether `--allowlist-db` is on durable storage; false makes the CDS warn at startup that operator-added digests and the mesh CA do not survive a restart | | `--sandbox-inventory-cidr` | string slice | `nil` | No | CIDR(s) holding the node addresses the CDS may dial for a sandbox's admission inventory (repeatable). Required to accept sandbox tokens; bounds the callback to node addresses only | | `--allowlist-seed` | string | `""` | No | path to a JSON allowlist (version + digests map) seeded into the store at startup before serving; missing digests are added, existing entries are left untouched (empty disables seeding) | | `--operator-keys` | string | `""` | No | path to a PEM bundle of pinned operator EC public keys; /allowlist writes (POST/PUT/DELETE) require an operator token signed by one of them (empty = writes disabled, reads still served) | | `--handoff-measurements` | string slice | `nil` | No | SHA-384 hex launch measurements allowed to pull the mesh CA and allowlist via /handoff; requires `--operator-keys` so both replicas attest the same policy (empty = /handoff disabled) | | `--handoff-peer-url` | string | `""` | No | https URL of a surviving CDS peer to adopt the mesh CA and allowlist from on startup via attested /handoff (empty = generate a fresh CA). When set, startup fails closed if the peer cannot be reached, denies handoff, or attests a different operator-key policy. Pins the peer with `--handoff-measurements` | | `--handoff-peer-timeout` | duration | `2m` | No | deadline for adopting the CA from `--handoff-peer-url` before failing startup | | `--rate-limit` | float64 | `10` | No | max requests per second per source IP on attestation endpoints | | `--rate-burst` | int | `20` | No | max burst size per source IP | | `--rate-limiter-max-entries` | int | `10000` | No | max entries in the per-IP rate limiter | | `--rate-limiter-evict-interval` | duration | `1m` | No | interval for per-IP rate limiter eviction sweep | | `--rate-limiter-idle-timeout` | duration | `5m` | No | idle duration before a per-IP rate limiter entry is evicted | | `--secrets-max-paths` | int | `1024` | No | max distinct secret paths held in memory | | `--secrets-max-value-bytes` | int | `4096` | No | max bytes in one secret value | | `--sandbox-ledger-max-entries` | int | `10000` | No | max sandbox-to-inventory bindings held in memory | | `--token-signer-rotation-interval` | duration | `720h` | No | EAR signing key rotation interval (0 disables) | | `--token-signer-overlap` | duration | `25h` | No | how long a retired EAR key stays in JWKS | | `--token-signer-rotation-jitter` | float64 | `0.1` | No | EAR key rotation jitter | | `--ratls-platform` | string | `sev-snp` | No | TEE platform for the RA-TLS serving cert: `sev-snp` or `tdx` (snp/az-snp/gcp-snp and az-tdx/gcp-tdx aliases normalized). Empty disables TLS — UNSAFE outside tests | | `--ratls-cert-ttl` | duration | `24h` | No | RA-TLS certificate TTL | {/* flags:cds-request-handoff */} ### `c8s cds request-handoff` Drive the client side of the attested mesh-CA handoff (`/handoff`) end to end: generate an in-memory signer key, obtain a TEE-bound EAR for it from the peer's `/attest-key`, pull the recipient-encrypted CA material, and confirm the handed-off CA certificate is the trust root the peer serves on `GET /ca`. Runs standalone as a live-cluster rollout-continuity probe, inside an attested TEE with access to the local attestation-api. The peer admits only the launch measurements pinned in its `--handoff-measurements`. Prints a one-line JSON report on stdout; the pulled CA private key never leaves process memory. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--peer-url` | string | `""` | **Yes** | https URL of the CDS peer to pull the mesh CA from; any other scheme is rejected | | `--attestation-api-url` | string | `""` | **Yes** | URL of the local attestation-api, which produces this caller's evidence | | `--measurements` | string slice | `nil` | **Yes** | SHA-384 hex launch measurement(s) the peer may present (repeatable / comma-separated); pins both its RA-TLS serving cert and its handoff issuer EAR. Handoff has no accept-any mode — an empty pin is a usage error | | `--operator-keys` | string | `""` | **Yes** | PEM bundle of operator EC public keys; its canonical hash is bound into both handoff attestations and must match the peer's | | `--expected-issuer` | string | `cds` | No | EAR JWT issuer claim required on the peer's handoff EAR | | `--timeout` | duration | `2m` | No | overall deadline, including retries while the peer's handoff EAR bootstraps; must be positive | | `--log-level` | string | `info` | No | log level: debug, info, warn, error | Exit codes: `0` verified · `1` usage · `2` handoff or verification failed · `3` endpoint unavailable (unreachable / disabled / still bootstrapping past `--timeout`). {/* flags:allowlist */} ## `c8s allowlist` Read and mutate the CDS-served allowlist that `nri-image-policy` / `policy-monitor` enforce. It has two layers: a **floor** of image digests admitted by digest alone, and named **workload entries** that pin an init/main container set with per-container argv and path policy. Reads are unauthenticated; writes are signed with the operator EC private key whose public half the CDS pins (`cds.operatorKeys`, set by `install --operator-keys`). See [The Allowlist](/docs/c8s/how-to/manage-the-allowlist) for usage. | Subcommand | Arguments | Purpose | | --- | --- | --- | | `list` | — | list the current allowlist floor and workload entries | | `export` | `[file]` | write the full allowlist as canonical JSON to a file (default stdout) for backup or re-upload | | `diff` | `` | show how an allowlist file differs from the live allowlist | | `lint` | `` | validate an allowlist file and report semantic findings; never contacts the CDS | | `inspect-image` | `` | resolve an image reference via crane and print its digest plus baked Entrypoint/Cmd; registry-only | | `add` | ` ` | add a single image digest to the floor (operator key required) | | `remove` | ` [...]` | remove one or more image digests from the floor (operator key required) | | `upload` | `` | atomically replace the entire allowlist — floor and workload entries — with the contents of a file (operator key required) | | `workload list` | — | list workload entries | | `workload get` | `` | print one workload entry as canonical JSON | | `workload apply` | `` | upsert workload entries from a file, each replaced whole (operator key required) | | `workload edit` | `` | fetch a workload entry, edit it in `$EDITOR`, and apply the result (operator key required) | | `workload delete` | ` [...]` | delete one or more workload entries (operator key required) | Persistent flags, registered on every subcommand: | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--url` | string | `""` | **Yes** | tls-lb or direct CDS base URL; required by every subcommand that contacts the CDS (`lint` and `inspect-image` do not). A tls-lb front door is trusted through its discovery document and is accepted only when it serves CDS-issued public TLS (discovery reports `public_tls.mode=cds`); a WebPKI front door is refused, since its public certificate is not bound to the attestation evidence. A direct URL is verified by RA-TLS on the CDS serving cert. Plaintext `http://` is refused without `--insecure` | | `--measurements` | string slice | `nil` | No | allowed SHA-384 hex launch measurement(s) of the attested endpoint (repeatable / comma-separated) — the tls-lb's value for a front door, the CDS's value for a direct URL; empty accepts any attested build (UNSAFE) | | `--measurements-file` | string | `""` | No | file of allowed launch measurements, one hex digest per line | | `--timeout` | duration | `15s` | No | per-request timeout | | `--operator-key` | string | `""` | No | operator EC private key PEM file whose public key is pinned on CDS via `--operator-keys` (env `C8S_OPERATOR_KEY`; the flag wins); required for writes | | `-o, --output` | string | `text` | No | output format: `text` or `json` | | `--insecure` | bool | `false` | No | dev/test only: allow a plaintext http:// CDS URL, skipping RA-TLS attestation of CDS | Per-subcommand flags: | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--exit-code` | bool | `false` | No | `diff` only: exit non-zero when the file and the live allowlist differ | | `--online` | bool | `false` | No | `lint` only: also check each digest exists in its registry via crane | | `--strict` | bool | `false` | No | `lint`, `upload`: treat lint warnings as fatal (lint errors already fail on their own) | | `--dry-run` | bool | `false` | No | `add`, `remove`: print the intended change without calling CDS; `upload`: show the diff without replacing the allowlist; `workload apply`: show the diff without writing any entry | | `--force` | bool | `false` | No | `upload` only: upload even if core c8s components are missing from the file | | `--require` | string slice | `nil` | No | `upload` only: component identifiers that must appear in the uploaded image refs (overrides the default set `cds`, `ratls-mesh`, `nri-image-policy`, `attestation-api`, `nginx`) | {/* flags:secrets */} ## `c8s secrets` Write operator-supplied values into the CDS secret store, and report why a sandbox does or does not receive them. CDS releases a value to a pod when the containers running in that pod's sandbox match a workload entry whose `secrets` grant covers the path; the grant is written with [`c8s allowlist workload apply`](#c8s-allowlist). Values are held in the CDS process and nowhere else, so a CDS restart empties the store. See [Secrets](/docs/c8s/concepts/application-secrets). | Subcommand | Arguments | Purpose | | --- | --- | --- | | `put` | `` | store a value at a secret path, read from stdin or `--from-file` | | `explain` | — | print the release decision CDS reaches for one sandbox | `put` sends the bytes exactly as read, including any trailing newline, and prints the byte count. Both subcommands sign their request with the operator EC private key whose public half the CDS pins (`cds.operatorKeys`, set by `install --operator-keys`). Persistent flags, registered on every subcommand: | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--url` | string | `""` | **Yes** | tls-lb or direct CDS base URL. A tls-lb front door is trusted through its discovery document and accepted only when that reports `public_tls.mode=cds`; a direct URL is verified by RA-TLS on the CDS serving cert. Plaintext `http://` is refused without `--insecure` | | `--measurements` | string slice | `nil` | No | allowed SHA-384 hex launch measurement(s) of the attested endpoint (repeatable / comma-separated) — the tls-lb's value for a front door, the CDS's value for a direct URL; empty accepts any attested build (UNSAFE) | | `--measurements-file` | string | `""` | No | file of allowed launch measurements, one hex digest per line | | `--timeout` | duration | `15s` | No | per-request timeout | | `--operator-key` | string | `""` | No | operator EC private key PEM file whose public key is pinned on CDS via `--operator-keys` (env `C8S_OPERATOR_KEY`; the flag wins). Both subcommands sign, so one of the two must be set | | `--insecure` | bool | `false` | No | dev/test only: allow a plaintext http:// CDS URL, skipping RA-TLS attestation of CDS | Per-subcommand flags: | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--from-file` | string | `""` | No | `put` only: read the value from this file instead of stdin | | `--overwrite` | bool | `false` | No | `put` only: replace a value already at the path. Without it an occupied path is refused, and the error names what is there | | `--dry-run` | bool | `false` | No | `put` only: print the intended change without calling CDS | | `--sandbox` | string | `""` | **Yes** | `explain` only: sandbox ID to report on. It is carried on the pod's certificate; `c8s verify` prints it | | `--json` | bool | `false` | No | `explain` only: print the raw report as JSON | {/* flags:volume */} ## `c8s volume` Build an encrypted volume and store its key in the CDS secret store. `create` packages `--source` into an erofs image, formats a dm-verity tree over it, encrypts the pair to `--out`, writes the key blob to `--escrow-out`, and puts the key at `--path` in the store. It then prints the pod annotations and the allowlist grant to apply; it modifies no workload. The image is ciphertext and can reach the node by any route, including through the untrusted host, attached as a raw block device with virtio serial `c8s-vol-`. Needs `mkfs.erofs` and `veritysetup` on `PATH`. The key exists in exactly two places — the CDS process and the escrow file — and a key path is create-only: CDS refuses a `--path` that already holds a value. See [Volumes](/docs/c8s/concepts/encrypted-volumes). | Subcommand | Arguments | Purpose | | --- | --- | --- | | `create` | — | build an encrypted volume image and store its key | Persistent flags, registered on every subcommand: | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--url` | string | `""` | **Yes** | tls-lb or direct CDS base URL, verified as for [`c8s secrets`](#c8s-secrets); not read under `--dry-run` | | `--measurements` | string slice | `nil` | No | allowed SHA-384 hex launch measurement(s) of the attested endpoint (repeatable / comma-separated); empty accepts any attested build (UNSAFE) | | `--measurements-file` | string | `""` | No | file of allowed launch measurements, one hex digest per line | | `--timeout` | duration | `15s` | No | per-request timeout | | `--operator-key` | string | `""` | No | operator EC private key PEM file whose public key is pinned on CDS via `--operator-keys` (env `C8S_OPERATOR_KEY`; the flag wins); required for the store write | | `--insecure` | bool | `false` | No | dev/test only: allow a plaintext http:// CDS URL, skipping RA-TLS attestation of CDS | `create` flags: | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--name` | string | `""` | **Yes** | volume name; forms the device serial `c8s-vol-`, so at most 12 characters | | `--source` | string | `""` | **Yes** | directory whose contents become the volume | | `--out` | string | `""` | **Yes** | path the encrypted image is written to; must not exist | | `--path` | string | `""` | **Yes** | secret-store path for the key, e.g. `/tenant-a/volumes/weights`; absolute, clean, no wildcards | | `--escrow-out` | string | `""` | **Yes** | path the key blob is written to, mode `0600`; must not exist. It is the only copy of the key outside CDS | | `--node` | string | `""` | No | node holding the device; emitted as a `nodeSelector` on the printed annotations | | `--work-dir` | string | `""` | No | directory for build intermediates (default: a temp dir); they are removed either way | | `--dry-run` | bool | `false` | No | build the image and write the escrow file, but do not call CDS | {/* flags:cds-attest */} ## `c8s cds-attest` Run the **tls-lb attestation + over-encryption sidecar** (the `c8s-verify/v1` server side). Fronted by the tls-lb nginx (chart flag `tlsLb.attest.enabled`), it serves the LB's attestation at `/.well-known/c8s/attestation`, runs the post-quantum handshake, and forwards decrypted tunnel traffic to the backend. It's a chart-managed component, not normally run by hand — see [Consumer Verification](/docs/c8s/concepts/verification-model) for the protocol. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--host` | string | `127.0.0.1` | No | listen host (loopback: nginx proxies to it) | | `-p, --port` | int | `8800` | No | listen port | | `--log-level` | string | `info` | No | log level: debug, info, warn, error | | `--attestation-api-url` | string | `""` | No | attestation-api URL (production evidence source) | | `--platform` | string | `snp` | No | TEE platform: `snp` \| `az-snp` \| `az-tdx` \| `tdx` | | `--generation` | string | `genoa` | No | AMD processor generation for the browser's bare-SNP verifier (`--platform snp` only, ignored otherwise): `milan` \| `genoa` \| `turin` | | `--serving-cert-file` | string | `""` | No | path to the LB serving-leaf PEM (the cert nginx presents); enables the tls-cert attestation binding (`?pq=false`, `report_data` binds the leaf's SPKI), re-read per request to follow get-cert rotation | | `--mesh-identity-cert-file` | string | `""` | No | TEE-held mesh leaf PEM for the identity-bound post-quantum binding, re-read per request. Separate from `--serving-cert-file`, which may name a host-visible public TLS credential | | `--mesh-identity-key-file` | string | `""` | No | private key for `--mesh-identity-cert-file`, re-read per request | | `--mesh-identity-ca-file` | string | `""` | No | mesh CA bundle that issued the identity leaf, re-read per request. All three `--mesh-identity-*` flags must be set; otherwise the identity-bound post-quantum handshake returns `501 binding_unavailable` | | `--evidence-fixture` | string | `""` | No | **dev only**: serve recorded TEE evidence from this file instead of the attestation-api | | `--session-ttl` | duration | `5m` | No | pending-handshake TTL and established-session idle TTL | | `--read-header-timeout` | duration | `5s` | No | HTTP read-header timeout | | `--upstream` | string | `""` | No | backend base URL to forward decrypted traffic to (`http://` rides the RA-TLS mesh; `https://` does mTLS). Empty uses an echo backend (demo) | | `--upstream-ca` | string | `""` | No | PEM CA bundle to verify an https upstream (the mesh CA) | | `--upstream-cert` | string | `""` | No | client cert presented to an https upstream (the CDS-issued LB cert) | | `--upstream-key` | string | `""` | No | client key for `--upstream-cert` | | `--upstream-server-name` | string | `""` | No | SNI / verification name for an https upstream | {/* flags:allowlist-proxy */} ## `c8s allowlist-proxy` Publish the CDS allowlist API through tls-lb. Public TLS terminates at the tls-lb nginx; this process makes the second trust hop, verifying CDS's RA-TLS serving certificate before forwarding the request. Path, query, and `Authorization` header are forwarded byte-for-byte, so the operator token's method/path/body binding survives the hop. It routes `/allowlist` and `/allowlist/` to CDS, answers `/healthz` itself, and binds loopback only. A chart-managed component, not run by hand — see [`c8s allowlist`](#c8s-allowlist) for the client side. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--host` | string | `127.0.0.1` | No | listen host; must be a loopback IP (nginx is the public listener) | | `-p, --port` | int | `8801` | No | listen port; 1–65535 | | `--cds-url` | string | `""` | **Yes** | CDS base URL; must be an https (RA-TLS) origin with no credentials, path, query, or fragment | | `--cds-measurements` | string slice | `nil` | No | allowed CDS SHA-384 hex launch measurement(s) (repeatable / comma-separated); empty accepts any attested CDS (UNSAFE) | | `--attestation-api-url` | string | `""` | No | attestation-api URL used to verify CDS evidence | | `--request-timeout` | duration | `30s` | No | timeout for one request to CDS; must be positive | | `--read-header-timeout` | duration | `5s` | No | HTTP request-header timeout; must be positive | {/* flags:verify */} ## `c8s verify` Verify a deployed component's TEE attestation evidence (AMD SEV-SNP or Intel TDX) against the hardware signature chain plus a measurement, TCB, and policy check. Verification runs **in-process** with `attestation-go` (the Go port of the attestation-rs engine the cluster runs) — no container runtime is needed; a bare SEV-SNP report needs outbound HTTPS to AMD KDS (`kdsintf.amd.com`) to fetch its VCEK, bounded by `--timeout`. Takes a `[target]` (a URL or `host:port`) as its positional argument, or `--url`. `c8s cds verify` is the same command with CDS presets (`--kind cds`, default port `8443`); `--mode` stays `auto`, which resolves to `ratls-cert` for a CDS target. See the [verification guide](/docs/c8s/how-to/verify-a-deployment) for the end-to-end workflow. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--url` | string | `""` | No | target URL or host:port (alternative to the positional argument) | | `--kind` | string | `auto` | No | component being verified: `cds`, `lb`, `workload`, or `auto` | | `--mode` | string | `auto` | No | evidence mode: `auto`, `ratls-cert`, `discovery`, or `attestation-endpoint` | | `--discovery-path` | string | `/v1/discovery` | No | path of the LB discovery document (discovery mode) | | `--server-name` | string | `""` | No | TLS SNI server name (for port-forward / routed domains) | | `--timeout` | duration | `15s` | No | per-attempt timeout (evidence fetch and AMD KDS collateral fetch) | | `--from-file` | string | `""` | No | verify evidence from a saved PEM certificate or attestation-response JSON instead of dialing | | `--measurements` | string slice | `nil` | No | allowed SHA-384 hex launch measurement(s) (repeatable / comma-separated); empty = no pinning (UNSAFE) | | `--measurements-file` | string | `""` | No | file of allowed launch measurements, one hex digest per line | | `--operator-keys` | string | `""` | No | PEM bundle of expected operator public keys; verification fails unless the key set the attested target serves at `/operator-keys` matches it (`--kind cds` targets) | | `--sandbox-id` | string | `""` | No | expected CRI pod sandbox ID on the target's leaf; requires `--mesh-ca` | | `--mesh-ca` | string | `""` | No | PEM bundle of the CDS mesh CA; when set, the target's leaf must chain to it, which is what authenticates the reported sandbox ID | | `--allow-debug` | bool | `false` | No | accept debug-enabled guests | | `--min-tcb-bootloader` | uint | `0` | No | minimum bootloader TCB component (0–255) | | `--min-tcb-tee` | uint | `0` | No | minimum TEE TCB component (0–255) | | `--min-tcb-snp` | uint | `0` | No | minimum SNP firmware TCB component (0–255) | | `--min-tcb-microcode` | uint | `0` | No | minimum microcode TCB component (0–255) | | `--expected-report-data` | string | `""` | No | hex REPORTDATA / TPM-nonce anchor override for bare evidence files (1–64 bytes, exactly as bound by the producer) | | `-o, --output` | string | `text` | No | output format: `text` or `json` | | `--show-evidence` | bool | `false` | No | print the raw report fields | Exit codes are a CI contract: `0` verified · `1` usage · `2` verification/policy failed · `3` evidence unavailable (unreachable / unparseable). {/* flags:get-cert */} ## `c8s get-cert` Obtain a signed certificate via the CDS attestation flow. Requires `--cds-url`, `--attestation-api-url`, and `--san`. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--cds-url` | string | `""` | **Yes** | URL of the CDS service (e.g. https://cds:8443) | | `--cds-measurements` | string | `""` | No | comma-separated SHA-384 hex launch measurements for CDS RA-TLS verification (empty = accept any attested CDS) | | `--attestation-api-url` | string | `""` | **Yes** | URL of the local attestation-api (e.g. http://localhost:8400) | | `-o, --out` | string | `""` | No | path to write the signed certificate chain PEM (stdout if omitted) | | `--ca-out` | string | `""` | No | path to write just the mesh CA bundle PEM (the issuer certs trailing the leaf in the CDS chain), e.g. for nginx to serve at a discovery endpoint without a separate ConfigMap | | `--key` | string | `""` | No | path to a PEM private key to use for the CSR (ephemeral if omitted) | | `--key-out` | string | `""` | No | path to write the generated private key PEM (ephemeral keys only) | | `--key-mode` | string | `0600` | No | octal mode for generated private key | | `--san` | string | `""` | **Yes** | Subject Alternative Name for the certificate (IP address or hostname) | | `-v, --verbose` | bool | `false` | No | enable debug logging | | `--renew-interval` | duration | `0` | No | re-obtain the certificate at this interval (0 = run once and exit) | | `--initial-retry-timeout` | duration | `2m` | No | retry the first certificate request in-process for up to this long before failing, so a transient CDS/mesh outage during a roll does not crash the init container into kubelet backoff (0 = try once) | | `--initial-retry-interval` | duration | `2s` | No | delay between in-process retries of the first certificate request | | `--reload-nginx` | bool | `true` | No | SIGHUP nginx after certificate renewal or watched file changes | | `--continue-on-initial-error` | bool | `false` | No | in renewal mode, keep running when the first certificate request fails | | `--reload-watch` | string array | `nil` | No | file path to poll for changes and reload nginx when it changes (repeatable) | | `--reload-watch-interval` | duration | `1m` | No | poll interval for --reload-watch paths | | `--discovery-out` | string | `""` | No | path to write JSON discovery metadata for the issued certificate and attestation evidence | | `--discovery-cds-cert-url` | string | `""` | No | public URL path where the CDS certificate PEM is served | | `--discovery-mesh-ca-url` | string | `""` | No | public URL path where the mesh CA PEM is served | | `--discovery-public-tls-mode` | string | `cds` | No | public TLS mode to report in discovery metadata (cds or webpki) | | `--workload-claims` | bool | `false` | No | request an inventory-signed sandbox token — which the CDS verifies and stamps into the issued leaf — from the local admission inventory at get-cert's compiled-in Unix socket path (`nri-image-policy` on Node-as-CVM, `policy-monitor` in the Kata guest). The path is baked in, not supplied; fails closed if the inventory is unreachable | | `--workload-claims-guest` | bool | `false` | No | reach the inventory on the Kata guest's loopback address instead of the Node-as-CVM Unix socket. Both endpoints are compiled in; this only selects which shape applies, and a wrong setting fails closed | | `--workload-claims-timeout` | duration | `5s` | No | timeout for the admission inventory request | {/* flags:get-secret */} ## `c8s get-secret` Fetch the secrets this pod is granted and write each one to a file under `--out-dir`. Injected by the webhook as a native sidecar. It authenticates to CDS with the pod's CDS-issued certificate and a sandbox token redeemed from the node's admission inventory over a compiled-in socket path, and CDS releases only once every main container in the sandbox is running — earlier attempts are refused and retried, so the files appear shortly after the workload starts and a consumer must wait for them. A path the store does not hold yet is created with a CDS-generated value, which does not survive a CDS restart. After writing, the process idles until the pod is torn down. See [Secrets](/docs/c8s/concepts/application-secrets). | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--cds-url` | string | `""` | **Yes** | base URL of CDS; must be `https` (RA-TLS) | | `--attestation-api-url` | string | `""` | **Yes** | local attestation-api used to verify CDS's RA-TLS certificate | | `--measurements` | string slice | `nil` | No | SHA-384 hex launch measurement(s) CDS must present (repeatable / comma-separated); empty pins none (UNSAFE) | | `--cert` | string | `/run/c8s/certs/tls.crt` | No | the pod's CDS-issued certificate, presented to CDS | | `--key` | string | `/run/c8s/certs/tls.key` | No | private key for `--cert` | | `--secret` | string slice | `nil` | **Yes** | `NAME=/store/path` to fetch; `NAME` is the filename written under `--out-dir` (repeatable). At least one is required, each `NAME` must be distinct and must not be a path | | `--out-dir` | string | `/run/c8s/secrets` | No | directory the secret files are written to; must be memory-backed | | `--file-mode` | string | `0640` | No | octal mode for the written files | | `--attempts` | int | `60` | No | how many times to try before failing; must be positive | | `--retry-interval` | duration | `5s` | No | wait between attempts; must be positive | | `--request-timeout` | duration | `10s` | No | per-request timeout against CDS; must be positive | | `--inventory-timeout` | duration | `5s` | No | timeout for redeeming a sandbox token from the node's admission inventory; must be positive | {/* flags:get-volume */} ## `c8s get-volume` Fetch the key for each encrypted volume this pod is granted and hand it to [`c8s volumed`](#c8s-volumed) on the node, which opens the device and mounts it read-only into this pod. Injected by the webhook as a native sidecar, with the same authentication, release rule, and retry behavior as [`c8s get-secret`](#c8s-get-secret): the volume appears shortly after the workload starts. The key must already be in the store, put there by [`c8s volume create`](#c8s-volume); nothing here creates one. **Linux only** — the subcommand is registered only in Linux builds of the CLI. See [Volumes](/docs/c8s/concepts/encrypted-volumes). | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--cds-url` | string | `""` | **Yes** | base URL of CDS; must be `https` (RA-TLS) | | `--attestation-api-url` | string | `""` | **Yes** | local attestation-api used to verify CDS's RA-TLS certificate | | `--measurements` | string slice | `nil` | No | SHA-384 hex launch measurement(s) CDS must present (repeatable / comma-separated); empty pins none (UNSAFE) | | `--cert` | string | `/run/c8s/certs/tls.crt` | No | the pod's CDS-issued certificate, presented to CDS | | `--key` | string | `/run/c8s/certs/tls.key` | No | private key for `--cert` | | `--volume` | string slice | `nil` | **Yes** | `NAME=/store/path` to open; `NAME` selects the device by serial `c8s-vol-` and names the mounted volume (repeatable). At least one is required and each `NAME` must be distinct | | `--socket-dir` | string | `/run/c8s/workload-claims` | No | directory holding the node agent's socket, as this pod sees it | | `--attempts` | int | `60` | No | how many times to try before failing; must be positive | | `--retry-interval` | duration | `5s` | No | wait between attempts; must be positive | | `--request-timeout` | duration | `10s` | No | per-request timeout against CDS and the node agent; must be positive | | `--inventory-timeout` | duration | `5s` | No | timeout for redeeming a sandbox token from the node's admission inventory; must be positive | {/* flags:volumed */} ## `c8s volumed` Node agent that opens encrypted volumes for the pods on its node. An injected [`c8s get-volume`](#c8s-get-volume) sidecar hands it a key over `volumed.sock`, created inside the admission inventory's socket directory; the daemon resolves the calling pod from kernel peer credentials, opens dm-crypt and dm-verity, and mounts the result read-only into that pod and no other. The mount target is never taken from what the caller reports about itself. Teardown is driven by the pod's cgroup slice disappearing. Runs privileged on every node, and takes no positional arguments. **Linux only** — the subcommand is registered only in Linux builds of the CLI. See [Volumes](/docs/c8s/concepts/encrypted-volumes). | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--socket-dir` | string | `""` | **Yes** | host directory holding the admission inventory's socket, where this daemon creates `volumed.sock`. No default: the in-pod path a sidecar sees is not where the daemon serves | | `--kubelet-root` | string | `/var/lib/kubelet` | No | kubelet's root directory, holding per-pod volume directories | | `--cgroup-root` | string | `/sys/fs/cgroup` | No | cgroup mount, where a pod's slice going away is what triggers teardown | | `--reap-interval` | duration | `15s` | No | how often to tear down volumes whose pod has gone; must be positive | | `--max-mounts` | int | `64` | No | maximum volumes open on this node at once; must be positive | {/* flags:cred-release */} ## `c8s cred-release` Issue a short-lived kube client certificate to a caller that proves possession of the operator key whose public half was measured into RTMR[3] at node launch. It serves `POST /release-credential` over RA-TLS — the endpoint [`c8s get-kubeconfig`](#c8s-get-kubeconfig) consumes — signing the caller's CSR with the cluster's client CA and returning a kubeconfig anchored to the serving CA. Startup fails closed if the measured key does not match RTMR[3], and an empty `--platform` is rejected because RA-TLS is mandatory here. Intel TDX only: the operator-key binding lives in RTMR[3]. Baked as a systemd unit in the c8s node image; not run by hand. The CA defaults are the RKE2 paths — on kubeadm, set all three to `/etc/kubernetes/pki/ca.crt` and `ca.key`. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--listen` | string | `:8443` | No | HTTPS (RA-TLS) bind address | | `--attestation-api-url` | string | `http://127.0.0.1:8400` | No | local attestation-api base URL, the source of the RA-TLS serving cert's TDX quote | | `--platform` | string | `tdx` | No | TEE platform (RTMR is TDX-only); empty is rejected | | `--client-ca-cert` | string | `/var/lib/rancher/rke2/server/tls/client-ca.crt` | No | cluster client-CA cert that signs kube client certs | | `--client-ca-key` | string | `/var/lib/rancher/rke2/server/tls/client-ca.key` | No | cluster client-CA key | | `--server-ca-cert` | string | `/var/lib/rancher/rke2/server/tls/server-ca.crt` | No | CA that signs the apiserver serving cert; embedded in the released kubeconfig | | `--cert-ttl` | duration | `24h` | No | lifetime of issued operator certs | | `--cert-org` | string | `system:masters` | No | Kubernetes group (cert Subject O) for the issued cert | | `--cert-cn` | string | `operator` | No | Kubernetes user (cert Subject CN) for the issued cert | {/* flags:get-kubeconfig */} ## `c8s get-kubeconfig` Obtain an operator kubeconfig from a measured Intel TDX CVM. The node is attested through its attestation-api and its quote's `rtmr_3` must equal the value the operator public key implies (the node was launched to trust that key); a locally generated CSR is then exchanged for a short-lived kube client certificate over the node's [`cred-release`](#c8s-cred-release) endpoint, and the kubeconfig is written to `--out`. Verification runs in-process with `attestation-go`, including the RA-TLS check on the `cred-release` dial. TDX only — the operator-key binding lives in RTMR[3], so a node reporting any other platform is rejected. Requires `--operator-key`, `--out`, and either `--node` or all of `--attest-url`, `--release-url`, and `--apiserver-url`. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--node` | string | `""` | No | guest host/IP that fills `--attest-url`, `--release-url`, and `--apiserver-url` with the standard ports (`8400`/`8443`/`6443`); each explicit URL flag overrides it | | `--attest-url` | string | `""` | No | attestation-api `/attest` URL | | `--release-url` | string | `""` | No | `cred-release` base URL | | `--apiserver-url` | string | `""` | No | apiserver URL written into the kubeconfig | | `--operator-key` | string | `""` | **Yes** | operator ECDSA private key PEM; its public half is the one bound into RTMR[3] at node launch, and it signs the credential-release request | | `--context` | string | `c8s` | No | kubeconfig cluster/context/user name | | `--tls-server-name` | string | `c8s-cvm` | No | kubeconfig `tls-server-name`: pins apiserver certificate verification to this SAN (baked into the node image's `tls-san`) instead of the dialed IP. Empty omits the field | | `--out` | string | `""` | **Yes** | path the kubeconfig is written to, mode `0600` | | `--timeout` | duration | `30s` | No | per-step network timeout | {/* flags:ratls-mesh */} ## `c8s ratls-mesh` Run the RA-TLS L4 mesh proxy or its iptables side commands. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--platform` | string | `auto` | No | TEE platform: `sev-snp`, `tdx`, or `auto` (probes `/dev/{tdx_guest,sev-guest}`) | | `--attestation-api-url` | string | `""` | No | URL of the local attestation-api (e.g. http://localhost:8400) | | `--outbound-port` | int | `15001` | No | outbound listener port (intercepted app traffic) | | `--inbound-port` | int | `15006` | No | inbound listener port (RA-TLS from peer nodes) | | `--node-ip` | string | `""` | No | this node's IP (auto-detected from NODE_IP env if unset) | | `--cert-dns-san` | string | `""` | No | DNS SAN placed on the CDS-issued mesh cert (must match CDS `--dns-san-pattern`; empty omits SANs). Not used for peer verification, which is attestation-based | | `--log-level` | string | `info` | No | log level: debug, info, warn, error | | `--dial-timeout` | duration | `5s` | No | plain TCP dial timeout | | `--tls-dial-timeout` | duration | `10s` | No | RA-TLS dial timeout | | `--dest-header-timeout` | duration | `5s` | No | inbound destination header read timeout | | `--drain-timeout` | duration | `30s` | No | graceful shutdown drain timeout | | `--keepalive` | duration | `30s` | No | TCP keepalive interval (0 to disable) | | `--idle-timeout` | duration | `0` | No | close connections idle longer than this (0=disabled) | | `--max-conns` | int | `0` | No | max concurrent connections (0=unlimited) | | `--max-conns-per-source` | int | `0` | No | max concurrent connections per source IP (0=unlimited) | | `--health-port` | int | `15021` | No | health/metrics HTTP port | | `--measurements` | string | `""` | No | comma-separated hex SHA-384 launch measurements (empty = accept any TEE) | | `--cert-ttl` | duration | `24h` | No | RA-TLS certificate lifetime (rotates at 50%) | | `--rotation-timeout` | duration | `30s` | No | max time for background certificate rotation | | `--cert-mode` | string | `self-signed` | No | certificate mode: `self-signed` (default), or `cds` (boots self-signed, upgrades to CDS-issued in background) | | `--cds-url` | string | `""` | No | CDS service URL for attestation and CA bundle retrieval (required for cds mode) | | `--ca-cert` | string | `""` | No | path to CA certificate file for peer verification | | `--ca-poll-interval` | duration | `5m` | No | interval to poll CDS /ca for CA bundle updates | | `--cds-measurements` | string | `""` | No | comma-separated SHA-384 hex launch measurements that CDS's RA-TLS peer cert must match; empty = accept any (UNSAFE outside development) | | `--session-cache-size` | int | `64` | No | TLS session cache size per node (0 disables session resumption) | | `--access-log` | bool | `true` | No | emit per-connection structured access log | | `--cert-pipeline-probe-url` | string | `""` | No | CDS /readyz URL for pipeline health probing (empty = disabled) | | `--cds-retry-backoff` | duration | `2s` | No | initial backoff duration for CDS certificate upgrade retries | | `--cds-retry-max-backoff` | duration | `60s` | No | maximum backoff duration for CDS certificate upgrade retries | | `--max-dest-header-size` | int | `256` | No | maximum destination header size in bytes | | `--pipe-buffer-size` | int | `32768` | No | buffer size for TCP pipe forwarding | | `--accept-error-threshold` | int64 | `10` | No | consecutive accept errors before marking unhealthy | | `--health-read-timeout` | duration | `5s` | No | health server read timeout | | `--health-write-timeout` | duration | `10s` | No | health server write timeout | | `--metrics-update-interval` | duration | `10s` | No | interval for resolver cache and cert expiry metric updates | | `--local-cidr-boot-timeout` | duration | `1s` | No | synchronous retry budget at startup for host pod-network CIDR discovery; past this the mesh falls through to the async refresh loop and local-destination validation uses Kubernetes pod HostIP ownership until discovery recovers | | `--iptables-metrics-file` | string | `/tmp/ratls-iptables-metrics.json` | No | shared file where iptables-sync publishes counters (empty disables) | | `--cds-op-timeout` | duration | `30s` | No | per-operation timeout for CDS certificate upgrade and CA bundle refresh | | `--cert-pipeline-probe-timeout` | duration | `5s` | No | HTTP client timeout for cert pipeline health probe requests | | `--cert-pipeline-probe-interval` | duration | `60s` | No | interval between cert pipeline health probe requests | ### `c8s ratls-mesh iptables-sync` Watch Kubernetes pods and maintain the iptables/ipset rules that redirect mesh traffic. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--outbound-port` | int | `15001` | No | outbound listener port | | `--uid` | int | `1337` | No | UID to exclude from redirect | | `--exclude-uids` | string | `0` | No | comma-separated UIDs to skip (e.g. root=0 so kubelet/containerd can reach registries) | | `--exclude-source-namespaces` | string | `kube-system` | No | comma-separated local source namespaces excluded from transparent mesh interception | | `--node-ip` | string slice | `nil` | No | local node IP(s); repeat or comma-separate for dual-stack (one per family). Defaults to the `NODE_IP` env. Each address must be a non-loopback, non-unspecified IP bound to a local interface | | `--resync-period` | duration | `30s` | No | periodic full ipset reconciliation interval | | `--watchdog-period` | duration | `2s` | No | interval at which the base-chain jump rules are re-asserted at position 1 (bounds the race window against kube-proxy reinserting KUBE-SERVICES) | | `--ipset-maxelem` | int | `262144` | No | maximum members per managed ipset | | `--cw-inbound-passthrough` | string | `udp:53,tcp:53` | No | comma-separated `proto:source-port` replies exempted from the always-on cw inbound guard (which drops FORWARD-path traffic to `confidential.ai/cw` pods so only mesh-delivered and host-local traffic reaches them). Empty = strict drop-all; DNS is the default | | `--ready-file` | string | `""` | No | path to write after initial ipset and iptables sync succeeds | | `--iptables-metrics-file` | string | `/tmp/ratls-iptables-metrics.json` | No | shared file where iptables-sync publishes counters (empty disables) | | `--log-level` | string | `info` | No | log level: debug, info, warn, error | The `ratls-mesh iptables-cleanup` subcommand takes no flags. {/* flags:nri-image-policy */} ## `c8s nri-image-policy` Run the NRI image-policy plugin. Uses the standard Go `flag` package; most configuration comes from the YAML config file rather than CLI flags. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--config` | string | `/etc/nri/conf.d/image-policy.yaml` | No | path to config file | | `--health-addr` | string | `:8080` | No | health check listen address | | `--read-timeout` | duration | `5s` | No | HTTP server read timeout | | `--write-timeout` | duration | `10s` | No | HTTP server write timeout | {/* flags:probe-file */} ## `c8s probe-file` Exit 0 if `` exists and is non-empty — a file-existence helper for distroless containers, where `/bin/test` is not available. One-shot by default (for a kubelet exec probe); with `--wait` it blocks until the path passes (or `--timeout` elapses), so it can be the entrypoint of an init container that gates a workload on a file another container writes — the exec-free equivalent of a startup probe, needed on locked kata guests where exec probes are denied by policy. The non-empty check rules out passing on a half-written file. | Flag | Type | Default | Required | Description | | --- | --- | --- | --- | --- | | `--wait` | bool | `false` | No | block until `` passes the check instead of probing once | | `--poll-interval` | duration | `1s` | No | how often to re-check `` in `--wait` mode | | `--timeout` | duration | `0` | No | give up (non-zero exit) after this long in `--wait` mode; 0 waits forever | --- # Helm values `c8s install` collects everything its flags imply into one computed values file and passes it to Helm as the **last** `-f`. Precedence is therefore: chart defaults, then your `-f` files in the order you list them, then the installer's computed values — a flag always wins on the keys it sets. This page covers the values you are likely to set by hand. It is not the whole chart; run `helm show values` against the embedded chart, or read `internal/helmchart/c8s/values.yaml` in the c8s repo, for the full surface. Passing any `-f` hands you ownership of the installer's default-path checks, which it then skips — including the `--operator-keys` guard. See [Choosing the install shape](/docs/c8s/how-to/install#choosing-the-install-shape). ## Trust pinning The values that decide whether the cluster pins anything at all. | Value | Default | Binds | | --- | --- | --- | | `cds.measurements` | `[]` | Launch digests CDS admits to `/attest`, `/sign-csr`, and `/handoff`; also what `ratls-mesh` and `nri-image-policy` pin when they dial CDS. **Empty means no pinning.** | | `ratlsMesh.measurements` | `[]` | Launch digests mesh peers accept from each other. Empty means no pinning. | | `cds.operatorKeys` | `""` | PEM **content** of the operator EC public keys authorized to write the allowlist. Empty rejects every write; reads still serve. | | `cds.sandboxInventoryCIDRs` | `[]` | The only addresses CDS dials for a node's admission inventory. **Required for sandbox identity** — unset, CDS refuses every request carrying a sandbox token. | | `cds.ratlsPlatform` | `snp` | Platform for the CDS serving cert's evidence. **Empty disables TLS entirely** — unsafe outside tests. | `c8s install --measurements ` sets the first two together; `--operator-keys` and `--node-cidr` set the next two. Prefer the flags: `cds.operatorKeys` is the PEM *content*, not a path, and the flag reads the file for you. **`cds.measurements` is also the handoff allowlist.** With `cds.handoff.enabled`, every digest listed there can pull the mesh CA private key over `/handoff`. There is no separate, narrower handoff allowlist. ## Certificate authority and handoff | Value | Default | Effect | | --- | --- | --- | | `cds.ca.commonName` | `c8s Mesh CA` | CN of the generated mesh CA. | | `cds.ca.certValidity` | `8760h` | CA lifetime (one year). | | `cds.ca.minValidity` | `1h` | `/readyz` starts failing below this remaining validity, so a successor is brought up in time. | | `cds.handoff.enabled` | `false` | Register `/handoff` so a joining replica can adopt the live CA. Requires `cds.measurements` **and** `cds.operatorKeys`. | | `cds.handoff.peerUrl` | `""` | Where a starting CDS adopts its CA from. The sentinel `self` expands to the CDS Service URL. Setting it flips the rollout to RollingUpdate so the new pod adopts before the old one retires. Cannot combine with `cds.persistence.enabled`. | | `cds.tokenSignerRotationInterval` | `720h` | How often a new EAR-signing key becomes active. | | `cds.tokenSignerOverlap` | `25h` | How long a retired signing key stays valid and published. | ## Allowlist seeding | Value | Default | Effect | | --- | --- | --- | | `nriImagePolicy.bootstrapAllowlist.digests` | `{}` | Digest → image reference pairs seeded into the floor before CDS serves its first request. The GitOps-friendly alternative to `c8s allowlist add`. | | `nriImagePolicy.bootstrapAllowlist.workloads` | `{}` | Named workload entries seeded the same way. | | `nriImagePolicy.bootstrapAllowlist.deriveComponents` | `false` | Auto-add the chart's own component images to the floor whenever their `image.digest` is set. `c8s install --resolve-digests` (the default) turns this on. | | `nriImagePolicy.refresh.interval` | `30s` | How often enforcers re-poll `GET /allowlist`. | | `nriImagePolicy.policy.mode` | `fail-closed` | `audit` logs the would-be denial and admits — a bring-up setting, never production. | Seeding is **additive**: it inserts only what is missing and leaves existing entries untouched. Any seed error halts CDS startup. See [Seeding and bootstrap](/docs/c8s/concepts/image-allowlist#seeding-and-bootstrap). ## Encrypted volumes | Value | Default | Effect | | --- | --- | --- | | `volumed.enabled` | `false` | The node agent that opens devices and mounts them. **Nothing about encrypted volumes works without it.** | | `volumed.maxMounts` | `64` | Live volumes per node; each costs two dm devices and a mount. | | `volumed.reapInterval` | `15s` | How often teardown checks which pods have gone. | | `volumed.nodeSelector` | `{}` | Confine the daemon to the nodes that carry volume devices. | ## The front door (`tls-lb`) | Value | Default | Effect | | --- | --- | --- | | `tlsLb.enabled` | `true` | The public front door: nginx plus the `cds-attest` sidecar. | | `tlsLb.attest.enabled` | `true` | Serve `/.well-known/c8s/` — the endpoints browser and CLI verifiers read. `c8s install --attest=false` omits it. | | `tlsLb.allowlist.enabled` | `true` | Publish the CDS allowlist API through the front door. Turning it off leaves only the direct CDS URL. | | `tlsLb.upstream` | — | Host and port of the upstream. A `c8s-..svc.cluster.local` address is recognized as mesh-wrapped and may be plaintext; anything else must be `https` with verification. | | `tlsLb.publicTLS.secretName` | `""` | Serve your own public certificate instead of the CDS-issued one. **This makes the allowlist CLI refuse the endpoint** — that certificate is not bound to the discovery attestation. | | `tlsLb.hostPort.enabled` | `true` | Publish nginx's TLS listener on the node's host port. Grabs the node's `443`; turn it off on clusters whose ingress already owns it (notably RKE2). | | `tlsLb.cors.enabled` | `false` | Your own CORS block, which **replaces** the built-in policy everywhere, protocol endpoints included. | | `tlsLb.cors.protocolEndpoints` | `true` | Wide-open CORS on the c8s protocol endpoints only, so any browser can verify the cluster. | | `tlsLb.routes` | `[]` | Extra routes. A route backend must be `https` with `tls.verify: true`. | **`tlsLb.hostPort.enabled` forces a `Recreate` rollout.** With the port bound, every `tls-lb` roll drops the node-IP front door for the restart window — under Pod-as-CVM, a full CVM boot. For gapless rolls set `enabled: false` and expose `tls-lb` through its Service. ## Scheduling | Value | Default | Effect | | --- | --- | --- | | `cds.node.selector` | `role: cds` | Pins the singleton CDS to a known node. `{}` for a single-node cluster — `c8s install --single-node` sets this. | | `cds.node.tolerations` | `dedicated=cds:NoSchedule` | Lets CDS land on a node dedicated to it. `[]` for an untainted node. | | `cds.persistence.enabled` | `false` | An RWO volume for CDS. Cannot combine with `handoff.peerUrl` — a surge pod cannot share it. | | `hostNamespacePolicy.enabled` | `true` | Withholds host namespaces from tenant pods, which is what reserves port `1019` for the admission inventory. | ## See also - [Install c8s](/docs/c8s/how-to/install) — where `-f` fits in the install. - [CLI reference → `c8s install`](/docs/c8s/reference/cli#c8s-install) — the flags that set these for you. - [Workload annotations](/docs/c8s/reference/workload-annotations) — the per-pod overrides. --- # Workload annotations The **c8s** admission webhook reads these annotations off a pod template and injects accordingly. All of them are namespaced `confidential.ai/`. Only the first is required; setting any other without it is rejected at admission. ## Opt in | Annotation | Default | Effect | | --- | --- | --- | | `confidential.ai/cw` | — (**required**) | The **workload id**, not a boolean. Opts the pod into injection, is mirrored as a pod label so the operator-managed headless Service can select it, and derives the Service name `c8s-` and the certificate SAN `c8s-..svc`. | | `confidential.ai/c8s-injected` | — | Stamped by the webhook after a successful mutation so re-invocation is a no-op. **Do not set it yourself.** | **The value is an identity, not a flag.** `confidential.ai/cw: my-inference` yields the Service `c8s-my-inference`, which is what `tls-lb` dials over the RA-TLS mesh. A value like `"true"` would only produce a Service named `c8s-true`. Use a stable name that is a valid DNS-1035 label once prefixed with `c8s-`; an id that cannot form one yields no Service at all. ## Certificate The injected `c8s-cert` sidecar writes the pod's mesh leaf. These override where and how. | Annotation | Default | Effect | | --- | --- | --- | | `confidential.ai/c8s-san` | derived from the cw id | Override the DNS SAN `get-cert` requests. For workloads adopted into c8s whose clients already dial an existing Service name. | | `confidential.ai/c8s-cert-volume` | `c8s-certs` | Name of the volume the cert is written to. | | `confidential.ai/c8s-cert-dir` | `/etc/c8s/certs` | Directory the cert and key land in. | | `confidential.ai/c8s-cert-file` | `tls.crt` | Certificate filename. | | `confidential.ai/c8s-key-file` | `tls.key` | Private-key filename. | | `confidential.ai/c8s-renew-interval` | `6h` | Renewal interval for the issued certificate. A Go duration. | ## Secrets and volumes | Annotation | Default | Effect | | --- | --- | --- | | `confidential.ai/c8s-secrets` | — | Comma-separated `NAME=/store/path` pairs. Injects the `c8s-secret` fetcher sidecar and the `c8s-secrets` volume. See [Use application secrets](/docs/c8s/how-to/use-application-secrets). | | `confidential.ai/c8s-secret-dir` | `/run/c8s/secrets` | Where the secret files land. | | `confidential.ai/c8s-volumes` | — | Comma-separated `NAME=/store/path` pairs. `NAME` selects the node device by its `c8s-vol-` serial. Injects the `c8s-volume` sidecar. See [Create an encrypted volume](/docs/c8s/how-to/create-an-encrypted-volume). | | `confidential.ai/c8s-volume-dir` | `/run/c8s/volumes` | Where opened volumes are mounted, one directory each. | Both are rejected at admission under **Pod-as-CVM** — the node-side machinery they need does not exist inside a per-pod guest. ## nginx reload For a workload that must be signalled when its certificate is renewed. | Annotation | Default | Effect | | --- | --- | --- | | `confidential.ai/c8s-reload-nginx` | `false` | Reload nginx after a certificate is written. | | `confidential.ai/c8s-reload-watch-paths` | — | Additional paths to watch for change. | | `confidential.ai/c8s-reload-watch-volume` | — | Volume holding the watched paths. | | `confidential.ai/c8s-reload-watch-mount-path` | — | Where that volume is mounted. | ## Discovery output For a pod that publishes its own attestation discovery document. | Annotation | Default | Effect | | --- | --- | --- | | `confidential.ai/c8s-discovery-out` | — | Path the discovery document is written to. | | `confidential.ai/c8s-discovery-volume` | — | Volume it is written to. | | `confidential.ai/c8s-discovery-mount-path` | — | Where that volume is mounted. | | `confidential.ai/c8s-discovery-cds-cert-url` | — | URL the document advertises for the CDS certificate. | | `confidential.ai/c8s-discovery-mesh-ca-url` | — | URL the document advertises for the mesh CA bundle. | | `confidential.ai/c8s-discovery-public-tls-mode` | `cds` | `cds` or `webpki` — which certificate the front door serves publicly. | ## get-cert process identity | Annotation | Default | Effect | | --- | --- | --- | | `confidential.ai/c8s-get-cert-run-as-user` | `65532` | `runAsUser` for the injected container. | | `confidential.ai/c8s-get-cert-run-as-group` | `65532` | `runAsGroup`. | | `confidential.ai/c8s-get-cert-run-as-non-root` | `true` | `runAsNonRoot`. | | `confidential.ai/c8s-get-cert-verbose` | `false` | Verbose `get-cert` logging. | The cluster-wide defaults behind these are set at install with the `--webhook-get-cert-*` flags — see the [CLI reference](/docs/c8s/reference/cli#c8s-install). ## Reserved names A pod that declares any of these itself is **rejected at admission**. They are reserved because the webhook rebuilds them on every call, so a pre-declared one would either be silently replaced or let a host-authored spec choose where released plaintext lands. | Name | Kind | What it is | | --- | --- | --- | | `c8s-cert` | container | the `get-cert` sidecar that obtains the mesh leaf | | `c8s-cert-wait` | container | the gate that blocks the workload until the first cert is written | | `c8s-secret` | container | the secret fetcher | | `c8s-volume` | container | the encrypted-volume fetcher | | `c8s-certs` | volume | default destination for the mesh leaf | | `c8s-secrets` | volume | memory-backed, mounted read-only into every container | | `c8s-volume-` | volume | one per requested volume, default-medium `emptyDir` | | `c8s-workload-claims` | volume | the admission inventory's socket directory | The `c8s-secrets` volume must be a **memory-backed** `emptyDir` if pre-declared at all; a `c8s-volume-` must be a **default-medium** `emptyDir`. A `hostPath` or PVC in either position is refused — it would write released plaintext to storage the host reads. See [Reserved containers and volumes](/docs/c8s/concepts/kata-runtime#reserved-containers-and-volumes). ## RuntimeClasses Injected under `--cvm-mode=pod`, selected by the cluster's `--hardware-platform` and by whether the pod requests an `nvidia.com/*` resource. These are a fixed contract with the RuntimeClasses the chart installs and with the kata-enforcement policy — a custom class is rejected. | RuntimeClass | Selected when | | --- | --- | | `kata-qemu-snp` | SEV-SNP, no GPU | | `kata-qemu-snp-nvidia` | SEV-SNP, pod requests a GPU | | `kata-qemu-tdx` | Intel TDX, no GPU | | `kata-qemu-tdx-nvidia` | Intel TDX, pod requests a GPU | | `kata-qemu` | VM-isolated but **not** confidential | ## See also - [Run a confidential workload](/docs/c8s/how-to/run-a-confidential-workload) — the minimum a pod needs. - [Helm values](/docs/c8s/reference/helm-values) — the cluster-wide defaults these override. - [Kata containers](/docs/c8s/concepts/kata-runtime) — why the names are reserved. --- # Your first confidential cluster By the end of this tutorial you will have a working **c8s** cluster and will have proved, from outside it, that your workloads run inside hardware you can check. It goes from an empty Azure subscription to a verified cluster in a single pass — every choice is already made for you, so follow it top to bottom rather than picking pieces out of it. You need no prior confidential-computing knowledge. Every term is introduced where it first matters, and nothing here has to be read first. The cluster you build runs the **node-as-CVM** shape: the node is one confidential VM and your pods are ordinary containers inside it. That's the only shape Azure supports — see [why](#why-pod-as-cvm-is-not-available-on-azure) at the end. You run everything **from your laptop** — `kubectl` and `c8s install` talk to the managed cluster over `az aks get-credentials`. There's nothing to SSH into. ## Prerequisites On your **laptop**: - An **Azure subscription**, with the [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) installed and logged in (`az login`). - **`kubectl`** (`az aks install-cli` if you don't have it). Step 3 adds the toolchain for building the CLI (`make`, Go, Helm, `crane`) — nothing to install up front. There's no access token to create and no registry login to do either: the c8s source repo and every c8s component image are public. If you mirror the images into a registry that *does* require authentication, step 4 has a short note on wiring that up. ## What you'll build - A single-node **AKS** cluster with a `Standard_DC*as_v5` **confidential node pool** — AMD SEV-SNP, with a vTPM. - **c8s** installed — operator, CDS, attestation-api, RA-TLS mesh, image policy, and the tls-lb front door — and verified. AKS confidential node pools use the AMD SEV-SNP **DCas_v5** family; `Standard_DC4as_v5` is a comfortable single-node dev size. Confidential VM sizes exist only in some regions (for example `northeurope`) — check `az vm list-skus --location --size Standard_DC` first. The [provisioning guide](/docs/c8s/how-to/provision-azure-aks#confidential-vm-sizes) has the full SKU table, including Azure's **Intel TDX** sizes, which c8s supports on the same vTPM path. ## Step by step ### 1. Create the AKS cluster with a confidential node pool A `DCas_v5` node size makes the pool a SEV-SNP confidential VM node pool — each node boots as a CVM with a vTPM. `--node-count 1` keeps it single-node. ```bash az group create --name c8s-rg --location northeurope az aks create \ --resource-group c8s-rg \ --name c8s-aks \ --node-count 1 \ --node-vm-size Standard_DC4as_v5 \ --os-sku Ubuntu \ --generate-ssh-keys ``` Prefer Intel TDX? Use a `DCes_v5` / `ECes_v5` size instead — `Standard_DC4es_v5` is the direct counterpart of the one above. Nothing else in this step changes; the difference is a single install flag in step 4. **Verify the node-pool shape.** If your subscription/region only allows the confidential size on a **user** pool, create a regular system pool and add the confidential pool with `az aks nodepool add --mode User --node-vm-size Standard_DC4as_v5 --os-sku Ubuntu …` — then drop `--single-node` in step 4 and let the CDS schedule onto the confidential pool. See the [provisioning guide](/docs/c8s/how-to/provision-azure-aks#step-by-step). ### 2. Point kubectl at the cluster ```bash az aks get-credentials --resource-group c8s-rg --name c8s-aks # wait until the node reports Ready kubectl get nodes -o wide ``` `get-credentials` merges the cluster into your kubeconfig, so `kubectl` (and `c8s install`) talk to it directly from your laptop. While you're looking at that output, check the `KERNEL-VERSION` column. c8s needs a kernel new enough for the TEE — **6.11 or later** on AMD SEV-SNP, **6.16 or later** on Intel TDX. Both clear the Linux 6.5 floor that `SO_PEERPIDFD` imposes, which `nri-image-policy`'s admission inventory depends on; on an older kernel, sandbox-token fetches fail closed. If the node image Azure gave you is older than that, pick a newer node image before going on. ### 3. Install the c8s CLI There's no prebuilt binary yet, so build the CLI from source on your laptop — Linux and macOS both work. It shells out to `helm`, `kubectl`, and `crane`.
Install the build tools (make, Go, Helm, crane) On Linux: ```bash # make + Go 1.26+ (the c8s go.mod floor) sudo apt-get update && sudo apt-get install -y make curl -sL https://go.dev/dl/go1.26.3.linux-amd64.tar.gz | sudo tar -C /usr/local -xz echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc && export PATH=$PATH:/usr/local/go/bin # Helm curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash # crane (only needed with the default --resolve-digests=true) CRANE_VER=$(curl -s https://api.github.com/repos/google/go-containerregistry/releases/latest | grep -oP '"tag_name": "\K[^"]+') curl -sL "https://github.com/google/go-containerregistry/releases/download/${CRANE_VER}/go-containerregistry_Linux_x86_64.tar.gz" \ | sudo tar -xz -C /usr/local/bin crane ``` On macOS, `brew install make go helm crane` covers all four.
Clone the repo and build. `make install` runs `go install`, which drops the binary in `$(go env GOPATH)/bin` (default `~/go/bin`) — add that directory to your `PATH`: ```bash git clone https://github.com/confidential-dot-ai/c8s.git cd c8s make install # builds and installs `c8s` to $(go env GOPATH)/bin (default ~/go/bin) export PATH=$PATH:$(go env GOPATH)/bin c8s --help ``` ### 4. Install c8s On Azure, attestation comes from the **vTPM at `/dev/tpm0`**, not a native `/dev/sev-guest` or `/dev/tdx-guest` device. Select that path with `--cvm-mode aks` — despite the name it means "Azure vTPM". Note this is a node-as-CVM shape, **not** `--cvm-mode=pod`: per-pod CVMs need nesting Azure doesn't provide. `--cvm-mode` (the deployment shape) and `--hardware-platform` (the CPU TEE) are separate axes, and both matter on Azure. The vTPM's HCL report wraps an SNP report on a SEV-SNP CVM and a TD quote on an Intel TDX CVM, so the platform still has to be declared: | Your node pool | Flags | | --- | --- | | AMD SEV-SNP (`DCas_v5`) | `--cvm-mode aks` (`--hardware-platform` defaults to `sev-snp`) | | Intel TDX (`DCes_v5` / `ECes_v5`) | `--cvm-mode aks --hardware-platform tdx` | First mint the operator keypair whose public half the install pins into the CDS — the private key is what will authorize [allowlist](/docs/c8s/concepts/image-allowlist) writes later, and without `--operator-keys` the installer stops and asks for `--force`: ```bash openssl ecparam -name prime256v1 -genkey -noout -out operator.key openssl ec -in operator.key -pubout -out operator.pub ``` Then install: ```bash c8s install \ --single-node \ --cvm-mode aks \ --operator-keys operator.pub ``` `--single-node` makes every node CDS-eligible (no dedicated CDS node). When it finishes, the CLI tells you the one thing still outstanding: ```text + tls-lb attestation sidecar enabled, but the mesh is UNPINNED (accepts any attested TEE). Pin it with the node image's launch measurement M (its manifest.json): reinstall with --measurements . Clients verify with the same M: c8s verify https:// --measurements . ``` That's the production homework: re-run the same install with `--measurements ` and c8s pins both internal trust boundaries at once — CDS's launch digest (which the mesh and NRI pin) and the mesh peers against each other. On AKS the node image is Azure's, so there's no manifest to read `M` out of; step 5 reads it off a live attestation report instead. One value covers everything, because every pod on a node-as-CVM node shares the node's measurement — see [Obtaining launch measurements](/docs/c8s/how-to/obtain-launch-measurements).
Pulling the c8s images from a private registry The c8s images are public, so the default install needs no credentials. If you mirror them somewhere authenticated, you need two independent logins: - **Locally**, so `crane` can resolve each image tag to a digest while `c8s install` runs (the default `--resolve-digests=true` shells out to `crane` on your laptop, reading `~/.docker/config.json`). Skipping it produces `crane digest ...: UNAUTHORIZED: authentication required`. - **In the cluster**, via a pull Secret in the release namespace, so the kubelet can pull at runtime. Create it **before** installing, then name it with `--image-pull-secret`. The install fails fast if the Secret is missing or is not a `kubernetes.io/dockerconfigjson`. ```bash REGISTRY= REGISTRY_USER= REGISTRY_TOKEN= # 1. Local login so `crane` can resolve digests during `c8s install` echo "$REGISTRY_TOKEN" | crane auth login "$REGISTRY" -u "$REGISTRY_USER" --password-stdin # 2. Cluster-side pull Secret so the kubelet can pull at runtime kubectl create namespace c8s-system kubectl create secret docker-registry mirror-secret \ --namespace c8s-system \ --docker-server="$REGISTRY" \ --docker-username="$REGISTRY_USER" \ --docker-password="$REGISTRY_TOKEN" c8s install --single-node --cvm-mode aks --operator-keys operator.pub \ --image-pull-secret mirror-secret ```
**Validate on managed nodes.** This install runs privileged DaemonSets that touch the node host — `nri-image-policy` patches containerd and restarts it, `ratls-mesh` sets up `iptables`, and `attestation-api` mounts `/dev/tpm0`. Confirm these complete on your AKS node pool (and survive node image upgrades) before relying on it. `helm --wait` makes `c8s install` **block** until every component reports Ready, so the command can sit for a few minutes — that's expected, not a hang. To watch progress, in another terminal: ```bash kubectl get pods -A -w ``` A brief `ImageInspectError` or a single restart on `ratls-mesh` early in the rollout is normal: `nri-image-policy` restarts containerd as it installs, and the mesh pod retries through that and self-heals. The wait is **not** unbounded, though — outside `--cvm-mode=pod` the install runs `helm --wait --timeout=5m`, so if a pod stays stuck in `ImagePullBackOff` or `CrashLoopBackOff`, `c8s install` **fails** once the timeout elapses. The resources it already applied stay in place, so fix the cause and **re-run** `c8s install` — it's idempotent (`helm upgrade --install`) and completes once every pod is Ready. Read the [install guide](/docs/c8s/how-to/install) for the rest of the flags. ### 5. Verify the install Confirm the control plane came up. A node-as-CVM install on AKS runs six components — `c8s-cds`, `c8s-operator`, `c8s-attestation-api`, `c8s-ratls-mesh`, `c8s-nri-image-policy`, and `c8s-tls-lb` — and all should be `Running` with every container ready: ```bash kubectl get pods -n c8s-system ``` Check the admission webhook is wired up — the operator patches its `caBundle` once at startup, so it should be non-empty: ```bash # the config is named -pod-injector (release defaults to c8s): kubectl get mutatingwebhookconfiguration c8s-pod-injector \ -o jsonpath='{.webhooks[*].clientConfig.caBundle}' | wc -c # any non-zero count means it's patched ``` Now the real test: prove confidentiality from **outside** the cluster. The tls-lb attestation sidecar is on by default, so the front door is already publishing its evidence. `c8s-tls-lb` is a `ClusterIP` Service at this point, so reach it with a port-forward and point the [`c8s verify` CLI](/docs/c8s/how-to/verify-a-deployment) at it: ```bash kubectl port-forward -n c8s-system svc/c8s-tls-lb 8443:443 & c8s verify https://localhost:8443 --kind lb --server-name c8s-tls-lb.c8s-system.svc ``` ```text ✓ VERIFIED (attestation-go backend) source: discovery document https://localhost:8443/v1/discovery verified at: 2026-07-31T12:04:11Z platform: az-snp measurement: 9f2c… ← this is M … note: freshness NOT proven (no per-request nonce bound) WARNING: no --measurements pinned — any genuine TEE is accepted (UNSAFE for production) ``` `--server-name` is the SNI the serving certificate carries — the chart-managed Service DNS name, which is what a port-forward would otherwise miss. `platform: az-snp` is the Azure vTPM SEV-SNP shape you installed (`az-tdx` on a TDX pool). The `freshness NOT proven` note is expected for `--kind lb`: the discovery document carries a challenge fixed at certificate-issuance time rather than a per-request nonce, in exchange for shipping the VCEK so the evidence verifies offline. That `measurement` line is the `M` from step 4 — the node's launch digest, read straight off a live attestation report. Close the loop: re-run `c8s install` with `--measurements ` to pin the mesh, then re-run `c8s verify … --measurements ` and the warning goes away. Exit code 0 means verified, 2 means the evidence or the policy failed, and 3 means no evidence could be fetched — so the command drops straight into CI. To do the same check from a browser or a Node client — and to open a sealed channel on top of it — continue to [Consumer verification](/docs/c8s/concepts/verification-model). ## Why pod-as-CVM is not available on Azure Azure's hypervisor does **not** expose nested virtualization. A [pod-as-CVM](/docs/c8s/concepts/trust-boundaries) design launches a *new* confidential VM inside the node for every pod — that requires nesting, which Azure does not provide. So each AKS confidential node is a single node-level CVM, and **all pods on it share that one confidential boundary.** Don't use `--cvm-mode=pod` on Azure. You still get strong guarantees: pod identities are issued by the CDS, the image allowlist is enforced at the node, inter-node traffic (once you scale past one node) is RA-TLS-protected, and secrets are released only to attested nodes. The tradeoff is attestation *granularity* — you can't distinguish two pods on the same node, because they're inside the same TEE. If your threat model needs per-pod isolation from the operator, you need a bare-metal-class confidential host that allows nesting; GKE (native `/dev/sev-guest`, `--cvm-mode gke`) is the other platform where per-pod CVMs are an option. See [pod-as-CVM vs node-as-CVM](/docs/c8s/concepts/trust-boundaries) for the full comparison. ## What you've built A managed Kubernetes cluster whose node is a confidential VM, with c8s installed: an attestation root of trust that will not issue a certificate to a pod it cannot vouch for, a mesh that refuses unattested peers, and image-policy enforcement at container creation. You verified the result from outside the cluster rather than taking its word for it. ## Where to go next **Continue the tutorial:** [Verified chat over confidential vLLM](/docs/c8s/tutorials/verified-chat) — deploy a vLLM model server onto **this cluster**, attest that it's running in a TEE, and chat with it over an encrypted channel from your laptop. Or bring your own serving stack onto it: - [Deploy NVIDIA Dynamo](/docs/c8s/how-to/deploy-dynamo) — a multi-component serving graph run confidentially. - [Serve a model with KServe](/docs/c8s/how-to/deploy-kserve) — `InferenceService` serving, every image digest-allowlisted. - [Run a confidential workload](/docs/c8s/how-to/run-a-confidential-workload) — any Deployment, one annotation. Or understand what you just deployed: - [The trust root](/docs/c8s/concepts/trust-root) — what the CDS vouches for and how. - [The image allowlist](/docs/c8s/concepts/image-allowlist) — how c8s decides what may run. - [Limitations](/docs/c8s/concepts/limitations) — the honest list of what this milestone does not do. --- # Verified chat over confidential vLLM This tutorial builds a tiny chat client that talks to a **vLLM** model server running inside a **c8s** cluster — but only *after* cryptographically verifying it's a genuine TEE. Using [`c8s-verify`](/docs/c8s/reference/c8s-verify-js), the client fetches the Load Balancer's attestation, checks it against a pinned measurement and your cluster's mesh CA, and only then opens a **post-quantum over-encrypted channel** to the LB enclave. Every prompt and completion is sealed end-to-end — a TLS-terminating proxy in front of the LB sees only ciphertext. You do not need to read anything first. If you want the design behind what you are about to do, [the verification model](/docs/c8s/concepts/verification-model) has it — afterwards. What you verify here is the **LB**, plus that it belongs to *your* cluster (its CDS-issued cert chains to your pinned mesh CA). By [transitivity of trust](/docs/c8s/concepts/verification-model#transitivity-of-trust), an attested LB only forwards to attested pods over the RA-TLS mesh — so verifying the LB transitively covers the vLLM backend behind it. ## Prerequisites - A running c8s cluster (see [Your first confidential cluster](/docs/c8s/tutorials/first-confidential-cluster)). - **Node ≥ 20** (or a browser) and [`c8s-verify`](/docs/c8s/concepts/verification-model) — see its [installation](/docs/c8s/reference/c8s-verify-js#installation). - The two **pinned values**, supplied out of band (Step 4): the LB **launch measurement** and your cluster's **mesh CA** PEM. ## Step by step ### 1. Deploy a confidential vLLM backend Run vLLM as a confidential workload. The `confidential.ai/cw: vllm` annotation opts the pod in — the value `vllm` is the **workload id**. The c8s webhook injects two `get-cert` containers, the `c8s-cert` identity sidecar and a `c8s-cert-wait` gate that holds your container until the first CDS-issued certificate lands (and, under `--cvm-mode=pod`, the confidential RuntimeClass), and the operator mints a headless Service `c8s-vllm..svc`. Headless DNS returns pod IPs, which the node mesh wraps in attested mTLS. We serve **`facebook/opt-125m`**, a tiny model that's quick to load on CPU. **This tutorial uses a CPU build of vLLM.** vLLM's published `vllm/vllm-openai` image is a **CUDA** build that won't start without a GPU, and vLLM ships no official prebuilt CPU image — so build one from [vLLM's CPU Dockerfile](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/), push it to a registry your cluster can pull, and reference *that* image below (pinning its digest in Step 2). **Running the GPU build instead?** Your cluster must be GPU-provisioned for confidential passthrough *before* c8s installs — GPUs bound to vfio-pci, GPU CC mode on, BAR resize done (this is host provisioning's job; do **not** install the NVIDIA GPU Operator — it assumes host-visible GPUs with a host driver and conflicts with the passthrough model). Then use the CUDA image and add a GPU resource limit with the per-model name your node advertises (`kubectl describe node | grep nvidia.com`), e.g. `nvidia.com/GB202GL_RTX_PRO_6000_BLACKWELL_SERVER_EDITION: 1` — the webhook injects the confidential GPU RuntimeClass from the GPU request alone. Don't set a `memory` limit on GPU pods, and expect one vCPU per GPU pod on SEV-SNP — see [Kata containers](/docs/c8s/concepts/kata-runtime). ```yaml # vllm.yaml apiVersion: apps/v1 kind: Deployment metadata: { name: vllm, namespace: workloads } spec: replicas: 1 selector: { matchLabels: { app: vllm } } template: metadata: labels: { app: vllm } annotations: { confidential.ai/cw: vllm } # workload id → headless Service c8s-vllm spec: containers: - name: vllm image: /vllm-cpu:v0.6.3 # your CPU build (see the callout above) args: ["--model", "facebook/opt-125m"] # tiny test model ports: [{ containerPort: 8000 }] # vLLM's OpenAI server port ``` ```bash kubectl create namespace workloads kubectl apply -f vllm.yaml ``` You don't create a Service yourself — the operator mints the `c8s-vllm` headless Service from the annotation. (The pod won't actually start until its image is allowlisted, next.) ### 2. Allow the vLLM image Nothing runs unless its image **digest** is on the [allowlist](/docs/c8s/concepts/image-allowlist) — otherwise enforcement blocks the container at creation. Resolve the digest of the exact image you deployed: ```bash crane digest /vllm-cpu:v0.6.3 # sha256:… ``` Add it to the live allowlist with `c8s allowlist`, signed by the operator key you created when installing the cluster (`operator.key` — its public half is what you pinned with `--operator-keys`). Where you point `--url` depends on your front door. The chart publishes the complete `/allowlist` API through tls-lb by default, so on a cluster whose front door still carries its CDS-issued certificate you use the same URL as application traffic. Not here: the front door you build in Step 3 terminates a real hostname with a WebPKI certificate, which isn't yet bound to the discovery attestation — so the CLI deliberately refuses it. Talk to the CDS directly instead. It has no public ingress, so port-forward it: ```bash kubectl port-forward -n c8s-system svc/c8s-cds 8443:8443 & c8s allowlist add sha256: /vllm-cpu:v0.6.3 \ --url https://localhost:8443 \ --measurements \ --operator-key operator.key # added sha256: ``` That's the only port-forward you need — the CLI verifies the CDS's RA-TLS attestation **in-process**, so nothing else has to be reachable. `--measurements` is the CDS launch digest; on a node-as-CVM cluster that's the node measurement every pod on the node reports, which `c8s verify` prints for you. Omit it and you get: ```text warning: no --measurements set; accepting any attested endpoint build (UNSAFE) ``` Fine for a tutorial cluster, but pin it in production so a write can never land on a rogue CDS. The enforcement plugins poll the CDS-served list, so the vLLM pod from Step 1 starts on its next retry: ```bash kubectl get pods -n workloads -w # vllm-… 2/2 Running ← the vLLM container plus the injected c8s-cert sidecar ``` Prefer a version-controlled allowlist? The **bootstrap** path pins digests in a values file instead (`nriImagePolicy.bootstrapAllowlist.digests`), baking them into the boot-time floor the chart seeds into the CDS — better suited to GitOps flows. See [Seeding and bootstrap](/docs/c8s/concepts/image-allowlist#seeding-and-bootstrap) and [Managing it with the c8s CLI](/docs/c8s/how-to/manage-the-allowlist). ### 3. Point the Load Balancer at vLLM You don't hand-edit nginx. The chart wires the front door — **tls-lb (public TLS + over-encryption) → your engine**, over the node mesh's attested mTLS — and you aim it at your vLLM workload with `--workload-ref` + `--upstream`. The attestation and over-encryption endpoints your client needs are served by default, so there's nothing extra to switch on. Apply the routing with one upgrade — re-run your original `c8s install` with the two flags added: ```bash c8s install \ --single-node \ --cvm-mode aks \ --operator-keys operator.pub \ --workload-ref vllm=workloads/deployment/vllm:8000 \ --upstream vllm ``` Repeat the flags you first installed with — `--cvm-mode` is required on every install, and if you pass a `-f values.yaml` too, the values these flags derive win on the keys they set. The ref's `:8000` (vLLM's default port) derives `tlsLb.upstream = c8s-vllm.workloads.svc.cluster.local:8000`, so tls-lb dials vLLM's headless Service directly and that hop rides the node mesh's attested mTLS — hence no nginx surgery. The `confidential.ai/cw: vllm` stamp the ref applies is a no-op — Step 1's manifest already carries it. A request your client sends over the over-encryption tunnel (Step 6) is decrypted **inside the LB enclave** and forwarded over the mesh to vLLM's OpenAI-compatible `/v1/chat/completions`. Giving that front door a public address is a separate, cluster-specific choice, and it's made in a values file rather than with flags: `tlsLb.service.type: LoadBalancer` for a public IP on AKS, `tlsLb.san` for the hostnames nginx answers on, and `tlsLb.publicTLS.secretName` for the certificate it presents. Pass the file with `-f values.yaml` alongside the flags above. The outer TLS layer is untrusted transport either way — Step 5's attestation is what establishes trust — but a WebPKI certificate is what lets a browser, or a stock Node client, reach `https://chat.example.com` without relaxing its own PKI check. ### 4. Gather the two pinned values Verification is meaningless without pinning *your* cluster's identity out of band: - **LB launch measurement** — the SHA-384 launch digest of the LB enclave. Use the published or recomputed digest (see [Obtaining launch measurements](/docs/c8s/how-to/obtain-launch-measurements)). On a node-as-CVM cluster this is the node's measurement, the same value the operator passed to `c8s install --measurements` — and the same one you pinned for the CDS in Step 2. - **Mesh CA certificate** — your cluster's CDS mesh CA (PEM). Capture it from a trusted context at install time (the CDS publishes its public CA bundle at `GET /ca`). This is the cluster-unique anchor — [why](/docs/c8s/concepts/verification-model#why-cluster-identity-is-pinned). ```bash export LB_MEASUREMENT= # plus mesh-ca.pem, captured out of band from your cluster operator / CDS bootstrap ``` Both values come to you *out of band* on purpose. Reading them off the endpoint you are about to verify would prove nothing. ### 5. Verify and connect Create a `C8sClient` with the LB origin and your pinned values, then `connect()`. That generates a nonce, fetches the LB attestation, verifies the attestation evidence in WASM, checks the measurement and the `report_data` binding, confirms the CDS cert chains to your mesh CA, and runs the X25519 + ML-KEM-768 handshake — all **fail-closed**: ```js import { C8sClient } from "c8s-verify"; import { readFileSync } from "node:fs"; const client = new C8sClient({ baseUrl: "https://chat.example.com", // your c8s LB measurements: [process.env.LB_MEASUREMENT], // pinned LB launch digest meshCaPem: readFileSync("mesh-ca.pem", "utf8"), // pinned cluster anchor }); const session = await client.connect(); // throws C8sVerifyError on any failure console.error(`✓ verified LB enclave — measurement ${session.attestation.measurement}`); ``` If anything is off — wrong measurement, tampered evidence, or a cert that doesn't chain to your mesh CA — `connect()` throws and **no channel is opened**. See [what it verifies](/docs/c8s/reference/c8s-verify-js#what-it-verifies). ### 6. Chat over the sealed channel `session.fetch` seals the **entire** request and sends it down the tunnel; the LB enclave decrypts it, forwards it to vLLM over the mesh, and seals the response back. Call vLLM's OpenAI-compatible endpoint — the `model` must match what you served in Step 1: ```js async function ask(session, messages) { const res = await session.fetch("/v1/chat/completions", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "facebook/opt-125m", messages }), }); if (res.status !== 200) throw new Error(`vLLM returned HTTP ${res.status}: ${res.text()}`); return JSON.parse(res.text()).choices[0].message.content; } ``` Wrap it in a REPL for an actual chat loop (`chat.mjs`): ```js import { createInterface } from "node:readline/promises"; import { stdin as input, stdout as output } from "node:process"; const rl = createInterface({ input, output }); const history = []; for (;;) { const prompt = await rl.question("you> "); if (!prompt || prompt === "/exit") break; history.push({ role: "user", content: prompt }); const reply = await ask(session, history); // session from Step 5 history.push({ role: "assistant", content: reply }); console.log(`bot> ${reply}\n`); } rl.close(); ``` Run it: ```bash LB_MEASUREMENT=$LB_MEASUREMENT node chat.mjs ``` Keep `"stream": true` out of these requests. The tunnel seals one whole request envelope and one whole response envelope, so a token stream can't be delivered incrementally through it. Streaming does work through tls-lb's ordinary route — nginx forwards Server-Sent Events straight through rather than buffering them — but that path gives you TLS to the front door, not a channel sealed to the enclave. Pick per endpoint: sealed for the prompts that matter, streamed for the ones where latency does. ## What you've proven By the time the first reply comes back, you've established — from an untrusted client, over an untrusted network — that: 1. the LB you reached is a **genuine TEE** running the **exact** measured image you pinned; 2. it belongs to **your** cluster (its cert chains to your mesh CA), not an attacker's look-alike; 3. the session key was minted **inside** that enclave (the `report_data` binding), so the over-encrypted channel terminates there and nowhere else; 4. by transitivity, the **vLLM** backend it forwards to is itself an attested pod on the c8s mesh. Your prompts and the model's completions are sealed end-to-end to the enclave — the TLS terminator, the host, and the infrastructure operator never see plaintext. ## Where to go next - **Verify without the tunnel** — the bare-evidence path and the full client API are in [Consumer verification](/docs/c8s/concepts/verification-model); the same check from a shell or CI is [`c8s verify`](/docs/c8s/how-to/verify-a-deployment). - **A bigger serving graph** — [NVIDIA Dynamo](/docs/c8s/how-to/deploy-dynamo) runs frontend, router, worker, and discovery as confidential workloads, and [KServe](/docs/c8s/how-to/deploy-kserve) does the same for `InferenceService` model serving. - **Automate the digests** — wire `c8s allowlist add` into the pipeline that builds your images: [Automating the allowlist](/docs/c8s/how-to/manage-the-allowlist#automating-the-allowlist). --- # Introduction to TEEs This document is a high-level introduction to what TEEs are, how they work, and what their limitations are. For a detailed technical deep dive, see the [Confidential Computing Primer](/docs/concepts/confidential-computing-primer). A Trusted Execution Environment (TEE) is a tamper-proof isolated computing environment that maintains data confidentiality and allows verifiability of its running workloads. TEEs have recently been implemented in modern CPU and GPU hardware. Intel and AMD have provided CPU-based TEEs for several years. More critically for AI workloads, NVIDIA introduced TEE capabilities in their GPUs starting 2024. ## Core Security Properties A TEE provides four critical security properties: **Integrity**: Once software is initialized in a TEE, it becomes tamper-proof. An attacker with root access to the host system cannot modify the running software without detection. **Confidentiality**: Data within the TEE is encrypted at the hardware level in memory pages. Data being processed remains private even from privileged system administrators. **Verifiability**: The TEE cryptographically measures exactly what software is running. Clients can verify they're communicating with the specific software they expect. **Attestation**: The TEE produces signed attestation reports that provide cryptographic proof of all the above properties. These attestations are signed by hardware manufacturers (Intel, AMD, NVIDIA) and cannot be forged. ## The HTTPS Analogy A useful analogy is the transition from HTTP to HTTPS. With HTTP you sent your data in plaintext, and you could not confirm who you're talking to. HTTPS allows you to confirm who you're talking to and encrypt your data in transit. TEE attestation goes a step further and allows you to prove that your data is encrypted during computation, letting you verify not just who you're talking to, but also what software they're running. All of this, like HTTPS, is available at a negligible performance cost. ## How TEEs Work in Practice Here's how this works in practice: Suppose that a user wants to submit a sensitive query to a large language model running on a remote server. They require that their query and the model's response remain confidential from all parties, including the server operator, and they need cryptographic proof that only the specified model version processed their data without any unauthorized access. The server runs the LLM inside a TEE. When the user sends their query, it gets encrypted with the TEE's public key before transmission. Inside the TEE, the model processes the encrypted query using weights held in hardware-protected memory, never accessible to any software outside the enclave. The model returns an encrypted response along with a cryptographic attestation that proves exactly which model version and code processed the query and verifies that no other software could access the user's data or the model weights. Without a TEE, this interaction would: - Expose the user's query in plaintext during processing - Allow system administrators and other processes to access both the query and model weights in memory - Enable potential interception by malicious software on the host system - Provide no verifiable guarantee about which software actually processed the request The TEE provides hardware-backed isolation and cryptographic verification that creates a secure computation environment even on untrusted infrastructure. ## How TEEs Actually Work The security guarantees of TEEs emerge from cryptographic primitives built into the silicon itself. Understanding how these guarantees work requires walking through the chain of trust from hardware to application. ### The TEE Module TEE implementations embed dedicated security modules directly into CPU hardware. These modules introduce new CPU execution modes and partition system resources at the hardware level. The TEE module operates in a privileged mode that sits alongside or above the hypervisor. Memory is divided between secure regions (accessible only to the TEE module) and regular regions. Current TEE module implementations can utilize the full resources of the CPU. ### Hardware Root of Trust TEE security properties are built from a hardware root of trust. During manufacturing, chip manufacturers fuse unique private keys directly into the chip using specialized one-time programmable memory. The fusing methods are designed so that keys cannot be extracted without destroying the chip. Attempts to read the keys through physical analysis would destroy the chip. These manufacturer-unique root keys establish the hardware root of trust. Manufacturers then can generate certificates which can be used to verify the authenticity of TEEs. The root keys are never used directly. Instead, TEEs implement key derivation hierarchies where application-specific keys are derived from root keys. This ensures that compromising one derived key doesn't expose others or the root. These derived keys have different use cases, most notably they are used to encrypt and decrypt memory pages for confidentiality guarantees. ### Establishing Confidentiality To establish confidentiality, clients first verify they're communicating with a legitimate TEE running on genuine hardware. A client receives an attestation report signed with keys that trace back to the hardware root of trust. Once verified, clients can verify that they are speaking with a legitimate TEE module. Clients can then encrypt their data with keys derived uniquely from the TEE, trusting that only the legitimate TEE can decrypt it. Confidentiality is maintained through memory encryption whenever the data is written to memory. Even privileged system software like hypervisors can only access encrypted data without the corresponding decryption keys. ### Establishing Verifiability The verifiability guarantee emerges from a process called "measured boot", a cryptographic chain that measures every piece of software before it executes. It begins with the hardware root of trust and builds a chain of confidence, step by step, through the entire system: 1. **Hardware Root Established**: The TEE module's authenticity is verified using the unforgeable hardware keys, establishing the foundation of trust. 2. **Firmware Trust**: Now that we trust the TEE module, it measures and cryptographically signs the firmware before execution. The trusted TEE vouches for the firmware's integrity. 3. **Kernel Trust**: The now-trusted firmware measures and signs the kernel before transferring control, extending the chain of trust. 4. **Application Trust**: The trusted kernel measures each application component before execution, completing the chain. Each measurement gets cryptographically signed and chained to the previous measurement using keys derived from the hardware root. The result is a tamper-evident record of exactly what software is running, anchored to the hardware root of trust. Any modification to any component in the chain produces a completely different measurement signature. ### The Attestation Proof TEEs generate attestation reports that can be handed to external parties to verify they are communicating with a legitimate TEE. These reports also contain the complete chain of measured boot with all measurements signed and traceable back to the hardware manufacturer's certificate chain. The client can verify this entire chain independently, confirming they're communicating with the exact software they expect, running with hardware-level confidentiality guarantees. ## TEE Vulnerabilities TEEs remain vulnerable to sophisticated attack vectors that exploit both hardware limitations and implementation flaws. Side-channel attacks represent the most prominent threat, where attackers analyze power consumption, electromagnetic emissions, timing variations, or cache behavior to extract sensitive information from supposedly secure enclaves. Physical access attacks, including fault injection and glitching techniques, can manipulate hardware behavior to bypass security mechanisms or extract cryptographic keys. These attacks typically require specialized equipment and physical proximity to the target system, making them difficult to execute at scale but representing real vulnerabilities that have been demonstrated against commercial TEE implementations in research environments. Supply chain attacks pose an emerging and particularly concerning threat vector. Malicious actors could compromise TEE hardware during the manufacturing process, embed backdoors in firmware or microcode, or subvert the cryptographic foundations that underpin attestation mechanisms. Nation-state actors with manufacturing capabilities represent the most significant risk in this category. Software vulnerabilities within TEE implementations also create attack surfaces, including bugs in the secure kernel, improper memory management, or flawed cryptographic implementations that could allow privilege escalation or information disclosure. Additionally, TEEs face architectural limitations that sophisticated adversaries can exploit. Rollback attacks can revert TEE state to previous versions, potentially exposing stale cryptographic material. Denial of service attacks can render TEE services unavailable, and in some implementations, certain classes of speculative execution vulnerabilities can leak information across security boundaries. While these attacks often require significant resources and expertise, they demonstrate that TEEs provide strong but not absolute security guarantees. ## Further Reading For a detailed technical deep dive into how confidential computing works under the hood using AMD SEV-SNP as a reference implementation, see the [Confidential Computing Primer](/docs/concepts/confidential-computing-primer). For how TEEs apply specifically to AI security, see our blog [Secure AI Needs TEEs](/blog/secure-ai-needs-tees). --- # Confidential Computing Primer This is a six-part series explaining confidential computing, the technology behind Confidential. Confidential uses Trusted Execution Environments (TEEs) to provide secure, private, verifiable AI. This series explains how TEEs work under the hood. We will be using AMD SEV-SNP as the reference for CPU based confidential computing. The concepts here (threat models, attestation, measurement) apply broadly to TEE technologies, while the implementation details are AMD-specific. The series assumes you're comfortable with basic virtualization concepts (VMs, hypervisors, memory management) and public key cryptography. It doesn't assume prior knowledge of AMD architecture or confidential computing. This series is for people who want to understand what's happening beneath the surface in detail, verify our claims, or evaluate confidential computing for their use case. ## What You'll Understand After reading this series, you'll be able to: - Explain what Confidential's TEE infrastructure protects against and what it doesn't - Trace how memory encryption and integrity protection work at the hardware level - Understand how workloads communicate safely with an untrusted hypervisor - Verify attestation reports and reason about what they prove - Understand how Confidential extends trust from hardware measurement to your application code ## The Documents ### [01. Threat Model & Security Boundaries](/docs/concepts/confidential-computing-primer/01-threat-model) Starts with the question: what does it mean to treat the hypervisor as adversarial? Defines the three guarantees (confidentiality, integrity, attestation), who you trust (AMD, the cryptographic primitives, your own code), and who you don't (cloud provider software stack, employees, other tenants). Explicitly covers what SEV-SNP does not protect against: denial of service, side channels, bugs in guest code, physical attacks. ### [02. Hardware Foundations](/docs/concepts/confidential-computing-primer/02-hardware-foundations) Explains the hardware that makes SEV-SNP possible. Covers AMD-V virtualization (the VMCB, VMRUN/#VMEXIT cycle, nested paging), the Platform Security Processor (the ARM core that's the root of trust), how memory encryption actually works (AES in the memory controller, per-address tweak, ASID-based key selection), and the evolution from SME through SEV, SEV-ES, to SEV-SNP. ### [03. Memory Integrity](/docs/concepts/confidential-computing-primer/03-memory-integrity) Addresses the gap that encryption alone leaves open. Explains the Reverse Map Table (RMP): its structure, how hardware performs RMP checks inline with every memory access, and how page validation works. Walks through specific attack scenarios (remapping, replay, aliasing) and shows exactly how the RMP catches them. ### [04. Privilege & Communication](/docs/concepts/confidential-computing-primer/04-privilege-and-communication) Covers how guests operate with an untrusted hypervisor. Explains VMPLs (the four privilege levels within an encrypted guest), how CPU state is protected in the encrypted VMSA, and the GHCB protocol (how the guest explicitly shares information with the hypervisor and validates responses). Introduces SVSM and restricted/alternate injection modes. ### [05. Attestation & Verification](/docs/concepts/confidential-computing-primer/05-attestation) Follows a measurement from launch through verification. Explains the SNP launch sequence (LAUNCH_START, LAUNCH_UPDATE, LAUNCH_FINISH), what gets measured and how, attestation report contents (measurement, policy, TCB, REPORT_DATA), the key hierarchy (ARK → ASK → VCEK), and practical verification steps. ### [06. Measurement Strategies](/docs/concepts/confidential-computing-primer/06-measurement-strategies) Addresses the gap between what the PSP measures at launch and what you actually run. Compares two approaches: packing everything into initramfs (PSP measures it directly) versus using dm-verity (kernel verifies filesystem against a root hash embedded in the measured initramfs). Covers SVSM and vTPM for key sealing when you need secrets bound to specific system state. ## Reading Order The documents build on each other. Document 1 (threat model) is conceptual and can be read standalone. Documents 2-5 should be read in order; each assumes concepts from the previous. Document 6 can be read after Document 5. If you're short on time: read Document 1 for the security model, skim Document 2 for hardware context, then jump to Document 5 for attestation. --- # Threat Model & Security Boundaries This threat model is the main driver behind Confidential's architecture. We designed our platform around the principle that even our own infrastructure shouldn't be able to see customer data. This document explains what confidential computing protects against and what it doesn't. After reading it, you'll understand where the trust boundaries lie, what an attacker with hypervisor control can and cannot do, and which guarantees you actually get from the hardware. This assumes you're comfortable with basic virtualization concepts (VMs, hypervisors, memory) but doesn't require any prior knowledge of confidential computing. ## The Security Guarantees SEV-SNP provides three guarantees. Understanding what each means, and what it doesn't mean, is essential for reasoning about what the technology actually gives you. **Confidentiality.** Guest memory is encrypted with a per-VM key that the hypervisor cannot access. Register state is encrypted when the VM exits to the hypervisor. The hypervisor sees ciphertext, not plaintext. This is a cryptographic guarantee: breaking it requires breaking AES-128 or compromising AMD's key management. **Integrity.** The hardware detects if the hypervisor tampers with guest memory. Substituting pages, replaying old data, or remapping addresses all trigger faults the guest can catch. The hypervisor cannot silently corrupt guest state. This is also a cryptographic guarantee, enforced by hardware checks on every memory access. **Attestation.** The guest can prove to a remote party what code it's running. AMD's security processor measures the initial guest image and signs a report with a key rooted in the silicon. A verifier can check this signature against AMD's certificate chain to confirm the report came from real hardware, not from software pretending to be a confidential VM. SEV-SNP does not provide availability. The hypervisor controls whether your VM runs, and can terminate it at any time. This is architectural: you can't enforce availability against someone who controls the power switch. ## Who You Trust SEV-SNP narrows the set of entities you must trust. Here's the explicit list. **You trust AMD.** You trust that AMD's manufacturing process didn't embed backdoors in the silicon. You trust that AMD's key management practices keep root keys secure. You trust that the Platform Security Processor (PSP) firmware doesn't have exploitable vulnerabilities. You trust that AMD's key distribution service serves authentic certificates. **You trust the cryptographic primitives.** AES-128 for memory encryption, ECDSA P-384 for attestation signatures, SHA-384 for measurements. If these break, the guarantees break. **You trust the physical host of the hardware.** There is a class of non-trivial physical attacks, requiring specialized hardware and knowledge, that can break TEE security. Who ever physically hosts the hardware can implement these attacks. **You trust your own software.** SEV-SNP protects the execution environment, not the code running in it. If your kernel has a vulnerability, an attacker can exploit it. Attestation proves what code loaded, not that the code is correct. ## Who You Don't Trust The value of SEV-SNP comes from what you no longer need to trust. **You don't trust the cloud provider's software stack.** The hypervisor, host OS, management plane, orchestration systems, and monitoring agents are all outside the trust boundary. They can be compromised, malicious, or buggy without affecting the confidentiality or integrity of your guest. **You don't trust the cloud provider's employees.** Administrators with root access to the hypervisor, datacenter technicians with physical access to servers, and anyone in the operational chain cannot read your guest's memory or tamper with it undetected. **You don't trust other tenants.** Even if another VM on the same physical host is compromised and escalates to hypervisor level, it cannot access your guest's data. **You don't trust the network or storage infrastructure.** Data leaving the encrypted memory boundary (to disk, to network) passes through untrusted channels. You use software encryption (TLS, LUKS) to protect it there. SEV-SNP protects data while it's being processed in memory. ## The Traditional Trust Model In conventional virtualization, a guest virtual machine trusts everything below it. The guest trusts the hypervisor to correctly isolate it from other guests. The hypervisor trusts the host operating system to manage hardware resources fairly. The entire stack trusts the hardware to execute instructions correctly. This forms a hierarchy where each layer assumes the layers beneath it are honest. This model works fine when you control the infrastructure. If you own the physical server, you trust it because you trust yourself. But cloud computing inverts this: you're running on someone else's hardware, managed by someone else's software, operated by someone else's employees. The traditional trust model asks you to trust all of them. Cloud providers implement extensive security controls: employee background checks, access audits, network segmentation, encrypted storage. These are meaningful and reduce risk. But they don't change the fundamental architecture: the hypervisor has complete visibility into and control over your VM's memory, registers, and execution. ## The Adversarial Hypervisor SEV-SNP treats the hypervisor as adversarial. This isn't a statement about cloud provider intentions. It's a design principle: assume the worst-case attacker has full control of the hypervisor and host OS, then design hardware that protects the guest anyway. What does "full hypervisor control" mean in practice? The attacker can: - Read and write any region of host memory - Inspect and modify the nested page tables that translate guest physical addresses to system physical addresses - Control which physical pages back the guest's memory - Intercept and inject interrupts - Pause, resume, snapshot, and migrate the VM at will - Control the I/O devices the guest thinks it's talking to In traditional virtualization, this level of access means total compromise. The attacker can dump the guest's memory, extract encryption keys, modify code in flight, and observe every piece of data the guest processes. SEV-SNP's goal is to make most of these attacks impossible despite the attacker having this level of access. ## What SEV-SNP Protects Against SEV-SNP provides hardware-enforced protections against a specific set of attacks. The protections fall into two categories: memory confidentiality and memory integrity. ### Memory Confidentiality **Reading guest memory.** The memory controller encrypts all guest memory with a per-VM key (the VEK, VM Encryption Key) that only the AMD security processor knows. When the hypervisor reads a guest's memory region, it gets ciphertext. The encryption is AES-128 with a physical-address-based tweak, meaning the same plaintext encrypts differently at different addresses. The hypervisor cannot derive the key: it's generated by hardware, stored in hardware, and never exposed to any x86 software. **Inspecting register state.** When a VM exits to the hypervisor (for interrupt handling, I/O, etc.), traditional virtualization stores the guest's register state in a memory structure the hypervisor reads. SEV-SNP encrypts this save area (called the VMSA) with the guest's VEK. The hypervisor can trigger VM exits but cannot read the register values. ### Memory Integrity **Modifying guest memory.** Encryption alone doesn't prevent modification attacks. An attacker who can't read your data might still corrupt it strategically. SEV-SNP adds the Reverse Map Table (RMP): a hardware structure that tracks which VM owns each physical page and what guest address it should map to. Every memory access by a guest triggers an RMP check. If the hypervisor modifies the page contents, the guest gets cryptographic garbage (wrong key). If the hypervisor tries to substitute a different physical page, the RMP check fails because the expected guest address won't match. **Replaying old memory contents.** Without integrity protection, an attacker could snapshot a guest's memory, let it run for a while, then restore the old snapshot. This could revert security-critical state (nonces, sequence numbers, key material). The RMP validation mechanism prevents this: each page has a "validated" bit that the guest sets after verifying the page. If the hypervisor swaps in a page from an old snapshot, it won't have the validated bit set, and the hardware rejects the access. **Remapping guest addresses.** The hypervisor controls the nested page tables that translate guest physical addresses (GPAs) to system physical addresses (SPAs). Without protection, it could silently remap GPA 0x1000 from pointing at physical page A to pointing at physical page B. SEV-SNP prevents this by storing the expected GPA in each RMP entry. The hardware checks that the GPA from the page table walk matches the GPA recorded in the RMP. A mismatch causes a fault. ## What SEV-SNP Does NOT Protect Against The security boundary has clear limits. These aren't weaknesses to be fixed in a future version. They're architectural constraints that follow from what confidential computing is trying to achieve. ### Availability and Resource Control **Denial of service.** The hypervisor can always refuse to schedule your VM. It can terminate your VM at any time. It controls the power and the physical resources. SEV-SNP protects confidentiality and integrity, not availability. If you need guaranteed uptime, you need operational controls (SLAs, redundancy, multi-cloud), not hardware encryption. **Resource starvation.** The hypervisor controls CPU scheduling, memory allocation, and I/O bandwidth. It can slow your VM to a crawl without technically killing it. Timing-sensitive workloads remain vulnerable to performance interference. ### Information Leakage Channels **Side-channel attacks.** The CPU still shares microarchitectural state between the guest and the hypervisor. Cache timing, branch prediction, power analysis, and electromagnetic emissions can leak information. AMD has added mitigations for some attacks (separate branch prediction state, cache partitioning options), but side channels remain an active research area. SEV-SNP significantly raises the bar compared to unprotected VMs, but it doesn't eliminate all side-channel risk. **I/O visibility.** I/O devices sit outside the encrypted memory boundary. When the guest reads from disk or sends network traffic, that data passes through shared memory regions the hypervisor can observe. Guests typically use software encryption (TLS, LUKS) to protect data in transit and at rest. SEV-SNP protects data while it's being processed in memory, not while it's moving through I/O channels. **Interrupt timing.** The hypervisor controls when interrupts are delivered. Even with restricted injection mode, a malicious hypervisor can observe timing patterns: when the guest handles interrupts, how long operations take, what the guest is doing when it's interruptible. This is a form of side channel. ### Software Vulnerabilities **Bugs in guest code.** SEV-SNP protects the execution environment, not the software running in it. If your application has a buffer overflow, an attacker who can send it malicious input can exploit it. The attacker might be the cloud operator sending data through a network interface, or it might be a legitimate user whose requests happen to trigger the bug. The hardware can't tell the difference between intended and malicious computation. **Compromised guest images.** Attestation proves that a specific set of code was loaded at boot. It doesn't prove that code is correct, secure, or does what you want. If you deploy a backdoored kernel, attestation faithfully reports the backdoored kernel's measurement. The verifier needs to know what measurements to expect and trust that those measurements correspond to trustworthy software. ### Physical Attacks **Memory Interposer attacks.** If an attacker can physically freeze the DRAM and move it to another system, they might recover data. SEV-SNP's encryption happens in the memory controller on the CPU die; once data leaves the CPU package and sits in DRAM, it's encrypted. However, sophisticated physical attacks against DRAM (voltage glitching, probing) are an active research area. SEV-SNP raises the bar significantly compared to unencrypted memory, but it's not designed to resist nation-state-level hardware attacks. **Hardware tampering.** If an attacker can modify the CPU itself, replace the memory controller, or intercept signals on the CPU package, all bets are off. SEV-SNP assumes the CPU die is intact. Protecting against supply chain attacks on the silicon is outside its scope. ### Trust Anchor Compromise **Compromised AMD hardware or firmware.** The trust anchor is AMD's security processor and the keys fused into the silicon at manufacturing. If AMD's manufacturing process is compromised, or if the security processor firmware has vulnerabilities, the guarantees break. This is a much smaller attack surface than "trust the entire cloud stack," but it's not zero. You're trading trust in your cloud provider for trust in AMD. ## The Trust Anchor Every security system has a root of trust: something you assume is correct because you can't verify it further down. In SEV-SNP, the trust anchor is AMD's Platform Security Processor (PSP) and the cryptographic keys fused into the CPU during manufacturing. The PSP is an ARM core embedded in the AMD processor die. It runs its own firmware, independent of the x86 cores running your hypervisor and guests. It generates the per-VM encryption keys, manages attestation, and enforces security policies. The PSP has access to hardware key material that no x86 software, at any privilege level, can read. Attestation reports are signed by a key derived from fuses in the silicon: the VCEK (Versioned Chip Endorsement Key). AMD's key distribution service provides certificates that chain back to AMD's root certificate. A verifier checks this chain to confirm that the attestation report really came from an AMD processor, not from software pretending to be one. This design means you're trusting: (1) AMD's manufacturing process didn't embed backdoors in the silicon, (2) AMD's key management practices keep the root keys secure, (3) the PSP firmware doesn't have vulnerabilities that allow bypass, and (4) the cryptographic primitives (AES, ECDSA, SHA) remain secure. These are meaningful assumptions, but they're substantially narrower than trusting every layer of a cloud provider's software stack. ## Summary SEV-SNP shifts the trust boundary from "trust the cloud provider's entire stack" to "trust AMD's silicon." The hypervisor is treated as adversarial: it can observe and manipulate the VM's external interfaces but cannot read memory, registers, or corrupt pages without detection. The protection is enforced by hardware (the memory controller's encryption engine and the RMP) using keys the hypervisor never sees. The boundaries are precise. You get strong confidentiality and integrity for data in memory during processing. You don't get availability guarantees, side-channel immunity, or protection against bugs in your own code. The trust anchor is AMD's manufacturing and key management; if those fail, the guarantees fail. Understanding these boundaries is essential for reasoning about what confidential computing actually gives you. The following documents explain how the hardware enforces these boundaries: how encryption works, how the RMP provides integrity, how guests communicate with the hypervisor safely, and how attestation lets you verify the whole thing. Confidential's platform is built on this threat model. Our infrastructure services—firewalls, load balancers, routing—run in TEEs themselves. We can't see your data, and you don't have to trust that we won't look. The hardware enforces it. --- # Hardware Foundations This document explains the hardware components that make SEV-SNP possible. After reading it, you'll understand how AMD-V virtualization works at the hardware level, what the Platform Security Processor does and why it's the root of trust, how memory encryption actually happens, and how SEV evolved through four generations to reach SNP. This assumes you're comfortable with basic operating system concepts (virtual memory, privilege levels) but doesn't require prior knowledge of AMD-specific architecture. ## AMD-V Virtualization SEV-SNP builds on AMD-V, the hardware virtualization extension that AMD introduced in 2006. Understanding AMD-V is essential because SEV-SNP doesn't replace virtualization; it adds security properties on top of it. ### The Problem Virtualization Solves You want to run multiple operating systems on one physical machine, each believing it has the entire computer to itself. The software that creates this illusion is the hypervisor. The challenge is that an operating system expects total control: it wants to manage memory, handle interrupts, and execute privileged instructions. If you ran two OSes side by side without mediation, they'd fight over resources and crash. Early hypervisors solved this through software trapping: intercept every privileged operation, simulate its effect, return control. This worked but was slow. AMD-V (and Intel's equivalent, VT-x) added explicit CPU modes for "running a guest" versus "running the hypervisor", letting hardware handle most isolation directly. ### The VMCB The VMCB (Virtual Machine Control Block) is a data structure in memory that captures everything the hardware needs to run a guest VM. It has two parts. ``` ┌───────────────────────────────────────────────────────────────┐ │ VMCB │ ├───────────────────────────────────────────────────────────────┤ │ CONTROL AREA (hypervisor readable/writable) │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Intercepts: which operations cause exits │ │ │ │ ASID: which encryption key to use │ │ │ │ nCR3: pointer to nested page tables │ │ │ │ Exit code: why did we exit? (filled by hardware) │ │ │ │ Exit info: fault address, etc. │ │ │ └─────────────────────────────────────────────────────────┘ │ ├───────────────────────────────────────────────────────────────┤ │ SAVE AREA / VMSA (encrypted in SEV-ES/SNP) │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ RAX, RBX, RCX, RDX, RSI, RDI, RBP, RSP, R8-R15 │ │ │ │ RIP (instruction pointer) │ │ │ │ RFLAGS │ │ │ │ CS, DS, SS, ES, FS, GS (segment registers) │ │ │ │ CR0, CR3, CR4 (control registers) │ │ │ └─────────────────────────────────────────────────────────┘ │ └───────────────────────────────────────────────────────────────┘ ``` The **Control Area** contains instructions for the hardware: which operations should cause exits to the hypervisor, which ASID to use (we'll explain this shortly), the pointer to nested page tables, and fields that hardware fills in when an exit occurs (exit code, fault address, etc.). The hypervisor reads and writes this area freely. The **Save Area** contains the guest's actual CPU state: all general-purpose registers (RAX through R15), the instruction pointer (RIP), flags, segment registers, and control registers. This is the complete "brain state" of the guest at any moment. In SEV-ES and SEV-SNP, this save area becomes the VMSA (VM Save Area) and gets encrypted. ### The VMRUN / #VMEXIT Cycle Guest execution follows a simple loop: 1. The hypervisor sets up a VMCB for the guest it wants to run and executes the VMRUN instruction. 2. Hardware saves some hypervisor state, loads the guest's registers from the VMCB save area, switches to "guest mode," and jumps to the guest's instruction pointer. 3. The guest executes at near-native speed until something triggers an exit: an intercepted instruction (like CPUID), an I/O access, a page fault, or an external interrupt. 4. Hardware saves the guest's registers back to the VMCB save area, writes exit information to the control area, restores hypervisor state, and resumes hypervisor execution right after the VMRUN. 5. The hypervisor reads the exit code, handles whatever caused the exit, and calls VMRUN again. This cycle repeats millions of times per second. Most guest execution happens at full speed; the hypervisor only gets involved when something interesting happens. ``` Hypervisor Guest │ │ │ VMRUN (load VMCB, switch to guest) │ ├──────────────────────────────────────►│ │ │ executing... │ │ executing... │ │ CPUID instruction │◄──────────────────────────────────────┤ #VMEXIT │ │ │ read exit code from VMCB │ │ handle the exit (emulate CPUID) │ │ update VMCB if needed │ │ │ │ VMRUN │ ├──────────────────────────────────────►│ │ │ continues executing... │ │ ``` ### Why This Matters for Security On every #VMEXIT, the guest's complete register state is written to the VMCB save area. Without encryption, the hypervisor can see everything: what code the guest is executing (RIP), all data in registers (maybe passwords, keys, sensitive data), and what the guest is about to do. It can also modify any of it before calling VMRUN again. This is the gap that SEV-ES closes by encrypting the save area. But we're getting ahead of ourselves. ### Nested Paging Before hardware nested paging, hypervisors maintained "shadow page tables": intercepting every guest page table modification and maintaining a parallel structure that mapped guest virtual addresses directly to physical addresses. This caused constant VM exits and was slow. AMD-V introduced nested paging (Intel calls it EPT, Extended Page Tables). The CPU now understands two levels of address translation natively: ``` ┌────────────────────────────────────────────────────────────────────────┐ │ Two-Level Address Translation │ │ │ │ Guest Process │ │ ┌─────────────────┐ │ │ │ GVA: 0x7fff1234 │ Guest Virtual Address │ │ └────────┬────────┘ │ │ │ │ │ ▼ Guest page tables (guest OS controls) │ │ ┌─────────────────┐ │ │ │ GPA: 0x00050000 │ Guest Physical Address │ │ └────────┬────────┘ (what guest thinks is "physical") │ │ │ │ │ ▼ Nested page tables (hypervisor controls) │ │ ┌─────────────────┐ │ │ │ SPA: 0x1a500000 │ System Physical Address │ │ └────────┬────────┘ (actual DRAM location) │ │ │ │ │ ▼ │ │ [ DRAM ] │ └────────────────────────────────────────────────────────────────────────┘ ``` Hardware walks both translations automatically on every memory access. The nested page table root is stored in the VMCB's nCR3 field. Guest page table modifications no longer cause exits; the hypervisor only gets involved when the GPA-to-SPA translation fails (a "nested page fault," indicating the guest needs more memory). The hypervisor controls the nested page tables. It decides which SPA backs each GPA. This is necessary for virtualization, but it's also where attacks become possible. A malicious hypervisor could remap GPAs to wrong SPAs, alias multiple GPAs to the same SPA, or replay old memory contents. Encryption alone doesn't prevent these integrity attacks. This is why SNP adds the RMP check at the end of every memory access. ## The Platform Security Processor The AMD Secure Processor (also called AMD-SP or PSP) is a dedicated ARM Cortex-A5 microcontroller physically integrated onto the CPU die. This is not a separate chip that could be bypassed; it's part of the silicon. ### What It Does The PSP runs its own firmware completely independently of the x86 cores. When you boot an AMD EPYC system, the PSP boots first and initializes before the x86 cores even start executing. This makes it the hardware root of trust for SEV. The PSP manages: **Key generation.** All VM encryption keys (VEKs) are generated by the PSP using an on-chip NIST SP 800-90 compliant hardware random number generator. The VEK is a 128-bit AES key, unique per guest instance. Even if you launch the same guest image twice, each gets a different VEK. **Guest context.** The PSP maintains an encrypted, integrity-protected data structure for each SNP guest called the GCTX (Guest Context). This contains the VEK, measurement digest, policy, VMPCKs (VM Platform Communication Keys), and other security state. The hypervisor can see the GCTX page exists but cannot read or modify its contents. **VM lifecycle.** Launch, activate, deactivate, and migration operations all go through the PSP. The hypervisor requests these operations; the PSP performs them and enforces security invariants. **Attestation.** The PSP generates and signs attestation reports. The signing key (VCEK or VLEK) is derived from keys fused into the silicon at manufacturing. ### Communication with the PSP The x86 cores communicate with the PSP through MMIO mailbox registers in PCI space. The protocol is request/response: 1. x86 software allocates a command buffer in DRAM. 2. Writes the physical address to CmdBufAddr registers. 3. Writes the command ID to the CmdResp register. 4. PSP reads the command, executes it, writes results back, and signals completion. This is intentionally slow and course-grained. The PSP doesn't sit in the fast path of normal guest execution; it only handles security-critical operations. ### Why This Architecture Matters The PSP being a separate processor with its own firmware means the x86 hypervisor cannot read PSP internal state. VEKs never leave the PSP in cleartext; they go directly to the memory controller's key slots through a hardware interface the hypervisor cannot access. Even if an attacker has full control of the hypervisor, they cannot extract VM encryption keys. ## Memory Encryption The actual encryption happens in dedicated AES engines located in the on-die memory controllers, not in the CPU cores. ### Location in the Data Path ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ CPU Die (trusted) │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ CPU Core │ │ CPU Core │ │ AMD-SP │ │ │ │ │ │ │ │ (PSP) │ │ │ │ Registers │ │ Registers │ │ │ │ │ │ (plaintext) │ │ (plaintext) │ │ VEK store │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ │ └─────────┬─────────┘ │ programs keys │ │ ▼ ▼ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ L1 / L2 / L3 Cache (plaintext) │ │ │ └───────────────────────────────┬────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Memory Controller │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ │ │ Key Slots: ASID 1 → VEK_1, ASID 2 → VEK_2, ... │ │ │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ │ │ AES-128 Engine │ │ │ │ │ │ encrypt on write, decrypt on read │ │ │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ └───────────────────────────────┬────────────────────────────────────┘ │ │ │ │ └──────────────────────────────────┼───────────────────────────────────────┘ │ ════════════════╪════════════════ (SoC boundary) │ ▼ ┌──────────────────────────────────────────────────────────────────────────┐ │ DRAM (ciphertext only) │ │ │ │ Physical probing, cold boot attacks, stolen DIMMs → encrypted garbage │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` Encryption and decryption occur at the boundary between the SoC and DRAM. Data inside the CPU caches is plaintext. This is a deliberate design choice: encrypting on every cache hit would be slow, and the threat model assumes the CPU package is trusted. An attacker who can probe inside the CPU die has capabilities beyond what memory encryption addresses. ### How Encryption Works The memory controller maintains key slots, one per ASID. When a memory transaction arrives: 1. The transaction is tagged with the originating ASID (set in the VMCB and carried through the entire memory pipeline). 2. The memory controller looks up the VEK for that ASID. 3. On writes, plaintext from cache is encrypted with the VEK before going to DRAM. 4. On reads, ciphertext from DRAM is decrypted with the VEK before going to cache. The encryption uses AES-128 in XEX mode (a variant of XTS) with the physical address as a "tweak." The tweak ensures that the same plaintext at different physical addresses encrypts to different ciphertext. This prevents ciphertext block move attacks where an attacker copies encrypted data from one location to another. ### The ASID The ASID (Address Space Identifier) is a 16-bit identifier that tags memory transactions. The hypervisor sets the ASID in the VMCB before VMRUN; hardware ensures all memory accesses from that guest carry the correct ASID. ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ ASID-Based Key Selection │ │ │ │ VMCB for VM #2: Memory Controller Key Slots: │ │ ┌──────────────────┐ ┌───────────────────────────┐ │ │ │ ASID = 2 │ │ Slot 1: VEK for VM #1 │ │ │ │ ... │ │ Slot 2: VEK for VM #2 ◄──┼─┐ │ │ └──────────────────┘ │ Slot 3: VEK for VM #3 │ │ │ │ │ ... │ │ │ │ └───────────────────────────┘ │ │ │ │ │ │ Guest memory access tagged with ASID=2 ─────────────────────────────┘ │ │ │ │ Memory controller uses ASID to select correct VEK automatically │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` ASIDs are a limited resource. The exact count varies by processor (exposed via CPUID Fn8000_001F[ECX]), typically a few hundred. If you have more VMs than ASIDs, you need ASID overcommit: deactivate one guest, flush caches, activate another. The SNP_ACTIVATE command installs a guest's VEK into the key slot for a given ASID; SNP_DEACTIVATE removes it. ### The C-bit The C-bit (bit 47 of guest physical addresses in page table entries, though the exact position is processor-dependent and reported by CPUID) controls whether a page is encrypted: | C-bit | Meaning | Encryption | | ----- | ------- | ------------------------------ | | 1 | Private | Encrypted with guest's VEK | | 0 | Shared | Not encrypted with guest's VEK | The guest sets the C-bit in its own page tables. The hypervisor cannot override this. Most pages are private (C=1). Shared pages (C=0) are used for communication with the hypervisor (the GHCB) and for DMA buffers (since devices doing DMA don't have a guest ASID). ### DMA and Bounce Buffers Devices doing DMA don't execute in guest context, so they don't have a guest ASID. If a device DMAs to an encrypted page, it would read/write with the wrong key (or no key). The solution: guests must use shared (C=0) pages for any memory that devices will DMA to. For I/O operations, guests use "bounce buffers": allocate a shared buffer, copy data there, let the device DMA, copy the result back to private memory. This adds overhead but preserves confidentiality. Linux's SWIOTLB (Software I/O TLB) implements this transparently for SEV guests. ## The Evolution: SME → SEV → SEV-ES → SEV-SNP Each generation addressed security gaps left by its predecessor. Understanding the progression clarifies what protections you actually get. ### SME (Secure Memory Encryption) SME introduced the core encryption mechanism: the AES engine in the memory controller and the C-bit in page tables. **What it does:** Uses a single AES-128 key for all memory encryption. The OS or hypervisor controls which pages are encrypted via the C-bit. **What it protects against:** Physical attacks only. Cold boot attacks, memory bus probing, stolen DIMMs. If all memory is encrypted, physical access to DRAM gets you ciphertext. **What it doesn't protect against:** The hypervisor, other VMs, anything with software access to memory. There's only one key for the whole system; everyone who can access memory can decrypt. ### SEV (Secure Encrypted Virtualization) SEV's breakthrough was per-VM encryption keys. **What it added:** Each VM gets its own VEK, selected via the ASID. The hypervisor reading VM memory sees encrypted garbage. **What it protects against:** Hypervisor reading guest memory contents. The hypervisor can see ciphertext but cannot decrypt. **What it doesn't protect against:** On #VMEXIT, guest registers are saved to the VMCB save area in plaintext. The hypervisor can read all register values, including sensitive data being computed, and can modify register state before the VM resumes. ### SEV-ES (Encrypted State) SEV-ES closes the register exposure gap. **What it added:** The VMCB save area (now called VMSA) is encrypted with the guest's VEK on every #VMEXIT. The hypervisor sees ciphertext. **The #VC exception:** With encrypted registers, the hypervisor can't see what the guest needs when an exit occurs. SEV-ES introduces the #VC (VMM Communication) exception. When the guest hits an intercept, #VC fires inside the guest. The guest's handler decides what information to share, puts only the necessary data in the GHCB (a shared page), and executes VMGEXIT. The guest controls exactly what crosses the trust boundary. **What it doesn't protect against:** Memory integrity attacks. A malicious hypervisor can still: - **Replay:** Replace a memory page with an old copy. - **Corrupt:** Flip bits in encrypted memory (decrypts to garbage, potentially exploitable). - **Alias:** Map two different guest addresses to the same physical page. - **Remap:** Silently swap which physical page backs a guest address. These don't let the attacker read data, but they can cause the guest to operate on incorrect data. ### SEV-SNP (Secure Nested Paging) SEV-SNP adds hardware-enforced integrity. The fundamental guarantee: if a guest reads a private page, it sees the value it last wrote, or gets an exception. Never stale data, never corrupted data, never another page's data. **The RMP (Reverse Map Table):** A single system-wide table with one entry per 4KB physical page, tracking: is this page assigned to a VM? Which VM (ASID)? What GPA should it map to? Is it validated? What are the VMPL permissions? Every memory access to a private guest page triggers an RMP check. If the access doesn't match what the RMP says (wrong ASID, wrong GPA, not validated), you get an exception. **Page validation:** Two-step process: 1. Hypervisor uses RMPUPDATE to assign a page to a guest at a specific GPA. The page becomes Guest-Invalid. 2. Guest uses PVALIDATE to validate the page. The page becomes Guest-Valid. The guest must explicitly accept each mapping. If the hypervisor swaps the backing page, the new page won't be validated, and the guest gets a #VC when accessing it. **VMPLs (VM Privilege Levels):** Four privilege levels (0-3) within the guest. Each RMP entry has separate permissions per VMPL. VMPL0 is highest privilege; it can run an SVSM (Secure VM Service Module) that provides services to the guest OS at VMPL2/3. **Additional protections:** Restricted interrupt injection (hypervisor can only inject a doorbell, not arbitrary interrupts), BTB protection (hardware flushes branch predictor entries from untrusted sources), TCB versioning (attestation reports include firmware versions, enabling verifiers to require minimum versions). ### Summary: The Protection Matrix | Threat | SME | SEV | SEV-ES | SEV-SNP | | ----------------------------- | --- | --- | ------ | ------- | | Some Phyiscal DRAM attacks | Yes | Yes | Yes | Yes | | Hypervisor reads VM memory | No | Yes | Yes | Yes | | Hypervisor reads VM registers | No | No | Yes | Yes | | Memory replay | No | No | No | Yes | | Memory corruption | No | No | No | Yes | | Memory aliasing | No | No | No | Yes | | Memory remapping | No | No | No | Yes | ## Key Hardware Structures Summary ### ASID Key Slots Limited slots in the memory controller, each holding one VEK. The SNP_ACTIVATE command programs a VEK into a slot; SNP_DEACTIVATE clears it. If you need more guests than slots, you overcommit by deactivating idle guests. ### The RMP One 16-byte entry per 4KB physical page. Indexed by system physical address. Modified only through specific privileged instructions (RMPUPDATE, PVALIDATE) or PSP commands. RMP checks are performed by CPU microarchitecture on every relevant memory access; this is hardware logic, not software the hypervisor could bypass. ### The GCTX A 4KB page maintained by the PSP for each guest. Contains the VEK, measurement digest, VMPCKs, policy, and other state. Encrypted by the PSP (hypervisor sees ciphertext) and marked immutable in the RMP (hypervisor cannot modify the RMP entry). All guest secrets live here. ### The VMSA The encrypted save area containing guest register state. Encrypted with the guest's VEK. The hypervisor specifies a pointer to the VMSA in the VMCB control area, but cannot read or write the VMSA contents. ### The GHCB A shared (C=0) page for guest-hypervisor communication. The guest explicitly puts information here when it needs hypervisor services. The hypervisor can read and write the GHCB, but the guest controls what information is placed there. ## The Trust Boundary Putting this together, here's where the security boundary lives: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ TRUSTED (inside CPU die) │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │ │ │ CPU Cores │ │ AMD-SP (PSP) │ │ Memory Controller │ │ │ │ │ │ │ │ │ │ │ │ • Registers │ │ • VEK storage │ │ • Key slots │ │ │ │ • RMP checks │ │ • GCTX mgmt │ │ • AES engine │ │ │ │ • Execution │ │ • Attestation │ │ • RMP table │ │ │ └──────────────────┘ └──────────────────┘ └───────────────────┘ │ │ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ L1 / L2 / L3 Cache (plaintext) │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ ════════════════╪════════════════ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ UNTRUSTED (outside CPU die) │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │ │ │ DRAM │ │ Memory Bus │ │ DMA Devices │ │ │ │ (ciphertext) │ │ (ciphertext) │ │ (shared pages │ │ │ │ │ │ │ │ only) │ │ │ └──────────────────┘ └──────────────────┘ └───────────────────┘ │ │ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Hypervisor │ │ │ │ • Sees ciphertext (cannot decrypt) │ │ │ │ • Cannot modify guest memory without RMP detection │ │ │ │ • Cannot read encrypted VMSA registers │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` The PSP programs keys into the memory controller. The CPU performs RMP checks inline with memory access. The memory controller encrypts and decrypts. All of this is hardware that the hypervisor cannot modify. --- # Memory Integrity This document explains how SEV-SNP protects the integrity of guest memory. After reading this, you'll understand how the Reverse Map Table (RMP) works, why guests must validate pages, and how these mechanisms together prevent memory manipulation attacks. This assumes you've read [02-hardware-foundations.md](/docs/concepts/confidential-computing-primer/02-hardware-foundations) and understand how memory encryption works. ## The Problem Encryption Doesn't Solve Memory encryption (covered in the previous document) protects confidentiality: the hypervisor sees ciphertext, not plaintext. But encryption alone doesn't prevent the hypervisor from manipulating which ciphertext the guest sees. Consider this: the hypervisor controls the nested page tables that map guest physical addresses (GPAs) to system physical addresses (SPAs). Even with encryption, a malicious hypervisor could: **Remapping attack.** The guest writes sensitive data to GPA 0x1000, which backs to SPA X. Later, the hypervisor changes the nested page table so GPA 0x1000 maps to SPA Y instead. The guest reads GPA 0x1000 expecting its data, but gets whatever was at SPA Y. The hypervisor didn't need to decrypt anything; it just swapped the backing page. **Replay attack.** The hypervisor captures the ciphertext at SPA X at time T₁. Later, after the guest updates that memory, the hypervisor restores the old ciphertext. The guest now sees stale data without realizing it. **Aliasing attack.** The hypervisor maps two different guest addresses (GPA A and GPA B) to the same physical page (SPA X). Writes to GPA A appear at GPA B and vice versa. This can corrupt data structures or leak information between components that thought they had separate memory. **Corruption attack.** The hypervisor writes random bytes to a guest's physical memory. The ciphertext is garbage, which decrypts to garbage. The hypervisor can't control what the garbage says, but corrupting critical data structures can crash the guest or cause exploitable behavior. These are all integrity attacks. SEV and SEV-ES had no defense against them. SEV-SNP adds the RMP to close this gap. ## The Reverse Map Table The RMP is a system-wide table in DRAM with one entry per 4KB physical page. It answers the question: who owns this physical page, and where should it appear in their address space? ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ RMP Structure in Memory │ │ │ │ System Physical Address Space │ │ ┌──────────────┬──────────────┬──────────────┬──────┬──────────────┐ │ │ │ 0x0000_0000 │ 0x0000_1000 │ 0x0000_2000 │ ... │ 0xFFFF_F000 │ │ │ │ Page 0 │ Page 1 │ Page 2 │ │ Page N │ │ │ └──────┬───────┴──────┬───────┴──────┬───────┴──────┴──────┬───────┘ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ │ │ RMP Table (indexed by SPA >> 12) │ │ ┌──────────────┬──────────────┬──────────────┬──────┬──────────────┐ │ │ │ Entry 0 │ Entry 1 │ Entry 2 │ ... │ Entry N │ │ │ │ 16 bytes │ 16 bytes │ 16 bytes │ │ 16 bytes │ │ │ └──────────────┴──────────────┴──────────────┴──────┴──────────────┘ │ │ │ │ Each entry tracks: owner, expected GPA, validated?, permissions │ └──────────────────────────────────────────────────────────────────────────┘ ``` The RMP is configured via MSRs (RMP_BASE and RMP_END) and must be 1MB aligned. There's one RMP for the entire system, even with multiple sockets. ### RMP Entry Fields Each 16-byte entry contains: | Field | Purpose | | ---------- | -------------------------------------------------- | | Assigned | Is this page assigned to a guest (or firmware)? | | ASID | Which guest owns this page (0 = hypervisor/free) | | GPA | The guest physical address this page should map to | | Validated | Has the guest explicitly accepted this page? | | Immutable | Is this page locked for firmware operations? | | VMSA | Is this a VM Save Area page? | | Page_Size | 4KB or 2MB | | VMPL perms | Read/write/execute permissions per VMPL (0-3) | The combination of these fields determines the page state. ### Page States ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Page State Overview │ │ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ MUTABLE STATES │ │ │ │ (can be modified by RMPUPDATE, PVALIDATE, or RMPADJUST) │ │ │ │ │ │ │ │ Hypervisor Guest-Invalid Guest-Valid │ │ │ │ ┌────────────┐ ┌────────────────┐ ┌────────────────┐ │ │ │ │ │ Assigned=0 │ │ Assigned=1 │ │ Assigned=1 │ │ │ │ │ │ ASID=0 │ │ Validated=0 │ │ Validated=1 │ │ │ │ │ │ │ │ ASID=guest │ │ ASID=guest │ │ │ │ │ └────────────┘ └────────────────┘ └────────────────┘ │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ IMMUTABLE STATES │ │ │ │ (only AMD-SP firmware can modify) │ │ │ │ │ │ │ │ Pre-Guest Pre-Swap Firmware Metadata Context │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ │ │ Launch │ │ Swap-out│ │ AMD-SP │ │ Swap │ │ Guest │ │ │ │ │ │ prep │ │ prep │ │ working │ │ tags │ │ context │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ │ └────────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ``` The three mutable states are the ones you'll encounter most often: **Hypervisor.** The default state. The page belongs to the hypervisor. It can be used for hypervisor data, or as shared (C=0) memory that guests can access. **Guest-Invalid.** The page is assigned to a guest at a specific GPA, but the guest hasn't validated it yet. The hypervisor cannot write to it (the ciphertext would be garbage without the VEK). The guest cannot use it for private (C=1) memory because it's not validated. **Guest-Valid.** The page is assigned to the guest and validated. This is the normal state for guest private memory. The guest can read and write freely. The hypervisor sees only ciphertext. The immutable states exist for specific firmware operations (launch, swap, etc.) where neither hypervisor nor guest should interfere. ## The RMP Check Hardware performs an RMP check at the end of address translation for accesses that require it. This happens inline with every relevant memory access. ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ RMP Check Flow (Guest Access) │ │ │ │ 1. Guest executes load/store at GVA │ │ └───────────────────────┐ │ │ ▼ │ │ 2. Guest page tables ┌───────────────────────┐ │ │ translate GVA │ GVA → GPA │ │ │ │ 0x7fff1234 → 0x50000 │ │ │ └───────────┬───────────┘ │ │ ▼ │ │ 3. Nested page tables ┌───────────────────────────┐ │ │ translate GPA │ GPA → SPA │ │ │ (hypervisor controls) │ 0x50000 → 0x1a500000 │ │ │ └───────────┬───────────────┘ │ │ ▼ │ │ 4. RMP lookup ┌───────────────────────────────────────┐ │ │ RMP[SPA >> 12] │ RMP entry for SPA 0x1a500000: │ │ │ │ ASID = 7 │ │ │ │ GPA = 0x50000 │ │ │ │ Validated = 1 │ │ │ └───────────┬───────────────────────────┘ │ │ ▼ │ │ 5. Hardware checks: ┌───────────────────────────────────────┐ │ │ │ □ ASID matches current guest? (7==7) │ │ │ │ □ GPA matches translation? (0x50000) │ │ │ │ □ Validated bit set? (yes) │ │ │ │ □ VMPL permissions allow access? │ │ │ └───────────┬───────────────────────────┘ │ │ ▼ │ │ 6a. All checks pass ┌───────────────────────┐ │ │ → access proceeds │ Create TLB entry │ │ │ │ Complete access │ │ │ └───────────────────────┘ │ │ │ │ 6b. Any check fails ┌───────────────────────┐ │ │ → exception │ #NPF or #VC │ │ │ │ (guest sees error) │ │ │ └───────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ``` The check that matters most for integrity is the GPA comparison. The RMP entry records what GPA this physical page should appear at. If the translation produces a different GPA (because the hypervisor modified the nested page table), the check fails. ### When RMP Checks Occur | Access type | RMP check? | Why | | --------------------------- | ---------- | ---------------------------------------- | | Guest private read (C=1) | Yes | Must verify ownership and validation | | Guest private write (C=1) | Yes | Must verify ownership and validation | | Guest shared access (C=0) | No | Shared pages don't need RMP protection | | Hypervisor read guest page | No | Encryption protects confidentiality | | Hypervisor write guest page | Yes | Prevents corruption attacks | | Page table A/D bit updates | Yes | Any write to guest pages needs RMP check | Shared pages (C=0) bypass RMP checks because they're explicitly meant to be accessible by both guest and hypervisor. The guest chooses to make certain pages shared for I/O and communication. ## Page Validation The RMP tells hardware what the mapping should be. But how does it get populated correctly in the first place? This is where validation comes in. ### The Two-Step Assignment Adding a page to a guest requires cooperation between hypervisor and guest: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Page Assignment: Two-Step Process │ │ │ │ STEP 1: Hypervisor executes RMPUPDATE │ │ ───────────────────────────────────────── │ │ │ │ ┌─────────────────┐ ┌──────────────────────────────────────────┐ │ │ │ Hypervisor │ │ RMP Entry (before) RMP Entry (after) │ │ │ │ │ │ ┌───────────────┐ ┌───────────────┐ │ │ │ │ RMPUPDATE( │───────►│ │ Assigned=0 │ → │ Assigned=1 │ │ │ │ │ SPA=X, │ │ │ ASID=0 │ │ ASID=7 │ │ │ │ │ ASID=7, │ │ │ GPA=n/a │ │ GPA=0x50000 │ │ │ │ │ GPA=0x50000) │ │ │ Validated=0 │ │ Validated=0 │ │ │ │ │ │ │ └───────────────┘ └───────────────┘ │ │ │ └─────────────────┘ │ │ │ │ │ Page state: Hypervisor → Guest-Invalid │ │ │ └──────────────────────────────────────────┘ │ │ │ │ STEP 2: Guest executes PVALIDATE │ │ ───────────────────────────────────── │ │ │ │ ┌─────────────────┐ ┌──────────────────────────────────────────┐ │ │ │ Guest │ │ RMP Entry (before) RMP Entry (after) │ │ │ │ │ │ ┌───────────────┐ ┌───────────────┐ │ │ │ │ PVALIDATE( │───────►│ │ Assigned=1 │ → │ Assigned=1 │ │ │ │ │ GPA=0x50000) │ │ │ ASID=7 │ │ ASID=7 │ │ │ │ │ │ │ │ GPA=0x50000 │ │ GPA=0x50000 │ │ │ │ │ Hardware: │ │ │ Validated=0 │ │ Validated=1 │ │ │ │ │ 1. GPA→SPA │ │ └───────────────┘ └───────────────┘ │ │ │ │ 2. Check ASID │ │ │ │ │ │ 3. Check GPA │ │ Page state: Guest-Invalid → Guest-Valid │ │ │ │ 4. Set Valid=1 │ └──────────────────────────────────────────┘ │ │ └─────────────────┘ │ │ │ │ Key insight: RMPUPDATE always clears Validated. Only PVALIDATE sets it. │ └──────────────────────────────────────────────────────────────────────────┘ ``` The hypervisor can assign pages to guests, but it cannot validate them. The guest must explicitly accept each page. This separation is critical for security. ### Why RMPUPDATE Always Clears Validated This is the crux of remapping protection. When the hypervisor executes RMPUPDATE, the Validated bit is always cleared, regardless of its previous value. This is enforced in hardware. Why? If RMPUPDATE could preserve the Validated bit, the hypervisor could: 1. Wait for the guest to validate a page 2. Use RMPUPDATE to reassign that page to a different GPA (keeping Validated=1) 3. The guest would access the wrong physical page without knowing By forcing Validated=0 on every RMPUPDATE, any change to page assignment requires the guest to re-validate. And the guest should never re-validate a page it already validated. ### The Validation Discipline For SEV-SNP's integrity guarantee to hold, the guest must follow one strict rule: **never validate the same GPA twice.** If the guest validates GPA A, and later validates GPA A again, the security model breaks. Here's why: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ What Happens If Guest Re-validates │ │ │ │ Initial state: │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ GPA 0x50000 → SPA X │ │ │ │ RMP[X]: ASID=7, GPA=0x50000, Validated=1 │ │ │ │ Guest writes sensitive data to GPA 0x50000 │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ Attack setup (hypervisor): │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ 1. RMPUPDATE(SPA=Y, ASID=7, GPA=0x50000) │ │ │ │ RMP[Y]: ASID=7, GPA=0x50000, Validated=0 ← new page, invalid │ │ │ │ 2. Modify NPT: GPA 0x50000 → SPA Y (instead of X) │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ Guest accesses GPA 0x50000: │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Translation: GPA 0x50000 → SPA Y │ │ │ │ RMP check: RMP[Y].Validated = 0 │ │ │ │ Result: #VC exception (validation required) │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ CORRECT behavior: Guest recognizes attack, terminates │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Guest: "I already validated 0x50000. Getting #VC means someone │ │ │ │ changed the backing page. This is a security violation." │ │ │ │ Action: Terminate or enter safe state │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ BROKEN behavior: Guest foolishly re-validates │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Guest: PVALIDATE(GPA=0x50000) ← WRONG! │ │ │ │ RMP[Y]: Validated=1 │ │ │ │ Now hypervisor can switch between X and Y at will! │ │ │ │ Guest sees either old data (X) or new/garbage data (Y) │ │ │ │ depending on hypervisor's whim │ │ │ └────────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ``` ## Attack Detection Walkthrough Let's trace through exactly how the RMP catches a remapping attack: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Remapping Attack Detection │ │ │ │ Timeline │ │ ──────── │ │ │ │ T₁: Initial setup (legitimate) │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ • NPT: GPA A → SPA X │ │ │ │ • RMP[X]: ASID=guest, GPA=A, Validated=1 │ │ │ │ • Guest writes secret to GPA A │ │ │ │ • Ciphertext stored at SPA X │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ T₂: Hypervisor prepares attack │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ • RMPUPDATE(SPA=Y, ASID=guest, GPA=A) │ │ │ │ → RMP[Y]: ASID=guest, GPA=A, Validated=0 │ │ │ │ • Hypervisor modifies NPT: GPA A → SPA Y │ │ │ │ • (SPA X still has RMP[X].Validated=1, but NPT no longer points │ │ │ │ to it for GPA A) │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ T₃: Guest accesses GPA A │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ 1. Guest: load from GPA A │ │ │ │ 2. NPT translation: GPA A → SPA Y (hypervisor's malicious NPT) │ │ │ │ 3. RMP lookup: RMP[Y] │ │ │ │ 4. RMP check: │ │ │ │ • ASID match? ✓ (guest matches) │ │ │ │ • GPA match? ✓ (RMP[Y].GPA = A, translation gave A) │ │ │ │ • Validated? ✗ (RMP[Y].Validated = 0) │ │ │ │ 5. RMP check fails → #VC exception │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ T₄: Guest handles exception │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ • Guest sees #VC for GPA A │ │ │ │ • Guest checks: "Did I validate GPA A before?" → Yes │ │ │ │ • Conclusion: Hypervisor changed the backing page │ │ │ │ • Action: Terminate (attack detected) │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ The key: RMPUPDATE always clears Validated. The hypervisor cannot │ │ assign a new physical page to the same GPA without clearing Validated, │ │ which the guest detects. │ └──────────────────────────────────────────────────────────────────────────┘ ``` ## The Bijective Mapping Guarantee When the RMP and validation discipline work together, they establish a one-to-one correspondence between guest physical addresses and system physical addresses: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Bijective Mapping Property │ │ │ │ Guest Physical Addresses System Physical Addresses │ │ (GPAs) (SPAs) │ │ │ │ ┌───┐ ┌───┐ │ │ │ A │ ───────────────────────────────────│ X │ │ │ └───┘ └───┘ │ │ │ │ ┌───┐ ┌───┐ │ │ │ B │ ───────────────────────────────────│ Y │ │ │ └───┘ └───┘ │ │ │ │ ┌───┐ ┌───┐ │ │ │ C │ ───────────────────────────────────│ Z │ │ │ └───┘ └───┘ │ │ │ │ Guaranteed by: │ │ │ │ SPA → at most one GPA Enforced by RMP's GPA field. │ │ (no aliasing) If SPA X has RMP[X].GPA=A, mapping │ │ GPA B to SPA X fails the RMP check │ │ because B ≠ A. │ │ │ │ GPA → at most one SPA Enforced by validation discipline. │ │ (no remapping) Guest validates each GPA once. If │ │ hypervisor changes SPA backing a GPA, │ │ the new SPA is unvalidated and access │ │ fails. │ │ │ │ Together: GPA ↔ SPA is bijective (one-to-one and onto) │ └──────────────────────────────────────────────────────────────────────────┘ ``` This is the fundamental integrity guarantee. The guest's view of memory is consistent: reading an address returns what was last written there, never stale data, never another page's data. ## IOMMU Integration The RMP isn't only checked by CPUs. IOMMUs also perform RMP checks when devices attempt DMA. **Without IOMMU RMP checks:** 1. Hypervisor programs device to DMA to SPA X 2. SPA X is guest private memory 3. Device reads/writes guest memory! This is a "confused deputy" attack that bypasses CPU-side protections. **With IOMMU RMP checks:** 1. Device attempts DMA to SPA X 2. IOMMU translates device address to SPA X 3. IOMMU checks RMP[X] 4. If page is guest-private, DMA is blocked The hypervisor cannot use devices to access guest memory. **Device DMA is allowed to:** - Hypervisor pages (Assigned=0) - Shared guest pages (C=0) **Device DMA is blocked for:** - Guest private pages (Assigned=1, belongs to a guest) This is why guests use bounce buffers for I/O: data moves through shared (C=0) pages that devices can access, then gets copied to/from private (C=1) pages. ## TLB Caching For performance, RMP check results are cached in the TLB alongside translation results. This means: - Successful translations (including RMP checks) don't repeat the RMP lookup on every access - TLB entries include RMP state (ownership, permissions) - When RMP entries change, TLB entries must be invalidated Hardware automatically triggers TLB invalidation across all cores and IOMMUs when: - RMPUPDATE modifies an entry - PVALIDATE validates a page - RMPADJUST changes VMPL permissions - AMD-SP firmware commands change page states Stale TLB entries would be a security hole; the automatic invalidation prevents this. ## The Instructions Three instructions modify RMP entries: **RMPUPDATE** (hypervisor instruction). Assigns pages to guests or reclaims them. Sets ASID, GPA, and VMPL permissions. Always clears the Validated bit. Cannot modify immutable pages. **PVALIDATE** (guest instruction, VMPL0 only). Validates or invalidates pages owned by the current guest. The hardware verifies that the RMP entry's ASID matches the current guest and that the RMP entry's GPA matches the address being validated. **RMPADJUST** (guest instruction, VMPL0 only). Modifies VMPL permissions for pages owned by the current guest. Does not affect the Validated bit. Only VMPL0 can execute PVALIDATE and RMPADJUST. If the guest runs an SVSM at VMPL0. the guest OS (at VMPL2/3) must call the SVSM to proxy these operations. This allows the SVSM to enforce additional security policies, like zeroing pages before handing them to the guest OS. ## Launch-Time Validation During guest launch, pages are validated through a different path. The hypervisor provides each page to the AMD-SP via SNP_LAUNCH_UPDATE: 1. Hypervisor calls SNP_LAUNCH_UPDATE(plaintext_page, GPA) 2. AMD-SP: - Encrypts page with guest's VEK - Updates launch measurement (hash) - Sets RMP entry: - Assigned = 1 - ASID = guest - GPA = specified address - Validated = 1 (set directly by AMD-SP) 3. Page goes directly to Guest-Valid state **Why this is safe:** - AMD-SP is trusted (part of the hardware root of trust) - The page is measured before being encrypted - Attestation reports include the launch measurement - Verifiers confirm the guest started with expected code Launch pages don't need explicit PVALIDATE because the AMD-SP is a trusted party that can set the Validated bit directly. The measurement ensures the verifier knows exactly what was loaded. ## What RMP Protection Doesn't Cover The RMP provides strong integrity guarantees, but some attacks remain possible: **Denial of service.** The hypervisor can refuse to run the guest, refuse to provide memory, or provide memory that's intentionally slow to access. Availability is explicitly out of scope. **Side channels.** The hypervisor can observe which pages the guest accesses by watching page table access/dirty bits or nested page faults. This leaks information about the guest's memory access patterns, though not the data itself. **Shared page manipulation.** Pages marked as shared (C=0) are accessible to both guest and hypervisor. The guest must not put sensitive data in shared pages and must validate any data received through shared pages. **Guest bugs.** If the guest has a vulnerability (buffer overflow, etc.), attackers can exploit it through legitimate communication channels. The RMP protects the environment, not the code running in it. --- # Privilege & Communication Confidential's privacy-preserving infrastructure services—firewalls, DDoS protection, routing, load balancing—run in their own TEE-protected layer. This document explains the mechanisms that make this possible: privilege separation within TEEs and secure communication with untrusted components. After reading this, you'll understand VMPLs (the privilege hierarchy inside the guest), how CPU state is protected, and the GHCB protocol (safe guest-hypervisor communication). This assumes familiarity with basic virtualization concepts and the RMP from [Document 3](/docs/concepts/confidential-computing-primer/03-memory-integrity). ## The Problem: Cooperation Without Trust Even with memory encryption and integrity protection, an SNP guest still needs the hypervisor for certain things. The guest can't directly access hardware devices. It can't allocate physical memory. It needs information about the CPU it's running on. In traditional virtualization, the hypervisor handles all of this transparently. When the guest executes certain instructions, control transfers to the hypervisor, which can see the guest's complete state (all register values, what instruction was executing, everything) and handle the request. With SNP, this model breaks. The guest's state is encrypted. The hypervisor can't read register values, which means it can't see what the guest was trying to do or help it. But the guest still needs help. SEV-SNP solves this with two key ideas: 1. **The guest controls what information the hypervisor sees.** Instead of the hypervisor automatically seeing everything, the guest explicitly shares only what's needed for each specific request. 2. **The guest validates everything the hypervisor returns.** A malicious hypervisor might lie. The guest checks responses against known-good values before trusting them. This document covers how these ideas are implemented, along with a third mechanism: privilege separation within the guest itself. ## What Operations Need the Hypervisor? Before diving into mechanisms, let's understand what operations require hypervisor involvement. These will come up throughout the document. **CPUID** is an instruction that returns information about the CPU: what features it supports, cache sizes, vendor string, etc. Operating systems use CPUID at boot to know what they're running on. In virtualized environments, the hypervisor typically intercepts CPUID and returns values appropriate for the virtual machine (possibly hiding features the VM shouldn't use). **MSRs (Model-Specific Registers)** are CPU configuration registers. Reading or writing certain MSRs requires hypervisor involvement because they affect system-wide state or need emulation. **I/O operations** (reading/writing to devices) go through the hypervisor because the hypervisor mediates access to physical hardware. When your guest writes to a virtual disk, that eventually becomes a request to the hypervisor. **Memory allocation** happens when the guest needs more RAM. The hypervisor decides which physical pages to give the guest. In traditional virtualization, the hypervisor handles all of these by examining guest state directly. In SNP, the guest must explicitly participate in each interaction. ## VM Permission Levels (VMPLs) Before we get to guest-hypervisor communication, we need to understand VMPLs. They solve a different problem: what if you want privilege separation *inside* the encrypted guest? ### The Problem VMPLs Solve Consider what happens without VMPLs. Everything inside the SNP guest runs at the same privilege level from the hardware's perspective. The guest kernel can access all guest memory. Any code in the guest can execute any guest instruction. This is fine if you trust everything in the guest equally. But what if you want: - A security monitor that even a compromised guest kernel can't tamper with? - A virtual TPM whose keys are inaccessible to the guest OS? - To run an unmodified legacy OS that doesn't know about SNP, with a small trusted shim handling SNP-specific operations? Without some form of isolation, a bug or exploit in the guest kernel could compromise any security service running alongside it. VMPLs provide that isolation. ### The Four Levels VMPLs are a hardware-enforced privilege hierarchy with four levels: ``` ┌──────────────────────────────────────────────────────────────────────┐ │ VMPL Privilege Hierarchy │ │ │ │ VMPL0 ──────── Highest privilege │ │ │ │ │ │ This level can: │ │ │ • Access all guest memory │ │ │ • Validate pages (PVALIDATE instruction) │ │ │ • Control permissions for lower levels │ │ │ • Read/modify lower levels' saved CPU state │ │ │ │ │ VMPL1 ──────── Intermediate (rarely used in practice) │ │ │ │ │ VMPL2 ──────── Typically where the guest OS kernel runs │ │ │ │ │ VMPL3 ──────── Lowest privilege (guest userspace) │ │ │ │ The numbering is counterintuitive: lower number = higher privilege │ │ │ └──────────────────────────────────────────────────────────────────────┘ ``` Think of VMPLs as an additional privilege axis, separate from the familiar ring 0/ring 3 distinction. A process could be running at VMPL3 (lowest VMPL privilege) while in ring 0 (kernel mode). The two are independent. Every virtual CPU runs at exactly one VMPL at any time. Which VMPL it's running at determines what memory it can access and what instructions it can execute. ### How VMPL Permissions Work VMPLs are enforced through the RMP (the same structure that tracks page ownership, covered in Document 3). Each RMP entry has four permission fields, one per VMPL: ``` ┌──────────────────────────────────────────────────────────────────────┐ │ Per-Page VMPL Permissions │ │ │ │ For each 4KB page, the RMP stores: │ │ │ │ VMPL0 permissions: [Read] [Write] [Execute] │ │ VMPL1 permissions: [Read] [Write] [Execute] │ │ VMPL2 permissions: [Read] [Write] [Execute] │ │ VMPL3 permissions: [Read] [Write] [Execute] │ │ │ │ When code at VMPL2 accesses a page, hardware checks │ │ the VMPL2 permission bits. If the access isn't allowed, │ │ the CPU raises an exception. │ │ │ └──────────────────────────────────────────────────────────────────────┘ ``` When a page is first validated, VMPL0 gets full access and all other VMPLs get none. VMPL0 must explicitly grant permissions to lower levels. The RMPADJUST instruction modifies these permissions. It has restrictions: - You can only modify permissions for less-privileged VMPLs (VMPL0 can grant permissions to VMPL1/2/3; VMPL2 can only grant to VMPL3) - You can't grant more permission than you have (if you only have read access, you can only grant read access) - You can't increase your own permissions This creates a top-down delegation model: VMPL0 decides what each lower level can access. ### The SVSM: What Runs at VMPL0 With VMPLs, the natural question is: what runs at VMPL0? In most configurations, it's the **SVSM (Secure VM Service Module)**. The SVSM is a small piece of software that runs at the highest privilege level inside the guest. It's measured during launch (so it appears in attestation reports) and provides services to the guest OS running at a lower VMPL. Think of it as a trusted shim between the guest OS and the SNP hardware. Why is this necessary? Certain operations can only be executed at VMPL0: - **PVALIDATE**: The instruction that validates pages (see Document 3). If the guest OS runs at VMPL2, it can't validate pages directly. - **RMPADJUST**: Modifying VMPL permissions. Lower VMPLs can't escalate their own access. - **Creating new vCPU contexts**: Adding processors to the guest requires VMPL0 access. If you want the guest OS at a lower VMPL (for isolation), something at VMPL0 must proxy these operations. That's the SVSM. ``` ┌──────────────────────────────────────────────────────────────────────┐ │ SNP Guest with SVSM │ │ │ │ ┌────────────────────────────────────────────────────────────────┐ │ │ │ VMPL0: SVSM │ │ │ │ │ │ │ │ • Small, measured code │ │ │ │ • Validates pages on behalf of guest OS │ │ │ │ • Can provide additional services (vTPM, attestation) │ │ │ │ • Its memory is inaccessible to lower VMPLs │ │ │ └────────────────────────────────────────────────────────────────┘ │ │ │ │ │ SVSM call interface │ │ │ │ │ ┌────────────────────────────▼───────────────────────────────────┐ │ │ │ VMPL2: Guest OS Kernel │ │ │ │ │ │ │ │ • Linux, Windows, or other OS │ │ │ │ • Cannot read SVSM memory (hardware enforced) │ │ │ │ • Requests page validation through SVSM │ │ │ └────────────────────────────────────────────────────────────────┘ │ │ │ │ │ Normal syscall interface │ │ │ │ │ ┌────────────────────────────▼───────────────────────────────────┐ │ │ │ VMPL3: Guest Applications │ │ │ │ │ │ │ │ • Your workload │ │ │ └────────────────────────────────────────────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────┘ ``` The SVSM's memory region has permissions that grant access only to VMPL0. The guest kernel literally cannot read, write, or execute SVSM code, even if it's compromised. This is hardware-enforced isolation. Not all SNP deployments use SVSM. In the "enlightened guest" model, the guest OS itself runs at VMPL0 and handles SNP operations directly. This is simpler but provides less isolation. ## Protecting CPU State: The VMSA Now we can talk about how CPU state is protected. When a virtual machine pauses (to let the hypervisor run, handle an exception, etc.), the CPU's current state, registers, instruction pointer, flags, needs to be saved somewhere. This saved state is called the **VMSA (VM Save Area)**. ### Why the VMSA Matters The VMSA contains everything about the CPU's execution context: - **Instruction pointer**: Where execution will resume. If an attacker could modify this, they could redirect the guest to execute arbitrary code. - **Stack pointer**: Where the current stack is. Stack pivots are a common exploit technique. - **General registers**: These often hold sensitive data, function arguments, intermediate computations, cryptographic keys during operations. - **Control registers**: CR3 holds the page table base. Modifying it could remap memory. In traditional virtualization, the hypervisor can read and write all of this. SEV-SNP changes that. ### How the VMSA Is Protected The VMSA is a special page with three protections: 1. **Encryption**: The contents are encrypted with the guest's key. The hypervisor sees only ciphertext. 2. **Integrity**: The RMP marks VMSA pages specially (VMSA=1). The hypervisor cannot write to them. 3. **Controlled access**: Only hardware operations (saving/restoring state on VM transitions) can modify the VMSA. Software, even the guest itself at lower VMPLs, cannot directly write it. ``` ┌──────────────────────────────────────────────────────────────────────┐ │ VMSA Protection │ │ │ │ When guest is running: │ │ ┌────────────────────────────────────────────────────────────┐ │ │ │ CPU registers contain live values (plaintext in CPU) │ │ │ └────────────────────────────────────────────────────────────┘ │ │ │ │ When guest pauses (VM exit): │ │ ┌────────────────────────────────────────────────────────────┐ │ │ │ Hardware saves registers to VMSA │ │ │ │ Hardware encrypts the VMSA with guest's key │ │ │ └────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────┐ │ │ │ VMSA page in DRAM (encrypted) │ │ │ │ • Hypervisor sees ciphertext │ │ │ │ • RMP prevents hypervisor writes │ │ │ │ • Contains: RIP, RSP, RAX-R15, CR3, flags, etc. │ │ │ └────────────────────────────────────────────────────────────┘ │ │ │ │ When guest resumes: │ │ ┌────────────────────────────────────────────────────────────┐ │ │ │ Hardware decrypts VMSA │ │ │ │ Hardware loads values back into CPU registers │ │ │ └────────────────────────────────────────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────┘ ``` The hypervisor still has a control area it can access (to configure intercepts, read exit codes, etc.), but this is separate from the VMSA. The hypervisor knows *that* an exit happened and *why* (the exit code), but not the detailed register values. ### Multiple VMSAs An SNP guest has at least one VMSA per vCPU. With VMPL separation, there may be more: separate VMSAs for VMPL0 (SVSM) and VMPL2 (guest OS) on the same vCPU. This allows switching between privilege levels while keeping state separate. ## Guest-Hypervisor Communication Now we can address the main question: how does an encrypted guest communicate with the hypervisor it doesn't trust? ### The #VC Exception When the guest needs hypervisor involvement (CPUID, MSR access, I/O), a **#VC exception** is raised. The #VC (VMM Communication) exception is new with SEV-ES/SNP, exception vector 29. Instead of the hypervisor directly seeing what happened, the #VC exception runs a handler inside the guest. This handler decides what to share with the hypervisor. ``` ┌──────────────────────────────────────────────────────────────────────┐ │ #VC Exception Flow │ │ │ │ 1. Guest executes CPUID instruction │ │ │ │ │ ▼ │ │ 2. Hardware raises #VC exception (not #VMEXIT to hypervisor) │ │ │ │ │ ▼ │ │ 3. Guest's #VC handler runs │ │ • Examines what operation was requested │ │ • Decides what info to share with hypervisor │ │ • Prepares a message in the GHCB (shared page) │ │ │ │ │ ▼ │ │ 4. Guest executes VMGEXIT (voluntary exit to hypervisor) │ │ │ │ │ ▼ │ │ 5. Hypervisor reads request from GHCB, provides response │ │ │ │ │ ▼ │ │ 6. Guest resumes, #VC handler validates response │ │ │ │ │ ▼ │ │ 7. If valid, handler provides result to original code │ │ │ └──────────────────────────────────────────────────────────────────────┘ ``` The key insight: the guest controls the information flow. Instead of the hypervisor automatically seeing all state, the guest explicitly shares only what's needed. ### The GHCB: A Shared Mailbox The **GHCB (Guest-Hypervisor Communication Block)** is a shared memory page (marked C=0, unencrypted) that both guest and hypervisor can read and write. It's like a mailbox for passing messages. The guest puts its request in the GHCB, then exits to the hypervisor. The hypervisor reads the request, writes a response, and returns. The guest reads the response. ``` ┌──────────────────────────────────────────────────────────────────────┐ │ GHCB Layout │ │ │ │ The GHCB is a 4KB page with defined fields: │ │ │ │ ┌────────────────────────────────────────────────────────────────┐ │ │ │ Exit code: What operation the guest wants (CPUID, MSR, etc.) │ │ │ │ Exit info: Additional parameters for the operation │ │ │ │ Register fields: Space for register values (RAX, RBX, etc.) │ │ │ │ Valid bitmap: Which fields the guest actually filled in │ │ │ └────────────────────────────────────────────────────────────────┘ │ │ │ │ The valid bitmap is important: the guest marks only the fields │ │ it's actually using. This minimizes information exposure. │ │ │ └──────────────────────────────────────────────────────────────────────┘ ``` ### Example: CPUID Emulation Let's trace through a concrete example. The guest wants to execute CPUID to learn about CPU features. ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ CPUID Request Flow │ │ │ │ Guest Hypervisor │ │ │ │ │ │ 1. CPUID EAX=1 │ │ │ │ │ │ │ 2. #VC exception │ │ │ │ │ │ │ 3. #VC handler prepares GHCB: │ │ │ • EAX=1, exit code=CPUID │ │ │ • Mark valid fields │ │ │ │ │ │ │ │────────────── 4. VMGEXIT ───────────────────►│ │ │ │ │ │ │ │ 5. Read CPUID leaf │ │ │ │ │ │ │ 6. Look up values │ │ │ │ │ │ │ 7. Write EAX/EBX/ECX/EDX │ │ │ │ │ │ │◄───────────── 8. VM Entry ────────────────── │ │ │ │ │ │ │ 9. #VC handler validates: │ │ │ • Compare against CPUID page │ │ │ • Mismatch = attack │ │ │ • OK = return to caller │ │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` Step 8 is critical. The guest doesn't blindly trust the hypervisor's CPUID response. SNP guests have a "CPUID page" that's measured during launch and contains known-good values. The guest compares the hypervisor's response against this trusted data. ### Validating Hypervisor Responses For different operations, validation looks different: **CPUID**: Compare against the measured CPUID page. Mismatches indicate potential attack. **MSR reads**: Check that the value is within expected ranges for that MSR. Some MSRs have architecturally defined constraints. **I/O operations**: Often can't be validated directly. Device responses are whatever the device (which the hypervisor controls) says. For sensitive data, use application-level encryption (TLS, etc.). **Memory allocation**: The RMP and page validation mechanism handle this (see Document 3). ### What Can Go Wrong Even with this protocol, a malicious hypervisor has options: - **Return wrong data**: Lie about CPUID, MSR values. Mitigated by validation. - **Refuse to respond**: The hypervisor controls scheduling. It can hang the guest indefinitely. This is denial of service, explicitly out of scope for SNP. - **Return stale data**: Replay old responses. Mitigated by sequence numbers where applicable. - **Modify GHCB during processing**: Race condition. Mitigated by copying values to private memory before use. The guest must be paranoid about everything in the GHCB. ## Interrupt Handling There's one more communication channel to protect: interrupts. Traditionally, the hypervisor can inject interrupts into guests at will. A malicious hypervisor could: - Inject interrupts while they're supposed to be disabled - Inject spurious exceptions to confuse the guest - Manipulate interrupt priority to alter behavior SNP provides two optional modes to address this: ### Restricted Injection In restricted injection mode, the hypervisor can only inject one specific exception: **#HV** (hypervisor exception). This acts as a simple doorbell: "hey, something happened." ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ Restricted Injection │ │ │ │ Hypervisor Guest │ │ │ │ │ │ [Has event for guest] │ │ │ │ │ │ │ │──────────── Inject #HV ─────────────────────►│ │ │ │ (only allowed) │ │ │ │ │ │ │ │ #HV handler runs: │ │ │ • Check event queue │ │ │ • Process events │ │ │ • Return when done │ │ │ │ │ │ │ │ The guest controls when and how to process events. │ │ Hypervisor can only signal "check the queue." │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ### Alternate Injection Alternate injection mode keeps the standard interrupt interface but moves the control fields into the encrypted VMSA. Only code with VMSA access (VMPL0) can actually inject interrupts into lower VMPLs. The hypervisor can't directly inject. With both modes combined in a VMPL setup: - Hypervisor signals VMPL0 via #HV - VMPL0 (SVSM) decides what to inject into VMPL2 (guest OS) - Guest OS sees normal interrupts, doesn't need to know about the indirection ## The SVSM Calling Convention When the guest OS needs VMPL0 services (page validation, vTPM operations, etc.), it uses a defined calling convention: ``` ┌────────────────────────────────────────────────────────────────────────────────────┐ │ SVSM Call Flow │ │ │ │ Guest OS (VMPL2) SVSM (VMPL0) Hypervisor │ │ │ │ │ │ │ 1. Write request to │ │ │ │ Calling Area │ │ │ │ │ │ │ │ │ 2. Set SVSM_CALL_PENDING=1 │ │ │ │ │ │ │ │ │ │─────────────────── 3. VMGEXIT ────────────────────────────►│ │ │ │ │ │ │ │ │ │◄──── 4. Schedule VMPL0 ──────│ │ │ │ │ │ │ │ │ 5. Verify entry: │ │ │ │ • Real VMGEXIT? │ │ │ │ • PENDING set? │ │ │ │ │ │ │ │ │ 6. Execute operation │ │ │ │ (e.g., PVALIDATE) │ │ │ │ │ │ │ │ │ 7. Write result │ │ │ │ │ │ │ │ │ 8. Clear PENDING │ │ │ │ │ │ │ │ │◄───────────── 9. Return to guest ─────────────────────────►│ │ │ │ │ │ │ │ 10. Check SVSM_CALL_PENDING: │ │ │ │ • If 1: SVSM never ran, retry │ │ │ │ • If 0: call done, check result │ │ │ │ │ │ │ │ │ │ └────────────────────────────────────────────────────────────────────────────────────┘ ``` Step 10 is important. The guest OS can't trust that the hypervisor actually ran the SVSM. By checking the pending flag atomically, it detects if the hypervisor returned without invoking SVSM. ### SVSM Security The SVSM zeros VMPCK0 (the VMPL0 communication key) from the secrets page after startup. This prevents the guest OS from impersonating VMPL0 when talking to the PSP. The guest OS uses VMPCK2 or VMPCK3 for its own PSP communication. This matters for attestation: only SVSM can generate VMPL0 attestation reports. ## Security Summary ### What's Protected | Mechanism | Protection | | -------------------- | ------------------------------------------------------------------------------------- | | VMPLs | Code at VMPL2 can't access VMPL0 memory; hardware-enforced isolation within the guest | | VMSA encryption | Hypervisor can't read register values (stack pointer, instruction pointer, etc.) | | VMSA integrity | Hypervisor can't modify registers to redirect execution | | GHCB protocol | Guest controls what information hypervisor sees; validates all responses | | Interrupt protection | Hypervisor can't directly inject arbitrary interrupts | | SVSM | Trusted code at highest privilege level; proxies VMPL0-only operations | ### What's Not Protected **Timing information**: The hypervisor knows when exits occur and how long operations take. **Anything in shared pages**: The GHCB contents are visible to the hypervisor. Don't put secrets there. **I/O data paths**: Device communication goes through the hypervisor. Use TLS or application-level encryption for sensitive data. **Availability**: The hypervisor can refuse to schedule the guest, run SVSM, or provide memory. Denial of service is explicitly out of scope. ### Guest Responsibilities The guest must: 1. Validate hypervisor responses (check CPUID against measured values, verify MSR ranges) 2. Minimize GHCB exposure (only mark fields valid that are needed) 3. Verify SVSM call completion (check the pending flag) 4. Handle failures appropriately (terminate if validation fails) ## Further Reading - [Document 3: Memory Integrity](/docs/concepts/confidential-computing-primer/03-memory-integrity) for RMP and page validation details - [Document 5: Attestation](/docs/concepts/confidential-computing-primer/05-attestation) for how SVSM and VMPLs appear in attestation reports --- # Attestation & Verification Attestation is central to what Confidential enables. Every request-response flow can include cryptographic proof of what code is running, on what hardware, with what configuration. This document explains how that works under the hood. After reading this, you'll understand how measurements are computed at launch, what attestation reports contain, how the certificate chain works, and how to verify a report. This assumes familiarity with the concepts from [Documents 1-4](/docs/concepts/confidential-computing-primer/01-threat-model), particularly the RMP, VMPLs, and the role of the PSP (Platform Security Processor). ## The Problem Attestation Solves Encryption and integrity protection are only useful if you can prove what's actually running. Consider the situation from a customer's perspective: they want to run sensitive workloads on your infrastructure, but they don't trust you. You tell them "we're running your code in an encrypted VM, we can't see anything." They have no reason to believe you. Maybe you're running their code unencrypted and just claiming otherwise. Maybe you modified their code before loading it. Maybe you're running a completely different program that just pretends to be theirs. Without cryptographic proof, promises are worthless. Attestation provides that proof. The hardware itself measures what code was loaded, and AMD's keys (fused into the silicon at manufacturing) sign a statement about that measurement. A customer can verify: "This signature is valid and chains to AMD's root key. The measurement matches what I expected. Therefore, AMD's hardware is asserting that my code is running." This is the problem Confidential solves. We make attestation accessible such that you don't need to implement the verification flow yourself, but understanding it helps you reason about what guarantees you're actually getting. ## What Gets Measured Before diving into attestation reports, we need to understand what goes into the measurement. The measurement (called the **launch digest**) captures the initial state of the VM at launch time. ### The Launch Sequence When a hypervisor launches an SNP guest, three commands are involved: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ SNP Launch Commands │ │ │ │ SNP_LAUNCH_START │ │ ──────────────── │ │ • Creates guest context in PSP memory │ │ • Generates the VM Encryption Key (VEK) │ │ • Records guest policy (what the VM is allowed to do) │ │ • Initializes the launch digest to a starting value │ │ │ │ │ │ │ ▼ │ │ │ │ SNP_LAUNCH_UPDATE (called repeatedly, once per page) │ │ ───────────────── │ │ • For each page of initial VM memory: │ │ - Compute hash of page contents │ │ - Combine with current launch digest, GPA, page type, permissions │ │ - New digest = SHA-384(current_digest || page_info) │ │ - Encrypt page with VEK │ │ - Mark page in RMP as belonging to this guest │ │ │ │ │ │ │ ▼ │ │ │ │ SNP_LAUNCH_FINISH │ │ ───────────────── │ │ • Finalizes the launch digest (no more updates possible) │ │ • Optionally validates Identity Block (more on this later) │ │ • Records HOST_DATA (hypervisor-provided, appears in reports) │ │ • Transitions guest to running state │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` The critical insight is that `SNP_LAUNCH_UPDATE` is called for every page in the initial VM image. Each call extends the digest, creating a chain. The final digest depends on: - The content of every measured page - The guest physical address (GPA) where each page is placed - The type of each page (code, data, VMSA, secrets page, etc.) - The VMPL permissions assigned to each page - The order in which pages were installed ### What's In the Measurement In a typical SNP launch with direct boot (passing kernel and initrd to QEMU): **Measured components:** - OVMF firmware (the UEFI implementation) - Linux kernel - Initial ramdisk (initramfs/initrd) - Kernel command line - Initial CPU state (the VMSA, which sets where execution starts) **Not measured:** - Anything loaded after boot (filesystems mounted from disk, downloaded code) - Runtime state changes - Data written to memory during execution This is important: the measurement only proves initial state. If your application code isn't in the initramfs, it's not in the measurement. Document 6 covers strategies for extending trust to code loaded after boot. ### Page Types and Measurement Different page types are measured differently: | Page Type | What Gets Measured | Notes | | ---------- | -------------------------------- | ------------------------------------------------ | | NORMAL | SHA-384 of page contents | Standard code/data | | VMSA | SHA-384 of initial CPU state | Where execution starts, initial register values | | ZERO | Measured as all zeros | Content is encrypted zeros, but measures as zero | | UNMEASURED | GPA recorded, content not hashed | For pages the guest will initialize itself | | SECRETS | GPA recorded, content not hashed | PSP fills in keys after measurement | | CPUID | GPA recorded, content not hashed | Hypervisor-provided CPU feature information | The SECRETS page is particularly interesting. The measurement commits to where the secrets page will be (the GPA), but not what's in it. After measurement, the PSP fills it with the VMPCKs (the keys for guest-PSP communication). This way, the measurement is reproducible (doesn't depend on random key values) but the guest knows exactly where to find its keys. ### The Measurement Chain Each `SNP_LAUNCH_UPDATE` constructs a `PAGE_INFO` structure and hashes it: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ How the Launch Digest Accumulates │ │ │ │ Start: digest = initial_value │ │ │ │ For each page: │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ PAGE_INFO = { │ │ │ │ DIGEST_CUR: current launch digest (48 bytes) │ │ │ │ CONTENTS: SHA-384 of page contents (or 0 for special types) │ │ │ │ GPA: guest physical address where page will be │ │ │ │ PAGE_TYPE: NORMAL, VMSA, ZERO, etc. │ │ │ │ VMPL_PERMS: permissions for each VMPL (0-3) │ │ │ │ } │ │ │ │ │ │ │ │ new_digest = SHA-384(PAGE_INFO) │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ After all pages: final launch digest = MEASUREMENT in attestation report│ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` Because each step incorporates the previous digest, the final value depends on the exact sequence of operations. Load the same code at different addresses? Different measurement. Load pages in different order? Different measurement. Change one byte in one page? Different measurement. ## Attestation Reports Once a guest is running, it can ask the PSP for an attestation report. The report is a signed statement from the hardware about the guest's identity and current state. ### Requesting a Report The guest sends a `MSG_REPORT_REQ` to the PSP through the encrypted communication channel (using the VMPCKs from the secrets page). The request includes: - **REPORT_DATA**: 64 bytes of arbitrary data the guest wants to bind to the report - **VMPL**: Which privilege level is requesting (must be >= current VMPL) The PSP constructs the report and signs it, then returns it to the guest. ### What's In the Report The attestation report is a structured data blob (currently version 5) with several categories of information: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Attestation Report Contents │ │ │ │ Identity Information │ │ ──────────────────── │ │ VERSION: Report format version (currently 5) │ │ POLICY: Guest policy flags (debug allowed?, SMT?, migration?) │ │ FAMILY_ID: 16-byte family identifier (from launch, if IDB used) │ │ IMAGE_ID: 16-byte image identifier (from launch, if IDB used) │ │ GUEST_SVN: Guest security version number │ │ VMPL: Which VMPL requested this report (or 0xFFFFFFFF if host) │ │ │ │ The Measurement │ │ ─────────────── │ │ MEASUREMENT: 48-byte SHA-384 launch digest │ │ This is what proves which code was loaded at launch │ │ │ │ Custom Data │ │ ─────────── │ │ REPORT_DATA: 64 bytes provided by the guest when requesting │ │ HOST_DATA: 32 bytes provided by hypervisor at launch │ │ │ │ Platform Information │ │ ──────────────────── │ │ CURRENT_TCB: Current firmware versions on this machine │ │ REPORTED_TCB: TCB version used to derive the signing key │ │ COMMITTED_TCB: Minimum TCB version (anti-rollback) │ │ PLATFORM_INFO: Flags about the platform (SMT enabled?, etc.) │ │ │ │ Hardware Identification │ │ ─────────────────────── │ │ CHIP_ID: 64-byte unique identifier for this physical CPU │ │ CPUID info: Family, model, stepping of the processor │ │ │ │ Signing Key Information │ │ ─────────────────────── │ │ ID_KEY_DIGEST: Hash of the ID signing key (if IDB used) │ │ AUTHOR_KEY_DIGEST: Hash of the author signing key (if provided) │ │ REPORT_ID: Random identifier for this specific guest instance │ │ │ │ The Signature │ │ ───────────── │ │ SIGNATURE: 512-byte ECDSA P-384 signature over all the above │ │ Signed by VCEK (chip-unique key) or VLEK (provider key) │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` Confidential enables these attestation reports to be attached to any request-response flow. The integration approach depends on your architecture—attestation on every request, on session establishment, or on-demand. ### The REPORT_DATA Field The REPORT_DATA field deserves special attention. The guest provides these 64 bytes, and they're included in the signed report. This enables several important patterns: **Binding a TLS session to attestation:** 1. Guest generates an ephemeral TLS keypair 2. Guest computes SHA-256 of the TLS public key (32 bytes, fits in 64) 3. Guest requests attestation report with this hash as REPORT_DATA 4. Guest sends both the report and the TLS public key to the verifier 5. Verifier checks: does REPORT_DATA equal SHA-256 of this TLS key? 6. If yes, the TLS session is provably with the attested VM **Binding a nonce for freshness:** 1. Verifier sends a random nonce to guest 2. Guest includes nonce in REPORT_DATA 3. Verifier checks the nonce matches 4. Proves the report was generated recently (not replayed) **Binding a manifest of claims:** 1. Guest creates a JSON/CBOR document with all claims 2. Guest puts SHA-384 of the manifest in REPORT_DATA 3. Guest sends both report and manifest 4. Verifier checks hash matches, then validates manifest contents The constraint is size: only 64 bytes. For complex attestations, hash a larger document and include the hash. We use REPORT_DATA to bind attestations to specific sessions and include deployment metadata. For example, we include a hash of the git commit that produced the running code, linking the cryptographic proof back to your source repository. ### TCB Version and Security Patches The report contains multiple TCB (Trusted Computing Base) version fields. The TCB version is a composite of firmware component versions: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ TCB Version Structure (Milan/Genoa) │ │ │ │ Bits 7:0 BOOT_LOADER - PSP bootloader version │ │ Bits 15:8 TEE - PSP OS version │ │ Bits 55:48 SNP - SNP firmware version │ │ Bits 63:56 MICROCODE - CPU microcode patch level │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` Why multiple TCB fields in the report? - **CURRENT_TCB**: What's actually running on the machine right now - **REPORTED_TCB**: What was used to derive the signing key (can be older) - **COMMITTED_TCB**: The minimum version, for anti-rollback protection - **LAUNCH_TCB**: What was running when this guest was created The distinction between CURRENT_TCB and REPORTED_TCB exists because the hypervisor can control which key signs the report. After a firmware update, the hypervisor might continue signing with the old TCB's key until verifiers have fetched the new certificates. This decouples "updating firmware" from "changing what key signs reports." ## The Key Hierarchy The attestation report signature needs to chain back to something the verifier trusts. This is where AMD's key hierarchy comes in. ### From Silicon to Signature ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Key Hierarchy │ │ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ AMD Root Key (ARK) │ │ │ │ • RSA-4096 key │ │ │ │ • Self-signed certificate │ │ │ │ • Lives in AMD's HSMs, never leaves │ │ │ │ • Different ARK per product family (Milan, Genoa, Turin) │ │ │ │ • This is the ultimate trust anchor │ │ │ └─────────────────────────────────┬──────────────────────────────────┘ │ │ │ signs │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ AMD SEV Key (ASK) │ │ │ │ • RSA-4096 key │ │ │ │ • Intermediate CA specifically for SEV │ │ │ │ • Also product-family specific │ │ │ └─────────────────────────────────┬──────────────────────────────────┘ │ │ │ signs │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ VCEK (Versioned Chip Endorsement Key) │ │ │ │ • ECDSA P-384 key │ │ │ │ • Unique per physical chip AND per TCB version │ │ │ │ • Derived from fused secrets that never leave the chip │ │ │ │ • This is what signs attestation reports │ │ │ └─────────────────────────────────┬──────────────────────────────────┘ │ │ │ signs │ │ ▼ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Attestation Report │ │ │ │ • Contains measurement, policy, REPORT_DATA, etc. │ │ │ │ • Signature verifiable with VCEK public key │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` ### The VCEK: Chip-Unique, Version-Specific The VCEK is the key that signs attestation reports. It has two important properties: **Unique per chip:** Each physical AMD processor has secrets fused into it at manufacturing. The VCEK is derived from these secrets. Two different chips have different VCEKs, even with identical firmware. **Unique per TCB version:** The derivation also incorporates the firmware versions. The same chip with different firmware produces a different VCEK. This means you can't take a key from an old, vulnerable firmware version and use it with new firmware (or vice versa). The derivation uses a one-way function: knowing the VCEK for one TCB version doesn't let you compute the VCEK for any other version. And only the chip's fused secrets (inaccessible to software) can compute any VCEK. ### VCEK vs VLEK There's an alternative to VCEK: the **VLEK (Versioned Loaded Endorsement Key)**. VCEK reveals which specific chip is running (via the CHIP_ID in the report and the certificate). Some cloud providers don't want to expose this. VLEK provides the same security guarantees but with a different trust model: - AMD maintains VLEK seeds for enrolled cloud providers - The provider provisions their platforms with VLEK via a secure process - Reports can be signed with VLEK instead of VCEK - VLEK certificates are issued by AMD but don't contain CHIP_ID From a verifier's perspective, both VCEK and VLEK provide the same assurance: "This report was signed by genuine AMD hardware at this TCB version." The difference is operational: VCEK reveals which specific chip, VLEK doesn't. Depending on the hardware your workload runs on, our attestations are signed by AMD (for CPU TEEs), Intel (for TDX), or NVIDIA (for GPU confidential computing). The verification flow is similar across all three—the key hierarchy and certificate chain structure are analogous. ## Verifying a Report Here's the complete flow for verifying an attestation report. ### Step 1: Receive the Report The guest sends its attestation report to the verifier, typically over TLS. The report is a binary blob (around 1KB). ### Step 2: Parse and Extract Metadata From the report, extract: - CHIP_ID (64 bytes at offset 0x1A0) - REPORTED_TCB (8 bytes at offset 0x180) - CPUID family/model/stepping (bytes at 0x188-0x18A) These tell you which certificates to fetch. ### Step 3: Fetch Certificates from AMD AMD's Key Distribution Service (KDS) at `https://kdsintf.amd.com` provides certificates. **Certificate chain (ARK + ASK):** ``` GET /vcek/v1/{product}/cert_chain ``` - `product` = "Milan", "Genoa", "Turin", etc. (from CPUID) - Returns PEM with both certificates **VCEK certificate:** ``` GET /vcek/v1/{product}/{chip_id}?blSPL=X&teeSPL=X&snpSPL=X&ucodeSPL=X ``` - `chip_id` = hex encoding of CHIP_ID from report - SPL values from REPORTED_TCB in report - Returns DER-encoded certificate **Certificate revocation list:** ``` GET /vcek/v1/{product}/crl ``` - Check this to ensure cert hasn't been revoked ### Step 4: Verify the Certificate Chain 1. **Verify ARK is self-signed** - ARK signs itself, this is the root - You must trust AMD's ARK (compare against known-good copy) 2. **Verify ASK is signed by ARK** - Check signature - Check validity period 3. **Verify VCEK is signed by ASK** - Check signature - Check validity period - Check VCEK's X.509 extensions contain the correct TCB values - Check VCEK's X.509 extensions contain the correct CHIP_ID If any step fails, reject the report. ### Step 5: Verify the Report Signature Using the VCEK public key from the certificate: 1. Compute SHA-384 of report bytes 0x00 through 0x29F 2. Verify the ECDSA P-384 signature at offset 0x2A0 The signature format stores R and S components as zero-extended little-endian 72-byte values (144 bytes total, padded to 512). ### Step 6: Check Report Contents The signature being valid only proves the report came from genuine AMD hardware. You still need to check that the report says what you expect: **Check MEASUREMENT:** - Does it match your expected launch digest? - This proves the right code was loaded **Check POLICY:** - DEBUG bit should be 0 for production - Check other policy flags match expectations **Check REPORTED_TCB:** - Is it at or above your minimum required TCB? - Old TCB versions may have known vulnerabilities **Check REPORT_DATA:** - Does it contain the expected value? - For TLS binding: does it match hash of the TLS public key? - For freshness: does it contain your nonce? **Check VMPL:** - Which privilege level requested this report? - VMPL 0xFFFFFFFF means host-requested (no guest involvement) ### Computing Expected Measurements For verification to work, you need to know what measurement to expect. This requires knowing exactly what will be loaded at launch. The `sev-snp-measure` tool computes expected measurements given the launch components: ```bash sev-snp-measure \ --mode snp \ --vcpus 4 \ --ovmf /path/to/OVMF.fd \ --kernel /path/to/vmlinuz \ --initrd /path/to/initrd.img \ --append "console=ttyS0" ``` This outputs the expected launch digest. Your verifier compares the report's MEASUREMENT against this value. For this to work, your builds must be reproducible. If OVMF or the kernel aren't byte-for-byte identical to what you measured, the digest won't match. ## The Identity Block (IDB) The Identity Block is an optional mechanism for asserting control over launch validation at the firmware level. ### What the IDB Does The guest owner pre-computes what the measurement should be, signs it, and provides this signed assertion to the hypervisor during launch. The PSP verifies that the actual measurement matches the IDB before the guest can run. **Before launch:** 1. Guest owner builds VM image, computes expected measurement 2. Creates Identity Block containing: - Expected launch digest - Expected policy - FAMILY_ID, IMAGE_ID, GUEST_SVN (metadata) 3. Signs IDB with their ID Key 4. Provides IDB + signature + public key to hypervisor **During launch:** 5. PSP performs normal launch, computing measurement 6. At SNP_LAUNCH_FINISH, PSP checks: - Does computed measurement match IDB.LD? - Does actual policy match IDB.POLICY? - Is the signature valid? 7. If any check fails: launch fails, guest never runs 8. If all pass: launch succeeds, key digests appear in future reports ### Why Use IDB Without IDB, the hypervisor could launch a modified VM image, and the verifier would only detect this after the fact (when checking the attestation report). The guest would have already started running. With IDB, the launch fails immediately if the measurement doesn't match. The guest never executes unauthorized code. IDB also enables a signing authority model: the guest owner's key appears in attestation reports (as ID_KEY_DIGEST and optionally AUTHOR_KEY_DIGEST). Verifiers can check "this was launched with authorization from this entity" without needing to know the exact expected measurement themselves. ## Attestation with SVSM When an SVSM runs at VMPL0, attestation has additional considerations. ### The Measurement Includes SVSM If SVSM is present, it's loaded and measured during launch, before the guest OS. The final measurement reflects both SVSM and guest firmware. This means: - Verifiers need to know both what SVSM and what guest firmware to expect - Different SVSM versions produce different measurements - You're trusting the SVSM as part of your TCB ### VMPCK0 and Attestation Requests The SVSM zeros VMPCK0 (the VMPL0 communication key) after startup to prevent the guest OS from impersonating VMPL0 to the PSP. This affects attestation: - The guest OS (at VMPL2/3) uses VMPCK2/3 to request attestation - Those reports show VMPL=2 or VMPL=3 - Only SVSM can generate VMPL=0 reports If you need VMPL0 attestation, the SVSM must provide it. The SVSM specification includes attestation protocols for this. ### SVSM Attestation Protocol The SVSM can provide attestation reports that include a services manifest, describing what services (vTPM, etc.) are available and their properties. This is defined in SVSM Protocol 1: - `SVSM_ATTEST_SERVICES`: Get report with manifest of all services - `SVSM_ATTEST_SINGLE_SERVICE`: Get report for one specific service This allows a verifier to check "not only is the SVSM the expected version, but it's offering the expected services." ## What Attestation Does and Doesn't Prove ### Attestation Proves - The report was generated by genuine AMD hardware with SEV-SNP - The hardware has a specific TCB version (firmware/microcode) - The guest was launched with specific measured code - The guest provided specific REPORT_DATA at request time - (If IDB used) A specific entity authorized this launch ### Attestation Does NOT Prove - The guest code is correct or secure (only that it matches the measurement) - The guest hasn't been compromised since launch (only initial state) - The hypervisor is behaving correctly (only that SNP is active) - Side channels aren't leaking information - The code does what you think it does (measurement is identity, not behavior) For closed-source code, attestation proves "this binary is running" but not "this binary does what the vendor claims." You still need to trust the code author. ## Security Considerations ### Verifier Responsibilities **Pin the ARK:** Your verifier should have a known-good copy of AMD's ARK, not fetch it from KDS on every verification. An attacker who compromises the network path could serve a fake ARK. **Check the CRL:** Certificates can be revoked. Check AMD's CRL to ensure the VCEK hasn't been revoked. **Enforce TCB minimums:** Old firmware versions may have known vulnerabilities. Don't accept reports signed with TCB below your threshold. **Verify REPORT_DATA:** If you're using REPORT_DATA for key binding or nonces, actually check it. A report with valid signature but wrong REPORT_DATA might be a replay or misdirection. ### What Attacks Remain Possible **Denial of service:** The hypervisor can refuse to let the guest request attestation, or refuse to deliver reports to verifiers. Availability is out of scope. **Rollback within committed TCB:** The COMMITTED_TCB sets a floor, but if an older (but still above COMMITTED) TCB has vulnerabilities, the hypervisor might run that version. **Pre-attestation attacks:** If the guest can be compromised before it requests attestation, the attacker could make the guest lie in REPORT_DATA (put their key instead of the legitimate key). **Post-attestation compromise:** Attestation proves initial state. If the guest is exploited after boot, attestation won't detect it. ## Practical Verification Summary **Verification Checklist:** - Received attestation report from guest - Extracted CHIP_ID, REPORTED_TCB, CPUID info - Fetched certificate chain from AMD KDS - Verified ARK is self-signed (compare against pinned copy) - Verified ASK is signed by ARK - Verified VCEK is signed by ASK - Verified VCEK extensions match report (TCB, CHIP_ID) - Checked CRL to ensure VCEK not revoked - Verified report signature with VCEK public key - Checked MEASUREMENT matches expected launch digest - Checked POLICY flags are acceptable (DEBUG=0, etc.) - Checked REPORTED_TCB meets minimum requirements - Checked REPORT_DATA contains expected value - Checked VMPL is expected level All checks pass → trust that this is the expected code on genuine AMD hardware. We provide SDKs that handle this verification flow. You don't need to implement certificate fetching, chain validation, or signature verification yourself—though you can if you want to verify independently. ## Attestation on Confidential We enable attestation on every request-response flow. How you integrate is up to you: - **Session-based:** Verify attestation once when establishing a connection, then trust subsequent requests on that session - **Per-request:** Include attestation with each request for maximum assurance - **On-demand:** Request attestation when you need it, skip it when you don't **What our attestations prove:** - The code running matches a specific measurement (tied to your git commit) - The hardware is genuine AMD/Intel/NVIDIA TEE hardware - The TCB version meets your security requirements - The workload is running in a confidential environment **Our server and client SDKs** let you: - Generate attestation reports with custom REPORT_DATA - Verify attestation reports and certificate chains - Bind TLS sessions to attestation for end-to-end verification **Manual verification** is always possible. Our attestations use standard formats—you can verify them with AMD's tools, OpenSSL, or your own implementation. ## Further Reading - [Document 4: Privilege & Communication](/docs/concepts/confidential-computing-primer/04-privilege-and-communication) for VMPLs and SVSM details - [Document 6: Measurement Strategies](/docs/concepts/confidential-computing-primer/06-measurement-strategies) for extending trust beyond launch --- # Measurement Strategies This document explains the measurement problem and how we approach it at Confidential. The gap between what hardware measures at launch and what you actually run is fundamental to confidential computing—and bridging that gap is core to what we do. After reading this, you'll understand the measurement gap, two common approaches to bridge it (initramfs and dm-verity), and how SVSM can provide isolated services like a vTPM for key sealing. This assumes familiarity with [Document 5: Attestation](/docs/concepts/confidential-computing-primer/05-attestation), particularly how launch measurements work and what LAUNCH_DIGEST contains. ## The Problem: Your Code Isn't in the Measurement The launch digest covers what's loaded at VM startup: OVMF firmware, the kernel, and the initial boot filesystem. Your application code? Probably not measured. Consider a typical deployment: you have a Python service with dependencies. At launch, the PSP measures the kernel and initial boot image. Then the kernel boots, mounts a disk, and runs your code from that disk. The disk contents were never measured by the PSP. A malicious hypervisor could swap your disk image for a modified one, and the attestation report would look identical. ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ The Measurement Gap │ │ │ │ What the PSP measures at launch: │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ OVMF (UEFI firmware) │ │ │ │ vmlinuz (Linux kernel) │ │ │ │ initramfs (initial boot filesystem) │ │ │ │ cmdline (kernel parameters) │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ What you actually want to attest: │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ Your application code │ │ │ │ Python interpreter │ │ │ │ Dependencies and libraries │ │ │ │ Configuration files │ │ │ │ Model weights │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ │ The gap: application code loads from disk AFTER the measurement. │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` This isn't a flaw in SEV-SNP. The hardware can only measure what's in memory at launch. It has no way to measure a disk image that will be mounted later. The question is: how do you extend trust from the PSP measurement to your runtime code? This is the measurement gap we bridge. When you deploy on Confidential, your application code becomes part of an attested measurement chain—you don't need to manage IGVM files, initramfs builds, or dm-verity setup yourself. ## Background: Key Concepts Before diving into the strategies, let's define some terms that will come up repeatedly. ### initramfs (Initial RAM Filesystem) When a Linux system boots, it faces a chicken-and-egg problem: the kernel needs drivers to access the disk, but those drivers might be on the disk. The solution is initramfs, a small filesystem that gets loaded into RAM alongside the kernel. The initramfs is a compressed archive (cpio format, gzipped) containing a minimal Linux environment. The kernel unpacks it into a RAM-based filesystem and runs /init from there. Typically, initramfs contains just enough to mount the real root filesystem: disk drivers, filesystem tools, and a small init script. Once the real root is mounted, the system pivots to it and discards the initramfs. For confidential computing, initramfs has a special property: it's loaded at launch time, so the PSP measures it. Whatever is in the initramfs is part of the LAUNCH_DIGEST. ### dm-verity (Device Mapper Verity) dm-verity is a Linux kernel feature that provides integrity verification for block devices. It's used by Android, Chrome OS, and other systems that need verified boot. The idea is simple: before deployment, you compute a hash tree (Merkle tree) over your filesystem image. The root of this tree, a 32-byte hash, commits to the entire filesystem contents. At runtime, the kernel verifies each block as it's read by checking it against the hash tree. If any block has been modified, the verification fails and the kernel returns an I/O error. The critical insight for confidential computing: the root hash is tiny (32 bytes). You can embed it in the initramfs. The PSP measures the initramfs (including the root hash), and the kernel uses that hash to verify everything else. ### TPM (Trusted Platform Module) and vTPM A TPM is a dedicated security chip that provides several capabilities: secure key storage, random number generation, and platform measurement. The key feature for confidential computing is key sealing: the ability to encrypt data such that it can only be decrypted when the system is in a specific state. TPMs have Platform Configuration Registers (PCRs), which are special hash accumulators. You can "extend" a PCR by mixing in new data: new_value = hash(old_value || new_data). Critically, you cannot set a PCR to an arbitrary value or reset it (except by rebooting). This creates a tamper-evident log of what has happened on the system. You can seal keys to specific PCR values, meaning the key can only be unsealed if the PCRs match the expected state. A vTPM (virtual TPM) brings these capabilities to virtual machines. For confidential computing, the vTPM runs inside the SVSM at VMPL0, isolated from the guest kernel by hardware. The kernel can request TPM operations through a defined interface, but cannot directly access the vTPM's state. This isolation is what makes the vTPM trustworthy: even a compromised kernel cannot unseal keys that were sealed to different PCR values. ## Computing Expected Measurements with IGVM Before examining the strategies, we need to address a practical problem: how do you know what measurement to expect? The PSP computes the LAUNCH_DIGEST by hashing each page of initial memory along with its address and metadata. To verify an attestation report, you need to perform the same computation on your build machine and compare results. We use IGVM (Independent Guest Virtual Machine) format to solve this. IGVM is a standardized file format that packages everything needed to launch a confidential VM: firmware, kernel, initramfs, and metadata about how to lay out memory. The IGVM file specifies exactly which bytes go at which addresses, eliminating ambiguity. When you build your VM image, you produce an IGVM file. A measurement tool can parse this file and compute the expected LAUNCH_DIGEST deterministically. Because the IGVM format fully specifies the memory layout, the measurement is reproducible: same IGVM file always produces the same expected measurement. This matters because the PSP's measurement depends not just on file contents, but on where each page is placed in guest physical memory. Without a standardized format, different hypervisors might lay out memory differently, producing different measurements for the same logical content. IGVM eliminates this problem. We use IGVM to ensure attested measurements. When you push code, we build an IGVM file that captures the complete memory layout, making the resulting measurement deterministic and verifiable. ## Two Common Approaches There are various ways to extend trust to your application code. Here we cover two common, straightforward approaches: | Approach | What's Measured | Who Does the Measuring | Key Tradeoff | | --------- | -------------------------------------- | ---------------------- | -------------------- | | initramfs | Everything at launch | PSP (hardware) | Size limited to RAM | | dm-verity | Root hash at launch, blocks at runtime | PSP + kernel | Read-only filesystem | These aren't the only options, but they represent the most common patterns. Both can be combined with SVSM to demote the kernel and enable isolated services like a vTPM for key sealing. ## Option 1: Pack Everything Into initramfs The simplest approach: put your application code directly in the initramfs. The PSP measures it at launch. ### How It Works Normally, initramfs contains just enough to boot: a small init script, busybox for basic utilities, and drivers. But you can pack anything into it: your application code, the Python interpreter, all your dependencies, configuration files. The structure would include: an init script (the first process that runs), basic utilities, your language runtime, its standard library, your application code, and any shared libraries needed. All of this gets packed together into the cpio archive. The init script mounts the necessary virtual filesystems (proc, sysfs, devtmpfs) and then executes your application directly. There's no disk mounting, no package managers, no systemd. Just your code running. To build the initramfs, you create a cpio archive of the directory structure and compress it. The result is a single file that gets included in your IGVM image. ### The Trust Chain ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ initramfs Trust Chain │ │ │ │ AMD PSP │ │ │ measures │ │ ▼ │ │ OVMF + kernel + initramfs (contains your code) │ │ │ │ │ │ the measurement is: SHA-384(page contents || GPA || ...) │ │ ▼ │ │ LAUNCH_DIGEST in attestation report │ │ │ │ │ │ verifier compares to expected value │ │ ▼ │ │ If match: entire initramfs contents are trusted, including your app │ │ │ │ Trust basis: PSP hardware measurement │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` The PSP directly measures your application code. No software intermediary is trusted for integrity of the initial load. ### Practical Workflow The workflow is straightforward: copy your latest application code into the initramfs directory structure, rebuild the cpio archive, regenerate your IGVM file, and compute the new expected measurement from the IGVM. The measurement value is what your verifier should expect. Every time you change your code, you rebuild the initramfs, which changes the IGVM, which changes the expected measurement. We automate this workflow. Push your code, and we build the initramfs, generate the IGVM, and record the expected measurement. All this is tied to your specific git commit. ### Tradeoffs **Strengths:** - Simplest verification (just compare LAUNCH_DIGEST) - No runtime verification overhead for initial code - Minimal software in the trust chain for initial load **Limitations:** - Size constrained by available RAM (initramfs unpacks into tmpfs) - Every code change requires new initramfs and new expected measurement - Large dependencies (ML frameworks, model weights) may not fit - Filesystem is in memory, so writes consume RAM - No runtime measurement: once booted, the kernel is trusted to behave correctly **When to use:** When your total runtime footprint (code + deps + data) fits comfortably in memory. Good for small services, microservices, or situations where you want minimal complexity in the boot process. ## Option 2: dm-verity for Filesystem Verification When your application is too large for initramfs, use dm-verity to extend trust to a disk-based filesystem. ### How It Works You compute a Merkle tree (hash tree) over your filesystem image. The tree structure looks like this: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ dm-verity Merkle Tree │ │ │ │ Root Hash │ │ (32 bytes) │ │ / \ │ │ Hash01 Hash23 │ │ / \ / \ │ │ Hash0 Hash1 Hash2 Hash3 │ │ │ │ │ │ │ │ Block0 Block1 Block2 Block3 │ │ 4KB 4KB 4KB 4KB │ │ │ │ The root hash commits to the entire filesystem. │ │ Verifying a block = hash it, walk up the tree, check against root. │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` The critical insight: the root hash is only 32 bytes. You embed this in your initramfs. The PSP measures the initramfs (including the root hash). At runtime, the kernel verifies disk blocks against this hash. ### The Trust Chain ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ dm-verity Trust Chain │ │ │ │ AMD PSP │ │ │ measures │ │ ▼ │ │ OVMF + kernel + initramfs │ │ │ │ │ │ initramfs contains: root hash = 7f3b2c9e... │ │ ▼ │ │ LAUNCH_DIGEST commits to this root hash │ │ │ │ │ │ at boot, initramfs sets up dm-verity │ │ ▼ │ │ Kernel creates verified block device │ │ │ │ │ │ every block read is verified against root hash │ │ ▼ │ │ Your rootfs (contains Python, your app, etc.) │ │ │ │ Trust basis: PSP measures root hash, kernel verifies blocks │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` Let's trace why an attack would fail: **Attacker modifies rootfs.img:** The hash of the modified blocks won't match the Merkle tree. The kernel returns I/O error. Attack detected. **Attacker modifies the hash tree:** The root of the modified tree won't match the root hash in initramfs. Block verification fails. Attack detected. **Attacker modifies initramfs to use different root hash:** LAUNCH_DIGEST changes. Attestation verification fails. Attack detected. ### Setup: Build Time On your trusted build machine, you create your root filesystem with everything you need: the Python interpreter, your application, all dependencies, configuration files, and any other runtime components. You then create a filesystem image from this directory. Next, you compute the Merkle tree over the filesystem image. This produces two things: the hash tree (stored in a separate file) and the root hash (a 32-byte value). The root hash is what you'll embed in your initramfs. ### Setup: Boot Time The initramfs init script has the root hash baked into it. At boot, after mounting the standard virtual filesystems, the init script creates a dm-verity device. This takes the rootfs image, the hash tree image, and the root hash. If the root hash doesn't match the hash tree, device creation fails. Once the verified device is created, the init script mounts it read-only as the new root. It sets up tmpfs mounts for directories that need to be writable (like /tmp and /var), then pivots to the verified filesystem and starts the real init system. ### Who Verifies What This is a common point of confusion. Let's be explicit: | Component | Who Measures/Verifies | Where | | ------------------------ | ------------------------ | -------------------- | | OVMF, kernel, initramfs | AMD PSP | Hardware, at launch | | Root hash (in initramfs) | AMD PSP | Hardware, at launch | | Filesystem blocks | Linux kernel (dm-verity) | Software, at runtime | The kernel is trusted because it was measured by the PSP. If the kernel is modified, the measurement changes, attestation fails. A correct kernel will correctly verify dm-verity. So the chain holds. The kernel was always in your TCB anyway. You can't run userspace without trusting the kernel. dm-verity doesn't add to the TCB, it just leverages the kernel you already trust. ### Handling Writes dm-verity is read-only. If you modify a block, verification fails on the next read. For runtime state, use overlayfs: mount a tmpfs for the "upper" layer, with the verified filesystem as the "lower" layer. The combined view appears writable. Reads come from the verified lower layer; writes go to the RAM-backed upper layer. This is how Android and Chrome OS work. The OS is read-only and verified. User data goes to a separate writable partition. ### Practical Workflow When you update your application code, the workflow has several steps: 1. Copy your new code into the rootfs directory. 2. Rebuild the filesystem image. 3. Compute a new hash tree and root hash. 4. Update your initramfs init script with the new root hash. 5. Rebuild the initramfs archive. 6. Regenerate your IGVM file and compute the new expected measurement. The kernel and OVMF typically stay the same across updates. Only the initramfs changes (because it contains the new root hash), which means the LAUNCH_DIGEST changes. We automate this entire workflow. When you deploy, we build the filesystem image, compute the Merkle tree, embed the root hash in the initramfs, and generate the IGVM—all in a single pipeline tied to your git commit. ### Tradeoffs **Strengths:** - Full filesystem, no RAM constraints on size - Can include large dependencies, ML frameworks, model weights - Read verification is fast (microseconds per block, parallelizable) - Familiar filesystem semantics - Continuous verification: every block read is checked **Limitations:** - Filesystem is read-only (need overlayfs for writes) - Kernel is in TCB for verification (but it was anyway) - No runtime measurement of dynamic content **When to use:** When your runtime is too large for initramfs. Good for applications with large dependencies, or when you want a "normal" Linux environment. ## Adding SVSM: Kernel Demotion and Isolated Services Both initramfs and dm-verity solve the problem of getting trusted code into your VM. But in both cases, the kernel runs at VMPL0, the most privileged level. If you want to run services that are isolated from the kernel (like a vTPM for key sealing), you need to demote the kernel to a lower privilege level. That's what SVSM does. ### What SVSM Does The SVSM (Secure VM Service Module) runs at VMPL0, the most privileged level inside the guest. It's measured by the PSP at launch along with everything else. The key change is that the guest kernel now runs at VMPL2, a lower privilege level. This means: - The kernel cannot access SVSM memory (hardware enforced) - The SVSM can run services that the kernel cannot tamper with - Even a compromised kernel cannot access VMPL0 resources The SVSM provides a place to run isolated services. The most common is a vTPM for key sealing. ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ SVSM Architecture │ │ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ VMPL 0 (most privileged) │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ │ │ SVSM + isolated services (e.g., vTPM) │ │ │ │ │ │ • Measured by PSP at launch │ │ │ │ │ │ • Kernel CANNOT access this memory │ │ │ │ │ │ • Can provide vTPM, attestation proxy, other services │ │ │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ VMPL 2 (less privileged, where kernel now runs) │ │ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ │ │ Guest kernel + your application │ │ │ │ │ │ • Calls SVSM for vTPM operations │ │ │ │ │ │ • Cannot directly access vTPM state or keys │ │ │ │ │ │ • Cannot unseal keys sealed to different PCR values │ │ │ │ │ └──────────────────────────────────────────────────────────────┘ │ │ │ └────────────────────────────────────────────────────────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` ### vTPM and Key Sealing The primary use of the vTPM in this context is key sealing: encrypting keys so they can only be decrypted when PCRs have specific values. This binds secrets to a particular system state. PCRs are hash accumulators. You extend them by mixing in new data: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ PCR Extension │ │ │ │ Initial state: PCR[14] = 0x000...000 │ │ │ │ Extend with hash of /app/main.py: │ │ PCR[14] = SHA-256(PCR[14] || SHA-256(/app/main.py)) │ │ │ │ Extend with hash of config.json: │ │ PCR[14] = SHA-256(PCR[14] || SHA-256(config.json)) │ │ │ │ The final PCR value depends on all extended data, in order. │ │ Cannot be "undone" or reset to an intermediate value. │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` You can seal a key to PCR[14]. Later, the key can only be unsealed if PCR[14] has the exact same value. If the kernel were compromised and ran different code, the PCR would be different, and the sealed key would remain inaccessible. Note that you could measure things into PCRs without a vTPM (the kernel can maintain its own measurement log). The vTPM adds two things: (1) the measurement state is isolated from the kernel, so a compromised kernel can't forge it, and (2) you get standard TPM APIs for key sealing. ### The Combined Trust Chain With SVSM added to either base approach: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ Combined Trust Chain │ │ │ │ AMD PSP │ │ │ measures │ │ ▼ │ │ SVSM (at VMPL0) │ │ │ │ │ │ also measured: │ │ ▼ │ │ OVMF + kernel + initramfs (with dm-verity root hash, if using) │ │ │ │ │ │ LAUNCH_DIGEST covers all of this │ │ ▼ │ │ Guest boots, kernel runs at VMPL2 (demoted) │ │ │ │ │ │ kernel uses SVSM services (vTPM, etc.) │ │ ▼ │ │ Keys sealed to PCR values can only be unsealed if PCRs match │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` ### When to Add SVSM SVSM is useful when: - You need key sealing (binding secrets to specific PCR/system states) - You want services isolated from the kernel (vTPM, attestation proxy) - You want defense in depth: even if the kernel is compromised, sealed keys remain protected - You have multiple parties and want isolation between different components It adds complexity: you're adding another component to your TCB (the SVSM), and the kernel runs at a lower privilege level which may affect some operations. But for scenarios where you need isolated services or key sealing, this complexity is justified. ## PSP Communication: VMPCKs and Guest Messages Before closing, let's cover how the guest talks to the PSP for attestation requests. This applies to all approaches. ### The VMPCK Protocol The guest communicates with the PSP through an encrypted channel using VMPCKs (VM Platform Communication Keys). There are four keys, one per VMPL: ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ VMPCKs in the Secrets Page │ │ │ │ Offset 0x020: VMPCK0 (32 bytes) - for VMPL0 (SVSM, if present) │ │ Offset 0x040: VMPCK1 (32 bytes) - for VMPL1 │ │ Offset 0x060: VMPCK2 (32 bytes) - for VMPL2 (guest kernel, typically) │ │ Offset 0x080: VMPCK3 (32 bytes) - for VMPL3 │ │ │ │ These are AES-256 keys for AES-256-GCM encryption. │ │ The PSP has copies; the guest has copies in the secrets page. │ │ The hypervisor never sees them (secrets page is encrypted). │ │ │ └──────────────────────────────────────────────────────────────────────────┘ ``` When the guest wants an attestation report, it: 1. Constructs the request (MSG_REPORT_REQ with REPORT_DATA) 2. Encrypts with AES-256-GCM using its VMPCK 3. Sends through shared memory via VMGEXIT 4. Hypervisor forwards to PSP (can't decrypt it) 5. PSP decrypts, processes, encrypts response 6. Guest decrypts response Sequence numbers prevent replay attacks. If the hypervisor tries to replay an old message, the sequence number is wrong and the PSP rejects it. ### VMPCK0 Zeroing When an SVSM is present, it zeros VMPCK0 in the secrets page after reading it. This means: - The guest kernel (at VMPL2/3) cannot communicate as VMPL0 - Only the SVSM can request VMPL0 attestation - The guest kernel uses VMPCK2 or VMPCK3 If there's no SVSM and the guest kernel runs at VMPL0, it has VMPCK0 and can use it directly. ## Practical Recommendations 1. **Start with dm-verity for most workloads.** It handles arbitrary sizes, gives you a real filesystem, and the trust chain is solid. The read-only constraint is manageable with overlayfs. 2. **Use initramfs for simple, small workloads.** If you can fit everything in RAM and want minimal complexity, initramfs is straightforward. 3. **Add SVSM when you need isolated services or key sealing.** If you need to seal keys to PCR values, or want services that the kernel can't tamper with, add the SVSM layer to demote the kernel. 4. **Use IGVM for reproducible measurements.** Whatever approach you choose, IGVM ensures your expected measurements are deterministic and portable. ## How We Handle Measurement We abstract the measurement workflow so you don't need to manage IGVM files, initramfs builds, or dm-verity setup yourself. **Attested builds:** When you push code to your connected GitHub repo, we build an attested image. The build process is recorded, and the resulting measurement is tied to your git commit hash. **The measurement chain:** ``` Your git commit │ ▼ Our build pipeline (recorded, auditable) │ ▼ IGVM file with deterministic memory layout │ ▼ Expected LAUNCH_DIGEST (tied to your commit) │ ▼ Attestation report (signed by hardware) ``` **Verifying a deployment:** Every deployment on Confidential can produce an attestation report. That report contains the LAUNCH_DIGEST, which you can trace back to the git commit that produced it. If you want to verify that a deployment matches your source code, you can: 1. Get the attestation report from the running deployment 2. Check that the MEASUREMENT matches the expected value for your commit 3. Verify the certificate chain to confirm the hardware signature We provide SDKs that handle this verification, but the underlying data is standard—you can verify independently with AMD's tools or your own implementation. **What this means in practice:** You push code. We build it into an attested image. Every request to that deployment can include cryptographic proof that your code (identified by git commit) is running on genuine TEE hardware. The measurement gap is bridged automatically. ## Further Reading - [Document 5: Attestation & Verification](/docs/concepts/confidential-computing-primer/05-attestation) for how measurements end up in reports - [Document 4: Privilege & Communication](/docs/concepts/confidential-computing-primer/04-privilege-and-communication) for VMPLs and SVSM details --- # c8s: A Confidential Kubernetes Architecture Download PDF version **April 2026** --- ## Abstract This paper presents C8s, a confidential computing architecture for Kubernetes that provides cryptographically rooted confidentiality, integrity, and verifiability guarantees for Kubernetes clusters from infrastructure operators. These guarantees are cryptographically provable to any independent third party verifier. The architecture is built on hardware Trusted Execution Environments (TEEs), specifically AMD SEV-SNP, Intel TDX, and NVIDIA Confidential Computing support, to establish an attestation-rooted trust boundary around confidential VMs. This design is compatible with managed Kubernetes services such as Amazon EKS, Google GKE, and Microsoft AKS, where the control plane cannot be attested. Under this boundary, three groups gain guarantees that are absent from conventional deployments. Data and artifact owners can deploy sensitive workloads and proprietary artifacts on third-party infrastructure without risking exfiltration. Compute providers can offer execution services without revealing workloads to cloud operators. End users can submit requests that remain opaque to all parties except the attested TEE processing them. Representative workloads include AI inference, securing AI model weights, and training or fine-tuning on sensitive data. --- ## 1. Introduction ### 1.1 Motivation Running sensitive workloads on third-party infrastructure presents a three-sided confidentiality problem. Artifact owners invest substantial resources in proprietary assets such as trained model weights, curated datasets, cryptographic keys, or proprietary algorithms, and risk exfiltration when those assets are deployed on infrastructure they do not control. Compute providers running execution infrastructure need assurance that cloud providers cannot observe their workloads. End users submitting sensitive requests, in healthcare, legal, financial, and national security contexts, require guarantees that neither the compute provider nor the infrastructure operator can inspect their inputs or outputs. Today, the dominant mitigation for all three concerns is contractual. Model providers sign licensing agreements restricting weight access. Cloud providers publish compliance certifications. Service providers implement access controls. None of these mechanisms are cryptographically rooted. A sufficiently privileged insider, such as a cloud engineer with hypervisor access, a Kubernetes administrator with credentials, or a compromised node with root access, can read data from memory, intercept network traffic, or exfiltrate model weights from storage. Encryption at rest and encryption in transit address portions of this problem but leave a fundamental gap during computation where data is plaintext in memory. Workloads, user data, and proprietary intellectual property all reside in memory cleartext, readable by any party with sufficient host access. Kubernetes is a dominant orchestration layer for workloads in production, and therefore a generalized confidential architecture at the cluster level targets a substantial fraction of existing deployed compute infrastructure. Rather than requiring workloads to be rewritten against a specialized confidential runtime, the approach here preserves the Kubernetes surface area that operators and developers already know, and any workload already running on a Kubernetes cluster can inherit these guarantees with minimal modification. ``` ┌─────────────────────────┐ │ Artifact owner │ │ [weights, datasets, │ │ proprietary code] │ └────────────┬────────────┘ │ Deploys ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Shared infrastructure │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │ │ Hypervisor │ │ Host OS │ │ Cloud operator │ │ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Kubernetes control plane │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ All layers above can read plaintext memory, intercept │ │ network traffic, and exfiltrate artifacts from storage. │ │ │ └─────────────────────────────────────────────────────────────────────┘ ▲ ▲ │ │ │ Runs workload │ Submits query │ │ ┌────────────┴────────────┐ ┌─────────────┴───────────┐ │ Compute provider │ │ End user │ │ [training, inference, │ │ [health, legal, │ │ data processing] │ │ financial queries] │ └─────────────────────────┘ └─────────────────────────┘ ``` *Figure: The three-sided trust problem. Artifact owners, compute providers, and end users share infrastructure none of them controls; every confidentiality boundary between them is enforced by contract rather than by cryptography.* ### 1.2 Problem Statement The core challenge is to construct a deployment architecture for sensitive workloads on Kubernetes in which: 1. Workload inputs, outputs, and intermediate computation state are cryptographically protected from infrastructure operators. 2. Sensitive artifacts, such as model weights, datasets, cryptographic keys, or proprietary code, are encrypted at rest and decrypted only within hardware-attested environments. 3. Clients and customers can independently verify, through cryptographic attestation, that their data and workloads are processed exclusively within trusted hardware running expected code. 4. The architecture operates on managed Kubernetes services where the control plane is outside the operator's control and cannot be attested. ### 1.3 Scope This document describes the architecture, threat model, and core components of the C8s (Confidential Kubernetes) platform developed by Confidential. It covers the trust model, the attestation and certificate issuance flow, the client connection protocol, and the design decisions that enable compatibility with untrusted control planes across a general class of sensitive workloads. The attested build system (Kettle) is referenced but not detailed; it will be the subject of a separate publication. --- ## 2. Background ### 2.1 Trusted Execution Environments A Trusted Execution Environment (TEE) is a hardware-enforced isolation boundary that provides four security properties: - **Confidentiality.** Data within the TEE is encrypted at the hardware level. Memory pages are encrypted with keys managed by the CPU's secure processor. The hypervisor, host operating system, and DMA-capable devices cannot decrypt TEE memory contents. - **Integrity.** Replay attacks, data corruption, and memory remapping are protected by hardware. This maintains the integrity of the initialized software in the TEE. - **Verifiability.** The TEE produces a cryptographic measurement (a hash digest) of the software loaded at launch. This measurement covers firmware, kernel, initial filesystem state, and user space, and is computed by the CPU's secure processor before the guest operating system begins execution. - **Attestation.** The TEE generates signed attestation reports (termed *Evidence* in the RATS architecture [15]) binding the launch measurement to a hardware-rooted signing key. These reports are signed by keys that chain to the hardware manufacturer's root of trust and cannot be forged by any software on the host. The hardware manufacturer takes the role of the *Endorser* in RATS terminology, vouching for the platform via the chain of certificates rooted at its public key. ``` Outside: Hypervisor Host OS BIOS Device drivers Cloud operator ╔═══════════════════════ CVM boundary ════════════════════╗ ║ ║ ║ ┌────────────────────────────────┐ ╮ ║ ║ │ Firmware │ │ ║ ║ └────────────────────────────────┘ │ ║ ║ ┌────────────────────────────────┐ │ ║ ┌─────────────────────────────┐ ║ │ initrd │ │ ║ │ Attestation report │ ║ └────────────────────────────────┘ │ ║ ╞═════════════════════════════╡ ║ ┌────────────────────────────────┐ │ hashed ║ │ Measurement │ ║ │ Kernel │ ├── into a ──────╫─────►│ digest of all layers │ ║ └────────────────────────────────┘ │ single ║ ├─────────────────────────────┤ ║ ┌────────────────────────────────┐ │ measurement ║ │ Hardware signature │ ║ │ User space │ │ ║ │ chains to vendor root │ ║ └────────────────────────────────┘ │ ║ └─────────────────────────────┘ ║ ┌────────────────────────────────┐ │ ║ ║ │ Deployed workload │ ╯ ║ ║ └────────────────────────────────┘ ║ ║ ║ ╚═════════════════════════════════════════════════════════╝ ``` AMD Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP) [1] and Intel Trust Domain Extensions (TDX) [2] implement these properties at the virtual machine level. In both architectures, the entire VM operates as a Confidential VM (CVM), and all guest memory is encrypted with keys the hypervisor never possesses. NVIDIA Confidential Computing (CC) mode [3] extends the TEE boundary to the GPU. When a GPU operates in CC mode, data crossing the PCIe bus between the CPU TEE and GPU is encrypted and integrity-protected via bounce buffers, and the GPU isolates its protected memory and state from the host hypervisor and OS. The GPU is able to generate its own attestation confirming that it is operating in CC mode. To use CC mode on GPUs, the GPU must be paired with a CC-capable CPU. In that setup, the CPU is the attestation trust anchor. For GPU-accelerated workloads, GPU CC mode is significant because the working set, whether that consists of model parameters, training data, intermediate activations, or other sensitive state, resides in VRAM for the lifetime of the compute process. Without GPU CC mode, VRAM is the one location where plaintext data is not hardware-protected. NVIDIA H100 and later GPUs with CC mode enabled close this gap. The C8s architecture supports AMD SEV-SNP and Intel TDX for CPU-level CVM attestation, and NVIDIA GPUs operating in CC mode for GPU-level attestation. The same attestation machinery, including CDS verification, NRI policy enforcement, raTLS mesh, and client-side encryption, applies uniformly across CPU-only and GPU-accelerated nodes. On GPU-equipped nodes, the GPU's attestation evidence is included in the node's combined attestation report, and the CDS verifies it alongside the CPU measurement. ### 2.2 Standard Kubernetes Architecture A Kubernetes deployment is organized as a cluster of machines, called nodes, that collectively run containerized workloads. Each node is a physical or virtual host contributing CPU, memory, and optionally GPU resources to the cluster. Workloads run inside pods, which are the smallest deployable unit in Kubernetes. Each pod is a group of one or more containers scheduled together on the same node and sharing its network and storage namespaces. A control plane, running separately from the worker nodes, accepts desired-state declarations from operators and schedules pods onto nodes that can satisfy their resource requirements. A typical production deployment on this substrate comprises one or more application pods fronted by a routing layer. External traffic arrives at a load balancer or ingress controller, which terminates TLS and forwards requests into the cluster. The routing layer, whether a dedicated router, a service mesh gateway, or a Kubernetes Service, distributes requests across available pods via Kubernetes service discovery. Application pods run the workload with dedicated CPU, GPU, or memory resources, pulling container images from a registry and loading any required data or artifacts from persistent volumes, object storage, or configuration at startup. ``` ┌─────────────────────────────────────────────────────────────────────┐ │ Kubernetes Cluster │ │ │ │ ┌─────────────────────┐ │ │ │ Control plane │ │ │ └──────────┬──────────┘ │ │ │ Schedules │ │ ┌───────────────┴───────────────┐ │ │ ▼ ▼ │ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ │ Node 1 │ │ Node 2 │ │ │ │ │ │ │ │ │ │ ┌─────┐ ┌─────┐ │ │ ┌─────┐ ┌─────┐ │ │ │ │ │ Pod │ │ Pod │ │ │ │ Pod │ │ Pod │ │ │ │ │ └─────┘ └─────┘ │ │ └─────┘ └─────┘ │ │ │ └──────────▲──────────┘ └──────────▲──────────┘ │ │ │ │ │ │ └───────────────┬───────────────┘ │ │ │ Routes │ │ ┌──────────┴──────────┐ │ │ │ Load balancer │ │ │ └──────────▲──────────┘ │ │ │ │ └──────────────────────────────────┼──────────────────────────────────┘ │ HTTP ┌───────┴───────┐ │ Client │ └───────────────┘ ``` ### 2.3 Related Work Prior work on confidential computing in Kubernetes, including Kata Containers [6] and the Confidential Containers (CoCo) project [7], has focused on per-pod isolation by running each pod in its own micro-VM with an independent attestation. These projects establish much of the ecosystem's shared machinery, including confidential container runtimes, per-pod attestation agents, and key broker services that release secrets to individually attested containers. Single-tenant large-scale deployments, where container-level isolation is not strictly required, have historically been a poor fit for this model because every workload has to be adapted to run in a confidential container runtime, and the integration cost falls on every Kubernetes deployment that does not already run its containers inside VMs. The approach C8s takes is to have multiple configuration options that widen the deployment options and lower integration costs. The C8s architecture treats the choice of isolation boundary as a deployment-time configuration. Available options range from node level attestation, to per pod attestation, or hybrid configurations within a cluster. When the boundary is drawn at the pod, per-pod confidential VM lifecycle is orchestrated by the **C8s pod runtime**, which presents a direct per-pod attestation to the same CDS. A user or customer can verify these deployment configurations independently through the trust boundary that C8s draws. This deployment flexibility makes C8s compatible with existing managed Kubernetes services such as GKE, AKS, and EKS. ### 2.4 Multi-Recipient Encryption For deployments that opt into client-side encryption (§5.9.3), the architecture needs a mechanism for encrypting payloads to a pool of TEEs such that any single TEE in the pool can decrypt the payload. This serves two purposes. The first is availability, since TEE instances may be terminated or rescheduled at any time without requiring client-side key refresh. The second is operational simplicity, since clients encrypt once per request rather than per-TEE. The natural fit is a hybrid scheme that encrypts the body once under a fresh symmetric key and then wraps that key to each recipient's public key, so the body cost is independent of pool size and only the header grows. Several schemes satisfy this requirement. Elliptic Curve Integrated Encryption Scheme (ECIES) [8] supports multi-recipient wrapping by encrypting a symmetric data key to each recipient's public key. Broadcast encryption schemes [9] offer similar semantics with different tradeoff profiles. The current C8s reference implementation uses AGE (Actually Good Encryption) [10], a file encryption format designed for simplicity and composability; the protocol is not bound to AGE and can be substituted (e.g., for threshold, hybrid, or post-quantum schemes) without changing the rest of the architecture. AGE supports multiple recipients via a stanza-based header format. A file is encrypted with a single symmetric key, and each recipient receives an independently decryptable copy of that key wrapped to their public key. The encrypted payload body is identical regardless of the number of recipients; only the header grows, by approximately 64 bytes per additional recipient. For a pool of 100 TEEs, the header overhead is approximately 6.4 KB, negligible relative to payloads. Two practical properties motivated the choice. The header format supports custom recipient types via a plugin system, allowing routing metadata (such as hostnames) to ride alongside cryptographic material, and the plugin architecture itself provides crypto-agility for the schemes mentioned above. The routing hints embedded in recipient stanzas are discussed further in §5.10. The CDS (introduced in detail in §5.6) signs the manifest of TEE public keys distributed to clients, so clients can verify the authenticity of the key set using standard certificate chain validation without performing individual attestation verification on every request. --- ## 3. Threat Model ### 3.1 Assets Under Protection The architecture protects three categories of assets: 1. **Workload inputs and outputs.** The data submitted to the workload and the results it returns. Examples include user queries and generated tokens in LLM inference, patient records and diagnostic outputs in healthcare analytics, and transaction data and risk scores in financial pipelines. 2. **Sensitive artifacts.** Proprietary assets loaded into the workload at runtime, such as trained model weights, curated datasets, cryptographic keys, or proprietary code. 3. **Intermediate computation state.** Caches, activations, and working memory generated during execution. Examples include KV cache entries and attention activations in LLM inference, intermediate aggregates in analytics workloads, and partial witnesses in cryptographic proving. ### 3.2 Trust Model The trust boundary encompasses the hardware manufacturer, the code running inside attested TEEs, and the CDS (which is itself attested and explicitly trusted as the root of the certificate chain). All other parties are untrusted. | Trusted | Untrusted | | ------------------------------------------------- | -------------------------------- | | Hardware manufacturer (AMD, Intel) | Cloud provider / hypervisor | | CPU/GPU hardware and firmware | Host OS, BIOS, device drivers | | Physical hardware host (datacenter, colo) | Network infrastructure | | Code inside TEE (measured at launch) | Kubernetes control plane | | CDS (attested, trusted root of certificate chain) | Operators with privileged access | | | Storage backends | The trust model places the workload provider's own operators on the untrusted side. This is deliberate, because the architecture offers guarantees that do not depend on the operational security of the party running the infrastructure. One physical-security assumption is required and worth stating explicitly. The physical hardware host (the datacenter, colo, or party that racks and cables the machine) is trusted not to mount physical attacks against the silicon, including memory-bus probing, DIMM substitution, and JTAG or debug-port access. In cloud deployments the physical hardware host and the hypervisor operator are usually the same company (e.g., AWS, Azure, GCP). The model trusts that company in its physical hardware host role while explicitly distrusting it in its hypervisor-operator role. Recent results such as TEE.Fail (2025), Battering RAM (2025), and BadRAM (2024) demonstrate that current TEE designs are vulnerable to adversaries with physical access to running hardware. Protection against such adversaries is out of scope and is treated further in §3.4. ### 3.3 Attacks Addressed - **Memory snooping.** The hypervisor, host OS, and DMA-capable devices cannot read CVM memory. Data in DRAM, queries in process memory, and cache entries are encrypted with keys the hypervisor never possesses. - **Memory tampering.** Replay attacks, corruption, and memory remapping are detected by the hardware. The guest always observes the data it last wrote. - **Memory snapshots.** The TEE protects against memory snapshots taken by the hypervisor while the VM is running. The hypervisor can capture an image of the VM's memory pages, but the contents are encrypted with keys it does not possess, rendering the snapshot useless for data extraction. - **Code substitution.** Attestation binds a cryptographic measurement to the software running in the TEE. Substituting a malicious binary changes the measurement, which is detected during attestation verification. - **Man-in-the-middle (internal).** All intra-cluster traffic uses mutual TLS (mTLS) with certificates issued by the Verifier. Only workloads that have passed attestation possess valid certificates for mTLS. - **Man-in-the-middle (external).** Clients verify the ingress's attestation against a signed manifest and the freshness beacon (§5.9.2) before trusting the TLS session. The TLS certificate's public key is bound by the attestation report to a key generated inside the ingress's TEE, so a forged certificate cannot be presented with valid attestation. - **Unauthorized workload injection.** Workload gating happens inside the trust boundary: in pod-level deployments, the in-pod measured policy or per-customer signing key (§5.1.1) rejects unauthorized images before they execute; in node-level deployments, the NRI image policy enforcer (§5.7) rejects container images whose digests do not appear in a signed allow-list. Both paths are independent of control plane directives. - **Key exfiltration.** Decryption keys for sensitive artifacts are released by the CDS only to workloads that pass attestation, and exist exclusively in the pod's hardware-encrypted memory. They are never stored in Kubernetes Secrets, environment variables, or any location accessible to the control plane. ### 3.4 Attacks Not Addressed - **Side-channel attacks.** Microarchitectural timing attacks (cache timing, branch prediction, page-level access patterns) are not fully mitigated by current CVM implementations. While SEV-SNP and TDX address integrity-based attacks demonstrated against earlier SEV generations [11], cache-based and ciphertext side channels remain an active area of research. - **Denial of service.** A malicious operator or compromised control plane can refuse to schedule workloads, terminate VMs, or disrupt network connectivity. TEEs guarantee confidentiality and integrity, not availability. - **Application-layer vulnerabilities.** A bug in the workload code is exploitable inside the TEE. The hardware protects the execution environment, not the application logic. - **Application-layer extraction.** Repeated querying to distill a model, reconstruct a dataset, or infer proprietary logic from workload responses is an application-layer concern outside the scope of hardware-level protections. - **Physical attacks.** Active probing of the memory bus, DIMM substitution, debug-port access, and similar attacks requiring physical access to running hardware are out of scope. The physical hardware host is trusted under §3.2; an adversary with sustained physical access to the machine is outside the threat model. Recent attacks in this category include TEE.Fail (2025) [21], Battering RAM (2025) [22], and BadRAM (2024) [23]. ## 4. Architecture ### 4.1 Trust Boundary The central construct of the architecture is a **trust boundary**, a cryptographically enforced perimeter separating components that the architecture treats as trusted from those that it does not. Inside the boundary, components run in hardware-attested environments and hold identities issued on the basis of that attestation. Outside the boundary, components are treated as potentially adversarial, regardless of who operates them, and no confidentiality or integrity guarantee depends on their correct behavior. The trust boundary is defined entirely by attestation. A component is inside the boundary if and only if three conditions hold: 1. Its code or image is measurable, meaning a cryptographic digest can be computed over the software that will execute. 2. Its execution environment supports confidential computing, either as a CVM itself, or as a container running inside an already-attested CVM. 3. A verification chain exists from the hardware root of trust to the component's runtime identity such that a remote Verifier can confirm the environment is genuine and the running code matches an expected measurement. Any component that fails one of these conditions sits outside the boundary. Network infrastructure, storage backends, and managed Kubernetes control planes all fail at least the second condition, since the operator does not run the code in a TEE and cannot attest it. These components remain part of the deployment but are stripped of any role in confidentiality enforcement. The architecture depends on them for liveness and orchestration, not for secrecy or integrity. For components that cannot be placed inside the boundary, two strategies apply: 1. **Do not depend on them for confidentiality.** The control plane schedules workloads, but an image policy enforcer on each node independently verifies every container image digest against a signed allow-list. The control plane can request arbitrary workloads; the node rejects those not authorized. 2. **Encrypt before exposure.** Data passing through untrusted components is ciphertext. Storage backends hold encrypted data. The pod network carries only mTLS-encrypted traffic. Client traffic is anchored to an attested ingress over TLS (and may additionally be multi-recipient-encrypted for clients that opt into end-to-end-to-pod confidentiality). ### 4.2 Design Principles With the trust boundary defined, the architecture applies a repeatable formula to every component placed inside it. The formula operationalizes the boundary by making each component independently measurable, identifiable, and verifiable, so that trust composes cleanly across the system. For every component that should go inside the trust boundary, we secure it by implementing the following steps: 1. **Encrypt the runtime.** The component runs in hardware-encrypted memory. 2. **Measure the code.** A cryptographic digest of the running software is computed by hardware at launch. 3. **Bind identity to measurement.** Credentials are issued only after the measurement is verified against a known-good value. 4. **Verify before connecting.** Peers require attestation-rooted identity before accepting connections. 5. **Secure the egress.** All traffic leaving the component is encrypted and authenticated to verified destinations. Steps 1 to 4 enable external parties to verify they are communicating with a genuine TEE running expected code. Step 5 ensures the component does not leak data to unverified infrastructure. The component sections that follow describe each piece of the system once, but the discipline they share is this five-step formula applied at a different layer of the stack; a reader asking how a given component becomes confidential can check it against these five steps and find the answer. ### 4.3 Component Overview The C8s platform introduces the following components: - **CVM node image.** A VM image that boots as a confidential VM on supported TEE hardware. The hardware's secure processor measures the initial state at launch. When GPU CC mode is available, the measurement additionally attests the GPU, producing a combined attestation report covering the full compute path. All processes running inside inherit hardware memory encryption. - **Certificate Distribution Service (CDS).** The root of the certificate chain within the boundary. The CDS verifies attestation reports from CVMs, checks measurements against an allow-list, and issues per-pod raTLS mesh certificates to attested workloads. It also issues freshness beacons on request, letting external clients verify that an attested ingress's recent attestation is current (§5.9.2). External-facing TLS certificates at the ingress are obtained through ordinary public-CA mechanisms bound to a TEE-internal key (out of scope here), so the CDS itself does not issue any externally facing certificate. The CDS runs inside a CVM and is itself attested, and acts as the attestation-gated broker for application-layer secrets including model decryption keys (§5.6.4). - **Attestation service.** An in-CVM component that generates attestation reports and presents them to a verifying party. It runs inside every attested CVM in the cluster, as a DaemonSet inside the node's CVM in node-level deployments and as a per-pod component inside the pod's CVM in pod-level deployments. The service requests the hardware-signed report from the CVM, binds it to the workload's ephemeral public key, and submits it to the CDS for appraisal; in node-level deployments it additionally composes per-pod workload digests on top of the node's CVM report so each pod presents a distinct report. - **C8s pod runtime.** A pod-level runtime layer that launches each pod as its own confidential VM. The runtime produces a pre-computable launch digest covering the pod's boot chain and vCPU initial state, so every pod attests directly to the CDS at start with no dependence on node-level evidence. Workload gating moves into the pod's measured configuration because the host does not see the post-boot container pull. Used wherever the trust boundary is drawn at the pod rather than the node. - **NRI image policy enforcer.** A Node Resource Interface [13] plugin that intercepts every container launch request on each node. It checks the image digest against a signed policy manifest. If the digest is not in the allow-list, the container does not start. This is node-level enforcement, independent of control plane directives. It applies where the trust boundary is drawn at the node; where the boundary is drawn at the pod, the equivalent gating is enforced inside the pod's measured configuration. - **Kettle (attested build system).** A build pipeline that executes inside a TEE, producing container images with deterministic, reproducible digests. The build environment is attested, binding the inputs (source code, dependencies) to the output (image digest) with a hardware-signed attestation report. Kettle is out of scope for this paper and will be detailed in a separate publication. - **raTLS mesh.** A mutual TLS overlay that encrypts all traffic crossing the trust boundary. Every pod holds its own CDS-issued certificate, issued only after the pod's attestation evidence verifies against policy: a direct per-pod attestation where the boundary sits at the pod (default), or composed node-and-workload evidence where the boundary sits at the node. The mesh terminates at the pod, not the node: the pod network, which is untrusted, sees only ciphertext between attested pod endpoints. - **Attestation-aware client.** A client library that fetches a CDS-signed manifest of attested identities (the ingress and the destination TEE pool), verifies attestation evidence either independently or via the CDS signature, and uses the verified material to anchor a TLS connection to the ingress (default) or to encrypt payloads with multi-recipient encryption (optional, §5.9.3). - **Ingress router.** A load balancer running an attested image inside the trust boundary. By default it terminates a public-CA TLS connection bound to its TEE-internal key (clients verify the attached attestation against the CDS-signed manifest and the freshness beacon) and forwards over raTLS to the destination pod. An Encrypted Ingress Router variant (§5.10.2) handles client-side multi-recipient encrypted payloads by routing on header metadata without decrypting the body. - **Secrets Manager Proxy.** A slim service inside the trust boundary that brokers access to enterprise secrets managers sitting outside the trust boundary, releasing secrets only on presentation of a valid CDS-issued mesh identity. Role-based access can be gated on measurements from the attestation. See §5.6.4 for the integration pattern. ### 4.4 Architecture Diagram The following pair of diagrams illustrates what changes when a standard Kubernetes cluster becomes confidential. The top panel is a conventional deployment, and the bottom panel is the same workload under C8s. The trust boundary is hardware-enforced and shown with a double wall, while the control plane stays outside it. ``` Standard Kubernetes ┌─────────────────────────────────────────────────────────────────────────────┐ │ ┌─────────────────────┐ │ │ │ Control plane │ │ │ └──────────┬──────────┘ │ │ │ Schedules │ │ ┌─────────────┴──────────────┐ │ │ ▼ ▼ │ │ ┌─────────────────────────┐ ┌─────────────────────────┐ │ │ │ Node 1 │ │ Node 2 │ │ │ │ ┌───────┐ ┌───────┐ │ │ ┌───────┐ ┌───────┐ │ │ │ │ │ Pod │ │ Pod │ │ │ │ Pod │ │ Pod │ │ │ │ │ └───────┘ └───────┘ │ │ └───────┘ └───────┘ │ │ │ └────────────┬────────────┘ └────────────┬────────────┘ │ │ │ │ │ │ └─────────────┬──────────────┘ │ │ │ Routes │ │ ┌──────────┴──────────┐ │ │ │ Load balancer │ │ │ └──────────▲──────────┘ │ │ │ │ └───────────────────────────────────────┼─────────────────────────────────────┘ │ HTTP, plaintext ┌───────┴───────┐ │ Client │ └───────────────┘ C8s ┌─────────────────────┐ │ Control plane │ untrusted └──────────┬──────────┘ Outside boundary │ Schedules ┌───────────────┴────────────────┐ │ │ ╔════════════════════╪════════════════════════════════╪══════════════════╗ ║ ▼ ▼ ║ ║ ┌─────────────────────────────┐ ┌─────────────────────────────┐ ║ ║ │ Node 1 │ │ Node 2 │ ║ ║ │ │ │ │ ║ ║ │ ╔═══════════╗ ╔═══════════╗ │ │ ╔═══════════╗ ╔═══════════╗ │ ║ ║ │ ║ Pod CVM ║ ║ Pod CVM ║ │ │ ║ Pod CVM ║ ║ Pod CVM ║ │ ║ ║ │ ╚═══════════╝ ╚═══════════╝ │ │ ╚═══════════╝ ╚═══════════╝ │ ║ ║ │ │ │ │ ║ ║ └──────────────┬──────────────┘ └──────────────┬──────────────┘ ║ ║ │ │ ║ ║ │ raTLS mesh │ ║ ║ └───────────────┬────────────────┘ ║ ║ │ ║ ║ │ raTLS ║ ║ │ ║ ║ ╔════════════════════╗ Attests │ ║ ║ ║ CDS ╠──────────►│ ║ ║ ║ Root of trust ║ │ ║ ║ ╚════════════════════╝ │ ║ ║ ▼ ║ ║ ╔═════════════════════╗ ║ ║ ║ Ingress Router ║ ║ ║ ║ [attested, CVM] ║ ║ ║ ╚══════════╦══════════╝ ║ ╚════════════════════════════════════╪═══════════════════════════════════╝ │ TLS + attestation ┌───────┴───────┐ │ Client │ │ Verifies │ └───────────────┘ Every ╔══╗ box is attested and holds a CDS issued cert. ``` --- ## 5. Component Design The pod is the unit of confidentiality, identity, and attestation in C8s. Every component described in this section is in service of that arrangement, and the CVM boundary around the pod is a deployment-time choice rather than a fixed feature of the architecture. The sections below introduce each component once, in the order a reader needs them to understand how a request reaches an attested pod and leaves again. ### 5.1 Pod-level CVM (default) C8s draws the trust boundary at the pod by default. Each pod boots inside its own confidential VM and carries its own launch measurement. Per-pod lifecycle in this configuration is orchestrated by the **C8s pod runtime**, which builds on Kata Containers [6] machinery for the per-pod VM lifecycle and extends it with a pre-computable measured boot, direct CDS integration, and the signed-workload discipline described below. The worker node itself is no longer the attested unit. It hosts the runtime and proxies pod lifecycle to per-pod CVMs, but carries no workload trust of its own. A node-level boundary is also supported (§5.2) for deployments where it is preferable; §5.5 discusses when to choose each. The same design formula from §4.2 applies per-pod. Each pod runs in hardware-encrypted memory. The pod runtime produces a launch digest that covers the pod's entire boot chain and vCPU initial state, so the digest can be recomputed ahead of time and checked against an allow-list without inspecting the running system. The CDS verifies the per-pod attestation report directly and issues a certificate bound to the pod's ephemeral key. Peers verify that certificate before accepting connections, and egress is encrypted via the raTLS mesh exactly as it is at the node level. The surrounding architecture (CDS, raTLS, attestation-gated key release, client-side encryption) does not change; only the evidence shape does. ``` ┌─ Host: ordinary VM [not attested, control surface only] ─────────────────────┐ │ │ │ ╔═ Pod CVM ═══════════════════════════════════════════════════════════╗ │ │ ║ ║ │ │ ║ ┌─ Workload ──────────────────────────────────────────────┐ ║ │ │ ║ │ ┌──────────────────────┐ ┌──────────────────────┐ │ ║ │ │ ║ │ │ Customer container │ │ Customer container │ │ ║ │ │ ║ │ └──────────────────────┘ └──────────────────────┘ │ ║ │ │ ║ └─────────────────────────────────────────────────────────┘ ║ │ │ ║ ║ │ │ ║ ┌─ Pod services ──────────────────────────────────────────┐ ║ │ │ ║ │ ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │ ║ │ │ ║ │ │ Image │ │ Attestation │ │ raTLS │ │ ║ │ │ ║ │ │ policy │ │ client │ │ endpoint │ │ ║ │ │ ║ │ └──────────┘ └──────────────┘ └──────────────┘ │ ║ │ │ ║ └─────────────────────────────────────────────────────────┘ ║ │ │ ║ ║ │ │ ║ ┌─ Boot chain and hardware ───────────────────────────────┐ ║ │ │ ║ │ Pod runtime boot chain, vCPU initial state │ ║ │ │ ║ │ TEE CPU, assigned GPU │ ║ │ │ ║ └─────────────────────────────────────────────────────────┘ ║ │ │ ║ ║ │ │ ╚═════════════════════════════════════════════════════════════════════╝ │ │ │ └──────────────────────────────────────────────────────────────────────────────┘ ``` #### 5.1.1 Image-gap Mechanisms A per-pod launch digest covers the software that measures at boot, but not the container image, which is pulled from a registry after the pod is already running. This is the same gap that NRI enforcement closes at the node level; when the boundary moves to the pod, the host can no longer observe the pull, so the gating has to move inside. C8s supports three mechanisms, chosen per deployment or combined: 1. **Measured image pinning.** A policy baked into the pod's measured configuration restricts which image digests the pod may pull. Because the policy file is part of the launch digest, attestation transitively proves that the pod can only run the images the policy permits. This gives the strongest hardware guarantee but requires rebuilding the measured configuration each time the allow-list changes, which scales poorly across many short-lived workloads. 2. **Per-customer workload signing.** A customer's public key is baked into the pod's measured configuration, and the customer signs their image digests out-of-band (for example with cosign [20] or notation). The in-pod agent verifies every image pull against the measured key before accepting the image. Attestation then proves that the pod will only run images the customer authorized, without requiring a new measured configuration per workload. This is the default for multi-tenant platform deployments. Customers onboard by registering a key, and from that point on they can deploy signed workloads without further platform involvement. 3. **Encrypted container images.** Images are encrypted at rest, and the decryption key is released through the CDS only to pods whose attestation matches the key's release policy. This is complementary to the two mechanisms above. It protects the image's contents against unauthorized environments rather than proving to a Verifier which image is running, and is the right choice when image confidentiality itself is the concern. Each of the three mechanisms anchors on something that is measured at pod launch, whether an allow-list of digests, a customer signing key, or a CDS decryption key. What differs is where the policy decision lives, which determines operational cost and who is in the loop. ``` Measured pinning Customer signing Encrypted images ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐ │ Image │ │ Image │ │ Image [ciphertext] │ └──────────────────────┘ └──────────────────────┘ └──────────────────────┘ ▲ ▲ ▲ │ Pin │ Verify sig │ Decrypt │ │ │ ╔══════════╪═══════════╗ ╔══════════╪═══════════╗ ╔══════════╪═══════════╗ ║ Pod CVM ║ ║ Pod CVM ║ ║ Pod CVM ║ ║ │ ║ ║ │ ║ ║ │ ║ ║ │ ║ ║ ┌───────┴────────┐ ║ ║ │ ║ ║ │ ║ ║ │ In pod agent │ ║ ║ │ ║ ║ │ ║ ║ └────────────────┘ ║ ║ │ ║ ║ │ ║ ║ ▲ ║ ║ │ ║ ║ │ ║ ║ │ ║ ║ │ ║ ║ ─ ─ ─ ─ ─┼─ ─ ─ ─ ─ ║ ║ ─ ─ ─ ─ ─┼─ ─ ─ ─ ─ ║ ║ ─ ─ ─ ─ ─┼─ ─ ─ ─ ─ ║ ║ ┌───────┴────────┐ ║ ║ ┌───────┴────────┐ ║ ║ ┌───────┴────────┐ ║ ║ │ Policy │ ║ ║ │ Customer key │ ║ ║ │ CDS key │ ║ ║ │ [digest list] │ ║ ║ │ │ ║ ║ │ │ ║ ║ └────────────────┘ ║ ║ └────────────────┘ ║ ║ └────────────────┘ ║ ║ Measured at ║ ║ Measured at ║ ║ Measured at ║ ║ launch ║ ║ launch ║ ║ launch ║ ╚══════════════════════╝ ╚══════════════════════╝ ╚══════════════════════╝ Per digest allow list Per customer key Per pod decryption ``` *Figure: Three image-gap mechanisms. All three anchor on something inside the pod's launch digest; they differ in whether the measured anchor is an allow-list, a signing key, or a decryption key.* ### 5.2 Node-level CVM When the trust boundary is drawn around the worker node rather than the pod, each Kubernetes node runs as a CVM. At boot, the hardware's secure processor computes a launch measurement covering the firmware, kernel, and initial ramdisk. When GPU CC mode is available, the measurement additionally attests the GPU hardware and firmware, producing a combined report. This measurement is the foundation of the attestation chain for every pod that lands on the node. ``` ╔═════════════════════════════ CVM node ═════════════════════════════════╗ ║ ║ ║ ┌─ Customer pods ──────────────────────────────────────────────┐ ║ ║ │ ┌───────┐ ┌───────┐ ┌───────┐ │ ║ ║ │ │ Pod │ │ Pod │ │ Pod │ │ ║ ║ │ └───────┘ └───────┘ └───────┘ │ ║ ║ └──────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ ┌─ K8s node runtime ───────────────────────────────────────────┐ ║ ║ │ ┌─────────────┐ ┌────────────────────┐ │ ║ ║ │ │ kubelet │ │ containerd [CRI] │ │ ║ ║ │ └─────────────┘ └────────────────────┘ │ ║ ║ └──────────────────────────────────────────────────────────────┘ ║ ║ ┌─ C8s services ───────────────────────────────────────────────┐ ║ ║ │ ┌─────────────┐ ┌────────────┐ ┌──────────────────┐ │ ║ ║ │ │ Attestation │ │ NRI image │ │ raTLS mesh proxy │ │ ║ ║ │ │ service │ │ policy │ │ │ │ ║ ║ │ └─────────────┘ └────────────┘ └──────────────────┘ │ ║ ║ └──────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ ┌─ Identities ─────────────────────────────────────────────────┐ ║ ║ │ ┌──────────────┐ ┌──────────────────┐ │ ║ ║ │ │ Node cert │ │ Per pod certs │ │ ║ ║ │ └──────────────┘ └──────────────────┘ │ ║ ║ └──────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ ┌─ Boot chain and hardware ────────────────────────────────────┐ ║ ║ │ Firmware, kernel, initrd, GPU driver │ ║ ║ │ TEE CPU, optional TEE GPU │ ║ ║ └──────────────────────────────────────────────────────────────┘ ║ ║ ║ ╚════════════════════════════════════════════════════════════════════════╝ ``` Applying the design formula from §4.2: 1. The node runs as a CVM; the TEE encrypts VM memory with keys the hypervisor never possesses. 2. The secure processor computes a launch measurement at boot. 3. The CDS verifies the node's attestation report and issues a node-scoped credential. For every pod that subsequently launches on the node, the CDS issues a per-pod certificate based on composed evidence (the node's TEE report plus the pod's workload digest, signed by the node's attestation service), so each pod still holds its own attestation-rooted identity. 4. Other components verify the pod's attestation-rooted identity before accepting connections. 5. Traffic leaving any pod is encrypted via the raTLS mesh. All workloads running inside the CVM inherit its memory encryption. Containers within a single CVM share the node's hardware protection boundary and are separated by standard Linux kernel isolation mechanisms (namespaces, cgroups, seccomp). This is process-level isolation, not hardware isolation. Pods on the same CVM node can access each other's memory through a kernel exploit or misconfigured security context, just as they could on a standard node. For deployments requiring hard multi-tenancy, where different customers' workloads must be hardware-isolated from one another, each tenant requires dedicated CVM nodes. The C8s architecture supports this naturally. New CVM nodes are provisioned on demand through standard Kubernetes autoscaling, and each boots through the same attestation flow. Sovereign-cloud and high-compliance deployments should treat node-level isolation as the tenancy boundary and schedule at most one tenant per CVM node. A tighter hardware boundary than the node may be needed for per-pod hardware isolation, per-pod attestation independent of node posture, or compatibility with managed Kubernetes clusters that do not expose a confidential node pool. In that case the same design formula is applied one level down, with each pod running as its own confidential VM (§5.1). §5.5 discusses when to choose each. ### 5.3 Deployment Topologies On a self-hosted confidential host, per-pod CVMs are launched directly by the pod runtime on the node. Each pod gets its own measured VM; the host is responsible only for scheduling and for providing the runtime. On a managed Kubernetes cluster that does not expose a confidential node pool, per-pod CVMs are provisioned through a peer-pod path. The pod runtime delegates VM lifecycle to the cloud provider's API and each pod becomes a dedicated confidential cloud VM (for example Azure Confidential VM, AWS Nitro-based SNP instances, or GCP Confidential VM). The cluster's Kubernetes nodes remain ordinary VMs whose role is reduced to control surface (scheduling, networking, and proxying pod lifecycle calls to the cloud), which is what makes this path compatible with managed services that have no CVM node pool at all. The trade-off is that every pod launch now includes cloud VM provisioning on the cold path, which adds latency and bills per VM rather than per container; warm pools amortize this at the cost of idle capacity. Hybrid clusters, in which some workloads use node-level CVMs and others use per-pod CVMs on the same control plane, are supported. The CDS accepts both evidence shapes. ### 5.4 GPU Attachment GPU CC mode binds a GPU's protected state to a single confidential guest. When the boundary is at the pod, any GPU the pod uses must be attached to the pod's CVM at launch, and it cannot be multiplexed across pods while that pod is running. Per-pod deployments therefore allocate whole GPUs (or whole GPU partitions, where the hardware supports partitioning) per pod. This is a real constraint for GPU-dense workloads. When the boundary is at the node, the GPU attaches to the node's CVM and several pods on that node can share GPU capacity, all running inside the node's trust boundary. The same sharing is not available at the pod level, because the sharing mechanisms assume multiple tenants inside the GPU's single attested context, and per-pod CVMs each want their own attested context. For workload pools that rely on cross-pod GPU sharing to reach high utilization, this consideration alone often decides where the boundary should sit. ``` Node level Pod level ╔═ Node [CVM] ════════════════════╗ ╔═ Pod [CVM] ═══════════════════╗ ║ ║ ║ ║ ║ ║ ║ ┌─────────┐ ┌─────────┐ ║ ║ ┌─────┐ ┌─────┐ ┌─────┐ ║ ║ │ Pod ├───►│ GPU │ ║ ║ │ Pod │ │ Pod │ │ Pod │ ║ ║ └─────────┘ └─────────┘ ║ ║ └──┬──┘ └──┬──┘ └──┬──┘ ║ ║ ║ ║ │ │ │ ║ ╚═══════════════════════════════╝ ║ │ │ │ ║ ║ └─────────┼─────────┘ ║ ╔═ Pod [CVM] ═══════════════════╗ ║ │ ║ ║ ║ ║ ▼ ║ ║ ┌─────────┐ ┌─────────┐ ║ ║ ┌───────────────┐ ║ ║ │ Pod ├───►│ GPU │ ║ ║ │ GPU │ ║ ║ └─────────┘ └─────────┘ ║ ║ └───────────────┘ ║ ║ ║ ║ ║ ╚═══════════════════════════════╝ ║ ║ ╚═════════════════════════════════╝ ╔═ Pod [CVM] ═══════════════════╗ ║ ║ ║ ┌─────────┐ ┌─────────┐ ║ ║ │ Pod ├───►│ GPU │ ║ ║ └─────────┘ └─────────┘ ║ ║ ║ ╚═══════════════════════════════╝ Shared GPU across pods Whole GPU per pod CVM ``` *Figure: GPU attachment comparison. Node-level deployments can time-slice or partition a single attached GPU across pods; pod-level deployments dedicate a whole GPU (or whole partition) to each pod's CVM because CC mode binds the protected context to one guest.* ### 5.5 Choosing Where to Draw the Boundary The choice between a node-level and a pod-level boundary is a deployment-time configuration, not two separate products. The same CDS, the same raTLS mesh, the same key-release machinery, and the same client-side encryption apply in either case; what changes is the granularity of hardware isolation, the shape of attestation evidence, and a handful of operational consequences that follow from where the CVM actually lives. In practice the pod-level boundary is the C8s default and the appropriate choice for per-pod tenancy, for managed Kubernetes deployments that do not offer a confidential node pool, and for workloads that need per-pod attestation independent of the node's posture. The node-level boundary remains attractive for single-tenant high-throughput pools (inference, training, batch analytics) where per-pod hardware isolation is unnecessary and sharing GPU capacity across pods on the same node matters for utilization. The second-order consequence is worth stating explicitly. At the pod level, the worker node stops being a confidential VM in any meaningful sense and becomes a control surface that proxies pod lifecycle to per-pod CVMs. That shift reshapes what a "node" is operationally. Peer-pod and virtual-kubelet-style topologies become natural, node-level GPU sharing becomes unavailable, and per-pod VM provisioning latency becomes a real cost to budget for. The following side-by-side shows where each piece of the platform sits under the two configurations. The node-level view puts the CVM wall around the node; the pod-level view collapses the per-node DaemonSets into each pod's own measured configuration and reduces the host to a control surface. ``` Node level Pod level ╔═ Worker Node [CVM] ═══════════════════════════╗ ┌─ Host [VM] ──────────────────────────┐ ║ ║ │ │ ║ ┌───────────────┐ ║ │ ┌───────────────┐ │ ║ │ Kubelet │ ║ │ │ Kubelet │ │ ║ └───────┬───────┘ ║ │ └───────┬───────┘ │ ║ ▼ ║ │ ▼ │ ║ ┌───────────────┐ ║ │ ┌───────────────┐ │ ║ │ Containerd │ ║ │ │ Containerd │ │ ║ │ [with NRI] │ ║ │ │ [with Kata] │ │ ║ └───────┬───────┘ ║ │ └───────┬───────┘ │ ║ ┌─────────┴─────────┐ ║ │ ▼ │ ║ ▼ ▼ ║ │ ╔═ Pod [CVM] ═══════════════════╗ │ ║ ┌───────────────┐ ┌───────────────┐ ║ │ ║ ║ │ ║ │ Pod │ │ Pod │ ║ │ ║ ┌───────────────────────┐ ║ │ ║ │ ┌───────────┐ │ │ ┌───────────┐ │ ║ │ ║ │ Image policy │ ║ │ ║ │ │ Container │ │ │ │ Container │ │ ║ │ ║ │ verifier │ ║ │ ║ │ └───────────┘ │ │ └───────────┘ │ ║ │ ║ └───────────────────────┘ ║ │ ║ └───────┬───────┘ └───────┬───────┘ ║ │ ║ ┌───────────────────────┐ ║ │ ║ └─────────┬─────────┘ ║ │ ║ │ Attestation client │ ║ │ ║ ▼ ║ │ ║ └───────────────────────┘ ║ │ ║ ┌─────────────────────────────┐ ║ │ ║ ┌───────────────────────┐ ║ │ ║ │ Shared GPU │ ║ │ ║ │ raTLS endpoint │ ║ │ ║ │ [CC mode] │ ║ │ ║ └───────────────────────┘ ║ │ ║ └─────────────────────────────┘ ║ │ ║ ║ │ ║ ║ │ ║ ┌───────────────────────┐ ║ │ ║ ┌───────────────┐ ║ │ ║ │ Container │ ║ │ ║ │ Mesh proxy │ ║ │ ║ └───────────────────────┘ ║ │ ║ │ [DaemonSet] │ ║ │ ║ ║ │ ║ └───────────────┘ ║ │ ╚══════════════╪════════════════╝ │ ║ ║ │ ▼ │ ║ ┌───────────────┐ ║ │ ┌───────────┐ │ ║ │ Attest agent │ ║ │ │ GPU [CC] │ │ ║ │ [DaemonSet] │ ║ │ └───────────┘ │ ║ └───────────────┘ ║ │ │ ║ ║ └──────────────────────────────────────┘ ╚═══════════════════════════════════════════════╝ ``` *Figure: Node-level vs. pod-level boundary. Same CDS, same mesh, same key-release machinery on both sides; what moves is where the CVM wall sits and which components end up inside it.* The trade-offs in summary: | Axis | Node-level boundary | Pod-level boundary | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Isolation boundary | Node. Pods on the same node share kernel-level isolation. | Pod. Each pod is its own confidential VM with hardware-isolated memory. | | Attestation granularity | Composed: node attestation plus per-pod workload digest. | Direct: per-pod attestation bound to the pod's launch digest. | | Image and workload gating | Host-side NRI check against a signed allow-list before containers start. | In-pod measured policy, per-customer workload signing, or both, enforced by the in-pod agent before the image pull completes. | | Container-image proof | Image digest checked against the signed manifest at launch, on the host. | Image is not in the launch digest; proof comes from the in-pod signing or policy check, which itself is in the digest. | | Managed Kubernetes compatibility | Works directly on confidential node pools where the cloud offers them (AKS, EKS, GKE with confidential nodes). | Works anywhere, via peer-pod-style provisioning of confidential cloud VMs per pod, including clusters whose nodes have no TEE support. | | Startup latency | Pod scheduling plus container pull, on the order of seconds. | Bare-metal per-pod CVMs add a VM boot to the pod's cold path; peer-pod deployments additionally incur cloud VM provisioning (tens of seconds) per pod, which warm pools can amortize at the cost of idle capacity. | | GPU attachment | GPU attaches to the node's CVM and is shared across pods on that node through standard GPU-sharing mechanisms (time-slicing, MPS, MIG) inside the node's trust boundary. | GPUs are assigned to a pod's CVM at launch and cannot be shared across pods while the pod runs, because GPU CC mode binds the protected context to a single guest. Whole GPUs or whole GPU partitions are allocated per pod. | | Per-pod resource overhead | Low. Pods run as regular processes and share the node's guest kernel and agent stack. | Meaningful. Each pod carries its own guest kernel, in-pod agent, attestation agent, and policy enforcement, typically on the order of a few hundred megabytes of memory before the workload itself. | | Tenancy model | Hard multi-tenancy requires dedicated nodes per tenant. Within a single tenant, pods share the node's trust boundary. | Hard per-pod isolation by construction. Natural fit for multi-tenant clusters without dedicated tenant nodes. | | Operational complexity | Cluster looks like standard Kubernetes with a confidential node image, a signed image policy manifest, and the mesh DaemonSets. An ops team already running Kubernetes has to add one node image and one allow-list. | Cluster needs per-pod CVM lifecycle management (measured configurations per workload class or per customer), a signed-workload enrollment flow, and, on managed Kubernetes, the peer-pod provisioning path with its own cloud-API integration, quotas, and failure modes. Operationally this is closer to running a per-pod VM fleet than a Kubernetes pod fleet. | | Worker-node role | The worker node is a confidential VM and carries real workload trust. | The worker node is a control surface and holds no workload trust, because the workload's TEE is the pod itself. In the limit, the node can be a plain VM. | Mixing boundaries within a single cluster is supported, so node-level CVMs for throughput-bound pools and pod-level CVMs for tenant-isolated workloads can coexist under the same CDS. Further refinement (pod-level raTLS endpoints, per-workload policy caching, and the roll-out of the per-customer signing model) is discussed in §10. ### 5.6 Certificate Distribution Service (CDS) The CDS is the Verifier and the root of the certificate chain within the trust boundary. It sits in the RATS *background-check* role [15]: pods produce Evidence, the CDS appraises it, and mesh peers act as Relying Parties that trust the CDS's prior appraisal via the issued certificate. It runs as a set of attested pods and performs four functions: 1. **Attestation verification.** When an unattested pod requests attestation, the CDS issues it a challenge nonce and the Attester responds with an attestation report. The CDS verifies that: (a) the endorsement key chains to the hardware manufacturer's root of trust, (b) the report signature is valid, (c) the launch measurement matches a reference value in the allow-list, (d) the Trusted Computing Base (TCB) version meets minimum requirements, and (e) the signed attestation nonce matches the expected value. 2. **Certificate issuance.** Upon successful appraisal, the CDS issues a raTLS mesh certificate to the pod for use in inter-pod traffic. The CDS does not issue externally facing certificates; the ingress router uses a public-CA-issued certificate bound to a TEE-internal key, paired with a CDS-countersigned freshness beacon that clients verify (see §5.10.1 and §5.9.2). 3. **Manifest signing.** The CDS signs the image policy manifest consumed by the image policy enforcement layer (§5.7); the manifest carries the reference values appraisal uses. Relying Parties verify it via standard certificate chain validation. 4. **Key brokering.** The CDS releases attestation-gated secrets to attested pods whose attestation reports match per-key release policy. The full release flow is described in §5.6.4. These responsibilities are summarized in the following figure; each subsequent subsection drills into deployment, state management, and the minimal-key property. ``` ┌──────────────────┐ │ Operator │ └────────┬─────────┘ │ Manual attestation [bootstrap] ▼ ╔═ CDS [attested pods] ══════════════╗ ║ ║ ║ ┌──────────────────────────┐ ║ ╔══════════════╗ ║ │ Attestation verification ├─────╫─── Verifies attestation ────────►║ ║ ║ └──────────────────────────┘ ║ ║ ║ ║ ║ ║ ║ ║ ┌──────────────────────────┐ ║ ║ ║ ║ │ Certificate issuance ├─────╫───────── Issues certs ──────────►║ ║ ║ └──────────────────────────┘ ║ ║ Pod ║ ║ ║ ║ [attested] ║ ║ ┌──────────────────────────┐ ║ ║ ║ ║ │ Manifest signing ├─────╫─────── Signed allowlist ────────►║ ║ ║ └──────────────────────────┘ ║ ║ ║ ║ ║ ║ ║ ║ ┌──────────────────────────┐ ║ ║ ║ ║ │ Key brokering ├─────╫────── Workload specific keys ───►║ ║ ║ └──────────────────────────┘ ║ ╚══════════════╝ ║ ║ ╚════════════════════════════════════╝ ``` *Figure: CDS responsibilities. The CDS is itself a CVM manually attested by the operator at bootstrap; from there, every outgoing channel is anchored in an attestation decision.* #### 5.6.1 Deployment Model The CDS is deployed as an active/active pair so it does not become a single point of failure. Each replica runs as an attested pod with the same measurement and serves traffic independently. A CA-style signing operation is idempotent given the same inputs (the operator-signed allow-list and an incoming attestation report), so no consensus protocol is needed between replicas. All replicas share the same CDS measurement, signing key, and identity, so the operator's one-time bootstrap covers the set rather than each replica individually. The signing key is generated inside the first replica's CVM and never leaves CVM-encrypted memory in plaintext. When a new replica starts, it presents an attestation report binding a CVM-resident public key to the CDS measurement; an existing replica verifies the report against its own measurement and wraps the signing key to that public key (the same primitive used for application secrets in §5.6.4). The control plane handles only ciphertext wrapped to a key it cannot reach. If every replica becomes unavailable simultaneously, the in-memory signing key is lost; recovery is a re-bootstrap, in which the operator re-attests a fresh CDS, a new signing key is generated, and mesh certificates reissue under the new key as pods re-attest. The allow-list is operator-signed, so each replica independently verifies the signature on every update, and updates are pushed to every replica. The CDS exposes an attested endpoint that returns the currently active allow-list and the immediately previous version. Because the endpoint is served by the measured CDS image inside its CVM, a verifier reading from this endpoint receives the allow-list state the CDS is actually enforcing rather than what the operator claims it is. This narrows the gap between policy-as-deployed and policy-as-claimed without requiring the operator's trust. A fuller append-only history with signed consistency roots over the entire allow-list sequence is treated as future work in §10. Kubernetes ConfigMaps and Secrets, including those managed by SealedSecrets controllers, External Secrets Operator, or similar tooling, are unsuitable for CDS state because the kube-apiserver always has access to plaintext. These mechanisms protect against certain attacks (stolen etcd disks, leaked Git repositories) but do not protect against a compromised control plane, which is explicitly part of our threat model. Confidentiality against the control plane requires that decryption happen inside TEE-encrypted memory, with keys derived from hardware attestation, so that no Kubernetes component ever handles plaintext. Other infrastructure services that exist in node-level deployments (the pod attestation agent, the raTLS mesh interceptors, and the NRI image policy enforcers) run as DaemonSets and attest to the CDS on startup like any other pod, presenting composed evidence (the node's TEE report plus their own workload measurement) and receiving per-pod certificates through the standard automated flow. Where the boundary is drawn at the pod (the default), there are no DaemonSet attestation agents to bootstrap. Each pod is its own CVM and presents its own TEE report directly to the CDS at start, and the equivalent gating relocates into the pod's measured configuration (§5.1). #### 5.6.2 Bootstrapping and Trust Anchoring The CDS is the first component to start in a fresh cluster, because every other attested pod needs the CDS to issue its certificate. Someone has to be trusted first; in C8s that someone is the operator, and the decision is bounded to a one-time manual attestation event that proceeds as follows: 1. The operator installs the CDS via its Helm chart or equivalent. 2. Kubernetes schedules the CDS replicas, each as an attested pod running the CDS's measured image. 3. The operator retrieves attestation evidence from one replica and verifies it against the hardware manufacturer's root of trust. This verification establishes that the CDS measurement is running on genuine hardware, and that the measurement matches the expected CDS image digest. 4. Key material for management of the allow-list is bootstrapped by the cluster owner to ensure integrity of the allow-list and thus images permitted by image-policy enforcement. 5. The operator signs the initial allow-list and uploads it to the CDS. 6. Because every CDS replica is an attested pod with the same measurement, the operator's verification applies to all replicas at this measurement. Replacing the CDS image (e.g., upgrading versions) produces a new measurement and requires re-attestation. This trust anchoring is the architecture's one human-in-the-loop step. Every subsequent attestation in the cluster chains to the operator's verification of the CDS measurement. ``` Operator Hardware CDS replicas │ │ │ │ Install chart │ │ ├──────────────────────────────────────────►│ │ [image digest] │ │ │ │ │ │ │ Launch replicas │ │ │ [measure each boot] │ │ ├──────────────────────►│ │ │ │ │ Request evidence from one replica │ ├┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈►│ │ │ │ │ Signed report │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ ┌──────────────────────────────────┐ │ │ │ Verify report │ │ │ │ [1] Signature chains to │ │ │ │ hardware vendor root │ │ │ │ [2] Launch digest matches │ │ │ │ expected CDS image │ │ │ └──────────────────────────────────┘ │ │ │ │ Send allow list public key & signed list │ ├┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈►│ │ │ ``` *Figure: CDS bootstrapping. The operator attests the CDS measurement once; this covers every replica at that measurement, present and future.* **Consequences of unavailability.** If the CDS becomes unavailable, existing certificates keep working until they expire. New attested pods cannot obtain certificates, so new pods cannot join the mesh. Running workloads are unaffected for the lifetime of their existing certificates. This bounds the blast radius of CDS downtime to "can't onboard new pods" rather than "cluster goes dark." #### 5.6.3 Key Minimization An important property of the CDS is what it does not hold. The CDS possesses its signing key, which it uses to issue certificates and to sign freshness beacons (§5.9.2). It does not hold the raTLS session keys that protect data in transit, the memory-encryption keys used by any CVM, or any application-layer decryption key outside the release flow described in §5.6.4. Those keys live inside the boundary of CVMs and never leave them. Session keys are ephemeral and negotiated per connection by the endpoints. CVM memory keys never leave the hardware secure processor. Brokered application secrets are forwarded to the attested pod that requested them and held only in that pod's hardware-encrypted memory. This scoping narrows the blast radius of a CDS compromise. An adversary who obtains the CDS signing key can issue fraudulent certificates going forward and, at worst, impersonate attested workloads to peers that accept CDS-issued identity. They cannot decrypt past traffic retroactively, break in-flight sessions protected by ephemeral keys, or read CVM memory whose keys the CDS has never possessed. The CDS's core responsibility is a certificate authority and attestation-gated broker. Application-layer keys flow through the brokering protocol in §5.6.4, which supports two modes. In neither mode does the CDS hold application keys at rest. The recommended wrapped-brokering mode has the CDS handle only ciphertext, never seeing plaintext keys at all. The fallback direct-brokering mode has the CDS proxy plaintext keys from a customer KMS to attested pods, with each key exposed only in the CDS's CVM memory for the duration of its proxy hop. A CDS compromise active during a direct-mode hop can read the key in flight, and an adversary persisting in the CDS will see every subsequent direct-mode key until detection. No keys are retroactively recoverable from past completed hops. **Certificate rotation is re-attestation.** One further implication follows from this scoping. Because the CDS has no long-lived secret corresponding to any pod's mesh identity beyond the outstanding certificate itself, rotating a pod's certificate is inseparable from re-attesting the pod. The CDS re-verifies the pod's current attestation report, re-checks the measurement against the allow-list, and re-checks any key-binding fields before issuing a replacement cert. Renewal fails if the pod's TCB has fallen below the minimum or its measurement has been revoked, and the pod drops out of the mesh at the old certificate's expiry. Expiry plus re-attestation is the revocation mechanism; there is no separate revocation channel to operate or to fail. §5.8.2 details how short-lived certificates and key-digest binding make this work without per-handshake attestation overhead. #### 5.6.4 Attestation-gated Key Brokering Beyond certificate issuance, the CDS is the broker for application-layer secrets whose release should be tied to a specific attested measurement. The pattern fits any artifact whose release policy can be expressed in terms of measurement, workload identity, and per-key release rules: encrypted datasets, signing keys for downstream services, decryption keys for proprietary code, and model weights are all instances of the same shape. C8s supports two brokering modes. Wrapped brokering is the default and recommended mode and is described first. Direct brokering is a fallback for customers whose key-management workflow cannot supply a deposit service. **Release decision.** Both modes share the same release decision. When an attested workload requests a secret, the CDS evaluates three conditions before authorizing release. 1. **Substrate attestation.** The CVM's launch measurement is on the allow-list, and the firmware or TCB version meets the minimum. 2. **Workload authorization.** The container image digest matches the signed policy manifest (enforced by the NRI image policy enforcer at the node, or by the in-pod measured policy at the pod). 3. **Per-key policy.** The specific secret is authorized for release to this measurement-and-digest combination. Only when all three are satisfied does the CDS proceed with release. What "release" means differs between the two modes. **Wrapped brokering (default).** The customer runs a key deposit service that holds the plaintext secret. At pod startup, the attested pod generates an ephemeral keypair inside its CVM and presents an attestation report binding its public key to the measurement, using the same key-binding mechanism §5.8.2 uses for raTLS certificates. The CDS verifies the three release conditions above and forwards the request, together with the pod's attestation report and bound public key, to the customer's deposit service. The deposit service independently verifies the attestation against the hardware vendor's root of trust, wraps the secret to the pod's public key (for example with HPKE [16]), and returns the ciphertext. The CDS forwards the ciphertext to the pod over the existing raTLS connection. The pod unwraps the secret inside its CVM-encrypted memory. In this mode, the CDS handles ciphertext only. It never holds the plaintext secret, and a CDS compromise does not yield secret plaintext. The customer remains in the loop on every release and can deny releases their deposit service does not approve. Two onboarding paths sit on top of this mode, illustrated here for the canonical case of encrypted model weights. In the *platform-managed* path, the artifact owner uploads plaintext and the platform encrypts on ingest, leaving the owner's workflow unchanged. In the *provider-managed* path, the artifact owner encrypts before upload and operates the deposit service themselves, retaining direct control over both the encryption and the release decision. The same two paths apply to other artifact classes (datasets, code, downstream credentials). **Direct brokering.** Where a customer cannot supply a deposit service that wraps keys to per-pod public keys, the CDS instead acts as an attested proxy to the customer's existing KMS. The CDS holds no keys at rest in this mode. On a matching request, the CDS authenticates to the customer's KMS by presenting its own attestation report; the customer's KMS attests the CDS against the hardware vendor's root of trust and releases the plaintext key over an authenticated channel. The CDS forwards the plaintext key to the requesting pod over the raTLS connection. The key transits the CDS's CVM-encrypted memory only for the duration of the proxy hop and is never persisted within the CDS. This mode is appropriate when the customer's KMS supports "release to an authenticated client" but not "wrap to an arbitrary external public key," which is the common case for cloud-native KMS products. The trade-off relative to wrapped brokering is exposure during the proxy hop. In either mode, the released material reaches only an attested pod over a raTLS connection, never persists outside CVM-encrypted memory, and is rotated by re-attestation rather than by a separate revocation channel. ``` Workload Pod [CVM] CDS Storage [encrypted at rest] │ │ │ │ Request secret │ │ │ [attestation report, │ │ │ workload digest, │ │ │ secret ID] │ │ ├───────────────────────►│ │ │ │ │ │ │ ┌──────────────────────┐ │ │ │ │ Verify 3 conditions │ │ │ │ │ [1] Substrate attest │ │ │ │ │ [2] Workload digest │ │ │ │ │ [3] Release policy │ │ │ │ └──────────────────────┘ │ │ │ │ │ Release secret │ │ │ [to CVM memory only] │ │ │◄───────────────────────┤ │ │ │ │ │ Deny [fail closed] │ │ │┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ │ GET artifact │ │ ├───────────────────────────────────────────────────►│ │ │ │ │ Ciphertext blob │ │ │ [AES 256 GCM] │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ Decrypt in CVM memory │ │ │ [never persisted] │ │ │ │ │ ``` *Figure: Attestation-gated key delivery. The CDS releases a secret only when all three release conditions match; the attested pod fetches ciphertext from untrusted storage and decrypts it exclusively inside CVM-encrypted memory.* **Multi-key isolation.** Each secret has its own release policy, and the CDS ties release to specific workload measurements rather than to which node or pod hosts the workload. Two providers' secrets can coexist on shared physical hardware. Provider A's secret is never released to provider B's workload, even on the same node, because the workload digests differ. **Fit and limits.** The CDS is deliberately scoped to secrets whose release can be tied to an attested measurement. Long-lived operational credentials and general-purpose application secrets are a poor fit. For those, C8s integrates with an external secrets manager such as HashiCorp Vault [12] through a proxy layer that sits inside the trust boundary: pods authenticate to the proxy with their per-pod CDS-issued identity, and the proxy brokers access to the external store. The CDS itself stays scoped to attestation-gated secrets; general secret management lives in the system organizations already use. ``` ╔══════════════════════════════╗ ║ Attestation gated proxy ║ ║ ║ ╔══════════════╗ ║ ┌────────────────────────┐ ║ ║ ║────►║ │ Cert verify │ ║ ┌──────────────────┐ ║ Attested ║ ║ └────────────────────────┘ ║ │ │ ║ workload ║ ║ ┌────────────────────────┐ ║─── Token ────────►│ │ ║ ║ ║ │ Policy check │ ║ │ External vault │ ║ ║◄────║ └────────────────────────┘ ║◄── Secret ────────┤ │ ║ ║ ║ ┌────────────────────────┐ ║ │ │ ║ ║ ║ │ Vault token mint │ ║ │ │ ╚══════════════╝ ║ └────────────────────────┘ ║ └──────────────────┘ ╚══════════════════════════════╝ ``` *Figure: Vault integration. The attestation-gated proxy sits inside the trust boundary; the external vault never sees a workload directly, only the proxy's identity after attestation and policy pass.* ### 5.7 NRI Image Policy Enforcer In standard Kubernetes, the control plane dictates what runs on each node. The scheduler selects a node, the API server instructs the kubelet, and the kubelet directs the container runtime to launch the container. The node has no independent authority to refuse. This is incompatible with a trust model that excludes the control plane. A compromised or misconfigured control plane could schedule a malicious sidecar alongside the workload, one that reads decrypted secrets from shared memory or exfiltrates query data. The NRI image policy enforcer addresses this by intercepting every container launch request at the node level. The enforcer operates as an NRI plugin, receiving container creation events from the container runtime. For each event, it extracts the image digest and checks it against a signed policy manifest. If the digest does not appear in the manifest, the container launch is rejected. The policy manifest is signed by the CDS and carries the reference values the enforcer compares against. Digests in the manifest correspond to images built by Kettle, the attested build system, which acts as the Reference Value Provider for image identity; the operator is the Reference Value Provider for which of those digests are authorized to run. The chain of trust runs as follows. Kettle attests the build, the output digest is recorded in the operator-signed allow-list, the CDS signs the resulting manifest, and the NRI enforcer checks incoming containers against it. The control plane is excluded from this chain entirely. The following table summarizes the attacks a compromised control plane can attempt and the mechanism that prevents each: | Attack | Mitigation | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Schedule malicious sidecar | NRI enforcer rejects: digest not in signed policy | | Leak secrets via environment variables | Application-layer secrets are released by the CDS to attested pods and held only in hardware-encrypted memory, never in Kubernetes Secrets or environment variables | | Inject a malicious node into the cluster | Node must attest to CDS; attacker cannot produce valid attestation report | | Read workload traffic on the pod network | All traffic is raTLS-encrypted; control plane has no valid mesh identity | | Man-in-the-middle internal traffic | raTLS requires CDS-issued certificates; attacker cannot obtain valid credentials | The enforcement path is shown below. The NRI hook interposes between containerd and the running container, so a rejection is indistinguishable from "container never started" as far as the control plane is concerned. There is no partial start, no ephemeral running state, and no opportunity to leak data before the policy check completes. ``` Control Plane Containerd NRI Shim Policy Manifest [untrusted] [C8s plugin] [CDS signed, local] │ │ │ │ │ │ │ │ │ ┌────────────────────────────┐ │ │ │ │ 1. CreateContainer │ │ │ │ │ [Image digest X] │ │ │ │ └──────────────┬─────────────┘ │ │ ├────────────────►│ │ │ │ │ │ │ │ │ ┌────────────────────────────┐ │ │ │ │ 2. Intercept pre start │ │ │ │ │ [NRI hook] │ │ │ │ └──────────────┬─────────────┘ │ │ ├────────────────►│ │ │ │ │ │ │ │ │ 3. Lookup digest X │ │ │ ├──────────────────────►│ │ │ │ │ │ │ │ 4. Allow list hit? │ │ │ │◄──────────────────────┤ │ │ │ │ │ │ │ │ │ │ 5a. Approved │ │ │ │◄────────────────┤ │ │ │ │ │ │ │ ┌────────────────────────────┐ │ │ │ │ Container starts │ │ │ │ └────────────────────────────┘ │ │ │ │ │ │ │ │ │ │ │ 5b. Rejected │ │ │ │◄────────────────┤ │ │ │ │ │ │ │ ┌──────────────────────────────────┐ │ │ │ │ Container does not start, │ │ │ │ │ regardless of control plane │ │ │ │ │ request [fail closed] │ │ │ │ └──────────────────────────────────┘ │ │ │ │ │ ``` *Figure: Container startup under NRI enforcement. The policy manifest is local and CDS-signed, so the rejection decision is independent of any live control-plane interaction.* The NRI enforcer as described here operates at the node, which is appropriate when the node is the CVM and the host can see every container before it starts. When the trust boundary is drawn at the pod, the host cannot observe the post-boot container pull at all, and the same gating role relocates into the pod's measured configuration. A policy file or a customer key is baked into the launch digest and enforced by the in-pod agent before the image is allowed to load. The chain of trust (Kettle-attested build → signed manifest or signed workload → measured enforcement) is preserved; only the enforcement point moves. ### 5.8 raTLS Mesh Pod-to-pod traffic in Kubernetes flows through the cluster network. The CNI plugin handles routing, and traffic may traverse virtual switches, physical infrastructure, or cloud networking layers outside the operator's control. By default, this traffic is plaintext. The network cannot be placed inside the trust boundary because the operator does not control the CNI, and network infrastructure cannot be attested. The architecture therefore applies the encrypt-before-exposure strategy via a remote attestation TLS (raTLS) mesh. The pattern will be familiar to readers acquainted with service meshes such as Istio [18], where each workload receives a SPIFFE [19] identity and a proxy terminates mTLS on its behalf. The raTLS mesh applies the same pattern with one substitution. Identity is rooted in hardware attestation via the CDS rather than in a Kubernetes-derived identity such as a service account. The certificates change meaning, not the mechanics. A certificate in a standard service mesh proves that a trusted CA issued it to a workload with a particular name. A raTLS certificate additionally proves that the workload launched on hardware-attested substrate and that its measured code matches policy. Where the trust boundary is drawn at the node, per-pod mesh identity is produced by composed attestation. The node boots as a CVM and presents its TEE report to the CDS, receiving a node-scoped credential rooted in the hardware manufacturer chain. An attestation service (a DaemonSet whose own identity chains through the node's attestation) runs inside the CVM. When a new pod launches, the NRI path verifies its image digest against the signed policy manifest, and the attestation service produces composed evidence covering the node's TEE report, the pod's workload digest, the pod's identity (namespace, pod UID), and the pod-generated ephemeral public key, all signed by the attestation service's CDS-issued key. The pod presents this evidence to the CDS, which verifies the hardware chain, the service's signature, and the workload digest against policy, then issues a per-pod certificate bound to the pod's ephemeral key. Where the boundary is drawn at the pod, the mechanism is simpler. The pod's own TEE report directly measures its boot state, and the pod presents that hardware report to the CDS without an intermediate agent. The CDS interface is unchanged. It receives attestation evidence, verifies the chain, checks the measurement against policy, and issues a per-pod certificate. The mesh, the certificates it issues, and the semantic guarantee read identically from the network's point of view. Once a per-pod certificate is issued, it needs to reach the mesh proxy that will present it on the pod's behalf, and the pod's outbound traffic needs to reach that proxy. An init container in each pod handles both concerns. It requests the pod's certificate from the CDS at start, hands the certificate to the node-level mesh proxy, and arranges that the pod's TCP traffic is transparently redirected through the mesh. The workload itself is unchanged. It sends plaintext to pod-IP destinations exactly as it would on any Kubernetes cluster, and the mesh handles interception, identity presentation, and encryption. Nothing in the application code has to know about attestation, certificates, or confidential compute. The mesh terminates at the pod, not the node. Every raTLS certificate corresponds to one pod whose evidence verified against policy at issuance time. Inter-pod traffic is intercepted at pod boundaries and encrypted end-to-end between attested pod endpoints. Pods that share a node still talk to each other over the mesh, and there is no in-node plaintext hop. The semantic guarantee of a raTLS connection is stronger than standard mTLS. Rather than proving only that a trusted CA issued the certificate, it proves that the CDS issued this certificate to a specific pod whose workload digest matched policy, launched on a hardware-attested CVM substrate. Both ends of every internal connection hold attestation-rooted identity at pod granularity. The underlying network infrastructure sees only ciphertext between verified pod peers. An attacker who compromises the network can observe encrypted traffic but cannot participate in it, inject traffic, or decrypt it. The following sequence shows a single pod-to-pod call under a node-level configuration with a shared mesh proxy. The workload code sends plaintext to a pod IP; interception, identity presentation, and raTLS happen at the mesh boundary. Under a pod-level configuration the mesh proxy lives inside each pod's CVM and the two hops collapse, but the on-the-wire guarantee is the same. ``` Pod A Mesh1 Mesh2 Pod B │ │ │ │ │ Plaintext TCP │ │ │ │ [to Pod B IP] │ │ │ ├──────────────────►│ │ │ │ │ │ │ │ ┌───────────────┴───────────────┐ │ │ │ │ Mesh selects Pod A │ │ │ │ │ per pod cert │ │ │ │ └───────────────┬───────────────┘ │ │ │ │ │ │ │ │ raTLS handshake │ │ │ │ [presents Pod A │ │ │ │ per pod cert] │ │ │ ├────────────────────►│ │ │ │ │ │ │ │ ┌───────────────┴───────────────┐ │ │ │ │ Mesh2 verifies Pod A cert │ │ │ │ │ chains to CDS, measurement │ │ │ │ │ on allow list │ │ │ │ └───────────────┬───────────────┘ │ │ │ │ │ │ │ │ Plaintext TCP │ │ │ ├───────────────────►│ │ │ │ │ │ │ │ Response │ │ │ │ plaintext │ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ │ │ Response raTLS │ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ │ │ Response │ │ │ │ plaintext │ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ │ │ ``` *Figure: Pod-to-pod mesh flow. Mesh1 and Mesh2 are the node-level mesh proxies on Pod A's and Pod B's nodes respectively. The workload observes plaintext on both ends; raTLS with per-pod certificates is terminated at the node-level proxy on each side.* #### 5.8.1 Identity Granularity Every pod receives its own CDS-issued certificate (the pod is always the unit of identity), but there are two reasonable choices for where that certificate actually lives on the wire, and they differ in how much machinery the customer has to tolerate in their pod spec. **Per-pod via node-level proxy.** A single mesh proxy runs on each node (typically as a DaemonSet) and holds the certificates for every pod on that node. When a pod opens a connection, the proxy selects the pod's certificate and presents it to the peer. The receiving side sees pod-level identity; the customer sees no additional sidecars in their workload. The trade-off is concentrated in the proxy, which now manages a growing and churning set of certificates rather than a single identity. **Per-pod via sidecar.** Each pod gets its own mesh proxy colocated as a sidecar container. Each sidecar holds exactly one certificate. This is the cleanest arrangement from an identity-accounting perspective and mirrors the Istio deployment model, but it requires injecting a sidecar into every customer pod. That is costly for deployments that were not designed around service-mesh conventions, and at odds with C8s's preference for minimal workload-side disruption. The on-the-wire semantics are identical in both cases. A raTLS handshake presents a per-pod certificate that chains to the CDS CA and corresponds to a measurement on the allow-list. The choice is operational. Deployments with very high pod churn, or those already standardized on per-pod sidecars for other reasons, may prefer the sidecar arrangement; most prefer the shared proxy. ``` Per pod via proxy Per pod via sidecar ╔═ Node [CVM] ═════════════════════════════╗ ╔═ Node [CVM] ═════════════════════════════╗ ║ ║ ║ ║ ║ ┌───────┐ ┌───────┐ ┌───────┐ ║ ║ ┌──────────┐ ┌──────────┐ ┌──────────┐ ║ ║ │ Pod A │ │ Pod B │ │ Pod C │ ║ ║ │ Pod A │ │ Pod B │ │ Pod C │ ║ ║ └───┬───┘ └───┬───┘ └───┬───┘ ║ ║ └────┬─────┘ └────┬─────┘ └────┬─────┘ ║ ║ │ │ │ ║ ║ │ │ │ ║ ║ └────────────┼────────────┘ ║ ║ ▼ ▼ ▼ ║ ║ │ ║ ║ ┌──────────┐ ┌──────────┐ ┌──────────┐ ║ ║ ▼ ║ ║ │ Sidecar │ │ Sidecar │ │ Sidecar │ ║ ║ ┌───────────────────┐ ║ ║ │ [A cert] │ │ [B cert] │ │ [C cert] │ ║ ║ │ Mesh proxy │ ║ ║ └──────────┘ └──────────┘ └──────────┘ ║ ║ │ [Pod A cert] │ ║ ║ ║ ║ │ [Pod B cert] │ ║ ║ ║ ║ │ [Pod C cert] │ ║ ║ ║ ║ └───────────────────┘ ║ ║ ║ ║ ║ ║ ║ ╚══════════════════════════════════════════╝ ╚══════════════════════════════════════════╝ N identities, 1 proxy N identities, N sidecars ``` *Figure: Two identity-holding arrangements. Both preserve pod-level identity on the wire; they differ in whether the certificate lives in a shared node-level proxy or in a sidecar colocated with each pod.* #### 5.8.2 Freshness and Binding Two concerns recur in attested TLS designs. Both have seen formal analysis and both admit a straightforward architectural answer. C8s addresses them by construction rather than as after-the-fact hardening. **Evidence-to-channel binding.** An attestation report proves that a particular code measurement ran at some point on genuine hardware. It does not, on its own, prove that the same hardware is one end of a given TLS session. This is the *channel binding* problem in the sense of RFC 5056 [17], applied to attestation evidence. If evidence were accepted independently of the session it arrived on, a compromised pod with a valid report, or an adversary who captured a report in transit, could present that report while proxying a session to a different key. The bind is what makes the report mean "this connection" rather than "some connection somewhere." C8s binds evidence to the session at certificate issuance time. The attested pod generates an ephemeral key pair inside its CVM's hardware-encrypted memory, and the attestation report it presents to the CDS includes a digest over that public key in a dedicated report field. The CDS verifies the digest before issuing the certificate and embeds the same public key in the certificate it returns. A peer completing a raTLS handshake against this certificate is therefore talking to the holder of the private key that was inside the TEE at attestation time. The report cannot be reused with a different key because the digest in the report field would not match. This puts the binding in the attestation evidence itself, not in a TLS extension or a separate channel. The CDS refuses to issue a certificate if the report's key digest does not match the presented public key, so a pod that failed to bind correctly never gets a certificate in the first place. Peers do not need to re-verify the binding on every connection; they verify the CDS's signature on the certificate, and the binding property is implied by the CDS's issuance policy. **Certificate freshness.** A bound certificate still represents attestation at a point in the past. If a pod's certificate is valid for days or weeks, the gap between "measurement verified" and "connection accepted" grows accordingly, and a post-attestation compromise (key extraction via a side channel, a firmware-level flaw disclosed after issuance, an operator exfiltrating a running pod's key) has longer to do damage. C8s issues short-lived certificates as the default posture. Lifetimes are measured in hours, not days, so the window in which a compromised or stale certificate is accepted is bounded. Short lifetimes are cheap in this architecture because issuance is automated and attested pods re-attest on renewal. The CDS re-verifies the pod's current attestation report, re-checks the key binding, and re-checks the measurement against the allow-list before issuing a replacement. If the pod's TCB has fallen below the minimum (because a firmware advisory has raised the floor), or the measurement is no longer on the allow-list (because the image has been revoked), renewal fails and the pod drops out of the mesh at the old certificate's expiry. There is no separate revocation channel to operate or to fail. Expiry plus re-attestation is the revocation mechanism. The net effect is that C8s's raTLS identity is bound to the session by construction (via key-digest-in-report) and short-lived by default (to bound staleness). Expiry plus re-attestation provides the freshness guarantee for the mesh; raTLS deliberately does not reuse the externally facing freshness beacon (§5.9.2) to avoid confused-deputy issues from sharing a single signing operation across two protocols. ### 5.9 Client Connection Protocol The client-side protocol decouples attestation verification from the transport layer. Rather than relying on TLS extensions (such as Exported Keying Material) to bind attestation to the transport session, the architecture moves attestation verification to the application layer. This decoupling improves client compatibility, so the attestation-aware client works in any HTTP client, including browsers, without requiring TLS extension support. #### 5.9.1 Bootstrapping On first connection, the client retrieves two artifacts from the CDS. The first is the CDS's own attestation report. The second is the CDS-signed manifest, which is the same allow-list the CDS uses internally for image policy enforcement (§5.6, §5.7), exposed for client-side use. The CDS attestation report anchors the trust chain. The client verifies it against the hardware vendor's root of trust, checks that the measurement matches the expected CDS image, and confirms that the public key bound by the report is the same key the manifest is signed under. From that point the client trusts the CDS public key on the basis of its own attestation check, not on operator assertion. The same CDS public key serves a second purpose. It is the trust anchor for any CDS-issued certificate the client subsequently encounters, including the certificate presented by whatever cluster component the client establishes a direct connection to (the ingress router in the default deployment, §5.10). Pool members are trusted transitively through the CDS signature on the manifest and the raTLS chain inside the cluster. The component the client connects to directly is attested separately on each connection window via the freshness beacon (§5.9.2). The manifest contains: - The CDS public key, bound by the CDS attestation the client has just verified. It is used both to verify the manifest signature and to anchor any CDS-issued certificate the client subsequently sees. - The allow-list of permitted measurements, including the expected measurement for the component the client connects to directly and any minimum TCB versions or other appraisal policy fields the CDS enforces. Because the client has already attested the CDS, verifying the manifest signature is sufficient to trust its contents. The client uses the allow-list to recognize a valid measurement when one is presented in an attestation report, but performs no per-machine appraisal. Verified data is cached locally. Subsequent requests use the cached values, with periodic refresh to keep up with allow-list changes. The bootstrapping cost is amortized, so only the first connection incurs the full CDS attestation. #### 5.9.2 Default: Traditional TLS + Attestation In the default configuration, the client establishes a connection to the cluster's ingress router over ordinary HTTPS. The ingress presents (a) a public-CA-issued TLS certificate whose key was generated and is held inside the ingress's TEE, and (b) an attestation report. The client verifies that the TLS certificate's public key matches the bound public key in the attestation report, that the attestation's measurement matches the ingress identity in the CDS-signed manifest, and that the attestation is fresh under the freshness beacon described below. Once these checks pass, the client treats the open TLS session as bound to an attested ingress and reuses it for subsequent requests until the freshness window or session lifetime elapses. A malicious control plane could in principle forge a public-CA TLS certificate, but it could not produce a valid attestation binding the certificate's key to an attested ingress measurement, so a verifying client refuses to use the session. The chain of trust runs end to end: 1. The operator attests the CDS on cluster install (§5.6.2). 2. The client attests the CDS on first connection (§5.9.1). 3. The CDS countersigns the freshness beacon used by the ingress. 4. The ingress's attestation binds its TLS certificate's public key to the measured ingress. 5. The ingress opens a raTLS connection to the destination pod using a CDS-issued mesh certificate, which the CDS issued only after the pod's own attestation. Every hop after the client's TLS termination is therefore between attested peers whose identities chain back to the same CDS the client itself has attested. **Freshness beacon.** TEE attestation signing has limited throughput, so requiring a fresh attestation per request is infeasible at scale. The CDS produces a freshness beacon (a timestamp-based freshness mechanism in the sense of RFC 9334 §10 [15], with the CDS signature serving as the unforgeable nonce) on demand that any Verifier can use to bound the staleness of a presented attestation: 1. When an attested workload (typically the ingress) detects that its current attestation is approaching the freshness window's edge, it requests a fresh beacon from the CDS over its existing raTLS connection. The CDS signs the current timestamp `T` with its signing key, applying a fixed domain separator (e.g., `"c8s/freshness-beacon/v1" || T`) so the beacon-signing operation cannot be confused with any other CDS signature. It returns `sig_CDS(T)` to the caller. 2. The workload asks the TEE to produce a new attestation whose `REPORT_DATA` field carries `sig_CDS(T)` and whose bound-pubkey field carries the workload's TLS public key. 3. When a client connects, the workload presents the attestation report along with `(T, sig_CDS(T))`. The client then checks each of the following: - `sig_CDS(T)` is a valid CDS signature over the domain-separated input under the CDS public key established at trust bootstrap. - `T` falls within the configured freshness window. - The attestation's `REPORT_DATA` equals `sig_CDS(T)` exactly. - The attestation's bound public key matches the TLS certificate's public key. - The measurement is on the allow-list and the TCB meets the minimum. A CDS signature is unforgeable to any party that does not hold the CDS private key, so it functions as an unforgeable nonce. Combined with the timestamp, the nonce bounds when the attestation could have been generated. The verifier needs no out-of-band data, since the timestamp and signature both arrive in-band with the attestation, and the CDS public key was already cached during bootstrapping (§5.9.1). The window is operator-configurable and typically on the order of minutes, large enough to amortize the cost of TEE attestation generation given hardware throughput limits, and small enough to limit the impact of a post-attestation compromise. If a workload's attestation goes too stale, clients refuse to talk to it, mirroring the behavior of expired raTLS certificates inside the mesh. #### 5.9.3 Optional: Multi-Recipient Client-Side Encryption For clients that want end-to-end encryption to the destination pod's own key, typically because they don't trust the ingress router to terminate TLS, or want crypto-agility for the payload independent of the transport, C8s supports client-side multi-recipient encryption (§2.4). The client encrypts the payload to the public keys of attested pods in the target pool, using a hybrid scheme. The body is encrypted once with a fresh symmetric key, and that key is wrapped to each recipient's public key. The wrapped header carries, per recipient: - The public-key material the corresponding pod uses to derive the symmetric decryption key, and - A plaintext routing hint (e.g., the pod's hostname), used by the Encrypted Ingress Router (§5.10) for forwarding decisions. Overhead per additional recipient is approximately 64 bytes in the header. Clients may encrypt to a subset of the pool rather than its entirety, trading routing flexibility for smaller headers. The reference implementation uses AGE (§2.4), but the protocol is not bound to a specific format. Threshold, hybrid, or post-quantum schemes can be substituted without changes elsewhere. This path is available to clients that opt in, and it requires the Encrypted Ingress Router variant on the cluster side. ``` Client CDS Ingress Router TEE pool │ │ │ │ │ │ │ │ │ GET manifest │ │ │ ├────────────────►│ │ │ │ │ │ │ │ Signed manifest │ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ │ │ │ ┌────────────────────────────┐ │ │ │ │ Verify + cache │ │ │ │ │ [chain or attestation] │ │ │ │ └────────────────────────────┘ │ │ │ │ │ │ │ ┌────────────────────────────┐ │ │ │ │ Optional multi recipient │ │ │ │ │ encrypt [§5.9.3] │ │ │ │ └────────────────────────────┘ │ │ │ │ │ │ │ HTTPS POST │ │ │ ├────────────────────────────────────►│ │ │ │ │ │ │ │ │ raTLS forward │ │ │ ├──────────────►│ │ │ │ │ │ Refresh │ │ │ │ [periodic] │ │ │ ├┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈►│ │ │ │ │ │ │ ``` *Figure: Client connection flow. The manifest fetch is amortized through caching and periodic refresh, and each request only exercises the lower half of the diagram. The multi-recipient encryption step is optional. By default the client submits the request over ordinary HTTPS and the ingress router does the raTLS forwarding.* #### 5.9.4 Response Path Attested workloads include attestation metadata in response headers, enabling the client to verify that the response originated from within the trust boundary. The response body is encrypted to the client using keys established during the session. The full request-response cycle operates over standard HTTPS; no custom transport protocol is required. ### 5.10 Ingress Router External clients need to reach a specific pod inside the trust boundary. C8s offers two interchangeable ways to do this, both running inside the cluster's trust boundary. They differ in how much they can see of the request payload. The router plays the same data-plane role as a standard Kubernetes ingress controller (nginx, Envoy, HAProxy, or a cloud-native equivalent). It accepts external traffic, terminates a public TLS endpoint, and forwards to a backing pool. Two things distinguish it from those primitives. First, the entire data path runs inside a CVM, so traffic in flight is never visible to the host or the control plane. Second, it refuses to forward to any peer that does not present a valid CDS-issued raTLS certificate, so a control-plane edit that points a Kubernetes Service at an unattested endpoint cannot redirect traffic. #### 5.10.1 Default: raTLS pass-through The default ingress router is an attested load balancer running inside a CVM alongside the rest of the C8s control-boundary components. Externally, it presents a public-CA-issued TLS certificate whose key is generated and held inside the ingress's TEE, paired with a recent attestation that the client verifies (§5.9.2). The TLS certificate itself is obtained through ordinary mechanisms (e.g., ACME running inside the attested ingress to bind the cert to the TEE-internal key). The precise issuance workflow is out of scope here. Internally, the ingress holds a CDS-issued raTLS mesh certificate for talking to other attested pods, and for each incoming request it opens a raTLS connection to the target pod over the mesh. **Service discovery and downstream membership.** Pool membership comes from the standard Kubernetes Service object the router watches; the router learns about new endpoints, scaling events, and pod churn through the same kube-apiserver mechanisms an ordinary ingress controller uses. A control plane that adds a malicious endpoint cannot weaponize this, because the raTLS handshake to that endpoint will fail unless the endpoint holds a CDS-issued mesh certificate, and the CDS only issues to attested measurements on the allow-list. Health-checking is similarly TLS-aware: a peer that fails the raTLS handshake (revoked, expired, or unattested) is dropped from the live pool until it can present a valid certificate again. Failover across multiple healthy peers uses the same load-balancing strategies any modern ingress supports; the difference is the admission criterion, not the load-balancing algorithm. The chain of trust runs end to end. The operator attests the CDS at bootstrap (§5.6.2). The CDS in turn countersigns the ingress's freshness beacon and validates its attestation against the allow-list. The ingress then validates the destination pod's mesh certificate. From the client's perspective the router is a normal HTTPS endpoint that additionally presents an attestation. From the cluster's perspective every hop after TLS termination is raTLS between attested peers. This gives end-to-end semantics without requiring the client to encrypt payloads to specific pod keys. The client still runs the standard raTLS verification path (attestation signature, freshness beacon, measurement allow-list, TLS key binding), but it does not need to manage per-pod recipient keys or wrap payloads itself the way it does in the multi-recipient encryption variant (§5.9.3). The request is in plaintext only inside the router's CVM memory (briefly, while it is being forwarded) and inside the destination pod's CVM memory. Neither endpoint is reachable from the hypervisor, host OS, or the control plane. Because the router itself is attested and mesh-resident, compromising the cluster's ordinary Kubernetes ingress does not yield access to request bodies. The attacker would have to compromise the router's CVM, which is the same threat model as compromising any other attested workload. ``` ┌─────────────────────────────┐ │ Cloud L7 ingress │ │ [standard, untrusted, │ │ distributes TCP only] │ └──────────────┬──────────────┘ │ ▼ ╔═════════════════════════════╗ ║ Ingress Router [CVM] ║ ║ Terminates external TLS ║ ║ Opens raTLS to target pod ║ ╚══════════════╦══════════════╝ ║ ║ raTLS mesh ┌────────────────────╬────────────────────┐ ▼ ▼ ▼ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ Attested pod │ │ Attested pod │ │ Attested pod │ └───────────────┘ └───────────────┘ └───────────────┘ ``` *Figure: Default ingress path. The ingress router is itself an attested CVM; it terminates external TLS, then relays requests to the destination pod over raTLS. No other cluster component ever sees request payloads in plaintext.* #### 5.10.2 Optional: Encrypted Ingress Router Some deployments want a stronger property, where the ingress never sees plaintext, not even briefly inside its own CVM. This is useful when the cluster operator is not the same entity as the workload owner (the workload owner trusts their own attested pod but not the router's), when multi-recipient crypto agility is valuable (threshold, hybrid, post-quantum), or as defense in depth against a router compromise. For these cases C8s ships an **Encrypted Ingress Router** variant, implemented as a per-node DaemonSet. Clients encrypt requests with multi-recipient encryption (§2.4, §5.9.3) to a pool of attested pods, submit the ciphertext through ordinary Kubernetes ingress, and the router parses the per-recipient header, extracts the plaintext routing hints from the recipient stanzas, and forwards the still-encrypted payload to the destination pod. The router never decrypts the payload body; it operates exclusively on the routing metadata in the header. A compromise of the router exposes routing metadata only, and workload data remains encrypted end-to-end to the destination pod. On each node the router runs as an attested pod. It boots with a measured image, attests to the CDS, receives a per-pod certificate, and participates in the raTLS mesh. When a request arrives: 1. The router parses the encrypted payload header and extracts the plaintext routing hints from the recipient stanzas. 2. If any hint matches an attested pod on the local node, the router forwards the still-encrypted payload to that pod. 3. Otherwise, the router forwards the payload across the raTLS mesh to a peer Encrypted Ingress Router on the node that hosts a matching pod. 4. The destination attested pod decrypts the payload with its private key (held only in CVM-encrypted memory) and processes the request. ``` ┌────────────────────────────┐ │ Cloud L7 ingress │ │ [standard, untrusted] │ └──────────────┬─────────────┘ │ Distributes to nodes ┌──────────────────────┴──────────────────────┐ ▼ ▼ ┌─ Node 1 ───────────────────────────────────┐ ┌─ Node 2 ───────────────────────────────────┐ │ │ │ │ │ ╔════════════════════════════════╗ │ │ ╔════════════════════════════════╗ │ │ ║ Encrypted Ingress ║ │ │ ║ Encrypted Ingress ║ │ │ ║ Router ║ │ │ ║ Router ║ │ │ ║ [DaemonSet, AGE aware] ║ │ │ ║ [DaemonSet, AGE aware] ║ │ │ ║ ║ │ │ ║ ║ │ │ ║ 1. Parse stanza hints ║ │ │ ║ 1. Parse stanza hints ║ │ │ ║ 2. Forward ciphertext ║ │ │ ║ 2. Forward ciphertext ║ │ │ ║ Local or mesh ║ │ │ ║ Local or mesh ║ │ │ ╚════════════════╦═══════════════╝ │ │ ╚════════════════╦═══════════════╝ │ │ │ Local │ │ │ Local │ │ ▼ │ │ ▼ │ │ ╔═══════════════════════╗ │ │ ╔═══════════════════════╗ │ │ ║ Attested pod ║ │ │ ║ Attested pod ║ │ │ ║ Decrypt + run ║ │ │ ║ Decrypt + run ║ │ │ ╚═══════════════════════╝ │ │ ╚═══════════════════════╝ │ │ │ │ │ └────────────────────────────────────────────┘ └────────────────────────────────────────────┘ ▲ ▲ │ raTLS mesh │ └──────────── [routed here if stanza ────────┘ targets peer node] ``` *Figure: Encrypted Ingress Router topology. Cloud ingress distributes TCP to nodes without understanding the encryption format. A per-node router (attested DaemonSet) parses recipient hints and forwards ciphertext to a local pod or a peer router on another node. No decryption happens in the router; at most one intra-cluster hop across the mesh.* **Why per-node rather than centralized.** A single centralized payload-aware load balancer would work, but it concentrates routing in one component and adds a hop that duplicates what Kubernetes ingress already does. Running the Encrypted Ingress Router as a per-node DaemonSet scales linearly with cluster size, eliminates the single-component concentration, and lets the cloud LB play its ordinary role (distributing TCP to nodes) without C8s-specific behavior. **Why not a sidecar in each customer pod.** Sidecars would require injecting C8s machinery into every customer workload manifest, which cuts against the architecture's preference for leaving customer workloads unchanged. The DaemonSet model lives at the node level and is invisible to the customer's pod spec. **Response path.** Responses are encrypted by the destination pod to the client's public key (carried in the request) and travel back through the same path, from pod to local router to cloud ingress to client. The mesh and the router both see ciphertext on the return leg as well. #### 5.10.3 Choosing between the two The two variants are compatible with the rest of the architecture; the choice is deployment-time and can be made per cluster or per workload. The default raTLS pass-through is simpler for operators and requires no client-side cryptography. The Encrypted Ingress Router is appropriate when the threat model includes an untrusted or potentially-compromised ingress component, or when the workload's clients already speak a multi-recipient encryption format for other reasons. Both variants can coexist in the same cluster. ### 5.11 Attested Build System (Kettle) Kettle is the build pipeline that produces container images with attested digests. Builds execute inside a TEE, and the build environment is itself attested, so the output of a Kettle build is a container image paired with a hardware-signed attestation report binding the build inputs to the output digest. This extends the chain of trust from source code to running container. Without an attested build system, an attacker who compromises the CI/CD pipeline could produce a malicious image and add it to the policy manifest. Kettle holds no signing key of its own; the attestation report is the signature. Each build produces a provenance document recording the build inputs and output image digests, and the SHA-256 digest of that document is committed into the report-data field of the attestation report, so the signing identity behind every Kettle build chains to the CPU vendor's root of trust rather than to a key held in guest memory. Digests enter the policy manifest (§5.7) only after that provenance has been verified against the vendor root and an allow-listed Kettle launch measurement; the manifest itself is signed by the CDS. An untrusted operator can deploy the manifest but cannot forge it: substituting a digest would require the CDS signing key, and fabricating provenance for a malicious image would require forging a hardware-signed attestation report. A detailed treatment of Kettle's design and implementation is the subject of a separate publication. --- ## 6. Request Lifecycle This section traces a single request through the system, identifying where each security guarantee applies. ### 6.1 Preconditions Before the first request, the following setup has occurred: 1. The CDS has been deployed inside a CVM and attested by the operator. 2. The CVM substrate that will host the workload has booted. In the default pod-level configuration, the C8s pod runtime is installed on each host and the measured per-pod configuration has been published. In a node-level configuration, worker nodes have booted as CVMs and each node's secure processor has computed a launch measurement. Either way, the CVM that will host the workload has a pre-computable launch digest that the CDS's allow-list recognizes. 3. The CVM substrate has attested to the CDS. In the pod-level configuration this happens once per pod at start, directly; in the node-level configuration the node attests first and then composes per-pod evidence on top. 4. Image-gap gating is in place. In the pod-level configuration, measured image pinning, per-customer workload signing, or both are baked into the pod's measured configuration and enforced by the in-pod agent at pull time. In the node-level configuration, the pod attestation agent, NRI image policy enforcer, and raTLS mesh interceptors run as DaemonSets, have attested to the CDS as pods, and hold their per-pod mesh certificates and the signed policy manifest. 5. Workload pods have been scheduled and started. Each workload pod has presented attestation evidence to the CDS (a direct per-pod attestation in the default configuration, or composed node-and-workload evidence in the node-level configuration) and received its own per-pod mesh certificate. All inter-pod connections use these certificates. 6. Any application secrets (if encrypted) have been decrypted inside TEE memory using keys released by the CDS to the attested pod under the brokering flow described in §5.6.4. The key exists only in the pod's hardware-encrypted memory; it is not persisted to pod storage. 7. The ingress router holds a public-CA-issued TLS certificate bound to its TEE-internal key, plus a recent attestation that incorporates the latest CDS freshness beacon. It also holds a CDS-issued raTLS mesh certificate for internal traffic. The following diagram traces a single request through the pipeline. Each dashed band corresponds to one of the subsections below; readers can use it as a map for the narrative that follows. ``` Client CDS Ingress [CVM] Mesh Workload Pod [CVM] │ │ │ │ │ ╌ §6.2 ╌ Client bootstrapping ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │ │ │ │ │ │ GET manifest │ │ │ │ ├─────────────────────────►│ │ │ │ │ Signed manifest │ │ │ │ │ [CDS pubkey, │ │ │ │ │ ingress ident, │ │ │ │ │ pod identities] │ │ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ Verify + cache │ │ │ │ │ │ │ │ │ ╌ §6.3 ╌ Connection establishment ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │ │ │ │ │ │ TLS handshake │ │ │ ├────────────────────────────────────────────►│ │ │ │ Public CA cert + attestation │ │ │ │ + [T, sig_CDS of T] │ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ Verify cert pubkey is attest.pubkey │ │ │ │ Verify sig_CDS over T + T fresh │ │ │ │ Verify measurement on allow list │ │ │ │ │ │ │ │ ╌ §6.4 ╌ Request submission ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │ │ │ │ │ │ HTTPS POST │ │ │ ├────────────────────────────────────────────►│ │ │ │ │ │ Pick attested │ │ │ │ │ pod from pool │ │ │ │ │ │ │ ╌ §6.5 ╌ Intra cluster transit ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │ │ │ raTLS │ │ │ │ │ [mesh certs] │ │ │ │ ├───────────────►│ │ │ │ │ ├───────────────────►│ │ │ │ │ │ ╌ §6.6 ╌ Workload ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │ │ │ │ │ Run workload | | | | | ╌ §6.7 ╌ Response ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ │ │ │ │ raTLS │ │ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ HTTPS 200 │ │ │ │◄┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┤ │ │ │ │ │ │ │ ``` *Figure: End-to-end request lifecycle in the default configuration. Dashed bands mark the subsection boundaries used in the rest of this section. The optional client-side multi-recipient encryption path (§5.9.3) replaces the `HTTPS POST` step with a ciphertext payload routed by hostname hints (see §5.10.2).* ### 6.2 Client Bootstrapping The client's attestation SDK contacts the CDS and retrieves the CDS-signed manifest containing the CDS public key, the ingress identity (its expected measurement and TLS public key), and the public keys and attestation references for the attested pod pool. The SDK verifies the manifest (either via certificate chain validation against the CDS public key or by independent attestation verification) and caches the verified data. This step occurs once and is amortized across subsequent requests; periodic refresh covers pool membership changes. ### 6.3 Connection Establishment The client opens a TLS connection to the ingress router. The ingress presents (a) a public-CA-issued TLS certificate whose key was generated and is held inside the ingress's TEE, and (b) a recent attestation report along with `(T, sig_CDS(T))` from the latest freshness beacon. The client verifies that the TLS certificate's public key matches the bound public key in the attestation, that the attestation's measurement matches the ingress identity in the cached manifest, and that `sig_CDS(T)` is a valid CDS signature over a sufficiently recent `T` (§5.9.2). Once these checks pass, the open TLS session is treated as bound to an attested ingress and reused for subsequent requests until the freshness window or session lifetime elapses. ### 6.4 Request Submission The client sends the request body over the established TLS connection. Inside the ingress's CVM-encrypted memory, the request is briefly in plaintext while the ingress selects an attested pod from the target pool. (Clients that prefer to keep the request body opaque to the ingress can use the optional client-side multi-recipient encryption path described in §5.9.3 with the Encrypted Ingress Router variant in §5.10.2; the ingress then operates only on the encrypted payload's routing hints.) ### 6.5 Intra-Cluster Transit The ingress opens a raTLS connection to the destination pod. Both endpoints hold CDS-issued per-pod mesh certificates whose key digests bound them to attestations the CDS verified. The mesh encrypts the request in transit; the underlying network sees only TLS ciphertext between attested peers. ### 6.6 Workload Execution The destination pod receives the request inside its CVM-encrypted memory and processes it as it would on any standard deployment. Inputs, intermediate state, working buffers, and any sensitive artifacts the workload has loaded (model weights, decryption keys, dataset shards, downstream credentials) reside in memory encrypted with keys the hypervisor never possesses. The execution itself is whatever the workload is — a database query, a batch job, an LLM generation step (tokenization, attention, KV cache updates, token sampling), a signing operation, etc. (Where the optional client-side multi-recipient encryption path is in use, the pod additionally decrypts the per-recipient envelope using its private key, held only in CVM-encrypted memory, before processing the request.) ### 6.7 Response The workload response is returned through the same path. It travels from pod to ingress via the raTLS mesh, then from ingress to client over the established TLS session. Attestation metadata is included in response headers, enabling the client to confirm the response originated from an attested TEE within the verified pool. --- ## 7. Untrusted Control Plane Compatibility ### 7.1 Design Rationale Managed Kubernetes services (AKS, EKS, GKE) operate the control plane as a service. The operator does not control the control plane's code, cannot run it in a TEE, and cannot attest it. Requiring the control plane to be inside the trust boundary would exclude the majority of production Kubernetes deployments. The architecture deliberately places the control plane outside the trust boundary. The control plane retains its scheduling and orchestration functions but is stripped of any role in confidentiality enforcement. Trust anchors are moved to the node: the NRI image policy enforcer gates workload execution, the CDS gates identity issuance, and the raTLS mesh gates network participation. ### 7.2 Blast Radius of Control Plane Compromise A compromised or malicious control plane is limited to denial of service. It can refuse to schedule workloads, terminate pods, or disrupt service discovery. It cannot: - Access workload data (encrypted in TEE memory and in transit). - Forge attestation reports (requires hardware-rooted signing keys). - Inject unauthorized workloads (rejected by the NRI enforcer). - Obtain mesh credentials (issued only by the CDS to attested pods). - Access model decryption keys (released by the CDS only to attested pods; present only in the pod's hardware-encrypted memory, never persisted or exposed to the control plane). When the trust boundary is drawn at the pod rather than the node, the "control plane outside the trust boundary" property extends even to managed Kubernetes clusters whose nodes have no TEE support at all. The pod's CVM is provisioned per-pod (for example as a confidential cloud VM), so the cluster's node image and scheduler stay on the untrusted side without weakening any of the guarantees above. ### 7.3 Self-Hosted Control Plane Organizations that operate their own control plane may optionally deploy it inside CVMs and attest it alongside worker nodes. This brings the control plane inside the trust boundary and protects cluster state (scheduling metadata, etcd contents, configuration) from the infrastructure operator. This option is not available on managed Kubernetes services and adds the operational cost of running the control-plane components (API server, scheduler, etcd, controller manager) as attested CVMs, with the corresponding attestation, upgrade, and key-management workflows applied to each. It is not required for the confidentiality guarantees described in this paper. --- ## 8. Configurable Boundaries The architecture is not a fixed blueprint. Components can be combined, extended, or omitted depending on the threat model and operational requirements. ### 8.1 Minimal Deployment The recommended starting point includes: - A CVM substrate attested to the CDS, with per-pod CVMs provisioned by the C8s pod runtime by default (§5.1), or CVM worker nodes where the boundary is drawn at the node (§5.2). - Measurement-bound workload gating, implemented as measured image pinning or per-customer workload signing inside each pod's configuration in the default pod-level setup, or as NRI image policy enforcement on each node (via DaemonSet) in a node-level setup; both can coexist in a mixed cluster. - raTLS mesh with CDS-issued per-pod certificates for all inter-pod traffic. - An attested ingress with a public-CA-issued TLS certificate bound to a TEE-internal key, and the CDS freshness beacon (§5.9.2) for client-side staleness checks. - Control plane remains untrusted. This configuration is compatible with managed Kubernetes and provides the full set of confidentiality guarantees described in this paper. ### 8.2 Oblivious HTTP (OHTTP) Standard TLS protects request contents but exposes metadata such as which clients connect, when, and how often. For applications where traffic analysis is a privacy concern, such as healthcare or legal queries, the architecture supports an OHTTP [14] relay layer. The relay observes client IP addresses but cannot read request payloads (which are encrypted to the attested gateway under multi-recipient encryption, §5.9.3). The gateway processes requests but sees only the relay's address, not the originating client. No single party learns both identity and content. ``` ┌─────────────┐ ┌───────────────┐ ┌───────────────────┐ ┌─────────────┐ │ │ AGE over │ │ OHTTP │ │ AGE │ │ │ Client ├──── OHTTP ─────────►│ OHTTP relay ├─── Ciphertext ───►│ Attested gateway ├─── Ciphertext ───►│ TEE │ │ │ ciphertext │ │ │ │ │ │ │ │ │ │ │ [unwraps OHTTP, │ │ │ │ │ │ │ │ forwards AGE] │ │ │ └─────────────┘ └───────────────┘ └───────────────────┘ └─────────────┘ No single party sees both client identity and plaintext content. ``` *Figure: OHTTP relay topology. Each hop sees either the client identity or the payload, never both; the relay cannot decrypt the OHTTP envelope, and the gateway cannot see the originating client.* --- ## 9. Discussion ### 9.1 Trust Assumptions The architecture concentrates trust in hardware manufacturers (AMD, Intel) and the code running inside measured TEEs. The CDS is explicitly trusted as the root of the certificate chain. A CDS compromise would allow an adversary to issue fraudulent certificates and, through them, impersonate attested workloads to peers that accept CDS-issued identity. It would not retroactively decrypt past traffic, break in-flight sessions protected by ephemeral keys, or reach CVM memory whose keys the CDS has never held, because the CDS is deliberately scoped to CA signing material and attestation-gated brokering (see §5.6.3). This trust is further mitigated by the CDS running inside a CVM whose attestation the operator manually verifies at deployment time. If the hardware manufacturer is compromised or malicious, the attestation guarantees do not hold. This is a narrower trust surface than the standard model (which additionally trusts cloud providers, their employees, and their software stack), but it is not zero trust. ### 9.2 Isolation Model Hardware isolation in C8s sits at the CVM boundary, and where that boundary falls depends on the deployment. Where the boundary is drawn at the node, the CVM provides hardware-level isolation from the hypervisor and host OS, and pods inside a single CVM node are separated by standard Linux kernel mechanisms. Hard isolation is at the node boundary; pod-to-pod isolation on the same node is kernel-level, so a kernel exploit or misconfigured security context can bridge pods that share a node. Deployments requiring hardware isolation between tenants in this configuration should dedicate CVM nodes per tenant. Where the boundary is drawn at the pod, the hardware boundary is the pod itself. Each pod's memory is encrypted with keys the hypervisor never possesses, and pods on the same physical host cannot reach each other's memory through a kernel exploit. Tenant isolation tightens to the pod level without requiring a node-per-tenant scheduling constraint. ### 9.3 Operational Impact For platform operators already running Kubernetes, the primary operational changes are: (a) replacing standard node images with CVM node images, (b) deploying the CDS, NRI enforcer (DaemonSet), and raTLS mesh (DaemonSet) as infrastructure components, and (c) integrating the attested build pipeline (Kettle) into CI/CD. Existing deployments, Helm charts, and application code require no modification. The workload runs the same code in the same way; only the underlying node and network layer change. C8s supports both all-CVM clusters and mixed clusters that combine CVM and non-CVM worker nodes under a single control plane. The per-pod raTLS mesh, NRI enforcer, and CDS operate unchanged on the CVM side; pods scheduled onto non-CVM nodes remain outside the trust boundary and do not receive mesh identity. Workload admission policy determines which workloads are permitted on which node type, letting operators run sensitive and non-sensitive workloads side by side in one cluster. ### 9.4 Limitations The architecture has the following bounded scope: - **GPU hardware without CC mode.** GPU-side protections are not provided on hardware that does not support confidential computing mode. - **Side channels.** Side-channel mitigations are outside the current scope. - **Runtime behavior.** The NRI image policy enforcer operates on image digests, not on runtime behavior; a vulnerability in an authorized image is exploitable within the TEE. - **Availability.** Availability guarantees are explicitly excluded; the architecture protects confidentiality and integrity, not uptime. --- ## 10. Conclusion and Open Directions The contribution of this paper is a single claim. Hardware-attested confidentiality for Kubernetes workloads is reachable without bringing the control plane inside the trust boundary, and the resulting architecture is compatible with the managed Kubernetes services (AKS, EKS, GKE) that most production deployments already run on. §5 describes the components that make this possible, including attestation-gated key brokering for sensitive artifacts such as model weights (§5.6.4). A handful of directions sit on top of the architecture described here and are treated at depth elsewhere: - **Attested build (Kettle).** A dedicated paper will cover the build environment, the attestation format, and the reproducibility guarantees (§5.11). - **Finer-grained mesh termination when the boundary is at the pod.** The raTLS mesh currently terminates at the node in node-level deployments. When the boundary is drawn at the pod (the default; §5.1), the mesh can terminate inside the pod's CVM, tightening the boundary between workloads that share a host. The implementation and the incremental guarantee are under investigation. - **Per-workload measured configuration and per-customer signing rollout.** The measured-image-pinning and per-customer-signing mechanisms in §5.1.1 are deployed for a small number of workloads today. Scaling signing to a platform-wide onboarding flow, caching measured configurations across workload classes, and sequencing key rotation without churning running pods are practical questions being worked through in production. - **Freshness of attestation.** TEE attestation throughput limits make per-connection attestation infeasible, which motivates the freshness beacon design in §5.9.2. Tightening the beacon window without saturating attestation throughput, and extending the same primitive to other verification points, remain open. - **Allow-list transparency log.** The CDS today exposes the current and previous allow-list from inside its attested CVM (§5.6.1). Extending this to a full append-only history with signed consistency roots, letting a third party verify that any past allow-list version is still in the log and that the log has not been rewritten, is a natural next step that would close the residual operator-side circularity in §9.1. Each of these sits on top of the architecture described here rather than changing it. The trust model, the threat model, and the design formula in §4.2 are stable, and new work slots into them rather than revises them. --- ## References [1] AMD. *AMD SEV-SNP: Strengthening VM Isolation with Integrity Protection and More*. AMD White Paper, 2020. [2] Intel. *Intel Trust Domain Extensions (Intel TDX) Module Base Architecture Specification*. Intel, 2023. [3] NVIDIA. *NVIDIA Confidential Computing*. NVIDIA Technical Brief, 2023. [4] confidential.ai. *TEE Performance on CPUs*. https://github.com/confidential-dot-ai/home/blob/main/blog/tee-performance-cpus.md, 2025. [5] W. Kwon, Z. Li, S. Zhuang, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." *Proceedings of the 29th Symposium on Operating Systems Principles (SOSP)*, 2023. [6] Kata Containers. *Kata Containers Architecture*. https://katacontainers.io, 2024. [7] Confidential Containers. *CoCo Project Documentation*. https://github.com/confidential-containers, 2024. [8] V. Shoup. "A Proposal for an ISO Standard for Public Key Encryption." *Cryptology ePrint Archive*, Report 2001/112, 2001. [9] A. Fiat and M. Naor. "Broadcast Encryption." *Advances in Cryptology (CRYPTO)*, 1993. [10] F. Valsorda. *The age File Encryption Format*. https://age-encryption.org/v1, 2019. [11] M. Morbitzer, M. Huber, J. Horsch, S. Wessel. "SEVered: Subverting AMD's Virtual Machine Encryption." *Proceedings of the 11th European Workshop on Systems Security (EuroSec)*, ACM, 2018. [12] HashiCorp. *Vault by HashiCorp*. https://www.vaultproject.io, 2024. [13] Kubernetes. *Node Resource Interface (NRI)*. https://github.com/containerd/nri, 2023. [14] M. Thomson, C. A. Wood. "Oblivious HTTP." *RFC 9458*, IETF, 2024. [15] H. Birkholz, D. Thaler, M. Richardson, N. Smith, W. Pan. "Remote ATtestation procedureS (RATS) Architecture." *RFC 9334*, IETF, 2023. https://datatracker.ietf.org/doc/html/rfc9334 [16] R. Barnes, K. Bhargavan, B. Lipp, C. Wood. "Hybrid Public Key Encryption." *RFC 9180*, IETF, 2022. https://datatracker.ietf.org/doc/html/rfc9180 [17] N. Williams. "On the Use of Channel Bindings to Secure Channels." *RFC 5056*, IETF, 2007. https://datatracker.ietf.org/doc/html/rfc5056 [18] Istio Authors. *Istio Service Mesh*. https://istio.io, 2024. [19] SPIFFE. *Secure Production Identity Framework for Everyone*. https://spiffe.io, 2024. [20] Sigstore. *Cosign: Container Signing, Verification and Storage in an OCI Registry*. https://github.com/sigstore/cosign, 2024. [21] J. Chuang, A. Seto, N. Berrios, S. van Schaik, C. Garman, D. Genkin. "TEE.fail: Breaking Trusted Execution Environments via DDR5 Memory Bus Interposition." *47th IEEE Symposium on Security and Privacy (S&P '26)*, IEEE, 2026. [22] J. De Meulemeester, D. Oswald, I. Verbauwhede, J. Van Bulck. "Battering RAM: Low-Cost Interposer Attacks on Confidential Computing via Dynamic Memory Aliasing." *47th IEEE Symposium on Security and Privacy (S&P '26)*, IEEE, 2026. [23] J. De Meulemeester, L. Wilke, D. Oswald, T. Eisenbarth, I. Verbauwhede, J. Van Bulck. "BadRAM: Practical Memory Aliasing Attacks on Trusted Execution Environments." *46th IEEE Symposium on Security and Privacy (S&P '25)*, IEEE, 2025. --- # Kettle: Attested Builds Download PDF version **April 2026** --- ## Abstract Kettle is an **attested build** system that produces cryptographically verifiable provenance for software built inside Trusted Execution Environments (TEEs). A Kettle build records the source commit, dependency set, toolchain, build environment, and output artifact digests in a provenance document produced inside a measured confidential VM. The SHA-256 digest of that document is committed to the TEE platform's attestation report-data field, so the hardware-signed attestation report is itself the signature on the provenance, with the signing identity chaining to the TEE manufacturer's root of trust rather than to the build infrastructure operator. Because the CVM image is itself reproducible, its launch measurement is public and stable, which lets a build requester pre-attest the CVM before submitting any input and optionally deliver source over a TLS channel terminated inside it, so the build runs end-to-end confidentially without the host ever seeing source code in plaintext. Verification reduces to one signature check against the vendor root and a small set of digest comparisons, with no need to re-execute the build. The result removes the build infrastructure, its operators, and the artifact distribution channel from the trust surface a verifier must accept when deciding whether a binary corresponds to its claimed inputs. --- ## 1. Introduction ### 1.1 Motivation Software consumers routinely execute artifacts built by a separate software provider. Examples include a developer installing a prebuilt binary from a registry, an enterprise deploying a vendor container image, or a customer running a hosted service whose source they were allowed to inspect. A software artifact is the end result of a rather complicated build system. Developers write source code, and that source is fetched from version control. Dependencies are resolved from package registries or caches. Compilers, linkers, and build scripts transform those inputs while running with specific environment variables and filesystem state. The operating system, kernel, and container or VM image define the execution environment in which the build occurs. Changes to any of these layers affect the bytes that are eventually distributed, and hence impact the behavior of the software that a consumer interacts with. Each of these layers also presents an attack surface. The software provider then must provide some claims regarding the integrity and behavior of the artifacts it produces. A consumer then uses those claims to establish trust in the software they interact with from that provider. The usual evidence available to that consumer is operational: CI logs, release notes, signed checksums, artifact registry metadata, or a provenance document emitted by the build pipeline. These records can be useful, but by themselves they prove only that some system made a statement. They do not prove that the statement is true. A compromised build runner can inject code during compilation while leaving the source repository clean. A privileged operator can alter the toolchain, cached dependencies, environment variables, or artifact upload step. An attacker with access to artifact storage can replace the binary after the build has finished. In each case the consumer sees a plausible release record, but the artifact may no longer correspond to the source and dependencies being claimed. The underlying gap is that most build systems do not bind the provider's claims to the actual software artifact in a way that is independently provable. A release may point to a source commit, publish a lockfile, name a toolchain, and distribute an artifact digest, but those facts are usually connected by process rather than by cryptographic evidence. Closing this gap is difficult because build systems are large, stateful, and highly variable across languages, package managers, operating systems, and deployment formats. Any useful binding must scale across that complexity, and verification must remain fast and low-friction enough for ordinary software consumption. ### 1.2 Problem Statement The problem is to establish, for a specific software artifact, whether the provider's claim about its origin is trustworthy. The aim is to turn build provenance from an assertion into cryptographic evidence that a consumer can verify independently. Any practical solution has to satisfy the following constraints: 1. A software consumer, or tooling acting on the consumer's behalf, can evaluate the evidence without inspecting the provider's internal systems. 2. Within the threat model, evidence cannot be forged or silently altered by registries, mirrors, caches, or deployment systems without detection by verification. 3. The approach scales across large volumes of artifacts and consumers. 4. Verification is efficient enough for ordinary software consumption, not only for manual audits. 5. The approach tolerates different languages, dependency managers, toolchains, operating systems, and artifact formats. The desired outcome is build provenance that is cryptographically verifiable. For each artifact, the provider produces a record of how it was built (the source it came from, the dependencies and toolchain it used, the environment it ran in) and binds that record to the artifact with hardware-rooted cryptographic evidence. A consumer checks the evidence directly. If any of the relevant inputs, the build process, the environment, or the artifact itself differs from what the record states, verification fails. ### 1.3 Scope This paper describes the threat model and architecture of Kettle, a build tool developed by Confidential.ai to create and verify attested builds. It covers the technologies that together make attested builds trustworthy, describes the evidence chain from source inputs to output artifacts, and presents Kettle as an open-source implementation of that system. It does not attempt to prove that source code is safe, that dependencies are free of backdoors, or that a compiler is semantically correct. Attestation proves provenance and build-process integrity, not software intent. --- ## 2. Background ### 2.1 Software Builds In the context of this paper, a software build is any transformation of inputs into outputs. Building can include preprocessing, compilation, linking, copying, packaging, or any kind of file generation. The specific aspects of a build that matter when creating builds that offer high confidence that the stated inputs map to a given output are: - Inputs: source repositories, dependencies, toolchains, configuration files - Process: build scripts run, tool options, environment variables - Environment: operating system, kernel and its parameters, filesystems and other devices - Outputs: binaries, containers, tarballs, or other resulting artifacts ``` ┌────────────────┐ │ Source │ ───╮ └────────────────┘ │ │ │ │ ┌────────────────┐ │ │ Deps │ ───┤ └────────────────┘ │ │ ┌──────────────────┐ ┌────────────┐ ├──────►│ Build execution │───────────►│ Artifact │ │ └──────────────────┘ └────────────┘ ┌────────────────┐ │ │ Toolchain │ ───┤ └────────────────┘ │ │ │ │ ┌────────────────┐ │ │ Environment │ ───╯ └────────────────┘ ``` *Figure: A build is a many-input transformation. Source identity is necessary but incomplete because dependencies, toolchains, build scripts, and execution state also affect the final bytes.* ### 2.2 SLSA Software Provenance As described by The Linux Foundation, "SLSA is a specification for describing and incrementally improving supply chain security, established by industry consensus".[2] Among other things, the SLSA specification offers a concrete format for **provenance**, which is defined as verifiable information that can be used to track an artifact back, through all the moving parts in a complex supply chain, to where it came from. It is the verifiable information about software artifacts describing **where, when, and how** something was produced.[3] By using the SLSA v1.2 build provenance format, Kettle is able to provide industry-standard and machine-readable information that connects an artifact back to the source and tools that were used to create that artifact. ### 2.3 Trusted Execution Environments (TEEs) A Trusted Execution Environment (TEE) is a hardware-enforced isolation boundary that provides four security properties: - **Confidentiality.** Private memory within the TEE is encrypted at the hardware level. Memory pages are encrypted with keys managed by the CPU's secure processor, so the hypervisor, host operating system, and DMA-capable devices cannot decrypt private TEE memory contents through software access. - **Integrity.** Once software is initialized in a TEE, modifications to protected guest state by external software are detected. Replay attacks, data corruption, and memory remapping trigger hardware-level faults within the platform's threat model. - **Verifiability.** The TEE produces a cryptographic measurement (a digest) of the software loaded at launch. This measurement covers firmware, kernel, initial filesystem state, and user space, and is computed by the CPU's secure processor before the guest operating system begins execution. - **Attestation.** The TEE generates signed attestation reports binding the launch measurement to guest-supplied report data. These reports are signed by keys that chain to the hardware manufacturer's root of trust and cannot be forged by any software on the host. ``` Outside: Hypervisor Host OS BIOS Device drivers Cloud operator ╔═══════════════════════ CVM boundary ════════════════════╗ ║ ║ ║ ┌────────────────────────────────┐ ╮ ║ ║ │ Firmware │ │ ║ ║ └────────────────────────────────┘ │ ║ ║ ┌────────────────────────────────┐ │ ║ ║ │ initrd │ │ ║ ┌──────────────────┐ ║ └────────────────────────────────┘ │ ║ │ Attestation │ ║ ┌────────────────────────────────┐ │ ║ │ report │ ║ │ Kernel │ ├──── digest ────╫─────►├──────────────────┤ ║ └────────────────────────────────┘ │ ║ │ Measurement │ ║ ┌────────────────────────────────┐ │ ║ │ Signature │ ║ │ User space │ │ ║ └──────────────────┘ ║ └────────────────────────────────┘ │ ║ ║ ┌────────────────────────────────┐ │ ║ ║ │ Deployed workload │ ╯ ║ ║ └────────────────────────────────┘ ║ ║ ║ ╚═════════════════════════════════════════════════════════╝ ``` **Implementations.** Current commercial TEEs from AMD [5] and Intel [6] implement all four properties at the boundary of a virtual machine. The resulting environments are referred to as Confidential Virtual Machines (CVMs), where private guest memory is encrypted with keys the hypervisor never possesses. The four properties reinforce one another. Host software can no longer directly read private CVM memory or silently rewrite measured guest state, which prevents software-level data exfiltration and tampering by the host. As long as the hardware manufacturers can be trusted and firmware vulnerabilities can be patched as they are found, the trust surface contracts to the CPU vendor and the cryptographic primitives the platform relies on, replacing administrators, hypervisors, and host operating systems on the trusted side of the boundary. --- ## 3. Prior Art ### 3.1 Reproducible Builds Reproducible builds address software provenance by making the build itself deterministic. A build is reproducible when the same source code, dependencies, toolchain, build instructions, and environment produce bit-for-bit identical output across separate machines. The goal is to make the relationship between declared inputs and output bytes behave like a stable one-to-one mapping for verification. A given input set should always produce the same artifact, and a matching artifact digest should identify the input set that produced it. Under that condition, a consumer does not need to trust the original build machine or its operator. The consumer, or some independent rebuilder, can rebuild the artifact and compare digests. If the locally produced digest matches the distributed artifact digest, the artifact corresponds to the published inputs. ``` Declared build inputs source, deps, toolchain, environment │ ┌───────────────┴───────────────┐ │ │ ▼ ▼ ┌──────────────────────┐ ┌──────────────────────┐ │ Provider build │ │ Independent │ │ original system │ │ rebuild │ └───────────┬──────────┘ └───────────┬──────────┘ │ │ ▼ ▼ ┌──────────────────────┐ ┌──────────────────────┐ │ Artifact digest A │ │ Artifact digest B │ └───────────┬──────────┘ └───────────┬──────────┘ │ │ └───────────────┬───────────────┘ ▼ ┌────────────────────┐ │ Accept if A = B │ └────────────────────┘ ``` *Figure: Reproducible builds verify provenance by independent reconstruction. The original build and the rebuild must converge on the same output digest from the same declared inputs.* A number of projects work on making this practical across the open source ecosystem. One of them is the Reproducible Builds project [1], which supports reproducibility efforts across many parts of that ecosystem. Lamb and Zacchiroli [2] survey the state of reproducibility across the Debian distribution and discuss its role in supply-chain integrity, including resistance to build-time tampering, independent verification of binaries, and reduced trust in build infrastructure. The limitation is practical rather than conceptual. Many production builds are not deterministic. Compilers embed timestamps, parallel builds produce outputs in varying order, linkers record full file paths, archive tools preserve inconsistent ordering, and generated files may depend on host state. Achieving bit-for-bit identical output requires controlling these factors across the entire dependency tree. A single non-reproducible component anywhere in the chain breaks the guarantee. Mapped against the constraints in §1.2, this is where reproducible builds run into trouble. Achieving bit-for-bit determinism across an arbitrary set of languages, package managers, and toolchains is hard, which strains constraint 5 (toolchain coverage). ### 3.2 Nix and Reproducible Build Environments Nix [3] is a package manager and build system that describes software builds as pure functions of their inputs, captured in files called derivations. Rather than requiring every build output to be bit-for-bit reproducible, Nix makes the build environment itself reproducible. A derivation or flake lock identifies the source, dependencies, toolchain, and build recipe used to construct an output, and Nix uses those identifiers to build the same software in the same way on any machine. This is a substantial improvement over package managers where dependency resolution can vary across machines or over time. The distinction matters. A reproducible build environment does not by itself imply reproducible build output. The same pinned source and compiler can still produce different artifacts if the compiler, linker, archive tooling, generated files, or build scripts contain non-determinism. Nix can support reproducible builds when the underlying toolchain is deterministic, but the reproducibility property comes from the combination of a controlled environment and deterministic build behavior, not from environment pinning alone. Nix also leaves a trust question at the build boundary. A derivation hash proves consistency with a particular Nix evaluation, but it does not prove that the evaluator, builder, or local toolchain executed honestly. A compromised build host can still alter the build process or output after evaluation. Nix therefore provides strong input and environment control, and can be part of a provenance strategy, but it does not by itself provide hardware-rooted evidence that a particular artifact was produced by a particular build execution. ### 3.3 Attested Builds An attested build is one that runs inside a Trusted Execution Environment and emits hardware-signed evidence about the environment, inputs, and outputs observed during the build. Verification then becomes a check of that evidence (signature, environment measurement, and digest comparisons) rather than a re-execution of the build itself. The trust anchor moves from the build infrastructure and its operators to the TEE hardware and its attestation chain. Two prior systems instantiate this idea concretely. **TEE Compile**, developed by Automata Network [4], runs a project's build inside an AWS Nitro Enclave. A worker process inside the enclave fetches dependencies through the host, builds the project, and emits a tuple of artifact bytes, an input/output hash report, and a Nitro attestation document whose PCR0 covers the enclave image. Verification compares input and output hashes against the report and checks that the enclave image is one the verifier trusts. The trust root is the AWS Nitro Attestation PKI rather than a CPU vendor's root, and Nitro Enclaves are constrained micro-VMs rather than general-purpose confidential VMs. How a verifier independently obtains a trustworthy reference measurement for the enclave image itself is not specified. **Hugenroth, Lins, Mayrhofer, and Beresford** [5] generalize the approach to AMD SEV-SNP, Intel TDX, and AWS Nitro Enclaves alike. Their design treats the host and the build process as untrusted, and introduces an integrity-protected Enclave Client inside the TEE. That client records the repository snapshot hash before the untrusted build starts, runs the build inside an inner sandbox built on `containerd` and `gVisor`, records the produced artifact hash, and requests a TEE attestation covering the launch measurement, source snapshot hash, and artifact hash. The sandbox matters because the build process itself may execute arbitrary project code: the TEE protects the Enclave Client from the host, and the sandbox protects the Enclave Client from the build it is observing. The authors acknowledge the bootstrap problem, that a verifier needs to know the expected launch measurement of a genuine Enclave Client image, and recommend that the very first such image be produced with reproducible builds, but defer that step to future work. This line of work changes the verification question. A reproducible build asks whether a second build produces the same bytes. An attested build asks whether a particular artifact is accompanied by hardware-rooted evidence that a measured build environment observed particular inputs and produced particular output bytes. Both approaches try to close the source-to-binary gap, but they make different tradeoffs. Reproducible builds minimize trust by requiring deterministic reconstruction, while attested builds reduce reconstruction cost by shifting trust to TEE hardware and its attestation chain. Kettle [10] builds on this prior work in three specific ways. First, the trust root is the CPU vendor's attestation chain (AMD VCEK, Intel TDX) rather than an operator-controlled PKI, so the signing identity behind every Kettle build is the same one that signs the underlying TEE attestation. Second, Kettle commits an explicit Merkle-rooted input manifest into a SLSA Provenance v1.2 document carried in a canonical in-toto Statement (§6) and binds the document to hardware via the report-data field, so the evidence format is one that existing supply-chain tooling can already consume. Third, Kettle closes the bootstrap loop that prior work defers: each Kettle release is itself reproducibly built using the Stagex deterministic toolchain [9], and the CVM image is reproducibly assembled from that Kettle binary and a published recipe (§5.3, §5.5.1, §7.3). A verifier rebuilds the image from public source, confirms that the resulting launch measurement matches the value published with the release, and adds the measurement to their allow-list. The trust question thereby reduces to source inspection plus the verifier's own toolchain, rather than trust in any image distributor. Kettle also offers an optional pre-attested confidential build flow (§5.6) for sensitive source, which has no equivalent in either system above. The longer-term goal of layering an inner sandbox on top of CVM isolation, in the spirit of [5], is described in §7.2. --- ## 4. Threat Model Every security system has a root of trust, a set of components assumed to be correct because they cannot be verified further down. Attested builds shift where the root of trust sits. In a conventional build, a verifier must trust the entire build infrastructure stack (the cloud provider's software, its operators, other tenants, and the build pipeline itself) to accept that a binary corresponds to its claimed source. Attested builds replace that trust surface with a much smaller one rooted in TEE hardware and a small set of cryptographic primitives. ### 4.1 Trust Model The trust boundary encompasses the TEE hardware and firmware and a small set of cryptographic primitives. Physical operators are legible attackers. Software-level access by an operator is untrusted, and invasive physical access can break the TEE threat model. The source repository is also trusted in the sense that attestation cannot reason about the intent of the code being built. All other parties are untrusted. ``` ┌────────────────────────────────────────────────────────────────────────┐ │ │ │ Hypervisor Host OS CI runner Cloud operator │ │ │ └────────────────────────────────────┬───────────────────────────────────┘ │ │ launches ▼ ╔═══════════════════════════ Kettle build CVM ═══════════════════════════╗ ║ ║ ║ ┌──────────┐ ┌──────────┐ ┌──────────┐ ║ ║ │ Source │ │ Deps │ │Toolchain │ ║ ║ └─────┬────┘ └─────┬────┘ └─────┬────┘ ║ ║ │ │ │ ║ ║ └─────────────────┼─────────────────┘ ║ ║ │ ║ ║ ▼ ║ ║ ┌────────────────┐ ║ ║ │Build execution │ ║ ║ └────────┬───────┘ ║ ║ │ produces ║ ║ ▼ ║ ║ ┌───────────────┴──────────────┐ ║ ║ │ │ ║ ║ │ SLSA provenance │ ║ ║ │ + attestation report │ ║ ║ │ │ ║ ║ └──────────────────────────────┘ ║ ║ ║ ╚════════════════════════════════════════════════════════════════════════╝ ``` *Figure: The host launches the CVM but cannot observe its memory. Inside the CVM, the build observes its declared inputs and produces a SLSA provenance document together with a hardware-signed attestation report that commits to it. The signed evidence is the only state the verifier later trusts.* | Trusted | Untrusted | Out of scope | | ------------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------ | | TEE hardware vendor (AMD, Intel) and signing keys | Hypervisor and host OS | Invasive physical attacks by the hardware operator | | TEE firmware (security processor microcode) | Cloud provider, datacenter, and CI operators (software access) | CPU probing, memory-bus interposers, chip decapsulation | | Cryptographic primitives (AES, ECDSA, SHA) | Other tenants on the same hardware | Physical fault injection against the CPU or memory subsystem | | Source repository and package registries (for identity, not intent) | Build orchestration, artifact storage, distribution | | | | Network infrastructure | | Two of these entries are subtle and worth elaborating. **Physical operators.** Whoever physically hosts the hardware can attack the TEE using active probing of the memory bus, hardware interposers placed between the CPU and memory, fault injection, or chip decapsulation. Those attacks can break the guarantees of the TEE. They require physical possession of the running machine and specialized equipment, and are outside Kettle's threat model. Software-level access by the same operator (administrators, datacenter technicians, monitoring agents, employees acting through cloud APIs) is untrusted and is in scope. The asymmetry is deliberate. The TEE threat model is scoped against software adversaries with arbitrary host privileges, not against an attacker with invasive physical access. **Source and package registries.** The source commit and any registry-fetched dependencies are trusted only for *identity*. The attestation proves that a specific commit and specific packages were used. This prevents package alteration or replacement, and provides a verifiable audit trail, but does not prove that any of them are free of backdoors. ### 4.2 Attacks Addressed Protection comes from two sources. The first is TEE isolation during the build itself. The second is cryptographic binding between inputs, the build process, and outputs. - **Tampering during the build.** An attacker who compromises the build platform and injects malicious behavior during compilation is normally undetectable. The source is clean but the binary is backdoored. Running the build inside a TEE places the environment outside the host's reach, and the attestation report binds the build to the exact code that was loaded. Any modification to the build process changes the launch measurement and causes verification to fail. - **Tampering after the build.** An attacker with access to artifact storage could substitute a malicious binary for the legitimate one. The provenance produced by an attested build records the digest of every output artifact, and a SHA-256 digest of the provenance document is in turn committed to the TEE attestation report-data field. A substituted artifact fails the digest check against the provenance, and a forged provenance document fails the digest check against the report-data field of the hardware-signed attestation. - **Forged provenance.** Without a hardware root of trust, provenance is a claim. Anyone with a signing key can produce a JSON document asserting that a binary came from a particular source. Attested provenance is bound to a hardware-signed attestation report by committing a SHA-256 digest of the document into the report-data field, with the report itself signed by the TEE platform's attestation key (the AMD VCEK on SEV-SNP, the equivalent on Intel TDX). The certificate chain terminates at the CPU vendor's root of trust, so forging provenance requires producing a valid hardware-signed attestation report whose report-data matches the forged document, which in turn requires compromising the TEE hardware itself. - **Dependency substitution.** A pinned dependency silently swapped at build time fails the checksum recorded in the lockfile and is rejected before the build starts. The provenance records the resolved version and digest of every dependency, so post-hoc substitution is also detectable by any verifier. ### 4.3 Attacks Not Addressed The following are architectural constraints rather than implementation gaps. Stating them explicitly clarifies the actual security posture provided by attested builds. - **Malicious source code.** Attested builds verify that a specific source was used, not that the source is safe. If the upstream repository contains a backdoor, the attested build faithfully reproduces the backdoor and the provenance accurately records the commit that introduced it. Attestation proves identity, not intent. - **Compromised upstream dependencies.** Dependencies fetched from registries are trusted for identity, not for the trustworthiness of their contents. If a registry publishes a malicious package, an attested build that consumes that package produces a verifiable record that the malicious version was used. It does not detect that the package is malicious. - **TEE hardware or firmware compromise.** The trust anchor is the CPU vendor's hardware and key management. If those are compromised, the guarantees break. Firmware vulnerabilities have been found in TEE implementations before and will be found again. The trust surface is substantially smaller than that of a conventional build pipeline, but not zero. - **Invasive physical attacks.** A physical operator with possession of the machine can attack the CPU package, memory bus, or memory subsystem directly. These attacks can break TEE confidentiality or integrity and are outside the software-adversary model Kettle relies on. - **Side-channel attacks.** TEEs share microarchitectural state with the host. Cache timing, branch prediction, and other side channels can leak information. Vendor mitigations have raised the bar significantly, but side channels remain an active research area. Attested builds reduce, but do not eliminate, information leakage during compilation. - **Availability.** The host controls whether the build runs at all. It can refuse to schedule the TEE, terminate it mid-build, or disrupt network connectivity. Attested builds protect confidentiality and integrity, not availability. - **Bugs in the built software.** The TEE protects the execution environment, not the code running in it. If the built application contains a vulnerability, an attacker can exploit it. Attestation proves which code was loaded, not that the code is correct. --- ## 5. Architecture A Kettle build moves a project from "source on a developer's machine" to "artifact accompanied by hardware-rooted evidence" without trusting anything between those two endpoints. The system has three logical pieces. The **client** provides a random nonce and a version control reference (such as a git repository and commit ID). The **host** schedules the build but is otherwise untrusted. The **Kettle CVM** is a confidential virtual machine that boots from a reproducible Kettle image, runs the build inside the TEE, and emits an evidence bundle verifiable against the CPU vendor's root of trust. ``` Developer machine Build host ┌──────────────────────────────┐ ┌─────────────────────────────────────┐ │ │ │ │ │ Client │ │ launches │ │ │ │ │ │ │ Commit inputs to VCS │ │ ▼ │ │ │ Build request │ ╔═══════════════════════════════╗ │ │ ├────────────────────►│ ║ Kettle CVM ║ │ │ │ │ ║ ║ │ │ │ Attestation │ ║ Measured boot ║ │ │ │◄────────────────────┤ ║ Verify inputs ║ │ │ Verify launch measurement │ │ ║ Generate provenance ║ │ │ Check vendor signature │ │ ║ Run build ║ │ │ │ │ ║ ║ │ │ │ │ ║ ┌─────────────────────┐ ║ │ │ │ │ ║ │ TEE processor signs │ ║ │ │ │ │ ║ │ with vendor key │ ║ │ │ │ │ ║ └─────────────────────┘ ║ │ │ │ │ ╚═══════════════╤═══════════════╝ │ │ │ │ ▼ │ │ │ │ ┌─────────────────────────────┐ │ │ │ Bundle │ │ artifact │ │ │ │◄────────────────────┤ │ + provenance.json │ │ │ Verify bundle │ │ │ + evidence [attestation] │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ └──────────────────────────────┘ └─────────────────────────────────────┘ ``` *Figure: Kettle build at a glance. The client commits the input and requests a build. The build runs inside a freshly-launched CVM, which generates a provenance document and emits an attestation signed by the TEE processor with a key chained to the CPU vendor root. The bundle returned to the client is verifiable locally against that root.* The result is a single bundle (the artifact, `provenance.json`, and the raw attestation `evidence`) that any verifier can check locally. The remainder of this section describes each part of that flow in detail. ### 5.1 Build Flow An attested build runs across four actors. These are the developer's CLI, the host (or CI runner) that schedules the work, the Kettle CVM that performs the build, and the TEE hardware that anchors the attestation. The host can start, stop, and route IO for the CVM, but cannot observe its memory. Everything between the CVM boot and the emitted evidence is shielded by the hardware boundary. ``` Developer CLI Host Kettle CVM TEE hardware │ │ │ │ │ 1. Create VCS commit │ │ │ │ for build │ │ │ │ │ │ │ │ 2. Submit request │ │ │ │ and build nonce │ │ │ ├──────────────────────►│ │ │ │ │ │ │ │ │ 3. Launch CVM │ │ │ ├──────────────────────────►│ │ │ │ │ │ │ │ │ 4. Measured boot │ │ │ │ request launch digest │ │ │ ├──────────────────────────────►│ │ │ │ │ Extend measurement │ │ │ Launch digest │ │ │ │◄──────────────────────────────┤ │ │ │ │ │ │ │ 5. Fetch repo, commit, │ │ │ │ dependencies │ │ │ │ │ │ │ │ 6. Run build, │ │ │ │ Digest outputs │ │ │ │ │ │ │ │ 7. Assemble SLSA 1.2 │ │ │ │ provenance JSON │ │ │ │ │ │ │ │ 8. Request attestation │ │ │ │ report_data = │ │ │ │ SHA-256(provenance.json) │ │ │ ├──────────────────────────────►│ │ │ │ │ Sign with vendor key │ │ │ Signed attestation report │ │ │ │◄──────────────────────────────┤ │ │ │ │ │ │ 9. Emit bundle │ │ │ │◄──────────────────────────┤ │ │ │ │ │ │ 10. Return bundle │ │ │ │◄──────────────────────┤ │ │ │ │ │ │ ``` *Figure: Kettle build sequence. The developer creates a VCS commit and submits the build request to the host (1–2). The host launches the CVM (3). Inside the CVM, measured boot extends the launch digest in TEE hardware (4). Kettle fetches the repo, commit, and dependencies, then (5) runs the build and digests outputs (6), and assembles the provenance document as SLSA 1.2 build provenance JSON (7). It then requests an attestation report whose `report_data` field contains `SHA-256(provenance.json)` and the build request nonce. (8). The hardware signs the report with the platform attestation key. The artifact and evidence are emitted back to the host (9) and returned to the developer (10).* §5.2 describes the input manifest committed to the provenance document. §5.3 covers measured boot and the attestation report. §5.4 covers provenance generation, build execution, output digesting, and how the provenance is committed to the attestation. §5.5 specifies the verification procedure. §5.6 describes an optional confidential variant of the flow above, in which the requester pre-attests the CVM and sends the source over a TLS channel terminated inside it. The verifier-facing chain is unchanged. ### 5.2 Input Merkle Tree Before any build starts, the project's dependencies and source state are locked. This step runs locally, outside the TEE, and produces the input manifest that the CVM will consume. **Locking dependencies.** The project must carry a lockfile that pins every dependency to a specific version and a cryptographic digest. This is standard practice in modern package managers, and the lockfile captures the exact dependency graph at a point in time. Ecosystems without first-class lockfiles require the developer to supply an equivalent pinned manifest of resolved versions and digests as part of the build configuration. **Input enumeration.** Kettle walks every build input and computes its cryptographic digest: - Source code: the git commit identifier, tree digest (a content-addressed digest of the file tree), and repository signature where present. - Dependencies: each package identified by name, version, and digest taken from the lockfile. - Toolchain: digests of compiler and build-tool binaries. Cached artifacts are checked against their expected digests at this stage, and any mismatch fails the build before the CVM is launched. **Merkle tree construction.** All input digests become leaves in a Merkle tree. The tree is constructed in a deterministic order, with git information first, then the lockfile digest, then dependencies in lexicographic order by name, and toolchain digests last. The ordering is fixed by convention so that any third party can reconstruct the same tree from the same inputs. Kettle uses SHA-256 for both leaf and node digests. The Merkle structure (fixed-size inputs concatenated with explicit length prefixes at each internal node) does not admit length-extension attacks against the underlying digest. ``` input_merkle_root │ ┌─────────────┴─────────────┐ h₀₁ h₂₃ ┌──────┴──────┐ ┌──────┴──────┐ h₀ h₁ h₂ h₃ │ │ │ │ Source Lockfile Deps Toolchain commit digest digests digests ``` *Figure: Input manifest as a Merkle tree. Each declared input is digested into a leaf, leaves are paired and hashed into internal nodes, and the procedure repeats until a single root remains. The root is recorded in the provenance document, whose digest is committed to the attestation report. Any change to any input bubbles up and changes the root.* The root of this tree is a single digest that uniquely identifies the complete set of build inputs. Any change to any input, even of a single byte, changes the root. The Merkle structure also enables selective disclosure. To prove inclusion of a specific dependency without revealing the rest of the input set, the builder publishes the path from that dependency's leaf to the root. The verifier checks the path against the published root and confirms that the dependency was part of the build. Selective disclosure matters most when the dependency set is itself sensitive, for example a vendor that does not wish to enumerate proprietary internal packages to an external auditor. **Optional developer signature.** The manifest may be signed locally before submission. The signature does not strengthen the cryptographic chain (the attestation report and provenance signature already pin inputs to outputs), but it adds an identity-bound declaration that a specific developer or release engineer approved this set of inputs. Verifiers that require human accountability check the signature, while verifiers that do not can ignore it. Manifest signing is orthogonal to the TEE-rooted evidence and protects against neither a compromised developer machine nor a malicious developer. ### 5.3 TEE Environment Setup The build environment boots inside a TEE with measured boot. From this point onward, every step of the build is bound to evidence rooted in hardware that no software on the host can forge. Concretely, Kettle currently targets confidential VMs implemented by AMD SEV-SNP [5] and Intel TDX [6]. The architecture below is described in vendor-agnostic terms but maps directly onto the launch-measurement and attestation primitives those platforms expose. **Measured boot chain.** As the TEE initializes, the hardware measures every component that loads. The security processor computes a digest of each component and extends it into a cumulative measurement register, where the extension operation is `new_measurement = digest(old_measurement || component_digest)`. Extension is one-way. A measurement register cannot be "un-extended", so the final value depends on exactly what was loaded and in what order. The measurement chain covers: | Component | What gets measured | | --------- | ------------------------------------------------------------- | | Firmware | UEFI/boot code that initializes the TEE | | Kernel | Kernel image and the kernel command line | | Initrd | Initial RAM filesystem used to bring up the build environment | | VM image | The hardened operating system image | | Kettle | The attested build orchestrator | Including the kernel command line in the measurement is important. A parameter passed at boot can change kernel behavior (security features, mount options, init binary) without altering the kernel image itself. A measurement chain that omitted the command line would let the host silently change those parameters between builds. Kettle itself is part of this initial measurement. The resulting attestation report identifies the environment as a TEE running a specific Kettle release at a known cumulative measurement, so a verifier can confirm that the entity producing the report is a genuine instance of Kettle rather than arbitrary code making the same claim. **Isolation as the primary defense.** By requiring a CVM, Kettle inherits hardware-enforced isolation rather than relying on container-level boundaries. Root access inside the CVM grants no visibility into other build jobs, the host, or other tenants on the same physical hardware, and each build runs in its own fresh CVM, so cross-build contamination is structurally precluded. The base VM image contains only the components required to run Kettle and execute builds, and is built deterministically so that its measurement is predictable across releases. **Reproducible CVM image.** Because the CVM image itself does not depend on the inputs of any individual build, it can be produced once per Kettle release as a fully reproducible artifact. Its digest, and the cumulative launch measurement that the platform will report when a CVM boots from it, are public, stable, and tied to a specific Kettle version. Anyone can rebuild the image from the published source and confirm that the resulting launch measurement matches the value pinned in their allow-list (§5.5.1). This is the foundation that lets a verifier recognize a "genuine Kettle build CVM" without trusting the operator that hosts it, and it is what enables the optional confidential-input flow described in §5.6. **Defence in depth (roadmap).** Several additional hardening layers complement CVM isolation but are not yet implemented in the current Kettle release. These include a mandatory access control policy applied to the build process, system-call filtering that allow-lists only the syscalls needed for compilation and file I/O, and severing network access at the kernel level once inputs are loaded. They are discussed alongside other planned work in §7.2. **Loading inputs.** The CVM consumes inputs in one of two ways. In the simpler mode, Kettle fetches each declared dependency over the network from inside the CVM and only then severs network access. In the stronger mode, all inputs are bundled with the source archive and become part of the initial CVM image, so they are themselves covered by the launch measurement. This eliminates network trust during the build at the cost of a larger image. In both cases, no input is used unless it is part of the commit from the build request, and any mismatch aborts the build before any project code runs. **Provenance binding via attestation.** Kettle binds the provenance document to hardware by committing its digest into the attestation report's report-data field. The attestation report itself, signed by the platform attestation key, is the signature on that document. The platform attestation key is provisioned by the hardware vendor and held by the CPU's security processor, so the signing identity behind every Kettle build chains directly to the vendor root of trust. Forging Kettle provenance requires forging an attestation report from genuine TEE hardware. **Attestation report.** Once the build has run and the provenance document has been assembled (§5.4), Kettle requests an attestation report from the platform. The report contains the cumulative launch measurement, platform information (CPU model, firmware version, enabled security features), and a *report-data* field that the guest fills in. Kettle places a SHA-256 digest of the provenance document and the one-time build nonce into the *report-data* field, ``` report_data = SHA-256( provenance_document ), nonce ``` where `provenance_document` is the canonical-JSON serialization of the in-toto Statement described in §6.1, and the digest covers everything in that document including the input Merkle root, the resolved-dependency list, the output artifact digests, the build metadata, and the build-request nonce supplied by the requester. SEV-SNP and TDX expose a 64-byte report-data field. The SHA-256 digest occupies the leading 32 bytes, and the remaining bytes carry the build-request nonce. The report signature chains to the CPU vendor's root certificate, so a verifier can confirm in one signature check that the report was produced by genuine TEE hardware, that the measured environment matches an expected Kettle build configuration, and that this report corresponds to one specific provenance document. Any change to the provenance document changes its digest and therefore the report-data the platform signed over. ``` ┌──────────────────────────── Attestation report ────────────────────────────┐ │ │ │ measurement : extend[firmware, kernel+cmdline, initrd, VM image, │ │ Kettle] │ │ │ │ report_data : SHA-256( provenance_document ) │ │ Build request nonce │ │ │ │ platform_info : { cpu_model, firmware_version, ... } │ │ │ │ signature : platform attestation key │ │ │ └────────────────────────────────────────────────────────────────────────────┘ ``` *Figure: Measured boot extends each loaded component into a single cumulative launch measurement, which becomes the `measurement` field of the attestation report. The `report_data` field carries a SHA-256 digest of the canonical provenance document and the nonce from the build request. The whole report is signed by the platform attestation key. The attestation report is therefore the signature on the provenance, with no additional in-guest key required.* The attestation report is the bridge between software claims and hardware proof. The VCS commit determines which inputs will be used, and the attestation proves that a measured Kettle environment used those inputs, ran the build, and produced exactly the provenance document the report-data digest commits to, all within a single hardware-signed object. ### 5.4 Build Execution and Provenance With the environment measured and inputs verified, the build executes inside the TEE. The output of this stage is a provenance document whose digest is then committed to hardware via the attestation report described in §5.3. **Build execution.** Kettle invokes the project's existing build toolchain according to the build configuration. Compilation, linking, and packaging proceed exactly as they would on any developer machine, and Kettle does not modify the build process, only the environment it runs in. CVM isolation remains active throughout, and any of the in-CVM hardening layers planned in §7.2 will apply at this stage when introduced. **Output digests.** On completion, Kettle computes a SHA-256 digest over each output artifact, and these digests are recorded in the provenance document. Because digest calculation happens inside the TEE, the output digests inherit the same isolation guarantees as the build itself, so a malicious host cannot substitute artifacts between the build finishing and the digests being recorded. **Provenance assembly.** Kettle assembles the provenance document inside the CVM. The document is an in-toto Statement carrying an SLSA Provenance v1.2 predicate (§6) and contains the input manifest (Merkle root and all input digests), the digests of all output artifacts, build metadata, and the build-request nonce supplied by the requester. Kettle then serializes the document under a canonical JSON encoding so that any verifier can recompute the exact byte sequence the platform will sign over. The SHA-256 digest of that canonical serialization is what Kettle places in the report-data field of the attestation request. **Hardware-bound provenance.** Provenance integrity comes from the attestation report itself. Once the platform returns a signed report whose report-data matches `SHA-256(provenance_document)`, the document is bound to the measured Kettle environment. A verifier reconstructs the chain by recomputing the digest from the provenance document, checking it against report-data, and validating the report signature against the CPU vendor's root. **Build outputs.** A complete build emits three evidence artifacts alongside the built outputs. `provenance.json` is the SLSA v1.2 build record carried inside an in-toto Statement, recording inputs, output digests, and build metadata, in the canonical JSON form whose SHA-256 digest the attestation report committed to. `evidence.json` includes the TEE attestation report whose report-data field carries the provenance digest and build nonce. Together with the source commit, these three artifacts form the complete evidence chain. The provenance binds inputs to outputs, and the attestation binds the whole document to a measured TEE for a specific build request. §6 specifies the format of these artifacts in detail. ### 5.5 Evidence Chain and Verification The result is a cryptographic chain from declared inputs to artifact bytes. Verification walks that chain in order, checking each binding. Each check is meaningful only if the previous check passed. ``` ┌────────────────┐ │ Vendor root │ └───────┬────────┘ │ │ Step 1: Report signature chains to vendor root, │ launch measurement is allow listed ▼ ┌────────────────┐ │ Attestation │ │ report │ └───────┬────────┘ │ │ Step 2: SHA-256(provenance.json) │ matches report_data ▼ ┌────────────────┐ │ Provenance │ └───────┬────────┘ │ │ Step 3: Expected digest of artifact │ ▼ ┌────────────────┐ │ Artifact │ │ bytes │ └───────┬────────┘ │ │ │ ▼ ┌──────────────────────────┐ │ Accept if all pass │ │ Reject if any fails │ └──────────────────────────┘ ``` *Figure: Evidence chain and verification. Each link binds to the next, and each numbered check verifies one link. Verification is local. Consumer tooling reads the artifact, `provenance.json`, and the raw `evidence` blob from a single bundle and runs the checks in order, with no remote service contacted.* | Step | What consumer tooling checks | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Step 1: Verify the TEE attestation | The attestation signature chains to the hardware vendor's root of trust, the cumulative launch measurement matches an allow-listed Kettle measurement (§5.5.1), and the freshness nonce embedded in the provenance matches the build-request nonce. | | Step 2: Verify the provenance binding | The verifier recomputes `SHA-256(provenance.json)` over the canonical serialization of the provenance document and checks that it matches the leading 32 bytes of the report-data field. This is the step that turns the attestation report into a signature on the provenance. | | Step 3: Verify the artifact binding | The digests of the distributed output artifacts match the digests recorded in the provenance. | Each link in the chain rejects a corresponding attack: | Attack | Mechanism that detects it | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Source modified after commit | The tree digest changes, the Merkle root changes, and the provenance file no longer matches the published build. | | Dependency substituted before the build | The dependency's digest does not match the lockfile entry, so input verification fails before the build starts. | | Build machine compromised | The build runs inside the TEE, so the host cannot read memory or inject code. Loading different inputs changes the provenance commitment or causes input verification to fail. | | Malicious Kettle binary | The launch measurement differs from the expected Kettle measurement, and the verifier rejects the attestation. | | Output artifact swapped after build | The artifact digest no longer matches the digest recorded in the provenance. | | Provenance forged | The SHA-256 digest of the forged provenance does not match the report-data field of the hardware-signed attestation report, so the binding check at step 2 fails. | When all checks pass, consumer tooling has cryptographic proof that the artifact in hand was produced by a specific build of Kettle running inside attested TEE hardware, over a specific list of inputs, and that every link in that chain is independently verifiable. Verification consists entirely of digest comparisons and one hardware-attestation signature check against the CPU vendor's root, all of which are local operations. #### 5.5.1 Measurement Allow-Lists Step 1 above presupposes that a verifier can recognize an allow-listed Kettle measurement. In practice the cumulative launch measurement covers firmware, kernel and command line, initrd, VM image, and Kettle itself, so an allow-list entry is not a single value but a set of acceptable measurements, one per signed Kettle release. A verifier maintains (or imports) a list of these measurements together with the Kettle version, target platform (SEV-SNP, TDX), and any required platform-firmware level it implies. Verification policy is then expressible as "accept attestations whose measurement appears in the allow-list at version `v` or later, on platform `p` with firmware `≥ f`." The allow-list itself needs trustworthy origin. Because each Kettle release ships as a reproducible CVM image (§5.3), its launch measurement is recomputable from the published source. A verifier, or any independent observer, rebuilds the image and confirms that the resulting measurement matches the value published with the release before adding it to their allow-list. Verifiers that do not wish to track Kettle releases directly can delegate to a third party that publishes a signed allow-list. ### 5.6 Confidential Builds The flow described so far yields *integrity* at the hardware level. A verifier can be sure that the bytes claimed as inputs were the bytes the build observed. It does not, on its own, hide those inputs from the host. In the simple flow the host receives the source archive in plaintext from the developer's CLI and hands it to the CVM at launch, so a malicious operator can still inspect or copy it before the CVM boots. When the source is itself sensitive (proprietary code, customer data, an unreleased product) the requester can take the additional step of refusing to send any input until the CVM has proven it is a genuine Kettle release, and then sending the input over a confidential channel terminated *inside* the CVM. The reproducible CVM image (§5.3) is what makes this practical. Because the launch measurement of a Kettle release is public and stable, the requester can decide, before transmitting any source, whether the CVM they are about to talk to is one they trust. The flow has three stages and reuses the same `report_data` mechanism that already binds inputs and outputs. ``` Requester CLI Host Kettle CVM TEE hardware │ │ │ │ │ Generate nonce_p │ │ │ │ Pre launch: requester knows │ │ │ │ expected Kettle measurement │ │ │ │ from public allow list │ │ │ │ │ │ │ │ Request CVM launch with nonce_p │ │ │ ├─────────────────────────────────────►│ │ │ │ │ Launch CVM with │ │ │ │ host_data = nonce_p │ │ │ ├─────────────────────────►│ │ │ │ │ Measured boot │ │ │ ├─────────────────────────────►│ │ │ │ Launch measurement │ │ │ │◄─────────────────────────────┤ │ │ │ │ │ │ │ Generate TLS keypair │ │ │ │ inside CVM │ │ │ │ │ │ │ │ Request attestation │ │ │ │ report_data: │ │ │ │ H[tls_pubkey || nonce_p] │ │ │ ├─────────────────────────────►│ │ │ │ Signed attestation: │ │ │ │ measurement │ │ │ │ host_data = nonce_p │ │ │ │ report_data │ │ │ │◄─────────────────────────────┤ │ │ │ │ │ Pre attestation │ │ │ │ [TLS pubkey + report] │ │ │ │◄─────────────────────────────────────┼──────────────────────────┤ │ │ │ │ │ │ Verify: │ │ │ │ - signature chains to vendor root │ │ │ │ - launch measurement is Kettle │ │ │ │ - host_data equals nonce_p │ │ │ │ - report_data equals │ │ │ │ H[tls_pubkey || nonce_p] │ │ │ │ │ │ │ │ Open TLS to CVM, send │ │ │ │ encrypted source │ │ │ ├──────────────────────────────────────┼─────────────────────────►│ Decrypt inside CVM │ │ │ │ │ │ │ │ Continue with §5.3 flow. │ │ │ │ Second attestation │ │ │ │ commits SHA-256 of the │ │ │ │ provenance document │ │ │ │ │ ``` *Figure: Pre-attested confidential build. The requester generates a single nonce that the hypervisor commits into the CVM's `host_data` field at launch and that Kettle inside the CVM also includes in `report_data`. The first attestation therefore proves three things at once. The CVM is running Kettle code, it was uniquely launched for this requester, and the report itself is fresh. Source is then delivered over a TLS channel terminated inside the CVM, and the standard §5.3 attestation over the provenance document follows.* **Stage 1, pre-attestation.** The requester generates a fresh `nonce_p` and submits it with the CVM-launch request. The hypervisor writes `nonce_p` into the CVM's `host_data` field, which the platform records before the guest boots and includes in every attestation report the CVM produces. Once booted, the CVM generates a TLS keypair inside its memory and requests an attestation report whose `report_data` field carries `SHA-256(tls_pubkey || nonce_p)`. The report is returned to the requester along with the TLS public key. The TLS keypair is used only to terminate the channel inside the CVM and plays no role in signing provenance. **Stage 2, verification before transmission.** The requester checks the attestation locally. The signature must chain to the CPU vendor's root of trust, the launch measurement must match an allow-listed Kettle release on the expected platform, `host_data` must equal the `nonce_p` the requester just submitted, and `report_data` must match `SHA-256(tls_pubkey || nonce_p)` for the TLS public key just received. Each check answers a different freshness question. The launch measurement covers code identity, `host_data` covers CVM uniqueness (this is a freshly launched VM provisioned for this requester and not a replay or shared instance), and `report_data` covers attestation freshness plus the channel binding. Because the Kettle CVM image is reproducible and its measurement is published with each release, the requester can perform every one of these checks without trusting the operator that hosts the CVM. If any check fails, the requester walks away without ever revealing the source. **Stage 3, confidential delivery and build.** The requester opens a TLS session to the CVM using the verified public key as the channel endpoint, and transmits the source archive over that session. The host can observe ciphertext and the launch measurement, but cannot read the inputs. Inside the CVM, Kettle decrypts the inputs, and proceeds with the standard build flow (§5.3, §5.4). A *second* attestation report is produced after the build, with `report_data = SHA-256(provenance_document)` exactly as in the non-confidential flow. The two attestations share a launch measurement (the same Kettle CVM produced both) and chain together. The first proves the channel, the second proves the build. This flow does not change the verifier-facing evidence chain at all. Verifiers consume the same `provenance.json` and `evidence` files and run the same checks (§5.5). What it changes is the *requester-facing* trust posture. Confidentiality of the source is enforced cryptographically rather than by trusting the host, so a build can run inside infrastructure the requester would otherwise be unwilling to send code to. The confidential flow is opt-in. Public open-source builds can use the simpler path of §5.1 with no loss of integrity guarantees, and only requester with sensitive inputs incur the extra round trip and the additional pre-launch verification step. --- ## 6. Provenance Format and Standards The evidence chain of §5 is only useful if its artifacts can be consumed by tools other than Kettle. A proprietary format would put attested builds in a silo. Only Kettle's own verifier could read them, and the rest of the supply-chain ecosystem (registries, scanners, deployment systems, audit tools) would have to integrate against a custom schema. Kettle therefore expresses provenance using two existing standards. The **in-toto attestation framework** [7] provides the outer signed-statement structure, and **SLSA Provenance v1.2** [8] is the predicate carried inside it. This section describes how those formats are populated and how they map onto the SLSA Build levels. ### 6.1 in-toto Statement Layer in-toto defines a generic envelope for statements about software artifacts. A statement names a set of *subjects* (the artifacts the claim is about, identified by digest), declares a *predicateType* (the schema of the claim), and carries the claim itself in the *predicate* field. Rather than wrapping the Statement in a separate signature envelope, Kettle commits the SHA-256 digest of its canonical serialization into the attestation report's `report_data` field, so the hardware-signed report is itself the binding evidence on the Statement. The concrete shape of a Kettle Statement is shown in §6.2. Subjects are identified by content digest, not filename, so a Statement is bound to specific bytes rather than a reusable label. The predicate type is a URI identifying the schema by which the predicate field should be interpreted. The framework is deliberately extensible. SBOMs, code-review attestations, test-result attestations, and vulnerability scans all reuse the same outer structure with different predicate types, and a single artifact can accumulate multiple in-toto Statements over its lifecycle. Kettle emits one such Statement per build, with the SLSA Provenance v1.2 predicate type, and binds it to hardware via the attestation report described in §5.3. ### 6.2 SLSA Provenance Predicate The SLSA Provenance v1.2 predicate has two top-level sections: `buildDefinition` describes what was built, and `runDetails` describes how the build executed. An abbreviated example illustrates the fields Kettle populates: ```json { "_type": "https://in-toto.io/Statement/v1", "subject": [ { "name": "my-app", "digest": { "sha256": "1d1ea25c371d4f6de8d6e3c26fdad2238..." } } ], "predicateType": "https://slsa.dev/provenance/v1", "predicate": { "buildDefinition": { "buildType": "https://kettle.confidential.ai/cargo-build/v1", "externalParameters": { "repository": "https://github.com/org/repo", "ref": "refs/heads/main" }, "internalParameters": { "tee_platform": "sev-snp", "kettle_version": "0.4.0" }, "resolvedDependencies": [ { "uri": "git+https://github.com/org/repo@refs/heads/main", "digest": { "gitCommit": "a1b2c3d4..." } }, { "uri": "pkg:cargo/serde@1.0.228", "digest": { "sha256": "9a8e94ea..." } } ] }, "runDetails": { "builder": { "id": "https://kettle.confidential.ai/tee-builder/v1" }, "metadata": { "invocationId": "build-12345", "startedOn": "2026-01-15T10:30:00Z", "finishedOn": "2026-01-15T10:35:00Z" } } } } ``` Three fields carry most of the security-relevant content: - **`buildType`.** A URI that fixes the schema of `externalParameters` and `internalParameters`. Different toolchains (Cargo, Nix, Bazel) and different build configurations get different build types, so verifiers can apply build-type-specific policy without parsing free-form fields. - **`externalParameters` vs. `internalParameters`.** External parameters are inputs supplied from outside the trusted control plane (which repository to build, which ref to resolve, which entry point to invoke), and a verifier *must* check these against expectations because they are attacker-influenceable. Internal parameters are set by Kettle itself inside the CVM (target platform, Kettle release, build-time configuration) and are trusted because the platform that emits them is trusted. Pushing build configuration into the source tree (where it is covered by the source commit digest) rather than into `externalParameters` reduces what a verifier has to police. - **`resolvedDependencies`.** Every input fetched during the build, recorded as a `ResourceDescriptor` with URI, digest, and (optionally) name and download location. Dependencies use Package URLs (PURLs), e.g. `pkg:cargo/serde@1.0.228`, so verifiers can cross-reference them against registries or vulnerability databases without bespoke parsers. The distinction between `externalParameters` (what was *requested*) and `resolvedDependencies` (what was *actually fetched*) is load-bearing: a request for `refs/heads/main` resolves to a specific commit, and the resolution is what the attestation witnesses. - **`builder.id`.** A URI identifying the build platform. Crucially, this represents the *transitive closure* of everything a verifier is trusting to faithfully run the build and emit accurate provenance, including the TEE platform, the Kettle release, and the measurement allow-list policy. Verifier policy is typically expressed as "accept builds with `builder.id` `X` at SLSA Build level *L*." ### 6.3 SLSA Build Levels SLSA's Build track defines three levels of supply-chain integrity, summarised in [2]: | Level | Summary | Key requirements | | ----- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | L1 | Provenance exists | Each artifact has provenance describing how it was built. May be unsigned. Catches mistakes, trivial to forge. | | L2 | Hosted build platform | Provenance is signed by a hosted build platform. Forging requires an explicit attack rather than a configuration error. | | L3 | Hardened builds | Build platform has strong tamper-resistance. Builds are isolated from one another. Signing keys are inaccessible to user-defined build steps. | Attested builds, as implemented by Kettle, target SLSA Build L3 with hardware enforcement of the L3 requirements: | L3 requirement | How Kettle satisfies it | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Provenance generated by build platform's trusted control plane | The Kettle orchestrator runs in a separate VM on the host and launches a fresh CVM for each build, so the user-supplied build never executes alongside the control plane and cannot interfere with provenance assembly (§5.4). | | Provenance signed by build platform | The SHA-256 digest of the provenance is committed to the attestation report-data field, and the report is signed by the TEE platform attestation key (e.g. AMD VCEK) which chains to the CPU vendor's root of trust. | | Builds isolated from one another | Each build runs in its own fresh CVM, and cross-build interference is precluded by hardware memory encryption rather than by soft policy. | | Signing keys inaccessible to user-defined build steps | There is no in-guest signing key. The platform attestation key is held by the CPU's security processor and is unreachable from any guest software, including user-supplied build steps. | Hardware enforcement of these requirements is the substantive difference from a typical L3 implementation. The build platform's signing identity is not a software key sitting on a build host but the platform attestation key, held in silicon and reachable only via the platform's attestation interface. A compromised build script, a privileged operator on the host, and a malicious other tenant on the same machine all face the same wall. ### 6.4 Build Outputs Recap Each Kettle build emits three artifacts whose roles map onto the format above: - **`provenance.json`**. The in-toto Statement described above, with the SLSA Provenance v1.2 predicate, serialized in canonical JSON. Contains the source commit and tree digest, the lockfile digest, the toolchain digests, and the `input_merkle_root` over all of them. This file's SHA-256 digest is what the attestation report's report-data field commits to. - **`evidence`**. The raw TEE attestation report. The report-data field carries `SHA-256(provenance.json)` (over the canonical JSON form), and the signature chains to the CPU vendor's root of trust. This is what roots the rest of the chain in hardware. --- ## 7. Conclusion Attested builds make it possible to definitively connect a specific executable with the inputs used to create that executable without requiring bit-for-bit reproducibility. By binding exact build inputs, a measured build environment, the assembled provenance document (via its digest in the attestation report-data field), and exact build outputs into one evidence chain, a verifier can evaluate a binary without trusting the build host or re-running the build. ### 7.1 Trust Surface Comparison The shift from a conventional build pipeline to attested builds is best understood as a change in what a verifier must trust to accept that a binary corresponds to its source. In a conventional pipeline, that trust extends across the entire build infrastructure, its operators, and its distribution channel. Verification is policy-based, and a consumer of the binary trusts that procedures were followed, that access controls were configured correctly, and that credentials were not compromised at any point along the chain. In the attested model, the trust surface contracts to the TEE hardware vendor, the TEE firmware, and the source infrastructure (for input identity). The build infrastructure and distribution channel move outside the trust boundary. Physical operators are not trusted for invasive attacks. They are treated as software adversaries when they act through host privileges, cloud APIs, monitoring agents, or operator access, and as out-of-scope attackers when they physically probe or modify the machine. Verification is cryptographic. The hardware attestation is checked against the vendor root, the provenance document is checked against the digest in the attestation report-data field, and artifact digests are checked against the provenance. The smaller, well-defined trust surface replaces a broad, opaque one. | Component | Conventional build | Attested build | | ----------------------- | ------------------ | -------------------------------------------------------- | | CI runner | trusted | untrusted | | Build host | trusted | untrusted | | Release signing key | trusted | replaced by TEE attestation key (vendor-rooted) | | Registry / distribution | trusted | untrusted | | Physical operator | trusted | software access untrusted, invasive attacks out of scope | | TEE hardware | not in trust path | trusted | | Hardware vendor root | not in trust path | trusted | | Source identity | trusted | trusted | *Table: What a verifier must trust. The attested model contracts the trust surface to TEE hardware and the vendor root, while the build infrastructure and distribution channel move outside the boundary.* ### 7.2 Open Directions - **In-CVM defense in depth.** Current Kettle releases rely on CVM isolation and a minimal VM image as the primary boundary around the build process. Several additional layers, discussed in §5.3, are planned but not yet implemented: a mandatory access control policy applied to the build process so that root inside the CVM does not imply unrestricted file access, system-call filtering that allow-lists only the syscalls needed for compilation and file I/O, and severing network access at the kernel level once inputs are loaded. None of these are required for the cryptographic chain to hold, but each shrinks the attack surface available to a malicious build script or compromised toolchain *inside* the CVM. - **Inner sandbox around the build process.** A complementary direction, in the spirit of Hugenroth et al. [7], is to run the build in an inner sandbox even within the CVM, separating the trusted Kettle observer (which records inputs and produces provenance) from the untrusted project code (which compiles and links). The CVM protects Kettle from the host, and the sandbox would protect Kettle from the build it is observing. - **Digest algorithms.** The Merkle tree of inputs and the report-data commitment currently use SHA-256 with explicit length prefixes at each internal node, which precludes length-extension attacks against the underlying digest. There are no known weaknesses in this construction, but SHA-3 or Blake3 would remove length-extension as an algorithmic concern at all and provide better performance on long inputs. - **Per-file input digests.** Git digests files, trees, and commits using SHA-1, which has documented practical collision attacks (the SHAttered work and successors). Kettle reduces the practical impact by recording both the commit digest and the tree digest in the provenance, so an attacker would need to manufacture a SHA-1 collision in two coupled values simultaneously. Enumerating and digesting each file in the source repository under SHA-2, or better yet SHA-3, would close this residual exposure entirely, at the cost of additional provenance construction time on large source trees. ### 7.3 Kettle [Kettle](https://github.com/confidential-dot-ai/kettle) is an open-source implementation of attested builds [10]. By committing the SHA-256 digest of each build's provenance document into the TEE attestation report and recording every input, output, and environment measurement in an in-toto Statement carrying an SLSA Provenance v1.2 predicate, Kettle provides supply-chain assertions of comparable strength to bit-for-bit reproducible builds without requiring deterministic compilation. Kettle releases are themselves fully reproducible, built on top of the Stagex deterministic toolchain [9] and published into the project repository. Any release of Kettle can be independently verified by cloning the repo and executing the reproducible build script, creating a new binary with the same checksum. --- ## References [1]: Reproducible Builds. *Reproducible Builds project*. https://reproducible-builds.org. [2]: Lamb, Chris, and Stefano Zacchiroli. *Reproducible Builds: Increasing the Integrity of Software Supply Chains*. IEEE Software 39(2):62–70, 2022. https://doi.org/10.1109/MS.2021.3073045. Preprint: https://arxiv.org/abs/2104.06020. [3]: Dolstra, Eelco, Merijn de Jonge, and Eelco Visser. *Nix: A Safe and Policy-Free System for Software Deployment*. In Proceedings of the 18th USENIX Conference on System Administration (LISA '04), pp. 79–92, 2004. [4]: Automata Network. *Creating Attested Builds with AWS Nitro Enclaves*. AWS Builder Center, September 2024. https://builder.aws.com/content/2lZdD8iDfAGbQpWK8g8fGvQ1kp8/creating-attested-builds-with-aws-nitro-enclaves. [5]: Hugenroth, Daniel, Mario Lins, René Mayrhofer, and Alastair R. Beresford. *Attested Builds: Compiling Verifiable Binaries on Untrusted Systems using Trusted Execution Environments*. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security (CCS '25), pp. 4514–4528, 2025. https://doi.org/10.1145/3719027.3765128. Preprint: https://arxiv.org/abs/2505.02521. [6]: AMD. *AMD SEV-SNP: Strengthening VM Isolation with Integrity Protection and More*. Document 70366, 2020. https://docs.amd.com/v/u/en-US/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more. [7]: Intel. *Intel Trust Domain Extensions (Intel TDX) Module Base Architecture Specification*. Document ID 853294, 2025. https://www.intel.com/content/www/us/en/content-details/853294/intel-trust-domain-extensions-intel-tdx-module-base-architecture-specification.html. [8]: Torres-Arias, Santiago, Hammad Afzali, Trishank Karthik Kuppusamy, Reza Curtmola, and Justin Cappos. *in-toto: Providing farm-to-table guarantees for bytes and bits*. In Proceedings of the 28th USENIX Security Symposium, pp. 1393–1410, 2019. https://www.usenix.org/conference/usenixsecurity19/presentation/torres-arias. Specification at https://github.com/in-toto/attestation. [9]: Stagex. *A container-native, full-source bootstrapped, reproducible toolchain*. https://stagex.tools, source at https://codeberg.org/stagex/stagex. [10]: Lunal Dev. *Kettle: attested builds for verifiable software provenance*. https://github.com/confidential-dot-ai/kettle. --- # Confidential Blog - **24 Jul 2026**: [Testing Code You Are Not Allowed To Read](/blog/confidential-ci.md) - **14 Jul 2026**: [Open-sourcing our Confidential Virtual Machine builder](/blog/cvm-builder.md) - **13 Jul 2026**: [c8s Is Now Generally Available](/blog/c8s-ga.md) - **06 Jul 2026**: [The Isolation Spectrum: From Plaintext to TEEs](/blog/isolation-spectrum.md) - **02 Jun 2026**: [Confidential Agents: Verifiably Private Agent Runtime](/blog/confidential-agents-launch.md) - **19 May 2026**: [A New Mark for Confidential](/blog/new-logo.md) - **28 Apr 2026**: [Announcing C8s: Confidential Kubernetes](/blog/c8s-launch.md) - **27 Apr 2026**: [PrivateClaw launches on Confidential: E2E Private AI Agents](/blog/privateclaw-launch.md) - **27 Mar 2026**: [Lunal Becomes Confidential AI](/blog/lunal-becomes-confidential.md) - **19 Mar 2026**: [Kettle: Attested Builds from a TEE](/blog/kettle-attested-builds.md) - **01 Mar 2026**: [Introduction to TEEs](../docs/intro-to-tees.md) - **27 Jan 2026**: [A Confidential Computing Primer](../docs/confidential-computing-primer/) - **26 Sep 2025**: [Secure AI Needs TEEs](/blog/secure-ai-needs-tees.md) - **26 Sep 2025**: [Benchmarking CPU TEE Performance](/blog/tee-performance-cpus.md) --- # c8s Is Now Generally Available Back in April, we released our [whitepaper](/docs/resources/c8s) detailing a Confidential Kubernetes based on Trusted Execution Environments (TEEs). Today we're launching the first general availability release of our implementation into the wild. c8s allows you to run workloads such as inference, training, or processing of customer or proprietary data privately, such that your dev team, your system administrators, and your cloud provider cannot see any of the runtime data. You can prove this cryptographically. To do this, we leverage TEEs: a secure space in hardware where code executes encrypted from the host machine that it's being run on. Despite having been around for more than a decade, nobody has built convenient tooling that spans the entire stack from bare metal to end user for use and verification — until now! ## How do TEEs work? TEEs are a hardware-embedded mode for your CPU for launching confidential virtual machines (CVMs). These CVMs create a secret key on launch that they use to encrypt the memory they use, such that the host OS can't read anything in it. They also add checksums to each memory region so if the host OS tries to tamper with it, the CVM detects it and can panic. CVMs can also request a signature from the manufacturer's key burned onto their chip to attest that they launched a specific VM image inside legitimate secure hardware. External parties or software can verify these attestations before sending any sensitive data into the CVM, gaining high confidence that the data is secure. The CVM becomes a trust boundary — anything running inside it can be trusted. Anything running outside it cannot be trusted, and anything crossing the boundary must be encrypted with a key that lives inside the trust boundary. ## How does Kubernetes work? Kubernetes is a container orchestration tool that automatically provisions and scales compute resources. If we are running an online store in a cloud provider, our traffic can change drastically based on things like the time of day or whether we're running a sale. If we over-provision resources to handle the peak of our traffic, they will stand idle when traffic is low. Kubernetes solves this by spinning up and down machines on cloud providers based on traffic, usage, and other metrics. It's become the de facto standard for managing cloud infrastructure at scale and is popular for bare-metal deployments too. Typically, your cloud provider runs the control plane for Kubernetes on your behalf. This control plane decides which containers to run where. When the control plane wants to start a new container on a node, it adds the container to its state served by its API server. The kubelet on the node polls the API server and identifies a drift between the node's state and the state desired by the control plane, and starts/stops the necessary containers to reach parity. This puts a lot of power into the hands of your control plane! ## How does c8s combine these for confidentiality? In c8s, we run individual pods or entire nodes inside CVMs to create a trust boundary. Though we leave the control plane outside this trust boundary, we run a shim for the container runtime that refuses to start any container that isn't on an allowlist managed by the operator. This means that untrusted workloads can't even start inside a c8s cluster! After starting, workloads perform mutual attestation between one another to establish TLS sessions before any communication happens, so that all intra-cluster communication is encrypted. Operators verify the attestations of the core components and the state of the cluster after installation. Users interacting with the cluster form a root of trust by verifying the attestation of the [Certificate Distribution Service](/docs/c8s/concepts/trust-root) and validating the allowlist that it's enforcing inside the cluster. From that, they can create a chain of trust with other components and be sure that their data is secure and confidential during processing. ## What's the performance like? On CPU, [the performance penalty for confidential computing is already less than 5%](/blog/tee-performance-cpus) for most workloads: the CPU enters a different mode, but much of the normal operation continues. Until recently, GPU performance paid a 20% penalty for confidential operations. After a bunch of research and optimization work, our team has achieved a reduction of the overhead from 20% to 1%! We'll be releasing a paper on how we did it in the next few weeks. ## How can I use it today? The code is available in the [c8s repository on GitHub](https://github.com/confidential-dot-ai/c8s). We've provided a variety of tutorials in [our docs](/docs/c8s), such as: - [Spinning up a confidential VM on a cloud provider and installing c8s on it](/docs/c8s/tutorials/first-confidential-cluster) - [Running vLLM inside c8s and attesting its confidentiality end-to-end](/docs/c8s/tutorials/verified-chat) - [Deploying NVIDIA Dynamo inside a c8s cluster for serving distributed inference with confidentiality](/docs/c8s/how-to/deploy-dynamo) ## Licensing For launch, we've licensed the code under AGPL. This is a copyleft license and means that all installations and derivations of the work must also be licensed under the AGPL. **This is not our final license.** We wanted to get c8s into the hands of users as soon as possible, and we haven't figured out our long-term licensing strategy yet. We are sure that we want to make the license more permissive, but there are a host of different options and we don't want to make the wrong decision. If you have a use case that's blocked by AGPL in the short term, [talk to us](mailto:hello@confidential.ai) and we can figure out a temporary license that can work for you. ## Roadmap A lot of work has gone into the c8s launch, but there's still a ton of work in flight to make it even better. In a few weeks, we'll be launching a key management system that's fully integrated into c8s so you can gate secret release by attestation: only specific workloads that can prove their code and prove they're running inside secure hardware can get an API key, model weight decryption key, or other secret. It will behave like and proxy to the secrets managers you're used to, but with the cryptographic benefits of trusted execution environments. --- # C8s: Kubernetes, Made Confidential Today we're publishing the C8s architecture whitepaper. C8s is confidential computing for Kubernetes. [Read the paper](/docs/c8s-whitepaper.md) ## The problem Running sensitive workloads on third-party infrastructure is a three-sided trust problem. Artifact owners ship model weights, datasets, and code onto infrastructure they do not control. Compute providers run workloads they cannot prove are private to their customers. End users send queries that are visible to every layer of the stack. Today, all three sides rely on contracts. Licensing agreements, compliance certifications, access controls. None of it is cryptographically rooted. A privileged insider with hypervisor access can read memory, intercept traffic, and exfiltrate weights from storage. Encryption at rest and in transit closes part of the gap. Data in memory during computation is still plaintext. ## What C8s does C8s is built on hardware Trusted Execution Environments. AMD SEV-SNP, Intel TDX, NVIDIA Confidential Computing on H100 and later. Memory is encrypted with keys the hypervisor never holds. The hardware signs an attestation report that proves what code is running on what silicon. C8s composes that hardware foundation up the stack. Every pod runs inside its own confidential VM. Every connection between pods is mutually authenticated based on what code each side is running. Sensitive artifacts like model weights stay encrypted at rest and only decrypt inside a pod that proves what it is. External clients can verify the cluster's entry point before sending data. The control plane stays outside the trust boundary. That's what makes C8s deployable on AKS, EKS, and GKE without modification. ## What you get Three groups gain guarantees they did not have before. * Artifact owners can deploy weights, datasets, and proprietary code on third-party infrastructure without exfiltration risk. * Compute providers can host sensitive workloads without seeing them. * End users can submit queries opaque to every party except the attested TEE processing them. All three are cryptographically provable to any independent verifier. ## Read the paper The full architecture is in the paper. Threat model, trust boundary, component design, request lifecycle, configurable boundaries. [C8s: A Confidential Kubernetes Architecture](/docs/c8s-whitepaper.md) If you want to run private inference, protect model weights, train on sensitive data, or deploy private agents, this is the substrate. [Get in touch](mailto:hello@confidential.ai). --- # Confidential Agents Today we launch Confidential Agents, the world's first verifiably private, secure agent runtime. Both the agents and their inference run in Trusted Execution Environments (TEEs), completely and verifiably encrypted. [Sign up for the beta here](https://forms.gle/QkfCfAjvDcujZLzB6). ## The Problem Two things are true when it comes to building AI agents. 1. The more data an agent can read and modify, the more useful it becomes. 2. The more data an agent can read and modify, the more dangerous it becomes. An agent worth running has your codebase, your patient records, your trading book, your legal files. The industry's best security measures simply gate access, but every party in the agent stack—the host, the inference provider, and the software vendor—can still read that data. Consider a hedge fund running an agent over its proprietary trading book on a frontier model. It signs a privacy agreement with the inference provider, but it can't *prove* the provider never saw its alpha. In principle, all data remains accessible. Providers simply promise not to look. Confidential Agents is the first agent architecture where clients can verify that the hosting provider and inference provider all *can't* see your data. ## Closing the gap with TEEs A Trusted Execution Environment (TEE) is a hardware-enforced boundary where the hypervisor and host only ever see ciphertext. The CPU itself encrypts memory with a per-CVM (confidential VM) key, locking the hypervisor and host out. A cryptographic measurement is sent to the client, which verifies that they're accessing compute within a TEE. This closes the host threat: even with full control of the machine, the operator reads noise, and the client can verify it. Verifiability comes from the **attestation report**. The CVM (for both the agent *and* inference) generates a signed report proving that they are running in TEEs and that the software matches the audited build. The client verifies that report against AMD's and NVIDIA's public key chains *before it sends anything*. Trust is established cryptographically, up front, with a mere 4–8% hit to throughput (tokens / second). ## Securing the agentic stack Every agent has three ingredients, and each has an attack surface. Each is closed through TEEs. - **Data.** The sensitive prompts and documents that agents receive. TLS protects the data en route to your compute, but once inside, it's vulnerable to a malicious hypervisor or host that can read your instance's RAM. TEEs prevent this by encrypting RAM, so the host sees only ciphertext. - **Runtime.** The agent harness that's reading data and making tool calls. [Attested builds](/blog/kettle-attested-builds), powered by TEEs, enables you to verify what software is running so it can be audited for security purposes. - **Intelligence.** Sensitive data is fed into large language models on inference hosts. The inference provider sees your sensitive data in the form of prompts. The GPU TEE closes this gap, similarly to the CPU TEE. When all three are sealed, no party in the stack — host, inference, or software vendor — can see your data, and every claim is verifiable to the client. ## What a TEE won't do A TEE doesn't make a model smarter, and it doesn't follow your data out the door. The seal covers the agent's three ingredients—data, runtime, and intelligence—and stops exactly there. The moment your agent calls an external tool, a third-party API, or an MCP server, that data crosses the boundary by design, and whatever is on the other side may see it in plaintext. Many real agent attacks have landed here, and TEEs are not built to solve it. What confidential computing gives you is a clean, provable line: everything inside is sealed, and every step outside is a deliberate decision, made with your eyes open. ## Try it out If your agent touches data that matters, the trust boundary is paramount to the product. Confidential Agents is how you close it. [Sign up for the beta here](https://forms.gle/QkfCfAjvDcujZLzB6), or visit our API docs [here](/docs/confidential-agents-api) to learn more. --- # Testing Code You Are Not Allowed To Read *Most of our components need a Trusted Execution Environment (TEE) to function, so our test suites cannot run on ordinary CI. We built infrastructure that runs them inside confidential VMs, in four environments, on every merge. The same machinery lets one party run tests against another party's code without either side handing over its secret.* --- Most of what we build doesn't work on an ordinary machine. Our attestation library asks the processor for a signed report describing the environment it is running in. Attestation is that exchange: the hardware states what booted, signs the statement with a key the chip vendor burned in at the factory, and someone outside checks it. We call that checker the verifier. The machine's owner cannot forge that signature, which is the whole point, and producing one at all needs real hardware. c8s, our confidential Kubernetes, issues a workload certificate only after the node proves which image it booted. Confidential OS Builder produces disk images whose purpose is to be measured by the hardware before they start. Point that at a GitHub-hosted runner, the ordinary machine GitHub runs a CI job on, and the tests that matter never execute at all. The attestation code sits behind a build flag, so a default build never compiles it. The hardware tests are marked ignored, so they run only when something asks for them. A skipped test doesn't turn a run red, so the suite goes green having tested none of it. So we built Confidential CI, which runs those suites inside confidential VMs (CVMs) at the scale a merge queue needs. This post is about the bugs that surfaced once the tests ran on real hardware, and what a green run from it actually proves. The last section gets to the case in the title: the same machinery lets one party benchmark another party's model without either side giving up its secret. ``` a merge lands │ ▼ ┌─────────────────────────────────────────────┐ │ off the runner: pick a random number │ └─────────────────────────────────────────────┘ │ ▼ ╔══ inside the CVM ═══════════════════════════╗ ║ run the tests ║ ║ ask the hardware to sign a report ║ ║ that includes that number ║ ╚═════════════════════════════════════════════╝ │ ▼ ┌─────────────────────────────────────────────┐ │ off the runner: check the signature, │ │ the image it booted, and the number │ └─────────────────────────────────────────────┘ │ ▼ pass or fail Two of the four steps happen where the tests cannot reach them. ``` ## The bugs only real hardware finds Without hardware, you test attestation code against a fixture. Record a report, return it from a fake device, compare against a known-good value. Those tests are worth having and they catch parsing and error handling bugs. Everything up to the verification boundary can be mocked. The verifier itself cannot. A test double that satisfies the verifier is, by definition, a forgery the verifier accepted. We found [two bugs](https://github.com/confidential-dot-ai/attestation-rs/pull/64) this way that a fixture couldn't have caught. In the first, the guest and the verifier disagreed about how to encode the random number that proves a report is fresh. Against a recorded report the encoding looked correct, because the recording came from the same code that produced it. The second showed up only on the machines we rent. There the report doesn't come straight from the processor: the cloud gives each machine a virtual Trusted Platform Module, an emulated security chip whose signing key the hardware report vouches for. Two of our test processes competed for its sessions, and the attestation failed. A fixture is a value, and no value contains a state machine. ## What green actually claims "Our tests run in a TEE" gets treated as one claim, which a pipeline either satisfies or does not. It is really five claims stacked, and each one removes a different assumption. ``` the code compiled ordinary CI ...on confidential hardware a TEE runner ...on the image we chose a pinned measurement ...during this run, not a replay a random number from outside ...and the guest did not grade itself an external verifier ``` Our own pipeline sat on the second rung for most of the year: confidential hardware, and nothing above it. The rest just shipped, and the sections below cover what each step cost. Supply-chain provenance frameworks such as [SLSA](https://slsa.dev) and [in-toto](https://in-toto.io) climb a ladder like this one for who built an artifact; ours is about where the tests ran. ## Where a test actually runs ``` Untrusted: the host, the hypervisor, the cloud operator, and us ╔══ inside the CVM ════════════════════════════════════════╗ ║ ║ ║ ┌──────┐ ┌╌╌╌╌╌╌┐ ┌╌╌╌╌╌╌┐ ┌╌╌╌╌╌╌┐ ║ ║ │ test │ ┊ test ┊ ┊ test ┊ ┊ test ┊ ║ ║ └──────┘ └╌╌╌╌╌╌┘ └╌╌╌╌╌╌┘ └╌╌╌╌╌╌┘ ║ ║ running created per CI job, destroyed after it ║ ║ ║ ╚══════════════════════════════════════════════════════════╝ ═══ a double wall is memory the host cannot read Same wall whether we own the machine or rent it. ``` The double wall is the hardware boundary. Everything outside it is untrusted, and on the left that includes us. We run four configurations: AMD SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging) and Intel TDX (Trust Domain Extensions), each on a machine we operate and a machine we rent. The evidence reaches the verifier by a different code path in each, and each path fails in its own way. The virtual TPM bug above surfaced on the rented ones alone. ## Why the guest cannot vouch for itself Asking a guest whether it is confidential is circular. A guest reads its own environment, and a host can present a convincing one, so the presence of an attestation device proves less than it appears to. The fix is a nonce, the name for that random number. The verifier picks one fresh for each run, and the guest has to fold it into the report the processor signs. ``` ┌────────────────────┐ ╔═ CVM ════════════════════╗ │ Verifier │─── nonce ────►║ ║ │ │ ║ ┌──────────────────┐ ║ │ [y] signature │ ║ │ Runner │ ║ │ [y] measurement │◄── report ────║ │ Tests │ ║ │ [y] nonce │ ║ └──────────────────┘ ║ └────────────────────┘ ╚══════════════════════════╝ ▲ ┊ fixture [n] The runner sits inside the boundary. The verifier stays outside it. ``` A separate CI job outside the CVM picks the nonce, so the answer has to be about this boot and no earlier one. A second job outside the CVM verifies the report, so the guest never grades itself. Both properties come from where those jobs run, and both jobs land on GitHub's ordinary runners. That's fine. Picking a random number and checking a signature are the two steps that don't need confidential hardware, and whoever controls GitHub already controls the source we're testing. ## What the boundary costs to renew ``` ┌────────────────────────────┐ │ Host │ outlives everything │ ╔═ CVM ══════════════════╗ │ provisioned once │ ║ ┌──────────────────┐ ║ │ │ ║ │ Runner │ ║ │ one job, then gone │ ║ │ Test process │ ║ │ │ ║ └──────────────────┘ ║ │ │ ╚════════════════════════╝ │ └────────────────────────────┘ The boundary is renewed less often than the thing inside it ``` A CVM booted fresh for every run carries its own evidence, because the boot produces the launch measurement, the hash of the image the hardware records as it starts. It's also the slow path. A standing CVM that jobs land on is quick, and its evidence is from whenever it was provisioned. The slowness and the freshness have the same cause. Booting is expensive, and booting refreshes the evidence, so you don't get to buy one without the other. Attesting per run on a standing CVM is nearly free: producing and checking the report take seconds, and the rest of the cost is scheduling jobs and setting up toolchains. It buys back the top of the ladder without a reboot, so how often you boot and how often you attest are independent choices. It still leaves you on the same machine, though. The report proves the same boot, later, and whatever earlier jobs changed on that guest rides along. More on that in the next section. There is also a third option: a pool of guests booted fresh and held idle. That makes the evidence current and the queue empty, and charges for both in idle hardware, which our merge rate doesn't yet justify. ## What the report covers ``` ╔══════════════════════════════╗ ║ ┌────────────────────────┐ ║ ╮ ║ │ Firmware │ ║ │ ║ │ Kernel │ ║ ├── launch measurement ║ │ initrd │ ║ │ ║ │ Command line │ ║ ╯ ║ └────────────────────────┘ ║ ║ ║ ║ ┌────────────────────────┐ ║ ║ │ CI agent │ ║ ╌╌╌ not covered ║ │ Fetched artifacts │ ║ ║ └────────────────────────┘ ║ ╚══════════════════════════════╝ The hardware measures what booted. Everything after that is on you. ``` A launch measurement covers the guest image at boot. It does not cover the CI agent installed afterwards, or anything a previous job left behind. TDX has runtime measurement registers that software can extend after boot. SEV-SNP doesn't, so post-boot measurement there rides on a software stack such as the virtual TPM. Either way, post-boot state is only measured if something has been written to measure it. Two limits on what we shipped, and both narrow the window rather than close it. Every chip signs with its own key, backed by a certificate chain ending at AMD's or Intel's root, and each vendor publishes a list of keys it has withdrawn. Our verifier walks that chain to the root on all four configurations, and on the Intel ones it also fetches the withdrawal lists. On the AMD ones it does not, so a report signed by a key AMD has since revoked would still pass. And attesting the runner is not the same as attesting the tests, since each job gets its own fresh pod and the pods share only the guest. ## Code you are not allowed to read We built this for our own repositories, but nothing in the mechanism requires the tests and the code under test to come from the same place. ``` ┌──────────────┐ ╔═══════════════╗ ┌──────────────┐ │ Model owner │ ║ Attested ║ │ Evaluator │ │ │─ sealed ─►║ guest ║◄─ test ───│ │ │ │ weights ║ ║ code │ │ │ checks it │◄─ report ─║ ║ │ never sees │ │ before the │ ║ opens them, ║── score ─►│ the weights │ │ key moves │── key ───►║ runs the test ║ │ │ └──────────────┘ ╚═══════════════╝ └──────────────┘ The key moves only after the report checks out. ``` Safety evaluation of a frontier model is where this bites hardest. The benchmark needs the weights, which usually leaves two options and neither is a good position for an audit. Either the owner runs the benchmark and reports a number, which asks you to trust the party being measured. Or the weights move to the evaluator, which is normally refused outright. A third arrangement avoids both. The benchmark runs as the payload, and the sealed weights come to it. The owner first checks a report showing which image booted and which harness is loaded. Only then does it release the model's decryption key, encrypted to a public key that the same report vouches for, so no other machine can open it. The score comes back with a report naming both, and the evaluator never holds the weights. In the simple version the owner reads the harness, because approving what runs against your weights is the basis on which you release the key. A stronger version keeps both secrets sealed. The owner approves the harness by its measurement instead of reading its source, and the evaluator's prompts stay hidden for the same reason the weights do. In that version the only thing holding the arrangement together is a runtime both parties checked. Three limits are worth stating. The first: this only works with the vendor chain checked to the end, revocation included, so the AMD gap above is the first thing to close here. The result channel is also an exfiltration channel. A harness that chooses the prompts and reads the answers can extract a great deal about a model, so the report has to pin the harness and not only the environment. And a model evaluation runs on a GPU, while these platforms encrypt CPU memory. Current NVIDIA hardware has a [confidential mode](https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/) that attests separately and protects device memory by controlling access rather than by encrypting it, to be checked alongside the CPU report. We have built the CPU side. [c8s](https://github.com/confidential-dot-ai/c8s), [Kettle](https://github.com/confidential-dot-ai/kettle), our [attestation libraries](https://github.com/confidential-dot-ai/attestation-rs) and [Confidential OS Builder](https://github.com/confidential-dot-ai/confidential-os-builder) are open source. --- # Open-sourcing our Confidential Virtual Machine builder > Attestable Virtual Machines, from firmware, to kernel, to OS Since we launched [Kettle](https://github.com/confidential-dot-ai/kettle), our attested build tool, we've been working to offer Confidential Virtual Machines that can be verified at every level. Today, we're launching [Confidential OS Builder](https://github.com/confidential-dot-ai/confidential-os-builder/), an open-source tool that builds images for Confidential Virtual Machines. Knowing the provenance of your own software is important, but you should be able to trace every piece of the system you are using — if your binary is trusted but the OS you run it inside can be compromised, it's pointless. **What, exactly, is running inside your VM?** Confidential OS Builder exists to make sure you know. Our CVM builder brings together TEEs (Trusted Execution Environments), a custom Linux kernel, and a custom OS build, to provide a hardware-signed proof of exactly what ran inside your VM. This proof doesn't require trust in the cloud host machine, or trust in anyone with root access to the host machine. Instead, it uses AMD's SEV-SNP or Intel's TDX to cryptographically sign a boot measurement. The signed boot measurement includes a checksum of the firmware and the initrd that will bootstrap the VM. The initrd includes a Merkle-tree checksum of the entire VM disk, including every package and binary you included. By using overlayfs and a read-only disk, we can prove that every boot matches what was originally built. By building your own software into the image, you have hardware measurements that trace all the way to the exact binaries that the CVM runs. ## The chain of trust Like Kettle, our security promises start with the TEE: an encrypted virtual machine whose memory cannot be read by the host, thanks to an encryption key provided directly to the VM by the CPU's dedicated security processor. Since the encryption key is never exposed to the host node, even root access on the host does not allow any visibility into what is running in the TEE VM. An attestation from a TEE includes boot measurements of the VM, the firmware and software used to bring the machine up, signed by a private key that verifiably belongs to the hardware manufacturer (in this case, AMD or Intel). In our images, the firmware and boot package include a cryptographic digest of the contents of the VM's entire boot disk. Once the firmware has loaded the kernel and started running the init process, the init process reads the disk checksum and uses [dm-verity](https://docs.kernel.org/admin-guide/device-mapper/verity.html) to ensure the entire disk exactly matches the build-time checksum. That disk was built read-only and compressed using the erofs filesystem. It can't be changed by any action the VM takes, because that would alter the dm-verity checksum and cause the kernel to terminate access to the disk. An overlayfs allows ephemeral writes, ensuring work can be done but changes will not persist to the next boot of the CVM. ## Using our images The easiest way to use our attested images is with our [Confidential Cloud](https://confidential.ai/cloud). If you want to replicate our VMs yourself, though, we've done our best to make it straightforward. If you have [qemu](https://www.qemu.org/) and [oras](https://oras.land), you can run one of our images in just a few seconds. We're going to use the `dev` image, since that includes console access to the VM after it boots. oras pull ghcr.io/confidential-dot-ai/confidential-os-builder:dev qemu-system-x86_64 \ -machine q35 \ -kernel uki.efi \ -drive if=pflash,format=raw,readonly=on,file=OVMF.fd \ -drive file=disk.raw,format=raw,if=virtio \ -smp 1 -m 4G -nographic That's it! You've got a running VM. For a Confidential VM running inside a TEE, you'll also need the host machine to have SEV-SNP or TDX. For SEV-SNP, you'll need qemu 10.1 or higher, compiled with the [IGVM extensions](https://www.qemu.org/docs/master/system/igvm.html) enabled as well. To make sure you're getting all the CLI flags right, you can use `confos` to help you, like this: confos pull ghcr.io/confidential-dot-ai/confidential-os-builder:dev confos run output/dev --scratch 20g That will build the full qemu command line to enable KVM and TEE mode if your host supports it, as well as create and attach a 20GB disk image the VM can use for encrypted, ephemeral storage. ## Building custom images Unlike `dev`, the `base` profile leaves it up to you to layer your own environment into your Confidential VM images. We've stripped things down to a locked-down kernel and [just 29 packages](https://github.com/confidential-dot-ai/confidential-os-builder/blob/main/mkosi/base/mkosi.conf#L30-L59), with a primary goal of ensuring the host doesn't have any ways to observe or interact with the CVM it's hosting. That means removing the SSH server, disabling the serial console, and counting on users to add their own packages. You can customize the software that will run inside the CVM by creating a `mkosi` profile or by supplying a cloud-init configuration file that will run when your VM boots. You'll want to think about the tradeoff that's ideal for your use case, where putting everything in the image gives you VMs that become ready faster and sign more of their code, but are bigger files to manage and less flexible if you need dynamic behavior. From inside the Confidential OS Builder repo, you can see some example mkosi profiles in `mkosi/base/mkosi.profiles`. You can build a profile by running `bin/confos build dev --profile dev`, which will create files in `output/dev`. There's also an example cloud-init config that installs the Caddy web server and hosts a static webpage in `examples/`. To include the example in your built image, run `bin/confos build web --cloud-init examples/caddy.yaml` to create an image in `output/web`. After you've built an image, Confidential OS Builder wraps the `oras` tool to push the disk image and associated files to any OCI registry via `confos push`. ## Share and enjoy Thanks to Confidential OS Builder, we were able to add a detailed verification mode to Kettle that attests not just the source code that went into your binary, but the entire build VM, from firmware, to kernel, to OS, with a signature anyone can verify after downloading the public VM image used to create the build. We're looking forward to seeing how this kind of tool enables new uses for TEEs and confidential VM workloads. If you're interested in building your own CVM images, check out the [documentation on GitHub](https://github.com/confidential-dot-ai/confidential-os-builder/tree/main/docs). We'd love to hear how Confidential OS Builder works for you! --- # The Isolation Spectrum: From Plaintext to TEEs *Sandboxes, containers, microVMs, and TEEs all claim to isolate your workload. They are not doing the same job. Here is the map.* --- There is one question that sorts every isolation technology into place: **Can the host read your memory?** The host is whoever runs the machine: the cloud provider, the platform operator, the admin with root. Ask the question at every tier and the map draws itself. ``` plaintext → isolate → container → microVM → TEE yes yes yes yes no ``` Four yeses and one no. Everything below explains why the line flips exactly there, and nowhere earlier. ## Tier 0: Plaintext No boundary. Your code and data sit in memory, readable by any process with sufficient privilege. This is a workload running directly on a machine, or in a standard VM where you only care about the guest's internals. - Protects: nobody - Enforced by: nothing - Host sees your data: yes ## Tier 1: Runtime isolates V8 isolates, as used by edge platforms like Cloudflare Workers. Many tenants share a single OS process. The boundary between them is the JavaScript engine's heap and context separation. No separate address space, no separate kernel namespaces. This is why isolates cold-start in microseconds. It is also why the boundary is thin. You are trusting the correctness of a large JIT-compiling engine, and platforms that run isolates at scale layer additional sandboxing around them, partly because of Spectre-class side channels between tenants in one process. - Protects: the host and co-tenants, from your code - Enforced by: the JavaScript engine - Host sees your data: yes ## Tier 2: Containers and sandboxes The default unit of cloud deployment. Namespaces, cgroups, and seccomp filters carve one kernel into many apparent machines. Sandboxes like gVisor add a syscall interception layer on top. Containers do their job well. That job is to stop a workload from escaping, damaging the machine, or touching other workloads. It points inwards. The kernel that enforces the boundary can read every byte inside it, and so can anyone who controls that kernel. - Protects: the host and other tenants, from your workload - Enforced by: the OS kernel, the thing you are trusting - Host sees your data: yes ## Tier 3: MicroVMs Firecracker, Kata Containers, and similar. Each workload gets its own guest kernel inside a minimal virtual machine. The boundary is enforced with hardware virtualization: VT-x, AMD-V, nested page tables. This is the strongest software-side boundary, and it is why AWS runs Lambda on Firecracker rather than bare containers. This tier is where people get confused, because "hardware" enters the vocabulary. So be precise: **Hardware-assisted is not hardware-enforced.** In a microVM, the CPU's virtualization features work for the hypervisor. The hypervisor builds the page tables, and it can read, dump, snapshot, or live-migrate guest memory whenever it likes. Guest memory is plaintext to the host. The virtualization extensions make the wall cheaper to build. The landlord still holds the master key. - Protects: the host and other tenants, from your workload - Enforced by: the CPU, on behalf of the hypervisor - Host sees your data: yes ## Tier 4: TEEs Trusted Execution Environments: AMD SEV-SNP, Intel TDX, NVIDIA Confidential Computing on the GPU side. Same virtualization machinery as tier 3, with one inversion that changes everything. The CPU encrypts the workload's memory with keys generated and held inside the silicon. The hypervisor is outside the trust boundary. It can schedule the workload, allocate its resources, and kill it. It cannot read it. Neither can the OS, the cloud provider, or an admin with root. To every layer of software on the host, your memory is ciphertext. And you do not have to take anyone's word for it. Remote attestation gives you a cryptographic proof, signed by the chip vendor's keys, of exactly what code booted inside the TEE and that the protections are on. Every other tier asks you to trust the operator. This one lets you verify. - Protects: your workload, from the host - Enforced by: the CPU, against the hypervisor - Host sees your data: no ## The direction of protection Line the tiers up and the real pattern is not strength. It is direction. Tiers 1 through 3 all point inwards. They protect the infrastructure from the code. Each one builds a thicker wall, but the wall is always operated by the host, for the host. Climbing from container to microVM changes how hard it is for your workload to break out. It changes nothing about who can look in. TEEs point outwards. They protect the code and data from the infrastructure. That is not a stronger version of the same guarantee. It is a different guarantee. Which means the tiers are not competitors. A sandbox inside a TEE gives you both directions at once: the host cannot see the workload, and the workload cannot harm the host. That is the correct architecture for running untrusted code on data that must stay private. One scoping note for the careful reader: TEEs defend against software on the host. Physical attacks on the machine and microarchitectural side channels are a separate threat model with their own mitigations, and vendors patch that surface continuously. For the threat model that matters in practice, an operator or provider reading your data, the guarantee holds. Condensed, the whole spectrum collapses to three distinctions: ``` ┌───────────────────────┬────────────┬─────────────────┬─────────────────┬───────────────────────┐ │ Model │ Isolation │ Protects │ Host sees data │ Enforced by │ ├───────────────────────┼────────────┼─────────────────┼─────────────────┼───────────────────────┤ │ Plaintext VM │ none │ nobody │ yes │ nothing │ ├───────────────────────┼────────────┼─────────────────┼─────────────────┼───────────────────────┤ │ Sandbox or container │ software │ the host │ yes │ the OS and hypervisor │ ├───────────────────────┼────────────┼─────────────────┼─────────────────┼───────────────────────┤ │ TEE │ hardware │ your workload │ no │ CPU with attestation │ └───────────────────────┴────────────┴─────────────────┴─────────────────┴───────────────────────┘ ``` ## Why this matters for AI AI workloads are the worst case for tiers 0 through 3. Prompts, model weights, training data, and agent context all sit in memory, in plaintext, on machines you usually do not own. Every "yes" on the map above is a party that can read them: the cloud provider, the platform, the inference host. If your threat model includes the infrastructure, and for proprietary weights, regulated data, or agents acting on your behalf it should, then software isolation is not a mitigation. It was never designed to be one. This is the boundary Confidential builds on. Private inference, private weights, private training, private agents: all of it runs inside TEEs, attested, with the host locked out by the silicon itself. The host cannot leak what the host cannot read. --- *Want the short version? Ask one question: can the host read your memory? If the answer is yes, it is not confidential computing.* --- # Kettle: Attested Builds from a TEE **Today we're open-sourcing Kettle, a tool for building and verifying attested builds.** Attested builds are packages that include cryptographically signed SLSA provenance, certifying exactly what source code, dependencies, and toolchain produced a binary. Anyone can verify it. No trust required. ## The problem You ship a binary. Someone has to trust it. They can audit your source code. They can review your dependencies. But there's a gap they can't close: did the binary they're running actually come from the code they reviewed? Traditional build systems can't answer that question. They ask users to trust that CI didn't tamper, that sysadmins didn't interfere, that the hypervisor behaved. That's a lot of trust. Attested builds close the gap. ## How Kettle works Kettle runs inside a TEE (Trusted Execution Environment). It generates a SLSA-compliant `provenance.json` covering your source, dependencies, and toolchain, then applies a hardware signature to everything via the TEE's attestation mechanism. The hardware signature is verified against certificates published by the hardware manufacturer. That chain links your binary directly to its source inputs, cryptographically, without requiring trust in any intermediate party. Two commands: ```bash # Build and apply a hardware signature from inside a TEE kettle attest # Verify the signed evidence and binary checksum kettle verify /kettle-build ``` Output: a `provenance.json`, an `evidence.json`, and your binaries. Anyone with Kettle can verify them. ## Why TEE-backed attestation matters GitHub's artifact attestations ask you to trust GitHub's word that their cloud VMs didn't tamper with your build. Kettle's attestation asks you to trust only the hardware manufacturer and the physical custodians of the build machine. No sysadmins. No hypervisor authors. No image maintainers. The TEE's memory is encrypted even against the hypervisor, so nothing running outside the enclave can observe or modify the build as it runs. That's a dramatically smaller trust surface. ## Who needs this **A customer deploying your service** wants to know it's running the code you claim, built from the source they audited. **A compliance team** wants evidence that a binary was built with specific dependency versions, not newer ones with unknown changes. **A security auditor** wants to verify that the toolchain used to compile a release matches the one specified in your security documentation. **A regulated enterprise** wants proof that sensitive data will be processed only by code that passed their review, not by a modified version. **A package consumer** wants to ensure the binary they downloaded corresponds to the source and dependencies they reviewed, not a tampered version. ## Getting started Kettle is open source and installs via Cargo: ```bash # From GitHub Releases curl -LO https://github.com/confidential-dot-ai/kettle/releases/latest/download/kettle # From source cargo install --git https://github.com/confidential-dot-ai/kettle # Inside a TEE (enables hardware attestation) apt-get install -y libtss2-dev cargo install --features attest --git https://github.com/confidential-dot-ai/kettle ``` Rust, Nix, JavaScript, and TypeScript (via pnpm) projects are supported today. Python and Go support is coming. Full documentation, architecture, and threat model: [github.com/confidential-dot-ai/kettle](https://github.com/confidential-dot-ai/kettle) --- Kettle is part of Confidential's broader infrastructure for confidential computing. If you're building systems where the integrity of code matters, this is the foundation. --- # Lunal Becomes Confidential Confidential AI is the most important infrastructure problem of today. Exposing our data and identity to the tech giants was bad. The same is happening with the frontier labs in fast forward mode. As AI infiltrates every workflow, confidentiality necessarily becomes table stakes. Yet the labs want this too. They want to protect their model weights. Their IP. Privacy is essential for all. Confidential computing is hitting an inflection point. It is crossing the chasm from theory and research into production. But this problem remains unsolved. That's why today we decided to rename ourselves Confidential. This new name describes exactly what gets the team out of bed every day. Lunal as a name has worked for us until now: an engineering team solely focused on shipping by daylight or moonlight. Until a few weeks ago, our domain lunal.dev was just our Github. A sign we cut the fluff and focus on the work. Now the name has changed. But the approach hasn't. Same team, same goal, same hellbent desire to ship industry-defining, privacy-preserving products. We've shipped confidential inference, training, fine-tuning, and agents. And now, finally, we got around to shipping a new name and a website. We are Confidential and we do Confidential AI. If you build with AI and care about privacy and security, [we should talk](mailto:hello@confidential.ai). --- # A New Mark for Confidential Today we are sunsetting the moon and shipping a new logo. The moon was a placeholder. An emoji. The fastest way to ship a mark, and it fit our old name, Lunal. Now we are Confidential. The moon needed an upgrade. Our new logo is a redacted bar inside square brackets. [███] You can read it in a few different ways, all valid. As a glyph. Redacted text inside brackets implies something is there but you cannot see it. Not deleted but private. Present, in use, deliberately withheld from view. That is confidential computing in one image. As code. Square brackets index into memory. The thing inside is a buffer. The bar is its contents. A buffer whose contents are opaque to everything outside it is a TEE. The logo is a picture of protected memory. As text. It is typeable. [███] renders in a terminal, a commit message, a Slack channel, a CLI prompt. Most logos die the moment they leave the brand folder. This one survives in plaintext. Grep-able. Whiteboard-able. Degrades to ASCII without losing meaning. It is not a padlock. It is not a shield. It is not a vault. Every other company in security reaches for those. They all mean: trust us, we will protect you. Ours is different. The contents are not protected by us. The contents are cryptographically unreachable. That is the entire point of hardware-enforced confidential computing, and now it is the logo. --- # PrivateClaw launches on Confidential: E2E Private AI Agents Your OpenClaw host can read every conversation you have with your AI agent. That's true for almost every hosted agent on the market. A provider runs the VM. A provider runs the inference. The providers, in theory and in practice, see your prompts and the model's responses. For some workloads, that's fine. For proprietary code, patient data, financial docs, or anything legal, it's a problem. [PrivateClaw](https://privateclaw.dev) fixes it. And it runs on Confidential's stack. ## What it is PrivateClaw is a managed [OpenClaw](https://docs.openclaw.ai/getting-started) instance with end-to-end encryption you can verify yourself. One command: ``` $ ssh privateclaw.dev ``` That's the install. SSH in, get a dedicated Confidential VM. There's a free tier. ## What "verify yourself" means Most "private AI" products are policy promises. "We won't look at your data." PrivateClaw is a cryptographic one. The host can't look at your data, and you can verify this. Two TEEs, connected privately: 1. **Your VM** runs in a TEE, an AMD SEV-SNP confidential virtual machine. Memory is hardware-encrypted. The cloud provider can't read it. 2. **Inference** runs in a separate TEE. Prompts are encrypted in transit and during processing. The [verification CLI](https://github.com/confidential-dot-ai/privateclaw-cli) is open source. Run it from your VM and confirm the properties yourself in seconds. ## The Confidential stack underneath PrivateClaw is what the Confidential platform is built for. The components doing the work: - **Confidential VMs (CPU and GPU)** provide the hardware-encrypted execution environments for the agent's VM and the inference endpoint. - **Attestation Service** generates the cryptographic proof that the code running is the code that was built. The PrivateClaw verify CLI wraps this for end users. - **Networking inside TEEs** keeps the path from VM to inference endpoint inside the encrypted boundary the whole way. - **Hardened VM image** is the base OS. Minimal, measured, no debugging tools in production. Overhead is negligible. Our [benchmarks](/blog/tee-performance-cpus) on SEV-SNP show ~1.3% across CPU-heavy workloads, and ~4% token throughput overhead for inference. ## Why this is the start, not the end OpenClaw is the first hosted agent. Qwen is the first model. More of both are coming. The pattern generalizes. Any agent that handles sensitive context wants the same thing: hardware-isolated execution, attested inference, credentials that never exist in plaintext outside the TEE. The Confidential stack provides all of it. If you're building agents and the trust boundary matters, the same components are available to you. [Read the docs](/docs/confidential-agents-api) or [get in touch](mailto:hello@confidential.ai). ## Try it ``` $ ssh privateclaw.dev ``` Free tier available. --- # Secure AI Needs TEEs AI security standards are being developed in real-time as the technology scales from research labs to billions of users. As these models take on more consequential roles in software systems in terms of delegating, making decisions, and acting on our behalf, questions remain about how to effectively secure these systems. In addition, frontier models are increasingly viewed as strategic national assets that require protection from sophisticated threat actors. The intersection of this rapid technological advancement, the scale of deployment, and this new complex threat landscape necessitates an equivalently accelerated effort to develop practical security standards and implement them at production scale. A core component of these emerging security standards and their implementations is Trusted Execution Environments (TEEs), often referred to as Confidential Computing. TEEs are already referenced in multiple AI security and safety standards. Notably, RAND's ["Securing AI Model Weights"](https://www.rand.org/pubs/research_reports/RRA2849-1.html) report identifies TEEs as essential infrastructure for securing modern AI systems. Anthropic has [started exploring](https://www.anthropic.com/research/confidential-inference-trusted-vms) confidential inference built on top of TEEs. Meta is [evaluating using TEEs](https://ai.meta.com/static-resource/private-processing-technical-whitepaper) to bring private AI processing to WhatsApp. Yet the tooling to make practical use of TEEs for AI infrastructure is nowhere near ready, let alone scalable and production-grade. Confidential exists to solve that problem: building the tooling and infrastructure that makes deploying secure AI using TEEs easy. ## What Are TEEs? A Trusted Execution Environment (TEE) is a tamper-proof isolated computing environment built into modern CPU and GPU hardware. TEEs provide four critical security properties: integrity (tamper-proof execution), confidentiality (hardware-level memory encryption), verifiability (cryptographic measurement of running software), and attestation (signed proofs of all the above). Think of it as HTTPS taken a step further: you can verify not just who you're talking to, but what software they're running, with negligible performance cost. For a full introduction to how TEEs work, see the [Introduction to TEEs](/docs/intro-to-tees.md). For a detailed technical deep dive, see the [Confidential Computing Primer](/docs/confidential-computing-primer/). ## Why TEEs Matter for AI ### Model Weight and IP Protection TEEs significantly improve secure model storage and distribution. Model weights can now be encrypted at rest and in transit, decrypted only within verified TEEs, and held in hardware-isolated GPU memory during compute. When AI systems span multiple components, this enables a zero trust architecture where each component itself runs inside a TEE and can verify the others' attestations before sharing model weights or derived data. This ensures that even within your own infrastructure, no component inherently trusts another and every interaction requires cryptographic verification of TEE integrity. Google is now deploying Gemini as part of its on premise Google Cloud Platform stack, with the condition that the weights are protected by a TEE. ### Confidential Inference Confidential inference allows AI service providers to run inference and guarantee that user data remains private by utilizing GPU TEEs. This is a new unlock for regulated industries such as healthcare, government, and finance. These industries have specific requirements and compliance standards for handling user data such as PII and PHI. These industries can now use external AI services while maintaining cryptographic proof that their sensitive data was never accessible to the service provider. ### Training and Fine-tuning TEEs extend beyond inference to training workflows that handle sensitive data. Healthcare organizations can train models on patient data without exposing records to cloud providers. Financial institutions can fine-tune models on transaction data while maintaining regulatory compliance. Federated learning scenarios benefit particularly from TEE guarantees. Instead of trusting participating organizations to honestly report gradient updates, TEEs provide cryptographic proof that computations were performed correctly on specified datasets. Verifiable training provides tamper-proof records of exactly what data was used to train a model. ### Agent Runtime Security AI agents require access to external services and credentials to be useful. An agent that can read your email, make calendar appointments, or execute financial transactions must be trusted with extremely sensitive access tokens and the corresponding data. TEEs provide credential isolation that ensures agent access tokens never exist in plaintext outside the secure environment. Even if the host system is compromised, attackers cannot extract the credentials needed to impersonate the agent. Multi-agent systems create additional trust requirements. Agents can verify each other via TEE attestation prior to sharing data or credentials. This enables new architectural patterns where agents can safely delegate tasks and communicate. ### AI Governance TEEs enable new approaches to AI governance, grounded in enforceable cryptographic proof rather than aspirational policy promises. Rate limiting becomes verifiable. Organizations can prove to auditors that usage quotas were enforced without the possibility of gaming or bypassing controls. Content policy enforcement gains similar verifiability. When safety filters are applied within TEEs, the system can provide cryptographic proof that all outputs were processed through specified content policies. This is particularly valuable for organizations that must demonstrate compliance with content regulations. Model version control becomes tamper-proof when implemented in TEEs. Organizations can prove exactly which model weights were used for specific inferences, creating audit trails that cannot be forged or modified after the fact. ## Confidential: Making TEEs Production-Ready TEEs represent an extremely valuable primitive for securing AI systems. The hardware is already deployed: the latest generation CPUs and GPUs from Intel, AMD, and NVIDIA all ship with TEE capabilities built in, sitting in data centers right now waiting to be leveraged. Using TEEs doesn't require changing or rebuilding your stack. Your existing applications, inference servers, and training workflows should run in TEE environments using the operational patterns you already use. Same code, same performance, same cost structure. So why isn't this happening? The tooling to make TEE deployment seamless simply doesn't exist yet and building a production TEE deployment stack has substantial technical challenges. ### Key Challenges **Attestation Complexity**: Attestation workflows become complex when you need to verify not just your application, but every component it depends on, from your code down to the firmware. These attestations must be generated, validated, and distributed in ways that third parties can independently verify. Additionally, TEE attestations have no unified standard, where every implementation you use has very different tooling. **Build Reproducibility**: Build reproducibility is an essential prerequisite step for TEE verifiability but difficult to achieve with current toolchains. Different compiler versions, dependency resolution order, and build environment variations create inconsistent binaries that break cryptographic verification. Achieving reproducible builds requires rethinking the compilation pipeline around hermetic environments and fixed toolchain versions. **Key Management at Scale**: There is also the problem of managing keys securely at scale. TEE keys must be automatically rotated while maintaining zero-downtime service. Data encrypted for one TEE instance needs to remain accessible as instances scale up and down. The key management system must independently verify TEE attestations before providing decryption keys, requiring orchestration between attestation verification and key distribution to maintain security guarantees. **Infrastructure Compatibility**: Standard infrastructure patterns create new security considerations in TEE environments. Middleware components like load balancers and rate limiters typically require plaintext access to requests, which undermines the end-to-end confidentiality that TEEs provide. Auto-scaling systems usually expose resource utilization patterns that can leak information about the workloads running inside TEEs. Observability tools need to collect telemetry without compromising the confidentiality guarantees that make TEEs valuable in the first place. Application services like caching, routing, and DDoS protection require rethinking when deployed alongside TEEs. These services must operate on encrypted data streams while maintaining their effectiveness, or they must run within TEE boundaries themselves. Either approach requires specialized implementation that differs from standard deployment patterns. There's no cohesive platform that handles the full stack of TEE deployment concerns while maintaining compatibility with existing development workflows. Confidential exists to solve this. We're building the standards, tooling, and infrastructure necessary to make using TEEs simple and seamless. ## Conclusion The rapid deployment of AI systems at scale has created an urgent need for robust security standards to handle the new threat models that have emerged. TEEs will be a core building block for these security standards. The hardware-backed confidentiality, integrity, and verifiability that TEEs provide establish a solid foundation for securing AI systems across inference, training, agent runtimes, and governance. However, the gap between TEE capabilities and production-ready tooling remains substantial. While the underlying hardware is already deployed across modern data centers, the complex requirements around attestation workflows, reproducible builds, key management, and infrastructure compatibility remain unsolved and prevent potential adoption. Confidential is bridging this gap by building the comprehensive tooling and infrastructure stack needed to make TEE deployment seamless for AI workloads. As AI systems become increasingly critical to business operations and national security, the ability to provide cryptographic guarantees about model security and data privacy will transition from a competitive advantage to a fundamental requirement. --- # TEE Performance on CPUs September 15, 2025 "What's the catch?" is a common question we get regarding TEEs. What is the performance tradeoff for getting the hardware-level security guarantees that TEEs provide? The de-facto assumption is that there must be significant performance penalties for the strong isolation and encryption that Trusted Execution Environments provide. We ran the tests and have the answer: **there isn't really a catch**. We ran comprehensive benchmarks comparing identical workloads inside TEE-enabled versus virtual machines, and the results show TEE overhead is remarkably minimal on CPUs, even for the most computationally intensive operations. ## The Bottom Line Performance difference across multiple metrics hovers around **1.5%**. The largest overhead we observed across any computational stage was **2.08%**. Here's a table summarizing the key performance metrics when running a proving workload on both TEE-protected and unprotected systems: | Metric | TEE Environment | Regular Environment | Performance Difference | | ---------------- | --------------- | ------------------- | ---------------------- | | Execution Speed | 5.16 kHz | 5.26 kHz | -1.90% | | Runtime Duration | 89.46s | 87.78s | +1.91% | | CPU Utilization | 26.61% | 26.23% | +1.45% | | Memory Usage | 19.02 GB | 18.87 GB | +0.80% | ## The Test Setup We chose to benchmark cryptographic proving systems because they represent some of the most computationally demanding workloads you can run in terms of CPU, and memory utilization. If TEE overhead is negligible for these applications, it will be negligible for almost anything else running on CPUs. ### The Workload: Zero-Knowledge Virtual Machine The specific system we tested was a zero-knowledge virtual machine (zkVM) that takes any computer program, executes it, and generates cryptographic proofs of correct execution. ### Proof System Architecture The zkVM uses STARK (Scalable Transparent Arguments of Knowledge) proofs backed by the Stwo prover from StarkWare. STARK proofs are particularly demanding because they heavily utilize hashes and involve intensive field arithmetic operations that are resource intensive. ### Test Programs We benchmarked standard computational workloads including Fibonacci sequence calculations and sorting algorithms. ### Hardware Configuration - **Processor:** AMD EPYC Series, 4 cores, 32GB RAM - **TEE Implementation:** AMD SEV-SNP (Secure Encrypted Virtualization - Secure Nested Paging) - **Comparison:** Identical machines, one with SEV-SNP enabled, one without TEE capabilities ### Benchmark Methodology The benchmarking process measured four distinct computational phases: 1. **Native Execution:** Direct program execution to establish baseline performance 2. **Program Emulation:** Running the program in a virtual machine to capture its execution trace 3. **Cryptographic Proving:** Generation of mathematical proofs of execution using the ZKPs 4. **Proof Verification:** Checking that the generated proofs are valid Each test ran multiple iterations to ensure statistical significance, measuring execution speed, duration, and resource utilization across all stages. ## Detailed Performance Breakdown ### Overall System Performance This table shows the aggregate performance metrics across the entire zkVM proving workflow, demonstrating that TEE overhead remains consistently minimal across all key system resources. | Metric | TEE VM | Regular VM | Difference | % Change | | ---------------- | ------ | ---------- | ---------- | -------- | | Avg Speed (kHz) | 5.16 | 5.26 | -0.10 | -1.90% | | Avg Duration (s) | 89.46 | 87.78 | +1.68 | +1.91% | | Peak CPU (%) | 26.61 | 26.23 | +0.38 | +1.45% | | Peak Memory (GB) | 19.02 | 18.87 | +0.15 | +0.80% | ### Stage-by-Stage Performance Analysis #### Native Execution Stage This table captures the performance when running programs directly without any ZK proving, establishing the baseline execution characteristics for both TEE and regular environments. | Metric | TEE VM | Regular VM | Difference | % Change | | ------------- | ------- | ---------- | ---------- | -------- | | Speed (kHz) | 7296.84 | 7437.21 | -140.37 | -1.88% | | Duration (s) | 0.064 | 0.063 | +0.001 | +1.59% | | CPU Usage (%) | 26.61 | 26.18 | +0.43 | +1.64% | | Memory (GB) | 18.17 | 18.05 | +0.12 | +0.66% | #### Program Emulation Stage This table shows the performance during program emulation, where the virtual machine captures execution traces—notably, this stage exhibits the highest TEE overhead at 2.08% for emulation overhead. | Metric | TEE VM | Regular VM | Difference | % Change | | ------------------ | ------- | ---------- | ---------- | -------- | | Speed (kHz) | 1616.27 | 1650.19 | -33.92 | -2.06% | | Duration (s) | 0.29 | 0.284 | +0.006 | +2.11% | | Emulation Overhead | 4.52x | 4.43x | +0.09 | +2.08% | | CPU Usage (%) | 25.00 | 24.65 | +0.35 | +1.42% | | Memory (GB) | 18.18 | 18.07 | +0.11 | +0.61% | *Emulation Overhead measures how much slower the emulated execution is compared to native execution* #### Cryptographic Proving Stage This table presents the most computationally intensive phase where STARK proofs are generated, yet TEE overhead remains under 2% even for these cryptographically demanding operations. | Metric | TEE VM | Regular VM | Difference | % Change | | ------------- | ------ | ---------- | ---------- | -------- | | Speed (kHz) | 5.18 | 5.27 | -0.09 | -1.71% | | Duration (s) | 89.17 | 87.67 | +1.50 | +1.71% | | CPU Usage (%) | 25.13 | 24.78 | +0.35 | +1.41% | | Memory (GB) | 19.02 | 18.89 | +0.13 | +0.69% | #### Proof Verification Stage This table shows the final verification phase where generated proofs are validated, demonstrating that TEE overhead remains minimal even for cryptographic verification operations. | Metric | TEE VM | Regular VM | Difference | % Change | | ------------- | ------ | ---------- | ---------- | -------- | | Speed (kHz) | 658.91 | 667.45 | -8.54 | -1.28% | | Duration (s) | 0.70 | 0.693 | +0.007 | +1.01% | | CPU Usage (%) | 26.15 | 25.92 | +0.23 | +0.89% | | Memory (GB) | 19.02 | 18.93 | +0.09 | +0.48% | --- # Careers at Confidential At Confidential, we wake up every day to protect the world's data with secure, private, verifiable AI. We believe a small, tenacious team can outrun, outmaneuver, and outperform giants. We live it. To build that team, we're looking for people with grit that get things done and want to be pivotal in an ascendant startup. We obsess over talent to an unusual degree, and are designing a company that's a haven for self-motivated contributors who want high ownership. We work together in person in sunny San Francisco, California. We believe in person is the best way to work. We also believe, today, San Francisco is the best place on earth to build an AI company. We're backed by [Initialized Capital](https://initialized.com/), [Notable Capital](https://www.notablecap.com/), [Soma Capital](https://somacap.com/), and some great angels. We're looking for relentless minds. Interested? Email us at [hello@confidential.ai](mailto:hello@confidential.ai). We'd love to hear from you. ## 🧱 Software Engineer Confidential is growing. Fast. And we're looking for software engineers to build the foundations of private, secure, and verifiable compute. Our [team](/team) is small but high-agency. You’ll own large pieces of the puzzle. ### What You'll Work on: As an engineer at Confidential, you'll design distributed systems, deploy infrastructure globally, and ship software daily. Here are a few examples of the hard problems you'll work on at Confidential: - Build software for end-to-end private, secure AI inference, training, and fine-tuning. - Publish open source industry standards for TEEs and TEE attestations. - Implement Zero Knowledge (ZK) circuits to secure TEEs, keys, and infrastructure. - Scale infra globally across on prem, bare metal, and every hyperscalar & neocloud. - Write Linux kernel modules for private logging and telemetry. - Optimize AI workloads across TEE hardware from NVIDIA, AMD, and Intel. - Build and package software securely in TEEs for attested builds and supply chain security. - Collaborate with the TEE hardware vendors on next generation TEE implementations. ### Requirements Confidential values these traits above all others: - Consistent: You ship daily. - Relentless: You find a way or make a way. - Curious: You thrive in the unfamiliar. New problems energize you. - Collaborative: People want to work with you. You communicate clearly. ### Qualifications Thats it. Really. We dont care where, or if, you went to school. ### Bonus Experience It's a bonus, not a requirement, if you have experience with any of the following: - Distributed systems - Cryptography - Low level systems engineering - Python, Golang, and/or Rust - Linux - Kubernetes - AMD, Intel, or ARM Trusted Execution Environments (TEEs) - NVIDIA Confidential Computing (CC) - AI inference, training, and/or fine-tuning ## Outro Don't see the right opening for you? [Email us](mailto:hello@confidential.ai). We'd love to hear from you. Confidential is an equal opportunity employer. We covet diversity and inclusiveness. We do not discriminate on race, religion, color, nationality, gender, gender identity, sexual orientation, age, or marital, veteran, or disability status.