Performance overhead of confidential computing
What confidential computing costs on NVIDIA B200 GPUs and on CPUs, per workload, with the setup, method and raw results. About 1 to 3% throughput for correctly configured confidential inference.
Confidential computing on Blackwell costs little when the stack is configured for it, and a lot when it isn't. Every GPU figure on this page comes from Benchmarking Confidential Computing Performance on NVIDIA Blackwell GPUs (Amean Asad and Ansgar Grunseid, arXiv 2608.26575, v2 1 Sep 2026). NVIDIA reports the same low single-digit result independently in Hardware-Rooted AI Security That Won't Slow You Down.
How much slower is confidential inference?
About 1 to 3% lower throughput for a correctly configured stack: 1.5% on a 229B mixture-of-experts model served on four B200s, about 3% on eight. A stock stack with CC switched on can lose 30 to 40%, from avoidable configuration, not hardware. Training on eight GPUs costs 10 to 13%. GPU compute, energy draw and usable memory are unaffected.
| Scenario | CC overhead | Where the cost comes from | Recoverable? |
|---|---|---|---|
| 1 GPU, overlap scheduler off | ~2% | Residual per-sync setup | Near floor |
| 1 GPU, overlap on, unpatched framework | ~35 to 39% | Lost compute/copy overlap | Yes, async device-to-host copy |
| 1 GPU, full CUDA graphs + CC patches | Under 1% | Per-step cost removed | At floor |
| MoE serving, 8 GPUs (TP8) | ~3% | Prefill all-reduce over encrypted NVLink | No, but small |
| MoE serving, 4 GPUs (TP4) | ~1.5% | Less encrypted NVLink traffic | No, but smaller |
| MoE serving, expert + data parallel | ~2% | No attention all-reduce | Near zero |
| 8-GPU training (TP/EP) | ~10 to 13% | Encrypted collectives | No, hardware floor |
MoE rows: MiniMax-M2.7 (228.7B total, 6B active, NVFP4), decode-heavy, 1024 tokens in, 2048 out, concurrency 32. TP8 is pinned by two five-repeat-median sessions (2.8% and 3.6%). TP4 keeps 94% of TP8's throughput. The expert/data-parallel row is Qwen3.5-397B at 32k context.
Where the cost comes from
Two independent costs, each with its own lever:
- Per host operation. Every kernel submission and host-device sync crosses an encrypted command channel, about 12 µs each in eager mode. It's a fixed cost per decode step, so it shrinks as batch size grows, and full CUDA graphs plus framework patches remove it.
- Per byte of NVLink traffic. Collectives between GPUs are encrypted (NVLE) at a roughly constant per-byte rate. It doesn't shrink with batch size. What you pay depends on how much of each step is cross-GPU communication.
Measured unit costs on four B200s on the same NUMA node:
| Metric | CC on | CC off | Change |
|---|---|---|---|
| NCCL all_reduce | 156 GB/s | 185 GB/s | −10% |
| NCCL all_to_all | 130 GB/s | 149 GB/s | −10% |
| Device-to-device read, copy engine | 8070 GB/s | 9170 GB/s | −12% |
| Device-to-device read, SM-based | 7693 GB/s | 9388 GB/s | −18% |
| Peer-to-peer short-write latency (median) | 14.5 µs | 3.7 µs | ~4× |
Host-device transfers under CC run through AES-GCM at about 10 GB/s (64 MB transfers), using about one CPU core per GPU, and don't scale across host threads. That matters at load time, not per token, as long as weights and KV cache stay on the GPU.
What changes the overhead
Production-default config, MiniMax-M2.7 at TP8 on eight B200s, single passes (±2 points run-to-run) except the pinned concurrency-32 point.
| Concurrency (1024 in / 2048 out) | Non-CC tokens/s | CC tokens/s | CC overhead |
|---|---|---|---|
| 1 | 172.7 | 176.7 | −2.4% |
| 8 | 892.1 | 895.6 | −0.4% |
| 16 | 1702.6 | 1644.0 | 3.4% |
| 32 | 2571.5 | 2449.3 | 4.8% |
| 64 | 4169.6 | 3942.6 | 5.4% |
| 128 | 6041.6 | 5717.2 | 5.4% |
| Input / output (concurrency 32) | Non-CC tokens/s | CC tokens/s | CC overhead |
|---|---|---|---|
| 512 / 512 | 2170.9 | 2009.9 | 7.4% |
| 1024 / 1024 | 2522.6 | 2392.1 | 5.2% |
| 2048 / 2048 | 2593.2 | 2481.5 | 4.3% |
| 4096 / 1024 | 2412.8 | 2319.5 | 3.9% |
| 32768 / 2048 (concurrency 16) | 1154.6 | 1159.3 | −0.4% |
Overhead falls from about 14% at 256-token outputs to about 1% at 2048, and is within noise at 32k context. Short, prefill-heavy traffic is the one corner that reaches the low teens.
Single GPU, shipped SGLang 0.5.13, Qwen3-8B bf16 with the overlap scheduler on (no CC patches): 34.4% at concurrency 16, 34.6% at 32, 38.9% at 64. The same class of workload with full CUDA graphs and the CC patches drops under 1%. Nemotron-3-Super-120B (overlap forced off): 2.8%, 2.4% and 1.7% at concurrency 16, 32 and 64.
Eight-GPU training, Megatron-core with TransformerEngine:
| Config | CC | Non-CC | Overhead |
|---|---|---|---|
| 41B dense, bf16, TP8 | 15.8 s/step, 550 TFLOPs | 13.95 s/step, 627 TFLOPs | 13% |
| 41B dense, delayed FP8, TP8 | 18.0 s/step, 480 TFLOPs | 14.5 s/step, 600 TFLOPs | 24% |
| ~43B MoE, EP8 | 2.40 s/step, 520 TFLOPs | 2.16 s/step, 575 TFLOPs | 11% |
What CC doesn't cost
- Compute: compute-bound prefill, GEMMs and training compute are unaffected.
- Energy: CC doesn't raise power draw. When it's slower, the GPU is idle, so it draws less.
- Memory: usable VRAM reads 183 GB on a B200 either way. CC reserves a fixed 700 MB at initialization; account for it when sizing KV cache.
How it was measured
Every number is a paired comparison on one physical host. The CC and non-CC runs use the same root disk, the same data disk with identical model bits, and the same GPUs, and differ only in the GPU CC bit and the TDX guest object in the VM launch. Multi-GPU runs are sequential: an all-eight CC-on pass, then an all-eight CC-off pass.
- Host: Intel Xeon 6767P (2 socket), eight NVIDIA B200s on an NVLink fabric.
- Host software: Ubuntu 24.04.3, kernel 7.0.9, QEMU 11.0.0, OVMF 2025.02, TDX SEAM 2.0.14.
- Guest GPU driver: NVIDIA 595.71.05 (open).
- Frameworks: SGLang 0.5.13.post1 and an SGLang branch with CC patches, vLLM 0.21.0 and 0.22.0, Megatron-core 0.19.0 with TransformerEngine 2.18.0.dev0.
- Models: Nemotron-3-Super-120B (NVFP4), Qwen3-8B (bf16), Qwen2.5-72B-Instruct (AWQ), MiniMax-M2.7 (NVFP4), Qwen3.5-397B (FP8), in-house 41B dense and ~43B MoE for training.
Measurement rules that change results under CC:
- Reboot between runs. A CC GPU left in a stale state by earlier CUDA contexts reads 100% utilization at lower throughput. One early run showed 16% on a workload that is really 2%.
- Don't use CUDA event timing. It's restricted under CC by design and can return negative values. Use the GPU
%globaltimerregister or wall-clock throughput. - Profile host-side. GPU kernel profiling is disabled under CC by design. Traced runs were used to attribute cost, never to report throughput.
Data was collected over several weeks across two frameworks and several versions, so compare within a row (each CC versus non-CC pair is controlled), not across rows. The paper also notes one gap: prefill/decode disaggregation under CC wasn't measured, because GPUDirect RDMA and pinned host buffers are unavailable with GPU CC on.
How to run confidential inference at the lowest overhead
- Use full CUDA graphs, always.
- Use a framework with the CC patches (an asynchronous device-to-host copy worker). Check for it before quoting any single-GPU number.
- Keep weights and KV cache resident on the GPU. Stock CC host offload is slow; Encrypted KV Cache Offloading is our fix for KV.
- Size tensor parallelism to what the model needs. Prefer expert and data parallelism over tensor parallelism.
- Favour decode-heavy, long-context workloads. Budget about 5% at very high concurrency.
Other measurements
| Measurement | Result | Setup | Source |
|---|---|---|---|
| CPU TEE overhead | About 1.5% across metrics, 2.08% worst stage | AMD SEV-SNP, zkVM proving workload | TEE Performance on CPUs |
| Encrypted KV cache offloading | Throughput up 141%, mean TTFT from 39 s to 1.5 s | B300s, Intel TDX, GPU CC on, llm-d benchmark | Encrypted KV Cache Offloading |
| GPU-host transfer of encrypted KV | About 4 GB/s standard CC copy, 50 GB/s with our path | B300, Intel TDX | Encrypted KV Cache Offloading |