Encrypted KV Cache Offloading for Confidential Computing

We're presenting Encrypted KV Cache Offloading, the first production implementation of KV cache offloading under confidential computing (CC). On NVIDIA B300s with CC on, it raised throughput 141% on llm-d's benchmark, matching the gain offloading gets without CC, and cut mean time to first token from 39 s to 1.5 s. To get there, we raised the speed of moving encrypted KV between GPU and host from about 4 GB/s through the standard CC copy to 50 GB/s, 12 times faster. Everything written to host memory stays encrypted under a key that never leaves the confidential VM and its GPU, and we still get the full gain of offloading. When the cache comes back, the GPU's copy engine checks every block in hardware, so anything the host tampers with is rejected before it reaches the KV cache.

This post covers why CC out of the box makes offloading slow, what we changed to make it fast, and the numbers we measured on two workloads: llm-d's public benchmark and our production MiniMax M3 deployment. New to CC? Start with the Introduction to TEEs.

Throughput gain over HBM-only serving on llm-d's benchmark. Each group is one of llm-d's load points, which we ran at half the rate on half the replicas. llm-d didn't publish a 15 QPS point.

GPU memory becomes a bottleneck in agent loops and long-context conversations. Every token a model has read leaves behind attention state, called the KV cache, and because agents resend their whole history on every turn, that cache grows until the GPU runs out of room. Once it does, the server throws old conversations away and recomputes them when they come back, spending GPU time on work it already did. This is typically solved with KV cache offloading, which parks the cache in the host's much larger memory and copies it back when needed. On a host without CC, llm-d measured a 2.4x throughput gain from doing this.

Under stock CC, KV cache offloading runs without changes, and it backfires. Host memory outside the confidential VM sits outside the trust boundary, so the KV cache can't be stored there in plaintext, and the safe path is slow. Every copy between GPU and CPU passes through an encrypted bounce buffer that moves about 4 GB/s from one process, and every call across that boundary pays a fixed cost of about 50 µs before any data moves. At those speeds restoring the cache takes longer than recomputing it, and with CC on, stock offloading cut throughput by more than half.

Why offloading is slow under CC

Under CC, the GPU's copy engine can still move bulk data at the full speed of the PCIe link. What gets slower is every interaction between the CPU and the GPU. On a B300 in an Intel TDX confidential VM, launching one kernel took 27 µs, launching one and waiting for it took 57 µs, and a 4 KiB copy took 45 to 55 µs before any data moved. Our benchmark of CC on NVIDIA Blackwell GPUs measures this as a fixed cost on every host operation.

Bulk copies pay even more. The GPU can't write directly into the confidential VM's encrypted memory, so a standard copy goes through a shared bounce buffer, with the GPU encrypting on one side and the CPU decrypting in software on the other. A standard GPU-to-host copy moved 3.3 to 4.3 GB/s from one process and 24 GB/s with eight processes copying at once. Setting up the host cache is slow as well, because the confidential VM won't let the GPU map its memory directly, so stock SGLang falls back to driver-managed pinned memory that allocates at about 4.4 GB per minute. A 169 GB host cache took 38 minutes to allocate.

A standard GPU-to-host copy under CC. The data crosses a shared bounce buffer, encrypted by the GPU on one side and decrypted in software by the CPU on the other.

On llm-d's benchmark, the result is that stock offloading makes serving slower than not offloading at all. With CC on, a stock SGLang HiCache replica served 5.4K tokens/s at the 10 QPS load point, compared with 14.5K tokens/s without offloading, and its mean time to first token (TTFT) rose from 6.6 s to 42 s. At 20 QPS, 141 of 150 requests timed out. The cache itself was doing its job, serving 89% to 100% of prompt tokens, but the GPU sat 100% busy while the CPUs were 94% idle, so the time was going into copies between the GPU and host memory.

How Encrypted KV Cache Offloading works

Encrypted KV Cache Offloading plugs into SGLang's HiCache, which still decides what to offload, restore and evict. What changes is how the data moves. Instead of copying through the bounce buffer and decrypting on the CPU, the GPU encrypts the KV cache itself before it leaves, so what lands in host memory is already ciphertext. The encryption key is generated inside the confidential VM at startup and never leaves the VM and its GPU, and the CPU never reads or writes KV data.

When a conversation returns, the copy engine checks the integrity of each block of KV in hardware as it brings it back, so anything the host changed or swapped is rejected before it reaches the KV cache.

Encrypted KV cache offloading. KV leaves the GPU only as ciphertext, and the copy engine checks each block in hardware on the way back.

Results on llm-d's benchmark

llm-d's tiered prefix-cache benchmark runs Qwen3-32B on SGLang and compares HiCache offloading with HBM-only serving, on eight servers of two H100s each, without CC. Its working set is larger than the GPU cache of all eight servers combined, which is the situation offloading is built for. We ran it on B300s, inside an Intel TDX confidential VM with GPU CC on for both configurations, using four single-B300 replicas each, half of llm-d's prefix groups and half its request rate at each load point. Both configurations served the same prompts at the same time behind SGLang's cache-aware router. Every replica kept llm-d's GPU cache size of 321,856 tokens, and ours added a host cache 2.4 times that size.

Measured against HBM-only serving, our implementation gained 141.3% at llm-d's 20 QPS load point, against 138.8% for stock offloading without CC, while stock offloading under CC lost more than 60% at every load point it completed, as the chart at the top of this post shows. Our implementation also cut mean TTFT at every load point, by 27 times at 20 QPS.

For comparison, llm-d's HiCache without CC cut mean TTFT from 40.6 s to 1.56 s at 20 QPS. The two baselines line up closely, with our HBM-only replicas serving 15.2K tokens/s at that point and llm-d's serving 14.7K, so one B300 with CC on starts from about the same place as one of llm-d's two-H100 servers without CC, and the gains compare directly.

Against stock HiCache under CC, encrypted offloading served 3.5 times the throughput at 10 QPS (18.9K compared with 5.4K tokens/s per replica) and 4.9 times at 15 QPS (29.0K compared with 6.0K), while mean TTFT dropped from 42.1 s to 0.61 s and from 97.7 s to 0.34 s.

Results on MiniMax M3

MiniMax M3 is our production deployment of Encrypted KV Cache Offloading, served at TP4 on B300s. Each worker holds 5.2M tokens of KV cache in GPU memory, and Encrypted KV Cache Offloading adds a host tier of 7.8M tokens. Unless noted, it runs in write-through mode, which copies each new piece of history to host memory once GPU memory runs low. We drive both workers with a replay of 800 multi-turn conversations whose prompts start around 24.5K tokens and grow by about 1.2K tokens each turn, and a second TP4 worker without offloading runs the identical replay at the same time, so both see exactly the same load.

The gain is largest once the working set outgrows GPU memory. With 512 conversations, 256 requests in flight and the GPU cache full, encrypted offloading produced 3.15 times the output of the worker without it (1,364 tokens/s), served 95% of prompt tokens from cache compared with 39%, and cut the median wait for a returning conversation's first token from 86 s to 26 s. Stock HiCache without CC gets 3.1 times in the same situation, so under CC, encrypted offloading delivers the same gain offloading delivers without it. Both workers are saturated at this load, so we also swept lower loads to see the latencies a real service would run at.

With 400 live conversations and at most 80 requests in flight per worker, we set the load by varying the pause between each conversation's turns. At every load, encrypted offloading kept more of each prompt cached and returned the first token sooner:

Each cell is with encrypted offloading / without offloading, measured at the same time.

A production service promises latency, so we also measured the most load each worker carries while staying inside a latency target, which we set as a median TTFT of at most 3 s, at most 80 requests in flight and at most 1% of requests waiting for a slot. Encrypted offloading stayed inside the target at 2.58 requests/s (646 output tokens/s) with a 99th-percentile TTFT of 1.25 s. At that load a quarter of the requests to the worker without offloading were waiting for a slot, and the most it carried inside the target was 2.18 requests/s (515 tokens/s). That is 1.19 times the requests and 1.25 times the output, and a repeat run gave 1.19 and 1.28 times. Each worker's limit falls between two load points we measured, so the true request ratio lies between 1.11 and 1.31.

We also held the load for a full hour at 150 s pauses, with conversations at mixed ages as they would be in production. For this run, offloading used write-back mode, which keeps each conversation's history in only one place, GPU or host memory. It served 86% of prompt tokens from cache compared with 39% without offloading, and delivered 1.38 times the requests and 1.37 times the output (656 compared with 477 tokens/s) at less than half the median TTFT (2.3 s compared with 5.0 s).

When the working set fits in GPU memory, offloading has nothing to save and ideally costs nothing. In that case encrypted offloading ran 4.5% to 6.5% below the worker without it, while stock HiCache without CC ran 27% below its own no-offload baseline in an earlier run of the same workload.

What made it fast

The first working version already served 4.14 times the output of stock HiCache under CC on MiniMax M3, but when the working set fit in GPU memory it ran 39% slower than not offloading at all. Almost all of that gap came from round trips between the CPU and the GPU, each of which costs tens of microseconds under CC. Cutting those round trips raised throughput 45% when the working set fits and 54% once the GPU cache fills, took the median restore from about 1 s to 88 ms, and brought the overhead when it fits down from 39% to 4.5% at best.

The same lesson applies to any GPU code under CC. Count the round trips between CPU and GPU on the hot path, batch them or capture them in CUDA graphs, and leave bulk data to the copy engine. In one test, 100 kernels launched one at a time took 2,833 µs, while the same kernels captured in a single CUDA graph took 147 µs.

Limitations

• Intel TDX only. Every result here comes from NVIDIA B300s in an Intel TDX confidential VM, and we haven't yet tested on AMD SEV-SNP. The CC-off numbers are llm-d's published H100 results and a MiniMax M3 run on the same hardware three days earlier, booted from the same guest image with CC off.
• Side channels. The host never sees KV contents, but it does see how large each block is and when it is written and restored, which can reveal how long conversations are, when users come back and which requests share a prefix. Timing side channels from KV cache sharing have already been shown to leak prompts in multi-tenant serving, and we don't yet hide these access patterns.
• Availability. The host can delete blocks, which turns a cache hit into recomputation, since availability is outside what CC protects. A block that fails its integrity check stops the server rather than serving bad data, so the system fails closed.

Toward CC by default

Our goal is to make confidential computing the default for AI inference. That only happens if turning CC on doesn't mean giving up the performance optimizations people already rely on. KV cache offloading is one of the biggest, and it now works under CC with the same gain. This is a big step toward getting there.

Further reading

• llm-d's Qwen3-32B offloading results
• llm-d's tiered prefix-cache guide
• SGLang HiCache
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Benchmarking Confidential Computing Performance on NVIDIA Blackwell GPUs
• NVIDIA confidential computing documentation
• Introduction to TEEs
• Private AI Inference: What It Actually Requires