Private AI Inference: What It Actually Requires

Everyone sells "private" inference. Almost nobody defines it. This post does: private AI inference means the provider cannot see your prompts, cannot see your responses, and can prove it. Anything less is a privacy policy with an API attached.

Three requirements. Miss any one and the privacy is a promise, not a property.

Requirement 1: protected during processing, not just in transit

TLS gets your prompt to the machine. It says nothing about what happens on the machine. The moment a normal server decrypts your request, the prompt sits in plaintext memory that the host OS, the hypervisor, and anyone with root can read. Most "private" inference is exactly this: encrypted to the door, exposed inside the house.

Private inference protects the data while the model runs. In hardware terms: prompts and responses stay encrypted in Confidential VM memory, the PCIe path between CPU and GPU is encrypted, and the model weights sit in the GPU's hardware-protected region, inaccessible to the host. That is what Trusted Execution Environments (TEEs) do. New to them? Start with the Introduction to TEEs.

Requirement 2: the operator locked out

Access controls do not count. IAM policies, audit logs, and "least privilege" are the operator's rules, running on the operator's machine, changeable by the operator. Privacy that the provider grants is privacy the provider can revoke.

The lockout has to come from below the operator. In a TEE, the CPU refuses to expose guest memory to the host, the hypervisor, or an admin with root. The provider can schedule your workload, bill it, and kill it. They cannot look inside it. Silicon, not policy. For where that boundary sits versus containers and microVMs, see The Isolation Spectrum.

Requirement 3: proof, not promises

The first two requirements are invisible from the outside. A provider claiming them looks identical to a provider faking them. So the third requirement is the one that makes the others real: attestation.

TEE hardware signs a report of exactly what code booted and what environment your request runs in. You verify that signature against the hardware vendor's root of trust, not against the provider. Done right, every inference response carries one. If you cannot verify it yourself, you are back to trusting someone.

What does not qualify

• "We don't train on your data." A training policy. Your data is still visible to the provider; they are just promising not to learn from it.
• Zero retention and SOC 2. Auditing the operator, not removing them. The auditor checks that the provider behaves. Private inference removes the provider's ability to misbehave.
• VPCs, private endpoints, dedicated instances. A private network path to compute the operator can still read. The packets are isolated. The memory is not.

None of these are worthless. They are just not private AI inference. They are trust, arranged more carefully.

The performance question

The old objection was cost: privacy meant giving up speed. Measured on our stack, confidential inference runs at 4% lower token throughput than non-confidential inference, with negligible impact on time to first token. On CPUs, TEE overhead measures around 1.5% on compute-heavy workloads (benchmarks here). Privacy is no longer a performance tax worth arguing about.

What it looks like in practice

Confidential Inference is private AI inference as an API. OpenAI-compatible: switch the base URL, keep your code.

client = OpenAI(
    base_url="https://api.confidential.ai/v1",
    api_key="$CONFIDENTIAL_API_KEY",
)

Prompts and responses invisible to us and our infrastructure. An attestation on every response, verifiable against the hardware vendor's root of trust. Per-token pricing, no commitments.

Private, done right, is confidential

There is a name for inference that meets all three requirements: confidential AI. Hardware-enforced privacy plus the receipt that proves it. "Private" is the claim anyone can make. Confidential is the version you can check. For the full picture of the category, see What is Private AI?