Testing Code You Are Not Allowed To Read
Most of our components need a Trusted Execution Environment (TEE) to function, so our test suites cannot run on ordinary CI. We built infrastructure that runs them inside confidential VMs, in four environments, on every merge. The same machinery lets one party run tests against another party's code without either side handing over its secret.
Most of what we build doesn't work on an ordinary machine. Our attestation library asks the processor for a signed report describing the environment it is running in. Attestation is that exchange: the hardware states what booted, signs the statement with a key the chip vendor burned in at the factory, and someone outside checks it. We call that checker the verifier. The machine's owner cannot forge that signature, which is the whole point, and producing one at all needs real hardware. c8s, our confidential Kubernetes, issues a workload certificate only after the node proves which image it booted. Confidential OS Builder produces disk images whose purpose is to be measured by the hardware before they start.
Point that at a GitHub-hosted runner, the ordinary machine GitHub runs a CI job on, and the tests that matter never execute at all. The attestation code sits behind a build flag, so a default build never compiles it. The hardware tests are marked ignored, so they run only when something asks for them. A skipped test doesn't turn a run red, so the suite goes green having tested none of it.
So we built Confidential CI, which runs those suites inside confidential VMs (CVMs) at the scale a merge queue needs. This post is about the bugs that surfaced once the tests ran on real hardware, and what a green run from it actually proves. The last section gets to the case in the title: the same machinery lets one party benchmark another party's model without either side giving up its secret.
a merge lands
│
▼
┌─────────────────────────────────────────────┐
│ off the runner: pick a random number │
└─────────────────────────────────────────────┘
│
▼
╔══ inside the CVM ═══════════════════════════╗
║ run the tests ║
║ ask the hardware to sign a report ║
║ that includes that number ║
╚═════════════════════════════════════════════╝
│
▼
┌─────────────────────────────────────────────┐
│ off the runner: check the signature, │
│ the image it booted, and the number │
└─────────────────────────────────────────────┘
│
▼
pass or fail
Two of the four steps happen where the tests cannot reach them.
The bugs only real hardware finds
Without hardware, you test attestation code against a fixture. Record a report, return it from a fake device, compare against a known-good value. Those tests are worth having and they catch parsing and error handling bugs.
Everything up to the verification boundary can be mocked. The verifier itself cannot. A test double that satisfies the verifier is, by definition, a forgery the verifier accepted.
We found two bugs this way that a fixture couldn't have caught. In the first, the guest and the verifier disagreed about how to encode the random number that proves a report is fresh. Against a recorded report the encoding looked correct, because the recording came from the same code that produced it. The second showed up only on the machines we rent. There the report doesn't come straight from the processor: the cloud gives each machine a virtual Trusted Platform Module, an emulated security chip whose signing key the hardware report vouches for. Two of our test processes competed for its sessions, and the attestation failed. A fixture is a value, and no value contains a state machine.
What green actually claims
"Our tests run in a TEE" gets treated as one claim, which a pipeline either satisfies or does not. It is really five claims stacked, and each one removes a different assumption.
the code compiled ordinary CI
...on confidential hardware a TEE runner
...on the image we chose a pinned measurement
...during this run, not a replay a random number from outside
...and the guest did not grade itself an external verifier
Our own pipeline sat on the second rung for most of the year: confidential hardware, and nothing above it. The rest just shipped, and the sections below cover what each step cost. Supply-chain provenance frameworks such as SLSA and in-toto climb a ladder like this one for who built an artifact; ours is about where the tests ran.
Where a test actually runs
Untrusted: the host, the hypervisor, the cloud operator, and us
╔══ inside the CVM ════════════════════════════════════════╗
║ ║
║ ┌──────┐ ┌╌╌╌╌╌╌┐ ┌╌╌╌╌╌╌┐ ┌╌╌╌╌╌╌┐ ║
║ │ test │ ┊ test ┊ ┊ test ┊ ┊ test ┊ ║
║ └──────┘ └╌╌╌╌╌╌┘ └╌╌╌╌╌╌┘ └╌╌╌╌╌╌┘ ║
║ running created per CI job, destroyed after it ║
║ ║
╚══════════════════════════════════════════════════════════╝
═══ a double wall is memory the host cannot read
Same wall whether we own the machine or rent it.
The double wall is the hardware boundary. Everything outside it is untrusted, and on the left that includes us. We run four configurations: AMD SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging) and Intel TDX (Trust Domain Extensions), each on a machine we operate and a machine we rent. The evidence reaches the verifier by a different code path in each, and each path fails in its own way. The virtual TPM bug above surfaced on the rented ones alone.
Why the guest cannot vouch for itself
Asking a guest whether it is confidential is circular. A guest reads its own environment, and a host can present a convincing one, so the presence of an attestation device proves less than it appears to. The fix is a nonce, the name for that random number. The verifier picks one fresh for each run, and the guest has to fold it into the report the processor signs.
┌────────────────────┐ ╔═ CVM ════════════════════╗
│ Verifier │─── nonce ────►║ ║
│ │ ║ ┌──────────────────┐ ║
│ [y] signature │ ║ │ Runner │ ║
│ [y] measurement │◄── report ────║ │ Tests │ ║
│ [y] nonce │ ║ └──────────────────┘ ║
└────────────────────┘ ╚══════════════════════════╝
▲
┊ fixture [n]
The runner sits inside the boundary. The verifier stays outside it.
A separate CI job outside the CVM picks the nonce, so the answer has to be about this boot and no earlier one. A second job outside the CVM verifies the report, so the guest never grades itself. Both properties come from where those jobs run, and both jobs land on GitHub's ordinary runners. That's fine. Picking a random number and checking a signature are the two steps that don't need confidential hardware, and whoever controls GitHub already controls the source we're testing.
What the boundary costs to renew
┌────────────────────────────┐
│ Host │ outlives everything
│ ╔═ CVM ══════════════════╗ │ provisioned once
│ ║ ┌──────────────────┐ ║ │
│ ║ │ Runner │ ║ │ one job, then gone
│ ║ │ Test process │ ║ │
│ ║ └──────────────────┘ ║ │
│ ╚════════════════════════╝ │
└────────────────────────────┘
The boundary is renewed less often than the thing inside it
A CVM booted fresh for every run carries its own evidence, because the boot produces the launch measurement, the hash of the image the hardware records as it starts. It's also the slow path. A standing CVM that jobs land on is quick, and its evidence is from whenever it was provisioned.
The slowness and the freshness have the same cause. Booting is expensive, and booting refreshes the evidence, so you don't get to buy one without the other.
Attesting per run on a standing CVM is nearly free: producing and checking the report take seconds, and the rest of the cost is scheduling jobs and setting up toolchains. It buys back the top of the ladder without a reboot, so how often you boot and how often you attest are independent choices. It still leaves you on the same machine, though. The report proves the same boot, later, and whatever earlier jobs changed on that guest rides along. More on that in the next section.
There is also a third option: a pool of guests booted fresh and held idle. That makes the evidence current and the queue empty, and charges for both in idle hardware, which our merge rate doesn't yet justify.
What the report covers
╔══════════════════════════════╗
║ ┌────────────────────────┐ ║ ╮
║ │ Firmware │ ║ │
║ │ Kernel │ ║ ├── launch measurement
║ │ initrd │ ║ │
║ │ Command line │ ║ ╯
║ └────────────────────────┘ ║
║ ║
║ ┌────────────────────────┐ ║
║ │ CI agent │ ║ ╌╌╌ not covered
║ │ Fetched artifacts │ ║
║ └────────────────────────┘ ║
╚══════════════════════════════╝
The hardware measures what booted. Everything after that is on you.
A launch measurement covers the guest image at boot. It does not cover the CI agent installed afterwards, or anything a previous job left behind. TDX has runtime measurement registers that software can extend after boot. SEV-SNP doesn't, so post-boot measurement there rides on a software stack such as the virtual TPM. Either way, post-boot state is only measured if something has been written to measure it.
Two limits on what we shipped, and both narrow the window rather than close it. Every chip signs with its own key, backed by a certificate chain ending at AMD's or Intel's root, and each vendor publishes a list of keys it has withdrawn. Our verifier walks that chain to the root on all four configurations, and on the Intel ones it also fetches the withdrawal lists. On the AMD ones it does not, so a report signed by a key AMD has since revoked would still pass. And attesting the runner is not the same as attesting the tests, since each job gets its own fresh pod and the pods share only the guest.
Code you are not allowed to read
We built this for our own repositories, but nothing in the mechanism requires the tests and the code under test to come from the same place.
┌──────────────┐ ╔═══════════════╗ ┌──────────────┐
│ Model owner │ ║ Attested ║ │ Evaluator │
│ │─ sealed ─►║ guest ║◄─ test ───│ │
│ │ weights ║ ║ code │ │
│ checks it │◄─ report ─║ ║ │ never sees │
│ before the │ ║ opens them, ║── score ─►│ the weights │
│ key moves │── key ───►║ runs the test ║ │ │
└──────────────┘ ╚═══════════════╝ └──────────────┘
The key moves only after the report checks out.
Safety evaluation of a frontier model is where this bites hardest. The benchmark needs the weights, which usually leaves two options and neither is a good position for an audit. Either the owner runs the benchmark and reports a number, which asks you to trust the party being measured. Or the weights move to the evaluator, which is normally refused outright.
A third arrangement avoids both. The benchmark runs as the payload, and the sealed weights come to it. The owner first checks a report showing which image booted and which harness is loaded. Only then does it release the model's decryption key, encrypted to a public key that the same report vouches for, so no other machine can open it. The score comes back with a report naming both, and the evaluator never holds the weights. In the simple version the owner reads the harness, because approving what runs against your weights is the basis on which you release the key. A stronger version keeps both secrets sealed. The owner approves the harness by its measurement instead of reading its source, and the evaluator's prompts stay hidden for the same reason the weights do. In that version the only thing holding the arrangement together is a runtime both parties checked.
Three limits are worth stating. The first: this only works with the vendor chain checked to the end, revocation included, so the AMD gap above is the first thing to close here. The result channel is also an exfiltration channel. A harness that chooses the prompts and reads the answers can extract a great deal about a model, so the report has to pin the harness and not only the environment. And a model evaluation runs on a GPU, while these platforms encrypt CPU memory. Current NVIDIA hardware has a confidential mode that attests separately and protects device memory by controlling access rather than by encrypting it, to be checked alongside the CPU report. We have built the CPU side.
c8s, Kettle, our attestation libraries and Confidential OS Builder are open source.