Securing the Edge: ConfidentialComputing in Distributed AI

The industry standard for securing high-performance compute environments, especially for LLM and AI workloads, is now Confidential Computing.

Standard encryption protects data at rest (storage) and in transit (network). However, the moment data is loaded into memory for processing, it is typically vulnerable. For our client, this gap was unacceptable. They needed a solution where the CPU, RAM, GPU, VRAM, NVLink, and PCIe bus communications were cryptographically protected, preventing unauthorized observation even from the host OS or a malicious administrator.

We identified NVIDIA Confidential Computing as the appropriate solution. By leveraging hardware-based Trusted Execution Environments (TEEs), we could ensure that the memory, CPU state, and GPU execution remained isolated from the host.

Because of the complexity of the stack, we needed to validate every attack vector. Our journey began with deep technical sessions with NVIDIA solution architects to validate our approach to GPU attestation and encryption layers.

We started by configuring a Confidential Virtual Machine (CVM) on an AMD-based server using AMD SEV-SNP with KVM. This was not a "plug-and-play" operation; it required significant updates and patching of the Linux kernel on Ubuntu to a specific version that supported the necessary confidential computing features.

This phase confirmed that we could successfully configure a CVM and verify GPU attestation against NVIDIA’s documentation, giving us the green light to move to production hardware.

The production environment was significantly more powerful. We moved to configuring four Supermicro GPU SuperServer SYS-821GE-TNHR units. These are beasts of computation, designed for LLM training and inference, each equipped with eight NVIDIA H100 GPUs connected via SXM.

Enabling Confidential Computing on this specific architecture presented unique hurdles. We encountered boot issues when enabling Intel TDX (Trusted Domain Extensions).

Razor worked closely with engineers from both Intel and Supermicro to troubleshoot the problem. We discovered the issue lay in the firmware; by installing the correct firmware versions for both the BIOS and the GPUs, we successfully enabled the host server for Confidential Computing.

Validating a complex hardware stack manually is neither scalable nor secure. To streamline this, Razor developed a custom Go-based application to perform system-level checks on all components required for Confidential Computing.

This tool acts as the gatekeeper for the distributed cluster:

1. Validation: It generates a detailed host validation report and securely transmits it to the client’s control servers.
2. Deployment: If, and only if, validation succeeds, the system automatically downloads and launches a pre-built Confidential Virtual Machine.
3. Attestation: Inside the CVM, a secure service provides attestation endpoints. This allows the client to verify the integrity of the CVM itself, while GPU attestation is performed against NVIDIA services to confirm the trusted state of each H100 GPU.

By rigorously implementing and testing these layers, the client gained the ability to run workloads inside encrypted Confidential Virtual Machines backed by verified GPU attestation.

This architecture ensures that no host operator or external actor can access or tamper with customer data. The client can now deploy AI workloads to their distributed cluster with total confidence, knowing that the environment is cryptographically isolated and verified before a single byte of data is processed.

Are you looking to implement Confidential Computing for your AI infrastructure? Get in touch to see how we can help secure your compute stack.