Cohere’s confidential AI beta checks the server before releasing a request

Encrypting a connection does not prevent a service from reading a request once it arrives. Cohere’s new Model Vault beta addresses the next stage: protecting the data while an AI model processes it. Announced on 22 October, the system uses confidential-computing capabilities on NVIDIA H100 GPUs in Microsoft Azure.[1]

The design combines a protected execution environment with a check performed before sensitive content leaves the customer. The client evaluates signed evidence about the destination’s hardware, firmware and software against an approved policy. If a required property fails—for example, the GPU is not in the expected confidential mode—the client withholds the request.[2]

The protection follows the request into inference

Cohere’s technical account describes Oblivious HTTP, or OHTTP, encrypting application payloads to a key associated with the attested environment. The corresponding private key stays within that environment. Routing infrastructure can forward the encrypted request without reading its prompts or documents, and the response returns encrypted.[2]

Inside the boundary, the model still processes plaintext. Hardware isolation protects the CPU and GPU workload and their connection; this is different from computing directly on encrypted values. The useful separation is between operating the infrastructure and accessing the request’s contents.[2]

Attestation adds a second distinction. A signed measurement establishes what is running; the approval policy decides whether that state is acceptable. The system only gains assurance from the check if the customer has sound reference values and trusts the measured software. A correctly identified but vulnerable application is still vulnerable.[2]

What the protected boundary leaves visible

Cohere lists hardware, firmware, measured software, reference values and attestation infrastructure among the remaining trusted components. The API credential and model identifier remain outside the OHTTP-protected payload, as do observable properties such as timing and message size. Confidential inference therefore does not make all service activity invisible.[2]

The same distinction applies after inference. Protecting a model’s memory does not determine who can read a saved answer or whether a downstream tool should act on it. Nor does it prevent prompt injection, guarantee availability or choose the jurisdiction where computation happens. These are separate parts of an application’s design.[2]

The beta makes data release conditional on the destination matching an approved configuration. Software updates therefore become verification decisions as well as deployments: a changed measurement must satisfy the client’s policy before protected requests can continue.[2]

Report an error or suggest a correction