Cohere encrypts AI inference in Model Vault, enhancing data privacy

Cohere: Cohere is a Canadian AI startup focused on developing enterprise-grade large language models and inference platforms. It offers Model Vault as a single-tenant deployment option for customers seeking dedicated model hosting. The company has added confidential computing to Model Vault to encrypt data during AI inference, ensuring neither Cohere nor cloud providers can access customer prompts or outputs. Nvidia: Nvidia produces GPUs that support confidential computing modes for isolating and encrypting AI workloads during execution. In Cohere's Model Vault, Nvidia GPUs operate alongside CPU technologies such as Intel TDX and AMD SEV-SNP to extend protection across the full inference path. This hardware enables the encrypted tier that prevents access by operators or the cloud provider. Aidan Gomez: Aidan Gomez is the CEO of Cohere. He has highlighted trust, security, and governability as core priorities for the company's AI offerings, particularly in enterprise settings. Gomez positioned recent developments around Model Vault alongside the company's merger with Aleph Alpha as advancing secure and auditable AI systems. Manoj Govindassamy: Manoj Govindassamy is Cohere's director of serving inference. He detailed the rollout of confidential computing support in Model Vault, confirming it runs workloads inside hardware-encrypted environments that protect data across CPU and GPU operations. Govindassamy noted that every inference includes an attestation report for customer verification and that the feature incurs no extra cost. Data Privacy Concerns: Enterprise worries about AI vendors accessing or retaining customer data have increased following policy adjustments at other major providers that affected usage by companies including Nvidia and Palantir. Verification Mechanisms: Attestation reports and open-source tools allow customers to independently confirm that inference runs inside verified secure environments with approved hardware and software configurations. Confidential Computing Adoption: Confidential computing relies on hardware encryption and attestation to protect data while it is actively processed by models, moving beyond traditional protections limited to data at rest or in transit.