Can Homomorphic Encryption Bring Privacy to Externally Hosted LLMs?

For enterprises handling highly sensitive data, keeping everything in-house has always been the gold standard for security. However, the rapid advancement of generative AI, especially large language models (LLMs) developed by companies like OpenAI and Anthropic, has created a dilemma. The cost of deploying, maintaining, and scaling these models on-premise complete with dedicated hardware and high-performance GPUs is steep. Meanwhile, third-party LLM providers offer powerful models as-a-service, at a fraction of the cost and with lower operational complexity.

This has forced some security-conscious organisations to bend their policies, allowing data to flow outside their networks in pursuit of AI-driven productivity and innovation. But what if there were a way to have both: full control over your data and the ability to use externally hosted AI services?

Enter homomorphic encryption.

What Is Homomorphic Encryption?

Homomorphic encryption (HE) is a cryptographic technique that allows computation to be performed directly on encrypted data, without ever needing to decrypt it. That means a cloud provider—or any third-party processor—can perform operations on your data without ever seeing its contents.

A Brief History: Who Invented It and Why?

The concept was first proposed in 1978 by Rivest, Adleman, and Dertouzos, but the first viable scheme wasn’t introduced until 2009 by Craig Gentry, then a Ph.D. student at Stanford. Gentry’s breakthrough introduced the first fully homomorphic encryption (FHE) system, which proved that it’s theoretically possible to compute arbitrary functions on ciphertexts.

The motivation behind HE has always been the same: secure delegation of computation. Whether in cloud computing, financial services, or now AI, the goal is to process data without revealing it—an ideal solution for privacy-preserving applications.

Types of Homomorphic Encryption

Homomorphic encryption is not one-size-fits-all. There are different types, each with trade-offs in terms of functionality and performance:

• Partially Homomorphic Encryption (PHE): Supports only one operation (either addition or multiplication, but not both). Useful in limited applications like encrypted voting or basic statistics.
• Somewhat Homomorphic Encryption (SWHE): Supports both operations but only for a limited number of times before the ciphertext becomes too noisy to be useful.
• Fully Homomorphic Encryption (FHE): Supports unlimited operations on ciphertexts. This is the holy grail, but it’s also the most computationally intensive.

FHE is the type that could, in theory, enable encrypted inference on large language models. But that brings us to the next question.

Can Homomorphic Encryption Be Used with LLMs?

Let’s consider the core desire of many enterprise customers: to retain full control over their data while still benefiting from the raw power and innovation of externally hosted LLMs.

The Case For It

In theory, FHE could allow a user to encrypt their data locally, send it to an external LLM, and receive an encrypted response—all without the model ever seeing the plaintext data. This would enable secure, privacy-preserving AI services in sensitive environments like finance, healthcare, or government.

For example, a healthcare provider could submit encrypted patient data to an LLM hosted by a third party and get back an encrypted summary, diagnosis recommendation, or insurance claim code—without violating HIPAA compliance.

The Case Against (At Least For Now)

While the theory is sound, the practical limitations are significant:

• Performance Overhead: Current FHE schemes are orders of magnitude slower than operations on plaintext. Even with hardware acceleration and algorithmic optimizations, processing encrypted data through something as compute-intensive as an LLM is still impractically slow.
• Model Complexity: LLMs involve billions of parameters and matrix multiplications—operations that are hard to efficiently perform on encrypted inputs using today’s FHE schemes.
• Limited Tooling: The FHE ecosystem is still young. While frameworks like Microsoft SEAL, IBM HELib, and OpenFHE are promising, they aren’t yet production-ready for real-time, large-scale LLM inference.

Why HE and LLMs Don’t Play Well Together

One critical technical mismatch is that homomorphic encryption was originally designed with CPUs in mind, not GPUs.

Most FHE implementations today rely on arithmetic circuits that are better suited for scalar operations on CPUs, not the parallel matrix multiplications that LLMs use. LLMs are optimized for hardware like NVIDIA GPUs or Google’s TPUs, which excel at dense tensor operations—operations that are currently inefficient or unsupported in FHE libraries.

In short, HE is CPU-friendly, but LLMs are GPU-hungry.

Efforts are underway to bridge this gap, including research into GPU-accelerated homomorphic encryption and hybrid privacy-preserving AI architectures. However, at present, this disconnect adds yet another barrier to using HE for real-world LLM inference at scale.

Summary: Are We There Yet?

Homomorphic encryption represents a compelling vision: a future where sensitive enterprise data never has to leave the safety of its network boundaries, yet can still be processed by the most advanced AI models on the planet. It’s an elegant solution to a long-standing tradeoff between privacy and utility.

But today, FHE is not yet ready for the scale and complexity of modern LLMs. The performance hit is too great, and the tooling too immature. However, research is advancing quickly. With continued investment and innovation, we may soon see hybrid architectures emerge—where certain parts of a pipeline are homomorphically encrypted, or where specialized hardware accelerates FHE operations to feasible speeds.

In the meantime, enterprises must weigh the risks and rewards. Some may continue to make security exceptions for GenAI access. Others may choose local model hosting despite the cost. And a few may begin to experiment with early FHE prototypes, preparing for a future where the phrase “your data is safe” no longer comes with an asterisk.

Bottom Line

While you can’t quite have your cake and eat it—yet homomorphic encryption might just be the technology that brings us closer to doing exactly that.