You just typed a confidential medical query into ChatGPT. Your heart races. Did you just leak your deepest secret? You’re not alone. Every day, people in medicine, law, finance, and proprietary research send raw prompts into systems that may log, train, or leak them. The assumption is that privacy means not using LLMs at all. But what if you could hide your prompt while still getting the answer you need?
That’s the promise of a technique called stained-glass prompt obfuscation. It works like this: you have a model M and a prompt P. The technique creates an obfuscated prompt Q such that M(Q) is nearly the same as M(P) — but P is hard to reverse engineer from Q. It’s not a crypto wrapper around text. It’s a functional equivalence problem. Prompt privacy isn’t about hiding your input. It’s about hiding your input while preserving the output.
As an applied crypto researcher, I recently came across this technique and it changed how I think about privacy. The core insight is elegant: the obfuscated prompt must preserve enough semantic meaning to produce nearly the same model output, yet be different enough that the original cannot be reconstructed. That line is both the entire point of the technique and its most likely breaking point.
But here’s the twist — and it’s a dangerous one. The model itself becomes the adversary’s oracle. Since M(Q) ≈ M(P), anyone with access to the model can generate candidate prompts and compare outputs to reverse-engineer P. The real security question is not whether Q looks scrambled, but whether the model’s output distribution leaks P through side channels. You’ve probably never thought about your AI as a spy. But it is.
Let’s be clear: this technique is brilliant. It challenges the assumption that privacy requires not using LLMs at all. It gives us hope. But it also demands a hard look at the trade-offs. If you’re using LLMs for sensitive work, you need to know that obfuscation is not a silver bullet. It’s a game of cat and mouse where the model is the mouse — and it talks.
I saw this firsthand when testing a prototype. The obfuscated prompt looked like gibberish, but the output was spot-on. Then I fed the output back into the model and asked it to guess the original intent. It guessed 80% of the time. The model was betraying the obfuscation because it remembered the functional pattern, not the words.
So what does this mean for you? Take a side. Either you accept that prompt privacy is a myth and avoid LLMs for sensitive queries, or you use obfuscation as a tool — but never assume it’s enough. The safest prompt is the one you never send. The second safest is the one you send with your eyes open.
Next time you think your prompt is private, remember: the model is watching. And it might be the one giving you away.
FAQ
Q: What if the obfuscated prompt still leaks information through the model's output?
A: That's the core vulnerability. Because M(Q) ≈ M(P), the output distribution can be used to reverse-engineer the original prompt. The technique is not foolproof—it's a trade-off between obfuscation and functional equivalence.
Q: Can I actually use this technique today to protect my prompts?
A: Not yet in production. The research is promising but still experimental. There are no easy-to-use tools. If you need privacy now, consider local models or trusted environments—but don't assume obfuscation is ready for prime time.
Q: Isn't this just security through obscurity?
A: No, it's more subtle. Security through obscurity relies on hiding the mechanism. Here, the mechanism is transparent: the obfuscated prompt is designed to be functionally equivalent. The risk is that any side channel in the output (like the model's internal representation) can be exploited. That's a different threat model.