I was obsessed with MTEB leaderboards. Every time a new embedding model hit the top, I’d drop everything to test it. I’d read the papers, run the benchmarks, and convince myself I’d found the holy grail of semantic search.
Then I’d plug it into my own retrieval pipeline. And it would flop.
The best embedding model on a generic benchmark is often the worst model for your actual data. That’s not hyperbole – it’s the dirty secret of the vector search world.
I spent months debugging why my RAG system was returning irrelevant results. The model was top of the MTEB chart. The code was clean. The vectors were indexed. But the results? Garbage. Because MTEB tests on news articles, scientific abstracts, and generic Q&A. My data? Legal contracts, product descriptions, and internal taxonomy. Different beasts entirely.
That’s when I realized: you can’t evaluate embedding models on someone else’s data and expect it to work on yours. It’s like testing a car on a racetrack and assuming it’ll handle a muddy mountain road.
So I built Embench – a playground where you can upload your own data, define your own taxonomy, and compare embedding models side by side on your exact retrieval tasks. No more trusting leaderboards. No more guesswork.
Embench started as an internal tool. I was working on clustering and matching problems, and I needed a quick way to see which model performed best on my specific corpus. The generic benchmarks were useless – they gave me false confidence. So I wrote a script that let me test multiple models on the same data with the same metrics. It worked. Then I thought: why not share this?
The result is a fully interactive web app. You can upload your documents, define your query types, and run comparisons on retrieval, clustering, or classification. The interface is clean, the feedback is instant, and the insights are real. For the first time, you can see exactly how a model behaves on your data – not on a curated academic dataset.
Here’s what I learned from building it: the gap between leaderboard performance and real-world performance is huge. Models that dominate on MTEB often fail on domain-specific language, rare entities, or noisy inputs. Conversely, models that score lower on generic benchmarks can be surprisingly effective when tuned to your data.
That’s why I’m sharing this publicly. If you’re building any system that relies on embeddings – search, RAG, clustering, recommendation – you owe it to yourself to test on your own data. Stop chasing the leaderboard. Start chasing relevance on your own terms.
Embench is free, open-source, and ready to use. Upload your data, pick your models, and see what really works. No sign-up required. No hidden costs. Just the truth about your embedding performance.
Your data deserves better than a generic benchmark. Give it the test it deserves.
FAQ
Q: Why can't I just use MTEB scores to choose an embedding model?
A: Because MTEB tests on generic tasks and datasets that may not match your data distribution. Your retrieval pipeline is unique – queries, documents, and relevance criteria are specific to your domain. A model that scores high on news articles might fail on legal contracts. You need to test on your own data.
Q: What's the practical implication of using Embench?
A: You can quickly compare multiple embedding models on your own data, see real retrieval metrics, and make an informed decision. It saves hours of trial and error and prevents costly mistakes in production RAG systems.
Q: Aren't leaderboards useful for initial filtering at least?
A: They're a starting point, but they can be misleading. Many models are fine-tuned on benchmark datasets, leading to overfitting. Embench lets you validate that initial pick on your actual data. Don't trust a model until you've tested it yourself.