Teams building retrieval-augmented generation (RAG) systems often claim that "dense retrieval beats BM25" or "hybrid search works better." Yet most of the time, no one involved has actually reproduced the baseline against which they're comparing. BM25—a keyword scoring method used for decades—still appears as the baseline in virtually every retrieval paper published today. Dense retrieval, now standard in production search, uses a trained model to convert text into embeddings and ranks documents by embedding similarity to the query. Learned sparse retrieval, a newer approach that sits conceptually between the two, is gaining traction rapidly. Many people repeat claims about BM25, dense retrieval, and SPLADE, but far fewer have run all three on real benchmark data and verified that their results match published scores.
I did—on a MacBook with 16GB of memory—using a public set of exercises called the Foundations of Retrieval onboarding path, maintained by the Castorini information retrieval research group. The path leverages two open-source toolkits from Castorini: Anserini (Java-based) and Pyserini (its Python counterpart). It tests them on two real document collections and provides exact expected scores for each exercise.
Every number matched. But getting there required overcoming an out-of-memory crash, a native library crash, and a gated model access request that a valid API key alone didn't satisfy. None of this appears in the official documentation, so this article covers it as carefully as the numbers themselves. By the end, you'll know how to run all three methods locally, what can break on typical hardware, and how to interpret the scores.
The practical takeaway: if you're building a RAG system, a document search product, or an internal knowledge base, retrieval quality is the first thing to verify. Before changing prompts, swapping large language models, or adding agents, you need to know whether the right documents are being found at all. Reproducing these baselines gives you a reference point: what BM25 can do, what dense retrieval adds, what learned sparse retrieval costs, and how to check whether your own scoring pipeline behaves correctly.
What BM25, Dense Retrieval, and SPLADE Actually Do
All three methods solve the same problem: given a search query and a large document collection, return the most relevant documents, ranked best first. They differ only in how they convert text into something a computer can compare and rank.
BM25 reads a document and a query, counts shared words, and weighs each shared word by its rarity and frequency. It produces a sparse vector—a long list where almost every entry is zero, and the non-zero entries represent word weights. It needs no training data; weights are determined by a fixed formula tuned by two constants (k1 and b), rather than learned from examples.
Dense retrieval replaces that formula with a trained neural network. A document encoder reads the document and outputs a fixed-length embedding—a list of numbers meant to capture meaning rather than exact wording. A query encoder does the same for the query. The two embeddings are compared via a dot product (multiplying matching positions and summing the results), and the highest-scoring documents win. This requires a model trained on large amounts of labeled query-document pairs.
Learned sparse retrieval, the category SPLADE belongs to, is a middle path. Like BM25, it produces a sparse vector where most entries are zero. Unlike BM25, a trained model determines which words matter and by how much, rather than relying on a fixed formula. SPLADE version 3 is the specific model used in this article.
All three follow the same broad pattern: encode the query, encode the document, then compare the two. This is called a bi-encoder architecture. Jimmy Lin's paper on a representational framework for information retrieval lays this out formally; Section 4 of this article verifies it by hand.
