Comparing semantic search, BM25, Reciprocal Rank Fusion, evidence grading, and Corrective RAG
In the first article of this series, I explored how raw documents become retrievable knowledge. I loaded different file formats, preserved their metadata, and divided their contents into chunks.
Once the chunks are ready, the next question is:
Given a user query, which chunks should the RAG system retrieve?
This is an information-retrieval problem, and it has a lot in common with traditional search. Concepts such as lexical matching, similarity scores, ranking, precision, and recall therefore feel familiar.
The experiments also revealed an important limitation: retrieval can find relevant chunks, but it does not determine whether those chunks are factually correct.
Retrieval ranks chunks according to their relevance to a query.
A highly relevant chunk can still contain inaccurate information. Relevance and correctness must therefore be evaluated separately.
Dense retrieval uses embeddings to capture semantic similarity.
Sparse retrieval methods such as BM25 emphasize matching words and phrases.
Hybrid retrieval combines dense and sparse rankings.
Dense retrieval works well for paraphrased questions, while BM25 remains useful when exact terminology matters.
Reciprocal Rank Fusion combines rankings without requiring their raw scores to use the same scale.
Corrective RAG adds another layer of evidence grading, confidence filtering, and corrective action when the retrieved evidence is insufficient.
After ingestion and chunking, a RAG knowledge base can be viewed as a collection of searchable passages.
When a user submits a query, the retriever:
compares the query with the stored chunks,
calculates a relevance signal,
ranks the chunks,
returns the top candidates.
The selected chunks are then passed to later stages of the RAG pipeline.
A retriever is not usually trying to prove that a passage is correct. Its immediate task is narrower: identify passages that appear most relevant to the query.
There are several ways to measure that relevance.
Dense retrieval represents both the query and every chunk as embeddings.
An embedding is a numerical vector intended to represent the meaning of a piece of text. Texts with similar meanings should ideally be located close to one another in the embedding space.
In the notebook, I generated embeddings using:
from sentence_transformers import SentenceTransformer
embedding_model = SentenceTransformer(
"BAAI/bge-small-en-v1.5"
)
chunk_embeddings = embedding_model.encode(
chunks,
normalize_embeddings=True,
)
The query is embedded using the same model:
query_embedding = embedding_model.encode(
[query],
normalize_embeddings=True,
)
I then used cosine similarity to compare the query vector with the chunk vectors:
scores = cosine_similarity(
query_embedding,
chunk_embeddings,
)[0]
Cosine similarity measures how closely two vectors point in the same direction. The chunks with the highest scores are returned first.
Dense retrieval is especially useful when the query and the relevant chunk express the same idea using different words.
For example, a user may ask:
Why does time pass differently for a moving object?
The relevant source might instead use terms such as time dilation, relative motion, or moving clocks without repeating the exact wording of the query.
Because dense retrieval compares semantic representations rather than only words, it can still identify the connection.
Embedding similarity is approximate. A dense retriever may return passages that are broadly related to the topic but do not answer the specific question.
It may also miss rare names, exact quotations, technical identifiers, or specialized terminology that lexical retrieval can identify more directly.
Sparse retrieval follows a more traditional search approach.
Instead of representing every passage as a dense semantic vector, BM25 scores chunks using the words they contain. It considers factors such as:
whether a query term appears in the chunk,
how frequently it appears,
how rare the term is across the corpus,
the length of the chunk.
A simplified implementation from the notebook looks like this:
from rank_bm25 import BM25Okapi
tokenized_chunks = [
chunk.lower().split()
for chunk in chunks
]
bm25 = BM25Okapi(tokenized_chunks)
query_tokens = query.lower().split()
bm25_scores = bm25.get_scores(query_tokens)
BM25 is effective when exact words matter.
For queries containing terms such as Lorentz transformation, Michelson–Morley experiment, or special relativity, lexical overlap may be a strong relevance signal.
It is also relatively transparent: a chunk ranks well largely because it contains important query terms.
BM25 does not naturally understand paraphrases.
If a query and a relevant passage use different vocabulary, lexical overlap may be low even when their meanings are closely related.
This creates a useful contrast:
Dense retrieval emphasizes meaning.
Sparse retrieval emphasizes words.
Neither is universally better.
Hybrid retrieval combines dense and sparse search.
The goal is to benefit from:
the semantic understanding of embeddings,
the lexical precision of BM25.
However, directly averaging their scores can be problematic. Cosine-similarity and BM25 scores use different scales and distributions.
Instead, I combined their ranked lists using Reciprocal Rank Fusion.
def reciprocal_rank_fusion(
ranked_lists,
k=60,
):
fused_scores = {}
for ranked_list in ranked_lists:
for rank, document_id in enumerate(
ranked_list,
start=1,
):
fused_scores[document_id] = (
fused_scores.get(document_id, 0)
+ 1 / (k + rank)
)
return sorted(
fused_scores.items(),
key=lambda item: item[1],
reverse=True,
)
Reciprocal Rank Fusion rewards chunks that appear near the top of one or more rankings. It works with rank positions rather than requiring the original scores to be directly comparable.
Here is one output from the notebook:
Query type: direct
Query: Why does a moving clock run slower?
Rank 1
Hybrid RRF score: 0.0325
Dense rank: 2
BM25 rank: 1
Blob:
A moving clock can run slow relative to another clock only if it is
physically observed through a telescope. Unobserved clocks are unaffected.
Hybrid retrieval can be more robust across different query types. A semantically phrased query may benefit from dense search, while an exact technical query may benefit from BM25.
However, combining two methods does not guarantee that every result will improve. Hybrid retrieval still depends on the quality of the component rankings and the fusion strategy.
The example above also exposes a more fundamental issue: the chunk ranked highly by both retrievers, but its claim is inaccurate.
This was the most important learning from the experiment.
A retriever does not normally evaluate whether a chunk is factually accurate. It evaluates whether the chunk matches the query according to its retrieval signal.
Consider a query about time dilation and these two chunks:
Moving clocks are measured to run more slowly.
Moving clocks are measured to run more quickly.
Both passages contain nearly identical terminology. Both may receive strong BM25 scores, and their embeddings may also be close to the query.
One is aligned with the trusted explanation. The other contradicts it.
From the retriever’s perspective, however, both are relevant.
This was also visible in the distribution of cosine-similarity scores for factually aligned and contradictory chunks. The same issue appeared in BM25 scores: relevance signals alone did not reliably separate correct evidence from incorrect evidence.
This leads to a critical distinction:
Relevance tells us whether a passage is about the query. It does not tell us whether the passage is correct.
The pipeline therefore needs additional controls when the source corpus may contain unreliable, conflicting, or uncertain information.
Dense, sparse, and hybrid retrieval all try to find chunks that match the user’s query. However, a strong retrieval score does not guarantee that the evidence is accurate or safe to use.
A retrieved chunk may be:
relevant and factually aligned,
relevant but contradictory,
related to the topic but too uncertain to support an answer.
This is where evidence grading becomes useful.
In the notebook, I added an LLM-based grader that evaluates each retrieved candidate against the query and a trusted reference. The grader assigns one of three labels:
aligned — the evidence supports the trusted explanation,
contradicts — the evidence conflicts with it,
uncertain — the evidence is related but insufficient for a confident decision.
The grader also returns a confidence score and a short rationale.
Here are two examples from the notebook:
Gold label: aligned
Predicted: aligned
Confidence: 1.0
Rationale:
The candidate's claim about a moving clock ticking more slowly becoming
substantial only near light speed directly matches the reference explanation
of special-relativity time dilation.
Text:
A clock moving rapidly relative to an observer is measured to tick more
slowly than a clock at rest with that observer. This effect becomes
substantial only when the relative speed is a significant fraction of
the speed of light.
Gold label: contradicts
Predicted: contradicts
Confidence: 1.0
Rationale:
The reference explicitly states that all observers do not share one
universal time and that time dilation causes moving clocks to record
different elapsed times.
Text:
All clocks in the universe share one absolute time regardless of their
motion. A fast-moving spacecraft and Earth must therefore record
identical elapsed times.
A simplified filtering step looks like this:
verified_chunks = graded_chunks[
(graded_chunks["predicted_label"] == "aligned")
& (
graded_chunks["verification_confidence"]
>= confidence_threshold
)
]
Only high-confidence, aligned chunks are allowed to continue through the pipeline. Contradictory and uncertain candidates are removed.
This creates a second layer of selection:
Initial retrieval
→ evidence grading
→ confidence filtering
→ verified chunks
The trusted reference provides an accuracy signal in this experiment. In a production system, equivalent checks may instead come from trusted-source metadata, domain rules, verified databases, human labels, or other validation mechanisms.
Filtering can leave the system with too little usable evidence.
For example, if the retriever returns five candidates but only one passes the verification threshold, the system may not have enough context to answer reliably.
Corrective RAG adds a recovery step for this situation:
Retrieve candidates
→ grade evidence
→ retain verified chunks
→ check whether enough evidence remains
→ trigger corrective retrieval or fallback when needed
The corrective action could involve:
retrieving more candidates,
rewriting the query,
searching a trusted source,
expanding the retrieval scope,
using a verified-reference fallback.
In this implementation, correction is triggered when the number of trusted candidates falls below a minimum threshold.
This distinction is important: filtering weak chunks is useful, but filtering alone is not a complete corrective pipeline. The system becomes corrective when it takes another action after determining that the original retrieval results are insufficient.
The notebook reinforced four main lessons.
First, dense and sparse retrieval solve different problems. Dense search is stronger when meaning is shared but wording differs. BM25 is stronger when exact terminology matters.
Second, hybrid retrieval can improve robustness by combining both signals, especially when user queries vary in style.
Third, retrieval scores should not be interpreted as truth or confidence. A high score indicates a strong match under that retrieval method—not factual correctness.
Finally, retrieval quality should be evaluated using a suite of queries and relevance labels rather than only a few manually inspected examples. Evidence quality must then be assessed separately using labels, verification rules, or an evidence grader.
In the next article, I will go beyond independent chunk retrieval and explore advanced techniques such as sliding-window, parent-child, summary-to-detail, and hierarchical retrieval.
The full implementation is available in the project repository on GitHub.
Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering.
Huang, J. RAG from First Principles: Mastering Retrieval-Augmented Generation.