Measuring RAG End to End: Retrieval, Evidence, and Answer Quality

In Part 3 of this series, I explored advanced retrieval techniques such as small-to-large retrieval, sentence windows, summary-to-detail retrieval, and LLM reranking. These methods aim to improve which passages are retrieved, how they are ranked, and how much useful context they provide.

But improving the pipeline raises a new question:

How do we know whether the RAG system is actually getting better?

A single metric cannot answer that question. A retriever may find relevant passages but include contradictory evidence. An answer may be factually correct but unsupported by the retrieved context. A strict evidence filter may improve trustworthiness while rejecting too much useful information.

In the final notebook of this series, I evaluated the RAG pipeline across four layers:

1. retrieval and ranking quality,

2. evidence alignment,

3. Corrective RAG behavior,

4. generated-answer quality.

TL;DR

• RAG quality cannot be represented by one score.

• Hit Rate, Precision, Recall, and nDCG measure different aspects of retrieval.

• Retrieval must be tested across multiple query types because different methods fail differently.

• Evidence alignment measures whether retrieved passages support the trusted reference.

• Corrective RAG needs operational metrics such as trigger rate, trusted-candidate retention, and false accepts.

• LLM-based evaluation can measure correctness, faithfulness, completeness, and false-premise handling.

• LLM judges are useful, but they should complement labeled data and deterministic metrics.

• A practical RAG scorecard should keep retrieval, evidence, correction, and answer metrics separate.

1. Retrieval Metrics

Retrieval evaluation requires query-specific relevance labels. A passage may be factually aligned with the source but irrelevant to a particular query. Similarly, a contradictory passage can still be highly relevant because it directly discusses the requested topic.

I evaluated retrieval using several query types:

• a direct question,

• a paraphrased question,

• a particle-physics example,

• a technology example involving GPS,

• a question containing an incorrect premise.

This matters because retrieval behavior depends on how the query is written. Dense retrieval may handle paraphrasing well, while BM25 may perform better when exact scientific terminology appears in both the query and the source.

Hit Rate@k

Hit Rate asks whether at least one relevant passage appears in the top k results.

It is useful when finding any relevant evidence is enough to count the retrieval as successful. However, it does not tell us how many results were relevant or how highly the best passage was ranked.

Precision@k

Precision@k measures the proportion of the top k results that are relevant.

High precision means the returned context contains little noise. This is important when only a small number of chunks can be passed to the LLM.

Recall@k

Recall@k measures how much of the total known relevant evidence appears in the top k results.

High recall is useful when the answer requires several supporting passages. Increasing recall often requires retrieving more candidates, which may reduce precision.

nDCG@k

Normalized Discounted Cumulative Gain evaluates both relevance and ranking position.

Relevant passages near the top receive more credit than those appearing later. The score is normalized against the ideal ranking, making it useful for comparing retrieval methods across queries.

# Add concise retrieval-metric functions from Notebook 4 here

Comparing Retrievers Across Query Types

The experiments showed why averages alone can be misleading.

Dense retrieval performed particularly well on the paraphrased query, reaching a Precision@3 of 1.00 and an nDCG@5 of approximately 0.85. BM25 performed worse on the same query because the wording differed from the relevant passages.

For the particle and GPS examples, both retrieval styles performed strongly because the queries contained recognizable domain terminology. Hybrid retrieval improved some rankings, but it was not consistently best for every query.

Averaged across the five query types:

Add notebook visual: Retrieval metrics by query type and retriever.

The purpose of this comparison is not to declare one universal winner. It is to understand which query patterns expose each retriever’s strengths and weaknesses.

2. Evidence Alignment

Retrieval relevance does not guarantee factual alignment.

To measure evidence quality, I calculated Aligned@k:

Of the top k retrieved passages, what proportion agrees with the trusted reference?

def aligned_at_k(results, k=5):
    return (
        results.head(k)["label"]
        .eq("aligned")
        .mean()
    )

This metric revealed a different picture from standard retrieval metrics.

For example, a retriever could achieve a perfect Hit Rate because it found a passage relevant to the query, while most of its top results still contradicted the trusted explanation.

This reinforces a central lesson from the series:

Retrieval quality and factual alignment are separate dimensions.

Precision@k answers, “Did I retrieve passages relevant to the query?”

Aligned@k answers, “How much of the retrieved evidence is factually aligned?”

Both are needed.

Add notebook visual: Aligned@3, Aligned@5, and Aligned@10 across retrievers.

3. Corrective RAG Operational Metrics

Corrective RAG grades the retrieved candidates, retains trusted evidence, and triggers a fallback when too little usable context remains.

Evaluating it requires more than measuring the final answer.

Correction trigger rate

This measures how often the corrective path is activated.

A very high trigger rate may indicate that retrieval is weak or that the verification threshold is too strict. A very low rate may mean the filter is permissive—or that the test set is not challenging enough.

Average trusted candidates retained

This measures how many candidates survive evidence grading and confidence filtering.

Too few retained candidates can reduce answer coverage. Too many may indicate that the verification layer is not filtering aggressively enough.

False accept rate

A false accept occurs when contradictory or unreliable evidence is incorrectly approved.

This is one of the most dangerous errors because it allows misinformation to pass into answer generation.

In my experiment, the pipeline retained an average of 3.2 trusted candidates per query. No correction was triggered, and no false accepts were recorded. The final evidence sets reached an average Aligned@3 of 1.0.

corrective_summary = {
    "correction_trigger_rate": ...,
    "average_trusted_candidates": ...,
    "total_false_accepts": ...,
    "mean_final_aligned_at_3": ...,
}

These results are encouraging, but they also show why a broader test set is needed. A correction trigger rate of zero may reflect strong retrieval—or an evaluation set that does not create enough failure cases.

4. LLM-Based Answer Evaluation

The final stage evaluates the generated answer itself.

I used an LLM judge to score four dimensions from 1 to 5.

Correctness

Does the answer agree with the trusted reference?

Correctness evaluates factual quality independently of whether the answer followed the retrieved evidence.

Faithfulness

Are the answer’s claims supported by the evidence supplied to the generator?

An answer can be correct but unfaithful if it introduces outside knowledge. It can also be faithful but incorrect if it accurately repeats bad evidence.

Completeness

Does the response sufficiently answer the question?

A concise answer may be accurate and grounded but still omit an important part of the explanation.

False-premise handling

Does the answer recognize and correct an incorrect assumption in the question?

This is especially important for queries such as, “Why do fast-moving clocks run faster?” A strong answer should correct the premise rather than answer it as written.

# Add the concise evaluate_answer() prompt and JSON output here

The notebook produced the following pipeline-level averages:

Corrective RAG improved faithfulness from 4.0 to 5.0, while completeness and false-premise handling remained unchanged. The slight correctness difference also shows why LLM-judge scores should not be overinterpreted from a small experiment.

Add notebook visual: Baseline versus Corrective RAG answer-quality scores.

Building an End-to-End Scorecard

The final scorecard combines the most useful metrics without collapsing them into one number:

Retrieval relevance
→ ranking quality
→ evidence alignment
→ correction behavior
→ answer faithfulness and correctness

This preserves visibility into where the system succeeds or fails.

A low answer score may originate from poor retrieval, contradictory evidence, overly strict filtering, or generation behavior. One overall score would hide those distinctions.

What the Experiments Showed

Seven lessons stood out:

1. Retrieval relevance and factual alignment must be evaluated separately.

2. Retrieval metrics require query-specific relevance labels.

3. Different query types are necessary to expose retriever trade-offs.

4. Evidence-grader false accepts and false rejects affect downstream behavior differently.

5. Corrective RAG should be evaluated as an operational system, not only by its final answers.

6. Correctness and faithfulness are separate answer properties.

7. LLM judges are useful but should be supported by manually reviewed samples and deterministic checks.

Most importantly:

No single metric represents RAG quality.

A practical evaluation framework must retain separate measures for retrieval, evidence, correction, and generation.

Closing the Series

This concludes my four-part RAG From First Principles series.

Across the series, I explored:

1. document loading and chunking,

2. dense, sparse, and hybrid retrieval,

3. advanced retrieval and context optimization,

4. end-to-end RAG evaluation.

The next step is to move beyond isolated techniques and notebook experiments.

In my next series, I will focus on building and operating a real RAG application: how these components are connected, deployed, monitored, evaluated, and improved in a production-style system.

The implementation for this series is available in the RAG From First Principles GitHub repository.

References and Further Reading

1. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.

2. Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering.

3. Järvelin, K. and Kekäläinen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques.

4. Huang, J. RAG from First Principles: Mastering Retrieval-Augmented Generation.

5. Es, S. et al. (2023). RAGAS: Automated Evaluation of Retrieval-Augmented Generation.

6. Saad-Falcon, J. et al. (2024). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems.

7. OpenAI evaluation best practices.

8. Project code and notebooks.