I write technical articles that analyze real world engineering problems through the lens of production systems.
My work focuses on architectural tradeoffs, UI layer system design, performance optimization, and reliability patterns derived from shipping and maintaining scalable applications.
Building a RAG system is relatively easy to demonstrate.
Building one where you can measure why it fails, reproduce the failure, and identify which stage caused it is a very different engineering problem.
In the first part of my Knowledge Vault journey, I focused on building and benchmarking the retrieval pipeline. I compared dense retrieval against BGE reranking and found something interesting: the reranker changed 27 out of 35 Top-1 results — 77.1% of them.
If you haven't read Part 1, start with Building Knowledge Vault: From RAG Implementation to Retrieval Engineering. That article covers the initial retrieval architecture and benchmark that led to this investigation.
That number looked significant.
But there was a problem.
A ranking change is not automatically an improvement.
The reranker could be correcting bad retrieval, or it could simply be replacing one passage with another passage that looks more relevant to the model but contains worse evidence.
At the end of that investigation, I had a hypothesis.
Document structure might be part of the problem.
Table-of-contents pages, indexes, glossaries, copyright pages, and similar structural content can be highly similar to a query while being poor sources of answer-bearing evidence.
But a hypothesis is not an engineering conclusion.
I needed to prove it.
That changed the direction of M1 completely.
Instead of asking whether the RAG system produced good answers, I started asking a much more useful question:
When the correct evidence exists, where does the system lose it?
And answering that question required building the measurement infrastructure first.
A RAG system has several independent failure boundaries.
A question can fail because the document parser extracted the content incorrectly.
It can fail because chunking split the relevant passage badly.
It can fail because dense retrieval never found the correct passage.
It can fail because the correct passage was retrieved but ranked too low.
Or the retrieval pipeline can work perfectly and the generation model can still produce a poor answer.
If I only evaluate the final response, all of those failures collapse into one number.
That makes debugging extremely difficult.
So I changed the evaluation boundary.
Instead of treating the generated answer as the only source of truth, I created passage-level gold evidence for the benchmark.
Knowledge Vault's evaluation set contains 35 controlled questions: 30 answerable questions used for strict retrieval metrics and 5 deliberately unanswerable questions.
For the answerable questions, the evaluator knows which passage contains the expected evidence.
That allows the pipeline to answer questions that are much more useful than "Was the final answer correct?"
For example:
Did dense retrieval find the correct evidence?
And if it did:
Did the reranker preserve that evidence?
Those are completely different failure modes.
The benchmark therefore tracks candidate Recall@k, Recall@1, Recall@5, Recall@20, MRR, nDCG@5, per-question evidence ranks, dense and reranker scores, and stage-level latency.
The important change wasn't the number of metrics.
It was the visibility into the pipeline.
I could finally observe where evidence entered the system and where it disappeared.
The dense retrieval baseline gave me a useful starting point.
Across the benchmark, dense retrieval achieved:
| Metric | Dense Baseline |
|---|---|
| Questions | 35 |
| Strict-metric questions | 30 |
| Recall@1 | 36.67% |
| Recall@5 | 66.67% |
| MRR | 0.4944 |
| nDCG@5 | 0.4274 |
These numbers weren't particularly interesting by themselves.
What mattered was what happened when I changed the candidate pool.
The retrieval pipeline exposes a parameter called candidate_k.
This controls how many passages dense retrieval sends to the reranker.
The final top_k controls how many passages eventually become evidence for the model.
That distinction sounds small.
It turned out to be one of the most important details in the investigation.
I ran a controlled comparison across dense retrieval and reranking:
| Configuration | Candidate k | Reranking | Recall@1 | Recall@5 | MRR | nDCG@5 |
|---|---|---|---|---|---|---|
| Dense | 5 | Off | 36.7% | 66.7% | 0.4944 | 0.4274 |
| Dense | 20 | Off | 36.7% | 66.7% | 0.4944 | 0.4274 |
| Rerank | 5 | On | 40.0% | 66.7% | 0.5067 | 0.4293 |
| Rerank | 20 | On | 20.0% | 66.7% | 0.3911 | 0.3479 |
Something immediately stood out.
With candidate_k=20, dense retrieval was able to find the gold evidence in the candidate pool at 83.3% Recall@20.
That sounds promising.
But after reranking, the final ranking quality actually got worse.
Recall@1 dropped from 40.0% to 20.0% when the reranker received 20 candidates instead of 5.
MRR and nDCG@5 dropped as well.
This changed the debugging question.
I no longer needed to ask:
"Can retrieval find the passage?"
I needed to ask:
"When retrieval already found the passage, what is causing the ranking stage to remove it?"
That is a much narrower problem.
And narrower problems are much easier to investigate.
The next step wasn't to change the reranker.
That would have made the experiment harder to interpret.
Instead, I instrumented the transition between candidate retrieval and final ranking.
For every evaluated question, I started preserving information such as:
candidate rank
final rank
dense score
reranker score
structural category
gold-evidence presence
whether the gold passage survived into the final top-k
This gave me something I didn't have before.
A forensic view of the ranking pipeline.
I could take a question where the final answer was wrong and trace the evidence backward:
Question
↓
Dense candidates
↓
Was gold evidence present?
↓
What rank did it have?
↓
What did the reranker score?
↓
What replaced it?
↓
Why did that replacement look attractive?
That changed the debugging process from speculation to evidence.
And a pattern started appearing.
Some of the passages being promoted by the reranker were genuinely related to the query.
They just weren't the passages that actually answered it.
This distinction is easy to miss.
Imagine a question asking:
What does the author mean by the fourth degree of action?
A table-of-contents passage might contain:
The Fourth Degree: Massive Action
A chapter passage might contain several paragraphs explaining what that concept actually means.
Both passages are semantically related to the query.
Only one is useful as evidence.
The first provides navigation information.
The second provides answer-bearing evidence.
A semantic retrieval system doesn't automatically know that difference.
And this is where document structure became interesting.
The book contained structural chunks such as:
CONTENTS
INDEX
COPYRIGHT
GLOSSARY
These chunks aren't bad data.
In fact, they can be extremely useful.
If someone asks:
Where does the book discuss massive action?
an index entry may be exactly what we want.
But if someone asks:
What does massive action mean according to the author?
the same index entry is insufficient.
The problem wasn't simply "structural chunks are bad."
The problem was more subtle:
Different document structures provide different kinds of relevance.
A table of contents is relevant for navigation.
An index is relevant for locating concepts.
A glossary is relevant for terminology.
A substantive passage is relevant for evidence.
Treating all of them as interchangeable evidence creates a ranking problem.
While investigating individual retrieval failures, I discovered something else that was important enough to stop the experiment and audit the benchmark.
An earlier evaluation artifact contained a stale gold-evidence identifier.
The underlying question/evidence mapping had subsequently been corrected in the evaluation data, but not every previously generated result artifact had been regenerated from the updated annotations.
That distinction matters.
A benchmark result isn't just a number.
It is the output of:
Question set
+
Evidence specification
+
Retriever configuration
+
Evaluation code
↓
Result artifact
Change one of those inputs and the result has a different provenance.
This led to a broader engineering lesson:
Evaluation artifacts need provenance.
If the benchmark changes, I need to know exactly which question set and evidence specification produced a particular result.
Otherwise, two JSON files can appear comparable while actually representing different versions of the experiment.
I also added an important validation requirement: evidence identifiers should not merely be syntactically valid; they should resolve to actual indexed passages.
The evaluator itself needs guardrails.
Otherwise, the system can end up confidently measuring an invalid experiment.
One of the clearest examples was Q022:
What is the relationship between 10X goals and 10X actions according to the author?
The important part wasn't that the answer failed.
It was where it failed.
The correct evidence was already inside the dense candidate pool.
Dense retrieval had done its job.
But after reranking, that evidence was displaced by a glossary chunk.
That meant the failure wasn't:
Dense retrieval
↓
✗ Could not find evidence
It was:
Dense retrieval
↓
✓ Found evidence
↓
Reranking
↓
✗ Discarded evidence
That distinction changes what you should fix.
Increasing the candidate pool won't solve a failure that happens after candidate retrieval.
Changing the embedding model isn't necessarily the right answer either.
The evidence was already there.
The failure boundary had moved downstream.
Across the forensic analysis, I found:
40 cases where gold evidence was present in candidate-k=20
24 cases where it survived into final top-5
16 cases where gold evidence was lost after reranking
For 12 of those losses, I could compare the reranker score of the gold passage against the replacement passage.
The results were surprisingly large:
| Measurement | Value |
|---|---|
| Mean gold reranker score | 0.1827 |
| Mean replacement score | 0.7915 |
| Mean score margin | 0.6088 |
| Median score margin | 0.6713 |
This was important.
The reranker wasn't always making a tiny mistake where two passages had almost identical scores.
In these cases, the replacement often received a substantially stronger score.
That made the problem more interesting.
The question became:
Why does a semantically plausible but less evidentiary passage look so much better to the reranker?
And that is a much harder question than "the reranker ranked the wrong thing."
At this point I had enough evidence to formulate a controlled hypothesis:
Structural candidates can interfere with reranking when they are treated as ordinary substantive evidence.
Notice what I wasn't claiming.
I wasn't claiming that structural chunks caused every reranking failure.
I wasn't claiming that the BGE reranker was fundamentally bad.
I wasn't claiming that filtering would solve RAG retrieval.
I was testing one specific contributor to the observed failure.
That distinction matters in engineering experiments.
If the hypothesis is too broad, almost any result can appear to support it.
A narrow hypothesis gives you something falsifiable.
I didn't remove structural chunks from the vector database.
That would have changed the corpus and potentially affected other query types.
Instead, I added an eligibility filter between dense retrieval and reranking.
The pipeline became:
Dense Retrieval
↓
Structural Eligibility Filter
↓
BGE Reranker
↓
Final Evidence
The filter excluded navigation-oriented structural categories such as:
CONTENTS
INDEX
COPYRIGHT
GLOSSARY
from the evidence-ranking path.
Everything else stayed the same.
Same corpus.
Same embeddings.
Same dense retrieval.
Same candidate depth.
Same reranker.
Same final top-k.
Only the eligibility decision changed.
That made the experiment much easier to reason about.
The controlled before/after experiment produced:
| Metric | Baseline | Structural Filter |
|---|---|---|
| Candidate Recall@20 | 83.3% | 83.3% |
| Recall@1 | 20.0% | 20.0% |
| Recall@5 | 66.7% | 70.0% |
| MRR | 0.3911 | 0.4022 |
| nDCG@5 | 0.3561 | 0.3722 |
The most important thing about these results is what didn't change.
Candidate Recall@20 remained exactly the same.
That is expected.
The filter didn't improve dense retrieval.
It acted downstream of dense retrieval.
Recall@1 also remained at 20%.
So the intervention didn't suddenly make the reranker capable of consistently selecting the best passage.
What changed was the quality of the final top-5.
Recall@5 improved:
66.7% → 70.0%
MRR improved:
0.3911 → 0.4022
nDCG@5 improved:
0.3561 → 0.3722
That is exactly the kind of result I wanted from the experiment.
Not a dramatic "we solved RAG" number.
A measurable improvement at the stage where the hypothesis predicted an improvement.
Q022 was one example.
Before filtering, the correct evidence was inside the candidate pool but was displaced by a glossary chunk.
After filtering, the substantive evidence survived into the final context.
Another useful example was Q004:
What does the fourth degree of action mean?
The gold passage was already present in the candidate pool.
Before filtering, it appeared at rank 5.
After filtering, it moved to rank 4.
The corresponding metrics improved:
MRR
0.20 → 0.25
nDCG@5
0.3869 → 0.4307
Again, this wasn't a retrieval breakthrough.
No new evidence was discovered.
The system simply ranked existing evidence slightly better.
And that is exactly why passage-level metrics are useful.
They tell you what kind of improvement actually occurred.
One of my early assumptions was straightforward:
More candidates should give the reranker more opportunities to find the right answer.
That's true.
But it is only half the equation.
More candidates also give the reranker more opportunities to encounter distractors.
A larger candidate pool provides:
More recall opportunity
+
More candidate diversity
+
More potential evidence
-
More candidate noise
So increasing candidate_k is not automatically an optimization.
It changes the ranking problem.
In this benchmark, candidate Recall@20 reached 83.3%, but final Recall@5 was still only 66.7% before structural filtering.
The system had the evidence.
The challenge was selecting it.
That is a very different engineering problem from simply "improve retrieval."
The structural filter improved Recall@5 by 3.3 percentage points.
That's useful.
But the more important improvement was observability.
Before instrumenting the pipeline, a failure looked like:
"The answer is wrong."
After instrumentation, I could say:
"The gold passage entered the top-20 dense candidate set but was displaced before the final top-5."
That is an entirely different debugging state.
The debugging loop became:
Was gold evidence retrieved?
│
├── No
│ ↓
│ Investigate dense retrieval
│
└── Yes
↓
What displaced it?
↓
What type of passage was it?
↓
What were the scores?
↓
What intervention could address it?
↓
Re-run the benchmark
This is what good evaluation infrastructure gives you.
Not just metrics.
A mechanism for turning an aggregate failure into a testable hypothesis.
At the beginning of this investigation, I was looking for a fix.
By the end, I was more interested in the remaining failure.
That is usually a good sign in systems engineering.
After structural filtering, the benchmark still had:
Candidate Recall@20 83.3%
Final Recall@5 70.0%
Recall@1 20.0%
The system can frequently retrieve the right evidence.
But it still struggles to put the best evidence at the top.
And structural filtering only explains part of that gap.
The remaining failures are heterogeneous.
Some are likely related to semantic similarity.
Some may be caused by the limitations of the reranking signal.
Some may be related to chunking.
Some may require complementary lexical retrieval.
That means the next investigation shouldn't simply be another isolated tweak to the reranker.
I need another retrieval signal.
This experiment does not establish that BGE is a poor reranker.
It does not establish that structural chunks cause all reranking failures.
It does not establish that the improvement will generalize to every document.
It does not establish that Recall@5 is now sufficient.
And it certainly doesn't mean the RAG system is production-grade.
The current evaluation is based on one 214-page text-layer PDF.
There are still major areas that require separate investigation: OCR and scanned documents, complex tables, multi-column layouts, reading-order corruption, larger and more diverse corpora, and broader evaluation sets.
The benchmark itself also contains related questions, so the raw count of questions should not automatically be interpreted as the same number of independent failure trials.
These limitations aren't footnotes to hide.
They define the boundary of what the experiment actually proves.
The most valuable outcome of M1 wasn't one metric.
It was the decomposition of the retrieval problem.
Instead of treating RAG as:
Question
↓
Magic retrieval
↓
Answer
I now have observable boundaries:
Document extraction
↓
Chunking
↓
Candidate retrieval
↓
Candidate eligibility
↓
Reranking
↓
Final evidence selection
↓
Generation
And each boundary can be evaluated independently.
That changes how I think about RAG engineering.
A low answer accuracy number is not a diagnosis.
It is a symptom.
The engineering work begins when you can determine which stage produced the symptom.
M1 started with a relatively simple question:
Can Knowledge Vault retrieve useful evidence from a PDF?
It ended with a much better one:
When the correct evidence exists, what causes the system to lose it before generation?
Dense retrieval at candidate-k=20 achieved 83.3% candidate Recall@20.
Before the structural intervention, final Recall@5 was 66.7%.
Forensic analysis showed that some of the missing evidence was displaced by structurally related but less useful passages.
A controlled structural eligibility filter improved Recall@5 from 66.7% to 70.0%, MRR from 0.3911 to 0.4022, and nDCG@5 from 0.3561 to 0.3722.
But Recall@1 remained at 20%.
That last number is important.
It tells me the problem isn't solved.
The structural filter improved one part of the ranking behavior, but it didn't close the fundamental gap between:
Evidence retrieved
↓
Evidence selected
And that's a much more useful conclusion than saying:
"Our RAG pipeline improved by 3.3%."
The real result is that the failure boundary is now measurable.
The implementation, evaluation harness, benchmark artifacts, and retrieval experiments are available in the Knowledge Vault GitHub repository
The next question is now much clearer:
If dense retrieval can find the correct evidence 83.3% of the time at k=20, why does final evidence ranking remain substantially worse?
The next experiment will investigate whether complementary retrieval signals can close some of that gap.
Specifically:
Query
│
┌───────┴───────┐
↓ ↓
Dense BM25
│ │
└───────┬───────┘
↓
RRF
↓
Reranking
↓
Evidence
The goal isn't to assume that hybrid retrieval will solve the problem.
The goal is to measure whether lexical and semantic retrieval provide complementary evidence coverage, and whether that improves candidate recall and final ranking independently.
That's where M2 begins.
The interesting part of building a RAG system isn't getting the first demo to work.
It's getting to the point where a bad answer stops being mysterious.
A production-minded retrieval system should let you ask:
Did we fail to find the evidence?
Did we find it but rank it incorrectly?
Did document structure confuse the ranking model?
Did the benchmark itself accurately describe the expected evidence?
Did the intervention actually improve the stage we intended to change?
That's the difference between building a RAG pipeline and engineering a retrieval system.
And for me, that was the real outcome of Knowledge Vault M1.
I didn't solve retrieval.
I made its failures observable enough to keep solving them.
If you're reading this for the first time, some of the terminology can feel unnecessarily dense. Here are the terms that matter most in this investigation.
| Keyword | Meaning |
|---|---|
| RAG | Retrieval-Augmented Generation. A system retrieves relevant information first and then gives that information to an LLM to generate an answer. |
| Dense Retrieval | Retrieval based on semantic embeddings. The query and document chunks are represented as vectors, and similar vectors are retrieved. |
| BM25 | A traditional lexical search algorithm that scores documents based largely on how well their words match the query. |
| Embedding | A numerical vector representation of text that captures semantic relationships between pieces of text. |
| Vector Database | A database optimized for storing and searching vector embeddings. Knowledge Vault uses Qdrant. |
| Candidate Pool | The set of passages retrieved before the final ranking stage. With candidate_k=20, the reranker receives up to 20 candidates. |
| candidate_k | The number of candidates retrieved for downstream processing. Increasing it can improve recall but can also introduce more distractors. |
| top_k | The number of passages ultimately selected as final context for the LLM. |
| Reranker | A model that takes an initial set of retrieved passages and reorders them based on query-passage relevance. |
| Cross-Encoder | A model that processes the query and passage together to produce a relevance score. BGE reranking uses this approach. |
| Gold Evidence | The passage identified by the benchmark as containing the evidence required to answer a question. |
| Candidate Recall@20 | The percentage of questions where the gold evidence appears somewhere within the first 20 retrieved candidates. |
| Recall@1 | The percentage of questions where the correct evidence is ranked first. |
| Recall@5 | The percentage of questions where the correct evidence appears within the top 5 results. |
| MRR | Mean Reciprocal Rank. Measures how high the first relevant result appears. Rank 1 contributes 1.0, rank 2 contributes 0.5, rank 3 contributes 0.33, and so on. Higher is better. |
| nDCG@5 | Normalized Discounted Cumulative Gain. Measures ranking quality while giving more importance to relevant results appearing near the top. Higher is better. |
| Ranking Failure | A case where the correct evidence was retrieved but ranked too low or displaced by another passage. |
| Retrieval Failure | A case where the correct evidence never entered the candidate pool in the first place. |
| Structural Chunk | A chunk representing document organization or navigation rather than substantive content, such as a TOC, index, glossary, or copyright page. |
| Structural Filtering | Preventing certain structural chunks from participating in the evidence-ranking stage while keeping them available in the underlying corpus. |
| Semantic Relevance | How conceptually related a passage is to a query. A passage can be semantically relevant without containing the answer. |
| Evidentiary Relevance | How useful a passage actually is as evidence for answering the question. This is the distinction that became central to the investigation. |
| RRF | Reciprocal Rank Fusion. A technique for combining rankings from different retrieval systems, such as dense retrieval and BM25. |
| Hybrid Retrieval | Combining multiple retrieval strategies, typically semantic and lexical retrieval, to improve coverage. |
| Latency | The time taken by a stage or the entire retrieval pipeline to produce a result. |
| Provenance | Information describing exactly which data, benchmark version, configuration, and evaluation specification produced a result. |
| Failure Boundary | The specific stage where a previously correct piece of information becomes lost, corrupted, or incorrectly ranked. |