In Part 2 of this series, I explored the basic retrieval mechanisms used in RAG: dense retrieval, BM25, and hybrid search. I also examined evidence grading and Corrective RAG as ways to prevent factually contradictory chunks from moving further through the pipeline.
This article focuses on a different problem:
Once a retriever finds relevant chunks, how do we ensure they contain enough context to be genuinely useful?
A small chunk may match the query precisely but omit the surrounding explanation. A large chunk may contain the complete answer but introduce unrelated information that weakens retrieval precision.
In this article, I explore four advanced strategies designed to improve retrieval relevance and context quality:
small-to-large retrieval,
sentence-window retrieval,
summary-to-detail retrieval,
LLM reranking.
The goal is not to determine whether a retrieved claim is factually true. Instead, it is to improve which passages are retrieved, how they are ordered, and how much context is ultimately provided.
TL;DR
Fixed-size chunks force one representation to support both precise retrieval and complete context.
Small-to-large retrieval searches small child chunks and returns their larger parent sections.
Sentence-window retrieval matches individual sentences and expands them with nearby sentences.
Summary-to-detail retrieval uses compact summaries to route queries toward relevant sections before searching their detailed chunks.
LLM reranking reorders an initial candidate set based on how directly and completely each candidate answers the query.
These techniques improve retrieval precision and context quality, but they do not verify factual correctness.
No single strategy is best for every query. Each introduces trade-offs in context size, noise, cost, latency, and complexity.
The Limitation of Fixed-Size Retrieval
A basic dense retriever usually splits every document into chunks of one fixed size, embeds each chunk independently, and returns the chunks most similar to the query.
This is a useful baseline, but the selected chunk size must perform two conflicting jobs:
remain small enough for precise matching,
remain large enough to provide complete context.
In my baseline, documents were divided into chunks of approximately 250 words with some overlap.
This worked reasonably well, but the retrieved chunks sometimes began or ended in the middle of an explanation. Increasing the chunk size would provide more context, but it could also dilute the embedding with unrelated information.
The following strategies try to separate the unit used for search from the unit returned as context.
1. Small-to-Large Retrieval
Small-to-large retrieval creates two linked levels of chunks:
small child chunks used for precise matching,
larger parent chunks returned after a child is selected.
In my implementation, parent sections contained approximately 900 words, while their child chunks contained around 150 words.
The system embeds and searches only the child chunks. When a child matches the query, the system maps it back to its parent and returns the larger context.
Query
→ search small child chunks
→ identify matched children
→ map children to parents
→ return larger parent context
def small_to_large_retrieve(
query,
child_top_k=10,
parent_top_k=3,
):
child_results = retrieve_dense(
query,
child_df,
top_k=child_top_k,
)
best_parent_scores = (
child_results
.groupby(
"parent_id",
as_index=False,
)["retrieval_score"]
.max()
.sort_values(
"retrieval_score",
ascending=False,
)
.head(parent_top_k)
)
expanded = best_parent_scores.merge(
parent_df,
on="parent_id",
how="left",
)
return child_results, expanded
One retrieved child looked like this:
Rank: 1
Source: theory of relativity.txt
Score: 0.5520
Child:
"...whether in the actual case the two events took place
simultaneously or not."
The child is highly focused, while its parent restores the broader explanation around simultaneity.
This strategy worked well when the answer required context surrounding the exact matching passage. However, the larger parent could also introduce unrelated material.
More context is not always better context.
2. Sentence-Window Retrieval
Sentence-window retrieval uses individual sentences as the searchable units.
Each sentence receives an embedding. When a sentence matches the query, the system adds nearby sentences from the original document to create a context window.
Query
→ retrieve the most similar sentence
→ add neighboring sentences
→ return the expanded window
WINDOW_SIZE = 2
sentence_rows = []
for document in documents:
sentences = split_sentences(
document["text"]
)
for i, sentence in enumerate(sentences):
start = max(
0,
i - WINDOW_SIZE,
)
end = min(
len(sentences),
i + WINDOW_SIZE + 1,
)
sentence_rows.append({
"sentence_id": (
f"{document['source']}"
f"::sentence::{i}"
),
"source": document["source"],
"sentence_text": sentence,
"window_text": " ".join(
sentences[start:end]
),
"window_start": start,
"window_end": end - 1,
})
sentence_df = pd.DataFrame(
sentence_rows
)
sentence_df["embedding"] = get_embeddings(
sentence_df[
"sentence_text"
].tolist(),
"sentences",
)
Sentence windows were compact and readable when the relevant explanation was concentrated within a few nearby sentences.
They work particularly well when:
the answer appears in one clear statement,
neighboring sentences provide sufficient explanation,
the source follows a coherent local structure.
The main limitation is that nearby text is not always the most useful text. An explanation may depend on another paragraph or section rather than the immediately adjacent sentences.
Sentence-window retrieval therefore assumes that useful context is locally concentrated.
3. Summary-to-Detail Retrieval
Summary-to-detail retrieval introduces a hierarchical routing step.
First, larger parent sections are summarized using an LLM. These summaries form a lightweight index representing the major ideas in each section.
At query time, the system:
retrieves the most relevant summaries,
identifies their corresponding parent sections,
searches detailed child chunks only within those sections.
Query
→ search section summaries
→ select relevant sections
→ search detailed chunks within those sections
→ return the strongest details
def summarize_text(text: str) -> str:
response = client.responses.create(
model=SUMMARY_MODEL,
input=[
{
"role": "system",
"content": (
"Summarize this passage for retrieval "
"in 2-3 concise sentences. Preserve "
"important concepts, terms, examples, "
"and claims. Add no new facts."
),
},
{
"role": "user",
"content": text,
},
],
)
return response.output_text.strip()
This reduces the second-stage search space and adds document-level structure to retrieval.
In my experiment, parent sections averaged approximately 883 words, while their summaries averaged around 102 words—a reduction of more than 88%.
Add notebook visual: Parent-section length versus summary length.
The summaries provide a compact representation for routing, while the child chunks preserve the original detail.
However, the approach depends heavily on summary quality. If an important concept is omitted from a summary, the corresponding section may never enter the detailed search stage.
The summary does not merely compress the source. It influences which sections remain discoverable.
4. LLM Reranking
The previous techniques change how the retrieval index is structured. LLM reranking instead changes the order of the retrieved candidates.
The first-stage retriever returns a wider candidate set to maximize recall. An LLM then compares those candidates with the query and reorders them according to how directly and completely they answer it.
Query
→ retrieve a wider candidate set
→ ask the LLM to compare candidates
→ reorder candidates
→ retain the strongest results
def llm_rerank(
query,
candidate_df,
text_column="child_text",
top_k=3,
):
candidates = [
{
"candidate_id": i,
"text": row[text_column],
}
for i, (_, row) in enumerate(
candidate_df.iterrows(),
start=1,
)
]
payload = {
"query": query,
"instruction": (
"Rank candidates by how directly "
"and completely they help answer "
f"the query. Return the best {top_k} IDs."
),
"candidates": candidates,
}
response = client.responses.create(
model=RERANK_MODEL,
input=[
{
"role": "system",
"content": (
'Return valid JSON only: '
'{"ranked_candidate_ids":[1,2,3]}'
),
},
{
"role": "user",
"content": json.dumps(payload),
},
],
)
ranked_ids = json.loads(
response.output_text
)["ranked_candidate_ids"][:top_k]
rows = []
for new_rank, candidate_id in enumerate(
ranked_ids,
start=1,
):
if 1 <= candidate_id <= len(candidate_df):
row = candidate_df.iloc[
candidate_id - 1
].copy()
row["initial_rank"] = candidate_id
row["reranked_rank"] = new_rank
rows.append(row)
return pd.DataFrame(rows)
Add notebook output: Before-and-after candidate rankings.
In one experiment, a chunk initially ranked third contained the clearest explanation of why observers disagree about simultaneity. After reranking, the LLM moved it to the first position.
Vector similarity identifies broadly related candidates. The reranker performs a second-stage comparison to determine which candidate best answers the actual question.
However, reranking improves ordering, not recall.
If the correct passage is absent from the original candidate set, the reranker cannot recover it. The first-stage retriever must therefore return a sufficiently broad set.
Comparing the Strategies
| Strategy | Search unit | Returned context | Primary benefit | Main trade-off |
|---|---|---|---|---|
| Fixed-size | Medium chunk | Same chunk | Simple baseline | One size must balance precision and context |
| Small-to-large | Small child | Large parent | Precise matching with fuller context | Parent expansion may add noise |
| Sentence-window | Sentence | Nearby sentences | Focused and readable context | Assumes useful context is nearby |
| Summary-to-detail | Summary, then child | Detailed child | Hierarchical routing and smaller search space | Summary omissions affect discoverability |
| LLM reranking | Existing candidates | Reordered candidates | Better candidate ordering | Adds cost and cannot improve recall |
These strategies can also be combined.
For example, a system could retrieve small child chunks, expand them to parents, and then rerank the expanded candidates.
The right architecture depends on document structure, query type, latency constraints, context limits, and acceptable cost.
What the Experiments Showed
Five lessons stood out.
First, fixed-size retrieval remains a useful baseline, but one chunk size rarely optimizes both precise search and complete context.
Second, small-to-large retrieval separates retrieval granularity from generation context, although larger parents can introduce noise.
Third, sentence windows work well when explanations are locally concentrated and become less effective when the required context is distributed across the document.
Fourth, summary-to-detail retrieval provides useful hierarchical routing, but summaries become part of the retrieval infrastructure. Missing information in a summary can hide the original details.
Finally, reranking can improve the ordering of retrieved candidates but cannot recover evidence that the initial retriever missed.
None of these techniques establishes factual truth. They optimize relevance, ranking, and context quality. Evidence verification, contradiction handling, and Corrective RAG remain separate responsibilities.
In the final article of this series, I will bring these components together and evaluate the complete RAG pipeline using retrieval metrics, evidence grading, answer evaluation, and an end-to-end scorecard.
The implementation is available in the RAG From First Principles GitHub repository.
References and Further Reading
Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering.
Huang, J. RAG from First Principles: Mastering Retrieval-Augmented Generation.