RAG From First Principles: How Documents Become Retrievable Knowledge
Understanding the complete RAG pipeline through document loading, metadata preservation, and chunking experiments
This series continues the journey from my earlier work on understanding language models from the inside out. In my first Hashnode series, beginning with Why I Built a GPT Model From Scratch, I assembled a GPT-style model to understand its core architecture. In Inside GPT: Experiments and Trade-offs, I used a smaller model as an experimental sandbox for training, architecture, fine-tuning, and decoding choices. With those model-level foundations in place, this series shifts toward application-level systems: how LLMs retrieve external knowledge, select evidence, generate grounded answers, and how the complete pipeline can be evaluated.
The implementation for this series is available in the RAG From First Principles GitHub repository.
TL;DR
RAG is more than embeddings and a vector database.
Documents must first be loaded, structured, and divided into useful chunks.
Chunking defines the unit of knowledge a retriever can return.
Retrieval, reranking, verification, and generation all depend on these early decisions.
In Notebook 1, I loaded Einstein-related TXT, PDF, PPTX, CSV, and HTML files.
I compared character, recursive, sentence-based, and semantic chunking.
The experiments showed that chunk quality depends on coherence, boundaries, metadata, and context—not only size.
The Full RAG Pipeline
Retrieval-Augmented Generation is often summarized as:
Documents → Embeddings → Vector Database → LLM Answer
That is a useful starting point, but a practical RAG system involves many more decisions:
Source documents
→ loading and parsing
→ cleaning and metadata preservation
→ chunking
→ embeddings and indexing
→ retrieval
→ reranking or evidence verification
→ context construction
→ answer generation
→ evaluation
Each stage affects the next.
If a PDF is parsed poorly, its chunks will also be poor. If a chunk combines unrelated ideas, retrieval may return noisy context. If weak evidence reaches the LLM, even a capable model may generate an incorrect answer.
RAG quality therefore cannot be improved only at the generation stage.
1. Loading and Parsing Documents
RAG systems rarely begin with clean text. Source material may come from PDFs, presentations, spreadsheets, HTML pages, websites, databases, images, or internal documents.
Each format has its own structure.
A PDF may contain page numbers, headers, footnotes, and multi-column layouts. A presentation organizes information by slide. A CSV stores rows and columns rather than prose. HTML contains headings, links, navigation elements, and markup.
The task is not merely to extract text. A useful ingestion process should also preserve information such as:
file name and source type
page or slide number
section heading
author or publication date
document category
This metadata later supports filtering, citations, debugging, and evaluation.
2. Chunking
Documents are usually too large to retrieve and pass to an LLM as a single unit. They must be divided into smaller pieces called chunks.
Chunking may look like a basic preprocessing task, but it determines the unit of knowledge available to the retriever.
When chunks are too small, ideas can lose context, definitions may be separated from explanations, and retrieved passages may feel incomplete.
When chunks are too large, multiple topics may be mixed together, embeddings must represent too many ideas, retrieval becomes less precise, and more tokens are passed downstream.
Common approaches include:
Character-based chunking
Text is divided after a fixed number of characters. It is simple and predictable, but it may split sentences or ideas at unnatural points.
Recursive chunking
The splitter attempts larger separators first—such as paragraphs—before falling back to sentences, words, and characters. This usually preserves structure better than fixed-character splitting.
Sentence-based chunking
Text is grouped around sentence boundaries. The output is more readable, although sentence count alone does not guarantee that a chunk contains a complete idea.
Format-aware chunking
Boundaries follow document structure, such as headings, slides, sections, or table rows.
Semantic chunking
Embeddings are used to identify changes in meaning. A new chunk begins when the semantic difference between nearby sentences crosses a threshold.
This is more adaptive, but it also depends on the embedding model, threshold, and source material.
3. Embeddings, Indexing, and Retrieval
After chunking, the system must make the chunks searchable.
In dense retrieval, each chunk is converted into an embedding—a numerical representation intended to capture meaning. The query is embedded in the same space, and similar chunks are retrieved.
Sparse methods such as BM25 rely more heavily on lexical overlap. Hybrid retrieval combines semantic and lexical signals.
Later articles in this series will explore:
dense and sparse retrieval
hybrid search and rank fusion
parent-child retrieval
hierarchical indexes
reranking and evidence grading
The key point is that indexing cannot repair weak input. An embedding cannot restore context that was removed during chunking or separate unrelated ideas that were placed in one chunk.
At query time, retrieval produces candidate evidence. But similarity is not the same as usefulness.
A passage may be semantically related but incomplete, relevant but too broad, or highly similar while still being factually incorrect. More advanced pipelines therefore add context expansion, reranking, metadata filtering, verification, or corrective retrieval.
4. Evaluating the Whole Pipeline
RAG cannot be evaluated using one metric because failures can occur at different stages.
A broader evaluation suite may include:
Retrieval quality: Hit Rate, Precision@k, and Recall@k
Ranking quality: Mean Reciprocal Rank and nDCG
Evidence quality: accuracy, precision, recall, F1, and confusion matrices
Answer quality: correctness, completeness, and faithfulness
Operational behavior: whether corrective retrieval or fallback was triggered appropriately
An LLM can act as a judge when evaluation requires semantic interpretation, such as determining whether an answer is faithful to supplied evidence.
However, an LLM judge is only one component of the evaluation pipeline. It should complement labeled datasets and deterministic metrics rather than replace them.
What I Built in Notebook 1
The first notebook focuses on document loading and chunking using Einstein-related material in five formats:
TXT
PDF
PPTX
CSV
HTML
I used both LangChain and LlamaIndex and inspected the resulting text, document boundaries, and metadata.
Loading CSV and PDF documents
from langchain_community.document_loaders import CSVLoader, PyPDFLoader
csv_loader = CSVLoader(
"../data/raw/csv/einstein_source_inventory.csv"
)
csv_documents = csv_loader.load()
pdf_loader = PyPDFLoader(
"../data/raw/pdf/theory_of_relativity.pdf"
)
pdf_documents = pdf_loader.load()
Notebook output: Add one example showing extracted text and its metadata.
Different loaders may return similar text while producing different metadata and document boundaries. Those differences matter when retrieved chunks must later be traced back to their sources.
Comparing Chunking Strategies
I compared character, recursive, sentence-based, and semantic chunking. For each method, I inspected the number of chunks, average size, minimum and maximum size, size distribution, and sample outputs.
Recursive chunking
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
separators=["\n\n", "\n", " ", ""],
)
recursive_chunks = text_splitter.split_documents(documents)
Semantic chunking
embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5"
)
semantic_splitter = SemanticSplitterNodeParser(
buffer_size=1,
breakpoint_percentile_threshold=95,
embed_model=embed_model,
)
semantic_chunks = semantic_splitter.get_nodes_from_documents(
documents
)
I also varied the semantic breakpoint threshold. This changed both the number and average size of the resulting chunks.
What the Experiments Showed
The experiments reinforced five lessons:
File format affects both extracted text and preserved metadata.
Configured chunk size is not the same as actual chunk size.
Semantic chunking is adaptive, but not automatically superior.
More chunks improve granularity but increase indexing and evaluation costs.
Chunk quality cannot be judged by length alone. A useful chunk should be coherent, understandable, and sufficiently self-contained.
In the next article, I will move from preparing knowledge to retrieving it by comparing dense retrieval, BM25, hybrid search, and Reciprocal Rank Fusion.
References and Further Reading
Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering.
Huang, J. RAG from First Principles: Mastering Retrieval-Augmented Generation.
