{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "RAG From First Principles: How Documents Become Retrievable Knowledge",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/rag-from-first-principles-how-documents-become-retrievable-knowledge",
  "publish_date": "2026-08-02",
  "corpus_stage": "",
  "notes": "",
  "id": "97f088d1eebf",
  "collected_at": "2026-10-02T14:11:43",
  "title": "RAG From First Principles: Document Loading and Chunking",
  "body_markdown": "RAG From First Principles: How Documents Become Retrievable Knowledge\nUnderstanding the complete RAG pipeline through document loading, metadata preservation, and chunking experiments\nUpdated\nAugust 2, 2026\n•\n7\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nRAG From First Principles\nThis series continues the journey from my earlier work on understanding language models from the inside out. In my first Hashnode series, beginning with\nWhy I Built a GPT Model From Scratch\n, I assembled a GPT-style model to understand its core architecture. In\nInside GPT: Experiments and Trade-offs\n, I used a smaller model as an experimental sandbox for training, architecture, fine-tuning, and decoding choices. With those model-level foundations in place, this series shifts toward application-level systems: how LLMs retrieve external knowledge, select evidence, generate grounded answers, and how the complete pipeline can be evaluated.\nThe implementation for this series is available in the\nRAG From First Principles GitHub repository\n.\nTL;DR\nRAG is more than embeddings and a vector database.\nDocuments must first be loaded, structured, and divided into useful chunks.\nChunking defines the unit of knowledge a retriever can return.\nRetrieval, reranking, verification, and generation all depend on these early decisions.\nIn Notebook 1, I loaded Einstein-related TXT, PDF, PPTX, CSV, and HTML files.\nI compared character, recursive, sentence-based, and semantic chunking.\nThe experiments showed that chunk quality depends on coherence, boundaries, metadata, and context—not only size.\nThe Full RAG Pipeline\nRetrieval-Augmented Generation is often summarized as:\nDocuments → Embeddings → Vector Database → LLM Answer\nThat is a useful starting point, but a practical RAG system involves many more decisions:\nSource documents\n→ loading and parsing\n→ cleaning and metadata preservation\n→ chunking\n→ embeddings and indexing\n→ retrieval\n→ reranking or evidence verification\n→ context construction\n→ answer generation\n→ evaluation\nEach stage affects the next.\nIf a PDF is parsed poorly, its chunks will also be poor. If a chunk combines unrelated ideas, retrieval may return noisy context. If weak evidence reaches the LLM, even a capable model may generate an incorrect answer.\nRAG quality therefore cannot be improved only at the generation stage.\n1. Loading and Parsing Documents\nRAG systems rarely begin with clean text. Source material may come from PDFs, presentations, spreadsheets, HTML pages, websites, databases, images, or internal documents.\nEach format has its own structure.\nA PDF may contain page numbers, headers, footnotes, and multi-column layouts. A presentation organizes information by slide. A CSV stores rows and columns rather than prose. HTML contains headings, links, navigation elements, and markup.\nThe task is not merely to extract text. A useful ingestion process should also preserve information such as:\nfile name and source type\npage or slide number\nsection heading\nauthor or publication date\ndocument category\nThis metadata later supports filtering, citations, debugging, and evaluation.\n2. Chunking\nDocuments are usually too large to retrieve and pass to an LLM as a single unit. They must be divided into smaller pieces called chunks.\nChunking may look like a basic preprocessing task, but it determines the unit of knowledge available to the retriever.\nWhen chunks are too small, ideas can lose context, definitions may be separated from explanations, and retrieved passages may feel incomplete.\nWhen chunks are too large, multiple topics may be mixed together, embeddings must represent too many ideas, retrieval becomes less precise, and more tokens are passed downstream.\nCommon approaches include:\nCharacter-based chunking\nText is divided after a fixed number of characters. It is simple and predictable, but it may split sentences or ideas at unnatural points.\nRecursive chunking\nThe splitter attempts larger separators first—such as paragraphs—before falling back to sentences, words, and characters. This usually preserves structure better than fixed-character splitting.\nSentence-based chunking\nText is grouped around sentence boundaries. The output is more readable, although sentence count alone does not guarantee that a chunk contains a complete idea.\nFormat-aware chunking\nBoundaries follow document structure, such as headings, slides, sections, or table rows.\nSemantic chunking\nEmbeddings are used to identify changes in meaning. A new chunk begins when the semantic difference between nearby sentences crosses a threshold.\nThis is more adaptive, but it also depends on the embedding model, threshold, and source material.\n3. Embeddings, Indexing, and Retrieval\nAfter chunking, the system must make the chunks searchable.\nIn dense retrieval, each chunk is converted into an embedding—a numerical representation intended to capture meaning. The query is embedded in the same space, and similar chunks are retrieved.\nSparse methods such as BM25 rely more heavily on lexical overlap. Hybrid retrieval combines semantic and lexical signals.\nLater articles in this series will explore:\ndense and sparse retrieval\nhybrid search and rank fusion\nparent-child retrieval\nhierarchical indexes\nreranking and evidence grading\nThe key point is that indexing cannot repair weak input. An embedding cannot restore context that was removed during chunking or separate unrelated ideas that were placed in one chunk.\nAt query time, retrieval produces candidate evidence. But similarity is not the same as usefulness.\nA passage may be semantically related but incomplete, relevant but too broad, or highly similar while still being factually incorrect. More advanced pipelines therefore add context expansion, reranking, metadata filtering, verification, or corrective retrieval.\n4. Evaluating the Whole Pipeline\nRAG cannot be evaluated using one metric because failures can occur at different stages.\nA broader evaluation suite may include:\nRetrieval quality:\nHit Rate, Precision@k, and Recall@k\nRanking quality:\nMean Reciprocal Rank and nDCG\nEvidence quality:\naccuracy, precision, recall, F1, and confusion matrices\nAnswer quality:\ncorrectness, completeness, and faithfulness\nOperational behavior:\nwhether corrective retrieval or fallback was triggered appropriately\nAn LLM can act as a judge when evaluation requires semantic interpretation, such as determining whether an answer is faithful to supplied evidence.\nHowever, an LLM judge is only one component of the evaluation pipeline. It should complement labeled datasets and deterministic metrics rather than replace them.\nWhat I Built in Notebook 1\nThe first notebook focuses on document loading and chunking using Einstein-related material in five formats:\nTXT\nPDF\nPPTX\nCSV\nHTML\nI used both LangChain and LlamaIndex and inspected the resulting text, document boundaries, and metadata.\nLoading CSV and PDF documents\nfrom langchain_community.document_loaders import CSVLoader, PyPDFLoader\n\ncsv_loader = CSVLoader(\n    \"../data/raw/csv/einstein_source_inventory.csv\"\n)\ncsv_documents = csv_loader.load()\n\npdf_loader = PyPDFLoader(\n    \"../data/raw/pdf/theory_of_relativity.pdf\"\n)\npdf_documents = pdf_loader.load()\nNotebook output:\nAdd one example showing extracted text and its metadata.\nDifferent loaders may return similar text while producing different metadata and document boundaries. Those differences matter when retrieved chunks must later be traced back to their sources.\nComparing Chunking Strategies\nI compared character, recursive, sentence-based, and semantic chunking. For each method, I inspected the number of chunks, average size, minimum and maximum size, size distribution, and sample outputs.\nRecursive chunking\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=500,\n    chunk_overlap=50,\n    separators=[\"\\n\\n\", \"\\n\", \" \", \"\"],\n)\n\nrecursive_chunks = text_splitter.split_documents(documents)\nSemantic chunking\nembed_model = HuggingFaceEmbedding(\n    model_name=\"BAAI/bge-small-en-v1.5\"\n)\n\nsemantic_splitter = SemanticSplitterNodeParser(\n    buffer_size=1,\n    breakpoint_percentile_threshold=95,\n    embed_model=embed_model,\n)\n\nsemantic_chunks = semantic_splitter.get_nodes_from_documents(\n    documents\n)\nI also varied the semantic breakpoint threshold. This changed both the number and average size of the resulting chunks.\nWhat the Experiments Showed\nThe experiments reinforced five lessons:\nFile format affects both extracted text and preserved metadata.\nConfigured chunk size is not the same as actual chunk size.\nSemantic chunking is adaptive, but not automatically superior.\nMore chunks improve granularity but increase indexing and evaluation costs.\nChunk quality cannot be judged by length alone. A useful chunk should be coherent, understandable, and sufficiently self-contained.\nIn the next article, I will move from preparing knowledge to retrieving it by comparing dense retrieval, BM25, hybrid search, and Reciprocal Rank Fusion.\nReferences and Further Reading\nLewis, P. et al. (2020).\nRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks\n.\nKarpukhin, V. et al. (2020).\nDense Passage Retrieval for Open-Domain Question Answering\n.\nHuang, J.\nRAG from First Principles: Mastering Retrieval-Augmented Generation\n.\nLangChain document-loader documentation\n.\nLangChain text-splitter documentation\n.\nLlamaIndex SemanticSplitterNodeParser documentation\n.\nBAAI/bge-small-en-v1.5 model card\n.\nProject code and notebooks\n.\n#\nrag\n#\nretrieval-augmented-generation\n#\ngenerative-ai\n10\nviews",
  "metadata": {
    "title": "RAG From First Principles: Document Loading and Chunking",
    "description": "A practical introduction to the RAG pipeline, experiments comparing document loaders and character, recursive, sentence-based, and semantic chunking.",
    "keywords": null,
    "author": null,
    "og:title": "RAG From First Principles: Document Loading and Chunking",
    "og:description": "A practical introduction to the RAG pipeline, experiments comparing document loaders and character, recursive, sentence-based, and semantic chunking.",
    "og:url": "https://curious-pm.hashnode.dev/rag-from-first-principles-how-documents-become-retrievable-knowledge",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/og-images/6a4f47d631bb27227f782535/aa3ccc4b-83d5-455f-90bf-fd9d91e55869.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "RAG From First Principles: Document Loading and Chunking",
    "twitter:description": "A practical introduction to the RAG pipeline, experiments comparing document loaders and character, recursive, sentence-based, and semantic chunking.",
    "twitter:image": "https://cdn.hashnode.com/uploads/og-images/6a4f47d631bb27227f782535/aa3ccc4b-83d5-455f-90bf-fd9d91e55869.png",
    "article:published_time": "2026-08-02T20:33:46.224Z",
    "article:modified_time": "2026-08-02T20:40:48.744Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/97f088d1eebf/code/body-000.txt",
      "chars": 54,
      "language": "text"
    },
    {
      "file": "attachments/97f088d1eebf/code/body-001.txt",
      "chars": 218,
      "language": "text"
    },
    {
      "file": "attachments/97f088d1eebf/code/body-002.txt",
      "chars": 295,
      "language": "text"
    },
    {
      "file": "attachments/97f088d1eebf/code/body-003.txt",
      "chars": 193,
      "language": "text"
    },
    {
      "file": "attachments/97f088d1eebf/code/body-004.txt",
      "chars": 296,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5962d039-e6a7-444e-9aa7-d1faf8e30d96.png",
      "bytes": 1094932,
      "reused": true,
      "local_path": "attachments/97f088d1eebf/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5962d039-e6a7-444e-9aa7-d1faf8e30d96.png",
      "alt": "RAG From First Principles: How Documents Become Retrievable Knowledge",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/b55465d4-e632-44c3-9797-8f6afeb8bd4e.png",
      "bytes": 20198,
      "reused": true,
      "local_path": "attachments/97f088d1eebf/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/b55465d4-e632-44c3-9797-8f6afeb8bd4e.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/da5b87af-7e2d-47ab-a759-6554849fb601.png",
      "bytes": 16368,
      "reused": true,
      "local_path": "attachments/97f088d1eebf/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/da5b87af-7e2d-47ab-a759-6554849fb601.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/7444b355-3da6-473d-8d9e-626f9467ed7b.png",
      "bytes": 24062,
      "reused": true,
      "local_path": "attachments/97f088d1eebf/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/7444b355-3da6-473d-8d9e-626f9467ed7b.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/rag-from-first-principles-how-documents-become-retrievable-knowledge.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/building-rag-from-scratch",
      "label": "Part of series RAG From First Principles",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/why-i-built-a-gpt-model-from-scratch-and-what-i-actually-built",
      "label": "Why I Built a GPT Model From Scratch",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/inside-gpt-experiments-and-trade-offs",
      "label": "Inside GPT: Experiments and Trade-offs",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/rag-from-scratch-work",
      "label": "RAG From First Principles GitHub repository",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.11401",
      "label": "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2004.04906",
      "label": "Dense Passage Retrieval for Open-Domain Question Answering",
      "type": "a"
    },
    {
      "href": "https://docs.langchain.com/oss/python/integrations/document_loaders",
      "label": "LangChain document-loader documentation",
      "type": "a"
    },
    {
      "href": "https://docs.langchain.com/oss/python/integrations/splitters/index",
      "label": "LangChain text-splitter documentation",
      "type": "a"
    },
    {
      "href": "https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/semantic_splitter/",
      "label": "LlamaIndex SemanticSplitterNodeParser documentation",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/BAAI/bge-small-en-v1.5",
      "label": "BAAI/bge-small-en-v1.5 model card",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/rag",
      "label": "# rag",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/retrieval-augmented-generation",
      "label": "# retrieval-augmented-generation",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/generative-ai",
      "label": "# generative-ai",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:08:50",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:53Z",
  "source_url": "https://curious-pm.hashnode.dev/rag-from-first-principles-how-documents-become-retrievable-knowledge",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.11401",
      "bytes": 885323,
      "reused": true,
      "local_path": "papers/repair/fc8d32f196e0f0c27ffb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2004.04906",
      "bytes": 383508,
      "reused": true,
      "local_path": "papers/repair/8b76986dbe24871ab2c9.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 9618,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/97f088d1eebf/pages/source.html",
  "local_html": "attachments/97f088d1eebf/pages/article.html",
  "body_base_path": "attachments/97f088d1eebf/pages/",
  "body_markdown_path": "attachments/97f088d1eebf/pages/article.md"
}