{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Angela Shi & Kezhan Shi",
  "title_listed": "One Document Type, a Million Files: Structured Extraction into the SQL Table RAG Queries",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/one-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries/",
  "publish_date": "2026-08-25",
  "corpus_stage": "",
  "notes": "",
  "id": "c2c4925fbd5c",
  "collected_at": "2026-10-02T01:02:29",
  "title": "One Document Type, a Million Files: Structured Extraction into the SQL Table RAG Queries | Towards Data Science",
  "body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# One Document Type, a Million Files: Structured Extraction into the SQL Table RAG Queries\nEnterprise Document Intelligence [Vol.1 #14C] - One hour with two people, six to ten fields, and the two signals that separate a real column from one that will break a filter later\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 25, 202616 min read\n![](../images/000.jpeg)\n[Photo](https://www.pexels.com/photo/36681344/) by Bor Jinson, via Pexels.\nA lot of RAG work right now goes into letting a model discover the structure of a document collection: read every file, pull out the entities, infer the relations, and a graph comes out that nobody had to design. On five thousand copies of the same contract, there is nothing left to discover.\nAsk the person who files those contracts what they look things up by. The answer takes about ten seconds: client, effective date, product, premium, renewal date, policy number.\nNobody found those six by reading the documents. They were designed years ago by whoever drew up the form, and every file since has been filled in against them. These files are the output of one repeatable process, and the process is what defines the fields.\nSo the collection is a database nobody typed. Its schema lives in the head of the person who handles the files, and the work is to get it out and write it into a table.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\nThis article assumes the diagnostic in Article 14A (three kinds of corpus, and what building for the wrong one costs) already came back with this shape: one document type, many copies. It covers what to do next:\n  * the interview that produces the columns, question by question\n  * the two signals that tell a real column from a field that will break a filter later\n  * the index as a table of contents for the collection, and why it keeps growing\n  * what gets paid once per document, and what gets paid on every question\n  * the arithmetic, at fifty questions a day, that says when the preparation pays for itself\n\n\nThis article is part of Part IV of [Enterprise Document Intelligence](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/), a series that builds an enterprise RAG system from four bricks. Part IV is where the input stops being a file and becomes a folder, and this is the folder shape where a table is the right answer.\n🧭 **New to the series? Start with the map** : [Prompt, Context, Loop](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/) sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.\n![](../images/001.png)\n_where this article sits: Part IV, the folder that becomes a table - Image by author_\n📓 **Run the column test on your own folder in the companion notebook** : declare the six columns, fill them across a folder of look-alike PDFs, then print the fill rate per column and watch two of the fields that sounded like columns come back half empty. Repo → [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1).\n![](../images/body-66d15da324dbbf109243.png)\n_The public companion-code repo at doc-intel/notebooks-vol1 - Image by author_\nThe worked example is an insurance portfolio, and it is fictional: the clients, policy numbers and amounts were authored for the series and match no real insured or insurer. Section 3.2 gives a public folder you can run the same test on.\n## 1. One document type, and the columns that came with it\nTwo things to settle before the interview: what makes this shape recognisable, and where its columns come from.\n### 1.1. What the shape looks like on a shelf\nTwelve years of renewal notices for one motor product. Every purchase order a procurement team has issued since the ERP went in. The monthly statements of one fund. Each of those is thousands of files, and each file is the same document with different values in it.\nThe count is not what matters. A hundred purchase orders behave like a hundred thousand of them: the questions people ask are about _which ones_ , not about where inside one of them.\nTwo things follow. Documents that came out of one process share a page-one layout, so the same value sits in roughly the same place every time. And the values themselves repeat: eleven products, forty-odd clients, four statuses. A column here has a small, stable set of values, which is what makes it worth filtering on.\n### 1.2. The columns existed before the corpus did\nA field like `effective_date` does not come from the text. It comes from whoever designed the form. A template has slots, a form has boxes, a back-office screen has required fields, and the PDF is that screen printed on paper.\nNobody wrote that decision down. There is no data dictionary, and the person who designed the form left in 2014. What survives is the practice: the fields people fill in, and the ones they check before answering a customer. That knowledge sits with the handlers, so getting it out is a conversation rather than an extraction job.\nThis is where the shape parts company with the folder in Article 14B (a collection whose documents share no field at all). There, no such decision was ever made, so there is nothing upstream to recover.\n## 2. The interview that gets you the columns\nOne hour, two people, and a list of six to ten fields at the end of it.\n### 2.1. Who you ask, and in what order\nAsk the person who handles the documents, not the manager who owns the process. The manager describes the process as designed. The handler knows which field is blank on the older files and which one gets typed in wrong.\nAsk two handlers, separately. The second hour is there to catch one thing: a field they both name the same way and mean differently.\nFour questions, in this order. Each one is easier to answer once the previous one has been.\n![](../images/body-61376b80ece8b141d2ea.png)\n_four questions, and what each answer settles about the schema - Image by author_\nWhat comes out is a list of names in the handler’s own words. Types and definitions come later.\n### 2.2. What tells you a column is real\nTwo signals, and both are about the answer rather than about the field.\n**It comes back without a pause.** Ask what someone filters on and a real column arrives at conversational speed, with an example attached: _“client, obviously, and the effective date, because that is how we know which version applies.”_ Take a field that needs thinking about. Nobody filters on it today, so nobody will keep it up to date either.\n**Two handlers name it the same way.** Same word, same meaning, same box on the page. The first signal cannot do this on its own, because a field can be named instantly by both handlers and still mean two different things to them.\nNeither signal needs a model or a sample. Both are settled in the room.\n![](../images/body-7142c790f6857b751655.png)\n_three fields the business named, and the two that are not columns - Image by author_\n### 2.3. Three answers that are not columns\n**Two meanings in one word.** One handler says _date_ and means the day the contract was signed. The other says _date_ and means the day the cover starts. Both answered fast, both used the same word, and the column that comes out of it holds two quantities mixed together. A query for policies in force in March then drops the rows carrying a signature date, and what comes back still looks like an answer. A missing field is safer, because a filter trusts what is there.\n**Empty on half the documents.** A field filled on 1,900 rows out of 5,000 is a subset with no name. Sometimes the subset is real, and then say so: the field belongs to a subtype, and the subtype is the column you were missing. Sometimes it is just a field that arrived in 2021, which is fine, as long as the filter reads a null as _not recorded yet_ rather than as _no_.\n**A judgement rather than a fact.** _“Is this client difficult?”_ is real knowledge and it is not printed anywhere. Ask a model to infer it and you get an opinion with a citation attached. Park those fields. Some come back later as something the business writes into the index by hand, which is not the same thing as extraction.\nThere is an alternative to that hour, and it is popular for good reasons. Point a model at the corpus, let it pull out whatever entities and relations it finds, and a vocabulary comes back that nobody had to specify. On a collection where no business user can name a field, that is the only move available.\nHere the trade is different. You pay a full extraction pass over five thousand documents to rediscover a list two people produce in an hour, and the vocabulary that comes back is one the business did not choose and cannot correct in place. Article 16 (where the corpus vocabulary comes from, curated against extracted) puts the two side by side on cost, audit trail and freshness. The ordering is what matters here. Ask first, and if the answer arrives in ten seconds, there is nothing left to discover.\n## 3. The index is a table of contents for the collection\nThe index has one job, and the quickest way to see it is to put it beside something every reader already uses.\n### 3.1. A map read before the content\nA table of contents is not part of a document’s content, it is a map of it. Six lines stand in for thirty-four pages, and a question about exclusions lands on section 4 before a page is read.\nThe index does the same job one level up. It is not the content of the corpus either. Five columns stand in for five thousand files, and a question about one client’s fleet cover lands on one row before a file is opened.\n![](../images/body-fb81a936ed54384291e3.png)\n_the same job at two scales: pick the section, pick the documents - Image by author_\nFollow that comparison all the way, because it tells you what belongs in the index. A table of contents does not hold the text of section 4, it holds enough to decide whether section 4 is where to look. An index row holds enough to decide whether a document is a candidate. Once it starts holding answers it is a second copy of the corpus, and the second copy drifts.\nRebuilding the map when a document ships none is a parsing job: [Building Document Structure with Loop Engineering](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/) recovers an outline from body typography for a PDF that never encoded one.\n### 3.2. The row, written down\nSix fields, in the handler’s words, with types on them.\npython\n\n```\nclass PolicyIndexRow(BaseModel):    \"\"\"One row of the corpus index: one policy, six business fields.\"\"\"    doc_id: str              # the file, so an answer can cite where it came from    policy_number: str       # what the business types into its own systems    client_name: str         # the party the policy is written for    product: str             # one of eleven, and the list never changes    effective_date: date     # the day the cover starts    renewal_date: date       # the day it lapses if nobody acts    premium_eur: float       # the number the finance team filters on\n```\n\nNothing in that declaration is a modelling decision. Every name came out of the interview. `doc_id` is the one field the business did not name, and it is there so an answer can point back at the file it was read from.\nWhether the table stays this flat once amendments and versions arrive is the subject of Article 15A (the schema a corpus gets written into), which replaces the single sheet with three tables and two date columns. How each field gets filled comes with the articles after it.\nTake a folder of arXiv papers if you want to try the shape without an insurance portfolio. Same thing: one document type, thousands of copies, and the first three columns come free. A file named `1706.03762v7.pdf` gives you the identifier, the version, and the submission month, all three read off the filename before a page is parsed. The primary category is a fourth column and it costs one lookup, because the modern identifier does not encode it: it comes from the arXiv metadata. Then add _the task_ and _the datasets used_ , and the line between a printed field and a judgement shows up within a dozen files. The papers carry the arXiv non-exclusive distribution license declared on each [abstract page](https://arxiv.org/).\n### 3.3. The index keeps growing, and that is the design\nA question turns up that the columns cannot answer. _Which of these policies carry a war exclusion?_ There is no such column, so the question falls back to opening documents one at a time.\nAdd the column, fill it on the documents it applies to, and every later question about war exclusions is a filter instead of a corpus scan. The cost is paid once and the benefit does not expire. On the naive path the same question pays full price every time somebody asks it.\nTwo habits keep that growth honest. Run each new field through the two signals from section 2.2 first, because a column added under pressure from one urgent question is the kind that means three things a year later. And keep a citation on every value, so that a number in the index can be checked against the page it was read from.\n## 4. Paid once, or paid on every question\nWhether the preparation is worth its price is arithmetic. Here it is.\n### 4.1. Two phases, and only one of them grows with the questions\nThe single-document pipeline does not change. It now sits inside two phases that cost very different things.\n![](../images/body-9c6d4cd70f21d2aff920.png)\n_the two phases, and the index they share - Image by author_\nIngestion runs once per document, when the file arrives and before anyone has asked anything. It parses, fills the agreed columns, keeps a page reference for each value, and appends a row. It runs again only when the file changes.\nQuerying runs once per question. It reads the scope fields out of the question, filters the index down to the candidates, and runs the pipeline on what survives. Article 17 (querying a corpus, filter first and retrieve second) builds that half.\nIngestion is bounded by the number of documents. Querying is bounded by the number of questions, and that is the number that keeps going up.\n### 4.2. Fifty questions a day, and what they add up to\nA claims handler uses a system like this about fifty times a day. That is the order of magnitude we keep seeing, and the exact number does not change what follows. Over a working year it is roughly twelve thousand questions from one person, and a team of six puts it past seventy thousand.\nWithout preparation, every one of those questions is priced against the whole shelf. The system works out from scratch which documents are candidates, and it does that by comparing the question against passages from files that had no business being consulted.\nWith preparation, the question opens with a filter over a table and the pipeline runs on what comes through. The work per question stops tracking the size of the corpus.\n![](../images/body-7eb27603846674dbc334.png)\n_what is paid once per document, and what is paid on every question - Image by author_\nIngestion is paid once per document and querying once per question. So as soon as the number of questions over the system’s life reaches the same order as the number of documents, the prepared version is ahead on compute. It was ahead earlier than that on two other counts: the answer comes from a document somebody chose, and the filter is something a business user can read and argue with.\n### 4.3. When the preparation is not worth it\nA document that gets read once, or never, does not justify a deep pass. Filling twelve columns on a one-off appendix costs more than reading the appendix when the question finally arrives, if it ever does.\nSo tier the work. A standard policy template consulted thousands of times over a portfolio’s life justifies a deep ingestion, with every field extracted and every cross-reference resolved. The long tail gets the cheap version, which here is the filename, the file metadata and the document type. A document can be promoted later, when its query count says it earned the deeper pass.\nThat is the same decision the generation brick makes about models: start cheap, and pay for the expensive path only where it changes the outcome. [Loop engineering for RAG generation](https://towardsdatascience.com/loop-engineering-for-rag-generation-an-llm-cascade-from-a-cheap-local-model-up-to-a-hosted-flagship/) works that cascade out on the model side.\n## 5. Conclusion\nOn a collection of one document type there is no schema to discover. There is a schema to write down, and it already exists in the practice of the people who handle the documents.\nThe preparation starts with an hour of conversation and two signals: a field named without a pause, and two people naming it the same way. What passes both goes into the index. What fails is a subtype, a judgement, or a word that means two things, and each of those is cheaper to catch in the room than after a pass over five thousand files.\nTry the two signals on a folder of your own before writing any ingestion code. Ask the person who handles those files what they filter on, then ask a second one, and see how much of the schema comes back in the first ten seconds.\nArticle 14B (the folder whose documents share no field) covers the collection with nothing to put in the columns. Article 14D (case files, one bundle about a single entity) covers the one where a row of the table is a case rather than a document.\n## 6. Further reading and sources\n**Earlier in the series:**\n  * [Document Intelligence: series intro](https://medium.com/@angela.shi/enterprise-document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-1cd0a3154abf). What the series builds, brick by brick, and in what order.\n\n\n_What works, what breaks_\n  * [Baseline Enterprise RAG, from PDF to highlighted answer](https://medium.com/data-science-collective/context-engineering-for-enterprise-rag-the-minimal-four-brick-pipeline-that-highlights-its-source-4de8c6776ecd). The four-brick pipeline end to end: PDF in, highlighted answer out.\n  * [Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval](https://medium.com/data-science-collective/embeddings-arent-magic-the-predictable-failure-modes-of-rag-retrieval-572620033db1). Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway.\n  * [RAG is not machine learning, and the ML toolkit solves the wrong problem](https://medium.com/@angela.shi/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem-a2fed3691e9f). Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead.\n  * [From regex to vision models: which RAG technique fits which problem](https://medium.com/data-science-collective/from-regex-to-vision-models-which-rag-technique-fits-which-problem-5afb3f142279). Two axes, document complexity and question control, that pick the technique for each case.\n    * [10 common RAG mistakes we keep seeing in production](https://medium.com/data-science-collective/10-common-rag-mistakes-we-keep-seeing-in-production-d0a2d4f45785). Ten production mistakes, organized brick by brick, with the fix for each.\n\n\n_Document parsing_\n  * [Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/). Rebuilding the outline from body typography when the PDF ships no contents page at all: six signals, one bounded loop.\n  * [Before Full Agentic RAG: Know How You Decide, and the Parsing Methods You Pick From](https://towardsdatascience.com/before-full-agentic-rag-know-how-you-decide-and-the-parsing-methods-you-pick-from/). The parsing methods as a catalogue, and the decision of which to run, before handing the loop to an agent.\n\n\n_Question parsing_\n  * [Loop engineering for RAG question parsing: the small loop that runs before retrieval](https://towardsdatascience.com/loop-engineering-for-rag-question-parsing-the-small-loop-that-runs-before-retrieval/). The bounded loop that runs before retrieval: parse, check the fields make sense, re-ask once when they do not.\n\n\n_Generation_\n  * [Loop Engineering for RAG Generation: Iterate top-k One at a Time](https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/). Reading the retrieved pages one at a time instead of all at once, and what that buys when the answer sits in only one of them.\n  * [Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract](https://towardsdatascience.com/most-rag-hallucinations-are-extraction-errors-seven-patterns-for-a-typed-generation-contract/). Seven recurring ways a model gets the extraction wrong, and the typed contract that catches each one.\n  * [Loop engineering for RAG generation: an LLM cascade from a cheap local model up to a hosted flagship](https://towardsdatascience.com/loop-engineering-for-rag-generation-an-llm-cascade-from-a-cheap-local-model-up-to-a-hosted-flagship/). Starting on a cheap local model and escalating only when the answer does not hold up, measured.\n\n\n_One-document pipelines_\n  * [Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations](https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/). Why a better prompt does not fix a wrong page, and what each of the four bricks contributes to the context instead.\n  * [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/). Cutting a pipeline’s latency and cost by calling the model less often and cheaper, not by buying a faster one.\n  * [Loop engineering for cross-references: when RAG answers ‘see Section 7.2’ instead of the actual answer](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/). When the answer says “see Section X”, the pipeline loops back and fetches it.\n  * [RAG workflow and loop engineering: the dispatcher that decides when to loop and when to stop](https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/). Feedback loops, bounded iteration, and the dispatcher, composed into one workflow.\n    * [Loop engineering for RAG: the small loops inside each step, the big loops across the pipeline](https://towardsdatascience.com/loop-engineering-for-rag-the-small-loops-inside-each-step-the-big-loops-across-the-pipeline/). The two scales of loop: small bounded loops inside each brick, big generation-triggered loops across them.\n\n\nAlso referenced above, listed here without links: Article 14A (three kinds of corpus, and what building for the wrong one costs), Article 14B (the folder whose documents share no field), Article 14D (case files, one bundle about a single entity), Article 15 (preparing a corpus, one column at a time), Article 16 (where the corpus vocabulary comes from, curated against extracted), and Article 17 (querying a corpus, filter first and retrieve second).\nExternal sources:\n  * Darren Edge et al., _From Local to Global: A Graph RAG Approach to Query-Focused Summarization_ , 2024 ([arXiv 2404.16130](https://arxiv.org/abs/2404.16130)). The opposite bet, stated properly: let a model extract the entities instead of asking the business to name the columns. Read it for the cases where discovery is the only option.\n  * Jinyang Li et al., _Can LLM Already Serve as A Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs_ , 2023 ([arXiv 2305.03111](https://arxiv.org/abs/2305.03111)). What a question over a typed table looks like once the table exists, measured on 95 real databases.\n  * Fatemeh Nargesian et al., _Data Lake Management: Challenges and Opportunities_ , PVLDB 2019 ([vldb.org](https://www.vldb.org/pvldb/vol12/p1986-nargesian.pdf)). The same problem in data management, years earlier: a pile of files with no schema is not queryable, and the fix is metadata written at ingestion.\n  * Scott Barnett et al., _Seven Failure Points When Engineering a Retrieval Augmented Generation System_ , 2024 ([arXiv 2401.05856](https://arxiv.org/abs/2401.05856)). Field-measured failure points from three production systems; the missing-document and wrong-document ones are what an index filter removes at the source.\n  * _Text-to-SQL over a structured index_ , LlamaIndex documentation ([docs.llamaindex.ai](https://docs.llamaindex.ai/en/stable/examples/index_structs/struct_indices/SQLIndexDemo/)). The typed-corpus path as a shipped open-source component, useful for seeing how small the query side gets once the table is there.\n\n\n* * *\n![](../images/body-249d024535392fb76067.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm Applications](https://towardsdatascience.com/tag/llm-applications/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)[Structured Output](https://towardsdatascience.com/tag/structured-output/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fone-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries%2F&title=One%20Document%20Type%2C%20a%20Million%20Files%3A%20Structured%20Extraction%20into%20the%20SQL%20Table%20RAG%20Queries)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fone-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries%2F&title=One%20Document%20Type%2C%20a%20Million%20Files%3A%20Structured%20Extraction%20into%20the%20SQL%20Table%20RAG%20Queries)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fone-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries%2F&text=One%20Document%20Type%2C%20a%20Million%20Files%3A%20Structured%20Extraction%20into%20the%20SQL%20Table%20RAG%20Queries)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/002.jpg)](https://towardsdatascience.com/multi-document-rag-a-folder-of-unrelated-pdfs-is-one-long-document-with-a-nested-outline/)\n## [Multi-Document RAG: A Folder of Unrelated PDFs Is One Long Document with a Nested Outline](https://towardsdatascience.com/multi-document-rag-a-folder-of-unrelated-pdfs-is-one-long-document-with-a-nested-outline/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #14B] - No shared fields means no index to build. One summary line per file plus each file’s own table of contents, and retrieval routes down two levels\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 22, 202616 min read\n  * [![](../images/003.jpg)](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/)\n## [Loop Engineering for Cross-References: When RAG Answers ‘see Section 7.2’ Instead of the Actual Answer](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #11] - When the first answer points elsewhere in the document, the pipeline loops back to fetch the linked context\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 6, 202620 min read\n  * [![](../images/004.jpg)](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n## [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 13, 202611 min read\n  * [![](../images/005.jpg)](https://towardsdatascience.com/retrieve-one-row-from-a-table-not-the-whole-table-row-level-chunks-for-rag/)\n## [Retrieve One Row from a Table, Not the Whole Table: Row-Level Chunks for RAG](https://towardsdatascience.com/retrieve-one-row-from-a-table-not-the-whole-table-row-level-chunks-for-rag/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #7sexies] - The unit of retrieval doesn’t have to be a page or a paragraph. When the corpus carries tables, each body row with its column headers is a chunk in its own right, and it’s often the one row the reader asked about\n[Kezhan Shi](https://towardsdatascience.com/author/kezhanshi/)\nAugust 21, 202613 min read\n  * [![](../images/006.png)](https://towardsdatascience.com/docling-the-document-alchemist/)\n## [Docling: The Document Alchemist](https://towardsdatascience.com/docling-the-document-alchemist/)\n[NLP](https://towardsdatascience.com/category/artificial-intelligence/nlp/)\nEffortless extraction of data and tables from PDF documents\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nSeptember 12, 202512 min read\n  * [![](../images/007.jpg)](https://towardsdatascience.com/zero-waste-agentic-rag-designing-caching-architectures-to-minimize-latency-and-llm-costs-at-scale/)\n## [Zero-Waste Agentic RAG: Designing Caching Architectures to Minimize Latency and LLM Costs at Scale](https://towardsdatascience.com/zero-waste-agentic-rag-designing-caching-architectures-to-minimize-latency-and-llm-costs-at-scale/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nReducing LLM costs by 30% with validation-aware, multi-tier caching\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nMarch 1, 202610 min read\n  * [![](../images/body-a18f4c36387322d732e9.jpg)](https://towardsdatascience.com/prompting-engineering-vs-rag-for-editing-resumes/)\n## [Prompt Engineering vs RAG for Editing Resumes](https://towardsdatascience.com/prompting-engineering-vs-rag-for-editing-resumes/)\n[LLM Applications](https://towardsdatascience.com/category/artificial-intelligence/llm-applications/)\nRunning a code-free comparison in Azure\n[Robert Etter](https://towardsdatascience.com/author/retter_42511/)\nJanuary 4, 202610 min read\n  * [![](../images/008.jpg)](https://towardsdatascience.com/one-rag-pipeline-four-very-different-pdfs-same-four-bricks-every-answer-typed-and-cited/)\n## [A Production RAG Pipeline in Action: Every Answer Typed and Cited](https://towardsdatascience.com/one-rag-pipeline-four-very-different-pdfs-same-four-bricks-every-answer-typed-and-cited/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #9B] - One call wires the four upgraded bricks together, run on a paper, a NIST standard, and a report with a broken TOC\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nJuly 17, 202625 min read\n  * [![](../images/009.jpg)](https://towardsdatascience.com/how-to-build-your-own-custom-llm-memory-layer-from-scratch/)\n## [How to Build Your Own Custom LLM Memory Layer from Scratch](https://towardsdatascience.com/how-to-build-your-own-custom-llm-memory-layer-from-scratch/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nStep-by-step guide to building autonomous memory retrieval systems\n[Avishek Biswas](https://towardsdatascience.com/author/neural.avb/)\nFebruary 4, 202614 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "One Document Type, a Million Files: Structured Extraction into the SQL Table RAG Queries | Towards Data Science",
    "description": "Enterprise Document Intelligence [Vol.1 #14C] - One hour with two people, six to ten fields, and the two signals that separate a real column from one that will break a filter later",
    "keywords": null,
    "author": "Angela and Kezhan Shi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "One Document Type, a Million Files: Structured Extraction into the SQL Table RAG Queries",
    "og:description": "Enterprise Document Intelligence [Vol.1 #14C] - One hour with two people, six to ten fields, and the two signals that separate a real column from one that will break a filter later",
    "og:url": "https://towardsdatascience.com/one-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-2.jpeg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Angela and Kezhan Shi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "16 min read",
    "article:published_time": "2026-08-25T06:00:01.694Z",
    "article:modified_time": "2026-08-25T06:00:01.694Z",
    "article:author": "https://towardsdatascience.com/author/angela.shi/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/c2c4925fbd5c/code/block_000.txt",
      "chars": 585
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-2.jpeg",
      "bytes": 171127,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/000.jpeg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-2.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-131-1024x572.png",
      "bytes": 136462,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/001.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-131-1024x572.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-133-1024x658.png",
      "bytes": 162802,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/c2c4925fbd5c/images/body-66d15da324dbbf109243.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-133-1024x658.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-130-1024x282.png",
      "bytes": 126794,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/c2c4925fbd5c/images/body-61376b80ece8b141d2ea.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-130-1024x282.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-134-1024x544.png",
      "bytes": 202279,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/c2c4925fbd5c/images/body-7142c790f6857b751655.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-134-1024x544.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-132-1024x521.png",
      "bytes": 163371,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/c2c4925fbd5c/images/body-fb81a936ed54384291e3.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-132-1024x521.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-136-1024x584.png",
      "bytes": 176754,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/c2c4925fbd5c/images/body-9c6d4cd70f21d2aff920.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-136-1024x584.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-135-1024x337.png",
      "bytes": 133222,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/c2c4925fbd5c/images/body-7eb27603846674dbc334.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-135-1024x337.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg",
      "bytes": 212225,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/c2c4925fbd5c/images/body-249d024535392fb76067.jpeg",
      "src": "https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/mixed_archive_pile_11952176_v3_card.jpg",
      "bytes": 160801,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/002.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/mixed_archive_pile_11952176_v3_card.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/bookmark_enNl3McVwSI_v3_card.jpg",
      "bytes": 143754,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/003.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/bookmark_enNl3McVwSI_v3_card.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/toll_plaza_aerial_34282750_card.jpg",
      "bytes": 258596,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/004.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/toll_plaza_aerial_34282750_card.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/museum_specimen_drawer_16566150_v3_card.jpg",
      "bytes": 165005,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/005.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/museum_specimen_drawer_16566150_v3_card.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/09/1_-M5Pq1pTepkZGSM4UKtP8Q.png",
      "bytes": 533398,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/006.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/09/1_-M5Pq1pTepkZGSM4UKtP8Q.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/Img-scaled-1.jpg",
      "bytes": 681586,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/007.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/Img-scaled-1.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/01/markus-winkler-XKKuY4ottJ0-unsplash-scaled-1.jpg",
      "bytes": 385887,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/c2c4925fbd5c/images/body-a18f4c36387322d732e9.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/01/markus-winkler-XKKuY4ottJ0-unsplash-scaled-1.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/pegboard_tools_909256_card.jpg",
      "bytes": 209097,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/008.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/pegboard_tools_909256_card.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/02/Article_Thumbnail2-1.jpg",
      "bytes": 69411,
      "reused": true,
      "local_path": "attachments/c2c4925fbd5c/images/009.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/02/Article_Thumbnail2-1.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/large-language-model/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/angela.shi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-rag-generation-an-llm-cascade-from-a-cheap-local-model-up-to-a-hosted-flagship/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/before-full-agentic-rag-know-how-you-decide-and-the-parsing-methods-you-pick-from/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-rag-question-parsing-the-small-loop-that-runs-before-retrieval/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/most-rag-hallucinations-are-extraction-errors-seven-patterns-for-a-typed-generation-contract/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-rag-the-small-loops-inside-each-step-the-big-loops-across-the-pipeline/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://www.pexels.com/photo/36681344/"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://github.com/doc-intel/notebooks-vol1"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/"
    },
    {
      "type": "external",
      "href": "https://medium.com/@angela.shi/enterprise-document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-1cd0a3154abf"
    },
    {
      "type": "external",
      "href": "https://medium.com/data-science-collective/context-engineering-for-enterprise-rag-the-minimal-four-brick-pipeline-that-highlights-its-source-4de8c6776ecd"
    },
    {
      "type": "external",
      "href": "https://medium.com/data-science-collective/embeddings-arent-magic-the-predictable-failure-modes-of-rag-retrieval-572620033db1"
    },
    {
      "type": "external",
      "href": "https://medium.com/@angela.shi/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem-a2fed3691e9f"
    },
    {
      "type": "external",
      "href": "https://medium.com/data-science-collective/from-regex-to-vision-models-which-rag-technique-fits-which-problem-5afb3f142279"
    },
    {
      "type": "external",
      "href": "https://medium.com/data-science-collective/10-common-rag-mistakes-we-keep-seeing-in-production-d0a2d4f45785"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2404.16130"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2305.03111"
    },
    {
      "type": "external",
      "href": "https://www.vldb.org/pvldb/vol12/p1986-nargesian.pdf"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2401.05856"
    },
    {
      "type": "external",
      "href": "https://docs.llamaindex.ai/en/stable/examples/index_structs/struct_indices/SQLIndexDemo/"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=One+Document+Type%2C+a+Million+Files%3A+Structured+Extraction+into+the+SQL+Table+RAG+Queries&u=https%3A%2F%2Ftowardsdatascience.com%2Fone-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=One+Document+Type%2C+a+Million+Files%3A+Structured+Extraction+into+the+SQL+Table+RAG+Queries&url=https%3A%2F%2Ftowardsdatascience.com%2Fone-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries%2F"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:26:14Z",
  "source_url": "https://towardsdatascience.com/one-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "http_403",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 18,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# One Document Type, a Million Files: Structured Extraction into the SQL Table RAG Queries\nEnterprise Document Intelligence [Vol.1 #14C] - One hour with two people, six to ten fields, and the two signals that separate a real column from one that will break a filter later\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 25, 202616 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-2.jpeg)\n[Photo](https://www.pexels.com/photo/36681344/) by Bor Jinson, via Pexels.\nA lot of RAG work right now goes into letting a model discover the structure of a document collection: read every file, pull out the entities, infer the relations, and a graph comes out that nobody had to design. On five thousand copies of the same contract, there is nothing left to discover.\nAsk the person who files those contracts what they look things up by. The answer takes about ten seconds: client, effective date, product, premium, renewal date, policy number.\nNobody found those six by reading the documents. They were designed years ago by whoever drew up the form, and every file since has been filled in against them. These files are the output of one repeatable process, and the process is what defines the fields.\nSo the collection is a database nobody typed. Its schema lives in the head of the person who handles the files, and the work is to get it out and write it into a table.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\nThis article assumes the diagnostic in Article 14A (three kinds of corpus, and what building for the wrong one costs) already came back with this shape: one document type, many copies. It covers what to do next:\n  * the interview that produces the columns, question by question\n  * the two signals that tell a real column from a field that will break a filter later\n  * the index as a table of contents for the collection, and why it keeps growing\n  * what gets paid once per document, and what gets paid on every question\n  * the arithmetic, at fifty questions a day, that says when the preparation pays for itself\n\n\nThis article is part of Part IV of [Enterprise Document Intelligence](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/), a series that builds an enterprise RAG system from four bricks. Part IV is where the input stops being a file and becomes a folder, and this is the folder shape where a table is the right answer.\n🧭 **New to the series? Start with the map** : [Prompt, Context, Loop](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/) sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-131-1024x572.png)\n_where this article sits: Part IV, the folder that becomes a table - Image by author_\n📓 **Run the column test on your own folder in the companion notebook** : declare the six columns, fill them across a folder of look-alike PDFs, then print the fill rate per column and watch two of the fields that sounded like columns come back half empty. Repo → [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1).\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-133-1024x658.png)\n_The public companion-code repo at doc-intel/notebooks-vol1 - Image by author_\nThe worked example is an insurance portfolio, and it is fictional: the clients, policy numbers and amounts were authored for the series and match no real insured or insurer. Section 3.2 gives a public folder you can run the same test on.\n## 1. One document type, and the columns that came with it\nTwo things to settle before the interview: what makes this shape recognisable, and where its columns come from.\n### 1.1. What the shape looks like on a shelf\nTwelve years of renewal notices for one motor product. Every purchase order a procurement team has issued since the ERP went in. The monthly statements of one fund. Each of those is thousands of files, and each file is the same document with different values in it.\nThe count is not what matters. A hundred purchase orders behave like a hundred thousand of them: the questions people ask are about _which ones_ , not about where inside one of them.\nTwo things follow. Documents that came out of one process share a page-one layout, so the same value sits in roughly the same place every time. And the values themselves repeat: eleven products, forty-odd clients, four statuses. A column here has a small, stable set of values, which is what makes it worth filtering on.\n### 1.2. The columns existed before the corpus did\nA field like `effective_date` does not come from the text. It comes from whoever designed the form. A template has slots, a form has boxes, a back-office screen has required fields, and the PDF is that screen printed on paper.\nNobody wrote that decision down. There is no data dictionary, and the person who designed the form left in 2014. What survives is the practice: the fields people fill in, and the ones they check before answering a customer. That knowledge sits with the handlers, so getting it out is a conversation rather than an extraction job.\nThis is where the shape parts company with the folder in Article 14B (a collection whose documents share no field at all). There, no such decision was ever made, so there is nothing upstream to recover.\n## 2. The interview that gets you the columns\nOne hour, two people, and a list of six to ten fields at the end of it.\n### 2.1. Who you ask, and in what order\nAsk the person who handles the documents, not the manager who owns the process. The manager describes the process as designed. The handler knows which field is blank on the older files and which one gets typed in wrong.\nAsk two handlers, separately. The second hour is there to catch one thing: a field they both name the same way and mean differently.\nFour questions, in this order. Each one is easier to answer once the previous one has been.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-130-1024x282.png)\n_four questions, and what each answer settles about the schema - Image by author_\nWhat comes out is a list of names in the handler’s own words. Types and definitions come later.\n### 2.2. What tells you a column is real\nTwo signals, and both are about the answer rather than about the field.\n**It comes back without a pause.** Ask what someone filters on and a real column arrives at conversational speed, with an example attached: _“client, obviously, and the effective date, because that is how we know which version applies.”_ Take a field that needs thinking about. Nobody filters on it today, so nobody will keep it up to date either.\n**Two handlers name it the same way.** Same word, same meaning, same box on the page. The first signal cannot do this on its own, because a field can be named instantly by both handlers and still mean two different things to them.\nNeither signal needs a model or a sample. Both are settled in the room.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-134-1024x544.png)\n_three fields the business named, and the two that are not columns - Image by author_\n### 2.3. Three answers that are not columns\n**Two meanings in one word.** One handler says _date_ and means the day the contract was signed. The other says _date_ and means the day the cover starts. Both answered fast, both used the same word, and the column that comes out of it holds two quantities mixed together. A query for policies in force in March then drops the rows carrying a signature date, and what comes back still looks like an answer. A missing field is safer, because a filter trusts what is there.\n**Empty on half the documents.** A field filled on 1,900 rows out of 5,000 is a subset with no name. Sometimes the subset is real, and then say so: the field belongs to a subtype, and the subtype is the column you were missing. Sometimes it is just a field that arrived in 2021, which is fine, as long as the filter reads a null as _not recorded yet_ rather than as _no_.\n**A judgement rather than a fact.** _“Is this client difficult?”_ is real knowledge and it is not printed anywhere. Ask a model to infer it and you get an opinion with a citation attached. Park those fields. Some come back later as something the business writes into the index by hand, which is not the same thing as extraction.\nThere is an alternative to that hour, and it is popular for good reasons. Point a model at the corpus, let it pull out whatever entities and relations it finds, and a vocabulary comes back that nobody had to specify. On a collection where no business user can name a field, that is the only move available.\nHere the trade is different. You pay a full extraction pass over five thousand documents to rediscover a list two people produce in an hour, and the vocabulary that comes back is one the business did not choose and cannot correct in place. Article 16 (where the corpus vocabulary comes from, curated against extracted) puts the two side by side on cost, audit trail and freshness. The ordering is what matters here. Ask first, and if the answer arrives in ten seconds, there is nothing left to discover.\n## 3. The index is a table of contents for the collection\nThe index has one job, and the quickest way to see it is to put it beside something every reader already uses.\n### 3.1. A map read before the content\nA table of contents is not part of a document’s content, it is a map of it. Six lines stand in for thirty-four pages, and a question about exclusions lands on section 4 before a page is read.\nThe index does the same job one level up. It is not the content of the corpus either. Five columns stand in for five thousand files, and a question about one client’s fleet cover lands on one row before a file is opened.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-132-1024x521.png)\n_the same job at two scales: pick the section, pick the documents - Image by author_\nFollow that comparison all the way, because it tells you what belongs in the index. A table of contents does not hold the text of section 4, it holds enough to decide whether section 4 is where to look. An index row holds enough to decide whether a document is a candidate. Once it starts holding answers it is a second copy of the corpus, and the second copy drifts.\nRebuilding the map when a document ships none is a parsing job: [Building Document Structure with Loop Engineering](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/) recovers an outline from body typography for a PDF that never encoded one.\n### 3.2. The row, written down\nSix fields, in the handler’s words, with types on them.\npython\n\n```\nclass PolicyIndexRow(BaseModel):    \"\"\"One row of the corpus index: one policy, six business fields.\"\"\"    doc_id: str              # the file, so an answer can cite where it came from    policy_number: str       # what the business types into its own systems    client_name: str         # the party the policy is written for    product: str             # one of eleven, and the list never changes    effective_date: date     # the day the cover starts    renewal_date: date       # the day it lapses if nobody acts    premium_eur: float       # the number the finance team filters on\n```\n\nNothing in that declaration is a modelling decision. Every name came out of the interview. `doc_id` is the one field the business did not name, and it is there so an answer can point back at the file it was read from.\nWhether the table stays this flat once amendments and versions arrive is the subject of Article 15A (the schema a corpus gets written into), which replaces the single sheet with three tables and two date columns. How each field gets filled comes with the articles after it.\nTake a folder of arXiv papers if you want to try the shape without an insurance portfolio. Same thing: one document type, thousands of copies, and the first three columns come free. A file named `1706.03762v7.pdf` gives you the identifier, the version, and the submission month, all three read off the filename before a page is parsed. The primary category is a fourth column and it costs one lookup, because the modern identifier does not encode it: it comes from the arXiv metadata. Then add _the task_ and _the datasets used_ , and the line between a printed field and a judgement shows up within a dozen files. The papers carry the arXiv non-exclusive distribution license declared on each [abstract page](https://arxiv.org/).\n### 3.3. The index keeps growing, and that is the design\nA question turns up that the columns cannot answer. _Which of these policies carry a war exclusion?_ There is no such column, so the question falls back to opening documents one at a time.\nAdd the column, fill it on the documents it applies to, and every later question about war exclusions is a filter instead of a corpus scan. The cost is paid once and the benefit does not expire. On the naive path the same question pays full price every time somebody asks it.\nTwo habits keep that growth honest. Run each new field through the two signals from section 2.2 first, because a column added under pressure from one urgent question is the kind that means three things a year later. And keep a citation on every value, so that a number in the index can be checked against the page it was read from.\n## 4. Paid once, or paid on every question\nWhether the preparation is worth its price is arithmetic. Here it is.\n### 4.1. Two phases, and only one of them grows with the questions\nThe single-document pipeline does not change. It now sits inside two phases that cost very different things.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-136-1024x584.png)\n_the two phases, and the index they share - Image by author_\nIngestion runs once per document, when the file arrives and before anyone has asked anything. It parses, fills the agreed columns, keeps a page reference for each value, and appends a row. It runs again only when the file changes.\nQuerying runs once per question. It reads the scope fields out of the question, filters the index down to the candidates, and runs the pipeline on what survives. Article 17 (querying a corpus, filter first and retrieve second) builds that half.\nIngestion is bounded by the number of documents. Querying is bounded by the number of questions, and that is the number that keeps going up.\n### 4.2. Fifty questions a day, and what they add up to\nA claims handler uses a system like this about fifty times a day. That is the order of magnitude we keep seeing, and the exact number does not change what follows. Over a working year it is roughly twelve thousand questions from one person, and a team of six puts it past seventy thousand.\nWithout preparation, every one of those questions is priced against the whole shelf. The system works out from scratch which documents are candidates, and it does that by comparing the question against passages from files that had no business being consulted.\nWith preparation, the question opens with a filter over a table and the pipeline runs on what comes through. The work per question stops tracking the size of the corpus.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-135-1024x337.png)\n_what is paid once per document, and what is paid on every question - Image by author_\nIngestion is paid once per document and querying once per question. So as soon as the number of questions over the system’s life reaches the same order as the number of documents, the prepared version is ahead on compute. It was ahead earlier than that on two other counts: the answer comes from a document somebody chose, and the filter is something a business user can read and argue with.\n### 4.3. When the preparation is not worth it\nA document that gets read once, or never, does not justify a deep pass. Filling twelve columns on a one-off appendix costs more than reading the appendix when the question finally arrives, if it ever does.\nSo tier the work. A standard policy template consulted thousands of times over a portfolio’s life justifies a deep ingestion, with every field extracted and every cross-reference resolved. The long tail gets the cheap version, which here is the filename, the file metadata and the document type. A document can be promoted later, when its query count says it earned the deeper pass.\nThat is the same decision the generation brick makes about models: start cheap, and pay for the expensive path only where it changes the outcome. [Loop engineering for RAG generation](https://towardsdatascience.com/loop-engineering-for-rag-generation-an-llm-cascade-from-a-cheap-local-model-up-to-a-hosted-flagship/) works that cascade out on the model side.\n## 5. Conclusion\nOn a collection of one document type there is no schema to discover. There is a schema to write down, and it already exists in the practice of the people who handle the documents.\nThe preparation starts with an hour of conversation and two signals: a field named without a pause, and two people naming it the same way. What passes both goes into the index. What fails is a subtype, a judgement, or a word that means two things, and each of those is cheaper to catch in the room than after a pass over five thousand files.\nTry the two signals on a folder of your own before writing any ingestion code. Ask the person who handles those files what they filter on, then ask a second one, and see how much of the schema comes back in the first ten seconds.\nArticle 14B (the folder whose documents share no field) covers the collection with nothing to put in the columns. Article 14D (case files, one bundle about a single entity) covers the one where a row of the table is a case rather than a document.\n## 6. Further reading and sources\n**Earlier in the series:**\n  * [Document Intelligence: series intro](https://medium.com/@angela.shi/enterprise-document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-1cd0a3154abf). What the series builds, brick by brick, and in what order.\n\n\n_What works, what breaks_\n  * [Baseline Enterprise RAG, from PDF to highlighted answer](https://medium.com/data-science-collective/context-engineering-for-enterprise-rag-the-minimal-four-brick-pipeline-that-highlights-its-source-4de8c6776ecd). The four-brick pipeline end to end: PDF in, highlighted answer out.\n  * [Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval](https://medium.com/data-science-collective/embeddings-arent-magic-the-predictable-failure-modes-of-rag-retrieval-572620033db1). Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway.\n  * [RAG is not machine learning, and the ML toolkit solves the wrong problem](https://medium.com/@angela.shi/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem-a2fed3691e9f). Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead.\n  * [From regex to vision models: which RAG technique fits which problem](https://medium.com/data-science-collective/from-regex-to-vision-models-which-rag-technique-fits-which-problem-5afb3f142279). Two axes, document complexity and question control, that pick the technique for each case.\n    * [10 common RAG mistakes we keep seeing in production](https://medium.com/data-science-collective/10-common-rag-mistakes-we-keep-seeing-in-production-d0a2d4f45785). Ten production mistakes, organized brick by brick, with the fix for each.\n\n\n_Document parsing_\n  * [Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/). Rebuilding the outline from body typography when the PDF ships no contents page at all: six signals, one bounded loop.\n  * [Before Full Agentic RAG: Know How You Decide, and the Parsing Methods You Pick From](https://towardsdatascience.com/before-full-agentic-rag-know-how-you-decide-and-the-parsing-methods-you-pick-from/). The parsing methods as a catalogue, and the decision of which to run, before handing the loop to an agent.\n\n\n_Question parsing_\n  * [Loop engineering for RAG question parsing: the small loop that runs before retrieval](https://towardsdatascience.com/loop-engineering-for-rag-question-parsing-the-small-loop-that-runs-before-retrieval/). The bounded loop that runs before retrieval: parse, check the fields make sense, re-ask once when they do not.\n\n\n_Generation_\n  * [Loop Engineering for RAG Generation: Iterate top-k One at a Time](https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/). Reading the retrieved pages one at a time instead of all at once, and what that buys when the answer sits in only one of them.\n  * [Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract](https://towardsdatascience.com/most-rag-hallucinations-are-extraction-errors-seven-patterns-for-a-typed-generation-contract/). Seven recurring ways a model gets the extraction wrong, and the typed contract that catches each one.\n  * [Loop engineering for RAG generation: an LLM cascade from a cheap local model up to a hosted flagship](https://towardsdatascience.com/loop-engineering-for-rag-generation-an-llm-cascade-from-a-cheap-local-model-up-to-a-hosted-flagship/). Starting on a cheap local model and escalating only when the answer does not hold up, measured.\n\n\n_One-document pipelines_\n  * [Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations](https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/). Why a better prompt does not fix a wrong page, and what each of the four bricks contributes to the context instead.\n  * [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/). Cutting a pipeline’s latency and cost by calling the model less often and cheaper, not by buying a faster one.\n  * [Loop engineering for cross-references: when RAG answers ‘see Section 7.2’ instead of the actual answer](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/). When the answer says “see Section X”, the pipeline loops back and fetches it.\n  * [RAG workflow and loop engineering: the dispatcher that decides when to loop and when to stop](https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/). Feedback loops, bounded iteration, and the dispatcher, composed into one workflow.\n    * [Loop engineering for RAG: the small loops inside each step, the big loops across the pipeline](https://towardsdatascience.com/loop-engineering-for-rag-the-small-loops-inside-each-step-the-big-loops-across-the-pipeline/). The two scales of loop: small bounded loops inside each brick, big generation-triggered loops across them.\n\n\nAlso referenced above, listed here without links: Article 14A (three kinds of corpus, and what building for the wrong one costs), Article 14B (the folder whose documents share no field), Article 14D (case files, one bundle about a single entity), Article 15 (preparing a corpus, one column at a time), Article 16 (where the corpus vocabulary comes from, curated against extracted), and Article 17 (querying a corpus, filter first and retrieve second).\nExternal sources:\n  * Darren Edge et al., _From Local to Global: A Graph RAG Approach to Query-Focused Summarization_ , 2024 ([arXiv 2404.16130](https://arxiv.org/abs/2404.16130)). The opposite bet, stated properly: let a model extract the entities instead of asking the business to name the columns. Read it for the cases where discovery is the only option.\n  * Jinyang Li et al., _Can LLM Already Serve as A Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs_ , 2023 ([arXiv 2305.03111](https://arxiv.org/abs/2305.03111)). What a question over a typed table looks like once the table exists, measured on 95 real databases.\n  * Fatemeh Nargesian et al., _Data Lake Management: Challenges and Opportunities_ , PVLDB 2019 ([vldb.org](https://www.vldb.org/pvldb/vol12/p1986-nargesian.pdf)). The same problem in data management, years earlier: a pile of files with no schema is not queryable, and the fix is metadata written at ingestion.\n  * Scott Barnett et al., _Seven Failure Points When Engineering a Retrieval Augmented Generation System_ , 2024 ([arXiv 2401.05856](https://arxiv.org/abs/2401.05856)). Field-measured failure points from three production systems; the missing-document and wrong-document ones are what an index filter removes at the source.\n  * _Text-to-SQL over a structured index_ , LlamaIndex documentation ([docs.llamaindex.ai](https://docs.llamaindex.ai/en/stable/examples/index_structs/struct_indices/SQLIndexDemo/)). The typed-corpus path as a shipped open-source component, useful for seeing how small the query side gets once the table is there.\n\n\n* * *\n![](https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm Applications](https://towardsdatascience.com/tag/llm-applications/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)[Structured Output](https://towardsdatascience.com/tag/structured-output/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fone-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries%2F&title=One%20Document%20Type%2C%20a%20Million%20Files%3A%20Structured%20Extraction%20into%20the%20SQL%20Table%20RAG%20Queries)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fone-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries%2F&title=One%20Document%20Type%2C%20a%20Million%20Files%3A%20Structured%20Extraction%20into%20the%20SQL%20Table%20RAG%20Queries)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fone-document-type-a-million-files-structured-extraction-into-the-sql-table-rag-queries%2F&text=One%20Document%20Type%2C%20a%20Million%20Files%3A%20Structured%20Extraction%20into%20the%20SQL%20Table%20RAG%20Queries)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/mixed_archive_pile_11952176_v3_card.jpg)](https://towardsdatascience.com/multi-document-rag-a-folder-of-unrelated-pdfs-is-one-long-document-with-a-nested-outline/)\n## [Multi-Document RAG: A Folder of Unrelated PDFs Is One Long Document with a Nested Outline](https://towardsdatascience.com/multi-document-rag-a-folder-of-unrelated-pdfs-is-one-long-document-with-a-nested-outline/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #14B] - No shared fields means no index to build. One summary line per file plus each file’s own table of contents, and retrieval routes down two levels\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 22, 202616 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/bookmark_enNl3McVwSI_v3_card.jpg)](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/)\n## [Loop Engineering for Cross-References: When RAG Answers ‘see Section 7.2’ Instead of the Actual Answer](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #11] - When the first answer points elsewhere in the document, the pipeline loops back to fetch the linked context\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 6, 202620 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/toll_plaza_aerial_34282750_card.jpg)](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n## [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 13, 202611 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/museum_specimen_drawer_16566150_v3_card.jpg)](https://towardsdatascience.com/retrieve-one-row-from-a-table-not-the-whole-table-row-level-chunks-for-rag/)\n## [Retrieve One Row from a Table, Not the Whole Table: Row-Level Chunks for RAG](https://towardsdatascience.com/retrieve-one-row-from-a-table-not-the-whole-table-row-level-chunks-for-rag/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #7sexies] - The unit of retrieval doesn’t have to be a page or a paragraph. When the corpus carries tables, each body row with its column headers is a chunk in its own right, and it’s often the one row the reader asked about\n[Kezhan Shi](https://towardsdatascience.com/author/kezhanshi/)\nAugust 21, 202613 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/09/1_-M5Pq1pTepkZGSM4UKtP8Q.png)](https://towardsdatascience.com/docling-the-document-alchemist/)\n## [Docling: The Document Alchemist](https://towardsdatascience.com/docling-the-document-alchemist/)\n[NLP](https://towardsdatascience.com/category/artificial-intelligence/nlp/)\nEffortless extraction of data and tables from PDF documents\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nSeptember 12, 202512 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/03/Img-scaled-1.jpg)](https://towardsdatascience.com/zero-waste-agentic-rag-designing-caching-architectures-to-minimize-latency-and-llm-costs-at-scale/)\n## [Zero-Waste Agentic RAG: Designing Caching Architectures to Minimize Latency and LLM Costs at Scale](https://towardsdatascience.com/zero-waste-agentic-rag-designing-caching-architectures-to-minimize-latency-and-llm-costs-at-scale/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nReducing LLM costs by 30% with validation-aware, multi-tier caching\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nMarch 1, 202610 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/01/markus-winkler-XKKuY4ottJ0-unsplash-scaled-1.jpg)](https://towardsdatascience.com/prompting-engineering-vs-rag-for-editing-resumes/)\n## [Prompt Engineering vs RAG for Editing Resumes](https://towardsdatascience.com/prompting-engineering-vs-rag-for-editing-resumes/)\n[LLM Applications](https://towardsdatascience.com/category/artificial-intelligence/llm-applications/)\nRunning a code-free comparison in Azure\n[Robert Etter](https://towardsdatascience.com/author/retter_42511/)\nJanuary 4, 202610 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/pegboard_tools_909256_card.jpg)](https://towardsdatascience.com/one-rag-pipeline-four-very-different-pdfs-same-four-bricks-every-answer-typed-and-cited/)\n## [A Production RAG Pipeline in Action: Every Answer Typed and Cited](https://towardsdatascience.com/one-rag-pipeline-four-very-different-pdfs-same-four-bricks-every-answer-typed-and-cited/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #9B] - One call wires the four upgraded bricks together, run on a paper, a NIST standard, and a report with a broken TOC\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nJuly 17, 202625 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/02/Article_Thumbnail2-1.jpg)](https://towardsdatascience.com/how-to-build-your-own-custom-llm-memory-layer-from-scratch/)\n## [How to Build Your Own Custom LLM Memory Layer from Scratch](https://towardsdatascience.com/how-to-build-your-own-custom-llm-memory-layer-from-scratch/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nStep-by-step guide to building autonomous memory retrieval systems\n[Avishek Biswas](https://towardsdatascience.com/author/neural.avb/)\nFebruary 4, 202614 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "body_base_path": "attachments/c2c4925fbd5c/pages/",
  "body_markdown_path": "attachments/c2c4925fbd5c/pages/archived-body.md",
  "local_html": "attachments/c2c4925fbd5c/pages/archived-body.html",
  "body_source": "prior_capture"
}