{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Angela Shi & Kezhan Shi",
  "title_listed": "Parse the Folder, Not Just the PDFs: The Relational Tables RAG Needs on a Case File",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/parse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file/",
  "publish_date": "2026-08-23",
  "corpus_stage": "",
  "notes": "",
  "id": "58b903da6998",
  "collected_at": "2026-10-02T01:02:30",
  "title": "Parse the Folder, Not Just the PDFs: The Relational Tables RAG Needs on a Case File | Towards Data Science",
  "body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# Parse the Folder, Not Just the PDFs: The Relational Tables RAG Needs on a Case File\nEnterprise Document Intelligence [Vol.1 #14D] - The index lists what the case type demands before any folder is opened, and the two questions worth building for are not retrieval questions at all\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 23, 202620 min read\n![](../images/000.jpg)\n[Photo](https://www.pexels.com/photo/34293529/) by Zulfugar Karimov, via Pexels.\nContext windows keep getting longer, and the advice that comes with them is simple: stop retrieving, put the whole thing in the prompt.\nOn a case file that advice nearly works. Eleven PDFs about one insurance claim come to sixty-odd pages, which fits inside a 200,000-token window several times over.\nIt still misses the two questions the person handling the claim has. Is the second repair invoice in the folder at all, and does the date of loss on the claim form match the one in the adjuster’s report?\nThe reason is in the shape of the thing. A case file is a bundle of unlike documents about one entity, and the unit of work is the bundle. A claim, a credit application, a medical record, a hiring file. The contract, the certificate, the photographs, the expert’s report, the letters: nothing in there resembles anything else, and the answer comes from reading across all of it.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\nSo neither question is a search. The first is about a document that does not exist, and no amount of context holds it. The second needs one value out of each of two files, compared. Models read the middle of a long input less reliably than the ends, and that has been measured. In a bundle of eleven, both values are in the middle.\nThis article works through what that changes:\n  * what separates a case file from a collection that happens to be small\n  * the list of pieces a case type expects, which indexes what should be there rather than what is\n  * why a missing document is a first-class answer, and what it takes to produce one\n  * the checks that compare one piece against another, and the typed values they need first\n  * the shape of the answer, which is the state of the case rather than a passage\n  * why this shape has the least prior art of the three, and how to evaluate it anyway\n\n\nThis article is part of Part IV of [Enterprise Document Intelligence](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/), a series that builds an enterprise RAG system from four bricks. Part IV is where the input stops being a file and becomes a folder, and this is the folder that behaves like one object rather than like a collection.\n🧭 **New to the series? Start with the map** : [Prompt, Context, Loop](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/) sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.\n![](../images/body-238f6cfcab49a64eafb6.png)\n_where this article sits: Part IV, the folder that is one case - Image by author_\n📓 **Run the completeness check on a folder of your own in the companion notebook** : write down the pieces one case type expects, point the check at a folder, and print the three lists that come back: the pieces present, the pieces missing, and the files that belong to some other case. Repo → [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1).\n![](../images/body-e236ef9d3ff984fa15a3.png)\n_The public companion-code repo at doc-intel/notebooks-vol1 - Image by author_\nThe worked example is a fire claim on a small joinery workshop, and it is fictional: the workshop, the dates, the amounts and the files were authored for the series and match no real insured or insurer. The list of expected pieces is written the way a broker’s back office would write one, not copied from anybody’s manual.\n## 1. A bundle is not a small corpus\nOne folder first, then the comparison that says why the other two architectures do not fit it.\n### 1.1. Eleven files about one fire\nA fire damages a joinery workshop on 12 March. Over the next six weeks the broker’s folder for that claim fills up with eleven PDFs: the claim form, the policy schedule in force that month, a certificate of premium payment, the loss adjuster’s report, two repair quotes, one repair invoice, a set of damage photographs, two letters, and a vehicle schedule that belongs to a different policy and was filed here by mistake.\nNothing in that list resembles anything else in it. The photographs carry no text. The adjuster’s report is prose. The quotes are tables. The claim form is boxes with values typed into them.\nAsk the handler what she does first and she does not search. She runs down a list she knows by heart and ticks off what is there. Fire claim, so the fire brigade report is required, and it is not in the folder. Two quotes accepted, so two invoices should follow, and only one has. That is a couple of minutes of work, and no document has been read yet.\nOnly after that does she read. The date of loss on the claim form says 12 March. The adjuster writes that he inspected the site on 3 March. One of those two is wrong, and until somebody says which, the file does not move.\n### 1.2. What the other two shapes give you, and what they cost here\nArticle 14A (three kinds of corpus, and what building for the wrong one costs) sets out three shapes a collection can take and a three-question test that tells you which one you are holding. This is the third shape: documents that arrive in bundles, several of them about one case.\nPut the three side by side, question by question.\n![](../images/body-988aa17754d3d5fe8fba.png)\n_five questions, three shapes, and where the bundle parts company - Image by author_\nTwo rows matter most here.\nThe index row. Article 14C (one document type, many copies, and the columns the business can already name) builds a table with one row per document and the business fields on it. That table describes what exists. Here the row is the case, and what goes in it is a list of pieces the case is supposed to hold, which is a different thing to write down and a different thing to fill.\nThe empty result. On the other two shapes, nothing found means the retrieval step failed and you go and fix it. On a case file, nothing found can be the correct answer, and often it is the answer the question was after.\nThe remaining comparison is the one that catches people out, because from a distance the two look identical. Article 14B (the folder whose documents share no field, indexed by one summary line each) treats a heterogeneous folder as one long document and routes down its nested outline. A case file is heterogeneous in exactly that way, so the routing runs fine here. It answers the wrong question: routing picks a piece, and these questions are about the set.\n## 2. The list of pieces a case type expects\nThe preparation for this shape is one short artefact: a list of the pieces the case type demands, with how many of each and under what condition.\nThis is parsing, one floor up. On a single PDF, the parsing brick returns a small set of relational tables, one row per line, one row per section, one row per table cell (Article 5B, _the relational tables RAG needs_). A folder parses the same way: one table of expected roles, one table of arrived files with the role each was assigned and a confidence on that assignment. Most of the work on a case file is here, in structuring and cleaning the bundle into those two tables. The completeness check, the contradiction check and the case state later in this article are all queries over them.\n### 2.1. An index of what should be there\nA list of what should be there, rather than a description of what is. The index in Article 14C describes documents that exist; this list names documents that ought to, and the two are read at different moments.\nAn index built by reading the corpus describes what the corpus contains. Point an extraction pass at five thousand contracts and you get a table with one row per contract and a client, a date and a premium on each. Ask it which contract is missing and the question does not parse. Everything in the table is there by construction.\nThe list here is written before the folder is opened, and it comes from the process rather than from the files. A property damage claim needs a claim form, a policy schedule, proof that the premium was paid, an adjuster’s report, at least one quote, an invoice for each accepted quote, photographs, and a fire brigade report when the cause is fire. That list holds for the case type whether or not a single one of those documents was ever filed.\nDerive the same list from the folders you already have and the missing piece disappears. Every role in such a list is a role that showed up in at least one folder, so a piece nobody ever files is simply not in the list.\n### 2.2. What a row of the list holds\nFour fields per piece: what it is, how many are expected, when it is required, and whether the case can move without it.\npython\n\n```\n# Written from the process, before any folder is opened.class ExpectedPiece(BaseModel):    \"\"\"One row of what a case type demands, filed or not.\"\"\"    role: str                  # the business word: \"loss adjuster report\"    min_count: int             # 0 when the piece is optional    max_count: int | None      # None when any number of them is fine    required_when: str | None  # a condition on the case's own fields    blocks_payment: bool       # can the case move on while this is absentPROPERTY_DAMAGE_CLAIM = [    ExpectedPiece(role=\"claim form\", min_count=1, max_count=1,                  required_when=None, blocks_payment=True),    ExpectedPiece(role=\"fire brigade report\", min_count=1, max_count=1,                  required_when=\"cause_of_loss == 'fire'\",                  blocks_payment=True),    ExpectedPiece(role=\"repair invoice\", min_count=1, max_count=None,                  required_when=\"accepted_quotes >= 1\",                  blocks_payment=True),    ExpectedPiece(role=\"correspondence\", min_count=0, max_count=None,                  required_when=None, blocks_payment=False),]\n```\n\n`required_when` is what stops the list producing false alarms. The fire brigade report is required on a fire and irrelevant on water damage, so a flat list of required pieces complains on every claim that is not a fire. A check that is wrong most of the time gets switched off. The condition reads fields off the case itself, which means the case has to carry a handful of fields of its own before the completeness check can run at all.\n`blocks_payment` is what makes the output usable. Two pieces are missing and one of them stops the money: a handler needs to see which, and a boolean per piece is enough to say it.\nWhere the list comes from is a conversation, and a short one. Somebody in the back office already has it, on a laminated card or in a procedures document, because a person has to check the same thing by hand today. Article 14C asks the business which fields it filters on. Here the question is narrower: what has to be in the folder before you sign this off. The answer usually comes back as a list.\n### 2.3. Matching the folder against the list\nWith the list on one side and eleven files on the other, the completeness check is a join with three outcomes.\n![](../images/body-db1eb0c2478395df5119.png)\n_nine expected pieces, eleven files, and the three outcomes of the join - Image by author_\nA piece is **present** when at least `min_count` files were assigned to its role. Seven of the nine roles pass here.\nA piece is **missing** when the role is required, its condition holds, and no file was assigned to it. Two roles fail: the fire brigade report, because the cause is fire and nothing was filed, and the second repair invoice, because two quotes were accepted and one invoice arrived.\nA file is **unmatched** when no role in the list fits it. Here that is the vehicle schedule from another policy. The outcome is worth reporting rather than dropping, because it is either a filing mistake or a role the list forgot.\nThe join rests on one step that can go wrong. Assigning a role to each file is a classification, and classifications make mistakes. An invoice filed under quotes reads as a missing invoice plus an extra quote, so the report then carries two wrong lines instead of none. The role assignment therefore keeps its own confidence, and a low-confidence assignment goes to the handler as a question rather than as a verdict.\n## 3. Two questions retrieval cannot answer\nBoth of the questions the handler asked in section 1.1 have the same property: no passage anywhere in the bundle answers them, so no amount of ranking gets you there.\n### 3.1. The missing piece\nAsk a document-by-document pipeline where the fire brigade report is and it returns an empty result. Ask it the same thing on a claim where the report is filed under a strange name and it also returns an empty result. Those are different situations and the pipeline cannot tell them apart, because an empty result carries no information about why it is empty.\nThe completeness check can, and that is what the list is for. It knows the role exists, it knows the condition holds, and it knows nothing was assigned. The answer it produces is a sentence with a reason attached: the fire brigade report is required because the cause of loss is fire, no file in the folder was assigned to that role, and payment is blocked until one is.\nGetting there needs three things the pipeline does not have by default. A list of roles, so absence has a name. A condition on each role, so absence is only claimed when the piece was actually due. And a confidence on the role assignment, so a badly filed document is reported as doubt rather than as a categorical no.\nThat last one matters more than it looks. A report that says _missing_ when the document is sitting in the folder under a bad name is worse than no report, because the handler then asks the client for a file the client already sent. The series already has this discipline one level down, on the generation side. A typed answer contract carries not-found as a value rather than as an empty string, which is what [the seven extraction patterns](https://towardsdatascience.com/most-rag-hallucinations-are-extraction-errors-seven-patterns-for-a-typed-generation-contract/) work out on a single document. Here the same move runs one floor up, on the bundle instead of on the field.\n### 3.2. The contradiction between two pieces\nThe second question compares. The date of loss appears on the claim form and again in the adjuster’s report, and the two do not agree.\nTop-k retrieval is the wrong instrument here, and not because of its quality. Ranking returns the passages most like the question. A contradiction is not like anything: both halves read as ordinary on their own, neither passage mentions the other, and nothing in either one is closer to the question than the rest of the file.\nWhat works instead is mechanical. Declare which fields are worth comparing and between which roles, extract that field from each piece with a typed contract and a citation, then compare the values.\n![](../images/body-29474212259c726a966e.png)\n_five declared pairs, the two values read for each, and what a mismatch means - Image by author_\nThree details decide whether this works in practice.\n**The pairs are declared, not discovered.** Nine pieces make thirty-six pairs, and multiplied by every field on every piece the comparison space is large and mostly pointless. What people actually check is a handful of pairs per case type, and a business user can name them in the same conversation that produced the list of expected pieces.\n**The values are typed before they are compared.** `12/03/2025` and `March 12, 2025` are the same date and two different strings. `Alpine Joinery SARL` and `Alpine Woodworks` are two different strings that may or may not be the same company. The comparison runs on parsed values, and the second case, where normalisation is a judgement rather than a format, is the one that needs a person in the loop. Turning a raw extraction into something joinable is its own piece of work, and Article 15 (preparing a corpus, one column at a time) takes it up.\n**Every value keeps its citation.** A report that only says the dates disagree gives the handler nothing to go and check. One that says the claim form gives 12 March on page 1 and the report gives 3 March on page 4 can be settled in ten seconds.\n## 4. The answer is the state of the case\nAsk a corpus a question and a good answer names documents. Ask a case file and a good answer describes the case. That answer is a composed object rather than a passage.\n### 4.1. What the object holds\nFour lists and a verdict. The lists are what section 2 and section 3 produce; the verdict is the sentence a handler would write at the top of the file.\npython\n\n```\nclass FieldConflict(BaseModel):    \"\"\"One declared pair that came back with two values.\"\"\"    field: str                 # \"date_of_loss\"    left: TypedValue           # value, doc_id, page, quoted line    right: TypedValueclass CaseState(BaseModel):    \"\"\"What a question about a case returns: the whole bundle.\"\"\"    case_id: str    case_type: str                  # \"property damage claim\"    present: list[MatchedPiece]     # role, doc_id, match confidence    missing: list[MissingPiece]     # role, why it was due, what it blocks    unmatched: list[str]            # files no role in the list fits    conflicts: list[FieldConflict]  # the declared pairs that disagree    verdict: Literal[\"complete\", \"incomplete\", \"conflicting\"]    verdict_reason: str             # one sentence naming what blocks\n```\n\nTwo properties of that object matter for how it gets filled.\nIt is a list-shaped answer, not a top-ranked one. Every entry counts and the ordering does not, which is the same contract a listing question needs inside one document. [Loop engineering for listing questions](https://towardsdatascience.com/loop-engineering-for-listing-questions-when-the-answer-is-every-passage-not-the-top-one/) works that out on a single PDF, and the constraint carries over to a bundle unchanged.\nIt is filled piece by piece rather than in one call. Eleven documents do fit in a window, and putting them all there is still the worse option: the extraction is per piece, the citation has to name the piece it came from, and reading one document at a time is what keeps both of those true. That trade, one call over everything against one call per item, is the subject of [iterating top-k one at a time](https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/) on a single document.\nThe object needs one more thing, and the series already has it. A piece can point at another piece: the adjuster’s report says _see the accepted quote_ , and the value being compared is in the other file. That is the same fetch-and-continue loop as [a document that answers ‘see Section 7.2’](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/), run across files instead of across sections.\n### 4.2. Where it breaks\nOf the three shapes, this is the one with the least prior art, both in the series and in the retrieval literature. So the failures below come with fewer known fixes than the ones in the other two articles.\nThe last row is the important one. Retrieval benchmarks measure whether the right passage came back. There is no standard set that measures whether a system correctly said a required document was never filed. The closest thing in the literature is the unanswerable half of a reading comprehension set, which is a different question asked of a single passage. So the evaluation stays local: take fifty cases a handler already checked by hand, run the completeness check, and count the two error types separately. A piece wrongly called missing costs a phone call to the client. A missing piece not caught means the claim is settled on an incomplete file, which is the thing the check exists to prevent.\nThree pieces of the build are not in this article. The index that holds the case and its fields is Article 15 (preparing a corpus, one column at a time). The vocabulary that says a _loss adjuster report_ and an _expert report_ are the same role is Article 16 (where the corpus vocabulary comes from). Routing a user’s question to the right case, and to the right mode once it is there, is Article 17 (querying a corpus, filter first and retrieve second).\n## 5. Conclusion\nA case file is neither a document nor a corpus. It is a bundle of unlike pieces about one entity, and the unit of work is the bundle.\nThe move that carries over from the single-document articles is relational. Parse the bundle into tables first, and every later question becomes a query; the model calls come after the structure, not instead of it.\nTwo things follow, and both are cheap. The index is a list of what the case type demands, written from the process before any folder is opened, which is what gives an absent document a name. And the answer is the state of the case, four lists and a verdict, each entry carrying the file and the page behind it.\nThe two questions worth building for are not retrieval questions, and that is the awkward part. A missing document is not in the corpus by definition, and a contradiction is a comparison between two typed values rather than a passage to rank. Both are easy to compute once the list exists and the pairs are declared, and neither happens unless you decide to build it.\nIf you have a folder like this, the first experiment costs an afternoon. Write down the pieces one case type expects, take twenty real cases, and run the join by hand. The count of missing pieces is the size of the problem, and it is worth knowing before anyone writes ingestion code.\nArticle 14A (three kinds of corpus, and what building for the wrong one costs) has the test that tells you whether this is the shape you are holding. Article 14B (the folder whose documents share no field) and Article 14C (one document type, many copies) take the other two.\n## 6. Further reading and sources\n**Earlier in the series:**\n  * [Document Intelligence: series intro](https://medium.com/@angela.shi/enterprise-document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-1cd0a3154abf). What the series builds, brick by brick, and in what order.\n\n\n_What works, what breaks_\n  * [Baseline Enterprise RAG, from PDF to highlighted answer](https://medium.com/data-science-collective/context-engineering-for-enterprise-rag-the-minimal-four-brick-pipeline-that-highlights-its-source-4de8c6776ecd). The four-brick pipeline end to end: PDF in, highlighted answer out.\n  * [Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval](https://medium.com/data-science-collective/embeddings-arent-magic-the-predictable-failure-modes-of-rag-retrieval-572620033db1). Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway.\n  * [RAG is not machine learning, and the ML toolkit solves the wrong problem](https://medium.com/@angela.shi/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem-a2fed3691e9f). Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead.\n  * [From regex to vision models: which RAG technique fits which problem](https://medium.com/data-science-collective/from-regex-to-vision-models-which-rag-technique-fits-which-problem-5afb3f142279). Two axes, document complexity and question control, that pick the technique for each case.\n    * [10 common RAG mistakes we keep seeing in production](https://medium.com/data-science-collective/10-common-rag-mistakes-we-keep-seeing-in-production-d0a2d4f45785). Ten production mistakes, organized brick by brick, with the fix for each.\n\n\n_Document parsing_\n  * [Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/). Rebuilding the outline from body typography when the PDF ships no contents page at all: six signals, one bounded loop.\n  * [Before Full Agentic RAG: Know How You Decide, and the Parsing Methods You Pick From](https://towardsdatascience.com/before-full-agentic-rag-know-how-you-decide-and-the-parsing-methods-you-pick-from/). The parsing methods as a catalogue, and the decision of which to run, before handing the loop to an agent.\n\n\n_Question parsing_\n  * [Loop engineering for RAG question parsing: the small loop that runs before retrieval](https://towardsdatascience.com/loop-engineering-for-rag-question-parsing-the-small-loop-that-runs-before-retrieval/). The bounded loop that runs before retrieval: parse, check the fields make sense, re-ask once when they do not.\n\n\n_Generation_\n  * [Loop Engineering for RAG Generation: Iterate top-k One at a Time](https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/). Reading the retrieved pages one at a time instead of all at once, and what that buys when the answer sits in only one of them.\n  * [Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract](https://towardsdatascience.com/most-rag-hallucinations-are-extraction-errors-seven-patterns-for-a-typed-generation-contract/). Seven recurring ways a model gets the extraction wrong, and the typed contract that catches each one.\n  * [Loop engineering for RAG generation: an LLM cascade from a cheap local model up to a hosted flagship](https://towardsdatascience.com/loop-engineering-for-rag-generation-an-llm-cascade-from-a-cheap-local-model-up-to-a-hosted-flagship/). Starting on a cheap local model and escalating only when the answer does not hold up, measured.\n\n\n_One-document pipelines_\n  * [Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations](https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/). Why a better prompt does not fix a wrong page, and what each of the four bricks contributes to the context instead.\n  * [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/). Cutting a pipeline’s latency and cost by calling the model less often and cheaper, not by buying a faster one.\n  * [Loop engineering for cross-references: when RAG answers ‘see Section 7.2’ instead of the actual answer](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/). When the answer says “see Section X”, the pipeline loops back and fetches it.\n  * [RAG workflow and loop engineering: the dispatcher that decides when to loop and when to stop](https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/). Feedback loops, bounded iteration, and the dispatcher, composed into one workflow.\n    * [Loop engineering for RAG: the small loops inside each step, the big loops across the pipeline](https://towardsdatascience.com/loop-engineering-for-rag-the-small-loops-inside-each-step-the-big-loops-across-the-pipeline/). The two scales of loop: small bounded loops inside each brick, big generation-triggered loops across them.\n\n\nAlso referenced above, listed here without links: Article 14A (three kinds of corpus, and what building for the wrong one costs), Article 14B (the folder whose documents share no field), Article 14C (one document type, many copies), Article 15 (preparing a corpus, one column at a time), Article 16 (where the corpus vocabulary comes from), and Article 17 (querying a corpus, filter first and retrieve second).\nExternal sources:\n  * Scott Barnett et al., _Seven Failure Points When Engineering a Retrieval Augmented Generation System_ , 2024 ([arXiv 2401.05856](https://arxiv.org/abs/2401.05856)). Missing content is their first failure point, measured on three production systems. On a case file it stops being a failure and becomes the output.\n  * Pranav Rajpurkar, Robin Jia and Percy Liang, _Know What You Don’t Know: Unanswerable Questions for SQuAD_ , 2018 ([arXiv 1806.03822](https://arxiv.org/abs/1806.03822)). Fifty thousand questions written so that the right answer is to abstain. The nearest thing in the benchmark literature to scoring a system on absence, and it is still about one passage rather than about a bundle.\n  * Harsh Trivedi et al., _MuSiQue: Multihop Questions via Single-hop Question Composition_ , 2022 ([arXiv 2108.00573](https://arxiv.org/abs/2108.00573)). Questions built so that no single passage can answer them, which is the property every cross-piece check in section 3.2 has.\n  * Yuta Koreeda and Christopher D. Manning, _ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts_ , 2021 ([arXiv 2110.01799](https://arxiv.org/abs/2110.01799)). 607 annotated contracts labelled entailed, contradicting or not mentioned, with the evidence spans. The document-level version of the contradiction check, and useful for how hard they found it.\n  * Nelson F. Liu et al., _Lost in the Middle: How Language Models Use Long Contexts_ , 2023 ([arXiv 2307.03172](https://arxiv.org/abs/2307.03172)). Performance is highest when the relevant text sits at the start or the end of the input and drops in between, which is the measured reason the whole bundle in one prompt is not the fix.\n\n\n* * *\n![](../images/body-249d024535392fb76067.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm](https://towardsdatascience.com/tag/llm/)[Llm Applications](https://towardsdatascience.com/tag/llm-applications/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F&title=Parse%20the%20Folder%2C%20Not%20Just%20the%20PDFs%3A%20The%20Relational%20Tables%20RAG%20Needs%20on%20a%20Case%20File)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F&title=Parse%20the%20Folder%2C%20Not%20Just%20the%20PDFs%3A%20The%20Relational%20Tables%20RAG%20Needs%20on%20a%20Case%20File)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F&text=Parse%20the%20Folder%2C%20Not%20Just%20the%20PDFs%3A%20The%20Relational%20Tables%20RAG%20Needs%20on%20a%20Case%20File)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/001.png)](https://towardsdatascience.com/from-chaos-to-clarity-streamlining-data-cleansing-using-large-language-models-a539fa0b2d90/)\n## [From Chaos to Clarity: Streamlining Data Cleansing Using Large Language Models](https://towardsdatascience.com/from-chaos-to-clarity-streamlining-data-cleansing-using-large-language-models-a539fa0b2d90/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nCleaning up survey responses using OpenAI's GPT Model. Full Code with Github link.\n[Naresh Ram](https://towardsdatascience.com/author/naresh-ram/)\nJune 7, 202317 min read\n  * [![](../images/002.jpg)](https://towardsdatascience.com/proxy-pointer-rag-multimodal-answers-without-multimodal-embeddings/)\n## [Proxy-Pointer RAG: Multimodal Answers Without Multimodal Embeddings](https://towardsdatascience.com/proxy-pointer-rag-multimodal-answers-without-multimodal-embeddings/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nStructure is all you need\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nApril 30, 202613 min read\n  * [![](../images/body-07c22e56eb6d56796a8f.jpeg)](https://towardsdatascience.com/how-to-make-the-most-out-of-llm-production-data-simulated-user-feedback-843c444febc7/)\n## [How to Make the Most Out of LLM Production Data: Simulated User Feedback](https://towardsdatascience.com/how-to-make-the-most-out-of-llm-production-data-simulated-user-feedback-843c444febc7/)\n[](https://towardsdatascience.com/parse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file/)\nA novel approach to use production data to simulate user feedback for testing and evaluating your LLM app.\n[Pasquale Antonante, Ph.D.](https://towardsdatascience.com/author/pasquale_68605/)\nApril 11, 20249 min read\n  * [![](../images/003.jpg)](https://towardsdatascience.com/step-by-step-guide-to-build-and-deploy-an-llm-powered-chat-with-memory-in-streamlit/)\n## [Step-by-Step Guide to Build and Deploy an LLM-Powered Chat with Memory in Streamlit](https://towardsdatascience.com/step-by-step-guide-to-build-and-deploy-an-llm-powered-chat-with-memory-in-streamlit/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nAnd monitor your API usage on Google Cloud Console\n[ALESSANDRA COSTA](https://towardsdatascience.com/author/alessandra.costa/)\nMay 1, 202515 min read\n  * [![](../images/004.jpeg)](https://towardsdatascience.com/learn-to-build-agentic-ai-systems-9e552d841525/)\n## [On a Time Crunch but Still Want to Learn to Develop Multi-Agent AI?](https://towardsdatascience.com/learn-to-build-agentic-ai-systems-9e552d841525/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nThese 3 starter projects only take a weekend (and a few cups of coffee, maybe)\n[Thuwarakesh Murallie](https://towardsdatascience.com/author/thuwarakesh/)\nJanuary 23, 202513 min read\n  * [![](../images/005.png)](https://towardsdatascience.com/deep-dive-into-lstms-xlstms-by-hand-%ef%b8%8f-c33e638bebb1/)\n## [Deep Dive into LSTMs & xLSTMs by Hand ✍️](https://towardsdatascience.com/deep-dive-into-lstms-xlstms-by-hand-%ef%b8%8f-c33e638bebb1/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nExplore the wisdom of LSTM leading into xLSTMs - a probable competition to the present-day LLMs\n[Srijanie Dey, PhD](https://towardsdatascience.com/author/srijanie.dey/)\nJuly 9, 202411 min read\n  * [![](../images/006.jpg)](https://towardsdatascience.com/proxy-pointer-rag-achieving-vectorless-accuracy-at-vector-rag-scale-and-cost/)\n## [Proxy-Pointer RAG: Achieving Vectorless Accuracy at Vector RAG Scale and Cost](https://towardsdatascience.com/proxy-pointer-rag-achieving-vectorless-accuracy-at-vector-rag-scale-and-cost/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nA new way to build vector RAG—structure-aware and reasoning-capable\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nApril 5, 202620 min read\n  * [![](../images/body-ba10697c3d323b1c0de6.jpeg)](https://towardsdatascience.com/understanding-long-rope-in-llms-29337dc7e4a9/)\n## [Understanding Long RoPE in LLMs](https://towardsdatascience.com/understanding-long-rope-in-llms-29337dc7e4a9/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThis blog post will go in detail about the new Long RoPE Methodology used to expand the context lengths LLMs can support without...\n[Matthew Gunton](https://towardsdatascience.com/author/mgunton7/)\nMay 15, 20248 min read\n  * [![](../images/007.jpg)](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/)\n## [Loop Engineering for Cross-References: When RAG Answers ‘see Section 7.2’ Instead of the Actual Answer](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #11] - When the first answer points elsewhere in the document, the pipeline loops back to fetch the linked context\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 6, 202620 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "Parse the Folder, Not Just the PDFs: The Relational Tables RAG Needs on a Case File | Towards Data Science",
    "description": "Enterprise Document Intelligence [Vol.1 #14D] - The index lists what the case type demands before any folder is opened, and the two questions worth building for are not retrieval questions at all",
    "keywords": null,
    "author": "Angela and Kezhan Shi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Parse the Folder, Not Just the PDFs: The Relational Tables RAG Needs on a Case File",
    "og:description": "Enterprise Document Intelligence [Vol.1 #14D] - The index lists what the case type demands before any folder is opened, and the two questions worth building for are not retrieval questions at all",
    "og:url": "https://towardsdatascience.com/parse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/binder_shelf_34293529_v3_card.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Angela and Kezhan Shi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "20 min read",
    "article:published_time": "2026-08-23T09:00:05.000Z",
    "article:modified_time": "2026-08-23T09:00:05.000Z",
    "article:author": "https://towardsdatascience.com/author/angela.shi/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/58b903da6998/code/block_000.txt",
      "chars": 1102
    },
    {
      "language": "txt",
      "file": "attachments/58b903da6998/code/block_001.txt",
      "chars": 803
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/binder_shelf_34293529_v3_card.jpg",
      "bytes": 133550,
      "reused": true,
      "local_path": "attachments/58b903da6998/images/000.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/binder_shelf_34293529_v3_card.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-139-1024x572.png",
      "bytes": 136462,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/58b903da6998/images/body-238f6cfcab49a64eafb6.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-139-1024x572.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-137-1024x658.png",
      "bytes": 162802,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/58b903da6998/images/body-e236ef9d3ff984fa15a3.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-137-1024x658.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-138-1024x309.png",
      "bytes": 132125,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/58b903da6998/images/body-988aa17754d3d5fe8fba.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-138-1024x309.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-141-1024x618.png",
      "bytes": 233664,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/58b903da6998/images/body-db1eb0c2478395df5119.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-141-1024x618.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-140-1024x383.png",
      "bytes": 155558,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/58b903da6998/images/body-29474212259c726a966e.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-140-1024x383.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg",
      "bytes": 212225,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/58b903da6998/images/body-249d024535392fb76067.jpeg",
      "src": "https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1EpbKIcJsaFLr_sUD1tPSVA.png",
      "bytes": 2872618,
      "reused": true,
      "local_path": "attachments/58b903da6998/images/001.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1EpbKIcJsaFLr_sUD1tPSVA.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/Gemini_Generated_Image_n4hxxdn4hxxdn4hx-scaled-1.jpg",
      "bytes": 537841,
      "reused": true,
      "local_path": "attachments/58b903da6998/images/002.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/Gemini_Generated_Image_n4hxxdn4hxxdn4hx-scaled-1.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/1r3Ov2EIMg_xfZJ1vavTbPA.jpeg",
      "bytes": 115916,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/58b903da6998/images/body-07c22e56eb6d56796a8f.jpeg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/1r3Ov2EIMg_xfZJ1vavTbPA.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/kaja-kadlecova-e04R6GDZdvY-unsplash-scaled-1.jpg",
      "bytes": 185564,
      "reused": true,
      "local_path": "attachments/58b903da6998/images/003.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/kaja-kadlecova-e04R6GDZdvY-unsplash-scaled-1.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/1f0PkshYadnjBu0el57GX7Q.jpeg",
      "bytes": 157878,
      "reused": true,
      "local_path": "attachments/58b903da6998/images/004.jpeg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/1f0PkshYadnjBu0el57GX7Q.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1Zf6XTb6jDQXVOt-N9S_YTg.png",
      "bytes": 1372282,
      "reused": true,
      "local_path": "attachments/58b903da6998/images/005.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1Zf6XTb6jDQXVOt-N9S_YTg.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/proxy-pointer-2048x1143-1.jpg",
      "bytes": 330039,
      "reused": true,
      "local_path": "attachments/58b903da6998/images/006.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/proxy-pointer-2048x1143-1.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1jWYfQqYTBijIMLoyOEjmaA.jpeg",
      "bytes": 114423,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/58b903da6998/images/body-ba10697c3d323b1c0de6.jpeg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1jWYfQqYTBijIMLoyOEjmaA.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/bookmark_enNl3McVwSI_v3_card.jpg",
      "bytes": 143754,
      "reused": true,
      "local_path": "attachments/58b903da6998/images/007.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/bookmark_enNl3McVwSI_v3_card.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/large-language-model/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/angela.shi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/most-rag-hallucinations-are-extraction-errors-seven-patterns-for-a-typed-generation-contract/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-listing-questions-when-the-answer-is-every-passage-not-the-top-one/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/before-full-agentic-rag-know-how-you-decide-and-the-parsing-methods-you-pick-from/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-rag-question-parsing-the-small-loop-that-runs-before-retrieval/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-rag-generation-an-llm-cascade-from-a-cheap-local-model-up-to-a-hosted-flagship/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://www.pexels.com/photo/34293529/"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://github.com/doc-intel/notebooks-vol1"
    },
    {
      "type": "external",
      "href": "https://medium.com/@angela.shi/enterprise-document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-1cd0a3154abf"
    },
    {
      "type": "external",
      "href": "https://medium.com/data-science-collective/context-engineering-for-enterprise-rag-the-minimal-four-brick-pipeline-that-highlights-its-source-4de8c6776ecd"
    },
    {
      "type": "external",
      "href": "https://medium.com/data-science-collective/embeddings-arent-magic-the-predictable-failure-modes-of-rag-retrieval-572620033db1"
    },
    {
      "type": "external",
      "href": "https://medium.com/@angela.shi/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem-a2fed3691e9f"
    },
    {
      "type": "external",
      "href": "https://medium.com/data-science-collective/from-regex-to-vision-models-which-rag-technique-fits-which-problem-5afb3f142279"
    },
    {
      "type": "external",
      "href": "https://medium.com/data-science-collective/10-common-rag-mistakes-we-keep-seeing-in-production-d0a2d4f45785"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2401.05856"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/1806.03822"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2108.00573"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2110.01799"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2307.03172"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Parse+the+Folder%2C+Not+Just+the+PDFs%3A+The+Relational+Tables+RAG+Needs+on+a+Case+File&u=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Parse+the+Folder%2C+Not+Just+the+PDFs%3A+The+Relational+Tables+RAG+Needs+on+a+Case+File&url=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Parse+the+Folder%2C+Not+Just+the+PDFs%3A+The+Relational+Tables+RAG+Needs+on+a+Case+File&url=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:25:29Z",
  "source_url": "https://towardsdatascience.com/parse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "http_403",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 16,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# Parse the Folder, Not Just the PDFs: The Relational Tables RAG Needs on a Case File\nEnterprise Document Intelligence [Vol.1 #14D] - The index lists what the case type demands before any folder is opened, and the two questions worth building for are not retrieval questions at all\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 23, 202620 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/binder_shelf_34293529_v3_card.jpg)\n[Photo](https://www.pexels.com/photo/34293529/) by Zulfugar Karimov, via Pexels.\nContext windows keep getting longer, and the advice that comes with them is simple: stop retrieving, put the whole thing in the prompt.\nOn a case file that advice nearly works. Eleven PDFs about one insurance claim come to sixty-odd pages, which fits inside a 200,000-token window several times over.\nIt still misses the two questions the person handling the claim has. Is the second repair invoice in the folder at all, and does the date of loss on the claim form match the one in the adjuster’s report?\nThe reason is in the shape of the thing. A case file is a bundle of unlike documents about one entity, and the unit of work is the bundle. A claim, a credit application, a medical record, a hiring file. The contract, the certificate, the photographs, the expert’s report, the letters: nothing in there resembles anything else, and the answer comes from reading across all of it.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\nSo neither question is a search. The first is about a document that does not exist, and no amount of context holds it. The second needs one value out of each of two files, compared. Models read the middle of a long input less reliably than the ends, and that has been measured. In a bundle of eleven, both values are in the middle.\nThis article works through what that changes:\n  * what separates a case file from a collection that happens to be small\n  * the list of pieces a case type expects, which indexes what should be there rather than what is\n  * why a missing document is a first-class answer, and what it takes to produce one\n  * the checks that compare one piece against another, and the typed values they need first\n  * the shape of the answer, which is the state of the case rather than a passage\n  * why this shape has the least prior art of the three, and how to evaluate it anyway\n\n\nThis article is part of Part IV of [Enterprise Document Intelligence](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/), a series that builds an enterprise RAG system from four bricks. Part IV is where the input stops being a file and becomes a folder, and this is the folder that behaves like one object rather than like a collection.\n🧭 **New to the series? Start with the map** : [Prompt, Context, Loop](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/) sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-139-1024x572.png)\n_where this article sits: Part IV, the folder that is one case - Image by author_\n📓 **Run the completeness check on a folder of your own in the companion notebook** : write down the pieces one case type expects, point the check at a folder, and print the three lists that come back: the pieces present, the pieces missing, and the files that belong to some other case. Repo → [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1).\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-137-1024x658.png)\n_The public companion-code repo at doc-intel/notebooks-vol1 - Image by author_\nThe worked example is a fire claim on a small joinery workshop, and it is fictional: the workshop, the dates, the amounts and the files were authored for the series and match no real insured or insurer. The list of expected pieces is written the way a broker’s back office would write one, not copied from anybody’s manual.\n## 1. A bundle is not a small corpus\nOne folder first, then the comparison that says why the other two architectures do not fit it.\n### 1.1. Eleven files about one fire\nA fire damages a joinery workshop on 12 March. Over the next six weeks the broker’s folder for that claim fills up with eleven PDFs: the claim form, the policy schedule in force that month, a certificate of premium payment, the loss adjuster’s report, two repair quotes, one repair invoice, a set of damage photographs, two letters, and a vehicle schedule that belongs to a different policy and was filed here by mistake.\nNothing in that list resembles anything else in it. The photographs carry no text. The adjuster’s report is prose. The quotes are tables. The claim form is boxes with values typed into them.\nAsk the handler what she does first and she does not search. She runs down a list she knows by heart and ticks off what is there. Fire claim, so the fire brigade report is required, and it is not in the folder. Two quotes accepted, so two invoices should follow, and only one has. That is a couple of minutes of work, and no document has been read yet.\nOnly after that does she read. The date of loss on the claim form says 12 March. The adjuster writes that he inspected the site on 3 March. One of those two is wrong, and until somebody says which, the file does not move.\n### 1.2. What the other two shapes give you, and what they cost here\nArticle 14A (three kinds of corpus, and what building for the wrong one costs) sets out three shapes a collection can take and a three-question test that tells you which one you are holding. This is the third shape: documents that arrive in bundles, several of them about one case.\nPut the three side by side, question by question.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-138-1024x309.png)\n_five questions, three shapes, and where the bundle parts company - Image by author_\nTwo rows matter most here.\nThe index row. Article 14C (one document type, many copies, and the columns the business can already name) builds a table with one row per document and the business fields on it. That table describes what exists. Here the row is the case, and what goes in it is a list of pieces the case is supposed to hold, which is a different thing to write down and a different thing to fill.\nThe empty result. On the other two shapes, nothing found means the retrieval step failed and you go and fix it. On a case file, nothing found can be the correct answer, and often it is the answer the question was after.\nThe remaining comparison is the one that catches people out, because from a distance the two look identical. Article 14B (the folder whose documents share no field, indexed by one summary line each) treats a heterogeneous folder as one long document and routes down its nested outline. A case file is heterogeneous in exactly that way, so the routing runs fine here. It answers the wrong question: routing picks a piece, and these questions are about the set.\n## 2. The list of pieces a case type expects\nThe preparation for this shape is one short artefact: a list of the pieces the case type demands, with how many of each and under what condition.\nThis is parsing, one floor up. On a single PDF, the parsing brick returns a small set of relational tables, one row per line, one row per section, one row per table cell (Article 5B, _the relational tables RAG needs_). A folder parses the same way: one table of expected roles, one table of arrived files with the role each was assigned and a confidence on that assignment. Most of the work on a case file is here, in structuring and cleaning the bundle into those two tables. The completeness check, the contradiction check and the case state later in this article are all queries over them.\n### 2.1. An index of what should be there\nA list of what should be there, rather than a description of what is. The index in Article 14C describes documents that exist; this list names documents that ought to, and the two are read at different moments.\nAn index built by reading the corpus describes what the corpus contains. Point an extraction pass at five thousand contracts and you get a table with one row per contract and a client, a date and a premium on each. Ask it which contract is missing and the question does not parse. Everything in the table is there by construction.\nThe list here is written before the folder is opened, and it comes from the process rather than from the files. A property damage claim needs a claim form, a policy schedule, proof that the premium was paid, an adjuster’s report, at least one quote, an invoice for each accepted quote, photographs, and a fire brigade report when the cause is fire. That list holds for the case type whether or not a single one of those documents was ever filed.\nDerive the same list from the folders you already have and the missing piece disappears. Every role in such a list is a role that showed up in at least one folder, so a piece nobody ever files is simply not in the list.\n### 2.2. What a row of the list holds\nFour fields per piece: what it is, how many are expected, when it is required, and whether the case can move without it.\npython\n\n```\n# Written from the process, before any folder is opened.class ExpectedPiece(BaseModel):    \"\"\"One row of what a case type demands, filed or not.\"\"\"    role: str                  # the business word: \"loss adjuster report\"    min_count: int             # 0 when the piece is optional    max_count: int | None      # None when any number of them is fine    required_when: str | None  # a condition on the case's own fields    blocks_payment: bool       # can the case move on while this is absentPROPERTY_DAMAGE_CLAIM = [    ExpectedPiece(role=\"claim form\", min_count=1, max_count=1,                  required_when=None, blocks_payment=True),    ExpectedPiece(role=\"fire brigade report\", min_count=1, max_count=1,                  required_when=\"cause_of_loss == 'fire'\",                  blocks_payment=True),    ExpectedPiece(role=\"repair invoice\", min_count=1, max_count=None,                  required_when=\"accepted_quotes >= 1\",                  blocks_payment=True),    ExpectedPiece(role=\"correspondence\", min_count=0, max_count=None,                  required_when=None, blocks_payment=False),]\n```\n\n`required_when` is what stops the list producing false alarms. The fire brigade report is required on a fire and irrelevant on water damage, so a flat list of required pieces complains on every claim that is not a fire. A check that is wrong most of the time gets switched off. The condition reads fields off the case itself, which means the case has to carry a handful of fields of its own before the completeness check can run at all.\n`blocks_payment` is what makes the output usable. Two pieces are missing and one of them stops the money: a handler needs to see which, and a boolean per piece is enough to say it.\nWhere the list comes from is a conversation, and a short one. Somebody in the back office already has it, on a laminated card or in a procedures document, because a person has to check the same thing by hand today. Article 14C asks the business which fields it filters on. Here the question is narrower: what has to be in the folder before you sign this off. The answer usually comes back as a list.\n### 2.3. Matching the folder against the list\nWith the list on one side and eleven files on the other, the completeness check is a join with three outcomes.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-141-1024x618.png)\n_nine expected pieces, eleven files, and the three outcomes of the join - Image by author_\nA piece is **present** when at least `min_count` files were assigned to its role. Seven of the nine roles pass here.\nA piece is **missing** when the role is required, its condition holds, and no file was assigned to it. Two roles fail: the fire brigade report, because the cause is fire and nothing was filed, and the second repair invoice, because two quotes were accepted and one invoice arrived.\nA file is **unmatched** when no role in the list fits it. Here that is the vehicle schedule from another policy. The outcome is worth reporting rather than dropping, because it is either a filing mistake or a role the list forgot.\nThe join rests on one step that can go wrong. Assigning a role to each file is a classification, and classifications make mistakes. An invoice filed under quotes reads as a missing invoice plus an extra quote, so the report then carries two wrong lines instead of none. The role assignment therefore keeps its own confidence, and a low-confidence assignment goes to the handler as a question rather than as a verdict.\n## 3. Two questions retrieval cannot answer\nBoth of the questions the handler asked in section 1.1 have the same property: no passage anywhere in the bundle answers them, so no amount of ranking gets you there.\n### 3.1. The missing piece\nAsk a document-by-document pipeline where the fire brigade report is and it returns an empty result. Ask it the same thing on a claim where the report is filed under a strange name and it also returns an empty result. Those are different situations and the pipeline cannot tell them apart, because an empty result carries no information about why it is empty.\nThe completeness check can, and that is what the list is for. It knows the role exists, it knows the condition holds, and it knows nothing was assigned. The answer it produces is a sentence with a reason attached: the fire brigade report is required because the cause of loss is fire, no file in the folder was assigned to that role, and payment is blocked until one is.\nGetting there needs three things the pipeline does not have by default. A list of roles, so absence has a name. A condition on each role, so absence is only claimed when the piece was actually due. And a confidence on the role assignment, so a badly filed document is reported as doubt rather than as a categorical no.\nThat last one matters more than it looks. A report that says _missing_ when the document is sitting in the folder under a bad name is worse than no report, because the handler then asks the client for a file the client already sent. The series already has this discipline one level down, on the generation side. A typed answer contract carries not-found as a value rather than as an empty string, which is what [the seven extraction patterns](https://towardsdatascience.com/most-rag-hallucinations-are-extraction-errors-seven-patterns-for-a-typed-generation-contract/) work out on a single document. Here the same move runs one floor up, on the bundle instead of on the field.\n### 3.2. The contradiction between two pieces\nThe second question compares. The date of loss appears on the claim form and again in the adjuster’s report, and the two do not agree.\nTop-k retrieval is the wrong instrument here, and not because of its quality. Ranking returns the passages most like the question. A contradiction is not like anything: both halves read as ordinary on their own, neither passage mentions the other, and nothing in either one is closer to the question than the rest of the file.\nWhat works instead is mechanical. Declare which fields are worth comparing and between which roles, extract that field from each piece with a typed contract and a citation, then compare the values.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-140-1024x383.png)\n_five declared pairs, the two values read for each, and what a mismatch means - Image by author_\nThree details decide whether this works in practice.\n**The pairs are declared, not discovered.** Nine pieces make thirty-six pairs, and multiplied by every field on every piece the comparison space is large and mostly pointless. What people actually check is a handful of pairs per case type, and a business user can name them in the same conversation that produced the list of expected pieces.\n**The values are typed before they are compared.** `12/03/2025` and `March 12, 2025` are the same date and two different strings. `Alpine Joinery SARL` and `Alpine Woodworks` are two different strings that may or may not be the same company. The comparison runs on parsed values, and the second case, where normalisation is a judgement rather than a format, is the one that needs a person in the loop. Turning a raw extraction into something joinable is its own piece of work, and Article 15 (preparing a corpus, one column at a time) takes it up.\n**Every value keeps its citation.** A report that only says the dates disagree gives the handler nothing to go and check. One that says the claim form gives 12 March on page 1 and the report gives 3 March on page 4 can be settled in ten seconds.\n## 4. The answer is the state of the case\nAsk a corpus a question and a good answer names documents. Ask a case file and a good answer describes the case. That answer is a composed object rather than a passage.\n### 4.1. What the object holds\nFour lists and a verdict. The lists are what section 2 and section 3 produce; the verdict is the sentence a handler would write at the top of the file.\npython\n\n```\nclass FieldConflict(BaseModel):    \"\"\"One declared pair that came back with two values.\"\"\"    field: str                 # \"date_of_loss\"    left: TypedValue           # value, doc_id, page, quoted line    right: TypedValueclass CaseState(BaseModel):    \"\"\"What a question about a case returns: the whole bundle.\"\"\"    case_id: str    case_type: str                  # \"property damage claim\"    present: list[MatchedPiece]     # role, doc_id, match confidence    missing: list[MissingPiece]     # role, why it was due, what it blocks    unmatched: list[str]            # files no role in the list fits    conflicts: list[FieldConflict]  # the declared pairs that disagree    verdict: Literal[\"complete\", \"incomplete\", \"conflicting\"]    verdict_reason: str             # one sentence naming what blocks\n```\n\nTwo properties of that object matter for how it gets filled.\nIt is a list-shaped answer, not a top-ranked one. Every entry counts and the ordering does not, which is the same contract a listing question needs inside one document. [Loop engineering for listing questions](https://towardsdatascience.com/loop-engineering-for-listing-questions-when-the-answer-is-every-passage-not-the-top-one/) works that out on a single PDF, and the constraint carries over to a bundle unchanged.\nIt is filled piece by piece rather than in one call. Eleven documents do fit in a window, and putting them all there is still the worse option: the extraction is per piece, the citation has to name the piece it came from, and reading one document at a time is what keeps both of those true. That trade, one call over everything against one call per item, is the subject of [iterating top-k one at a time](https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/) on a single document.\nThe object needs one more thing, and the series already has it. A piece can point at another piece: the adjuster’s report says _see the accepted quote_ , and the value being compared is in the other file. That is the same fetch-and-continue loop as [a document that answers ‘see Section 7.2’](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/), run across files instead of across sections.\n### 4.2. Where it breaks\nOf the three shapes, this is the one with the least prior art, both in the series and in the retrieval literature. So the failures below come with fewer known fixes than the ones in the other two articles.\nThe last row is the important one. Retrieval benchmarks measure whether the right passage came back. There is no standard set that measures whether a system correctly said a required document was never filed. The closest thing in the literature is the unanswerable half of a reading comprehension set, which is a different question asked of a single passage. So the evaluation stays local: take fifty cases a handler already checked by hand, run the completeness check, and count the two error types separately. A piece wrongly called missing costs a phone call to the client. A missing piece not caught means the claim is settled on an incomplete file, which is the thing the check exists to prevent.\nThree pieces of the build are not in this article. The index that holds the case and its fields is Article 15 (preparing a corpus, one column at a time). The vocabulary that says a _loss adjuster report_ and an _expert report_ are the same role is Article 16 (where the corpus vocabulary comes from). Routing a user’s question to the right case, and to the right mode once it is there, is Article 17 (querying a corpus, filter first and retrieve second).\n## 5. Conclusion\nA case file is neither a document nor a corpus. It is a bundle of unlike pieces about one entity, and the unit of work is the bundle.\nThe move that carries over from the single-document articles is relational. Parse the bundle into tables first, and every later question becomes a query; the model calls come after the structure, not instead of it.\nTwo things follow, and both are cheap. The index is a list of what the case type demands, written from the process before any folder is opened, which is what gives an absent document a name. And the answer is the state of the case, four lists and a verdict, each entry carrying the file and the page behind it.\nThe two questions worth building for are not retrieval questions, and that is the awkward part. A missing document is not in the corpus by definition, and a contradiction is a comparison between two typed values rather than a passage to rank. Both are easy to compute once the list exists and the pairs are declared, and neither happens unless you decide to build it.\nIf you have a folder like this, the first experiment costs an afternoon. Write down the pieces one case type expects, take twenty real cases, and run the join by hand. The count of missing pieces is the size of the problem, and it is worth knowing before anyone writes ingestion code.\nArticle 14A (three kinds of corpus, and what building for the wrong one costs) has the test that tells you whether this is the shape you are holding. Article 14B (the folder whose documents share no field) and Article 14C (one document type, many copies) take the other two.\n## 6. Further reading and sources\n**Earlier in the series:**\n  * [Document Intelligence: series intro](https://medium.com/@angela.shi/enterprise-document-intelligence-a-series-on-building-rag-brick-by-brick-from-minimal-to-corpus-1cd0a3154abf). What the series builds, brick by brick, and in what order.\n\n\n_What works, what breaks_\n  * [Baseline Enterprise RAG, from PDF to highlighted answer](https://medium.com/data-science-collective/context-engineering-for-enterprise-rag-the-minimal-four-brick-pipeline-that-highlights-its-source-4de8c6776ecd). The four-brick pipeline end to end: PDF in, highlighted answer out.\n  * [Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval](https://medium.com/data-science-collective/embeddings-arent-magic-the-predictable-failure-modes-of-rag-retrieval-572620033db1). Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway.\n  * [RAG is not machine learning, and the ML toolkit solves the wrong problem](https://medium.com/@angela.shi/rag-is-not-machine-learning-and-the-ml-toolkit-solves-the-wrong-problem-a2fed3691e9f). Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead.\n  * [From regex to vision models: which RAG technique fits which problem](https://medium.com/data-science-collective/from-regex-to-vision-models-which-rag-technique-fits-which-problem-5afb3f142279). Two axes, document complexity and question control, that pick the technique for each case.\n    * [10 common RAG mistakes we keep seeing in production](https://medium.com/data-science-collective/10-common-rag-mistakes-we-keep-seeing-in-production-d0a2d4f45785). Ten production mistakes, organized brick by brick, with the fix for each.\n\n\n_Document parsing_\n  * [Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/). Rebuilding the outline from body typography when the PDF ships no contents page at all: six signals, one bounded loop.\n  * [Before Full Agentic RAG: Know How You Decide, and the Parsing Methods You Pick From](https://towardsdatascience.com/before-full-agentic-rag-know-how-you-decide-and-the-parsing-methods-you-pick-from/). The parsing methods as a catalogue, and the decision of which to run, before handing the loop to an agent.\n\n\n_Question parsing_\n  * [Loop engineering for RAG question parsing: the small loop that runs before retrieval](https://towardsdatascience.com/loop-engineering-for-rag-question-parsing-the-small-loop-that-runs-before-retrieval/). The bounded loop that runs before retrieval: parse, check the fields make sense, re-ask once when they do not.\n\n\n_Generation_\n  * [Loop Engineering for RAG Generation: Iterate top-k One at a Time](https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/). Reading the retrieved pages one at a time instead of all at once, and what that buys when the answer sits in only one of them.\n  * [Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract](https://towardsdatascience.com/most-rag-hallucinations-are-extraction-errors-seven-patterns-for-a-typed-generation-contract/). Seven recurring ways a model gets the extraction wrong, and the typed contract that catches each one.\n  * [Loop engineering for RAG generation: an LLM cascade from a cheap local model up to a hosted flagship](https://towardsdatascience.com/loop-engineering-for-rag-generation-an-llm-cascade-from-a-cheap-local-model-up-to-a-hosted-flagship/). Starting on a cheap local model and escalating only when the answer does not hold up, measured.\n\n\n_One-document pipelines_\n  * [Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations](https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/). Why a better prompt does not fix a wrong page, and what each of the four bricks contributes to the context instead.\n  * [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/). Cutting a pipeline’s latency and cost by calling the model less often and cheaper, not by buying a faster one.\n  * [Loop engineering for cross-references: when RAG answers ‘see Section 7.2’ instead of the actual answer](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/). When the answer says “see Section X”, the pipeline loops back and fetches it.\n  * [RAG workflow and loop engineering: the dispatcher that decides when to loop and when to stop](https://towardsdatascience.com/rag-workflow-and-loop-engineering-the-dispatcher-that-decides-when-to-loop-and-when-to-stop/). Feedback loops, bounded iteration, and the dispatcher, composed into one workflow.\n    * [Loop engineering for RAG: the small loops inside each step, the big loops across the pipeline](https://towardsdatascience.com/loop-engineering-for-rag-the-small-loops-inside-each-step-the-big-loops-across-the-pipeline/). The two scales of loop: small bounded loops inside each brick, big generation-triggered loops across them.\n\n\nAlso referenced above, listed here without links: Article 14A (three kinds of corpus, and what building for the wrong one costs), Article 14B (the folder whose documents share no field), Article 14C (one document type, many copies), Article 15 (preparing a corpus, one column at a time), Article 16 (where the corpus vocabulary comes from), and Article 17 (querying a corpus, filter first and retrieve second).\nExternal sources:\n  * Scott Barnett et al., _Seven Failure Points When Engineering a Retrieval Augmented Generation System_ , 2024 ([arXiv 2401.05856](https://arxiv.org/abs/2401.05856)). Missing content is their first failure point, measured on three production systems. On a case file it stops being a failure and becomes the output.\n  * Pranav Rajpurkar, Robin Jia and Percy Liang, _Know What You Don’t Know: Unanswerable Questions for SQuAD_ , 2018 ([arXiv 1806.03822](https://arxiv.org/abs/1806.03822)). Fifty thousand questions written so that the right answer is to abstain. The nearest thing in the benchmark literature to scoring a system on absence, and it is still about one passage rather than about a bundle.\n  * Harsh Trivedi et al., _MuSiQue: Multihop Questions via Single-hop Question Composition_ , 2022 ([arXiv 2108.00573](https://arxiv.org/abs/2108.00573)). Questions built so that no single passage can answer them, which is the property every cross-piece check in section 3.2 has.\n  * Yuta Koreeda and Christopher D. Manning, _ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts_ , 2021 ([arXiv 2110.01799](https://arxiv.org/abs/2110.01799)). 607 annotated contracts labelled entailed, contradicting or not mentioned, with the evidence spans. The document-level version of the contradiction check, and useful for how hard they found it.\n  * Nelson F. Liu et al., _Lost in the Middle: How Language Models Use Long Contexts_ , 2023 ([arXiv 2307.03172](https://arxiv.org/abs/2307.03172)). Performance is highest when the relevant text sits at the start or the end of the input and drops in between, which is the measured reason the whole bundle in one prompt is not the fix.\n\n\n* * *\n![](https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm](https://towardsdatascience.com/tag/llm/)[Llm Applications](https://towardsdatascience.com/tag/llm-applications/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F&title=Parse%20the%20Folder%2C%20Not%20Just%20the%20PDFs%3A%20The%20Relational%20Tables%20RAG%20Needs%20on%20a%20Case%20File)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F&title=Parse%20the%20Folder%2C%20Not%20Just%20the%20PDFs%3A%20The%20Relational%20Tables%20RAG%20Needs%20on%20a%20Case%20File)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fparse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file%2F&text=Parse%20the%20Folder%2C%20Not%20Just%20the%20PDFs%3A%20The%20Relational%20Tables%20RAG%20Needs%20on%20a%20Case%20File)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1EpbKIcJsaFLr_sUD1tPSVA.png)](https://towardsdatascience.com/from-chaos-to-clarity-streamlining-data-cleansing-using-large-language-models-a539fa0b2d90/)\n## [From Chaos to Clarity: Streamlining Data Cleansing Using Large Language Models](https://towardsdatascience.com/from-chaos-to-clarity-streamlining-data-cleansing-using-large-language-models-a539fa0b2d90/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nCleaning up survey responses using OpenAI's GPT Model. Full Code with Github link.\n[Naresh Ram](https://towardsdatascience.com/author/naresh-ram/)\nJune 7, 202317 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/Gemini_Generated_Image_n4hxxdn4hxxdn4hx-scaled-1.jpg)](https://towardsdatascience.com/proxy-pointer-rag-multimodal-answers-without-multimodal-embeddings/)\n## [Proxy-Pointer RAG: Multimodal Answers Without Multimodal Embeddings](https://towardsdatascience.com/proxy-pointer-rag-multimodal-answers-without-multimodal-embeddings/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nStructure is all you need\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nApril 30, 202613 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/04/1r3Ov2EIMg_xfZJ1vavTbPA.jpeg)](https://towardsdatascience.com/how-to-make-the-most-out-of-llm-production-data-simulated-user-feedback-843c444febc7/)\n## [How to Make the Most Out of LLM Production Data: Simulated User Feedback](https://towardsdatascience.com/how-to-make-the-most-out-of-llm-production-data-simulated-user-feedback-843c444febc7/)\n[](https://towardsdatascience.com/parse-the-folder-not-just-the-pdfs-the-relational-tables-rag-needs-on-a-case-file/)\nA novel approach to use production data to simulate user feedback for testing and evaluating your LLM app.\n[Pasquale Antonante, Ph.D.](https://towardsdatascience.com/author/pasquale_68605/)\nApril 11, 20249 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/kaja-kadlecova-e04R6GDZdvY-unsplash-scaled-1.jpg)](https://towardsdatascience.com/step-by-step-guide-to-build-and-deploy-an-llm-powered-chat-with-memory-in-streamlit/)\n## [Step-by-Step Guide to Build and Deploy an LLM-Powered Chat with Memory in Streamlit](https://towardsdatascience.com/step-by-step-guide-to-build-and-deploy-an-llm-powered-chat-with-memory-in-streamlit/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nAnd monitor your API usage on Google Cloud Console\n[ALESSANDRA COSTA](https://towardsdatascience.com/author/alessandra.costa/)\nMay 1, 202515 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/1f0PkshYadnjBu0el57GX7Q.jpeg)](https://towardsdatascience.com/learn-to-build-agentic-ai-systems-9e552d841525/)\n## [On a Time Crunch but Still Want to Learn to Develop Multi-Agent AI?](https://towardsdatascience.com/learn-to-build-agentic-ai-systems-9e552d841525/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nThese 3 starter projects only take a weekend (and a few cups of coffee, maybe)\n[Thuwarakesh Murallie](https://towardsdatascience.com/author/thuwarakesh/)\nJanuary 23, 202513 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1Zf6XTb6jDQXVOt-N9S_YTg.png)](https://towardsdatascience.com/deep-dive-into-lstms-xlstms-by-hand-%ef%b8%8f-c33e638bebb1/)\n## [Deep Dive into LSTMs & xLSTMs by Hand ✍️](https://towardsdatascience.com/deep-dive-into-lstms-xlstms-by-hand-%ef%b8%8f-c33e638bebb1/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nExplore the wisdom of LSTM leading into xLSTMs - a probable competition to the present-day LLMs\n[Srijanie Dey, PhD](https://towardsdatascience.com/author/srijanie.dey/)\nJuly 9, 202411 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/proxy-pointer-2048x1143-1.jpg)](https://towardsdatascience.com/proxy-pointer-rag-achieving-vectorless-accuracy-at-vector-rag-scale-and-cost/)\n## [Proxy-Pointer RAG: Achieving Vectorless Accuracy at Vector RAG Scale and Cost](https://towardsdatascience.com/proxy-pointer-rag-achieving-vectorless-accuracy-at-vector-rag-scale-and-cost/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nA new way to build vector RAG—structure-aware and reasoning-capable\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nApril 5, 202620 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/1jWYfQqYTBijIMLoyOEjmaA.jpeg)](https://towardsdatascience.com/understanding-long-rope-in-llms-29337dc7e4a9/)\n## [Understanding Long RoPE in LLMs](https://towardsdatascience.com/understanding-long-rope-in-llms-29337dc7e4a9/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThis blog post will go in detail about the new Long RoPE Methodology used to expand the context lengths LLMs can support without...\n[Matthew Gunton](https://towardsdatascience.com/author/mgunton7/)\nMay 15, 20248 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/bookmark_enNl3McVwSI_v3_card.jpg)](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/)\n## [Loop Engineering for Cross-References: When RAG Answers ‘see Section 7.2’ Instead of the Actual Answer](https://towardsdatascience.com/loop-engineering-for-cross-references-when-rag-answers-see-section-7-2-instead-of-the-actual-answer/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #11] - When the first answer points elsewhere in the document, the pipeline loops back to fetch the linked context\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 6, 202620 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "body_base_path": "attachments/58b903da6998/pages/",
  "body_markdown_path": "attachments/58b903da6998/pages/archived-body.md",
  "local_html": "attachments/58b903da6998/pages/archived-body.html",
  "body_source": "prior_capture"
}