{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Angela Shi & Kezhan Shi",
  "title_listed": "Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/three-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one/",
  "publish_date": "2026-08-20",
  "corpus_stage": "",
  "notes": "",
  "id": "4fb8938e8405",
  "collected_at": "2026-10-02T01:04:52",
  "title": "Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One\nEnterprise Document Intelligence [Vol.1 #14A] - Three questions tell you which shape a document collection has, and each shape wants a different architecture\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 20, 202617 min read\n![](../images/000.jpg)\n[Photo](https://www.pexels.com/photo/20061882/) by Peter Holmboe, via Pexels.\nAsk an AI engineer how to take a RAG system past one document and the answer has been the same for three years. Put every page of every file into a vector store, embed the question, and let similarity search pick the passages.\nOn a demo folder that works. On an enterprise shelf it stops working, and not because a parameter is set wrong. Similarity search returns passages, so it never picks a document, and the answer comes back assembled out of three files that were never meant to be read together.\nThe step that gets skipped is the one before that. A collection of documents is not one kind of thing. A shared drive of unrelated reports, five thousand copies of the same contract, and a stack of claim folders are three different problems, and an architecture that suits any one of them suits the other two badly.\nThis article is the map of that decision:\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n  * five ways a flat pile fails once the shelf grows, and why a re-ranker fixes none of them\n  * three questions that tell you which of the three shapes you have\n  * what each shape asks you to prepare, and what building the wrong thing costs\n  * the four bricks of the single-document pipeline, seen one floor up\n  * a baseline you can run today, and the waste it makes visible on five real PDFs\n\n\nThis article opens Part IV of [Enterprise Document Intelligence](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/), a series that builds an enterprise RAG system from four bricks. Part IV is where the input stops being a file and becomes a folder, and the first thing to settle is which kind of folder is in front of you.\n🧭 **New to the series? Start with the map** : [Prompt, Context, Loop](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/) sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.\n![](../images/001.png)\n_where this article sits: the opening of Part IV - Image by author_\n📓 **Run the five-PDF baseline in the companion notebook** : one question, five model calls, and the per-document table that shows one real answer and four blanks. Then point the same loop at a folder of your own and watch the same ratio appear. Repo → [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1).\n![](../images/002.png)\n_The public companion-code repo at doc-intel/notebooks-vol1 - Image by author_\nThe worked example in section 4 is five public NIST publications: the [Cybersecurity Framework v1.1](https://doi.org/10.6028/NIST.CSWP.04162018) (NIST CSWP 04162018), FIPS 199, SP 800-207, AI 100-1 and CSWP 29. All five are US Government work, public domain in the US (see the [NIST copyright statement](https://www.nist.gov/director/copyright-fair-use-and-licensing-statements-srd-data-and-software)).\n## 1. A quarter of a million documents and one question\nEverything in the series so far could assume the system knew which document to look at, because there was only one. This section is what happens when that assumption goes.\n### 1.1. What the expert does, and what the pipeline does\nA mid-size insurance broker keeps about a quarter of a million documents. Fifteen years of policies, certificates, contracts, claims and correspondence, some scanned and some native, most of them never opened twice.\nA user asks: _“what are the seller’s obligations in our distribution agreements with retailer X?”_\nA senior claims handler answers in under a minute, and she does it without reading anything she does not need. Obligations sit in distribution agreements, so invoices and certificates are out. For retailer X there is one master from 2019 and two amendments. Obligations are usually section 4 of the master, and an amendment wins where it says so. She pulls three folders, reads three sections, comes back.\nPoint the textbook pipeline at the same question and it does something else entirely. Every passage of every document is already embedded in one store. The question gets embedded, the store returns the thirty nearest passages by cosine similarity, and those thirty go into one prompt.\nSometimes the answer is right. Often the thirty hold two paragraphs from a contract with a different retailer, a certificate that belongs to no contract at all, and a 2017 invoice whose only qualification is the word _obligation_. Nothing in the input tells the model that those three should never be read together, so it writes a paragraph that reads well and describes no document that exists.\nThe gap between the two is not accuracy. They solve different problems. The handler decides which documents count and then reads. The pipeline never decides anything about documents at all.\n![](../images/body-004876b7cbe07bfff079.png)\n_one question, two ways to answer it, and only one of them picks documents - Image by author_\n### 1.2. Five ways a flat pile fails at scale\nAt small scale the pile works often enough to demo. On a few hundred documents the right one usually lands in the top-k by luck, and the demo gets read as a validation of the approach rather than of the corpus size.\nFive things break as the shelf grows, and they break independently of each other.\n![](../images/body-e7945a6bc6774cc22ef6.png)\n_five failures, and why none of them is a tuning problem - Image by author_\nRead the last column first. A wider top-k pulls in noise at the same rate as signal. A different embedding model moves the failures around instead of removing them. Chunk size has nothing to say about a document boundary that was never recorded in the first place.\n### 1.3. Why a re-ranker does not rescue it\nThe usual next move is a cross-encoder: retrieve a wide top-100, re-score every candidate against the question one at a time, keep the best three.\nA re-ranker is a real tool and it is worth having, but not here and not yet. It re-orders the list the embedding step returned. If the right document never entered that list, and vocabulary drift is precisely the condition under which it does not, there is nothing to re-order. Article 2bis (what a cross-encoder adds over bi-encoder embeddings, measured) works through where the layer does pay, and the answer is that it pays after the candidate set is right, not instead of getting it right.\nThere is a second cost that gets less attention. The re-ranker adds a number. Asked why the system answered this way, you can now offer a cosine score, a cross-encoder score and a top-3 cut. Three numbers, none of which a claims handler can argue with.\nThe fix sits upstream of all of it. Record the shape of the collection before any question arrives, then use that shape to cut the candidate set. A clause like `doc_type = 'distribution agreement' AND client = 'retailer X'` is exact, fast, and readable by the person who will be held responsible for the answer. It takes a quarter of a million documents down to three before a single embedding is computed, and with three candidates the single-document pipeline is already enough.\n## 2. Which shape is your collection\nThe useful part is that you can tell which shape you have without writing anything. Three questions do it, and they are questions for a person rather than for a script.\n### 2.1. Three questions, asked before any code is written\nThe test costs one conversation with one business user, and it settles the architecture. Three questions, asked of the collection rather than of any one document.\n**Question one: do two documents in the collection ever point at each other?** An amendment names its master. A renewal names last year’s policy. A certificate belongs to a contract. A yes here does not pick a shape. It adds a requirement to whichever shape you land on, and section 3.2 says what that requirement is.\n**Question two: can a business user name a field that every document carries, meaning the same thing in each one?** Client, effective date, premium, policy number. What tells you the answer is the speed, not the words. If four examples come back in two seconds, the collection is a database that nobody has typed yet, and typing it is the job. If the answer arrives after a pause and with qualifications, treat it as a no.\n**Question three: do the documents arrive in bundles, several of them about one case?** A claim, a credit application, a medical record. Not a folder someone made to stay tidy: a bundle where a fixed set of pieces is expected, and where the interesting questions are about the bundle rather than about any piece in it.\n![](../images/body-54996bbb97c511aad978.png)\n_three questions, and the shape each answer points to - Image by author_\nQuestions two and three pick the shape. When both come back yes, and in insurance they often both do, the row of the table is the case rather than the document. That is the third shape, not a fourth one.\nOne practical note. Run the test on a subtree, not on a drive. A shared drive that has grown for ten years usually holds two of these next to each other, and the right answer for the whole drive is the right answer for none of it.\n### 2.2. Three shapes, three architectures\nHere they are side by side, with what each one asks you to build once and what the wrong call costs.\n![](../images/001.png)\n_three shapes, three preparations, three different bills for guessing wrong - Image by author_\n**A pile of unrelated files.** There is no field to filter on, so there is nothing to build a filter with. The preparation collapses to two artefacts, and the parser already returns one of them: a summary line per file saying what that file is about, and each file’s own table of contents. Retrieval routes down those two levels, which is the hierarchical retrieval of Part III applied one floor higher.\n**One document type, many copies.** Five thousand policies of the same product, ten years of invoices, every claim form the company has ever received. The documents look alike because the same process produced them, which is exactly why the columns exist and why someone can name them. This is the shape that gets an index, and the index is a table rather than a vector store.\n**Bundles about one case.** A claim folder holds a declaration, a police report, three invoices, a photograph and two letters. Heterogeneous inside, repetitive across: every claim folder holds roughly the same kinds of piece. The unit of work is the bundle, so retrieval assembles it first and reads across it second.\n### 2.3. What it costs to build for the wrong shape\nThree shapes means three ways to be wrong, and the bills are not the same size.\n**Treating a typed corpus as a pile.** This is the default, because it requires no decision at all. Everything goes into the vector store and the five failure modes of section 1 arrive on schedule. The measurement in section 4 is the small version of it. The real cost is not the compute: it is that the system is wrong in a way nobody can trace, on a corpus where a business user could have written the correct filter by hand in ten seconds.\n**Treating a pile as a typed corpus.** This one is expensive in a different way, because it looks like diligence. A team runs a classification pass and an extraction pass over every file to fill columns that no business user could name. Months later there is a table where most cells are empty and the filled ones mean different things per row: a date that is a publication date here, an effective date there, a review deadline somewhere else. Then a filter runs on that column and quietly drops the right document. A column that means three things is worse than no column, because a filter trusts it.\n**Treating case files as independent documents.** The quiet one. Every piece answers on its own, so nothing looks broken. What never happens is the comparison. The contract says the cover started on 1 March, the certificate says 1 April, and no question is ever asked that puts the two side by side. The other half is the missing piece: when the medical report was never filed, a document-by-document system returns an empty result, and an empty result reads as “no information” rather than “a required piece is not here”.\n## 3. The same four bricks, one floor up\nWhichever shape came out of the test, the pipeline underneath is the one from Part III. What moves is the input each of its four bricks sees, so laying the two side by side is the last thing to do before building anything.\n### 3.1. Nothing new, everything widened\nEach of the four bricks does the job it already did on one document, over a wider input. Part IV adds no fifth one.\n![](../images/body-188dce16352c968c7165.png)\n_the same four bricks, widened one floor up - Image by author_\nThe right-hand column is where the work goes. Parsing gains a second pass that writes one row per document into a table of the collection. Question parsing gains a handful of typed fields whose only job is to narrow that table. Retrieval gains a filtering step in front of the filtering it already does. Generation gains a second source, because sometimes the answer is a count of rows and no document is opened at all.\nThe names do not change and neither do the contracts between the bricks. What changes is how many documents reach the loop body.\n### 3.2. Versions cut across all three shapes\nA collection where documents supersede each other is not a fourth shape. It is a dimension that any of the three can carry, and question one is what detects it.\nIt adds two things to whatever you were going to build. A relations table, saying which document replaces or amends which. And a pair of dates on each row, saying from when and until when that document was the one in force.\nSkip them and the system answers correctly from a document that stopped applying in 2023. That failure is worse than a missing answer because it looks right. The 2019 master says the seller gives 30 days of notice, the 2024 amendment says 90, and a pipeline with no time columns has no reason to prefer one over the other.\nWhich version answers is a retrieval question, and Article 17 (querying a corpus, with the time window as part of the scope) works it out. What matters at this point is only that the two columns exist, because retrieval cannot filter on a field nobody wrote.\n## 4. The baseline, and the waste it makes visible\nBefore any of the three architectures, it helps to see what the collection problem costs when you ignore it. The simplest possible version wraps the single-document pipeline in a loop, and running it on five real PDFs puts the whole issue in one table.\n### 4.1. A for-loop around the pipeline you already have\nThe single-document pipeline of Article 1 (the four-brick baseline, a PDF in and a highlighted answer out) has the signature `pdf_qa_baseline(pdf_path, question)`. The collection version has the signature `corpus_qa_baseline(question, pdf_paths)`. Everything between the two is a for-loop and a DataFrame.\npython\n\n```\ndef corpus_qa_baseline(question, pdf_paths, *, client=None, top_k=3):\n    rows = []\n    for pdf in pdf_paths:\n        answer = pdf_qa_baseline(pdf, question, top_k=top_k, client=client)\n        text = answer.answer.strip()\n        rows.append({\n            \"doc_id\": Path(pdf).stem,\n            \"answer\": answer.answer,\n            \"start_page\": answer.start_page_num,\n            \"confidence\": answer.confidence,\n            \"has_answer\": bool(text) and text.upper() != \"NA\",\n        })\n    per_doc = pd.DataFrame(rows).sort_values(\n        [\"has_answer\", \"confidence\"], ascending=[False, False]\n    ).reset_index(drop=True)\n    return CorpusAnswer(question=question, per_doc=per_doc)\n```\n\nNo classification here, no index, no vocabulary, no filter. The function opens every document, asks every document the same question, and sorts what comes back. That is deliberate. It makes the mirror from section 3 visible with none of the machinery on top, and it is the honest place to measure from.\n### 4.2. Five PDFs, one question, four calls that returned nothing\nThe question is _“What is a Profile in the Cybersecurity Framework?”_. Four of the five NIST PDFs cover cybersecurity from a different angle and the fifth covers AI risk. The Profile concept is defined in the framework document and nowhere else in the set.\n![](../images/body-5fc95a01f102dc5aa2bb.png)\n_one question, five documents, one answer, four wasted calls - Image by author_\nOne document answered: page 5 of the framework, the definition quoted, confidence 0.95. The other four returned nothing usable. Two came back with an empty string, one with the literal `NA`, and one with a bare slash.\nThat last one is worth a second look. The baseline decides whether a document answered by testing the output string: non-empty, and not the literal `NA`. A single slash passes both tests, so the sort promoted a document that had said nothing. A heuristic on the output is not a substitute for knowing, before the call, which documents could hold the answer.\nThe run took 14.3 seconds for five documents, so about 2.9 seconds per document for a parse plus one model call. The total is linear in the number of documents, because nothing in the loop is shared between them. Five documents at that rate is a demo. The broker’s shelf at that rate is roughly eight days of wall time for one question.\nThe pipeline is not broken. The right answer came from the right document with a page behind it. What is broken is the ratio. Four calls out of five did no work, and that share only rises as the shelf grows.\n## 5. What the follow-ups build, and what stays out of scope\nThree articles follow this one, one per shape. They can be read in any order, because the shapes do not depend on each other.\n**The pile.** Article 14B (_RAG on a folder of unrelated PDFs: one summary line per file is the whole index_) takes the collection with no shared field. It needs the least new machinery of the three: the summary lines and the outlines are the whole index, and retrieval is the hierarchical routing of Part III run one level higher.\n**The typed corpus.** Article 14C (the homogeneous corpus, where the business already knows the columns) takes the collection where question two came back yes. Its claim is the one that saves the most time in practice: on a homogeneous corpus you do not discover the schema, the business already holds it, and the interview that produces the columns takes an afternoon.\n**The case file.** Article 14D (case files, one bundle about a single entity) takes the third shape. It has the least prior art of the three in the RAG literature, because its two hardest questions are not retrieval questions. Does piece A contradict piece B, and which expected piece is not here.\nTwo things stay outside this group and go to follow-up work.\n**Formats other than PDF.** Word, Excel, PowerPoint and mail each need their own parser before any of this applies. The three shapes carry over unchanged; the parsing does not.\n**Document comparison.** Asking what changed between version 1 and version 2 of a policy is a different operation from asking a question of a collection. The version chain is in scope here, because retrieval has to know which version answers. The diff itself is not.\nCase files are often kept out of a corpus pipeline as an oddity. They are common enough in insurance, lending and healthcare that leaving them out means leaving out most of the work, which is why they get an article rather than a caveat.\n## 6. Conclusion\nA folder is not a corpus, and a corpus is not one thing. Three questions decide which of three shapes you have, they cost one conversation with one business user, and they are worth asking before a line of ingestion code is written.\nWhat they buy is small and specific. Question one tells you whether you need relations and validity dates. Question two tells you whether there is a table to build. Question three tells you whether the row of that table is a document or a case.\nThe for-loop from section 4 survives all three answers. Every article in this group is about the same thing: shrinking what that loop iterates over, from every document on the shelf down to the handful that could plausibly answer.\n## 7. Further reading and sources\nEarlier in the series, on the pieces this article stands on:\n  * [Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/). The pile shape leans on each file’s table of contents, and this is what to do about the files that ship none.\n  * [Most RAG Hallucinations Are Retrieval Failures: Fix Retrieval, Not the Prompt](https://towardsdatascience.com/most-rag-hallucinations-are-retrieval-failures-how-the-retrieval-brick-decides-what-the-model-can-invent/). The single-document version of the argument in section 1: what the model can invent is decided upstream of the prompt.\n  * [Context engineering for RAG question parsing: from a raw question to typed fields that steer retrieval and generation](https://towardsdatascience.com/context-engineering-for-rag-question-parsing-from-a-raw-question-to-typed-fields-that-steer-retrieval-and-generation/). Where the typed fields in row 2 of the mirror table come from.\n  * [A Production RAG Pipeline in Action: Every Answer Typed and Cited](https://towardsdatascience.com/one-rag-pipeline-four-very-different-pdfs-same-four-bricks-every-answer-typed-and-cited/). The single-document pipeline that the for-loop in section 4 calls once per file.\n  * [Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations](https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/). The four bricks the mirror table widens, on one document.\n  * [Loop Engineering for RAG Generation: Iterate top-k One at a Time](https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/). The same loop-or-batch decision, inside a document rather than across a folder.\n\n\nAlso referenced above, listed here without links: Article 1 (the four-brick baseline, a PDF in and a highlighted answer out), Article 2bis (what a cross-encoder adds over bi-encoder embeddings, measured), Article 14B (the folder of unrelated PDFs), Article 14C (the homogeneous corpus), Article 14D (case files), and Article 17 (querying a corpus, with the time window as part of the scope).\nExternal sources:\n  * Yunfan Gao et al., _Retrieval-Augmented Generation for Large Language Models: A Survey_ , 2023 ([arXiv 2312.10997](https://arxiv.org/abs/2312.10997)). The naive, advanced, modular vocabulary the baseline in section 4 sits against.\n  * Scott Barnett et al., _Seven Failure Points When Engineering a Retrieval Augmented Generation System_ , 2024 ([arXiv 2401.05856](https://arxiv.org/abs/2401.05856)). Field-measured failure points from three production systems; the missing-document and wrong-document ones are the first two rows of the table in section 1.2.\n  * Anthropic, _Introducing Contextual Retrieval_ , 2024 ([anthropic.com](https://www.anthropic.com/news/contextual-retrieval)). Measured evidence that even a strong embedding and BM25 hybrid degrades once passages lose their document context.\n  * Darren Edge et al., _From Local to Global: A Graph RAG Approach to Query-Focused Summarization_ , 2024 ([arXiv 2404.16130](https://arxiv.org/abs/2404.16130)). The opposite bet on shape two: let a model discover the entities instead of asking the business to name the columns.\n  * Jinyang Li et al., _Can LLM Already Serve as A Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs_ , 2023 ([arXiv 2305.03111](https://arxiv.org/abs/2305.03111)). What querying the typed shape looks like once the table exists, measured on 95 real databases.\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-249d024535392fb76067.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm](https://towardsdatascience.com/tag/llm/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)[Retrieval Augmented Generation](https://towardsdatascience.com/tag/retrieval-augmented-generation-2/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F&title=Three%20Kinds%20of%20RAG%20Corpus%2C%20and%20What%20It%20Costs%20to%20Build%20for%20the%20Wrong%20One)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F&title=Three%20Kinds%20of%20RAG%20Corpus%2C%20and%20What%20It%20Costs%20to%20Build%20for%20the%20Wrong%20One)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F&text=Three%20Kinds%20of%20RAG%20Corpus%2C%20and%20What%20It%20Costs%20to%20Build%20for%20the%20Wrong%20One)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/003.jpg)](https://towardsdatascience.com/using-local-llms-to-discover-high-performance-algorithms/)\n## [Using Local LLMs to Discover High-Performance Algorithms](https://towardsdatascience.com/using-local-llms-to-discover-high-performance-algorithms/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nHow I used open-source models to explore new frontiers in efficient code generation, using my MacBook and local LLMs\n[Stefano Bosisio](https://towardsdatascience.com/author/stefanobosisio1/)\nJanuary 19, 20268 min read\n  * [![](../images/004.jpg)](https://towardsdatascience.com/vector-rag-isnt-enough-i-built-a-context-graph-layer-for-multi-agent-memory/)\n## [Vector RAG Isn’t Enough — I Built a Context Graph Layer for Multi-Agent Memory](https://towardsdatascience.com/vector-rag-isnt-enough-i-built-a-context-graph-layer-for-multi-agent-memory/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nA structured context graph beat a flat chat dump and a vector-only RAG pipeline on the same multi-agent conversations — full working code, real benchmark numbers, zero API calls\n[Emmimal P Alexander](https://towardsdatascience.com/author/emmimalp.alexander/)\nJune 26, 202617 min read\n  * [![](../images/005.jpg)](https://towardsdatascience.com/how-to-apply-powerful-ai-audio-models-to-real-world-applications/)\n## [How to Apply Powerful AI Audio Models to Real-World Applications](https://towardsdatascience.com/how-to-apply-powerful-ai-audio-models-to-real-world-applications/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLearn about different types of AI audio models and the application areas they can be used in.\n[Eivind Kjosbakken](https://towardsdatascience.com/author/oieivind/)\nOctober 27, 20257 min read\n  * [![](../images/006.jpeg)](https://towardsdatascience.com/the-next-frontier-in-llm-accuracy-cb2491a740d4/)\n## [The Next Frontier in LLM Accuracy](https://towardsdatascience.com/the-next-frontier-in-llm-accuracy-cb2491a740d4/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nExploring the Power of Lamini Memory Tuning\n[Mariya Mansurova](https://towardsdatascience.com/author/miptgirl/)\nJanuary 4, 202519 min read\n  * [![](../images/007.png)](https://towardsdatascience.com/the-future-of-ai-agent-communication-with-acp/)\n## [The Future of AI Agent Communication with ACP](https://towardsdatascience.com/the-future-of-ai-agent-communication-with-acp/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA practical guide to connecting and coordinating multiple AI agents\n[Mariya Mansurova](https://towardsdatascience.com/author/miptgirl/)\nJuly 15, 202516 min read\n  * [![](../images/008.jpg)](https://towardsdatascience.com/artificial-intelligence-machine-learning-deep-learning-and-generative-ai-clearly-explained/)\n## [Artificial Intelligence, Machine Learning, Deep Learning, and Generative AI — Clearly Explained](https://towardsdatascience.com/artificial-intelligence-machine-learning-deep-learning-and-generative-ai-clearly-explained/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUnderstanding AI in 2026 — from machine learning to generative models\n[Sabrine Bendimerad](https://towardsdatascience.com/author/sabrine.bendimerad1/)\nDecember 7, 202510 min read\n  * [![](../images/009.png)](https://towardsdatascience.com/improve-your-rag-context-recall-by-40-with-an-adapted-embedding-model-5d4a8f583f32/)\n## [Improve Your RAG Context Recall by 95% with an Adapted Embedding Model.](https://towardsdatascience.com/improve-your-rag-context-recall-by-40-with-an-adapted-embedding-model-5d4a8f583f32/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nStep by Step Model Adaptation Code and Results Attached.\n[Vignesh Baskaran](https://towardsdatascience.com/author/vignesh865/)\nOctober 12, 202410 min read\n  * [![](../images/010.jpg)](https://towardsdatascience.com/how-to-build-an-efficient-knowledge-base-for-ai-models/)\n## [How to Build an Efficient Knowledge Base for AI Models](https://towardsdatascience.com/how-to-build-an-efficient-knowledge-base-for-ai-models/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nBuilding a knowledge base for AI models isn’t a one-time task but an iterative process of refinement.\n[Nidhin Karunakaran Ponon](https://towardsdatascience.com/author/nidhin.karunakaranponon/)\nMay 4, 202615 min read\n  * [![](../images/011.jpg)](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n## [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 13, 202611 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "metadata": {
    "title": "Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One | Towards Data Science",
    "description": "Enterprise Document Intelligence [Vol.1 #14A] - Three questions tell you which shape a document collection has, and each shape wants a different architecture",
    "keywords": null,
    "author": "Angela and Kezhan Shi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One",
    "og:description": "Enterprise Document Intelligence [Vol.1 #14A] - Three questions tell you which shape a document collection has, and each shape wants a different architecture",
    "og:url": "https://towardsdatascience.com/three-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/forked_road_snow_20061882_v3_card.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Angela and Kezhan Shi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "17 min read",
    "article:published_time": "2026-08-20T09:00:04.000Z",
    "article:modified_time": "2026-08-20T09:00:04.000Z",
    "article:author": "https://towardsdatascience.com/author/angela.shi/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/4fb8938e8405/code/block_000.txt",
      "chars": 686
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:40:55Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:40:55Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:40:55Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:40:55Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/large-language-model/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/angela.shi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/most-rag-hallucinations-are-retrieval-failures-how-the-retrieval-brick-decides-what-the-model-can-invent/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/context-engineering-for-rag-question-parsing-from-a-raw-question-to-typed-fields-that-steer-retrieval-and-generation/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/one-rag-pipeline-four-very-different-pdfs-same-four-bricks-every-answer-typed-and-cited/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/document-processing/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/llm/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/rag/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/rag-architecture/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/retrieval-augmented-generation-2/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://www.pexels.com/photo/20061882/"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://github.com/doc-intel/notebooks-vol1"
    },
    {
      "type": "external",
      "href": "https://doi.org/10.6028/NIST.CSWP.04162018"
    },
    {
      "type": "external",
      "href": "https://www.nist.gov/director/copyright-fair-use-and-licensing-statements-srd-data-and-software"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2312.10997"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2401.05856"
    },
    {
      "type": "external",
      "href": "https://www.anthropic.com/news/contextual-retrieval"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2404.16130"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2305.03111"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Three+Kinds+of+RAG+Corpus%2C+and+What+It+Costs+to+Build+for+the+Wrong+One&u=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Three+Kinds+of+RAG+Corpus%2C+and+What+It+Costs+to+Build+for+the+Wrong+One&url=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Three+Kinds+of+RAG+Corpus%2C+and+What+It+Costs+to+Build+for+the+Wrong+One&url=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:24:48Z",
  "source_url": "https://towardsdatascience.com/three-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One\nEnterprise Document Intelligence [Vol.1 #14A] - Three questions tell you which shape a document collection has, and each shape wants a different architecture\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 20, 202617 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/forked_road_snow_20061882_v3_card.jpg)\n[Photo](https://www.pexels.com/photo/20061882/) by Peter Holmboe, via Pexels.\nAsk an AI engineer how to take a RAG system past one document and the answer has been the same for three years. Put every page of every file into a vector store, embed the question, and let similarity search pick the passages.\nOn a demo folder that works. On an enterprise shelf it stops working, and not because a parameter is set wrong. Similarity search returns passages, so it never picks a document, and the answer comes back assembled out of three files that were never meant to be read together.\nThe step that gets skipped is the one before that. A collection of documents is not one kind of thing. A shared drive of unrelated reports, five thousand copies of the same contract, and a stack of claim folders are three different problems, and an architecture that suits any one of them suits the other two badly.\nThis article is the map of that decision:\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n  * five ways a flat pile fails once the shelf grows, and why a re-ranker fixes none of them\n  * three questions that tell you which of the three shapes you have\n  * what each shape asks you to prepare, and what building the wrong thing costs\n  * the four bricks of the single-document pipeline, seen one floor up\n  * a baseline you can run today, and the waste it makes visible on five real PDFs\n\n\nThis article opens Part IV of [Enterprise Document Intelligence](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/), a series that builds an enterprise RAG system from four bricks. Part IV is where the input stops being a file and becomes a folder, and the first thing to settle is which kind of folder is in front of you.\n🧭 **New to the series? Start with the map** : [Prompt, Context, Loop](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/) sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-101-1024x238.png)\n_where this article sits: the opening of Part IV - Image by author_\n📓 **Run the five-PDF baseline in the companion notebook** : one question, five model calls, and the per-document table that shows one real answer and four blanks. Then point the same loop at a folder of your own and watch the same ratio appear. Repo → [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1).\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-100-1024x658.png)\n_The public companion-code repo at doc-intel/notebooks-vol1 - Image by author_\nThe worked example in section 4 is five public NIST publications: the [Cybersecurity Framework v1.1](https://doi.org/10.6028/NIST.CSWP.04162018) (NIST CSWP 04162018), FIPS 199, SP 800-207, AI 100-1 and CSWP 29. All five are US Government work, public domain in the US (see the [NIST copyright statement](https://www.nist.gov/director/copyright-fair-use-and-licensing-statements-srd-data-and-software)).\n## 1. A quarter of a million documents and one question\nEverything in the series so far could assume the system knew which document to look at, because there was only one. This section is what happens when that assumption goes.\n### 1.1. What the expert does, and what the pipeline does\nA mid-size insurance broker keeps about a quarter of a million documents. Fifteen years of policies, certificates, contracts, claims and correspondence, some scanned and some native, most of them never opened twice.\nA user asks: _“what are the seller’s obligations in our distribution agreements with retailer X?”_\nA senior claims handler answers in under a minute, and she does it without reading anything she does not need. Obligations sit in distribution agreements, so invoices and certificates are out. For retailer X there is one master from 2019 and two amendments. Obligations are usually section 4 of the master, and an amendment wins where it says so. She pulls three folders, reads three sections, comes back.\nPoint the textbook pipeline at the same question and it does something else entirely. Every passage of every document is already embedded in one store. The question gets embedded, the store returns the thirty nearest passages by cosine similarity, and those thirty go into one prompt.\nSometimes the answer is right. Often the thirty hold two paragraphs from a contract with a different retailer, a certificate that belongs to no contract at all, and a 2017 invoice whose only qualification is the word _obligation_. Nothing in the input tells the model that those three should never be read together, so it writes a paragraph that reads well and describes no document that exists.\nThe gap between the two is not accuracy. They solve different problems. The handler decides which documents count and then reads. The pipeline never decides anything about documents at all.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-103-1024x545.png)\n_one question, two ways to answer it, and only one of them picks documents - Image by author_\n### 1.2. Five ways a flat pile fails at scale\nAt small scale the pile works often enough to demo. On a few hundred documents the right one usually lands in the top-k by luck, and the demo gets read as a validation of the approach rather than of the corpus size.\nFive things break as the shelf grows, and they break independently of each other.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-98-1024x310.png)\n_five failures, and why none of them is a tuning problem - Image by author_\nRead the last column first. A wider top-k pulls in noise at the same rate as signal. A different embedding model moves the failures around instead of removing them. Chunk size has nothing to say about a document boundary that was never recorded in the first place.\n### 1.3. Why a re-ranker does not rescue it\nThe usual next move is a cross-encoder: retrieve a wide top-100, re-score every candidate against the question one at a time, keep the best three.\nA re-ranker is a real tool and it is worth having, but not here and not yet. It re-orders the list the embedding step returned. If the right document never entered that list, and vocabulary drift is precisely the condition under which it does not, there is nothing to re-order. Article 2bis (what a cross-encoder adds over bi-encoder embeddings, measured) works through where the layer does pay, and the answer is that it pays after the candidate set is right, not instead of getting it right.\nThere is a second cost that gets less attention. The re-ranker adds a number. Asked why the system answered this way, you can now offer a cosine score, a cross-encoder score and a top-3 cut. Three numbers, none of which a claims handler can argue with.\nThe fix sits upstream of all of it. Record the shape of the collection before any question arrives, then use that shape to cut the candidate set. A clause like `doc_type = 'distribution agreement' AND client = 'retailer X'` is exact, fast, and readable by the person who will be held responsible for the answer. It takes a quarter of a million documents down to three before a single embedding is computed, and with three candidates the single-document pipeline is already enough.\n## 2. Which shape is your collection\nThe useful part is that you can tell which shape you have without writing anything. Three questions do it, and they are questions for a person rather than for a script.\n### 2.1. Three questions, asked before any code is written\nThe test costs one conversation with one business user, and it settles the architecture. Three questions, asked of the collection rather than of any one document.\n**Question one: do two documents in the collection ever point at each other?** An amendment names its master. A renewal names last year’s policy. A certificate belongs to a contract. A yes here does not pick a shape. It adds a requirement to whichever shape you land on, and section 3.2 says what that requirement is.\n**Question two: can a business user name a field that every document carries, meaning the same thing in each one?** Client, effective date, premium, policy number. What tells you the answer is the speed, not the words. If four examples come back in two seconds, the collection is a database that nobody has typed yet, and typing it is the job. If the answer arrives after a pause and with qualifications, treat it as a no.\n**Question three: do the documents arrive in bundles, several of them about one case?** A claim, a credit application, a medical record. Not a folder someone made to stay tidy: a bundle where a fixed set of pieces is expected, and where the interesting questions are about the bundle rather than about any piece in it.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-99-1024x645.png)\n_three questions, and the shape each answer points to - Image by author_\nQuestions two and three pick the shape. When both come back yes, and in insurance they often both do, the row of the table is the case rather than the document. That is the third shape, not a fourth one.\nOne practical note. Run the test on a subtree, not on a drive. A shared drive that has grown for ten years usually holds two of these next to each other, and the right answer for the whole drive is the right answer for none of it.\n### 2.2. Three shapes, three architectures\nHere they are side by side, with what each one asks you to build once and what the wrong call costs.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-101-1024x238.png)\n_three shapes, three preparations, three different bills for guessing wrong - Image by author_\n**A pile of unrelated files.** There is no field to filter on, so there is nothing to build a filter with. The preparation collapses to two artefacts, and the parser already returns one of them: a summary line per file saying what that file is about, and each file’s own table of contents. Retrieval routes down those two levels, which is the hierarchical retrieval of Part III applied one floor higher.\n**One document type, many copies.** Five thousand policies of the same product, ten years of invoices, every claim form the company has ever received. The documents look alike because the same process produced them, which is exactly why the columns exist and why someone can name them. This is the shape that gets an index, and the index is a table rather than a vector store.\n**Bundles about one case.** A claim folder holds a declaration, a police report, three invoices, a photograph and two letters. Heterogeneous inside, repetitive across: every claim folder holds roughly the same kinds of piece. The unit of work is the bundle, so retrieval assembles it first and reads across it second.\n### 2.3. What it costs to build for the wrong shape\nThree shapes means three ways to be wrong, and the bills are not the same size.\n**Treating a typed corpus as a pile.** This is the default, because it requires no decision at all. Everything goes into the vector store and the five failure modes of section 1 arrive on schedule. The measurement in section 4 is the small version of it. The real cost is not the compute: it is that the system is wrong in a way nobody can trace, on a corpus where a business user could have written the correct filter by hand in ten seconds.\n**Treating a pile as a typed corpus.** This one is expensive in a different way, because it looks like diligence. A team runs a classification pass and an extraction pass over every file to fill columns that no business user could name. Months later there is a table where most cells are empty and the filled ones mean different things per row: a date that is a publication date here, an effective date there, a review deadline somewhere else. Then a filter runs on that column and quietly drops the right document. A column that means three things is worse than no column, because a filter trusts it.\n**Treating case files as independent documents.** The quiet one. Every piece answers on its own, so nothing looks broken. What never happens is the comparison. The contract says the cover started on 1 March, the certificate says 1 April, and no question is ever asked that puts the two side by side. The other half is the missing piece: when the medical report was never filed, a document-by-document system returns an empty result, and an empty result reads as “no information” rather than “a required piece is not here”.\n## 3. The same four bricks, one floor up\nWhichever shape came out of the test, the pipeline underneath is the one from Part III. What moves is the input each of its four bricks sees, so laying the two side by side is the last thing to do before building anything.\n### 3.1. Nothing new, everything widened\nEach of the four bricks does the job it already did on one document, over a wider input. Part IV adds no fifth one.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-102-1024x297.png)\n_the same four bricks, widened one floor up - Image by author_\nThe right-hand column is where the work goes. Parsing gains a second pass that writes one row per document into a table of the collection. Question parsing gains a handful of typed fields whose only job is to narrow that table. Retrieval gains a filtering step in front of the filtering it already does. Generation gains a second source, because sometimes the answer is a count of rows and no document is opened at all.\nThe names do not change and neither do the contracts between the bricks. What changes is how many documents reach the loop body.\n### 3.2. Versions cut across all three shapes\nA collection where documents supersede each other is not a fourth shape. It is a dimension that any of the three can carry, and question one is what detects it.\nIt adds two things to whatever you were going to build. A relations table, saying which document replaces or amends which. And a pair of dates on each row, saying from when and until when that document was the one in force.\nSkip them and the system answers correctly from a document that stopped applying in 2023. That failure is worse than a missing answer because it looks right. The 2019 master says the seller gives 30 days of notice, the 2024 amendment says 90, and a pipeline with no time columns has no reason to prefer one over the other.\nWhich version answers is a retrieval question, and Article 17 (querying a corpus, with the time window as part of the scope) works it out. What matters at this point is only that the two columns exist, because retrieval cannot filter on a field nobody wrote.\n## 4. The baseline, and the waste it makes visible\nBefore any of the three architectures, it helps to see what the collection problem costs when you ignore it. The simplest possible version wraps the single-document pipeline in a loop, and running it on five real PDFs puts the whole issue in one table.\n### 4.1. A for-loop around the pipeline you already have\nThe single-document pipeline of Article 1 (the four-brick baseline, a PDF in and a highlighted answer out) has the signature `pdf_qa_baseline(pdf_path, question)`. The collection version has the signature `corpus_qa_baseline(question, pdf_paths)`. Everything between the two is a for-loop and a DataFrame.\npython\n\n```\ndef corpus_qa_baseline(question, pdf_paths, *, client=None, top_k=3):\n    rows = []\n    for pdf in pdf_paths:\n        answer = pdf_qa_baseline(pdf, question, top_k=top_k, client=client)\n        text = answer.answer.strip()\n        rows.append({\n            \"doc_id\": Path(pdf).stem,\n            \"answer\": answer.answer,\n            \"start_page\": answer.start_page_num,\n            \"confidence\": answer.confidence,\n            \"has_answer\": bool(text) and text.upper() != \"NA\",\n        })\n    per_doc = pd.DataFrame(rows).sort_values(\n        [\"has_answer\", \"confidence\"], ascending=[False, False]\n    ).reset_index(drop=True)\n    return CorpusAnswer(question=question, per_doc=per_doc)\n```\n\nNo classification here, no index, no vocabulary, no filter. The function opens every document, asks every document the same question, and sorts what comes back. That is deliberate. It makes the mirror from section 3 visible with none of the machinery on top, and it is the honest place to measure from.\n### 4.2. Five PDFs, one question, four calls that returned nothing\nThe question is _“What is a Profile in the Cybersecurity Framework?”_. Four of the five NIST PDFs cover cybersecurity from a different angle and the fifth covers AI risk. The Profile concept is defined in the framework document and nowhere else in the set.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-104-1024x474.png)\n_one question, five documents, one answer, four wasted calls - Image by author_\nOne document answered: page 5 of the framework, the definition quoted, confidence 0.95. The other four returned nothing usable. Two came back with an empty string, one with the literal `NA`, and one with a bare slash.\nThat last one is worth a second look. The baseline decides whether a document answered by testing the output string: non-empty, and not the literal `NA`. A single slash passes both tests, so the sort promoted a document that had said nothing. A heuristic on the output is not a substitute for knowing, before the call, which documents could hold the answer.\nThe run took 14.3 seconds for five documents, so about 2.9 seconds per document for a parse plus one model call. The total is linear in the number of documents, because nothing in the loop is shared between them. Five documents at that rate is a demo. The broker’s shelf at that rate is roughly eight days of wall time for one question.\nThe pipeline is not broken. The right answer came from the right document with a page behind it. What is broken is the ratio. Four calls out of five did no work, and that share only rises as the shelf grows.\n## 5. What the follow-ups build, and what stays out of scope\nThree articles follow this one, one per shape. They can be read in any order, because the shapes do not depend on each other.\n**The pile.** Article 14B (_RAG on a folder of unrelated PDFs: one summary line per file is the whole index_) takes the collection with no shared field. It needs the least new machinery of the three: the summary lines and the outlines are the whole index, and retrieval is the hierarchical routing of Part III run one level higher.\n**The typed corpus.** Article 14C (the homogeneous corpus, where the business already knows the columns) takes the collection where question two came back yes. Its claim is the one that saves the most time in practice: on a homogeneous corpus you do not discover the schema, the business already holds it, and the interview that produces the columns takes an afternoon.\n**The case file.** Article 14D (case files, one bundle about a single entity) takes the third shape. It has the least prior art of the three in the RAG literature, because its two hardest questions are not retrieval questions. Does piece A contradict piece B, and which expected piece is not here.\nTwo things stay outside this group and go to follow-up work.\n**Formats other than PDF.** Word, Excel, PowerPoint and mail each need their own parser before any of this applies. The three shapes carry over unchanged; the parsing does not.\n**Document comparison.** Asking what changed between version 1 and version 2 of a policy is a different operation from asking a question of a collection. The version chain is in scope here, because retrieval has to know which version answers. The diff itself is not.\nCase files are often kept out of a corpus pipeline as an oddity. They are common enough in insurance, lending and healthcare that leaving them out means leaving out most of the work, which is why they get an article rather than a caveat.\n## 6. Conclusion\nA folder is not a corpus, and a corpus is not one thing. Three questions decide which of three shapes you have, they cost one conversation with one business user, and they are worth asking before a line of ingestion code is written.\nWhat they buy is small and specific. Question one tells you whether you need relations and validity dates. Question two tells you whether there is a table to build. Question three tells you whether the row of that table is a document or a case.\nThe for-loop from section 4 survives all three answers. Every article in this group is about the same thing: shrinking what that loop iterates over, from every document on the shelf down to the handful that could plausibly answer.\n## 7. Further reading and sources\nEarlier in the series, on the pieces this article stands on:\n  * [Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/). The pile shape leans on each file’s table of contents, and this is what to do about the files that ship none.\n  * [Most RAG Hallucinations Are Retrieval Failures: Fix Retrieval, Not the Prompt](https://towardsdatascience.com/most-rag-hallucinations-are-retrieval-failures-how-the-retrieval-brick-decides-what-the-model-can-invent/). The single-document version of the argument in section 1: what the model can invent is decided upstream of the prompt.\n  * [Context engineering for RAG question parsing: from a raw question to typed fields that steer retrieval and generation](https://towardsdatascience.com/context-engineering-for-rag-question-parsing-from-a-raw-question-to-typed-fields-that-steer-retrieval-and-generation/). Where the typed fields in row 2 of the mirror table come from.\n  * [A Production RAG Pipeline in Action: Every Answer Typed and Cited](https://towardsdatascience.com/one-rag-pipeline-four-very-different-pdfs-same-four-bricks-every-answer-typed-and-cited/). The single-document pipeline that the for-loop in section 4 calls once per file.\n  * [Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations](https://towardsdatascience.com/prompt-engineering-isnt-enough-how-four-bricks-of-context-engineering-stop-rag-hallucinations/). The four bricks the mirror table widens, on one document.\n  * [Loop Engineering for RAG Generation: Iterate top-k One at a Time](https://towardsdatascience.com/loop-engineering-for-rag-generation-when-top-1-is-enough-when-you-need-top-k/). The same loop-or-batch decision, inside a document rather than across a folder.\n\n\nAlso referenced above, listed here without links: Article 1 (the four-brick baseline, a PDF in and a highlighted answer out), Article 2bis (what a cross-encoder adds over bi-encoder embeddings, measured), Article 14B (the folder of unrelated PDFs), Article 14C (the homogeneous corpus), Article 14D (case files), and Article 17 (querying a corpus, with the time window as part of the scope).\nExternal sources:\n  * Yunfan Gao et al., _Retrieval-Augmented Generation for Large Language Models: A Survey_ , 2023 ([arXiv 2312.10997](https://arxiv.org/abs/2312.10997)). The naive, advanced, modular vocabulary the baseline in section 4 sits against.\n  * Scott Barnett et al., _Seven Failure Points When Engineering a Retrieval Augmented Generation System_ , 2024 ([arXiv 2401.05856](https://arxiv.org/abs/2401.05856)). Field-measured failure points from three production systems; the missing-document and wrong-document ones are the first two rows of the table in section 1.2.\n  * Anthropic, _Introducing Contextual Retrieval_ , 2024 ([anthropic.com](https://www.anthropic.com/news/contextual-retrieval)). Measured evidence that even a strong embedding and BM25 hybrid degrades once passages lose their document context.\n  * Darren Edge et al., _From Local to Global: A Graph RAG Approach to Query-Focused Summarization_ , 2024 ([arXiv 2404.16130](https://arxiv.org/abs/2404.16130)). The opposite bet on shape two: let a model discover the entities instead of asking the business to name the columns.\n  * Jinyang Li et al., _Can LLM Already Serve as A Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs_ , 2023 ([arXiv 2305.03111](https://arxiv.org/abs/2305.03111)). What querying the typed shape looks like once the table exists, measured on 95 real databases.\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm](https://towardsdatascience.com/tag/llm/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)[Retrieval Augmented Generation](https://towardsdatascience.com/tag/retrieval-augmented-generation-2/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F&title=Three%20Kinds%20of%20RAG%20Corpus%2C%20and%20What%20It%20Costs%20to%20Build%20for%20the%20Wrong%20One)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F&title=Three%20Kinds%20of%20RAG%20Corpus%2C%20and%20What%20It%20Costs%20to%20Build%20for%20the%20Wrong%20One)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthree-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one%2F&text=Three%20Kinds%20of%20RAG%20Corpus%2C%20and%20What%20It%20Costs%20to%20Build%20for%20the%20Wrong%20One)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/01/bruce-hong-asdR5r-2JxY-unsplash-scaled-1.jpg)](https://towardsdatascience.com/using-local-llms-to-discover-high-performance-algorithms/)\n## [Using Local LLMs to Discover High-Performance Algorithms](https://towardsdatascience.com/using-local-llms-to-discover-high-performance-algorithms/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nHow I used open-source models to explore new frontiers in efficient code generation, using my MacBook and local LLMs\n[Stefano Bosisio](https://towardsdatascience.com/author/stefanobosisio1/)\nJanuary 19, 20268 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/Context-Graph.jpg)](https://towardsdatascience.com/vector-rag-isnt-enough-i-built-a-context-graph-layer-for-multi-agent-memory/)\n## [Vector RAG Isn’t Enough — I Built a Context Graph Layer for Multi-Agent Memory](https://towardsdatascience.com/vector-rag-isnt-enough-i-built-a-context-graph-layer-for-multi-agent-memory/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nA structured context graph beat a flat chat dump and a vector-only RAG pipeline on the same multi-agent conversations — full working code, real benchmark numbers, zero API calls\n[Emmimal P Alexander](https://towardsdatascience.com/author/emmimalp.alexander/)\nJune 26, 202617 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/10/image-369.jpg)](https://towardsdatascience.com/how-to-apply-powerful-ai-audio-models-to-real-world-applications/)\n## [How to Apply Powerful AI Audio Models to Real-World Applications](https://towardsdatascience.com/how-to-apply-powerful-ai-audio-models-to-real-world-applications/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLearn about different types of AI audio models and the application areas they can be used in.\n[Eivind Kjosbakken](https://towardsdatascience.com/author/oieivind/)\nOctober 27, 20257 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/18T5JV9SpEDjyLaKQPLTngw.jpeg)](https://towardsdatascience.com/the-next-frontier-in-llm-accuracy-cb2491a740d4/)\n## [The Next Frontier in LLM Accuracy](https://towardsdatascience.com/the-next-frontier-in-llm-accuracy-cb2491a740d4/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nExploring the Power of Lamini Memory Tuning\n[Mariya Mansurova](https://towardsdatascience.com/author/miptgirl/)\nJanuary 4, 202519 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/07/image_2025-07-13_17-25-46.png)](https://towardsdatascience.com/the-future-of-ai-agent-communication-with-acp/)\n## [The Future of AI Agent Communication with ACP](https://towardsdatascience.com/the-future-of-ai-agent-communication-with-acp/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA practical guide to connecting and coordinating multiple AI agents\n[Mariya Mansurova](https://towardsdatascience.com/author/miptgirl/)\nJuly 15, 202516 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/matryoshka-dolls-8228919_1280.jpg)](https://towardsdatascience.com/artificial-intelligence-machine-learning-deep-learning-and-generative-ai-clearly-explained/)\n## [Artificial Intelligence, Machine Learning, Deep Learning, and Generative AI — Clearly Explained](https://towardsdatascience.com/artificial-intelligence-machine-learning-deep-learning-and-generative-ai-clearly-explained/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUnderstanding AI in 2026 — from machine learning to generative models\n[Sabrine Bendimerad](https://towardsdatascience.com/author/sabrine.bendimerad1/)\nDecember 7, 202510 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/1wO9_v_1pUsZKsy6nktd0ug.png)](https://towardsdatascience.com/improve-your-rag-context-recall-by-40-with-an-adapted-embedding-model-5d4a8f583f32/)\n## [Improve Your RAG Context Recall by 95% with an Adapted Embedding Model.](https://towardsdatascience.com/improve-your-rag-context-recall-by-40-with-an-adapted-embedding-model-5d4a8f583f32/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nStep by Step Model Adaptation Code and Results Attached.\n[Vignesh Baskaran](https://towardsdatascience.com/author/vignesh865/)\nOctober 12, 202410 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/05/Knowledge-base-AI-models.jpg)](https://towardsdatascience.com/how-to-build-an-efficient-knowledge-base-for-ai-models/)\n## [How to Build an Efficient Knowledge Base for AI Models](https://towardsdatascience.com/how-to-build-an-efficient-knowledge-base-for-ai-models/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nBuilding a knowledge base for AI models isn’t a one-time task but an iterative process of refinement.\n[Nidhin Karunakaran Ponon](https://towardsdatascience.com/author/nidhin.karunakaranponon/)\nMay 4, 202615 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/toll_plaza_aerial_34282750_card.jpg)](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n## [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 13, 202611 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "body_base_path": "attachments/4fb8938e8405/pages/",
  "body_markdown_path": "attachments/4fb8938e8405/pages/archived-body.md",
  "local_html": "attachments/4fb8938e8405/pages/archived-body.html",
  "body_source": "prior_capture"
}