{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Angela Shi & Kezhan Shi",
  "title_listed": "Cut an Enterprise RAG Pipeline's Latency and Cost by Calling the LLM Less",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/",
  "publish_date": "2026-08-13",
  "corpus_stage": "",
  "notes": "",
  "id": "1e6ce4d0981f",
  "collected_at": "2026-10-02T01:12:23",
  "title": "Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model\nEnterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 13, 202611 min read\n![](../images/000.jpg)\n[Photo](https://www.pexels.com/photo/34282750/) by Chen EdisoN, via Pexels.\nMore and more, RAG pipelines are built the agentic way: put a model wherever a decision needs judgment, and let it decide. That means a lot of model calls. The pipeline from [Article 9](https://towardsdatascience.com/a-production-rag-pipeline-for-pdfs-relational-parsing-toc-retrieval-typed-answers/) (a production RAG pipeline for PDFs) is built exactly like that. Ask _“What is the annual premium?”_ and it runs three model calls in series:\n  * one to parse the question,\n  * one to arbitrate the retrieved candidates,\n  * one to generate the typed answer.\n\n\nEach call buys something real. The parsing call absorbs typos and vague phrasing before retrieval runs. The arbiter call keeps a wrong page from reaching generation. The generation call returns a typed answer with a quote you can check on the page. On a hard question, these three calls are what make the answer trustworthy, and that is why Article 9 put them there.\nThe problem is that we cannot afford this on every question, because it is slow. Three calls in series means the user waits about two seconds, and pays tokens, on every question, easy ones included. And many enterprise questions were never hard: _“What is the annual premium?”_ has one answer, on one line, and the deterministic keyword match had already isolated it on the first pass. So this article adds a router: a cheap per-question signal, read before any model call, that sends easy questions down the fast path and keeps all three calls for the hard ones. This article is the map of that routing: the signal (a score the pipeline already computes), the margin that makes it honest, what it saves (about two seconds on a keyword match), and the questions it must refuse to fast-path. Every number is a real run on the fictional broker corpus, reproducible in the companion notebook.\n🧭 **New to the series? Start with the map** : [Prompt, Context, Loop](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/) sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n![](../images/001.png)\n_Where this article sits: a Part III companion to Article 9, optimizing the pipeline it built - Image by author_\n📓 **The runnable notebook for this article is on GitHub** : [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1). It runs the router on the broker corpus, prints the per-question confidence score, and shows which questions skip the model and which keep it.\n## 1. Every question pays for the whole pipeline\nThe upgraded pipeline is worth its cost on hard questions. The problem is that it charges the same cost on easy ones. Trace a single question through it and the model calls stack up: one to normalize the question and pull its keywords, one for the arbiter that ranks the retrieved candidates, one to generate the typed answer. Three round-trips to a hosted model, in series, before the user sees a word.\nOn a hard question that is money well spent. On _“what is the premium?”_ it is three network hops to return a value that a keyword match had already isolated on the first pass. The latency is real and it is the part the user feels. The token cost is smaller per call, but it is billed on every question, forever, and the easy questions are the ones users repeat most.\nBefore any of those three calls fire, the pipeline can make one decision with no model call, from the parsed question and the document’s `line_df`: can this question be answered on the keyword path alone?\nThe decision needs a signal. It has to be cheap, because the whole point is to spend nothing when the answer is easy, and it has to be honest, because a router that sends a hard question down the fast path returns a confident wrong answer, the exact failure the series works to avoid.\n## 2. Not every question needs the model\nThe signal is already in the pipeline. Retrieval scores each line by how strongly the question’s keywords co-occur on it, and that score, before any arbiter or generation call, already separates the questions that answer themselves from the ones that do not.\n### 2.1 A question that answers itself\nTake _“what is the annual premium?”_ against a homeowner’s policy. The deterministic keyword scorer from Article 7 (retrieval as filtering on `line_df`), `co_occurrence_score`, counts how many of the question’s keywords land on each line, weighting a primary term (`premium`) together with the co-signals that mark a real answer (`EUR`, `annual`, `payable`). Run it and one line wins outright.\nThe top line, _“The annual premium is EUR 1,200, payable…”_ , scores 5. Every other line scores 0. One clear winner, a margin of 5 over the runner-up, and the winning line already carries the shape of the answer, a currency and an amount. There is nothing for the model to disambiguate. The keyword path has answered the question, and a generation call here would only reformat a value that is already isolated.\n### 2.2 A question that needs the model\nNow ask _“which guarantees can I avoid in my case?”_ against the same policy. The keyword scorer runs the same way, but the result is different in kind.\nThree lines tie at a score of 2: the two optional guarantees and the line that mentions adjusting them. The margin between first and second is zero. No single line stands out, because the question is not asking for a line. _“In my case”_ asks the model to weigh the user’s situation against several optional guarantees and reason about which ones are safe to drop. That is generation work, and the flat score is the tell: the keyword path found candidates, not an answer.\n![](../images/body-de95aff2c3599cd6f2e0.png)\n_One clear winner routes past the model; a flat score routes to it - Image by author_\n## 3. The signal is the margin\nThe router does not need a new model. It needs the number the retrieval brick already produced. Two features of the keyword scores decide the route: the top score, and the margin between the top line and the next one.\nA high top score with a wide margin, Q1’s 5 against 0, means one line matched strongly and nothing else came close. The answer is on that line. Route it to a deterministic extractor that pulls the value and skips the model entirely. A low top score, or a top score with no margin, Q2’s flat 2s, means the keywords spread across several lines without settling. Send that question through the full pipeline: arbiter, then generation, the model calls justified on exactly the questions that need them.\nThe whole router is that decision, and it needs no model of its own. Score the lines, read the top score and its margin, and return a route.\npython\n\n```\n# co_occurrence_score: the keyword scorer from Article 7, in the companion notebook.\ndef route_question(line_df, primary, secondary, *, min_score=4, min_margin=3):\n    \"\"\"Decide, with no model call, whether the keyword path already answers.\"\"\"\n    scores = [co_occurrence_score(t, primary, secondary) for t in line_df[\"text\"]]\n    top, second = sorted(scores, reverse=True)[:2]\n\n    # The signal is the retrieval brick's own output: a high top score with a\n    # clear margin means one line answered; a flat score means it did not.\n    confident = top >= min_score and (top - second) >= min_margin\n\n    # \"fast\" skips the model. \"full\" runs the arbiter and generation.\n    return \"fast\" if confident else \"full\"\n```\n\nThe threshold is not a universal constant. What counts as a confident margin depends on the domain: the vocabulary an insurer uses, how templated the questions are, how many lines a typical answer spans. That is the honest limit of this technique. The signal is cheap and general, but the cutoff is a business decision, tuned per corpus against a labelled set the way Article 20 (evaluation) measures every other brick. What travels is the shape: read the confidence the retrieval brick already computed, and spend the model only when it is low.\nThis also composes with the dispatcher from Article 6C (dispatching the parsed question: chunk strategy, model tier, activations). The dispatcher already decides, per question, which bricks to activate. The route is one more activation: a `skip_generation` flag the dispatcher sets when the keyword margin clears the bar, alongside the ones it already sets for chunk strategy and model tier. The router is not a new stage bolted on: it is the dispatcher learning to say “this one does not need the model.”\n## 4. What it saves: two seconds, or nothing\nThe saving is the whole point, so measure it. The fast path is deterministic: score the lines, read the margin, return an answer. Timed on the broker corpus, the entire routing decision runs in about **0.1 millisecond** , with no network call at all.\nThe full pipeline is three hosted-model calls in series. Question parsing, the retrieval arbiter, and generation each cost a few hundred milliseconds to over a second, and they run one after another. For an easy question that adds up to roughly **two seconds** of the user watching a spinner, to return a number a keyword match had already isolated.\n![](../images/body-9fe6e215859bce105d15.png)\n_Two seconds through the full pipeline, or a tenth of a millisecond routed past it - Image by author_\nTwo seconds against a tenth of a millisecond is a different order of magnitude, not a tuning gain, and it lands on exactly the questions users ask most: the simple, templated ones. The cost side moves the same way. Every routed question is three model calls not billed, and at a support desk running the same fifty templated questions across ten thousand contracts, those easy questions are most of the traffic. So the fast path is close to free as well as fast.\n## 5. Indicators, not a bigger model\nThe margin is one indicator. It is the cheapest one and it settles the clean cases, but it is not the whole optimization. Making a thorough pipeline fast is a matter of finding, for each decision the pipeline currently hands to a model, a signal that decides it without one. There are several fronts, and each is real work.\n### 5.1 The fronts\n**The answer that already exists.** Most enterprise questions get asked again. If a question matches one already answered, on a document that has not changed, the answer is on file: return it, with no retrieval and no model. The catch is the match itself. Deciding “this is the same question” is its own classification: normalize the wording, compare the keywords, compare the question’s embedding against a store of past questions, and accept only a close hit. That store is the templated-question base from Article 6C (dispatching the parsed question) and the corpus tables from Article 25 (the storage model); the indicator reads them.\n**The retrieval that already answered.** The signal this article uses. Score the lines, and when one clearly carries the answer’s shape, the answer is found, and generation has nothing left to do.\n**The question type, from the expert dictionary.** To know a question wants one line and not a paragraph, the pipeline needs its answer shape, and today a model reads it. But for a _known_ concept the shape is already written down: the dictionary maps `premium` to `(single, amount)`. Look the concept up and you have the type deterministically, no model. The model is left for the genuinely new question the dictionary has never seen. This is _amplify the expert_ moved to the control flow: the expert’s dictionary decides the route, not the LLM.\n### 5.2 Where the indicators end and the agent begins\nTwo boundaries sit at the edge of this. The full engineering of latency and cost, the small-model tiers, the caches, context reduction, streaming, and the measurement of cost-per-query and tail latency, is Article 21 (cost and latency). This article works the single highest lever, not calling the model, and points there for the rest.\nThe other boundary is agentic RAG, and it falls on one precise line: who makes the route decision. Made by an indicator, a keyword score, a cache hit, a dictionary lookup, the decision is deterministic, and it belongs to the Vol.1 engineering this article is part of. Made by the model, classifying the intent and planning the steps, it is what makes a system agentic. The agentic follow-up to this series carries the same deterministic-first, model-only-if-ambiguous pattern up a level, to picking a tool rather than just a route. The two are not rivals but layers: the deterministic indicators handle the easy majority for free, and the agent is the fallback for the questions the indicators cannot classify. Building indicators that are fast, and often just classical rules or a small trained model, is how you keep the agent, and its latency, for the cases that truly need it.\n## 6. Conclusion\nThe reflex when a RAG pipeline feels slow is to reach for a faster model. The cheaper and larger win is to call the model less. The retrieval brick already computes, for free, a signal that separates the questions answered by a keyword match from the ones that need the model to reason: the top keyword score and its margin. Route on it. Send _“what is the premium?”_ , one line at score 5, straight to a deterministic extractor, and keep the model for _“which guarantees can I avoid in my case?”_ , where the flat score is the pipeline telling you it needs help.\nThe threshold is yours to tune, per corpus, against a labelled set. The mechanism is portable, and it is the same lesson the series keeps landing on: the expert’s keyword work is not a fallback under the model, it is often the whole answer, and knowing when it suffices is what keeps the pipeline fast.\n## 7. Further reading and sources\n**Earlier in the series:**\n  * [A production RAG pipeline for PDFs: relational parsing, TOC retrieval, typed answers](https://towardsdatascience.com/a-production-rag-pipeline-for-pdfs-relational-parsing-toc-retrieval-typed-answers/). The upgraded pipeline this article optimizes, brick by brick.\n  * Dispatching the parsed RAG question: chunk strategy, model tier, activations, audit. The dispatcher the router extends with a skip-the-model activation.\n  * Retrieval is filtering, not search: a mental model for enterprise RAG. Where `co_occurrence_score` comes from: keyword filtering on `line_df`.\n\n\n_Sources_\n  * The cascade idea, run cheap stages first and stop as soon as one is confident, is the classic detection cascade of Viola and Jones ([Rapid Object Detection using a Boosted Cascade of Simple Features](https://doi.org/10.1109/CVPR.2001.990517), CVPR 2001), applied here to model calls instead of image windows.\n  * FrugalGPT (Chen, Zaharia, and Zou, [arXiv:2305.05176](https://arxiv.org/abs/2305.05176), 2023) makes the same move for LLM APIs: a cascade that queries a cheap model first and escalates only when the cheap answer is not confident.\n  * RouteLLM (Ong et al., [arXiv:2406.18665](https://arxiv.org/abs/2406.18665), 2024) learns a router that sends easy queries to a small model and hard ones to a large one, the same triage this article applies one step earlier, before any model at all.\n\n\nThe broader cost-and-latency treatment is Article 21 (cost and latency).\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-249d024535392fb76067.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm](https://towardsdatascience.com/tag/llm/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)[Token Management](https://towardsdatascience.com/tag/token-management/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F&title=Cut%20an%20Enterprise%20RAG%20Pipeline%E2%80%99s%20Latency%20and%20Cost%20by%20Calling%20the%20LLM%20Less%2C%20Not%20by%20Buying%20a%20Faster%20Model)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F&title=Cut%20an%20Enterprise%20RAG%20Pipeline%E2%80%99s%20Latency%20and%20Cost%20by%20Calling%20the%20LLM%20Less%2C%20Not%20by%20Buying%20a%20Faster%20Model)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F&text=Cut%20an%20Enterprise%20RAG%20Pipeline%E2%80%99s%20Latency%20and%20Cost%20by%20Calling%20the%20LLM%20Less%2C%20Not%20by%20Buying%20a%20Faster%20Model)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/002.jpeg)](https://towardsdatascience.com/10-ways-to-improve-the-performance-of-retrieval-augmented-generation-systems-5fa2cee7cd5c/)\n## [10 Ways to Improve the Performance of Retrieval Augmented Generation Systems](https://towardsdatascience.com/10-ways-to-improve-the-performance-of-retrieval-augmented-generation-systems-5fa2cee7cd5c/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nTools to go from prototype to production\n[Matt Ambrogi](https://towardsdatascience.com/author/mattambrogi/)\nSeptember 18, 20239 min read\n  * [![](../images/003.png)](https://towardsdatascience.com/ai-agent-with-multi-session-memory/)\n## [AI Agent with Multi-Session Memory](https://towardsdatascience.com/ai-agent-with-multi-session-memory/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nBuild from scratch using only Python & Ollama (no GPU, no APIKEY)\n[Mauro Di Pietro](https://towardsdatascience.com/author/maurodp/)\nJune 26, 20258 min read\n  * [![](../images/004.png)](https://towardsdatascience.com/clip-model-overview-unlocking-the-power-of-multimodal-ai/)\n## [CLIP Model Overview : Unlocking the Power of Multimodal AI](https://towardsdatascience.com/clip-model-overview-unlocking-the-power-of-multimodal-ai/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThe magic behind multimodal models unlocked through contrastive learning\n[Slava Efimov](https://towardsdatascience.com/author/slavahead/)\nJuly 14, 20256 min read\n  * [![](../images/005.jpg)](https://towardsdatascience.com/vector-rag-isnt-enough-i-built-a-context-graph-layer-for-multi-agent-memory/)\n## [Vector RAG Isn’t Enough — I Built a Context Graph Layer for Multi-Agent Memory](https://towardsdatascience.com/vector-rag-isnt-enough-i-built-a-context-graph-layer-for-multi-agent-memory/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nA structured context graph beat a flat chat dump and a vector-only RAG pipeline on the same multi-agent conversations — full working code, real benchmark numbers, zero API calls\n[Emmimal P Alexander](https://towardsdatascience.com/author/emmimalp.alexander/)\nJune 26, 202617 min read\n  * [![](../images/006.jpg)](https://towardsdatascience.com/ai-in-social-research-and-polling/)\n## [AI in Social Research and Polling](https://towardsdatascience.com/ai-in-social-research-and-polling/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat do we do when nobody answers the phone?\n[Stephanie Kirmer](https://towardsdatascience.com/author/s.kirmer/)\nApril 1, 202511 min read\n  * [![](../images/007.jpeg)](https://towardsdatascience.com/why-90-accuracy-in-text-to-sql-is-100-useless/)\n## [Why 90% Accuracy in Text-to-SQL is 100% Useless](https://towardsdatascience.com/why-90-accuracy-in-text-to-sql-is-100-useless/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nThe eternal promise of self-service analytics\n[Gary Zavaleta](https://towardsdatascience.com/author/gary.zavaleta/)\nJanuary 12, 20268 min read\n  * [![](../images/008.jpg)](https://towardsdatascience.com/how-to-fine-tune-an-llm-an-end-to-end-guide/)\n## [How to Fine-Tune an LLM: An End-to-End Guide](https://towardsdatascience.com/how-to-fine-tune-an-llm-an-end-to-end-guide/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nA hands-on guide to fine-tuning LLMs for the real world\n[Sam Black](https://towardsdatascience.com/author/sam-black/)\nAugust 20, 202626 min read\n  * [![](../images/009.jpg)](https://towardsdatascience.com/three-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one/)\n## [Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One](https://towardsdatascience.com/three-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #14A] - Three questions tell you which shape a document collection has, and each shape wants a different architecture\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 20, 202617 min read\n  * [![](../images/010.jpeg)](https://towardsdatascience.com/a-comprehensive-guide-to-collaborative-ai-agents-in-practice-1f4048947d9c/)\n## [A Comprehensive Guide to Collaborative AI Agents in Practice](https://towardsdatascience.com/a-comprehensive-guide-to-collaborative-ai-agents-in-practice-1f4048947d9c/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nthe definition, and building a team of agents that refine your CV and Cover Letter for job applications\n[Hesam Sheikh](https://towardsdatascience.com/author/itshesamsheikh/)\nJuly 3, 202414 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "metadata": {
    "title": "Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model | Towards Data Science",
    "description": "Enterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.",
    "keywords": null,
    "author": "Angela and Kezhan Shi",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model",
    "og:description": "Enterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.",
    "og:url": "https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/toll_plaza_aerial_34282750_card.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Angela and Kezhan Shi",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "11 min read",
    "article:published_time": "2026-08-13T10:35:01.970Z",
    "article:modified_time": "2026-08-13T10:35:01.970Z",
    "article:author": "https://towardsdatascience.com/author/angela.shi/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/1e6ce4d0981f/code/block_000.txt",
      "chars": 712
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:55:56Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:55:56Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:55:56Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:55:56Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/large-language-model/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/angela.shi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/a-production-rag-pipeline-for-pdfs-relational-parsing-toc-retrieval-typed-answers/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/document-processing/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/llm/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/rag/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/rag-architecture/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/token-management/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/10-ways-to-improve-the-performance-of-retrieval-augmented-generation-systems-5fa2cee7cd5c/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/large-language-models/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/mattambrogi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/ai-agent-with-multi-session-memory/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://www.pexels.com/photo/34282750/"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://github.com/doc-intel/notebooks-vol1"
    },
    {
      "type": "external",
      "href": "https://doi.org/10.1109/CVPR.2001.990517"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2305.05176"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2406.18665"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Cut+an+Enterprise+RAG+Pipeline%E2%80%99s+Latency+and+Cost+by+Calling+the+LLM+Less%2C+Not+by+Buying+a+Faster+Model&u=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Cut+an+Enterprise+RAG+Pipeline%E2%80%99s+Latency+and+Cost+by+Calling+the+LLM+Less%2C+Not+by+Buying+a+Faster+Model&url=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Cut+an+Enterprise+RAG+Pipeline%E2%80%99s+Latency+and+Cost+by+Calling+the+LLM+Less%2C+Not+by+Buying+a+Faster+Model&url=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh"
    },
    {
      "type": "external",
      "href": "https://insightmediagroup.io/creator-network"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:28:34Z",
  "source_url": "https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model\nEnterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 13, 202611 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/toll_plaza_aerial_34282750_card.jpg)\n[Photo](https://www.pexels.com/photo/34282750/) by Chen EdisoN, via Pexels.\nMore and more, RAG pipelines are built the agentic way: put a model wherever a decision needs judgment, and let it decide. That means a lot of model calls. The pipeline from [Article 9](https://towardsdatascience.com/a-production-rag-pipeline-for-pdfs-relational-parsing-toc-retrieval-typed-answers/) (a production RAG pipeline for PDFs) is built exactly like that. Ask _“What is the annual premium?”_ and it runs three model calls in series:\n  * one to parse the question,\n  * one to arbitrate the retrieved candidates,\n  * one to generate the typed answer.\n\n\nEach call buys something real. The parsing call absorbs typos and vague phrasing before retrieval runs. The arbiter call keeps a wrong page from reaching generation. The generation call returns a typed answer with a quote you can check on the page. On a hard question, these three calls are what make the answer trustworthy, and that is why Article 9 put them there.\nThe problem is that we cannot afford this on every question, because it is slow. Three calls in series means the user waits about two seconds, and pays tokens, on every question, easy ones included. And many enterprise questions were never hard: _“What is the annual premium?”_ has one answer, on one line, and the deterministic keyword match had already isolated it on the first pass. So this article adds a router: a cheap per-question signal, read before any model call, that sends easy questions down the fast path and keeps all three calls for the hard ones. This article is the map of that routing: the signal (a score the pipeline already computes), the margin that makes it honest, what it saves (about two seconds on a keyword match), and the questions it must refuse to fast-path. Every number is a real run on the fictional broker corpus, reproducible in the companion notebook.\n🧭 **New to the series? Start with the map** : [Prompt, Context, Loop](https://towardsdatascience.com/prompt-context-loop-the-three-engineering-layers-every-rag-system-is-built-on/) sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-42-1024x572.png)\n_Where this article sits: a Part III companion to Article 9, optimizing the pipeline it built - Image by author_\n📓 **The runnable notebook for this article is on GitHub** : [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1). It runs the router on the broker corpus, prints the per-question confidence score, and shows which questions skip the model and which keep it.\n## 1. Every question pays for the whole pipeline\nThe upgraded pipeline is worth its cost on hard questions. The problem is that it charges the same cost on easy ones. Trace a single question through it and the model calls stack up: one to normalize the question and pull its keywords, one for the arbiter that ranks the retrieved candidates, one to generate the typed answer. Three round-trips to a hosted model, in series, before the user sees a word.\nOn a hard question that is money well spent. On _“what is the premium?”_ it is three network hops to return a value that a keyword match had already isolated on the first pass. The latency is real and it is the part the user feels. The token cost is smaller per call, but it is billed on every question, forever, and the easy questions are the ones users repeat most.\nBefore any of those three calls fire, the pipeline can make one decision with no model call, from the parsed question and the document’s `line_df`: can this question be answered on the keyword path alone?\nThe decision needs a signal. It has to be cheap, because the whole point is to spend nothing when the answer is easy, and it has to be honest, because a router that sends a hard question down the fast path returns a confident wrong answer, the exact failure the series works to avoid.\n## 2. Not every question needs the model\nThe signal is already in the pipeline. Retrieval scores each line by how strongly the question’s keywords co-occur on it, and that score, before any arbiter or generation call, already separates the questions that answer themselves from the ones that do not.\n### 2.1 A question that answers itself\nTake _“what is the annual premium?”_ against a homeowner’s policy. The deterministic keyword scorer from Article 7 (retrieval as filtering on `line_df`), `co_occurrence_score`, counts how many of the question’s keywords land on each line, weighting a primary term (`premium`) together with the co-signals that mark a real answer (`EUR`, `annual`, `payable`). Run it and one line wins outright.\nThe top line, _“The annual premium is EUR 1,200, payable…”_ , scores 5. Every other line scores 0. One clear winner, a margin of 5 over the runner-up, and the winning line already carries the shape of the answer, a currency and an amount. There is nothing for the model to disambiguate. The keyword path has answered the question, and a generation call here would only reformat a value that is already isolated.\n### 2.2 A question that needs the model\nNow ask _“which guarantees can I avoid in my case?”_ against the same policy. The keyword scorer runs the same way, but the result is different in kind.\nThree lines tie at a score of 2: the two optional guarantees and the line that mentions adjusting them. The margin between first and second is zero. No single line stands out, because the question is not asking for a line. _“In my case”_ asks the model to weigh the user’s situation against several optional guarantees and reason about which ones are safe to drop. That is generation work, and the flat score is the tell: the keyword path found candidates, not an answer.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-41-1024x394.png)\n_One clear winner routes past the model; a flat score routes to it - Image by author_\n## 3. The signal is the margin\nThe router does not need a new model. It needs the number the retrieval brick already produced. Two features of the keyword scores decide the route: the top score, and the margin between the top line and the next one.\nA high top score with a wide margin, Q1’s 5 against 0, means one line matched strongly and nothing else came close. The answer is on that line. Route it to a deterministic extractor that pulls the value and skips the model entirely. A low top score, or a top score with no margin, Q2’s flat 2s, means the keywords spread across several lines without settling. Send that question through the full pipeline: arbiter, then generation, the model calls justified on exactly the questions that need them.\nThe whole router is that decision, and it needs no model of its own. Score the lines, read the top score and its margin, and return a route.\npython\n\n```\n# co_occurrence_score: the keyword scorer from Article 7, in the companion notebook.\ndef route_question(line_df, primary, secondary, *, min_score=4, min_margin=3):\n    \"\"\"Decide, with no model call, whether the keyword path already answers.\"\"\"\n    scores = [co_occurrence_score(t, primary, secondary) for t in line_df[\"text\"]]\n    top, second = sorted(scores, reverse=True)[:2]\n\n    # The signal is the retrieval brick's own output: a high top score with a\n    # clear margin means one line answered; a flat score means it did not.\n    confident = top >= min_score and (top - second) >= min_margin\n\n    # \"fast\" skips the model. \"full\" runs the arbiter and generation.\n    return \"fast\" if confident else \"full\"\n```\n\nThe threshold is not a universal constant. What counts as a confident margin depends on the domain: the vocabulary an insurer uses, how templated the questions are, how many lines a typical answer spans. That is the honest limit of this technique. The signal is cheap and general, but the cutoff is a business decision, tuned per corpus against a labelled set the way Article 20 (evaluation) measures every other brick. What travels is the shape: read the confidence the retrieval brick already computed, and spend the model only when it is low.\nThis also composes with the dispatcher from Article 6C (dispatching the parsed question: chunk strategy, model tier, activations). The dispatcher already decides, per question, which bricks to activate. The route is one more activation: a `skip_generation` flag the dispatcher sets when the keyword margin clears the bar, alongside the ones it already sets for chunk strategy and model tier. The router is not a new stage bolted on: it is the dispatcher learning to say “this one does not need the model.”\n## 4. What it saves: two seconds, or nothing\nThe saving is the whole point, so measure it. The fast path is deterministic: score the lines, read the margin, return an answer. Timed on the broker corpus, the entire routing decision runs in about **0.1 millisecond** , with no network call at all.\nThe full pipeline is three hosted-model calls in series. Question parsing, the retrieval arbiter, and generation each cost a few hundred milliseconds to over a second, and they run one after another. For an easy question that adds up to roughly **two seconds** of the user watching a spinner, to return a number a keyword match had already isolated.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/image-43-1024x377.png)\n_Two seconds through the full pipeline, or a tenth of a millisecond routed past it - Image by author_\nTwo seconds against a tenth of a millisecond is a different order of magnitude, not a tuning gain, and it lands on exactly the questions users ask most: the simple, templated ones. The cost side moves the same way. Every routed question is three model calls not billed, and at a support desk running the same fifty templated questions across ten thousand contracts, those easy questions are most of the traffic. So the fast path is close to free as well as fast.\n## 5. Indicators, not a bigger model\nThe margin is one indicator. It is the cheapest one and it settles the clean cases, but it is not the whole optimization. Making a thorough pipeline fast is a matter of finding, for each decision the pipeline currently hands to a model, a signal that decides it without one. There are several fronts, and each is real work.\n### 5.1 The fronts\n**The answer that already exists.** Most enterprise questions get asked again. If a question matches one already answered, on a document that has not changed, the answer is on file: return it, with no retrieval and no model. The catch is the match itself. Deciding “this is the same question” is its own classification: normalize the wording, compare the keywords, compare the question’s embedding against a store of past questions, and accept only a close hit. That store is the templated-question base from Article 6C (dispatching the parsed question) and the corpus tables from Article 25 (the storage model); the indicator reads them.\n**The retrieval that already answered.** The signal this article uses. Score the lines, and when one clearly carries the answer’s shape, the answer is found, and generation has nothing left to do.\n**The question type, from the expert dictionary.** To know a question wants one line and not a paragraph, the pipeline needs its answer shape, and today a model reads it. But for a _known_ concept the shape is already written down: the dictionary maps `premium` to `(single, amount)`. Look the concept up and you have the type deterministically, no model. The model is left for the genuinely new question the dictionary has never seen. This is _amplify the expert_ moved to the control flow: the expert’s dictionary decides the route, not the LLM.\n### 5.2 Where the indicators end and the agent begins\nTwo boundaries sit at the edge of this. The full engineering of latency and cost, the small-model tiers, the caches, context reduction, streaming, and the measurement of cost-per-query and tail latency, is Article 21 (cost and latency). This article works the single highest lever, not calling the model, and points there for the rest.\nThe other boundary is agentic RAG, and it falls on one precise line: who makes the route decision. Made by an indicator, a keyword score, a cache hit, a dictionary lookup, the decision is deterministic, and it belongs to the Vol.1 engineering this article is part of. Made by the model, classifying the intent and planning the steps, it is what makes a system agentic. The agentic follow-up to this series carries the same deterministic-first, model-only-if-ambiguous pattern up a level, to picking a tool rather than just a route. The two are not rivals but layers: the deterministic indicators handle the easy majority for free, and the agent is the fallback for the questions the indicators cannot classify. Building indicators that are fast, and often just classical rules or a small trained model, is how you keep the agent, and its latency, for the cases that truly need it.\n## 6. Conclusion\nThe reflex when a RAG pipeline feels slow is to reach for a faster model. The cheaper and larger win is to call the model less. The retrieval brick already computes, for free, a signal that separates the questions answered by a keyword match from the ones that need the model to reason: the top keyword score and its margin. Route on it. Send _“what is the premium?”_ , one line at score 5, straight to a deterministic extractor, and keep the model for _“which guarantees can I avoid in my case?”_ , where the flat score is the pipeline telling you it needs help.\nThe threshold is yours to tune, per corpus, against a labelled set. The mechanism is portable, and it is the same lesson the series keeps landing on: the expert’s keyword work is not a fallback under the model, it is often the whole answer, and knowing when it suffices is what keeps the pipeline fast.\n## 7. Further reading and sources\n**Earlier in the series:**\n  * [A production RAG pipeline for PDFs: relational parsing, TOC retrieval, typed answers](https://towardsdatascience.com/a-production-rag-pipeline-for-pdfs-relational-parsing-toc-retrieval-typed-answers/). The upgraded pipeline this article optimizes, brick by brick.\n  * Dispatching the parsed RAG question: chunk strategy, model tier, activations, audit. The dispatcher the router extends with a skip-the-model activation.\n  * Retrieval is filtering, not search: a mental model for enterprise RAG. Where `co_occurrence_score` comes from: keyword filtering on `line_df`.\n\n\n_Sources_\n  * The cascade idea, run cheap stages first and stop as soon as one is confident, is the classic detection cascade of Viola and Jones ([Rapid Object Detection using a Boosted Cascade of Simple Features](https://doi.org/10.1109/CVPR.2001.990517), CVPR 2001), applied here to model calls instead of image windows.\n  * FrugalGPT (Chen, Zaharia, and Zou, [arXiv:2305.05176](https://arxiv.org/abs/2305.05176), 2023) makes the same move for LLM APIs: a cascade that queries a cheap model first and escalates only when the cheap answer is not confident.\n  * RouteLLM (Ong et al., [arXiv:2406.18665](https://arxiv.org/abs/2406.18665), 2024) learns a router that sends easy queries to a small model and hard ones to a large one, the same triage this article applies one step earlier, before any model at all.\n\n\nThe broader cost-and-latency treatment is Article 21 (cost and latency).\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm](https://towardsdatascience.com/tag/llm/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)[Token Management](https://towardsdatascience.com/tag/token-management/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F&title=Cut%20an%20Enterprise%20RAG%20Pipeline%E2%80%99s%20Latency%20and%20Cost%20by%20Calling%20the%20LLM%20Less%2C%20Not%20by%20Buying%20a%20Faster%20Model)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F&title=Cut%20an%20Enterprise%20RAG%20Pipeline%E2%80%99s%20Latency%20and%20Cost%20by%20Calling%20the%20LLM%20Less%2C%20Not%20by%20Buying%20a%20Faster%20Model)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fcut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model%2F&text=Cut%20an%20Enterprise%20RAG%20Pipeline%E2%80%99s%20Latency%20and%20Cost%20by%20Calling%20the%20LLM%20Less%2C%20Not%20by%20Buying%20a%20Faster%20Model)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/1J16KiNF4M_XxPkALZFUJHw-scaled.jpeg)](https://towardsdatascience.com/10-ways-to-improve-the-performance-of-retrieval-augmented-generation-systems-5fa2cee7cd5c/)\n## [10 Ways to Improve the Performance of Retrieval Augmented Generation Systems](https://towardsdatascience.com/10-ways-to-improve-the-performance-of-retrieval-augmented-generation-systems-5fa2cee7cd5c/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nTools to go from prototype to production\n[Matt Ambrogi](https://towardsdatascience.com/author/mattambrogi/)\nSeptember 18, 20239 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/06/image-127.png)](https://towardsdatascience.com/ai-agent-with-multi-session-memory/)\n## [AI Agent with Multi-Session Memory](https://towardsdatascience.com/ai-agent-with-multi-session-memory/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nBuild from scratch using only Python & Ollama (no GPU, no APIKEY)\n[Mauro Di Pietro](https://towardsdatascience.com/author/maurodp/)\nJune 26, 20258 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/07/featured_image-scaled-1.png)](https://towardsdatascience.com/clip-model-overview-unlocking-the-power-of-multimodal-ai/)\n## [CLIP Model Overview : Unlocking the Power of Multimodal AI](https://towardsdatascience.com/clip-model-overview-unlocking-the-power-of-multimodal-ai/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThe magic behind multimodal models unlocked through contrastive learning\n[Slava Efimov](https://towardsdatascience.com/author/slavahead/)\nJuly 14, 20256 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/Context-Graph.jpg)](https://towardsdatascience.com/vector-rag-isnt-enough-i-built-a-context-graph-layer-for-multi-agent-memory/)\n## [Vector RAG Isn’t Enough — I Built a Context Graph Layer for Multi-Agent Memory](https://towardsdatascience.com/vector-rag-isnt-enough-i-built-a-context-graph-layer-for-multi-agent-memory/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nA structured context graph beat a flat chat dump and a vector-only RAG pipeline on the same multi-agent conversations — full working code, real benchmark numbers, zero API calls\n[Emmimal P Alexander](https://towardsdatascience.com/author/emmimalp.alexander/)\nJune 26, 202617 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/04/pawel-czerwinski-0xCCPIbl3M-unsplash.jpg)](https://towardsdatascience.com/ai-in-social-research-and-polling/)\n## [AI in Social Research and Polling](https://towardsdatascience.com/ai-in-social-research-and-polling/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat do we do when nobody answers the phone?\n[Stephanie Kirmer](https://towardsdatascience.com/author/s.kirmer/)\nApril 1, 202511 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/01/image01-scaled-1.jpeg)](https://towardsdatascience.com/why-90-accuracy-in-text-to-sql-is-100-useless/)\n## [Why 90% Accuracy in Text-to-SQL is 100% Useless](https://towardsdatascience.com/why-90-accuracy-in-text-to-sql-is-100-useless/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nThe eternal promise of self-service analytics\n[Gary Zavaleta](https://towardsdatascience.com/author/gary.zavaleta/)\nJanuary 12, 20268 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/google-deepmind-LcgLq78WZCQ-unsplash-scaled-1.jpg)](https://towardsdatascience.com/how-to-fine-tune-an-llm-an-end-to-end-guide/)\n## [How to Fine-Tune an LLM: An End-to-End Guide](https://towardsdatascience.com/how-to-fine-tune-an-llm-an-end-to-end-guide/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nA hands-on guide to fine-tuning LLMs for the real world\n[Sam Black](https://towardsdatascience.com/author/sam-black/)\nAugust 20, 202626 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/forked_road_snow_20061882_v3_card.jpg)](https://towardsdatascience.com/three-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one/)\n## [Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One](https://towardsdatascience.com/three-kinds-of-rag-corpus-and-what-it-costs-to-build-for-the-wrong-one/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #14A] - Three questions tell you which shape a document collection has, and each shape wants a different architecture\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 20, 202617 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1enxKj0VEHcec6IewiDat4g.jpeg)](https://towardsdatascience.com/a-comprehensive-guide-to-collaborative-ai-agents-in-practice-1f4048947d9c/)\n## [A Comprehensive Guide to Collaborative AI Agents in Practice](https://towardsdatascience.com/a-comprehensive-guide-to-collaborative-ai-agents-in-practice-1f4048947d9c/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nthe definition, and building a team of agents that refine your CV and Cover Letter for job applications\n[Hesam Sheikh](https://towardsdatascience.com/author/itshesamsheikh/)\nJuly 3, 202414 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "body_base_path": "attachments/1e6ce4d0981f/pages/",
  "body_markdown_path": "attachments/1e6ce4d0981f/pages/archived-body.md",
  "local_html": "attachments/1e6ce4d0981f/pages/archived-body.html",
  "body_source": "prior_capture"
}