保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Large Language Model

Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model

Enterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match. Angela and Kezhan Shi August 13, 202611 min read Photo by Chen EdisoN, via Pexels. More and more, RAG pipelines are built the agentic way: put a model wherever a decision needs judgment, and let it decide. That means a lot of model calls. The pipeline from Article 9 (a production RAG pipeline for PDFs) is built exactly like that. Ask “What is the annual premium?” and it runs three model calls in series:

Each call buys something real. The parsing call absorbs typos and vague phrasing before retrieval runs. The arbiter call keeps a wrong page from reaching generation. The generation call returns a typed answer with a quote you can check on the page. On a hard question, these three calls are what make the answer trustworthy, and that is why Article 9 put them there. The problem is that we cannot afford this on every question, because it is slow. Three calls in series means the user waits about two seconds, and pays tokens, on every question, easy ones included. And many enterprise questions were never hard: “What is the annual premium?” has one answer, on one line, and the deterministic keyword match had already isolated it on the first pass. So this article adds a router: a cheap per-question signal, read before any model call, that sends easy questions down the fast path and keeps all three calls for the hard ones. This article is the map of that routing: the signal (a score the pipeline already computes), the margin that makes it honest, what it saves (about two seconds on a keyword match), and the questions it must refuse to fast-path. Every number is a real run on the fictional broker corpus, reproducible in the companion notebook. 🧭 New to the series? Start with the map : Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits. Learn this step by step with the interactive AI Engineer roadmap. Where this article sits: a Part III companion to Article 9, optimizing the pipeline it built - Image by author 📓 The runnable notebook for this article is on GitHub : doc-intel/notebooks-vol1. It runs the router on the broker corpus, prints the per-question confidence score, and shows which questions skip the model and which keep it.

1. Every question pays for the whole pipeline

The upgraded pipeline is worth its cost on hard questions. The problem is that it charges the same cost on easy ones. Trace a single question through it and the model calls stack up: one to normalize the question and pull its keywords, one for the arbiter that ranks the retrieved candidates, one to generate the typed answer. Three round-trips to a hosted model, in series, before the user sees a word. On a hard question that is money well spent. On “what is the premium?” it is three network hops to return a value that a keyword match had already isolated on the first pass. The latency is real and it is the part the user feels. The token cost is smaller per call, but it is billed on every question, forever, and the easy questions are the ones users repeat most. Before any of those three calls fire, the pipeline can make one decision with no model call, from the parsed question and the document’s line_df: can this question be answered on the keyword path alone? The decision needs a signal. It has to be cheap, because the whole point is to spend nothing when the answer is easy, and it has to be honest, because a router that sends a hard question down the fast path returns a confident wrong answer, the exact failure the series works to avoid.

2. Not every question needs the model

The signal is already in the pipeline. Retrieval scores each line by how strongly the question’s keywords co-occur on it, and that score, before any arbiter or generation call, already separates the questions that answer themselves from the ones that do not.

2.1 A question that answers itself

Take “what is the annual premium?” against a homeowner’s policy. The deterministic keyword scorer from Article 7 (retrieval as filtering on line_df), co_occurrence_score, counts how many of the question’s keywords land on each line, weighting a primary term (premium) together with the co-signals that mark a real answer (EUR, annual, payable). Run it and one line wins outright. The top line, “The annual premium is EUR 1,200, payable…” , scores 5. Every other line scores 0. One clear winner, a margin of 5 over the runner-up, and the winning line already carries the shape of the answer, a currency and an amount. There is nothing for the model to disambiguate. The keyword path has answered the question, and a generation call here would only reformat a value that is already isolated.

2.2 A question that needs the model

Now ask “which guarantees can I avoid in my case?” against the same policy. The keyword scorer runs the same way, but the result is different in kind. Three lines tie at a score of 2: the two optional guarantees and the line that mentions adjusting them. The margin between first and second is zero. No single line stands out, because the question is not asking for a line. “In my case” asks the model to weigh the user’s situation against several optional guarantees and reason about which ones are safe to drop. That is generation work, and the flat score is the tell: the keyword path found candidates, not an answer. One clear winner routes past the model; a flat score routes to it - Image by author

3. The signal is the margin

The router does not need a new model. It needs the number the retrieval brick already produced. Two features of the keyword scores decide the route: the top score, and the margin between the top line and the next one. A high top score with a wide margin, Q1’s 5 against 0, means one line matched strongly and nothing else came close. The answer is on that line. Route it to a deterministic extractor that pulls the value and skips the model entirely. A low top score, or a top score with no margin, Q2’s flat 2s, means the keywords spread across several lines without settling. Send that question through the full pipeline: arbiter, then generation, the model calls justified on exactly the questions that need them. The whole router is that decision, and it needs no model of its own. Score the lines, read the top score and its margin, and return a route. python

# co_occurrence_score: the keyword scorer from Article 7, in the companion notebook.
def route_question(line_df, primary, secondary, *, min_score=4, min_margin=3):
    """Decide, with no model call, whether the keyword path already answers."""
    scores = [co_occurrence_score(t, primary, secondary) for t in line_df["text"]]
    top, second = sorted(scores, reverse=True)[:2]

    # The signal is the retrieval brick's own output: a high top score with a
    # clear margin means one line answered; a flat score means it did not.
    confident = top >= min_score and (top - second) >= min_margin

    # "fast" skips the model. "full" runs the arbiter and generation.
    return "fast" if confident else "full"

The threshold is not a universal constant. What counts as a confident margin depends on the domain: the vocabulary an insurer uses, how templated the questions are, how many lines a typical answer spans. That is the honest limit of this technique. The signal is cheap and general, but the cutoff is a business decision, tuned per corpus against a labelled set the way Article 20 (evaluation) measures every other brick. What travels is the shape: read the confidence the retrieval brick already computed, and spend the model only when it is low. This also composes with the dispatcher from Article 6C (dispatching the parsed question: chunk strategy, model tier, activations). The dispatcher already decides, per question, which bricks to activate. The route is one more activation: a skip_generation flag the dispatcher sets when the keyword margin clears the bar, alongside the ones it already sets for chunk strategy and model tier. The router is not a new stage bolted on: it is the dispatcher learning to say “this one does not need the model.”

4. What it saves: two seconds, or nothing

The saving is the whole point, so measure it. The fast path is deterministic: score the lines, read the margin, return an answer. Timed on the broker corpus, the entire routing decision runs in about 0.1 millisecond , with no network call at all. The full pipeline is three hosted-model calls in series. Question parsing, the retrieval arbiter, and generation each cost a few hundred milliseconds to over a second, and they run one after another. For an easy question that adds up to roughly two seconds of the user watching a spinner, to return a number a keyword match had already isolated. Two seconds through the full pipeline, or a tenth of a millisecond routed past it - Image by author Two seconds against a tenth of a millisecond is a different order of magnitude, not a tuning gain, and it lands on exactly the questions users ask most: the simple, templated ones. The cost side moves the same way. Every routed question is three model calls not billed, and at a support desk running the same fifty templated questions across ten thousand contracts, those easy questions are most of the traffic. So the fast path is close to free as well as fast.

5. Indicators, not a bigger model

The margin is one indicator. It is the cheapest one and it settles the clean cases, but it is not the whole optimization. Making a thorough pipeline fast is a matter of finding, for each decision the pipeline currently hands to a model, a signal that decides it without one. There are several fronts, and each is real work.

5.1 The fronts

The answer that already exists. Most enterprise questions get asked again. If a question matches one already answered, on a document that has not changed, the answer is on file: return it, with no retrieval and no model. The catch is the match itself. Deciding “this is the same question” is its own classification: normalize the wording, compare the keywords, compare the question’s embedding against a store of past questions, and accept only a close hit. That store is the templated-question base from Article 6C (dispatching the parsed question) and the corpus tables from Article 25 (the storage model); the indicator reads them. The retrieval that already answered. The signal this article uses. Score the lines, and when one clearly carries the answer’s shape, the answer is found, and generation has nothing left to do. The question type, from the expert dictionary. To know a question wants one line and not a paragraph, the pipeline needs its answer shape, and today a model reads it. But for a known concept the shape is already written down: the dictionary maps premium to (single, amount). Look the concept up and you have the type deterministically, no model. The model is left for the genuinely new question the dictionary has never seen. This is amplify the expert moved to the control flow: the expert’s dictionary decides the route, not the LLM.

5.2 Where the indicators end and the agent begins

Two boundaries sit at the edge of this. The full engineering of latency and cost, the small-model tiers, the caches, context reduction, streaming, and the measurement of cost-per-query and tail latency, is Article 21 (cost and latency). This article works the single highest lever, not calling the model, and points there for the rest. The other boundary is agentic RAG, and it falls on one precise line: who makes the route decision. Made by an indicator, a keyword score, a cache hit, a dictionary lookup, the decision is deterministic, and it belongs to the Vol.1 engineering this article is part of. Made by the model, classifying the intent and planning the steps, it is what makes a system agentic. The agentic follow-up to this series carries the same deterministic-first, model-only-if-ambiguous pattern up a level, to picking a tool rather than just a route. The two are not rivals but layers: the deterministic indicators handle the easy majority for free, and the agent is the fallback for the questions the indicators cannot classify. Building indicators that are fast, and often just classical rules or a small trained model, is how you keep the agent, and its latency, for the cases that truly need it.

6. Conclusion

The reflex when a RAG pipeline feels slow is to reach for a faster model. The cheaper and larger win is to call the model less. The retrieval brick already computes, for free, a signal that separates the questions answered by a keyword match from the ones that need the model to reason: the top keyword score and its margin. Route on it. Send “what is the premium?” , one line at score 5, straight to a deterministic extractor, and keep the model for “which guarantees can I avoid in my case?” , where the flat score is the pipeline telling you it needs help. The threshold is yours to tune, per corpus, against a labelled set. The mechanism is portable, and it is the same lesson the series keeps landing on: the expert’s keyword work is not a fallback under the model, it is often the whole answer, and knowing when it suffices is what keeps the pipeline fast.

7. Further reading and sources

Earlier in the series:

Sources

The broader cost-and-latency treatment is Article 21 (cost and latency).

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Angela and Kezhan Shi

See all from Angela and Kezhan Shi Document ProcessingLlmRagRag ArchitectureToken Management Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

10 Ways to Improve the Performance of Retrieval Augmented Generation Systems

Large Language Models Tools to go from prototype to production Matt Ambrogi September 18, 20239 min read

AI Agent with Multi-Session Memory

Artificial Intelligence Build from scratch using only Python & Ollama (no GPU, no APIKEY) Mauro Di Pietro June 26, 20258 min read

CLIP Model Overview : Unlocking the Power of Multimodal AI

Machine Learning The magic behind multimodal models unlocked through contrastive learning Slava Efimov July 14, 20256 min read

Vector RAG Isn’t Enough — I Built a Context Graph Layer for Multi-Agent Memory

Large Language Model A structured context graph beat a flat chat dump and a vector-only RAG pipeline on the same multi-agent conversations — full working code, real benchmark numbers, zero API calls Emmimal P Alexander June 26, 202617 min read

AI in Social Research and Polling

Artificial Intelligence What do we do when nobody answers the phone? Stephanie Kirmer April 1, 202511 min read

Why 90% Accuracy in Text-to-SQL is 100% Useless

Large Language Models The eternal promise of self-service analytics Gary Zavaleta January 12, 20268 min read

How to Fine-Tune an LLM: An End-to-End Guide

Large Language Models A hands-on guide to fine-tuning LLMs for the real world Sam Black August 20, 202626 min read

Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One

Large Language Model Enterprise Document Intelligence [Vol.1 #14A] - Three questions tell you which shape a document collection has, and each shape wants a different architecture Angela and Kezhan Shi August 20, 202617 min read

A Comprehensive Guide to Collaborative AI Agents in Practice

Large Language Models the definition, and building a team of agents that refine your CV and Cover Letter for job applications Hesam Sheikh July 3, 202414 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close