保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Large Language Model

Loop Engineering for RAG: The Small Loops Inside Each Step, the Big Loops Across the Pipeline

Enterprise Document Intelligence [Vol.1 #13bis] - The four bricks return useful results most of the time. Loop engineering is what the system does the rest of the time: when retrieval misses, when generation fails the schema, when the listing comes back incomplete, when an API call times out. Three control surfaces (trigger, termination, recovery) and one rule that separates a useful loop from a spinning one Angela and Kezhan Shi August 17, 202614 min read Photo by Nina Simková, via Pexels. AI engineers today talk a lot about loop engineering. The phrase took off after Boris Cherny, the Anthropic engineer behind Claude Code, said he had mostly stopped prompting: he writes loops that prompt the model until a condition is met. For a coding agent with a test suite to check against, that works. For an enterprise RAG pipeline, the loop itself is the part that needs engineering, because the failures it has to absorb are ordinary: the parser flattens the one table the answer lived in, retrieval returns the page next to the right one, the model sends back JSON that fails the schema, the API times out in the middle of a batch. The four bricks handle the clean path; loop engineering is what the system does on every other path, and doing it well is what lets the pipeline recover instead of spin. A one-shot pipeline commits to its first try: parse once, retrieve once, generate once, return whatever comes out. When retrieval comes back empty or the answer is half-formed, there is no second chance. A loop gives the pipeline one: notice the miss, adjust, and run the weak step again before the user ever sees it. This article is a standalone companion to Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks. It sits next to Article 7bis (context engineering for single-document RAG) and treats the layer just above it. Article 13 (the composite pipeline) glued the bricks together and named the dispatcher. This article zooms into what the dispatcher does when a single call is not enough. Volume 4 develops the agentic version of the same discipline. This piece stays at single-document scope. The rest is the map: what loop engineering is, the anatomy of one loop (trigger, termination, recovery), the two scales it runs at, the failure modes to guard against, and the outer loop the human holds. Learn this step by step with the interactive AI Engineer roadmap. 🧭 New to the series? Start with the map : Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits. where this article sits in the series: Article 13bis (loop engineering), a companion to the composite pipeline - Image by author 📓 The runnable companion shows the loops firing: you trip llm_parse on a simulated timeout and watch the backoff schedule count down, then flip complete_answer_found to false on a listing answer and watch the dispatcher widen the retrieval scope and regenerate. On GitHub: doc-intel/notebooks-vol1. The public companion-code repo at doc-intel/notebooks-vol1 - Image by author

1. What loop engineering is

1.1 The third layer of the agent stack

Three disciplines stack on top of an LLM call. Prompt engineering writes the call: system message, user message, schema. Context engineering chooses what enters and exits the model’s context window between calls (see Article 7bis). Loop engineering decides when the next call happens , what triggers it , when the loop stops , and how the system recovers when something goes wrong. A polished prompt with a clean retrieval upstream produces a correct answer most of the time. The rest of the time the call fails in one of the ways the opening listed: an invalid JSON, a self-flagged incomplete listing, a 429 at the deployment’s rate cap. Loop engineering is the discipline of designing what happens next. The shortest critique of bad loop engineering, paraphrasing the practitioner literature, is this: a loop that retries the same action on the same error is not learning, it is spinning. The difference between a loop that helps and a loop that wastes tokens is whether each iteration changes something the previous iteration did not address.

1.2 The lineage in one paragraph

Bounded retry is the oldest pattern in the catalogue: Erlang’s let it crash model has carried some version of it since the 1980s. ReAct (Princeton and Google, October 2022) put it inside an LLM call for the first time: reason about the result, decide to act again or stop. AutoGPT (March 2023) made the autonomous version public. Reflexion (NeurIPS 2023) added self-evaluation. Plan-and-Execute separated planning from execution. Geoffrey Huntley’s Ralph Loop (July 2025) put the goal on disk so a context reset never lost track. Anthropic released the /goal command in Claude Code (May 2026) and packaged the whole stack two weeks later as Dynamic Workflows. The single-document case in this article uses a much smaller subset: retry-with-backoff, schema-fail retry, completeness check, dispatcher branching. None of the heavier orchestration primitives bite at single-document scope. They will, in V1 Part IV (corpus) and V4 (agentic).

2. The anatomy of a loop

Before cataloguing the loops in a RAG pipeline, it helps to fix the parts of a single one. Every loop, small or big, has the same three control surfaces and answers to the same one rule.

2.1 Three control surfaces: trigger, termination, recovery

A loop has three control surfaces that are easier to design separately than together. Trigger. The condition under which a new call fires after the first one returned. Three triggers show up in the single-document case. Schema validation failed : the model produced JSON that the Pydantic schema rejected. The answer carries a self-flag : the listing schema’s complete_answer_found field is false, the answer schema’s confidence is below 0.6, the model wrote needs_clarification. Transient API failure : an APITimeoutError, an APIConnectionError, a 429, a 5xx. Each trigger is a single named predicate. The trace store logs which one fired; the dispatcher (Article 13) decides which one to handle and how. Termination. The condition under which the loop stops. Three classes show up in the single-document case. Loop-until-done terminates when the predicate is finally satisfied (the schema validates, complete_answer_found is true, the confidence is above threshold). Loop-until-budget terminates when a quota is exhausted (six retries reached, sixty seconds of wall-clock burned). Hard cap terminates unconditionally after N attempts (the llm_parse wrapper hard-stops at six retries with RuntimeError). A loop without an explicit termination is the canonical failure mode the Anthropic Cookbook flags first. Recovery. What happens when a call fails. The library this series provides does the simplest answer in its LLM wrapper: retry-with-backoff. Six attempts, delays of 2 / 4 / 8 / 16 / 32 / 60 seconds plus jitter, on a fixed list of retriable exceptions (APITimeoutError, APIConnectionError, RateLimitError, 429, 500, 502, 503, 504). Three richer answers fit different cases: fallback to a different model (the 3B failed schema, the 7B might), escalate to a human reviewer (any answer below 0.6 confidence on a high-stakes question), or skip the item and return what was computed (the batch result carries the list of successes and the list of failures, the consumer decides whether to retry the failures later). These three controls describe most of what a loop does between LLM calls. One LLM call, three control surfaces around it: trigger, termination, recovery. - Image by author The one control surface the library provides as a ready-made primitive is recovery, in the shape of retry-with-backoff. It lives in llm_parse and wraps every parse call: bounded attempts, an exponential delay with jitter, a hard cap that raises rather than spins. python

# the library's LLM wrapper - retry-with-backoff, built into every parse call

def llm_parse(*, input, text_format, max_retries=6, cache=True, **opts):
    for attempt in range(max_retries):
        try:
            return client.responses.parse(input=input, text_format=text_format)
        except Exception as err:
            if not is_retriable(err):              # timeout/429/5xx retry ;
                raise                              # context-length never does
            if attempt == max_retries - 1:
                raise                              # hard cap -> propagate
            delay = _compute_delay(attempt, base=2, cap=60, jitter=0.3,
                                   retry_after=extract_retry_after(err))  # 2,4,..,60s
            time.sleep(delay)                      # honour Retry-After floor

2.2 The anti-spinning rule

A loop that fires its trigger, retries with the same payload, gets the same failure, and retries again is not a loop. It is a denial-of-service against the company’s API budget. The rule that prevents this is the simplest one in the discipline. A loop should only retry when something has changed between the previous attempt and the next. Three legitimate things to change. The payload : the dispatcher widened the retrieval scope (50 lines became 200 lines), the system prompt got an extra constraint (“the previous answer did not validate; produce ONLY a JSON matching this schema with no prose around it”), the question was rephrased (the user clarified what they meant by “in Q3”). The model : the small model returned garbage, the large model gets the second shot. Article 8quater (adaptive model selection) builds this loop: start with the cheapest model the job allows, escalate only when validation rejects. The strategy : the keyword retrieval missed, the second attempt uses dense retrieval. This is the dispatcher’s job (Article 13) and the V4 catalogue calls it classify-and-act. Retry that does not change one of the three is spinning. The implementation rule is operational: every retry should carry a trace block that names what changed. If the trace block is empty, the retry should not have fired. The fix is to bound retries at a small N (the library uses six) and to log the changed-since-last field on every iteration. The day a user complains that “the same question now costs four times as much” , the trace tells you whether the loop was working or spinning.

3. Two scales of loop

A loop is not one magic mechanism bolted onto the pipeline. Each one is a named, bounded response to a specific way a brick can fall short, and they sort into two scales: small loops that run inside a single brick, and big loops that cross bricks when generation distrusts its own input.

3.1 Small loops: one inside each brick

Small loops live inside a single brick. Document parsing runs the image cascade: filter the page’s images cheaply, classify what each one is, and pay a vision model to describe only the ones worth reading (Article 5). It also rebuilds a missing outline from body typography, looping until the outline stabilises (Article 5octies), and dispatches each page to the parser its signals call for (Article 5nonies). Retrieval walks the table of contents down to the right subsection (Article 7quater). Question parsing asks one clarifying question and learns the default (Article 6bis). Generation retries when the JSON fails the schema, and reruns retrieval and generation when the listing comes back incomplete (Articles 8 and 12). Each of these has the same three control surfaces as any loop, a trigger, a termination, and a bounded depth; it just iterates over its own brick’s material rather than reacting to a downstream result. Retry-with-backoff wraps every API call underneath all of them, the one loop that is pure plumbing rather than a brick’s own reasoning. Small loops sit inside a brick, big loops cross bricks when generation distrusts its input. - Image by author

3.2 Big loops: across bricks

Big loops cross bricks. Generation reads its own input, decides it was not good enough to answer from, and sends the pipeline back to an earlier brick. The single-document case has three: adaptive parsing (generation’s context_structured=false re-parses the offending page with a deeper parser, Articles 10A and 10B), reference resolution (generation reports pending_references, the orchestrator resolves the pointer against the parsing tables, re-retrieves the target region, regenerates, Article 11), and scope feedback (generation’s complete_answer_found=false widens retrieval, Article 13). Same shape each time: a typed flag on the answer, a bounded trip back upstream, a second generation pass that re-checks itself. The check is a typed flag, the escalation is bounded to the flagged page, and the loop stops the moment the re-checked answer trusts its context. The adaptive-parsing big loop: verify the parse, escalate the flagged page, regenerate, re-verify. - Image by author

3.3 The named patterns the series already has

Looking at the existing V1 code with the 2026 loop-engineering vocabulary on hand: most of the named primitives have a counterpart already in the code, under a less catchy name. The five primitives the single-document case uses already exist in the series under quieter names. - Image by author Loop-until-done : the V1 Article 12 listing completeness loop. The generator answers, the schema carries the complete_answer_found flag, the loop reruns retrieval and generation if the flag is false. The pattern came with the listing schema and predates the name. Retry-with-backoff : the library’s llm_parse wrapper. Six attempts with the exponential schedule. The cost analysis in Article 21 Section 4.6 details when this triggers and what the operational impact looks like. Generate-and-filter : V1 Article 8’s structured-output schema. The LLM returns JSON, the Pydantic validator rejects anything that does not match, the loop retries if validation fails. The 2026 vocabulary makes the loop layer explicit; the discipline has been in the series since the schema-first generation article. Classify-and-act : V1 Article 13’s composite dispatcher. The parsed question carries a shape (single-value, listing, aggregation, comparison), the dispatcher routes to the matching handler. Anthropic’s classify-and-act applies the same shape to free-form classification rather than to a parsed-question taxonomy; the underlying machinery is the same. Feedback-loop-between-bricks : V1 Article 13’s between-bricks feedback. Generation flags the retrieval scope as insufficient, the dispatcher widens the scope, generation reruns. The 2026 vocabulary calls this evaluator-optimiser ; the series called it the dispatcher’s feedback loop and built it in 2024. What is not yet in V1: adversarial verification (a separate agent that tries to refute the answer), the tournament pattern (N candidate answers, judges pick the winner), explicit loop-until-dry termination (K consecutive empty rounds). Those are V4-scope; the single-document case rarely needs them.

3.4 The loops at a glance

Every loop in the single-document pipeline, small or big, wears the same three control surfaces from Section 2. Read down the trigger column and the shape is the same each time: the loop fires on one specific, named signal, terminates on a bound, and changes exactly one thing before it retries. Every loop as one row against the three control surfaces of Section 2. - Image by author

4. Failure modes the discipline guards against

Three failure modes are routine in production single-document RAG and each one maps to a specific loop-engineering rule. Infinite-loop on transient failure. The retry layer has no hard cap. The deployment hits a 429, the loop retries, the next 429 arrives within the rate-limit window, the loop retries again, the API budget burns through the night. Fix : a hard cap on retries (the library uses six), an explicit RuntimeError after the cap, and an alert on the trace store when the cap is hit. Spinning on a deterministic failure. The Pydantic schema rejects the output because the model wrote a field name with a typo. The loop retries with the same prompt. The model produces the same typo. Six retries, six rejections, the user sees a RuntimeError. Fix : on schema-fail retry, modify the prompt to include the validator error message verbatim (“ your previous answer failed validation with:_field_name_should be _field_name_actual_. Produce a JSON matching the schema exactly. ”). The next attempt has new information. Confidence-flag ignored. The answer comes back with confidence = 0.3 and the dispatcher returns it to the user without flagging anything. Fix : the dispatcher applies a confidence threshold, low-confidence answers either escalate to a richer retrieval pass (widen scope, switch from keyword to dense) or carry an explicit low_confidence warning to the consumer.

5. Loops within loops: the human holds the outer one

Loop engineering as covered so far is one loop: the dispatcher fires a call, reads the result, decides to retry, terminates when satisfied. Zoom out and there are three loops running at different cadences around the same system, borrowing the frame Andrew Ng laid out in his “Three Key Loops for Building Great Software” letter:

Ng’s point is that as long as the human knows something the AI does not, that outer loop cannot be closed by more automation, and treating it as noise to be optimised away is the mistake. “As long as the human knows something the AI does not, human-in-the-loop is needed to inject that knowledge into the system.” This is the series’ amplify-the-expert thesis in 2026 vocabulary. The dispatcher owns the inner loop, the engineer owns the dev loop, the expert owns the outer loop, and none of them can substitute for the others. The corollary for design: keep the outer loop cheap for the expert. If correcting a wrong field takes ten clicks, the outer loop breaks and the system’s error rate stops improving. The refresh-field UX, the review queue, and the audit trail keep the outer loop fast enough to compound.

6. Out of scope, with pointers

Three siblings stay open here.

7. Conclusion

Loop engineering is the third discipline after prompt engineering and context engineering. At single-document scope, it owns four primitives: retry-with-backoff for transient API failures, generate-and-filter for schema-fail retries, loop-until-done for completeness checks, classify-and-act for dispatcher branching. The single rule that separates a useful loop from a spinning one is every retry changes something. The library this series provides implements the first primitive directly; the other three are in the dispatcher (Article 13) and the listing schema (Article 12). The 2026 vocabulary puts a name on a discipline the series has been practising since 2024.

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Angela and Kezhan Shi

See all from Angela and Kezhan Shi Document ProcessingLlmRagRag Architectureloop engineering Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Code understanding on your own hardware

Setting up an LLM to talk about your code - with LangChain and local hardware Dorian Drost July 5, 20237 min read

How to Increase Coding Iteration Speed

LLM Applications Learn how to become a more efficient programmer with local testing Eivind Kjosbakken December 13, 20257 min read

Prompt Engineering Isn’t Enough — I Built a Control Layer That Works in Production

Large Language Model Most production LLM integrations treat prompts like the final layer. That gets you 0% reliability on structured output. I built an 8-component system that brought it to 100% — without changing a single prompt. Emmimal P Alexander May 21, 202620 min read

LLM Fallbacks Break Agent Pipelines — I Built the Missing Recovery Layer

Large Language Model Standard model swapping forces your production AI to accept corrupted data. Here is the zero-dependency, state-aware recovery layer I built to preserve schema integrity during failovers. Emmimal P Alexander June 16, 202614 min read

Stop Returning Text from RAG: The Typed Answer Contract That Prevents Hallucination

Large Language Model Enterprise Document Intelligence [Vol.1 #8A] - The schema is the contract: every field is a question the pipeline asks the model, and every answer is checkable Kezhan Shi July 4, 202626 min read

Building a Knowledge Graph From Scratch Using LLMs

Artificial Intelligence Turn your Pandas Data Frame into a Knowledge Graph using LLMs. Build your own LLM graph-builder and QA your KG. Cristian Leo November 25, 202439 min read

LLMs + Pandas: How I Use Generative AI to Generate Pandas DataFrame Summaries

Large Language Models Local Large Language Models can convert massive DataFrames to presentable Markdown reports — here's how. Dario Radečić June 2, 202510 min read

A Production RAG Pipeline in Action: Every Answer Typed and Cited

Large Language Model Enterprise Document Intelligence [Vol.1 #9B] - One call wires the four upgraded bricks together, run on a paper, a NIST standard, and a report with a broken TOC Angela and Kezhan Shi July 17, 202625 min read

Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model

Large Language Model Enterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match. Angela and Kezhan Shi August 13, 202611 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close