{ "source_file": "0930", "source_platform": "Medium", "author": "Angela Shi & Kezhan Shi", "title_listed": "Building Document Structure with Loop Engineering: Recovering a PDF's Outline from Body Typography for RAG", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/", "publish_date": "2026-08-05", "corpus_stage": "", "notes": "", "id": "5b1794197a78", "collected_at": "2026-10-02T01:23:22", "title": "Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG\nEnterprise Document Intelligence [Vol.1 #5octies] - Rules propose, LLM validates: six deterministic signals on span-level typography surface heading candidates, one bounded loop keeps the real ones, and the same toc_df drops back into the RAG pipeline\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 5, 202622 min read\n![](../images/000.jpg)\n[Photo](https://www.pexels.com/photo/226612/) by icon0.com, via Pexels.\nA structured RAG pipeline needs the document’s table of contents: retrieval scopes by section, and the chunker cuts on heading boundaries instead of mid-sentence. But some documents don’t have one. A paper exported straight from LaTeX has no native outline and no printed contents page, so there is no `toc_df` for the pipeline to read. The headings are still right there on the body, section by section, bigger and bolder than the text around them. [Article 5septies (TOC reconstruction from a table of content)](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/) handled the document that at least prints a contents page; this one prints none. So the pipeline rebuilds the `toc_df` from the one signal left: the way the headings look.\nThis article is a document parsing companion in [Enterprise Document Intelligence](https://towardsdatascience.com/amplify-the-expert-a-philosophy-for-building-enterprise-rag/), the series that builds an enterprise RAG system from four bricks. It sits in **brick 1 (document parsing)** and closes the TOC-reconstruction thread that [Article 5 (document parsing)](https://towardsdatascience.com/beyond-extract_text-the-two-layers-of-a-pdf-that-drive-rag-quality/), [Article 5B (the relational data model)](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/), and [Article 5septies (TOC reconstruction from a table of content)](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/) opened.\n![](../images/001.png)\n_where this article sits: brick 1 (document parsing), Part II, the TOC-reconstruction case deferred by Article 5septies - Image by author_\n📓 **The runnable notebook** walks the attention paper (`data/paper/1706.03762v7.pdf`) through the loop from `line_df` + `span_df` to a 24-entry `toc_df` with 21 real headings and 3 false positives that the LLM validation drops: [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1).\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n## 1. Where this article sits and what it does\n[Article 5septies (TOC reconstruction from a ](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/)[](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/)[table of content](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/)[)](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/) drew a line: TOC _detection_ reads a page that already exists (a native outline or a printed table of content); recovering structure from body typography belonged to summarisation. That line was pragmatic. It kept the table of content cascade small and testable. But the summarisation intent has different failure modes (long-context, prompt-first) and different guarantees (no fixed shape, no source column). For a document that has clear typographic headings on its body, we do not need a summarisation loop. We need the same `toc_df` shape as everywhere else, filled from the signals the body carries. So the frontier moves: body-typography reconstruction is a fourth _detection_ case, not a summarisation fallback.\n> **In:** a `line_df` (plus a `span_df` when the parser exposes typography) whose native TOC and printed table of content both came up empty. **Out:** a `toc_df` in the standard [Article 5B](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/) shape, so retrieval, chunking, and summarisation keep working unchanged.\nOnce you have that fourth case, three neighbouring situations come along for free:\n * The PDF has **no structure at all** , the case just described.\n * The PDF has a **partial native outline** that stops at level 2 while the body clearly has a level 3 (`3.2.1 ...`, `3.2.2 ...`). The body-typography pass deepens what the outline gives you.\n * The PDF is **composite** : two or more documents concatenated. Section numbering re-inits mid-file, and the native outline (if any) is flat. The same pass, with a merge step, reconciles the streams.\n\n\nThe rest of the article walks each in turn, in the same order the cascade tries them.\n## 2. Four cases, one cascade\nThe cascade grew by one. Same order: cheapest first, drop through to the next when a case does not fire or returns too little. Case 4 is the new one; the earlier three come from [Article 5septies (TOC reconstruction from a table of content)](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/) unchanged.\n![](../images/002.png)\n_Case 4 is the new one: the only case where the LLM does real detection, not coherence-checking - Image by author_\n * **Case 1** , native outline, and **Case 2** , contents page with links. Article 5septies (TOC reconstruction from a table of content) covers both. Deterministic, cheap, exact when they work.\n * **Case 3** , printed contents page without links. Same article, text patterns plus label-to-page alignment.\n * **Case 4** , no contents page at all. This article. Body typography surfaces heading candidates; an LLM loop keeps the real ones.\n\n\nCase 4 is opt-in in the cascade because it is genuinely the most expensive: several passes of an LLM in the worst case, one pass when the deterministic signals are clean enough. Enable it by passing `methods=(\"links\", \"contents_text\", \"llm\", \"body_structure\")` to `reconstruct_toc_df`, or call `reconstruct_toc_from_body` directly when you know upfront that the file has no contents page.\n## 3. How the cascade reconstructs a TOC\nWith the four cases mapped, here is how the cascade actually turns a document’s pages into a `toc_df`, one stage at a time, from the signals it reads off the page to the LLM check at the end.\n### 3.1 The parser matrix: what each parsing method gives us\nEvery rule the body-typography detector reads is a _column of the extracted table_. So the question is not _“what are the six signals”_ first, it is _“which columns does the parser give me at all”_. The answer depends on how the PDF was parsed. The four bricks live one level up; here we live inside brick 1, and brick 1 has (already, in the earlier articles) several implementations.\n![](../images/body-20086ceec4cc9598ae5e.png)\n_How four parsers differ on TOC signals, from fitz (richest) down to EasyOCR (position and text only) - Image by author_\nTwo practical points that shape the code:\n 1. **PyMuPDF exposes typography per** _**span**_**, not per** _**line**_**.** A span is one contiguous run of characters in one font, size, weight, italic and color. A line that mixes bold and non-bold text expands to multiple spans. So the module provides a helper `enrich_line_df_with_style(line_df, span_df)` that aggregates the spans of each line down to one row and appends `font_size` (character-weighted), `bold_ratio`, `is_bold`, `is_italic`, `dominant_font_name`. Consumers pass a `line_df` when the parser has no typography, or a `line_df` **plus** a `span_df` when it does.\n 2. **Missing typography does not break the loop.** Each of the six signal functions checks the columns it needs and returns a zero series when they are absent. Font size and bold contribute 0.0 to the score in that case; the four position and text signals (numeric prefix, short length, left alignment, blank-line-above) carry the load. Azure OCR Layout and EasyOCR both fall in this regime.\n\n\n`Mistral OCR` is a special case: it returns the document already in markdown, and its `# / ## / ###` headings _are_ the reconstructed outline. When you have a Mistral output you skip Case 4 entirely and read the markdown structure. That is the parser matrix’s shortcut column.\n### 3.2 Six signals a heading gives off\nNow the six signals themselves. Each is one function of one input frame, each returns a `pd.Series[float]` in `[0, 1]`, and the combined score is a plain weighted sum.\n 1. **Font size ratio**. A heading uses a larger font than body prose. Score = `(font_size / median_body_size) - 1`, clipped to [0, 1]. A line at 12pt in a body of 10pt scores 0.2; at 16pt in a 10pt body it saturates to 1.0.\n 2. **Boldness**. Headings are often bold. When the enriched line_df carries `bold_ratio`, we use that; otherwise `is_bold` as a 0/1 fallback.\n 3. **Numeric prefix**. `1.`, `1.2`, `1.2.3`, `I.`, `A.`. Regex `^\\s*(?:\\d+(?:\\.\\d+)*\\.?|[IVXLCDM]+\\.|[A-Z]\\.)(?:\\s|$)`. Present ⇒ 1, absent ⇒ 0. Also carries the _level_ signal: `1.2.3` is a level-3 heading. **Fitz artefact.** A LaTeX-exported PDF often emits `\"1\"` and `\"Introduction\"` as two separate lines; the loop pre-merges bare-number lines with their next line via `merge_split_headings` before scoring, so the numeric prefix is recovered on the merged line.\n 4. **Short length**. A heading is short. Score linearly decays from 1 for lines shorter than 20 characters to 0 at 90 characters. A paragraph rarely lives on a single line; a heading rarely spills across two.\n 5. **Left alignment**. A heading starts at the left margin (or the block’s own left indent). Score = 1 when `x0` matches the page’s median left margin within a small tolerance, decays with distance.\n 6. **Blank line above**. Headings sit in visual isolation. Score = 1 when the vertical gap above the line is wider than the median gap between prose lines on that page.\n\n\nNone of these is decisive on its own. Bold-and-large by itself catches figure captions (_“Figure 3: architecture”_ is often bold and larger). Numeric prefix by itself catches enumerated lists inside a paragraph. The point is to combine them, and combining them is where the weights live.\npython\n\n```\n# The six per-line signals, cheap, deterministic, engine-agnostic.\nfrom docintel.parsing.pdf.toc.body_structure import (\n enrich_line_df_with_style,\n score_font_size_ratio,\n score_is_bold,\n score_has_numeric_prefix,\n score_is_short,\n score_is_left_aligned,\n score_has_blank_before,\n)\n\nline_df = enrich_line_df_with_style(line_df, span_df)\n\nsignals = {\n \"font_size_ratio\": score_font_size_ratio(line_df),\n \"is_bold\": score_is_bold(line_df),\n \"has_numeric_prefix\":score_has_numeric_prefix(line_df),\n \"is_short\": score_is_short(line_df),\n \"is_left_aligned\": score_is_left_aligned(line_df),\n \"has_blank_before\": score_has_blank_before(line_df),\n}\n```\n\n### 3.3 Combining signals into candidates\nA weighted sum turns six scores into one. Weights come from prior belief, not from a training run. Numeric prefix is the strongest single signal (1.6). Font size ratio next (1.4). Boldness middle (1.0). Blank-line-above (0.8), short length (0.6), left alignment (0.4) are supporting evidence. Sum of weights is 5.8. The default threshold is 3.0 (about half the sum), which on the attention paper picks up every real heading plus a handful of table-cell false positives.\npython\n\n```\ncandidates_df = detect_body_headings(line_df, threshold=3.0)\n\ncandidates_df[[\"text\", \"heading_score\", \"candidate_level\",\n \"signal_font_size_ratio\", \"signal_has_numeric_prefix\"]].head()\n```\n\n**On the attention paper (Vaswani et al. 2017, an arXiv LaTeX export), the deterministic pass at threshold 3.5 finds 24 candidates** : 21 real section headings (from `1 Introduction` to `7 Conclusion`, all subsections included) and 3 false positives (`28.4`, `4.33`, `26.4`, BLEU scores from the results table on page 8-9 that happen to be bold and short). Recall on real headings is 91 % (21 of 23 real headings, `1 Introduction` sits just below the threshold and comes back at 3.0). Precision is 88 %. Level accuracy from numeric prefix is 100 % on the recovered rows. Those are the numbers from a real run, not a projection.\n### 3.4 Rules propose, LLM validates\nHere is where the LLM does real detection work. Not free-form. Fixed schema. The prompt describes what a heading is, what a false positive looks like (figure caption, table row label, bold in-body emphasis, BLEU score in a results table), gives the candidates in reading order with page and snippet, and asks for a JSON list of the kept entries. Missed headings (small caps, italics, an unnumbered epilogue) can be added by the same pass because it also sees a slice of the surrounding lines.\nThe loop is bounded. At most `max_passes` iterations (3 by default). Each pass reads the current keep list and can drop entries or propose additions. Convergence is when a pass changes nothing.\npython\n\n```\ndef my_llm_parse(system_prompt: str, user_content: str) -> list[dict]:\n ... # user's LLM client, returns the kept-entries JSON list\n\ntoc_df = reconstruct_toc_from_body(\n line_df,\n span_df=span_df,\n mode=\"no_toc\",\n max_passes=3,\n llm_parse=my_llm_parse,\n)\n```\n\nTwo things worth spelling out. First, the LLM is told what it may return. The response schema is fixed (`title`, `page`, `level`, `source`), matching the table of content cascade so downstream code does not branch on which case fired. Second, `llm_parse` is an _injected_ callable. Tests hand in a mock; CI can replay from a JSON cache; the module never opens a socket by itself. That is the discipline every LLM-touching module in this series follows.\n### 3.5 Extending a partial native outline\nSome PDFs have a native outline that stops at level 2. `1. Introduction`, `1.1 Motivation`, `2. Method`, `2.1 Data`, and the file is done. But the body clearly runs `2.1.1`, `2.1.2`, `2.2.1`. The retrieval brick wants those. So does the chunker, if the section runs long.\n`reconstruct_toc_from_body` handles this with `mode=\"extend_native\"`. The native outline is preserved verbatim (each row carries `source=\"native\"`). The body-typography pass runs on top, and any candidate whose title-and-page pair does not already exist in the native outline is added with `source=\"body_structure\"`. The reader can see, row by row, which entries came from the file and which came from the body pass. That is what `source` is for.\npython\n\n```\ndeeper_toc = reconstruct_toc_from_body(\n line_df,\n existing_toc_df=native_toc, # from doc.get_toc(), stops at level 2\n span_df=span_df,\n mode=\"extend_native\",\n)\n\ndeeper_toc[deeper_toc[\"source\"] == \"body_structure\"].head()\n```\n\n`mode=\"auto\"` (the default) picks this branch when `existing_toc_df` is non-empty and its max level is `≤ 2`, so most callers do not have to think about which mode to pass.\n### 3.6 Composite documents, and the API in one call\nThe other case: a single PDF file that concatenates two or more documents. An arXiv paper followed by a supplementary memo. A request-for-proposal bundle where each vendor answer is its own document. A review dossier of the letter and the response side-by-side. The native outline (if any) is flat and confusing: two `\"1. Introduction\"` entries, page numbers that do not increase monotonically, a re-init of the numbering half-way through.\n`detect_document_boundaries` looks for three deterministic signals: numbering re-init (a `\"1.\"` on a page after `\"N.something\"`), style rupture (a jump in median font size between pages N-1 and N), and cover pages (a page with a large title, low line count, no body text). Each returns a small frame of boundary rows. When any boundary is confirmed, `reconstruct_toc_from_body(..., mode=\"composite\")` re-roots the outline: each internal document becomes a top-level entry (`\"Document 1\"`, `\"Document 2\"`, …) with the original outlines nested under.\nThis is the case where the merge is worth an LLM pass, because a re-init in numbering could also just be a document that legitimately restarts a counter inside a chapter. The `MERGE_PROMPT` asks the LLM to confirm each boundary or reject it, then to return the merged outline as a JSON list. The whole subpackage funnels through one public entry point:\npython\n\n```\ntoc_df = reconstruct_toc_from_body(line_df, span_df=span_df, mode=\"no_toc\")\ntoc_df = reconstruct_toc_from_body(\n line_df, existing_toc_df=native_toc, span_df=span_df, mode=\"extend_native\",\n)\ntoc_df = reconstruct_toc_from_body(\n line_df, existing_toc_df=native_toc, span_df=span_df, mode=\"composite\",\n)\ntoc_df = reconstruct_toc_from_body(line_df, existing_toc_df=native_toc, span_df=span_df, mode=\"auto\")\n```\n\nFor the pipeline caller who already uses `reconstruct_toc_df` and wants the body-typography case as a _cascade fallback_ rather than a separate call, add `\"body_structure\"` to the methods tuple: `reconstruct_toc_df(pdf_path, methods=(\"links\", \"contents_text\", \"llm\", \"body_structure\"))`. The cascade will try the table of content methods first and fall through to Case 4 when they return empty.\n## 4. Testing against native outlines: how well does it work\nThe honest way to score this is against ground truth. Take six PDFs that DO carry a native outline, hide it, run the body-typography loop on the body, and compare the reconstruction to the outline the file was hiding. The six fixtures are all tier-1 open sources with different flavours of structure:\n * Vaswani et al., [_Attention Is All You Need_](https://arxiv.org/abs/1706.03762) (arXiv 1706.03762, arXiv non-exclusive distribution). A LaTeX-exported NIPS paper, decimal numbering.\n * NIST, [_Zero Trust Architecture_](https://doi.org/10.6028/NIST.SP.800-207) (SP 800-207, US government work, public domain). Decimal numbering, dot-leader table of content.\n * NIST, [_Protecting Controlled Unclassified Information in Nonfederal Systems and Organizations_](https://doi.org/10.6028/NIST.SP.800-171r2) (SP 800-171r2, public domain). Nested numeric prefixes, heavy front matter.\n * NIST, [_Standards for Security Categorization of Federal Information and Information Systems_](https://doi.org/10.6028/NIST.FIPS.199) (FIPS 199, public domain). Very short, appendix-heavy.\n * NIST, [_Securing Distributed Energy Resources: An Example of Industrial Internet of Things Cybersecurity_](https://doi.org/10.6028/NIST.SP.1800-32) (SP 1800-32, public domain). A large practice guide, 152 pages, mixed numbering.\n * FEMA, [_NFIP Flood Insurance Manual, Appendices (Policy Forms)_](https://www.fema.gov/flood-insurance/work-with-nfip/manuals) (US government work, public domain). The hardest case in the set, appendices titled with Roman numerals (`IV. PROPERTY NOT INSURED`) and named sub-forms.\n\n\nThe script that runs the eval lives at `scripts/checks/eval_toc_body_structure_vs_native.py`; each row below is the deterministic pass ONLY (no LLM validation), threshold 3.0 with the default weights, `span_df` typography enrich on.\n![](../images/body-1d635493ed0bf71b3632.png)\n_High recall, lower precision, and level assignment that breaks on Roman-numeral outlines; page accuracy holds at 100% - Image by author_\nThree things read off the table.\n**Aggregate reporting: macro and micro tell different stories.** The `aggregate` row is the macro-average (each doc weighs one). The micro-average, weighted by native rows, is different: `299 / 415 = 72%` micro-recall and `299 / 1481 = 20%` micro-precision. The gap between macro (77% recall) and micro (72% recall) has a specific driver: FEMA NFIP (186 natives, 69% recall) and NIST SP 1800-32 (104 natives, 62% recall) carry most of the native count, and both sit below the macro. The macro over-represents Vaswani (only 22 natives at 100% recall). Both numbers matter; neither alone is enough.\n**The deterministic pass is a high-recall filter on decimal-numbered docs.** 75 to 100% recall on Vaswani, SP 800-207, SP 800-171r2 and FIPS 199. The 100% recall on Vaswani is the fitz-artefact merge doing its job: every `\"1\"` + `\"Introduction\"` pair reunited on the fly. Recall drops on the two non-decimal fixtures (FEMA NFIP 69%, NIST SP 1800-32 62%). The missed rows are the ones whose numbering the numeric-prefix regex does not match, plus level-3 named sections (`National Flood Insurance Program Dwelling Form`) that carry no prefix at all.\n**Precision without LLM is poor.** 7 to 38% across the corpus. The pass drags in a lot of noise, and the noise falls into five predictable categories.\n![](../images/body-44fedf90b9248fe6c901.png)\n_Every category is systematic; Section 3.4’s validation pass targets each, though the drop rate is not yet measured - Image by author_\n**Level accuracy is reported conditional on match, and that framing hides a bias.** The 100% level accuracy on the four decimal-numbered docs is `(matched rows where recon-level equals native-level) / matched`. It tells you that when the algorithm identifies a heading whose title matches a native, the numeric-prefix regex reads the level correctly. On FEMA NFIP and NIST SP 1800-32 the recon regex fails to assign a level at all (Roman numerals, named appendices), the matched rows carry NaN, and the conditional ratio collapses to 0. The honest global metric is `(native rows where recon predicted the correct level) / all native rows`. It lands at exactly `105 / 415 = 25%` micro across the six fixtures. **Page accuracy stays at 100% within ±1** on every matched row, but that near-tautology sits because both recon and native read `page_num` from fitz. The rows where it would matter (a heading whose native page is off by one from where fitz shows the text) are not represented in this fixture set.\n### 4.1 What the algorithm sees on a real page\nThe best way to make the decision visible is to overlay the algorithm’s per-line output on the raster of the page. Green boxes are lines the deterministic loop scored above the 3.0 threshold and kept as heading candidates. Grey outlines are the other lines the algorithm looked at (body prose, page headers, footnotes) but did not keep. The tag on the right shows what `span_df` exposed for each candidate: `(level, bold flag, font size, heading score)`.\n![](../images/body-00005a08e94a9abbf293.png)\n_Attention paper page 2, algorithm overlay on the split “1” + “Introduction” heading pair - Image by author_\n![](../images/body-baad4976f732213e2c21.png)\n_NIST SP 800-207 page 10, the same overlay pattern on a public-domain doc - Image by author_\nTwo things read off the overlay. First, the number of green boxes per page is small (four on Vaswani p2, two on NIST p10), which confirms the threshold is picking headings rather than sprinkling them across the page. Second, the tags show the algorithm sees the right numbers: 11.9 pt bold for NIST section titles, 11.9 pt bold for Vaswani chapter titles, both with heading scores near 4.28 (well above the 3.0 threshold). The signals `span_df` exposes match what a human reader would flag as a heading, and the scorer combines them without needing per-document tuning.\nHere is the exact code that ran to produce these overlays. It reads the same `line_df` + `span_df` the rest of the pipeline reads, then loops over the candidates for the page and draws the boxes.\npython\n\n```\npdf = \"data/nist/NIST.SP.800-207.pdf\"\nline_df = fitz_pdf_to_line_df(pdf)\nspan_df = build_span_df(pdf)\nenriched = enrich_line_df_with_style(line_df, span_df) # per-line font + bold\ncandidates = detect_body_headings(enriched, threshold=3.0)\n\npage = 10\nfor _, row in candidates[candidates[\"page_num\"] == page].iterrows():\n print(row[\"text\"], row[\"heading_score\"], row[\"candidate_level\"],\n row[\"font_size\"], bool(row[\"is_bold\"]))\n# 1 Introduction 4.28 1 12.0 True\n```\n\n### 4.2 What the LLM validation buys\nEvery category in the taxonomy above is one the LLM is _designed_ to catch on the first pass.\n * Page numbers alone: a candidate whose text is a single digit is never a heading.\n * Table of content dot-leader lines: `\"1 Introduction ...........\"` is a contents-page entry, not a body heading.\n * Cover-page author names: no numeric prefix, wrong page.\n * Table cells: bold and short but not a section title.\n * The `HEADING_VALIDATION_PROMPT` names each category explicitly. Here is the measured effect of a single validation pass with `gpt-4.1` (Azure OpenAI, cache on) on the six-fixture eval.\n\n\n![](../images/body-206d8618c8b2299e069c.png)\n_One validation pass lifts precision from 20% to 87% across six fixtures; recall holds on five of six - Image by author_\nThree things read off the measurement.\n**Precision lifts massively.** From 15% up to 96% on the attention paper, 38% to 98% on NIST SP 800-207, 21% to 100% on FIPS 199, 31% to 82% on NIST SP 1800-32, 7% to 72% on SP 800-171r2. The LLM removes exactly the FP categories the taxonomy above named.\n**Recall is preserved on five of six fixtures.** The LLM validation is conservative in the good sense: it drops false positives without touching the real headings the deterministic pass already found. On Vaswani, 100% recall is preserved; on SP 800-207, 75%; on SP 800-171r2, 97%; on FIPS 199, 60%; on SP 1800-32, 62%. The micro-aggregate on those five docs stays close to before.\n**FEMA NFIP is the outlier where the LLM over-drops.** Recall collapses from 69% to 22%. The Roman-numeral appendices (`IV. PROPERTY NOT INSURED`) and named sub-forms (`National Flood Insurance Program Dwelling Form`) do not fit the LLM’s mental model of “section heading with a numeric prefix”, and it drops them alongside the false positives. This is a known limit: the LLM prompt was tuned on decimal-numbered outlines. A per-domain prompt or a two-pass loop with a corrective critic would recover here.\nThe catch that even the LLM does not fix is deeper. Look at what SP 800-207 has that the deterministic pass genuinely misses: unnumbered top-level titles like `\"References\"`, `\"Acronyms\"`, and the front-page `\"NIST SP 800-207, Zero Trust Architecture\"`. These do not carry a numeric prefix, and their font signal blends with the body. The LLM can propose them when it sees the surrounding context, but the deterministic recall does not include them by construction. That is the ceiling of the method: a document that wants a perfectly-nested reconstructed outline has to carry visible signals for every heading.\n### 4.3 Limitations of this eval\nThe numbers above are honest but small. Four caveats that a rigorous reader should know:\n * **The threshold 3.0 is a default, not a tuned choice.** No cross-validation, no ROC / precision-recall sweep. A per-corpus sweep would likely lift precision meaningfully on the noisier fixtures.\n * **The**`**matching**`**step is loose.** Titles normalize by stripping leading numbers, then compare exactly. A document with two sections named _Introduction_ (a body intro and an appendix intro) can match either, and the greedy assignment picks the closest page.\n * **The**`**page-within-1**`**metric is almost tautological.** Native and recon both derive `page` from fitz’s `page_num`, so an off-by-one at that stage is rare by construction. A stress test would randomize the ground-truth page assignment or use an OCR fixture where the page number is inferred.\n * **Six fixtures, English only, tier-1 open sources only.** No confidence intervals. No multilingual document. No RTL script. No handwritten / scanned insurance CG (which is tier-2 anyway). The pattern generalizes in ways this eval does not confirm.\n\n\nSection 3.4’s LLM validation pass has now been measured on the six-fixture eval (see Section 4.2 above). Micro precision moves from 20% to 87%, recall from 72% to 51% (the drop is entirely FEMA NFIP, whose Roman appendices confuse the LLM). Per-corpus prompt tuning is a real follow-up; the current prompt was written against decimal-numbered outlines.\n## 5. Beyond a single hierarchical TOC: multi-tag enriched structure\nA recovered TOC is one dimension of a document’s structure. It is not the whole thing. Two examples make this clear.\n**Insurance auto contract.** A section titled _3.2 Collision Guarantee_ explains coverage, then names the plafond, then names the exclusions that apply. The exclusions do not necessarily live in a separate chapter titled _Exclusions_. They live inside the guarantee section. A retrieval that scopes by section (via `toc_df.start_page`, `end_page`) and asks _“what exclusions apply to the collision guarantee?”_ pulls back the whole section. The answer is there, but so is a lot of noise, and the query cannot filter on _exclusion_ alone.\n**Composite / poorly-written docs.** A transcription, a meeting minute, a legal opinion, a forensic report. None of these have a clean hierarchical TOC. Themes recur across paragraphs, actors appear and reappear, and the useful navigation is not _“which section”_ but _“which topic”_. A hierarchical outline of a transcription is close to useless.\n### 5.1 Two layers per line or paragraph: level from typography, tags from the business\nThe idea, in one line: the algorithm recovers the **level** (5octies loop), and the business adds **tags** on top. Two layers stack on the same line or paragraph. Neither replaces the other.\n![](../images/body-747cfa898fde95335818.png)\n_Two axes on the same paragraph: section level from the typographic loop, thematic tags from a business taxonomy - Image by author_\n * **Layer 1: LEVEL.** From the body-typography loop above. Chapter 3, section 3.2, subsection 3.2.1. This is `toc_df` and the section anchors on every paragraph via `start_page`/`end_page`.\n * **Layer 2: TAGS.** A `list[str]` per paragraph, drawn from a domain taxonomy. In the insurance example: `[\"garantie\", \"garantie:collision\", \"plafond\", \"exclusion\", \"exclusion:vitesse\"]`. Tags come from a fixed taxonomy defined by the business (or bootstrapped from the corpus and validated by an expert), then an LLM proposes tags per paragraph.\n\n\n**Retrieval implications** : the query _“what exclusions apply to the collision guarantee?”_ becomes an intersection on Layer 2 (`garantie:collision AND exclusion:*`), which is exact even when the answer lives inside a guarantee section, not an exclusions section. The query _“section 3.2 in full”_ still works on Layer 1. The two are complementary.\n### 5.2 A concrete example, drawn from a synthetic contract\nIllustration on a synthesized auto-contract paragraph (fabricated for the article, no real insurer named, the point is the pattern, not the fixture):\n![](../images/body-9866ec88c44b19cbf09d.png)\n_One paragraph holds both a guarantee and an exclusion, and Layer 2 tagging surfaces each theme separately - Image by author_\nThe example is synthetic on purpose. Real insurance conditions générales are tier-2 documents and cannot be published verbatim on TDS; the pattern this figure shows is representative, not a specific insurer’s wording.\n### 5.3 Honest limitations of the multi-tag layer\nThe idea is powerful. It is also hard. Four costs to name upfront:\n * **Redundancy with Layer 1.** Tagging a paragraph _garantie_ when it lives inside a section titled _Auto guarantees_ often duplicates information already in `toc_df.breadcrumb`. Useful tags are the ones that **cross** the section boundary: _exclusion_ inside a _guarantee_ section, _condition_ that applies to two different guarantees.\n * **Taxonomy chicken-and-egg.** Without a fixed taxonomy, an LLM invents new tags each pass, producing label proliferation and cross-document inconsistency. With a fixed taxonomy, the tag set is brittle across domains (an auto taxonomy will not work on a health-insurance contract without adaptation).\n * **Paragraph unit is fuzzy.** Fitz’s paragraph boundary is not the semantic paragraph a human would draw. A single displayed paragraph can span multiple `line_df` blocks, be broken by page footers, or run across pages. Layer 2 needs a paragraph model, not just a line stream.\n * **Cost per document.** One thousand paragraphs is one thousand LLM calls if the tagger runs one paragraph at a time. Batching, cache, and hierarchical clustering (tag a group of similar paragraphs together) matter for practical use.\n\n\n### 5.4 Where this fits and what closes it\nThe tagging pass belongs to a later brick in the series. Two natural homes:\n * **Volume 2 (Multi-format, multi-intent documents).** The classification intent tags each paragraph against a fixed taxonomy. That is Layer 2, packaged as a first-class RAG intent.\n * **Volume 3 (Agentic Bricks).** An agent reads the taxonomy, walks the paragraphs it has not tagged yet, proposes tags, checks consistency across paragraphs of the same section, and re-runs on the paragraphs whose tags were rejected. That is Layer 2 with a feedback loop.\n\n\nTables are a third axis I did not open here. The parsing brick keeps each table entire; the retrieval brick decides whether to score the table as one chunk or as individual rows serialised with their column headers. A follow-up article in Part III builds that row-level retrieval loop and pairs it with this parsing choice.\nThe TOC loop this article built stays useful on its own. It is the first axis of the super structure. Layers 2 and 3 come on top when the use case calls for them.\n## 6. Conclusion\nA PDF that has neither an outline nor a printed contents page is not a dead end. The body carries every signal a heading gives off: font size, boldness, numeric prefix, short length, left alignment, blank line above. Six per-line scoring functions read from an enriched `line_df` (the parser matrix says which parsers give what), a weighted sum surfaces candidates, and the LLM validates them in a bounded loop that stops on convergence. Rules propose, LLM validates. The same subpackage extends a partial native outline (mode `extend_native`) and reconciles composite files (mode `composite`), all through one entry point that returns the same `toc_df` shape as the table of content cascade, with a `source` column so an audit can see which case fired. Retrieval, chunking, summarisation keep working unchanged. Case 4 is the first place in the parsing brick where the LLM does real detection work rather than checking coherence, and it needs the harness discipline around it: injected `llm_parse` callable, fixed response schema, bounded pass count, cached raw responses.\nThe next article (5nonies, in the queue) closes the parsing brick with an **agentic parsing** loop: given a document, pick which parsing methods to run at all (fitz, Azure, Docling, EasyOCR, vision LLM), execute them in the right order, synthesise their outputs into one enriched corpus. Case 4 of this cascade is one of the tools that agent will pick from.\n## 7. Further reading and sources\n**Earlier in the series** (payable-window links, out-of-window titles listed without a link):\n * [Baseline Enterprise RAG, from PDF to highlighted answer](https://towardsdatascience.com/baseline-enterprise-rag-from-pdf-to-highlighted-answer-enterprise-document-intelligence-vol-1-1/). The four-brick pipeline that reads `toc_df` at the retrieval and chunking steps.\n * [Beyond extract_text: the two layers of a PDF that drive RAG quality](https://towardsdatascience.com/beyond-extract_text-the-two-layers-of-a-pdf-that-drive-rag-quality/). The first half of the parsing brick: the document’s nature, signals, and summary.\n * [Stop returning flat text from a PDF: the relational tables RAG needs](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/). The `toc_df` this article fills sits in the middle of that data model.\n * [Reconstructing the table of contents a PDF forgot to ship](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/). Cases 1 to 3 of the same cascade, the table of content path this article extends.\n\n\n**External sources** (worked examples and prior art):\n * Vaswani et al., _Attention Is All You Need_ , [arXiv:1706.03762](https://arxiv.org/abs/1706.03762), NeurIPS 2017. The paper we run the body-typography loop on in Section 4 (arXiv non-exclusive distribution).\n * NIST, _Framework for Improving Critical Infrastructure Cybersecurity, v1.1_ ([NIST CSF](https://doi.org/10.6028/NIST.CSWP.04162018), 2018). The second worked example (US government work, public domain).\n * PyMuPDF (fitz) documentation, [`page.get_text(\"dict\")`](https://pymupdf.readthedocs.io/en/latest/textpage.html#TextPage.extractDICT). The span-level API `build_span_df` reads.\n * Manning et al., _Introduction to Information Retrieval_ (Cambridge University Press, 2008), ch. 20. The “sections as retrieval scope” premise that motivates a rich `toc_df`.\n * Bast et al., _Extracting the table of contents of PDF documents_ , [DocEng 2010](https://doi.org/10.1145/1860559.1860594). Prior art on TOC recovery from body typography, focused on the deterministic layer this article extends with an LLM loop.\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-249d024535392fb76067.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm Applications](https://towardsdatascience.com/tag/llm-applications/)[Parsing](https://towardsdatascience.com/tag/parsing/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F&title=Building%20Document%20Structure%20with%20Loop%20Engineering%3A%20Recovering%20a%20PDF%E2%80%99s%20Outline%20from%20Body%20Typography%20for%20RAG)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F&title=Building%20Document%20Structure%20with%20Loop%20Engineering%3A%20Recovering%20a%20PDF%E2%80%99s%20Outline%20from%20Body%20Typography%20for%20RAG)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F&text=Building%20Document%20Structure%20with%20Loop%20Engineering%3A%20Recovering%20a%20PDF%E2%80%99s%20Outline%20from%20Body%20Typography%20for%20RAG)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/003.jpg)](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n## [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 13, 202611 min read\n * [![](../images/004.jpg)](https://towardsdatascience.com/dispatching-the-parsed-rag-question-chunk-strategy-model-tier-activations-audit/)\n## [Dispatching the Parsed RAG Question: Chunk Strategy, Model Tier, Activations, Audit](https://towardsdatascience.com/dispatching-the-parsed-rag-question-chunk-strategy-model-tier-activations-audit/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nEnterprise Document Intelligence [Vol.1 #6c] - The decisions the parser makes on top of the user string, using the document’s profile: dispatch, activations, full schema, three approaches to deciding what fires, the audit _meta block, and a broker-corpus walkthrough\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nJune 18, 202624 min read\n * [![](../images/005.jpg)](https://towardsdatascience.com/build-your-own-local-ai-coding-agent-with-gemma-4-and-opencode-2/)\n## [Build Your Own Local AI Coding Agent with Gemma 4 and OpenCode](https://towardsdatascience.com/build-your-own-local-ai-coding-agent-with-gemma-4-and-opencode-2/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nFrom installing Ollama to launching OpenCode with a local model, step by step.\n[Shuai Guo](https://towardsdatascience.com/author/shuaiguo/)\nJune 23, 20266 min read\n * [![](../images/006.jpg)](https://towardsdatascience.com/how-to-build-an-efficient-knowledge-base-for-ai-models/)\n## [How to Build an Efficient Knowledge Base for AI Models](https://towardsdatascience.com/how-to-build-an-efficient-knowledge-base-for-ai-models/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nBuilding a knowledge base for AI models isn’t a one-time task but an iterative process of refinement.\n[Nidhin Karunakaran Ponon](https://towardsdatascience.com/author/nidhin.karunakaranponon/)\nMay 4, 202615 min read\n * [![](../images/007.jpg)](https://towardsdatascience.com/mcp-in-practice/)\n## [MCP in Practice](https://towardsdatascience.com/mcp-in-practice/)\n[Agentic AI](https://towardsdatascience.com/category/artificial-intelligence/agentic-ai/)\nMapping power, concentration, and usage in the emerging AI developer ecosystem\n[Sruly Rosenblat](https://towardsdatascience.com/author/sruly.rosenblat/)\nSeptember 29, 202511 min read\n * [![](../images/008.jpg)](https://towardsdatascience.com/the-untaught-lessons-of-rag-retrieval-cosine-is-not-the-foundation/)\n## [6 common RAG retrieval mistakes we keep seeing in production](https://towardsdatascience.com/the-untaught-lessons-of-rag-retrieval-cosine-is-not-the-foundation/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #7ter] - Six positions on the retrieval brick that contradict the cosine-first reflex of mainstream RAG\n[Kezhan Shi](https://towardsdatascience.com/author/kezhanshi/)\nJuly 3, 20268 min read\n * [![](../images/009.jpeg)](https://towardsdatascience.com/hands-on-genai-for-product-engineering-leaders-6ee6ad94e058/)\n## [Hands-On GenAI for Product & Engineering Leaders](https://towardsdatascience.com/hands-on-genai-for-product-engineering-leaders-6ee6ad94e058/)\n[](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/)\nMake better product decisions by taking a peek under the hood of LLM-based products\n[Ninad Sohoni](https://towardsdatascience.com/author/ninadsohoni/)\nNovember 28, 202336 min read\n * [![](../images/010.jpg)](https://towardsdatascience.com/build-and-run-an-intelligent-document-processing-idp-system-in-the-cloud/)\n## [Build and Run an Intelligent Document Processing (IDP) System in the Cloud](https://towardsdatascience.com/build-and-run-an-intelligent-document-processing-idp-system-in-the-cloud/)\n[Programming](https://towardsdatascience.com/category/programming/)\nAutomating the classification and extraction of PII from emails using AWS\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nJuly 24, 202612 min read\n * [![](../images/011.jpg)](https://towardsdatascience.com/building-a-multimodal-rag-with-text-images-tables-from-sources-in-response/)\n## [Building a Multimodal RAG That Responds with Text, Images, and Tables from Sources](https://towardsdatascience.com/building-a-multimodal-rag-with-text-images-tables-from-sources-in-response/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nWhy do few chatbots return figures from source documents in their responses?\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nNovember 3, 202510 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n", "metadata": { "title": "Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG | Towards Data Science", "description": "Enterprise Document Intelligence [Vol.1 #5octies] - Rules propose, LLM validates: six deterministic signals on span-level typography surface heading candidates, one bounded loop keeps the real ones, and the same toc_df drops back into the RAG pipeline", "keywords": null, "author": "Angela and Kezhan Shi", "og:locale": "en_US", "og:type": "article", "og:title": "Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG", "og:description": "Enterprise Document Intelligence [Vol.1 #5octies] - Rules propose, LLM validates: six deterministic signals on span-level typography surface heading candidates, one bounded loop keeps the real ones, and the same toc_df drops back into the RAG pipeline", "og:url": "https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/open_dictionary_pen_226612_card.jpg", "og:image:width": "1200", "og:image:height": "630", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Angela and Kezhan Shi", "twitter:label2": "Est. reading time", "twitter:data2": "22 min read", "article:published_time": "2026-08-05T06:00:05.000Z", "article:modified_time": "2026-08-05T06:00:05.000Z", "article:author": "https://towardsdatascience.com/author/angela.shi/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 202, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/5b1794197a78/code/block_000.txt", "chars": 706 }, { "language": "txt", "file": "attachments/5b1794197a78/code/block_001.txt", "chars": 199 }, { "language": "txt", "file": "attachments/5b1794197a78/code/block_002.txt", "chars": 274 }, { "language": "txt", "file": "attachments/5b1794197a78/code/block_003.txt", "chars": 236 }, { "language": "txt", "file": "attachments/5b1794197a78/code/block_004.txt", "chars": 410 }, { "language": "txt", "file": "attachments/5b1794197a78/code/block_005.txt", "chars": 474 } ], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:10:55Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:10:55Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:10:55Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:10:55Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/large-language-model/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/angela.shi/" }, { "type": "internal", "href": "https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/" }, { "type": "internal", "href": "https://towardsdatascience.com/amplify-the-expert-a-philosophy-for-building-enterprise-rag/" }, { "type": "internal", "href": "https://towardsdatascience.com/beyond-extract_text-the-two-layers-of-a-pdf-that-drive-rag-quality/" }, { "type": "internal", "href": "https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/" }, { "type": "internal", "href": "https://towardsdatascience.com/baseline-enterprise-rag-from-pdf-to-highlighted-answer-enterprise-document-intelligence-vol-1-1/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/document-processing/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/llm-applications/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/parsing/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/rag/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/rag-architecture/" }, { "type": "internal", "href": "https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/" }, { "type": "internal", "href": "https://towardsdatascience.com/dispatching-the-parsed-rag-question-chunk-strategy-model-tier-activations-audit/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://www.pexels.com/photo/226612/" }, { "type": "external", "href": "https://github.com/doc-intel/notebooks-vol1" }, { "type": "external", "href": "https://roadmap.sh/ai-engineer" }, { "type": "external", "href": "https://arxiv.org/abs/1706.03762" }, { "type": "external", "href": "https://doi.org/10.6028/NIST.SP.800-207" }, { "type": "external", "href": "https://doi.org/10.6028/NIST.SP.800-171r2" }, { "type": "external", "href": "https://doi.org/10.6028/NIST.FIPS.199" }, { "type": "external", "href": "https://doi.org/10.6028/NIST.SP.1800-32" }, { "type": "external", "href": "https://www.fema.gov/flood-insurance/work-with-nfip/manuals" }, { "type": "external", "href": "https://doi.org/10.6028/NIST.CSWP.04162018" }, { "type": "external", "href": "https://pymupdf.readthedocs.io/en/latest/textpage.html" }, { "type": "external", "href": "https://doi.org/10.1145/1860559.1860594" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=Building+Document+Structure+with+Loop+Engineering%3A+Recovering+a+PDF%E2%80%99s+Outline+from+Body+Typography+for+RAG&u=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=Building+Document+Structure+with+Loop+Engineering%3A+Recovering+a+PDF%E2%80%99s+Outline+from+Body+Typography+for+RAG&url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F" }, { "type": "external", "href": "https://x.com/share?text=Building+Document+Structure+with+Loop+Engineering%3A+Recovering+a+PDF%E2%80%99s+Outline+from+Body+Typography+for+RAG&url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F" }, { "type": "external", "href": "https://www.youtube.com/c/TowardsDataScience" }, { "type": "external", "href": "https://www.threads.net/@towardsdatascience" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:34:49Z", "source_url": "https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\n# Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG\nEnterprise Document Intelligence [Vol.1 #5octies] - Rules propose, LLM validates: six deterministic signals on span-level typography surface heading candidates, one bounded loop keeps the real ones, and the same toc_df drops back into the RAG pipeline\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 5, 202622 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/open_dictionary_pen_226612_card.jpg)\n[Photo](https://www.pexels.com/photo/226612/) by icon0.com, via Pexels.\nA structured RAG pipeline needs the document’s table of contents: retrieval scopes by section, and the chunker cuts on heading boundaries instead of mid-sentence. But some documents don’t have one. A paper exported straight from LaTeX has no native outline and no printed contents page, so there is no `toc_df` for the pipeline to read. The headings are still right there on the body, section by section, bigger and bolder than the text around them. [Article 5septies (TOC reconstruction from a table of content)](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/) handled the document that at least prints a contents page; this one prints none. So the pipeline rebuilds the `toc_df` from the one signal left: the way the headings look.\nThis article is a document parsing companion in [Enterprise Document Intelligence](https://towardsdatascience.com/amplify-the-expert-a-philosophy-for-building-enterprise-rag/), the series that builds an enterprise RAG system from four bricks. It sits in **brick 1 (document parsing)** and closes the TOC-reconstruction thread that [Article 5 (document parsing)](https://towardsdatascience.com/beyond-extract_text-the-two-layers-of-a-pdf-that-drive-rag-quality/), [Article 5B (the relational data model)](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/), and [Article 5septies (TOC reconstruction from a table of content)](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/) opened.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-498-1024x572.png)\n_where this article sits: brick 1 (document parsing), Part II, the TOC-reconstruction case deferred by Article 5septies - Image by author_\n📓 **The runnable notebook** walks the attention paper (`data/paper/1706.03762v7.pdf`) through the loop from `line_df` + `span_df` to a 24-entry `toc_df` with 21 real headings and 3 false positives that the LLM validation drops: [doc-intel/notebooks-vol1](https://github.com/doc-intel/notebooks-vol1).\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n## 1. Where this article sits and what it does\n[Article 5septies (TOC reconstruction from a ](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/)[](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/)[table of content](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/)[)](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/) drew a line: TOC _detection_ reads a page that already exists (a native outline or a printed table of content); recovering structure from body typography belonged to summarisation. That line was pragmatic. It kept the table of content cascade small and testable. But the summarisation intent has different failure modes (long-context, prompt-first) and different guarantees (no fixed shape, no source column). For a document that has clear typographic headings on its body, we do not need a summarisation loop. We need the same `toc_df` shape as everywhere else, filled from the signals the body carries. So the frontier moves: body-typography reconstruction is a fourth _detection_ case, not a summarisation fallback.\n> **In:** a `line_df` (plus a `span_df` when the parser exposes typography) whose native TOC and printed table of content both came up empty. **Out:** a `toc_df` in the standard [Article 5B](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/) shape, so retrieval, chunking, and summarisation keep working unchanged.\nOnce you have that fourth case, three neighbouring situations come along for free:\n * The PDF has **no structure at all** , the case just described.\n * The PDF has a **partial native outline** that stops at level 2 while the body clearly has a level 3 (`3.2.1 ...`, `3.2.2 ...`). The body-typography pass deepens what the outline gives you.\n * The PDF is **composite** : two or more documents concatenated. Section numbering re-inits mid-file, and the native outline (if any) is flat. The same pass, with a merge step, reconciles the streams.\n\n\nThe rest of the article walks each in turn, in the same order the cascade tries them.\n## 2. Four cases, one cascade\nThe cascade grew by one. Same order: cheapest first, drop through to the next when a case does not fire or returns too little. Case 4 is the new one; the earlier three come from [Article 5septies (TOC reconstruction from a table of content)](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/) unchanged.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-497-1024x316.png)\n_Case 4 is the new one: the only case where the LLM does real detection, not coherence-checking - Image by author_\n * **Case 1** , native outline, and **Case 2** , contents page with links. Article 5septies (TOC reconstruction from a table of content) covers both. Deterministic, cheap, exact when they work.\n * **Case 3** , printed contents page without links. Same article, text patterns plus label-to-page alignment.\n * **Case 4** , no contents page at all. This article. Body typography surfaces heading candidates; an LLM loop keeps the real ones.\n\n\nCase 4 is opt-in in the cascade because it is genuinely the most expensive: several passes of an LLM in the worst case, one pass when the deterministic signals are clean enough. Enable it by passing `methods=(\"links\", \"contents_text\", \"llm\", \"body_structure\")` to `reconstruct_toc_df`, or call `reconstruct_toc_from_body` directly when you know upfront that the file has no contents page.\n## 3. How the cascade reconstructs a TOC\nWith the four cases mapped, here is how the cascade actually turns a document’s pages into a `toc_df`, one stage at a time, from the signals it reads off the page to the LLM check at the end.\n### 3.1 The parser matrix: what each parsing method gives us\nEvery rule the body-typography detector reads is a _column of the extracted table_. So the question is not _“what are the six signals”_ first, it is _“which columns does the parser give me at all”_. The answer depends on how the PDF was parsed. The four bricks live one level up; here we live inside brick 1, and brick 1 has (already, in the earlier articles) several implementations.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-499-1024x338.png)\n_How four parsers differ on TOC signals, from fitz (richest) down to EasyOCR (position and text only) - Image by author_\nTwo practical points that shape the code:\n 1. **PyMuPDF exposes typography per** _**span**_**, not per** _**line**_**.** A span is one contiguous run of characters in one font, size, weight, italic and color. A line that mixes bold and non-bold text expands to multiple spans. So the module provides a helper `enrich_line_df_with_style(line_df, span_df)` that aggregates the spans of each line down to one row and appends `font_size` (character-weighted), `bold_ratio`, `is_bold`, `is_italic`, `dominant_font_name`. Consumers pass a `line_df` when the parser has no typography, or a `line_df` **plus** a `span_df` when it does.\n 2. **Missing typography does not break the loop.** Each of the six signal functions checks the columns it needs and returns a zero series when they are absent. Font size and bold contribute 0.0 to the score in that case; the four position and text signals (numeric prefix, short length, left alignment, blank-line-above) carry the load. Azure OCR Layout and EasyOCR both fall in this regime.\n\n\n`Mistral OCR` is a special case: it returns the document already in markdown, and its `# / ## / ###` headings _are_ the reconstructed outline. When you have a Mistral output you skip Case 4 entirely and read the markdown structure. That is the parser matrix’s shortcut column.\n### 3.2 Six signals a heading gives off\nNow the six signals themselves. Each is one function of one input frame, each returns a `pd.Series[float]` in `[0, 1]`, and the combined score is a plain weighted sum.\n 1. **Font size ratio**. A heading uses a larger font than body prose. Score = `(font_size / median_body_size) - 1`, clipped to [0, 1]. A line at 12pt in a body of 10pt scores 0.2; at 16pt in a 10pt body it saturates to 1.0.\n 2. **Boldness**. Headings are often bold. When the enriched line_df carries `bold_ratio`, we use that; otherwise `is_bold` as a 0/1 fallback.\n 3. **Numeric prefix**. `1.`, `1.2`, `1.2.3`, `I.`, `A.`. Regex `^\\s*(?:\\d+(?:\\.\\d+)*\\.?|[IVXLCDM]+\\.|[A-Z]\\.)(?:\\s|$)`. Present ⇒ 1, absent ⇒ 0. Also carries the _level_ signal: `1.2.3` is a level-3 heading. **Fitz artefact.** A LaTeX-exported PDF often emits `\"1\"` and `\"Introduction\"` as two separate lines; the loop pre-merges bare-number lines with their next line via `merge_split_headings` before scoring, so the numeric prefix is recovered on the merged line.\n 4. **Short length**. A heading is short. Score linearly decays from 1 for lines shorter than 20 characters to 0 at 90 characters. A paragraph rarely lives on a single line; a heading rarely spills across two.\n 5. **Left alignment**. A heading starts at the left margin (or the block’s own left indent). Score = 1 when `x0` matches the page’s median left margin within a small tolerance, decays with distance.\n 6. **Blank line above**. Headings sit in visual isolation. Score = 1 when the vertical gap above the line is wider than the median gap between prose lines on that page.\n\n\nNone of these is decisive on its own. Bold-and-large by itself catches figure captions (_“Figure 3: architecture”_ is often bold and larger). Numeric prefix by itself catches enumerated lists inside a paragraph. The point is to combine them, and combining them is where the weights live.\npython\n\n```\n# The six per-line signals, cheap, deterministic, engine-agnostic.\nfrom docintel.parsing.pdf.toc.body_structure import (\n enrich_line_df_with_style,\n score_font_size_ratio,\n score_is_bold,\n score_has_numeric_prefix,\n score_is_short,\n score_is_left_aligned,\n score_has_blank_before,\n)\n\nline_df = enrich_line_df_with_style(line_df, span_df)\n\nsignals = {\n \"font_size_ratio\": score_font_size_ratio(line_df),\n \"is_bold\": score_is_bold(line_df),\n \"has_numeric_prefix\":score_has_numeric_prefix(line_df),\n \"is_short\": score_is_short(line_df),\n \"is_left_aligned\": score_is_left_aligned(line_df),\n \"has_blank_before\": score_has_blank_before(line_df),\n}\n```\n\n### 3.3 Combining signals into candidates\nA weighted sum turns six scores into one. Weights come from prior belief, not from a training run. Numeric prefix is the strongest single signal (1.6). Font size ratio next (1.4). Boldness middle (1.0). Blank-line-above (0.8), short length (0.6), left alignment (0.4) are supporting evidence. Sum of weights is 5.8. The default threshold is 3.0 (about half the sum), which on the attention paper picks up every real heading plus a handful of table-cell false positives.\npython\n\n```\ncandidates_df = detect_body_headings(line_df, threshold=3.0)\n\ncandidates_df[[\"text\", \"heading_score\", \"candidate_level\",\n \"signal_font_size_ratio\", \"signal_has_numeric_prefix\"]].head()\n```\n\n**On the attention paper (Vaswani et al. 2017, an arXiv LaTeX export), the deterministic pass at threshold 3.5 finds 24 candidates** : 21 real section headings (from `1 Introduction` to `7 Conclusion`, all subsections included) and 3 false positives (`28.4`, `4.33`, `26.4`, BLEU scores from the results table on page 8-9 that happen to be bold and short). Recall on real headings is 91 % (21 of 23 real headings, `1 Introduction` sits just below the threshold and comes back at 3.0). Precision is 88 %. Level accuracy from numeric prefix is 100 % on the recovered rows. Those are the numbers from a real run, not a projection.\n### 3.4 Rules propose, LLM validates\nHere is where the LLM does real detection work. Not free-form. Fixed schema. The prompt describes what a heading is, what a false positive looks like (figure caption, table row label, bold in-body emphasis, BLEU score in a results table), gives the candidates in reading order with page and snippet, and asks for a JSON list of the kept entries. Missed headings (small caps, italics, an unnumbered epilogue) can be added by the same pass because it also sees a slice of the surrounding lines.\nThe loop is bounded. At most `max_passes` iterations (3 by default). Each pass reads the current keep list and can drop entries or propose additions. Convergence is when a pass changes nothing.\npython\n\n```\ndef my_llm_parse(system_prompt: str, user_content: str) -> list[dict]:\n ... # user's LLM client, returns the kept-entries JSON list\n\ntoc_df = reconstruct_toc_from_body(\n line_df,\n span_df=span_df,\n mode=\"no_toc\",\n max_passes=3,\n llm_parse=my_llm_parse,\n)\n```\n\nTwo things worth spelling out. First, the LLM is told what it may return. The response schema is fixed (`title`, `page`, `level`, `source`), matching the table of content cascade so downstream code does not branch on which case fired. Second, `llm_parse` is an _injected_ callable. Tests hand in a mock; CI can replay from a JSON cache; the module never opens a socket by itself. That is the discipline every LLM-touching module in this series follows.\n### 3.5 Extending a partial native outline\nSome PDFs have a native outline that stops at level 2. `1. Introduction`, `1.1 Motivation`, `2. Method`, `2.1 Data`, and the file is done. But the body clearly runs `2.1.1`, `2.1.2`, `2.2.1`. The retrieval brick wants those. So does the chunker, if the section runs long.\n`reconstruct_toc_from_body` handles this with `mode=\"extend_native\"`. The native outline is preserved verbatim (each row carries `source=\"native\"`). The body-typography pass runs on top, and any candidate whose title-and-page pair does not already exist in the native outline is added with `source=\"body_structure\"`. The reader can see, row by row, which entries came from the file and which came from the body pass. That is what `source` is for.\npython\n\n```\ndeeper_toc = reconstruct_toc_from_body(\n line_df,\n existing_toc_df=native_toc, # from doc.get_toc(), stops at level 2\n span_df=span_df,\n mode=\"extend_native\",\n)\n\ndeeper_toc[deeper_toc[\"source\"] == \"body_structure\"].head()\n```\n\n`mode=\"auto\"` (the default) picks this branch when `existing_toc_df` is non-empty and its max level is `≤ 2`, so most callers do not have to think about which mode to pass.\n### 3.6 Composite documents, and the API in one call\nThe other case: a single PDF file that concatenates two or more documents. An arXiv paper followed by a supplementary memo. A request-for-proposal bundle where each vendor answer is its own document. A review dossier of the letter and the response side-by-side. The native outline (if any) is flat and confusing: two `\"1. Introduction\"` entries, page numbers that do not increase monotonically, a re-init of the numbering half-way through.\n`detect_document_boundaries` looks for three deterministic signals: numbering re-init (a `\"1.\"` on a page after `\"N.something\"`), style rupture (a jump in median font size between pages N-1 and N), and cover pages (a page with a large title, low line count, no body text). Each returns a small frame of boundary rows. When any boundary is confirmed, `reconstruct_toc_from_body(..., mode=\"composite\")` re-roots the outline: each internal document becomes a top-level entry (`\"Document 1\"`, `\"Document 2\"`, …) with the original outlines nested under.\nThis is the case where the merge is worth an LLM pass, because a re-init in numbering could also just be a document that legitimately restarts a counter inside a chapter. The `MERGE_PROMPT` asks the LLM to confirm each boundary or reject it, then to return the merged outline as a JSON list. The whole subpackage funnels through one public entry point:\npython\n\n```\ntoc_df = reconstruct_toc_from_body(line_df, span_df=span_df, mode=\"no_toc\")\ntoc_df = reconstruct_toc_from_body(\n line_df, existing_toc_df=native_toc, span_df=span_df, mode=\"extend_native\",\n)\ntoc_df = reconstruct_toc_from_body(\n line_df, existing_toc_df=native_toc, span_df=span_df, mode=\"composite\",\n)\ntoc_df = reconstruct_toc_from_body(line_df, existing_toc_df=native_toc, span_df=span_df, mode=\"auto\")\n```\n\nFor the pipeline caller who already uses `reconstruct_toc_df` and wants the body-typography case as a _cascade fallback_ rather than a separate call, add `\"body_structure\"` to the methods tuple: `reconstruct_toc_df(pdf_path, methods=(\"links\", \"contents_text\", \"llm\", \"body_structure\"))`. The cascade will try the table of content methods first and fall through to Case 4 when they return empty.\n## 4. Testing against native outlines: how well does it work\nThe honest way to score this is against ground truth. Take six PDFs that DO carry a native outline, hide it, run the body-typography loop on the body, and compare the reconstruction to the outline the file was hiding. The six fixtures are all tier-1 open sources with different flavours of structure:\n * Vaswani et al., [_Attention Is All You Need_](https://arxiv.org/abs/1706.03762) (arXiv 1706.03762, arXiv non-exclusive distribution). A LaTeX-exported NIPS paper, decimal numbering.\n * NIST, [_Zero Trust Architecture_](https://doi.org/10.6028/NIST.SP.800-207) (SP 800-207, US government work, public domain). Decimal numbering, dot-leader table of content.\n * NIST, [_Protecting Controlled Unclassified Information in Nonfederal Systems and Organizations_](https://doi.org/10.6028/NIST.SP.800-171r2) (SP 800-171r2, public domain). Nested numeric prefixes, heavy front matter.\n * NIST, [_Standards for Security Categorization of Federal Information and Information Systems_](https://doi.org/10.6028/NIST.FIPS.199) (FIPS 199, public domain). Very short, appendix-heavy.\n * NIST, [_Securing Distributed Energy Resources: An Example of Industrial Internet of Things Cybersecurity_](https://doi.org/10.6028/NIST.SP.1800-32) (SP 1800-32, public domain). A large practice guide, 152 pages, mixed numbering.\n * FEMA, [_NFIP Flood Insurance Manual, Appendices (Policy Forms)_](https://www.fema.gov/flood-insurance/work-with-nfip/manuals) (US government work, public domain). The hardest case in the set, appendices titled with Roman numerals (`IV. PROPERTY NOT INSURED`) and named sub-forms.\n\n\nThe script that runs the eval lives at `scripts/checks/eval_toc_body_structure_vs_native.py`; each row below is the deterministic pass ONLY (no LLM validation), threshold 3.0 with the default weights, `span_df` typography enrich on.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-500-1024x293.png)\n_High recall, lower precision, and level assignment that breaks on Roman-numeral outlines; page accuracy holds at 100% - Image by author_\nThree things read off the table.\n**Aggregate reporting: macro and micro tell different stories.** The `aggregate` row is the macro-average (each doc weighs one). The micro-average, weighted by native rows, is different: `299 / 415 = 72%` micro-recall and `299 / 1481 = 20%` micro-precision. The gap between macro (77% recall) and micro (72% recall) has a specific driver: FEMA NFIP (186 natives, 69% recall) and NIST SP 1800-32 (104 natives, 62% recall) carry most of the native count, and both sit below the macro. The macro over-represents Vaswani (only 22 natives at 100% recall). Both numbers matter; neither alone is enough.\n**The deterministic pass is a high-recall filter on decimal-numbered docs.** 75 to 100% recall on Vaswani, SP 800-207, SP 800-171r2 and FIPS 199. The 100% recall on Vaswani is the fitz-artefact merge doing its job: every `\"1\"` + `\"Introduction\"` pair reunited on the fly. Recall drops on the two non-decimal fixtures (FEMA NFIP 69%, NIST SP 1800-32 62%). The missed rows are the ones whose numbering the numeric-prefix regex does not match, plus level-3 named sections (`National Flood Insurance Program Dwelling Form`) that carry no prefix at all.\n**Precision without LLM is poor.** 7 to 38% across the corpus. The pass drags in a lot of noise, and the noise falls into five predictable categories.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-501-1024x337.png)\n_Every category is systematic; Section 3.4’s validation pass targets each, though the drop rate is not yet measured - Image by author_\n**Level accuracy is reported conditional on match, and that framing hides a bias.** The 100% level accuracy on the four decimal-numbered docs is `(matched rows where recon-level equals native-level) / matched`. It tells you that when the algorithm identifies a heading whose title matches a native, the numeric-prefix regex reads the level correctly. On FEMA NFIP and NIST SP 1800-32 the recon regex fails to assign a level at all (Roman numerals, named appendices), the matched rows carry NaN, and the conditional ratio collapses to 0. The honest global metric is `(native rows where recon predicted the correct level) / all native rows`. It lands at exactly `105 / 415 = 25%` micro across the six fixtures. **Page accuracy stays at 100% within ±1** on every matched row, but that near-tautology sits because both recon and native read `page_num` from fitz. The rows where it would matter (a heading whose native page is off by one from where fitz shows the text) are not represented in this fixture set.\n### 4.1 What the algorithm sees on a real page\nThe best way to make the decision visible is to overlay the algorithm’s per-line output on the raster of the page. Green boxes are lines the deterministic loop scored above the 3.0 threshold and kept as heading candidates. Grey outlines are the other lines the algorithm looked at (body prose, page headers, footnotes) but did not keep. The tag on the right shows what `span_df` exposed for each candidate: `(level, bold flag, font size, heading score)`.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-502-886x1024.png)\n_Attention paper page 2, algorithm overlay on the split “1” + “Introduction” heading pair - Image by author_\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-503-875x1024.png)\n_NIST SP 800-207 page 10, the same overlay pattern on a public-domain doc - Image by author_\nTwo things read off the overlay. First, the number of green boxes per page is small (four on Vaswani p2, two on NIST p10), which confirms the threshold is picking headings rather than sprinkling them across the page. Second, the tags show the algorithm sees the right numbers: 11.9 pt bold for NIST section titles, 11.9 pt bold for Vaswani chapter titles, both with heading scores near 4.28 (well above the 3.0 threshold). The signals `span_df` exposes match what a human reader would flag as a heading, and the scorer combines them without needing per-document tuning.\nHere is the exact code that ran to produce these overlays. It reads the same `line_df` + `span_df` the rest of the pipeline reads, then loops over the candidates for the page and draws the boxes.\npython\n\n```\npdf = \"data/nist/NIST.SP.800-207.pdf\"\nline_df = fitz_pdf_to_line_df(pdf)\nspan_df = build_span_df(pdf)\nenriched = enrich_line_df_with_style(line_df, span_df) # per-line font + bold\ncandidates = detect_body_headings(enriched, threshold=3.0)\n\npage = 10\nfor _, row in candidates[candidates[\"page_num\"] == page].iterrows():\n print(row[\"text\"], row[\"heading_score\"], row[\"candidate_level\"],\n row[\"font_size\"], bool(row[\"is_bold\"]))\n# 1 Introduction 4.28 1 12.0 True\n```\n\n### 4.2 What the LLM validation buys\nEvery category in the taxonomy above is one the LLM is _designed_ to catch on the first pass.\n * Page numbers alone: a candidate whose text is a single digit is never a heading.\n * Table of content dot-leader lines: `\"1 Introduction ...........\"` is a contents-page entry, not a body heading.\n * Cover-page author names: no numeric prefix, wrong page.\n * Table cells: bold and short but not a section title.\n * The `HEADING_VALIDATION_PROMPT` names each category explicitly. Here is the measured effect of a single validation pass with `gpt-4.1` (Azure OpenAI, cache on) on the six-fixture eval.\n\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-506-1024x382.png)\n_One validation pass lifts precision from 20% to 87% across six fixtures; recall holds on five of six - Image by author_\nThree things read off the measurement.\n**Precision lifts massively.** From 15% up to 96% on the attention paper, 38% to 98% on NIST SP 800-207, 21% to 100% on FIPS 199, 31% to 82% on NIST SP 1800-32, 7% to 72% on SP 800-171r2. The LLM removes exactly the FP categories the taxonomy above named.\n**Recall is preserved on five of six fixtures.** The LLM validation is conservative in the good sense: it drops false positives without touching the real headings the deterministic pass already found. On Vaswani, 100% recall is preserved; on SP 800-207, 75%; on SP 800-171r2, 97%; on FIPS 199, 60%; on SP 1800-32, 62%. The micro-aggregate on those five docs stays close to before.\n**FEMA NFIP is the outlier where the LLM over-drops.** Recall collapses from 69% to 22%. The Roman-numeral appendices (`IV. PROPERTY NOT INSURED`) and named sub-forms (`National Flood Insurance Program Dwelling Form`) do not fit the LLM’s mental model of “section heading with a numeric prefix”, and it drops them alongside the false positives. This is a known limit: the LLM prompt was tuned on decimal-numbered outlines. A per-domain prompt or a two-pass loop with a corrective critic would recover here.\nThe catch that even the LLM does not fix is deeper. Look at what SP 800-207 has that the deterministic pass genuinely misses: unnumbered top-level titles like `\"References\"`, `\"Acronyms\"`, and the front-page `\"NIST SP 800-207, Zero Trust Architecture\"`. These do not carry a numeric prefix, and their font signal blends with the body. The LLM can propose them when it sees the surrounding context, but the deterministic recall does not include them by construction. That is the ceiling of the method: a document that wants a perfectly-nested reconstructed outline has to carry visible signals for every heading.\n### 4.3 Limitations of this eval\nThe numbers above are honest but small. Four caveats that a rigorous reader should know:\n * **The threshold 3.0 is a default, not a tuned choice.** No cross-validation, no ROC / precision-recall sweep. A per-corpus sweep would likely lift precision meaningfully on the noisier fixtures.\n * **The**`**matching**`**step is loose.** Titles normalize by stripping leading numbers, then compare exactly. A document with two sections named _Introduction_ (a body intro and an appendix intro) can match either, and the greedy assignment picks the closest page.\n * **The**`**page-within-1**`**metric is almost tautological.** Native and recon both derive `page` from fitz’s `page_num`, so an off-by-one at that stage is rare by construction. A stress test would randomize the ground-truth page assignment or use an OCR fixture where the page number is inferred.\n * **Six fixtures, English only, tier-1 open sources only.** No confidence intervals. No multilingual document. No RTL script. No handwritten / scanned insurance CG (which is tier-2 anyway). The pattern generalizes in ways this eval does not confirm.\n\n\nSection 3.4’s LLM validation pass has now been measured on the six-fixture eval (see Section 4.2 above). Micro precision moves from 20% to 87%, recall from 72% to 51% (the drop is entirely FEMA NFIP, whose Roman appendices confuse the LLM). Per-corpus prompt tuning is a real follow-up; the current prompt was written against decimal-numbered outlines.\n## 5. Beyond a single hierarchical TOC: multi-tag enriched structure\nA recovered TOC is one dimension of a document’s structure. It is not the whole thing. Two examples make this clear.\n**Insurance auto contract.** A section titled _3.2 Collision Guarantee_ explains coverage, then names the plafond, then names the exclusions that apply. The exclusions do not necessarily live in a separate chapter titled _Exclusions_. They live inside the guarantee section. A retrieval that scopes by section (via `toc_df.start_page`, `end_page`) and asks _“what exclusions apply to the collision guarantee?”_ pulls back the whole section. The answer is there, but so is a lot of noise, and the query cannot filter on _exclusion_ alone.\n**Composite / poorly-written docs.** A transcription, a meeting minute, a legal opinion, a forensic report. None of these have a clean hierarchical TOC. Themes recur across paragraphs, actors appear and reappear, and the useful navigation is not _“which section”_ but _“which topic”_. A hierarchical outline of a transcription is close to useless.\n### 5.1 Two layers per line or paragraph: level from typography, tags from the business\nThe idea, in one line: the algorithm recovers the **level** (5octies loop), and the business adds **tags** on top. Two layers stack on the same line or paragraph. Neither replaces the other.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-504-1024x429.png)\n_Two axes on the same paragraph: section level from the typographic loop, thematic tags from a business taxonomy - Image by author_\n * **Layer 1: LEVEL.** From the body-typography loop above. Chapter 3, section 3.2, subsection 3.2.1. This is `toc_df` and the section anchors on every paragraph via `start_page`/`end_page`.\n * **Layer 2: TAGS.** A `list[str]` per paragraph, drawn from a domain taxonomy. In the insurance example: `[\"garantie\", \"garantie:collision\", \"plafond\", \"exclusion\", \"exclusion:vitesse\"]`. Tags come from a fixed taxonomy defined by the business (or bootstrapped from the corpus and validated by an expert), then an LLM proposes tags per paragraph.\n\n\n**Retrieval implications** : the query _“what exclusions apply to the collision guarantee?”_ becomes an intersection on Layer 2 (`garantie:collision AND exclusion:*`), which is exact even when the answer lives inside a guarantee section, not an exclusions section. The query _“section 3.2 in full”_ still works on Layer 1. The two are complementary.\n### 5.2 A concrete example, drawn from a synthetic contract\nIllustration on a synthesized auto-contract paragraph (fabricated for the article, no real insurer named, the point is the pattern, not the fixture):\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/image-505-1024x580.png)\n_One paragraph holds both a guarantee and an exclusion, and Layer 2 tagging surfaces each theme separately - Image by author_\nThe example is synthetic on purpose. Real insurance conditions générales are tier-2 documents and cannot be published verbatim on TDS; the pattern this figure shows is representative, not a specific insurer’s wording.\n### 5.3 Honest limitations of the multi-tag layer\nThe idea is powerful. It is also hard. Four costs to name upfront:\n * **Redundancy with Layer 1.** Tagging a paragraph _garantie_ when it lives inside a section titled _Auto guarantees_ often duplicates information already in `toc_df.breadcrumb`. Useful tags are the ones that **cross** the section boundary: _exclusion_ inside a _guarantee_ section, _condition_ that applies to two different guarantees.\n * **Taxonomy chicken-and-egg.** Without a fixed taxonomy, an LLM invents new tags each pass, producing label proliferation and cross-document inconsistency. With a fixed taxonomy, the tag set is brittle across domains (an auto taxonomy will not work on a health-insurance contract without adaptation).\n * **Paragraph unit is fuzzy.** Fitz’s paragraph boundary is not the semantic paragraph a human would draw. A single displayed paragraph can span multiple `line_df` blocks, be broken by page footers, or run across pages. Layer 2 needs a paragraph model, not just a line stream.\n * **Cost per document.** One thousand paragraphs is one thousand LLM calls if the tagger runs one paragraph at a time. Batching, cache, and hierarchical clustering (tag a group of similar paragraphs together) matter for practical use.\n\n\n### 5.4 Where this fits and what closes it\nThe tagging pass belongs to a later brick in the series. Two natural homes:\n * **Volume 2 (Multi-format, multi-intent documents).** The classification intent tags each paragraph against a fixed taxonomy. That is Layer 2, packaged as a first-class RAG intent.\n * **Volume 3 (Agentic Bricks).** An agent reads the taxonomy, walks the paragraphs it has not tagged yet, proposes tags, checks consistency across paragraphs of the same section, and re-runs on the paragraphs whose tags were rejected. That is Layer 2 with a feedback loop.\n\n\nTables are a third axis I did not open here. The parsing brick keeps each table entire; the retrieval brick decides whether to score the table as one chunk or as individual rows serialised with their column headers. A follow-up article in Part III builds that row-level retrieval loop and pairs it with this parsing choice.\nThe TOC loop this article built stays useful on its own. It is the first axis of the super structure. Layers 2 and 3 come on top when the use case calls for them.\n## 6. Conclusion\nA PDF that has neither an outline nor a printed contents page is not a dead end. The body carries every signal a heading gives off: font size, boldness, numeric prefix, short length, left alignment, blank line above. Six per-line scoring functions read from an enriched `line_df` (the parser matrix says which parsers give what), a weighted sum surfaces candidates, and the LLM validates them in a bounded loop that stops on convergence. Rules propose, LLM validates. The same subpackage extends a partial native outline (mode `extend_native`) and reconciles composite files (mode `composite`), all through one entry point that returns the same `toc_df` shape as the table of content cascade, with a `source` column so an audit can see which case fired. Retrieval, chunking, summarisation keep working unchanged. Case 4 is the first place in the parsing brick where the LLM does real detection work rather than checking coherence, and it needs the harness discipline around it: injected `llm_parse` callable, fixed response schema, bounded pass count, cached raw responses.\nThe next article (5nonies, in the queue) closes the parsing brick with an **agentic parsing** loop: given a document, pick which parsing methods to run at all (fitz, Azure, Docling, EasyOCR, vision LLM), execute them in the right order, synthesise their outputs into one enriched corpus. Case 4 of this cascade is one of the tools that agent will pick from.\n## 7. Further reading and sources\n**Earlier in the series** (payable-window links, out-of-window titles listed without a link):\n * [Baseline Enterprise RAG, from PDF to highlighted answer](https://towardsdatascience.com/baseline-enterprise-rag-from-pdf-to-highlighted-answer-enterprise-document-intelligence-vol-1-1/). The four-brick pipeline that reads `toc_df` at the retrieval and chunking steps.\n * [Beyond extract_text: the two layers of a PDF that drive RAG quality](https://towardsdatascience.com/beyond-extract_text-the-two-layers-of-a-pdf-that-drive-rag-quality/). The first half of the parsing brick: the document’s nature, signals, and summary.\n * [Stop returning flat text from a PDF: the relational tables RAG needs](https://towardsdatascience.com/stop-returning-flat-text-from-a-pdf-the-relational-shape-rag-needs/). The `toc_df` this article fills sits in the middle of that data model.\n * [Reconstructing the table of contents a PDF forgot to ship](https://towardsdatascience.com/reconstructing-the-table-of-contents-a-pdf-forgot-to-ship/). Cases 1 to 3 of the same cascade, the table of content path this article extends.\n\n\n**External sources** (worked examples and prior art):\n * Vaswani et al., _Attention Is All You Need_ , [arXiv:1706.03762](https://arxiv.org/abs/1706.03762), NeurIPS 2017. The paper we run the body-typography loop on in Section 4 (arXiv non-exclusive distribution).\n * NIST, _Framework for Improving Critical Infrastructure Cybersecurity, v1.1_ ([NIST CSF](https://doi.org/10.6028/NIST.CSWP.04162018), 2018). The second worked example (US government work, public domain).\n * PyMuPDF (fitz) documentation, [`page.get_text(\"dict\")`](https://pymupdf.readthedocs.io/en/latest/textpage.html#TextPage.extractDICT). The span-level API `build_span_df` reads.\n * Manning et al., _Introduction to Information Retrieval_ (Cambridge University Press, 2008), ch. 20. The “sections as retrieval scope” premise that motivates a rich `toc_df`.\n * Bast et al., _Extracting the table of contents of PDF documents_ , [DocEng 2010](https://doi.org/10.1145/1860559.1860594). Prior art on TOC recovery from body typography, focused on the deterministic layer this article extends with an LLM loop.\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657699750_yw6uqg.jpeg)\nWritten By\n## [Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[See all from Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\n[Document Processing](https://towardsdatascience.com/tag/document-processing/)[Llm Applications](https://towardsdatascience.com/tag/llm-applications/)[Parsing](https://towardsdatascience.com/tag/parsing/)[Rag](https://towardsdatascience.com/tag/rag/)[Rag Architecture](https://towardsdatascience.com/tag/rag-architecture/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F&title=Building%20Document%20Structure%20with%20Loop%20Engineering%3A%20Recovering%20a%20PDF%E2%80%99s%20Outline%20from%20Body%20Typography%20for%20RAG)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F&title=Building%20Document%20Structure%20with%20Loop%20Engineering%3A%20Recovering%20a%20PDF%E2%80%99s%20Outline%20from%20Body%20Typography%20for%20RAG)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fbuilding-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag%2F&text=Building%20Document%20Structure%20with%20Loop%20Engineering%3A%20Recovering%20a%20PDF%E2%80%99s%20Outline%20from%20Body%20Typography%20for%20RAG)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/08/toll_plaza_aerial_34282750_card.jpg)](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n## [Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model](https://towardsdatascience.com/cut-an-enterprise-rag-pipelines-latency-and-cost-by-calling-the-llm-less-not-by-buying-a-faster-model/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #9ter] - The pipeline from Article 9 calls a model at several steps to be sure it is right. On easy questions that is needless latency. A per-question signal routes them past the model, about two seconds saved for a keyword match.\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nAugust 13, 202611 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/compare_pressed_flowers_7341163_v3_card.jpg)](https://towardsdatascience.com/dispatching-the-parsed-rag-question-chunk-strategy-model-tier-activations-audit/)\n## [Dispatching the Parsed RAG Question: Chunk Strategy, Model Tier, Activations, Audit](https://towardsdatascience.com/dispatching-the-parsed-rag-question-chunk-strategy-model-tier-activations-audit/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nEnterprise Document Intelligence [Vol.1 #6c] - The decisions the parser makes on top of the user string, using the document’s profile: dispatch, activations, full schema, three approaches to deciding what fires, the audit _meta block, and a broker-corpus walkthrough\n[Angela and Kezhan Shi](https://towardsdatascience.com/author/angela.shi/)\nJune 18, 202624 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/ChatGPT-Image-Jun-18-2026-10_36_02-PM.jpg)](https://towardsdatascience.com/build-your-own-local-ai-coding-agent-with-gemma-4-and-opencode-2/)\n## [Build Your Own Local AI Coding Agent with Gemma 4 and OpenCode](https://towardsdatascience.com/build-your-own-local-ai-coding-agent-with-gemma-4-and-opencode-2/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nFrom installing Ollama to launching OpenCode with a local model, step by step.\n[Shuai Guo](https://towardsdatascience.com/author/shuaiguo/)\nJune 23, 20266 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/05/Knowledge-base-AI-models.jpg)](https://towardsdatascience.com/how-to-build-an-efficient-knowledge-base-for-ai-models/)\n## [How to Build an Efficient Knowledge Base for AI Models](https://towardsdatascience.com/how-to-build-an-efficient-knowledge-base-for-ai-models/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nBuilding a knowledge base for AI models isn’t a one-time task but an iterative process of refinement.\n[Nidhin Karunakaran Ponon](https://towardsdatascience.com/author/nidhin.karunakaranponon/)\nMay 4, 202615 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/09/3502055a-222e-40db-a0c8-50b40f12a0ac-1.jpg)](https://towardsdatascience.com/mcp-in-practice/)\n## [MCP in Practice](https://towardsdatascience.com/mcp-in-practice/)\n[Agentic AI](https://towardsdatascience.com/category/artificial-intelligence/agentic-ai/)\nMapping power, concentration, and usage in the emerging AI developer ecosystem\n[Sruly Rosenblat](https://towardsdatascience.com/author/sruly.rosenblat/)\nSeptember 29, 202511 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/card_catalog_kAQo6CJCPN4_v3_card.jpg)](https://towardsdatascience.com/the-untaught-lessons-of-rag-retrieval-cosine-is-not-the-foundation/)\n## [6 common RAG retrieval mistakes we keep seeing in production](https://towardsdatascience.com/the-untaught-lessons-of-rag-retrieval-cosine-is-not-the-foundation/)\n[Large Language Model](https://towardsdatascience.com/category/large-language-model/)\nEnterprise Document Intelligence [Vol.1 #7ter] - Six positions on the retrieval brick that contradict the cosine-first reflex of mainstream RAG\n[Kezhan Shi](https://towardsdatascience.com/author/kezhanshi/)\nJuly 3, 20268 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/11/1Lq_eXvsxDws0Wu48yVwyyw.jpeg)](https://towardsdatascience.com/hands-on-genai-for-product-engineering-leaders-6ee6ad94e058/)\n## [Hands-On GenAI for Product & Engineering Leaders](https://towardsdatascience.com/hands-on-genai-for-product-engineering-leaders-6ee6ad94e058/)\n[](https://towardsdatascience.com/building-document-structure-with-loop-engineering-recovering-a-pdfs-outline-from-body-typography-for-rag/)\nMake better product decisions by taking a peek under the hood of LLM-based products\n[Ninad Sohoni](https://towardsdatascience.com/author/ninadsohoni/)\nNovember 28, 202336 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/1_RurwGGGw2TXjPrXRLBdnnA.jpg)](https://towardsdatascience.com/build-and-run-an-intelligent-document-processing-idp-system-in-the-cloud/)\n## [Build and Run an Intelligent Document Processing (IDP) System in the Cloud](https://towardsdatascience.com/build-and-run-an-intelligent-document-processing-idp-system-in-the-cloud/)\n[Programming](https://towardsdatascience.com/category/programming/)\nAutomating the classification and extraction of PII from emails using AWS\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nJuly 24, 202612 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/11/Unsplash.jpg)](https://towardsdatascience.com/building-a-multimodal-rag-with-text-images-tables-from-sources-in-response/)\n## [Building a Multimodal RAG That Responds with Text, Images, and Tables from Sources](https://towardsdatascience.com/building-a-multimodal-rag-with-text-images-tables-from-sources-in-response/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nWhy do few chatbots return figures from source documents in their responses?\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nNovember 3, 202510 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n", "body_base_path": "attachments/5b1794197a78/pages/", "body_markdown_path": "attachments/5b1794197a78/pages/archived-body.md", "local_html": "attachments/5b1794197a78/pages/archived-body.html", "body_source": "prior_capture" }