{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Panagiotis Gkilis",
  "title_listed": "I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/bedvibe_studios/i-assumed-my-retriever-failed-at-stage-one-the-bigger-failure-was-at-stage-three-2k0k",
  "publish_date": "2026-08-31",
  "corpus_stage": "",
  "notes": "",
  "id": "ee49ac802890",
  "collected_at": "2026-10-02T13:52:14",
  "title": "I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three. - DEV Community",
  "body_markdown": "Hierarchical retrieval is supposed to help on long documents. Pick the right chapter, then search inside it. Narrow the haystack, then find the needle.\nOn my long-book benchmark it came\nthird of five\n, behind a chapter-summary chain and flat chunk RAG, and only barely ahead of naively grabbing the end of the book.\nmethod\ncontext precision\ncontext recall\nnaive_first_context\n0.1475\n0.1458\nnaive_last_context\n0.4150\n0.3302\nflat_chunk_rag\n0.3375\n0.4302\nchapter_summary_chain\n0.4000\n0.4771\nhierarchical_book_rag\n0.3475\n0.3365\nMy hypothesis was\nerror compounding\n: if stage one picks the wrong chapter, every later stage searches the wrong text. That turned out to be true and incomplete — which is worse than being wrong, because it would have sent me to fix the right thing and stop there.\nThe ablation\nSame corpus (240,767 words), same 80 gold questions, five variants, 400 scored rows. Two of the variants are\ngiven\nthe correct chapter — diagnostics, not deployable methods — purely to measure how much is lost before versus after chapter selection.\nmethod\nrecall\nprecision\nhit@1\nhit@3\nhit@5\nhier_current\n0.3365\n0.3475\n0.2000\n0.3875\n0.4375\nhier_no_neighbors\n0.4771\n0.4000\n0.2000\n0.3875\n0.4375\nchapter_summary_chain\n0.4771\n0.4000\n0.2000\n0.3875\n0.4375\nhier_oracle_chapter\n0.7844\n0.6796\n1.0000\n1.0000\n1.0000\nhier_oracle_chapter_neighbors\n0.7688\n0.6925\n1.0000\n1.0000\n1.0000\nDeleting a feature closed the entire gap.\nhier_no_neighbors\nlands on 0.4771 / 0.4000 — the same numbers as the baseline, to four decimals. The method I thought was structurally worse wasn't worse. It was carrying a stage that was hurting it.\nAnd look at hit@k.\nIdentical across the first three rows. Chapter selection did not change at all between\nhier_current\nand\nhier_no_neighbors\n— same code. Every point of recall that moved, moved downstream, in the stage that pads results with adjacent chunks.\nCounting failures directly\nfailure type\ncount\nneighbor_dilution\n32\nwrong_chapter\n27\nright_chapter_wrong_chunk\n7\nok\n14\nMy hypothesis accounted for 27 questions. The stage I hadn't suspected accounted for 32. I would have fixed chapter routing, seen a real improvement, and never looked at expansion — because the improvement would have confirmed the theory I walked in with.\nNeighbour expansion is not simply bad\nWith the oracle chapter, adding neighbours moves recall 0.7844 → 0.7688 and precision 0.6796 →\n0.6925\n. Recall down slightly, precision\nup\n. When you're already in the right chapter, expansion is roughly a wash and can help precision.\nThe damage happens when expansion runs on top of an uncertain chapter choice, crowding out the good chunks you did find with text that is only adjacent, not relevant. It is a stage whose value depends on the confidence of the stage above it — exactly the interaction a single end-to-end score cannot show you.\nSo the finding is not \"turn off neighbour expansion.\" It is\ntreat it as a tunable stage rather than a default\n, conditioned on router confidence.\nThe sequence that worked\nDisable each optional stage in turn.\nCheap, and it found the larger of my two problems.\nInsert an oracle at each boundary.\nNot deployable, but it partitions the loss into before/after.\nOnly then attribute the failure.\nIf I had skipped to step three, I'd have published \"hierarchical RAG underperforms chapter-summary retrieval on long narrative corpora\" — well-supported by my headline numbers, and wrong about the cause.\nWhat this does not show\nOne private narrative corpus, 80 gold questions written from the corpus rather than by independent annotators. Evidence-term overlap scoring, which is a lightweight audit signal and not full semantic correctness.\nNo confidence intervals\n— the package didn't compute them and I won't imply precision I didn't measure. The oracle variants read gold labels and are not production-realistic. This is not a universal rule against hierarchical retrieval.\nPaper:\nDiagnosing Hierarchical Retrieval Failure in Long-Document RAG\nDOI:\n10.5281/zenodo.20692450",
  "metadata": {
    "title": "I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three. - DEV Community",
    "description": "Hierarchical retrieval is supposed to help on long documents. Pick the right chapter, then search... Tagged with machinelearning, rag, llm, python.",
    "keywords": "machinelearning, rag, llm, python, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/bedvibe_studios/i-assumed-my-retriever-failed-at-stage-one-the-bigger-failure-was-at-stage-three-2k0k",
    "og:title": "I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three.",
    "og:description": "Hierarchical retrieval is supposed to help on long documents. Pick the right chapter, then search...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9crrgevzh4gm0yrx1mn5.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three.",
    "twitter:description": "Hierarchical retrieval is supposed to help on long documents. Pick the right chapter, then search...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9crrgevzh4gm0yrx1mn5.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://doi.org/10.5281/zenodo.20692450",
      "label": "10.5281/zenodo.20692450",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:44Z",
  "source_url": "https://dev.to/bedvibe_studios/i-assumed-my-retriever-failed-at-stage-one-the-bigger-failure-was-at-stage-three-2k0k",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3993,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ee49ac802890/pages/source.html",
  "local_html": "attachments/ee49ac802890/pages/article.html",
  "body_base_path": "attachments/ee49ac802890/pages/",
  "body_markdown_path": "attachments/ee49ac802890/pages/article.md"
}