{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Panagiotis Gkilis",
  "title_listed": "My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/bedvibe_studios/my-agent-answers-from-06-of-its-corpus-and-reports-it-like-a-full-read-35e7",
  "publish_date": "2026-08-25",
  "corpus_stage": "",
  "notes": "",
  "id": "8b0a2842aa9d",
  "collected_at": "2026-10-02T13:52:17",
  "title": "My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read - DEV Community",
  "body_markdown": "Originally published at\nai.bedvibe.studio\n.\nMy portfolio RAG agent holds 1,003 indexed chunks. A question retrieves six and answers.\nSix of 1,003 is 0.598%.\nNothing in the response, the logs or the trace says the other 997 were never looked at. It reads exactly like an answer built from reading everything.\nThat is fine for\n\"what does he say about X.\"\nIt is\nnot evidence for\n\"does he mention X anywhere\"\n— and there is nothing in a normal RAG stack that tells those two apart.\nThe claim retrieval cannot support\nA compliance corpus: 2,431 policy documents. Someone asks whether there is a remote-work reimbursement policy. Retrieval returns eight chunks. None mentions reimbursement. The agent answers:\nThere is no remote-work reimbursement policy in the corpus.\nThat claim requires knowledge of 2,431 documents. It was made from eight. The agent did not lie and it did not hallucinate — it correctly reported what it found, and the\nshape of the sentence\nquietly upgraded a statement about eight documents into a statement about the corpus.\nAbsence of evidence in a retrieved fragment is not evidence of absence in the whole. Everyone knows this. No retrieval stack I have used records enough to enforce it, because the denominator is not carried anywhere near the answer.\n\"But our faithfulness score is 1.0\"\nThis is the first objection and it deserves a straight answer:\nfaithfulness cannot catch this, by construction.\nFaithfulness asks whether the answer is supported by the retrieved context — did the model invent anything beyond what it was given. The compliance answer above invents nothing. It scores a perfect faithfulness and may be false about 2,423 documents.\nFaithfulness scores\nanswer against context\n. Coverage scores\ncontext against corpus\n. Different axes, and a system can be perfect on the first while completely silent on the second.\nMetric\nWhat it needs\nAvailable at answer time?\nFaithfulness\nanswer + context\nYes\nContext precision\nanswer + context\nYes\nContext\nrecall\nground-truth annotations\nNo\n— offline evaluation only\nCorpus coverage\ncorpus size + retrieved count\nYes — and it is not reported\nContext recall is the metric that would catch it, and it needs labelled ground truth, so it lives in your evaluation harness and not in production. Corpus coverage needs two integers you already have.\nThe control is the shape of the sentence, not a percentage\nThis took longest to see, and it is why there is no threshold anywhere in the implementation.\n0.598% is not a bad number. It is a bad number for one class of sentence.\nIt is entirely adequate for\n\"he mentions Rust\"\n— you need the one chunk you are quoting and nothing else. It cannot support\n\"he never mentions Rust.\"\nSame retrieval, same six chunks, opposite verdicts, because the claim changed.\nA single coverage threshold cannot serve both. Set it low and it licenses the second sentence; set it high and it forbids the first.\nClaim\nCoverage required\n\"The policy says X\"\nAny — you need only the units you cite\n\"There is no policy about X\"\nExhaustive\n\"All policies require X\"\nExhaustive\n\"The most recent policy is…\"\nExhaustive\n— the unread remainder may hold the true maximum\n\"There are three mentions\"\nExhaustive\n— a count over a sample is an estimate\nEvery row needing exhaustive coverage has the same reason: it asserts something about the units that were\nnot\nread. 999,999 of 1,000,000 is 99.9999% and still cannot establish absence — a threshold there would be a lie with a decimal point on it.\nThree failures wearing one label\nThe second thing the missing denominator costs you is diagnosis. When a RAG answer is wrong, the post-mortem usually terminates at\nthe model hallucinated\n. That sentence hides at least three separate engineering problems:\nWhat actually happened\nWhat to fix\nThe relevant chunk was never retrieved\nThe retriever, embedding or query —\nnot\nthe model\nIt was retrieved, then dropped during context assembly\nContext construction — the model never saw it\nIt reached the model, which reasoned past it\nThe prompt or the model — the only reasoning failure\nThree different people, three different days, one identical symptom. And the distinction is trivially recordable —\nretrieved\n,\nin_context\n, and whether the answer was right — but almost nobody logs the middle number, so the first two are permanently indistinguishable after the fact.\nWhat I built\nAn MCP server for\nnotchecked\n, the coverage-accounting schema I wrote after hitting the silent-skip failure in four domains. The library types the gaps a\nprogram\nleaves. The server does it for an agent, which is where the failure moved.\ncoverage_retrieval(\n  target      = \"portfolio corpus\",\n  query       = \"Rust experience\",\n  corpus_size = 1003,\n  retrieved   = 6,\n  in_context  = 6,\n  claim_type  = \"absence\",\n)\n\n→ scope:      6 of 1003 corpus units (0.598%)\n  exhaustive: false\n  claim_supported: false\n\n  REFUSE_THIS_CLAIM: An ABSENCE claim asserts something about every unit\n  you did NOT read, and 997 were never read. Downgrade the answer to what\n  you found, or inspect the whole corpus.\nEnter fullscreen mode\nExit fullscreen mode\nretrieved = 0\nis not a thin answer, it is a retrieval failure, and it records as one. Everything retrieved and then dropped is a context failure and records as that instead. Neither is a gap in the agent's reasoning, and calling them one sends someone to debug a prompt for a day.\nThe bug this found in its own implementation\nI wrote a suite that replays six investigations of my own that produced wrong claims — a page judged from 3,000 of its 10,828 words, a search that had stripped the HTML so anything named only in an\nhref\nwas invisible, three different counts from three broken filesystem walks.\nOn its first run it failed, and it failed on my code rather than on the cases.\nexhaustive\nwas a bare assertion: an agent could pass\nexhaustive: true\nalongside\nscope: \"3,000 of 10,828 words\"\nand the absence warning was dropped.\nThe tool committed the exact failure it exists to prevent, one layer above the schema it protects.\nWhere the scope carries \"N of M\", the contradiction is machine-visible and is now refused.\nThat is the third time this idea has caught its own implementation. I have stopped finding it funny.\nWhat this does not do\nIf your retriever reports\ncorpus_size=100000, retrieved=20, in_context=12, cited=3\n, this records those numbers and what they can support.\nIt does not know whether the retriever chose the right twenty.\nIt is not a retriever, a vector database, a reranker or a context assembler, and it is not competing with the one you have.\nThree limits are recorded as\npassing tests\nrather than left out of the README: an exhaustive search of the\nwrong instrument\nis still exhaustive; the target list is\nself-declared\n, so nothing can know what the caller failed to think of; the retrieval counts are\nself-reported\n, with only internal consistency enforced.\nWhat it removes is the\nsilence\n. That is a smaller claim than \"this makes agents honest,\" and it is the one the evidence supports.\nI have measured this in\none\nlive system, my own. I have not established what other RAG deployments report, and I am not claiming it from a sample of one.\npip install notchecked\n·\ngithub.com/Mormolykos/notchecked\n— MIT, zero runtime dependencies, MCP over stdio written from the JSON-RPC wire format.\nThe eight coverage states were reviewed publicly by\nBoris Teplitsky\n, an IBM Certified Expert IT Architect who hit the same shape in infrastructure compliance and gave three corrections that changed the schema. They are frozen.\nIf you run RAG in production and you\ndo\nrecord retrieval coverage alongside answers, I would like to know\n— that would make this a solved problem I had not found the solution to, which is a better outcome than being right.",
  "metadata": {
    "title": "My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read - DEV Community",
    "description": "Retrieval is not the evidence — it is a measurement instrument, and nobody reports its coverage. Faithfulness cannot catch this: an answer perfectly grounded in 8 retrieved chunks can still be false about 2,423 documents. Tagged with rag, ai, architecture, python.",
    "keywords": "rag, ai, architecture, python, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/bedvibe_studios/my-agent-answers-from-06-of-its-corpus-and-reports-it-like-a-full-read-35e7",
    "og:title": "My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read",
    "og:description": "Retrieval is not the evidence — it is a measurement instrument, and nobody reports its coverage. Faithfulness cannot catch this: an answer perfectly grounded in 8 retrieved chunks can still be false about 2,423 documents.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fawonc1rjidpiketlglzd.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read",
    "twitter:description": "Retrieval is not the evidence — it is a measurement instrument, and nobody reports its coverage. Faithfulness cannot catch this: an answer perfectly grounded in 8 retrieved chunks can still be false about 2,423 documents.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fawonc1rjidpiketlglzd.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/8b0a2842aa9d/code/body-000.txt",
      "chars": 462,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://ai.bedvibe.studio/retrieval-coverage/",
      "label": "ai.bedvibe.studio",
      "type": "a"
    },
    {
      "href": "https://github.com/Mormolykos/notchecked",
      "label": "notchecked",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:45Z",
  "source_url": "https://dev.to/bedvibe_studios/my-agent-answers-from-06-of-its-corpus-and-reports-it-like-a-full-read-35e7",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 7766,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/8b0a2842aa9d/pages/source.html",
  "local_html": "attachments/8b0a2842aa9d/pages/article.html",
  "body_base_path": "attachments/8b0a2842aa9d/pages/",
  "body_markdown_path": "attachments/8b0a2842aa9d/pages/article.md"
}