{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Edward Izgorodin",
  "title_listed": "The Same GraphRAG Comparison Wins and Loses. It Depends Which Instrument Judged It.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/izgorodin/the-same-graphrag-comparison-wins-and-loses-it-depends-which-instrument-judged-it-fm9",
  "publish_date": "2026-08-29",
  "corpus_stage": "",
  "notes": "",
  "id": "a77549f4ca75",
  "collected_at": "2026-10-02T14:03:24",
  "title": "The Same GraphRAG Comparison Wins and Loses. It Depends Which Instrument Judged It. - DEV Community",
  "body_markdown": "One finding, and it is not about graphs. It is about how the result you are quoting was produced.\nFor breadth-oriented sensemaking questions, GraphRAG community summaries reached\n72 to 83 percent comprehensiveness\nand\n62 to 82 percent diversity\nwin rates, and root-level summaries cut answer tokens by about 97 percent (\narXiv:2404.16130\n). Those are the numbers the category quotes.\nThey were produced by an LLM judge, with no gold answers.\nScore the same kind of comparison against ground truth and it turns over. On ROUGE-2 against gold answers, GraphRAG loses to plain RAG:\n6.99 against 10.08 on SQuALITY\n, and\n3.23 against 6.32 on QMSum\n. The same paper reports that summary-ordering position bias can flip the LLM judge's preference on its own (\narXiv:2502.11371\n).\nBoth results are true\nThat is the part worth sitting with. Neither number is wrong and neither paper is sloppy.\nOne measures which answer a model prefers. The other measures overlap with a gold answer. They are different questions, so they are allowed to have different answers, and a comparison that looks decisive is often just one of the two questions asked loudly.\nThe same seam runs through the retrieval verdicts. On GraphRAG-Bench, \"RAG (w rerank)\" beat \"GraphRAG (local)\"\n60.92 to 49.29\non fact retrieval, while \"HippoRAG2\" beat \"RAG (w rerank)\"\n53.38 to 42.93\non complex reasoning (\narXiv:2506.05690\n). One table, two verdicts, depending on which row of it you were looking for.\nCost has the same problem\n\"Graph equals expensive\" is too crude to act on. On the Novel split of the same benchmark, query cost spans\nmore than two orders of magnitude within the graph methods themselves\n: about 331,375 total query tokens for \"MS-GraphRAG(global)\", about 100,832 for LightRAG, about 4,204 for Fast-GraphRAG, about 1,008 for \"HippoRAG2\", against about 879 for vanilla RAG.\nIndex build shows the same spread: roughly 9.2 million tokens for \"HippoRAG2\" against roughly 115.5 million for GraphRAG, about twelvefold (\narXiv:2502.14802\n, Table 12).\nThose two figures come from different primaries with different method sets, and no single primary puts index-build cost and query cost for the same method on one axis. Which is itself the lesson: the tidy cost model you want does not exist yet, and assembling one from two papers gives you a useful heuristic, not a controlled comparison.\nThe question that survives\nBefore accepting any retrieval win, mine included:\nWas this judged by a model, or scored against ground truth?\nIt takes ten seconds to check and it changes what the number means. A model-judged win says an evaluator preferred that answer. A ground-truth score says the answer overlapped with a known correct one. Quoting the first while implying the second is how a benchmark verdict gets mistaken for a decision framework.\nDisclosure: I work on\nMnemoverse\n, a memory engine for AI agents. Every number above comes from a linked paper, none of them ours. The longer version, with the full decision matrix, is on the\noriginal page\n.",
  "metadata": {
    "title": "The Same GraphRAG Comparison Wins and Loses. It Depends Which Instrument Judged It. - DEV Community",
    "description": "One finding, and it is not about graphs. It is about how the result you are quoting was... Tagged with ai, rag, llm, benchmarks.",
    "keywords": "ai, rag, llm, benchmarks, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/izgorodin/the-same-graphrag-comparison-wins-and-loses-it-depends-which-instrument-judged-it-fm9",
    "og:title": "The Same GraphRAG Comparison Wins and Loses. It Depends Which Instrument Judged It.",
    "og:description": "One finding, and it is not about graphs. It is about how the result you are quoting was...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx93iimf3205tupl814jx.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "The Same GraphRAG Comparison Wins and Loses. It Depends Which Instrument Judged It.",
    "twitter:description": "One finding, and it is not about graphs. It is about how the result you are quoting was...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx93iimf3205tupl814jx.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://arxiv.org/abs/2404.16130",
      "label": "arXiv:2404.16130",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2502.11371",
      "label": "arXiv:2502.11371",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2506.05690",
      "label": "arXiv:2506.05690",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2502.14802",
      "label": "arXiv:2502.14802",
      "type": "a"
    },
    {
      "href": "https://mnemoverse.com/",
      "label": "Mnemoverse",
      "type": "a"
    },
    {
      "href": "https://mnemoverse.com/docs/library/graphrag-tax",
      "label": "original page",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:31Z",
  "source_url": "https://dev.to/izgorodin/the-same-graphrag-comparison-wins-and-loses-it-depends-which-instrument-judged-it-fm9",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2404.16130",
      "bytes": 6893854,
      "reused": true,
      "local_path": "papers/repair/b101ca0e7d5283641404.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2502.11371",
      "bytes": 849723,
      "reused": false,
      "local_path": "papers/repair/ec373aad51db83e5cac3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2506.05690",
      "bytes": 31150461,
      "reused": false,
      "local_path": "papers/repair/65355a9bd79f2598e862.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2502.14802",
      "bytes": 2498688,
      "reused": false,
      "local_path": "papers/repair/c73c6a41b6bd36ef1b63.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 3017,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/a77549f4ca75/pages/source.html",
  "local_html": "attachments/a77549f4ca75/pages/article.html",
  "body_base_path": "attachments/a77549f4ca75/pages/",
  "body_markdown_path": "attachments/a77549f4ca75/pages/article.md"
}