{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Jason Liu",
  "title_listed": "There Are Only 6 RAG Evals",
  "keywords": "评估,RAG,评测",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/",
  "publish_date": "2025-05",
  "corpus_stage": "推理部署",
  "notes": "RAG真正重要的6个评估指标",
  "id": "6b05a56c40e5",
  "collected_at": "2026-10-02T14:29:24",
  "title": "There Are Only 6 RAG Evals - Jason Liu",
  "body_markdown": "There Are Only 6 RAG Evals\n¶\nThe world of RAG evaluation feels needlessly complex. Everyone's building frameworks, creating metrics, and generating dashboards that make you feel like you need a PhD just to know if your system is working.\nBut what if I told you there are only 6 fundamental ways to evaluate a RAG system?\nJust math and symmetry.\nThe Insight\n¶\nRAG systems have three core components:\nA question (Q)\nRetrieved context (C)\nAn answer (A)\nThat's it. Three variables.\nExhaustive by Design\nThe power of focusing on Question (Q), Context (C), and Answer (A) is that these three components, and their conditional relationships, cover\nevery\npossible aspect of RAG evaluation. There are no hidden variables.\nIf we look at this through the lens of conditional relationships — the quality of one component given another — we get exactly six possible relationships. No more, no less.\nThink about it: when one thing breaks in your RAG system, it's always one of these relationships failing. This relates directly to what we've learned about the systematic approach to RAG improvement - identifying specific failure points rather than making vague statements about \"making the AI better.\"\nThe 6 Core Evaluation Metrics\n¶\nLet's break down each relationship (I'll use the notation X|Y to mean \"quality of X given Y\"). Rather than treating all metrics equally, I'll organize them into three practical tiers based on implementation complexity and business impact - similar to how we think about the improvement flywheel for RAG products:\nTier 1: Foundation Metrics (Before RAG Evaluation)\n¶\nBefore we even get to our six core metrics, we need to acknowledge the foundation of any retrieval system:\nRetrieval Precision & Recall\n: These traditional information retrieval metrics measure how well your retriever finds relevant documents from the corpus. They're fast to compute, don't require LLMs, and provide quick feedback for retriever tuning.\nThis matches our approach to starting the flywheel with synthetic data - you need to establish baseline retrieval performance with clear, measurable metrics before moving to more complex evaluation techniques. These metrics serve as the leading indicators that predict future success, rather than lagging indicators that just tell you about past performance.\nTier 2: Primary RAG Relationships\n¶\n1. Context Relevance (C|Q)\nDefinition\n: How well do the retrieved chunks address the question's information needs? This measures whether your retriever component is doing its job—finding passages that contain information relevant to answering the user's question.\nExample (Good)\n:\nQuestion: \"What are the health benefits of meditation?\"\nContext: \"Regular meditation has been shown to reduce stress hormones like cortisol. A 2018 study in the Journal of Cognitive Enhancement found meditation improves attention and working memory.\"\nReasoning\n: Strong relevance. The context directly addresses multiple health benefits with specific details.\nExample (Bad)\n:\nQuestion: \"What are the health benefits of meditation?\"\nContext: \"Meditation practices vary widely across different traditions. Mindfulness meditation, which originated in Buddhist practices, focuses on present-moment awareness, while transcendental meditation uses mantras to achieve deeper states of consciousness.\"\nReasoning\n: Low relevance. Despite being factually correct about meditation, this context discusses types and origins rather than any health benefits. The retriever has found topically related content but missed the specific information need.\nIrrelevant Context Dooms Generation\nIf your retriever pulls irrelevant context, your generator is doomed from the start. This is a common pitfall, reflecting \"absence blindness\" where teams obsess over generation quality while neglecting to ensure retrieval (C|Q) is even working correctly.\n2. Faithfulness/Groundedness (A|C)\nDefinition\n: To what extent does the answer restrict itself only to claims that can be verified from the retrieved context? This evaluates the generator's ability to avoid hallucinations.\nExample (Good)\n:\nContext: \"The Great Barrier Reef is the world's largest coral reef system.\"\nAnswer: \"The Great Barrier Reef is the largest coral reef system in the world.\"\nReasoning\n: Perfect faithfulness. The answer only states what's in the context.\nExample (Bad)\n:\nContext: \"The Great Barrier Reef is the world's largest coral reef system. It stretches for over 2,300 kilometers along the coast of Queensland, Australia.\"\nAnswer: \"The Great Barrier Reef, the world's largest coral reef system, stretches for over 2,300 kilometers along Australia's eastern coast and is home to about 10% of the world's fish species.\"\nReasoning\n: Mixed faithfulness. The first part is supported, but the claim about \"10% of the world's fish species\" isn't in the provided context. This subtle hallucination appears plausible and might be factually correct, but it's not grounded in the retrieved context.\nWhy It Matters\n: Hallucination undermines trust. This is why we implement validation patterns, interactive citations, and chain-of-thought reasoning in our RAG applications - to catch errors before they reach users and build trust through transparency.\n3. Answer Relevance (A|Q)\nDefinition\n: How directly does the answer address the specific information need expressed in the question? This evaluates the end-to-end system performance.\nExample (Good)\n:\nQuestion: \"How does compound interest work in investing?\"\nAnswer: \"Compound interest works by adding the interest earned back to your principal investment, so that future interest is calculated on the new, larger amount.\"\nReasoning\n: High relevance. The answer directly explains the concept asked about.\nExample (Bad)\n:\nQuestion: \"How does compound interest work in investing?\"\nAnswer: \"Interest in investing can be simple or compound. Compound interest is more powerful than simple interest and is an important concept in finance. It's the reason why starting to invest early is so beneficial for long-term wealth building.\"\nReasoning\n: Low relevance. Despite being about compound interest, the answer doesn't actually explain the mechanism of how it works. It tells you it's important but fails to address the specific how question.\nWhy It Matters\n: This is the primary user experience metric. It's also why we focus on building feedback mechanisms that specifically ask \"Did we answer your question?\" rather than vague \"How did we do?\" feedback prompts. Specific feedback matched to this metric increases response rates dramatically.\nTier 3: Advanced RAG Relationships\n¶\n4. Context Support Coverage (C|A)\nDefinition\n: Does the retrieved context contain all the information needed to fully support every claim in the answer? This measures whether the context is both sufficient and focused.\nThis metric matches what we've learned about specialized retrievers and the query routing architecture. Different content types may require different retrieval approaches to ensure complete coverage. For instance, when answering questions about blueprints in construction projects, you might need both image retrieval and document retrieval working together.\n5. Question Answerability (Q|C)\nDefinition\n: Given the context provided, is it actually possible to formulate a satisfactory answer to the question? This evaluates whether the question is reasonable given the available information.\nThis matches the strategic rejection pattern we've discussed. When a query can't be answered with the available context, the most honest response is to acknowledge this limitation. This builds trust through transparency rather than generating a hallucinated answer.\n6. Self-Containment (Q|A)\nDefinition\n: Can the original question be inferred from the answer alone? This measures whether the answer provides enough context to stand on its own.\nThis matches our discussion of monologues and chain-of-thought approaches that make thinking visible. Answers that restate and address the core question directly create better user experiences, especially in asynchronous communication contexts.\nImplementing Tiered Evaluation in Practice\n¶\nBased on recent academic research and practical experience, here's how to approach RAG evaluation with these metrics:\nStart with Tier 1\n: Implement fast retrieval metrics for daily development\nUse precision, recall, MAP@K, and MRR@K to tune your retriever\nThese don't require LLM evaluation and provide quick feedback cycles\nThis matches our approach of starting the improvement flywheel with synthetic data and focused evaluation metrics before moving to more complex approaches.\nFocus on Tier 2\n: Implement the three primary RAG relationships\nThese core metrics (C|Q, A|C, A|Q) directly assess how well your RAG system functions\nMost benchmarks prioritize these three metrics\nUse LLM-based evaluation for more nuanced assessment of these relationships\nThis matches our focus on building feedback mechanisms and quality-of-life improvements that enhance trust and transparency.\nExtend to Tier 3\n: Add advanced metrics when you need deeper insights\nThese metrics (C|A, Q|C, Q|A) connect technical performance to business outcomes\nUse them for monthly evaluations, major releases, and strategic decisions\nDifferent domains may require emphasis on different Tier 3 metrics (e.g., medical RAG needs stronger C|A)\nThis matches our discussion of topic modeling and capability identification, recognizing that different query types may require different evaluation emphasis.\nLLM-as-Judge\n¶\nMost modern RAG evaluations rely on LLMs as judges. This approach, while resource-intensive, provides the most nuanced assessment of our six core relationships.\nThe Nuance of LLM Judges\nWhile resource-intensive, using LLMs as judges is currently the most effective method for capturing the subtle nuances in the six core RAG relationships. Traditional metrics often fall short in this complex assessment.\nSeveral benchmarks, including RAGAs, ARES, and TruEra RAG Triad, now use LLM evaluation by default. While traditional metrics like BLEU, ROUGE, and BERTScore still have a place, only LLM-based evaluation can effectively capture the nuanced relationships in our framework.\nThis matches our discussion about using LLMs to analyze feedback and identify patterns in user queries - leveraging AI to understand AI.\nDomain-Specific Evaluation\n¶\nAn interesting insight from the DomainRAG benchmark is that different domains may require different emphasis within our framework:\nMedical RAG systems need higher faithfulness scores (A|C)\nCustomer service RAG demands better answer relevance (A|Q)\nTechnical documentation RAG requires stronger question answerability (Q|C)\nThis matches what we've learned about topic modeling and segmentation - different query types need different capabilities, and our evaluation should reflect those priorities. It's why we segment questions not just by topic but by the capability required to answer them.\nWhy This Framework Matters\n¶\nWhen your RAG system fails, it fails along one of these dimensions. Every time.\nAnswer seems wrong? Check faithfulness (A|C).\nAnswer seems irrelevant? Check answer relevance (A|Q).\nAnswer missing key info? Check context relevance (C|Q) or context support (C|A).\nThe beauty of this framework is that it's complete. There are no other relationships between Q, C, and A. We've covered every possible evaluation angle.\nThis systematic approach to diagnosing problems matches our product mindset for RAG - identifying specific failure points rather than making vague statements about \"making the AI better.\"\nSo What?\n¶\nNext time you're debugging a RAG system, don't waste time on complexity theater. Focus on these six relationships organized in practical tiers. Fix the ones that are broken. Ignore the rest.\nThis framework matches our improvement flywheel - start with the basics, collect feedback on specific aspects, analyze that feedback to identify patterns, and make targeted improvements based on what you learn.\nAnd if someone tries to sell you a RAG evaluation framework with 20 different metrics? Smile and ask which of the 6 core relationships they're actually measuring.\nBecause in RAG evaluation, as in RAG implementation, the systematic approach wins every time.\nWhat's your experience with RAG evaluation? Do you find yourself focusing on particular metrics more than others? Drop a comment below—I'd love to hear which of these 6 relationships causes you the most headaches.\nWant to learn more?\n¶\nI also wrote a 6 week email course on RAG, where I cover everything in my consulting work. It's free and you can:\nCheck out the free email course here\nComments",
  "metadata": {
    "title": "There Are Only 6 RAG Evals - Jason Liu",
    "description": "There are only 6 fundamental ways to evaluate a RAG system. Here's how to do it.",
    "keywords": null,
    "author": "['jxnl']",
    "og:type": "website",
    "og:title": "There Are Only 6 RAG Evals - Jason Liu",
    "og:description": "There are only 6 fundamental ways to evaluate a RAG system. Here's how to do it.",
    "og:image": "https://jxnl.co/assets/images/social/writing/2025/05/19/there-are-only-6-rag-evals.png",
    "og:image:type": "image/png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:url": "https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/jxnl/blog/edit/main/docs/writing/posts/rag-only-6-evals.md",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/jxnl/blog/raw/main/docs/writing/posts/rag-only-6-evals.md",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://dub.link/6wk-rag-email",
      "label": "Check out the free email course here",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:55Z",
  "source_url": "https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 12647,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6b05a56c40e5/pages/source.html",
  "local_html": "attachments/6b05a56c40e5/pages/article.html",
  "body_base_path": "attachments/6b05a56c40e5/pages/",
  "body_markdown_path": "attachments/6b05a56c40e5/pages/article.md"
}