{ "source_file": "0930", "source_platform": "个人博客", "author": "Jason Liu", "title_listed": "Systematically Improving RAG", "keywords": "RAG,改进,系统化", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://jxnl.co/writing/2024/05/22/systematically-improving-your-rag/", "publish_date": "2024-05", "corpus_stage": "推理部署", "notes": "系统化改进RAG的方法", "id": "338cb3bfedef", "collected_at": "2026-10-02T14:29:24", "title": "Systematically Improving Your RAG - Jason Liu", "body_markdown": "Systematically Improving Your RAG\n¶\nThis article explains how to make Retrieval-Augmented Generation (RAG) systems better. It's based on a talk I had with\nHamel\nand builds on other articles I've written about RAG. For a comprehensive understanding of RAG fundamentals, see my guide on\nwhat RAG is\n.\nIn\nRAG is More Than Just Embeddings\n, I talk about how RAG is more than just vector embeddings. This helps you understand RAG better. I also wrote\nHow to Build a Terrible RAG System\n, where I show what not to do, which can help you learn good practices.\nIf you want to learn about how complex RAG systems can be, check out\nLevels of RAG Complexity\n. This article breaks down RAG into smaller parts, making it easier to understand. For quick tips on making your RAG system better, read\nLow Hanging Fruit in RAG\n.\nI also wrote about what I think will happen with RAG in the future in\nPredictions for the Future of RAG\n. This article talks about how RAG might be used to create reports in the future.\nAll these articles work together to give you a full guide on how to make RAG systems better. They offer useful tips for developers and companies who want to improve their systems. For additional improvement strategies, check out my\nsix tips for improving RAG\nand insights on\nRAG anti-patterns\n. If you're interested in AI engineering in general, you might enjoy my talk at the\nAI Engineer Summit\n. In this talk, I explain how tools like Pydantic can help with prompt engineering, which is useful for building RAG systems.\nThrough all these articles, I try to give you a complete view of RAG systems. I cover everything from basic ideas to advanced uses and future predictions. This should help you understand and do well in this fast-changing field.\nBy the end of this post, you'll understand my step-by-step approach to making RAG applications better for the companies I work with. We'll look at important areas like:\nMaking fake questions and answers to quickly test how well your system works\nUsing both full-text search and vector search together for the best results\nSetting up the right ways to get feedback from users about what you want to study\nUsing grouping to find sets of questions that have problems, sorted by topics and abilities\nBuilding specific systems to improve abilities\nConstantly checking and testing as you get more real-world data\nThis step-by-step runbook shows how to incrementally improve the performance and utility of your RAG applications. Let's dive in and explore how to systematically improve your RAG systems.\nStart with Synthetic Data\n¶\nI think the biggest mistake around improving the system is that most people are spending too much time on the actual synthesis without actually understanding whether or not the data is being retrieved correctly. To avoid this:\nCreate synthetic questions for each text chunk in your database\nUse these questions to test your retrieval system\nCalculate precision and recall scores to establish a baseline\nIdentify areas for improvement based on the baseline scores\nWhat we should be finding with synthetic data is that synthetic data should just be around 97% recall precision. And synthetic data might just look like something very simple to begin with.\nWe might just say, for every text chunk, I want it to synthetically generate a set of questions that this text chunk answers. For those questions, can we retrieve those text chunks? And you might think the answer is always going to be yes. But I found in practice that when I was doing tests against essays, full text search and embeddings basically performed the same, except full text search was about 10 times faster. This approach is part of my broader\nRAG flywheel strategy\n.\nWhereas when I did the same experiment on pulling issues from a repository, it was the case that full text search got around 55% recall, and then embedding search got around 65% recall. And just knowing how challenging these questions are on the baseline is super important to figure out what kind of experimentation you need to perform better. This will give you a baseline to work with and help you identify areas for improvement. For a detailed breakdown of evaluation metrics, see my guide on\nthe only 6 RAG evaluations you need\n.\nUtilize Metadata\n¶\nExtract and make searchable relevant metadata (e.g., date ranges, file names, ownership) to improve search results.\nExtract relevant metadata from your documents\nInclude metadata in your search indexes\nUse query understanding to extract metadata from user queries\nExpand search queries with relevant metadata to improve results\nFor example, if someone asks, \"What is the latest x, y, and z?\" Text search will never get that answer. Semantic search will never get that answer.\nYou need to perform query understanding to extract date ranges. There will be some prompt engineering that needs to happen. For enterprise implementations, see my guide on\nRAG enterprise process\n. That's the metadata, and being aware that there will be questions that people aren't answering because those filters can never be caught by full text search and semantic search.\nAnd what this looks like in practice is if you ask the question, what are recent developments in the field, the search query is now expanded out to more terms. There's a date range where the language model has reasoned about what recent looks like for the research, and it's also decided that you should only be searching specific sources. If you don't do this, then you may not get trusted sources. You may be unable to figure out what recent means.\nYou'll need to do some query understanding to extract date ranges and include metadata in your search.\nUse Both Full-Text Search and Vector Search\n¶\nUtilize both full-text search and vector search (embeddings) for retrieving relevant documents. Ideally, you should use a single database system to avoid synchronization issues.\nImplement both full-text search and vector search\nTest the performance of each method on your specific use case\nConsider using a single database system to store both types of data\nEvaluate the trade-offs between speed and recall for your application\nIn my experience, full-text search can be faster, but vector search can provide better recall.\nWhat ended up being very complicated was if you have a single knowledge base, maybe that complexity is fine, because you have more configuration of each one.\nBut one of my clients who was doing construction data, they had to create separate indices per project, and now they just had this exploding array of different data sources that get in or out of sync. Like, maybe the database has an outage, and now the data is not in the database, but it's in another system. So if the embedding gets pulled up, then text is missing.\nAnd this complex configuration becomes a huge pain. And so, for example, some tools are able to do all 3 in a single object. And so even if you had a lot of partitioned data sources, you can do full text search, embedding search, and write SQL against a single data object. And that has been really helpful, especially when you think about these examples where you want to find the latest. Now you can just do a full text search query and then order by date and have a between clause.\nTest both and see what works best for your use case.\nImplement Clear User Feedback Mechanisms\n¶\nImplement clear user feedback systems (e.g., thumbs up/down) to gather data on your system's performance and identify areas for improvement.\nAdd user feedback mechanisms to your application\nMake sure the copy for these mechanisms clearly describes what you're measuring\nAsk specific questions like \"Did we answer the question correctly?\" instead of general ones like \"How did we do?\"\nUse the feedback data to identify areas for improvement and prioritize fixes\nI find that it's important to build out these feedback mechanisms as soon as possible. And making sure that the copy of these feedback mechanisms explicitly describe what you're worried about.\nSometimes, we'll get a thumbs down even if the answer is correct, but they didn't like the tone. Or the answer was correct, but the latency was too high. Or it took too many hops.\nThis means we couldn't actually produce an evaluation dataset just by figuring out what was a thumbs up and a thumbs down. It was a lot of confounding variables. We had to change the copy to just \"Did we answer the question correctly? Yes or no.\" We need to recognize that improvements in tone and improvements in latency will come eventually. But we needed the user feedback to build us that evaluation dataset.\nMake sure the copy for these feedback mechanisms explicitly describes what you're worried about. This will help you isolate the specific issues users are facing.\nCluster and Model Topics\n¶\nAnalyze user queries and feedback to identify topic clusters, capabilities, and areas of user dissatisfaction. This will help you prioritize improvements.\nWhy should we do this? Let me give you an example. I once worked with a company that provided a technical documentation search system. By clustering user queries, we identified two main issues:\nTopic Clusters: A significant portion of user queries were related to a specific product feature that had recently been updated. However, our system was not retrieving the most up-to-date documentation for this feature, leading to confusion and frustration among users.\nCapability Gaps: Another cluster of queries revealed that users were frequently asking for troubleshooting steps and error code explanations. While our system could retrieve relevant documentation, it struggled to provide direct, actionable answers to these types of questions.\nBased on these insights, we prioritized updating the product feature documentation and implementing a feature to extract step-by-step instructions and error code explanations. These targeted improvements led to higher user satisfaction and reduced support requests.\nLook for patterns like:\nTopic clusters: Are users asking about specific topics more than others? This could indicate a need for more content in those areas or better retrieval of existing content. I explore this concept further in my post on\nRAG decomposition\nand\ntopics and capabilities\n.\nCapabilities: Are there types of questions your system categorically cannot answer? This could indicate a need for new features or capabilities, such as direct answer extraction, multi-document summarization, or domain-specific reasoning.\nBy continuously analyzing topic clusters and capability gaps, you can identify high-impact areas for improvement and allocate your resources more effectively. This data-driven approach to prioritization ensures that you're always working on the most critical issues affecting your users.\nOnce you have this in place, once you have these topics and these clusters, you can talk to domain experts for a couple of weeks to figure out what these categories are explicitly. Then, you can build out systems to tag that as data comes in.\nIn the same way that when you open up ChatGPT and make a conversation, it creates an automatic title in the corner. You can now do that for every question. As part of that capability, you can add the classification, such as what are the topics and what are the capabilities. Capabilities could include ownership and responsibility, fetching tables, fetching images, fetching documents only, no synthesis, compare and contrast, deadlines, and so on. For more on selecting the right tools and capabilities, see my post on\ntrade-offs in tool selection\n.\nYou can then put this information into a tool like Amplitude or Sentry. This will give you a running stream of the types of queries people are asking, which can help you understand how to prioritize these capabilities and topics.\nContinuously Monitor and Experiment\n¶\nContinuously monitor your system's performance and run experiments to test improvements.\nSet up monitoring and logging to track system performance over time\nRegularly review the data to identify trends and issues\nDesign and run experiments to test potential improvements\nMeasure the impact of changes on precision, recall, and other relevant metrics\nImplement changes that show significant improvements\nThis could include tweaking search parameters, adding metadata, or trying different embedding models. Measure the impact on precision and recall to see if the changes are worthwhile.\nOnce you now have these questions in place, you have your synthetic data set and a bunch of user data with ratings. This is where the real work begins when it comes to systematically improving your RAG.\nThe system will be running many clusters of topic modeling around the questions, modeling that against the thumbs up and thumbs down ratings to figure out what clusters are underperforming. It will then determine the count and probability of user dissatisfaction for each cluster.\nThe system will be doing this on a regular cadence, figuring out for what volume of questions and user satisfaction levels it should focus on improving these specific use cases.\nWhat might happen is you onboard a new organization, and all of a sudden, those distributions shift because their use cases are different. That's when you can go in and say, \"We onboarded these new clients, and they very much care about deadlines. We knew we decided not to service deadlines, but now we know this is a priority, as it went from 2% of questions asking about deadlines to 80%.\" You can then determine what kind of education or improvements can be done around that.\nBalance Latency and Performance\n¶\nFinally, make informed decisions about trade-offs between system latency and search performance based on your specific use case and user requirements.\nUnderstand the latency and performance requirements for your application\nMeasure the impact of different configurations on latency and performance\nMake trade-offs based on what's most important for your users\nConsider different requirements for different use cases (e.g., medical diagnosis vs. general search)\nHere, this is where having the synthetic questions that test against will effectively answer that question. Because what we'll do is we'll run the query with and without this parent document retriever, and we will have a recall with and without that feature and the latency improvement of that feature.\nAnd so now we'll be able to say, okay. Well, recall doubles. The latency increases by 20%, then a conversation can happen. Or, is that worth the investment? But if latency goes up double and the recall goes up 1%, again, it depends on, okay.\nWell, if this is a medical diagnostic, maybe I do care that the 1% is included because the stakes are so high. But if it's for a doc page, maybe the increased latency will reduce in churn.\nIf you can improve recall by 1%, and the results are too complex, it's not worth deploying it in the future as well.\nFor example, if you're building a medical diagnostic tool, a slight increase in latency might be worth it for better recall. But if you're building a general-purpose search tool, faster results might be more important.\nWrapping Up\n¶\nThis is was written based off of a 30 conversation with a client, so I know I'm skipping over many details and implementation details. Leave a comment and let me know and we can get into specifics.\nWant to learn more?\n¶\nThis was based on a 30-minute conversation, so I'm definitely skipping implementation details. The real breakthrough happens when you stop random improvements and start measuring what actually moves the needle:\nFree 6-Week RAG Email Course\nFull Course Breakdown\nRAG FAQ\nComments", "metadata": { "title": "Systematically Improving Your RAG - Jason Liu", "description": "Discover systematic strategies to enhance your Retrieval-Augmented Generation (RAG) systems for better performance and user experience.", "keywords": null, "author": "Jason Liu", "og:type": "website", "og:title": "Systematically Improving Your RAG - Jason Liu", "og:description": "Discover systematic strategies to enhance your Retrieval-Augmented Generation (RAG) systems for better performance and user experience.", "og:image": "https://jxnl.co/assets/images/social/writing/2024/05/22/systematically-improving-your-rag.png", "og:image:type": "image/png", "og:image:width": "1200", "og:image:height": "630", "og:url": "https://jxnl.co/writing/2024/05/22/systematically-improving-your-rag/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [], "links": [ { "href": "https://github.com/jxnl/blog/edit/main/docs/writing/posts/rag-improving-rag.md", "label": "", "type": "a" }, { "href": "https://github.com/jxnl/blog/raw/main/docs/writing/posts/rag-improving-rag.md", "label": "", "type": "a" }, { "href": "https://hamel.dev/", "label": "Hamel", "type": "a" }, { "href": "https://jxnl.co/writing/2024/11/07/what-is-retrieval-augmented-generation/", "label": "what RAG is", "type": "a" }, { "href": "https://jxnl.co/writing/2023/09/17/rag-is-more-than-embeddings/", "label": "RAG is More Than Just Embeddings", "type": "a" }, { "href": "https://jxnl.co/writing/2024/01/07/inverted-thinking-rag/", "label": "How to Build a Terrible RAG System", "type": "a" }, { "href": "https://jxnl.co/writing/2024/02/28/levels-of-complexity-rag-applications/", "label": "Levels of RAG Complexity", "type": "a" }, { "href": "https://jxnl.co/writing/2024/05/11/low-hanging-fruit-for-rag-search/", "label": "Low Hanging Fruit in RAG", "type": "a" }, { "href": "https://jxnl.co/writing/2024/06/05/predictions-for-the-future-of-rag/", "label": "Predictions for the Future of RAG", "type": "a" }, { "href": "https://jxnl.co/writing/2024/11/04/how-to-improve-rag-applications-6-proven-strategies/", "label": "six tips for improving RAG", "type": "a" }, { "href": "https://jxnl.co/writing/2025/06/11/rag-anti-patterns-with-skylar-payne/", "label": "RAG anti-patterns", "type": "a" }, { "href": "https://jxnl.co/writing/2023/11/02/ai-engineer-keynote-pydantic-is-all-you-need/", "label": "AI Engineer Summit", "type": "a" }, { "href": "https://jxnl.co/writing/2024/08/19/rag-flywheel/", "label": "RAG flywheel strategy", "type": "a" }, { "href": "https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/", "label": "the only 6 RAG evaluations you need", "type": "a" }, { "href": "https://jxnl.co/writing/2025/06/09/how-to-invest-in-ai-w-mcps-and-data-analytics/", "label": "RAG enterprise process", "type": "a" }, { "href": "https://jxnl.co/writing/2024/11/18/decomposing-rag-systems-to-identify-bottlenecks/", "label": "RAG decomposition", "type": "a" }, { "href": "https://jxnl.co/writing/2024/06/29/art-of-looking-at-rag-data/", "label": "topics and capabilities", "type": "a" }, { "href": "https://jxnl.co/writing/2024/08/21/trade-off-tool-selection/", "label": "trade-offs in tool selection", "type": "a" }, { "href": "https://dub.link/6wk-rag-email", "label": "Free 6-Week RAG Email Course", "type": "a" }, { "href": "https://jxnl.co/writing/2025/01/24/systematically-improving-rag-applications/", "label": "Full Course Breakdown", "type": "a" }, { "href": "https://jxnl.co/writing/2024/10/26/faq-on-improving-rag-applications/", "label": "RAG FAQ", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:54Z", "source_url": "https://jxnl.co/writing/2024/05/22/systematically-improving-your-rag/", "papers": [], "actual_video_files": [], "body_chars": 15623, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/338cb3bfedef/pages/source.html", "local_html": "attachments/338cb3bfedef/pages/article.html", "body_base_path": "attachments/338cb3bfedef/pages/", "body_markdown_path": "attachments/338cb3bfedef/pages/article.md" }