{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Jason Liu",
  "title_listed": "Grep Beats Embeddings",
  "keywords": "grep,嵌入,编码Agent",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://jxnl.co/writing/2025/09/11/why-grep-beat-embeddings-in-our-swe-bench-agent-lessons-from-augment/",
  "publish_date": "2025-09",
  "corpus_stage": "推理部署",
  "notes": "grep为何胜过嵌入（SWE-bench教训）",
  "id": "863d5c4d3a23",
  "collected_at": "2026-10-02T14:29:28",
  "title": "Why Grep Beat Embeddings in Our SWE-Bench Agent (Lessons from Augment) - Jason Liu",
  "body_markdown": "Why Grep Beat Embeddings in Our SWE-Bench Agent (Lessons from Augment)\n¶\nI hosted Colin Flaherty, previously a founding engineer at Augment and co-author of Meta's Cicero AI, to discuss autonomous coding agents and retrieval systems. This session explores how agentic approaches are transforming traditional RAG systems, what we can learn from state-of-the-art coding agents, and how these insights might apply to other domains.\n▶️ See How Grep Beat Complex Embeddings\nDo agents make traditional RAG obsolete?\n¶\nColin shared his experience building an agent for SWE-Bench Verified, a canonical AI coding evaluation where agents implement code changes based on problem descriptions. His team's agent reached the top of the leaderboard with a surprising discovery: embedding-based retrieval wasn't the bottleneck they expected.\n\"We explored adding various embedding-based retrieval tools, but found that for SweeBench tasks this was not the bottleneck - grep and find were sufficient,\" Colin explained. This initially surprised him, as he expected embedding models to be significantly more powerful.\nThe evolution of code retrieval complexity has followed a clear progression:\n2023 (Completions): Simple retrieval with low latency requirements\n2024 (Chatbots): More complex retrieval across multiple files\n2025 (Agents): Highly complex retrieval across many parts of codebases\nWhen examining how the agent solved problems, Colin observed it would use simple tools like grep and find persistently, trying different approaches until it found what it needed. The agent's persistence effectively compensated for less sophisticated tools.\nKey Takeaway:\nAgents don't necessarily make traditional RAG obsolete, but they change how we should think about retrieval systems. The persistence and course-correction capabilities of agents can sometimes overcome limitations in the underlying retrieval tools.\nBenefits of agentic retrieval with simple tools\nAgentic retrieval with grep and find offers several advantages:\nIterative retrieval becomes much simpler - instead of complex multi-step embedding processes, agents can simply run multiple searches and refine as they go\nToken budget management is straightforward - when the agent hits token limits, it can simply truncate old tool calls and rerun them if needed\nImplementation is low-effort - no need to maintain vector databases, syncing mechanisms, or other infrastructure\nCourse correction happens naturally - if one search approach fails, the agent tries another\nHowever, these simple approaches have clear limitations:\nThey don't scale well to large codebases\nThey struggle with unstructured natural language content\nThey're relatively slow and expensive compared to optimized embedding lookups\nThe best approach might be combining both worlds - an agentic loop with access to high-quality embedding models as tools.\nHow to architect retrieval systems for different needs\nWhen deciding between traditional RAG, agent+grep/find, or agent+embeddings, Colin recommends considering several factors:\nQuality: How good is the final output?\nLatency: How quickly does the system respond?\nCost: What are the computational expenses?\nReliability: Does the system correct itself when it fails?\nScalability: How well does it handle large indices?\nMaintenance effort: How much engineering work is required?\nTraditional RAG offers decent quality with excellent speed, low cost, and high scalability, but lacks course correction. Agent+grep provides excellent quality and reliability but struggles with speed, cost, and scale. Agent+embeddings combines the best of both but remains slow and expensive.\nKey Takeaway:\nDon't throw away your existing retrieval systems - instead, expose them as tools to agents. This gives you the benefits of both approaches while allowing you to optimize based on your specific constraints. For implementation guidance, see the\nContext Engineering approach to tool response design\n, which shows how to structure tool outputs to give agents better peripheral vision of the information landscape.\nEvaluating agentic retrieval systems\nColin emphasized a \"vibe-first\" approach to evaluation:\n\"Start with 5-10 examples and do end-to-end vibe checks before moving to quantitative evaluation. With natural language systems, you can learn so much just from looking at a few examples.\"\nHe noted that improving embedding models doesn't necessarily improve end-to-end performance because agents are persistent - they'll eventually find what they need even with suboptimal tools. This makes traditional embedding evaluation metrics less useful for agentic systems.\nFor those starting from scratch, Colin recommends:\nBuild the simplest possible retrieval tool\nPut an agent loop on top\nIterate based on what causes the most pain for users\nOnly move to quantitative evaluation once you've addressed obvious issues\nI found it refreshing that Colin focused on specific examples of queries rather than abstract discussions of model architectures. As he put it, \"Being a researcher is actually very similar to being a product person - you're working backwards from use cases and examples.\"\nWhen to use embedding models vs. simple search tools\nWhile grep and find worked well for SWE-Bench's relatively small codebases, Colin identified several scenarios where embedding models become essential:\nSearching large codebases\nRetrieving from unstructured content like Slack messages or documentation\nSearching across third-party code that models haven't memorized\nRetrieving from non-text media like video recordings of user sessions\n\"If I was a human working on this use case, and I was a really persistent human that never got tired, would having this other search tool help me? If the answer is yes, then it's probably going to be useful for the agent.\"\nColin noted that SWE-Bench is somewhat artificial - its repositories are smaller than real-world codebases, and 90% of its problems take less than an hour for a good engineer to solve. In more complex environments, embedding models become increasingly valuable.\nImproving agentic retrieval systems\nTo enhance agentic retrieval, Colin recommends:\nAdding re-rankers to embedding tools to improve precision and reduce token usage\nTraining specialized embedding models for different tasks (e.g., one for code, another for Slack messages)\nPrompt-tuning tool schemas to guide agents toward efficient usage patterns\nCreating hierarchical retrieval systems that summarize files and directories\nLeveraging language server protocols as additional tools\nOne particularly effective technique is asynchronous pre-processing: \"I've taken songs and used an LLM to create a dossier about each one. This simple pre-processing step took a totally non-working search system and turned it into something that works really well.\"\nWhy aren't more people training great embedding models?\n¶\nWhen asked what question people aren't asking enough, Colin highlighted the lack of expertise in training embedding models: \"Very few people understand how to build and train good retrieval systems. It just confuses me why no one knows how to fine-tune really good embedding models.\"\nHe attributed this partly to the specialized nature of the skill and partly to data availability. For code, there's abundant data on GitHub, but most domains lack comparable resources. Additionally, the most talented engineers often prefer working on LLMs rather than embedding models.\nKey Takeaway:\nAs agents become more capable, the quality of their tools becomes increasingly important. Even though agents can compensate for suboptimal tools through persistence, providing them with better retrieval mechanisms significantly improves their efficiency and capabilities.\nFinal thoughts on the future of retrieval\nColin believes we're entering an era where the boundaries between traditional RAG and agentic systems are blurring. The ideal approach combines the strengths of both: the speed and efficiency of well-tuned embedding models with the persistence and course-correction of agents.\nAs these systems evolve, we'll likely see more specialized tools emerging for different retrieval contexts, along with more sophisticated pre-processing techniques that make retrieval more effective. The key is focusing on the specific problems you're trying to solve rather than getting caught up in architectural debates.\n\"Agents are getting radically smarter, but even Einstein preferred writing on paper instead of a stone tablet,\" Colin noted. \"Yes, these agents are persistent, but you should give them whatever you can to improve the odds that they find what they're looking for.\"\nFAQs\nWhat is agentic retrieval and how does it differ from traditional RAG?\n¶\nAgentic retrieval is an approach where AI agents use tools like grep, find, or embedding models to search through code and other content. Unlike traditional RAG (Retrieval-Augmented Generation), which typically uses embedding databases and vector searches, agentic retrieval gives the agent direct control over the search process. This allows the agent to be persistent, try multiple search strategies, and course-correct when initial attempts fail. Traditional RAG is more rigid but can be faster and more efficient for certain use cases.\nDo agents make traditional RAG obsolete?\n¶\nNo, agents don't make traditional RAG obsolete—they complement it. The best approach is often to build agentic retrieval on top of your existing retrieval system by exposing your embedding models and search capabilities as tools that an agent can use. This combines the strengths of both approaches: the persistence and flexibility of agents with the efficiency and scalability of well-tuned embedding models.\nWhat are the benefits of using grep and find tools with agents?\n¶\nUsing simple tools like grep and find with agents offers several advantages:\nIterative retrieval becomes much easier as agents can refine searches based on previous results\nToken budget management is simpler since old tool calls can be truncated and rerun if needed\nThe system is easier to build and maintain without complex vector database dependencies\nAgents can course-correct when searches don't yield useful results by trying different approaches\nWhat are the limitations of using grep and find for retrieval?\n¶\nWhile grep and find work well for certain scenarios, they have significant limitations:\nThey don't scale well to very large codebases (millions of files)\nThey're ineffective for searching through unstructured natural language content\nThey work best with highly structured content like code that contains distinctive keywords\nThey can be slower than optimized embedding-based searches for large datasets\nWhat's the ideal approach to retrieval for coding agents?\n¶\nThe best approach is often a hybrid system that combines:\nAn agentic loop that gives the agent control over the search process\nAccess to multiple search tools including grep, find, and embedding-based search\nThe ability to choose the most appropriate tool based on the specific search task\nCourse correction capabilities when initial searches don't yield useful results\nHow should I evaluate agentic retrieval systems?\n¶\nStart with a qualitative \"vibe check\" using 5-10 examples to understand how the system performs. Observe the agent's behavior, identify patterns in successes and failures, and develop an intuition for where improvements are needed. Only after this initial assessment should you move to quantitative end-to-end evaluations or specific evaluations of individual components like embedding tools. Remember that improving a single component (like an embedding model) may not necessarily improve the end-to-end performance if the agent is already persistent enough to overcome limitations.\nI already built a retrieval system with custom-trained embedding models. Should I replace it with agentic retrieval?\n¶\nNo, don't replace it—enhance it. Build agentic retrieval on top of your existing system by exposing your embedding models and search capabilities as tools that an agent can use. This gives you the best of both worlds: the quality and efficiency of your custom embeddings plus the persistence and flexibility of an agent.\nHow can I improve my agentic retrieval system?\n¶\nFocus on building better tools for your agent:\nAdd re-rankers to your embedding tools to improve precision and reduce token usage\nTrain different embedding models for different specific tasks\nPrompt-tune your tool schemas to help the agent use them effectively\nConsider hierarchical retrieval approaches like creating summaries of files or directories\nAdd specialized tools for specific retrieval tasks (like searching commit history)\nHow do memories work with agentic retrieval systems?\n¶\nMemories in agentic systems can be implemented by adding tools that save and read memories. These memories can serve as a semantic cache that speeds up future searches by storing information about the codebase structure, relevant interfaces, or other insights gained during previous searches. This can significantly improve performance on similar tasks in the future.\nWhy did embedding models not improve performance on SWE-Bench?\n¶\nFor the SWE-Bench coding evaluation, embedding models didn't significantly improve performance because:\nThe repositories were relatively small, making grep and find sufficient\nThe code was highly structured with distinctive keywords that made text-based search effective\nThe agent's persistence compensated for less sophisticated search tools\nThe tasks were relatively simple, typically solvable by a good engineer in under an hour\nThis doesn't mean embedding models aren't valuable—they become essential for larger codebases, less structured content, or more complex retrieval tasks.\n¶",
  "metadata": {
    "title": "Why Grep Beat Embeddings in Our SWE-Bench Agent (Lessons from Augment) - Jason Liu",
    "description": "Insights from Colin Flaherty on building autonomous coding agents and how agentic approaches reshape retrieval-augmented generation systems.",
    "keywords": null,
    "author": "Jason Liu",
    "og:type": "website",
    "og:title": "Why Grep Beat Embeddings in Our SWE-Bench Agent (Lessons from Augment) - Jason Liu",
    "og:description": "Insights from Colin Flaherty on building autonomous coding agents and how agentic approaches reshape retrieval-augmented generation systems.",
    "og:image": "https://jxnl.co/assets/images/social/writing/2025/09/11/why-grep-beat-embeddings-in-our-swe-bench-agent-lessons-from-augment.png",
    "og:image:type": "image/png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:url": "https://jxnl.co/writing/2025/09/11/why-grep-beat-embeddings-in-our-swe-bench-agent-lessons-from-augment/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/jxnl/blog/edit/main/docs/writing/posts/talks/colin-rag-agents.md",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/jxnl/blog/raw/main/docs/writing/posts/talks/colin-rag-agents.md",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://maven.com/p/5f4d74",
      "label": "▶️ See How Grep Beat Complex Embeddings",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2025/08/27/facets-context-engineering/",
      "label": "Context Engineering approach to tool response design",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:57Z",
  "source_url": "https://jxnl.co/writing/2025/09/11/why-grep-beat-embeddings-in-our-swe-bench-agent-lessons-from-augment/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 13745,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/863d5c4d3a23/pages/source.html",
  "local_html": "attachments/863d5c4d3a23/pages/article.html",
  "body_base_path": "attachments/863d5c4d3a23/pages/",
  "body_markdown_path": "attachments/863d5c4d3a23/pages/article.md"
}