{ "source_file": "0930", "source_platform": "个人博客", "author": "Hamel Husain / Parlance Labs", "title_listed": "Selecting The Right AI Evals Tool", "keywords": "评估工具,选型,evals", "domain": "大模型算法", "content_type": "行业分析", "is_paid_flag": "否", "url": "https://hamel.dev/blog/posts/eval-tools/", "publish_date": "2025-10", "corpus_stage": "推理部署", "notes": "AI评估工具选型指南", "id": "daa87ecbe328", "collected_at": "2026-10-02T14:28:58", "title": "Selecting The Right AI Evals Tool – Hamel’s Blog", "body_markdown": "Selecting The Right AI Evals Tool\nAI\nEvals\nAn example of how to assess AI eval tools like Langsmith, Braintrust, and Arize Phoenix.\nAuthor\nHamel Husain\nPublished\nOctober 1, 2025\nOver the past year, I’ve focused heavily on\nAI Evals\n, both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons. First, people focus too much on tools instead of the process, thinking the tool will be an off-the-shelf solution when it rarely is. Second, the tools change so quickly that comparisons become outdated immediately.\nHaving used many of the popular eval tools, I can genuinely say that no single one is superior in every dimension. The “best” tool depends on your team’s skillset, technical stack, and maturity.\nInstead of a feature-by-feature comparison, I think it’s more valuable to show you\nhow\na panel of data scientists skilled in evals assesses these tools. As part of my AI Evals\ncourse\n, we had three of the most dominant vendors—Langsmith, Braintrust, and Arize Phoenix complete the same homework\nassignment\n. This gave us a unique opportunity to see how they tackle the exact same challenge.\nWe recorded the entire process and live commentary, which is available below. We think this might be helpful in learning about the kinds of things you should consider when selecting a tool for your team.\nThanks to\nShreya Shankar\nand\nBryan Bischof\nfor serving as the panelists (alongside me).\nLangsmith\nWith\nHarrison Chase\n, CEO of LangChain.\nBraintrust\nWith\nWayde Gilliam\n, former developer relations at Braintrust.\nArize Phoenix\nWith\nSallyAnn DeLucia\n, Technical AI Product Leader at Arize.\nCriteria for Assessing AI Evals Tools\nHere are themes that consistently surfaced during our review.\n1. Workflow and Developer Experience\nReducing friction is more important than any single feature. Concretely, you should be mindful of the time it takes to go from observing a failure to iterating on a solution. For example, we appreciated the ability to go from viewing a single trace to experimenting with that same trace in a playground. For some teams with data-science backgrounds, a notebook-centric workflow is ideal as it provides transparency and control. This happens to be my preferred workflow as well.\nWhen considering a notebook-centric workflow, its important to pay attention to the ergonmics of the sdk. This often boils down to the quality of the documentation and integration with existing data tools.\n2. Human-in-the-Loop Support\nThe best tools don’t try to automate away the human; they empower them. Since error analysis is the highest ROI activity in AI engineering, a tool’s ability to support efficient human review is paramount. Prioritize tools with first-class support for manual annotation and error analysis. As of this writing, one thing that is missing from many tools is\naxial coding\n.\n3. Transparency and Control vs. “Magic”\nBe deeply skeptical of features that promise full automation without human validation, as these can create a powerful and dangerous illusion of confidence. For example, be wary of features where an AI agent both creates an evaluation rubric and then immediately scores the outputs. This “stacking of abstractions” often hides flaws behind a high score. Favor tools that give you control and visibility.\n4. Ecosystem Integration vs. Walled Gardens\nAn eval tool should fit your stack, not force you to fit its stack. Assess how well a tool integrates with your existing technologies. Also, beware of proprietary DSLs as they can add friction. Finally, the ability to export data into common formats for analysis in a variety of environments is a must-have.\nConclusion\nThe right choice of tool depends on your team’s workflow, skillset, and specific needs. I hope seeing how our panel approached this evaluation provides a better framework for making your own decision.\nAs for me personally, I tend to use these tools as a backend data store and use Jupyter notebooks as well as my own\ncustom built\nannotation interfaces for most of my needs.\nCaution\nAppendix: Vendor Snapshots (As of Mid-2025)\nYou should take these notes with a grain of salt. I recommend watch the videos above to get a sense of how we applied these criteria and where you might differ according to your neeeds.\nLangsmith Evaluation Notes\nOverall Sentiment\nThe overall workflow is intuitive, especially for those new to formal evaluation processes. The UI guides you through creating datasets, running experiments, and annotating results.\nPositive Feedback / What We Liked\nSeamless Workflow from Trace to Playground:\nThe transition from inspecting a trace to experimenting with it in the playground is very smooth.\nAI-Assisted Prompt Improvement:\nThe “Prompt Canvas” feature is a powerful tool for prompt engineering.\nDataset Creation and Management:\nYou can easily create datasets by uploading files, and the schema detection helps structure the data correctly.\nExperimentation and Evaluation:\nThe “Annotation Queue” is a dedicated interface for human review and labeling of traces, which is more efficient than using spreadsheets.\nCritiques and Areas for Improvement\nLimited Side-by-Side Comparison:\nThe UI doesn’t make it easy to see side-by-side comparisons of different prompt versions and their outputs.\nUI/UX Concerns:\nThe UI can feel a bit cluttered, with a lot of options and information presented at once.\nPotential for Over-Automation:\nFeatures like AI-generated examples, while convenient, can lead to homogenous data.\nBraintrust Evaluation Notes\nOverall Sentiment\nThe panel had a generally positive view of Braintrust, highlighting its clean UI and structured approach to evaluations. The tool’s emphasis on human-in-the-loop workflows was a significant strength.\nPositive Feedback / What We Liked\nFocus on a Structured Evals Process:\nThe demonstration emphasized a solid, methodical approach, starting by involving subject-matter experts to create an initial dataset.\nClean and Intuitive User Interface (UI):\nThe panel found the UI to be clean and easier to navigate than other tools, with a particularly readable trace viewing screen.\nStrong Support for Human-in-the-Loop Workflows:\nThe platform has dedicated UIs designed for human review and annotation, which is critical for creating high-quality datasets and performing error analysis.\nThe “Money Table”:\nAfter annotating traces with failure modes, the final dataset view is an actionable output that allows teams to quickly sort, filter, and quantify the most common failure modes.\nCritiques and Areas for Improvement\nThe “Loop” AI Scorer:\nThe most significant concern was the “Loop” feature, an AI agent that creates an evaluation rubric and then immediately scores the outputs, which could lead to a false sense of security.\nReliance on a Proprietary Query Language (BTQL):\nThe panel viewed the use of “BTQL” with mild skepticism, stating a preference for exporting data to a Jupyter notebook.\nClunky Data Workflows:\nThe process for generating and refining synthetic data seemed inefficient, requiring downloading and re-uploading data between steps.\nArize Phoenix Evaluation Notes\nOverall Sentiment\nThe panel had a generally positive view of Phoenix, with one panelist calling it one of his “favorite open source eval tools.” The tool is positioned as a developer-first, notebook-centric platform.\nPositive Feedback / What We Liked\nNotebook-Centric Workflow:\nThe entire evaluation process was driven from a Jupyter notebook, giving the developer transparency and control. The ability to export annotated data back into a Pandas DataFrame was a powerful feature.\nUI & Developer Experience:\nThe prompt management UI was praised for being clear and easy to understand. The tight integration between traces and the “Playground” was also noted as a smooth workflow.\nOpen Source & Local-First Approach:\nPhoenix can be run entirely locally, providing a sense of control and transparency. As an open-source tool, it was noted for being “hackable.”\nCritiques and Areas for Improvement\nUI Readability:\nThe text in the output panes was difficult to read during the demonstration, with a possible lack of markdown rendering for model outputs.\nMetrics and Visualization:\nThe tool displays point statistics for each run, but the panel found this of limited use and expressed a desire for aggregate visualizations like histograms to identify outliers.\nPrompt Management and Testing:\nThe prompt editor treats the system prompt as one large, monolithic block of text. A more component-based approach where individual instructions could be toggled on and off (“ablated”) would be preferable for systematic testing.", "metadata": { "title": "Selecting The Right AI Evals Tool – Hamel’s Blog", "description": "An example of how to assess AI eval tools like Langsmith, Braintrust, and Arize Phoenix.", "keywords": null, "author": "Hamel Husain", "og:type": "article", "og:title": "Selecting The Right AI Evals Tool – Hamel’s Blog", "og:description": "An example of how to assess AI eval tools like Langsmith, Braintrust, and Arize Phoenix.", "og:image": "https://hamel.dev/blog/posts/eval-tools/cover-img-2.png", "og:site_name": "Hamel’s Blog", "og:locale": "en_US", "og:image:height": "1024", "og:image:width": "1536", "twitter:title": "Selecting The Right AI Evals Tool – Hamel’s Blog", "twitter:description": "An example of how to assess AI eval tools like Langsmith, Braintrust, and Arize Phoenix.", "twitter:image": "https://hamel.dev/blog/posts/eval-tools/cover-img-2.png", "twitter:creator": "@HamelHusain", "twitter:site": "@HamelHusain", "twitter:card": "summary_large_image", "twitter:image-height": "1024", "twitter:image-width": "1536" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=y0vm_fjkejo", "src": "https://www.youtube.com/embed/y0vm_fjkejo", "status": "ok", "kind": "single_video", "local_path": "attachments/daa87ecbe328/videos/repair-y0vm_fjkejo.mp4", "bytes": 266684662, "valid": true, "height": 1080, "width": 1920, "duration": "2774.341000", "reused": false }, { "url": "https://www.youtube.com/watch?v=97iykOemOn4", "src": "https://www.youtube.com/embed/97iykOemOn4", "status": "ok", "kind": "single_video", "local_path": "attachments/daa87ecbe328/videos/repair-97iykOemOn4.mp4", "bytes": 240810574, "valid": true, "height": 1080, "width": 1920, "duration": "2501.666000", "reused": false }, { "url": "https://www.youtube.com/watch?v=wcYnzHJlUR0", "src": "https://www.youtube.com/embed/wcYnzHJlUR0", "status": "ok", "kind": "single_video", "local_path": "attachments/daa87ecbe328/videos/repair-wcYnzHJlUR0.mp4", "bytes": 122654078, "valid": true, "height": 1080, "width": 1920, "duration": "1959.881000", "reused": false } ], "links": [ { "href": "https://www.googletagmanager.com/ns.html?id=GTM-PKGWQMKL", "label": "", "type": "iframe" }, { "href": "https://hamel.dev/blog/posts/evals-faq/index.html", "label": "AI Evals", "type": "a" }, { "href": "https://maven.com/parlance-labs/evals?promoCode=evals-info-url", "label": "course", "type": "a" }, { "href": "https://github.com/ai-evals-course/recipe-chatbot", "label": "assignment", "type": "a" }, { "href": "https://www.sh-reya.com/", "label": "Shreya Shankar", "type": "a" }, { "href": "https://x.com/BEBischof", "label": "Bryan Bischof", "type": "a" }, { "href": "https://x.com/hwchase17", "label": "Harrison Chase", "type": "a" }, { "href": "https://www.youtube.com/embed/y0vm_fjkejo", "label": "", "type": "iframe" }, { "href": "https://x.com/waydegilliam", "label": "Wayde Gilliam", "type": "a" }, { "href": "https://www.youtube.com/embed/97iykOemOn4", "label": "", "type": "iframe" }, { "href": "https://www.linkedin.com/in/sallyann-delucia-59a381172/", "label": "SallyAnn DeLucia", "type": "a" }, { "href": "https://www.youtube.com/embed/wcYnzHJlUR0", "label": "", "type": "iframe" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:43Z", "source_url": "https://hamel.dev/blog/posts/eval-tools/", "papers": [], "actual_video_files": [], "body_chars": 8685, "expected_body_images": 0, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/daa87ecbe328/pages/source.html", "local_html": "attachments/daa87ecbe328/pages/article.html", "body_base_path": "attachments/daa87ecbe328/pages/", "body_markdown_path": "attachments/daa87ecbe328/pages/article.md" }