{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Hamel Husain / Parlance Labs",
  "title_listed": "AI Evals: Everything You Need to Know",
  "keywords": "评估,evals,FAQ",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://hamel.dev/blog/posts/evals-faq/",
  "publish_date": "2026-09",
  "corpus_stage": "推理部署",
  "notes": "AI评估完整FAQ",
  "id": "5bf201eed3d0",
  "collected_at": "2026-10-02T14:28:44",
  "title": "AI Evals: Everything You Need to Know – Hamel’s Blog",
  "body_markdown": "AI Evals: Everything You Need to Know\nLLMs\nevals\nA guide to AI and LLM evals for engineers and product managers. Learn how to test AI systems, analyze failures, and improve retrieval, agents, and AI products.\nAuthors\nHamel Husain\nShreya Shankar\nPublished\nSeptember 18, 2026\nModified\nSeptember 21, 2026\nThis document curates the most common questions Shreya and I received while\nteaching\n5,000+ engineers and PMs AI Evals.\nWarning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment.\nHow to use this FAQ\nBrowse the questions that interest you, or choose a guide below for a curated reading path through the FAQs and related articles.\nWhere are you?\nThis sounds like me\nI’m new to evals\nI’ve heard the term, but I’m not sure what evals involve or whether I need them.\nI don’t know what to test\nI’m building an AI product, but I haven’t figured out which failures to measure or what good performance looks like.\nI don’t trust my eval scores\nWe have evals, but the scores don’t match our judgment of the outputs, or tests pass while users still encounter problems.\nMy product feels too hard to evaluate\nOur outputs are subjective, long, or involve many steps. Even a knowledgeable person has trouble deciding whether they’re right.\nEvals take too much time or money\nWe’re spending too much effort reviewing outputs, maintaining tests, or running evaluators.\nAll questions\nBrowse all questions by section.\nGetting Started & Fundamentals\nQ: What are AI Evals?\nQ: What is a trace?\nQ: What’s a minimum viable evaluation setup?\nQ: How much of my development budget should I allocate to evals?\nQ: Will today’s evaluation methods still be relevant in 5-10 years given how fast AI is changing?\nQ: How do I make the case for investing in evaluations to my team?\nError Analysis & Data Collection\nQ: Why is \"error analysis\" so important in AI evals, and how is it performed?\nQ: Do I need a reference answer or rubric before annotating data?\nQ: Should I record problems that aren’t the model’s fault?\nQ: How many examples do I need for an eval?\nQ: How do I surface problematic traces for review beyond user feedback?\nQ: How often should I re-run error analysis on my production system?\nQ: What should I do when my \"gold\" eval dataset becomes stale?\nQ: What is the best approach for generating synthetic data?\nQ: Are there scenarios where synthetic data may not be reliable?\nQ: How can I do evals when traces contain sensitive data?\nQ: How do I approach evaluation when my system handles diverse user queries?\nQ: How can I efficiently sample production traces for review?\nEvaluation Design & Methodology\nQ: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?\nQ: How do I combine my evals into a single metric?\nQ: Should I practice eval-driven development?\nQ: Should I build automated evaluators for every failure mode I find?\nQ: What model or LLM should I use to build automated evals?\nQ: Can I use Jev for evals?\nQ: How do I know if I can trust my automated eval?\nQ: What should I do when I can’t get my LLM judge to agree with human reviewers?\nQ: Should I use \"ready-to-use\" evaluation metrics?\nQ: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?\nQ: Can I use the same model for both the main task and evaluation?\nQ: How much context should I give a LLM judge?\nQ: How do we evaluate a model’s ability to express uncertainty or \"know what it doesn’t know\"?\nHuman Annotation & Process\nQ: How many people should annotate my LLM outputs?\nQ: How can I make AI outputs easier for people to evaluate?\nQ: Should product managers and engineers collaborate on error analysis? How?\nQ: Can I help with evals if I’m not a domain expert?\nQ: How do I evaluate outputs in a language I don’t speak?\nQ: Should I outsource annotation & labeling to a third party?\nQ: How do you review a trace that is really large?\nQ: What parts of evals can be automated with LLMs?\nQ: Should I stop writing prompts manually in favor of automated tools?\nTools & Infrastructure\nQ: Should I build a custom annotation tool or use something off-the-shelf?\nQ: What makes a good custom interface for reviewing LLM outputs?\nQ: What gaps in eval tooling should I be prepared to fill myself?\nQ: What should an internal eval platform standardize across teams?\nQ: What’s your favorite eval vendor?\nQ: How should I version and manage prompts?\nQ: What should go in the system prompt vs. the user prompt?\nProduction & Deployment\nQ: How are evaluations used differently in CI/CD vs. monitoring production?\nQ: How often should I run my evals?\nQ: What’s the difference between guardrails & evaluators?\nQ: Can my evaluators also be used to automatically\nfix\nor\ncorrect\noutputs in production?\nQ: How much time should I spend on model selection?\nDomain-Specific Applications\nQ: Is RAG dead?\nQ: How should I evaluate a coding agent?\nQ: How should I approach evaluating my RAG system?\nQ: How do I choose the right chunk size for my document processing tasks?\nQ: How do I debug multi-turn conversation traces?\nQ: How do I evaluate sessions with human handoffs?\nQ: How do I evaluate complex multi-step workflows?\nQ: How do I evaluate agentic workflows?\nGetting Started & Fundamentals\nQ: What are AI Evals?\nAI evals are tests that tell you whether an AI system is doing what you want. They give your team feedback when the product drifts from user needs or business goals. The failures they catch also become data you can use to improve the system.\nMore formally, evaluation is the systematic measurement of quality. Each eval checks one behavior on relevant examples and returns a score or structured review. Most AI products need several evals because they can fail in different ways.\nWhen you hear the word “evals,” it usually refers to one of two things: model benchmarks or product evals.\nModel benchmarks\nModel benchmarks compare general-purpose models on shared tasks. Model providers publish these benchmark results when they release new models. Common examples include\nGPQA Diamond\nfor graduate-level science reasoning,\nTerminal-Bench\nfor agents doing complex work in command-line environments, and\nMMLU\nfor knowledge and reasoning across a wide range of subjects. These scores can help you choose a promising model as a starting point. To assess quality on your own tasks you need product evals, which we discuss next.\nProduct evals\nProduct evals measure whether your specific AI product does what you want it to do. They turn your judgment about what a good product experience looks like into metrics you can track. Product evals encompass all components of your product, including the model, prompts, retrieval, tools, and application code. This flavor of evals are focused on capturing failures that matter to users and the business.\nConsider an order-cancellation agent. Its product evals might check whether it selected the correct order and waited for the cancellation tool to succeed before telling the user the order was canceled. A high score on GPQA Diamond or Terminal-Bench gives you little information on this, because those benchmarks don’t have access to your systems.\nThere are several mechanisms you can use to implement product evals, including code assertions, human review, LLM judges, and online experiments. The right method depends on the failure being measured, and is discussed in greater detail\nin this series\n.\nIn the rest of the\nAI Evals FAQ\n, we focus on product evals. It starts with\nanalyzing traces\nto discover real failure modes. We then turn important failures into\ntargeted evals\nand use the results to guide changes. Finally,\nrerunning the evals\ntells us whether the system improved.\nWhere to start with evals\nIf you are completely new to product-specific evals, see these posts:\nGuide\nWhat it covers\nPart 1\n:\nYour AI Product Needs Evals\nBuild a domain-specific evaluation system with scoped tests, trace review, human evaluation, and experiments.\nPart 2\n:\nUsing LLM-as-a-Judge For Evaluation: A Complete Guide\nCapture a domain expert’s judgment, automate it with an LLM judge, and validate the judge against human labels.\nPart 3\n:\nA Field Guide to Rapidly Improving AI Products\nUse error analysis, realistic data, and trustworthy evals to run a sustained product-improvement loop.\n↗ Focus view\nQ: What is a trace?\nA trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response. It includes every step across all agents, tools, and system components in a session: multiple user messages, assistant responses, retrieved documents, and intermediate tool interactions.\nNote on terminology:\nDifferent observability vendors use varying definitions of traces and spans.\nAlex Strick van Linschoten’s analysis\nhighlights these differences (screenshot below):\nVendor differences in trace definitions as of 2025-07-02\n↗ Focus view\nQ: What’s a minimum viable evaluation setup?\nStart with\nerror analysis\n, not infrastructure. Spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes. Use one\ndomain expert\nwho understands your users as your quality decision maker (a “\nbenevolent dictator\n”).\nUse a notebook\nto review traces and analyze data, or build your own\ncustom annotation interface\nwith an AI coding assistant like Claude or Codex. Either way, you can write arbitrary code, visualize data, and iterate quickly. The\nvideo\nbelow shows a simple annotation interface built inside a notebook.\nWatch “Build Your Own Eval Tools With Notebooks!” on YouTube\n↗ Focus view\nQ: How much of my development budget should I allocate to evals?\nIt’s important to recognize that evaluation is part of the development process rather than a distinct line item, similar to how debugging is part of software development.\nYou should always be doing\nerror analysis\n. When you discover issues through error analysis, many will be straightforward bugs you’ll fix immediately. These fixes don’t require separate evaluation infrastructure as they’re just part of development.\nThe decision to build automated evaluators comes down to\ncost-benefit analysis\n. If you can catch an error with a simple assertion or regex check, the cost is minimal and probably worth it. But if you need to align an LLM-as-judge evaluator, consider whether the failure mode warrants that investment.\nIn the projects we’ve worked on,\nwe’ve spent 60-80% of our development time on error analysis and evaluation\n. Expect most of your effort to go toward understanding failures (i.e. looking at data) rather than building automated checks.\nBe\nwary of optimizing for high eval pass rates\n. If you’re passing 100% of your evals, you’re likely not challenging your system enough. A 70% pass rate might indicate a more meaningful evaluation that’s actually stress-testing your application. Focus on evals that help you catch real issues, not ones that make your metrics look good.\n↗ Focus view\nQ: Will today’s evaluation methods still be relevant in 5-10 years given how fast AI is changing?\nYes. Even with perfect models, you still need to verify they’re solving the right problem. The need for systematic\nerror analysis\n, domain-specific testing, and monitoring will still be important.\nToday’s prompt engineering tricks might become obsolete, but you’ll still need to understand failure modes. Additionally, a LLM cannot read your mind, and\nresearch shows\nthat people need to observe the LLM’s behavior in order to properly externalize their requirements.\nFor deeper perspective on this debate, see these two viewpoints:\n“The model is the product”\nversus\n“The model is NOT the product”\n.\n↗ Focus view\nQ: How do I make the case for investing in evaluations to my team?\nDon’t try to sell your team on “evals”. Instead, show them what you find when you look at the data.\nStart by doing the\nerror analysis\nyourself. Look at 50 to 100 real user conversations and find the most common ways the product is failing. Use these findings to tell a story with data.\nPresent your team with:\nA list of the top failure modes you discovered.\nMetrics showing how often high-impact errors are happening.\nSurprising ways that users are interacting with the product.\nReports on the bugs you found and fixed, framed as “prevented production issues”.\nFrame evaluation as part of development, not optional testing. Keep a running log of the errors you catch, what you learned, the fix, and the likely impact you avoided. Share it weekly or monthly. A concrete report such as “we caught 47 issues before users saw them” makes the value easier to see than an abstract pitch about evals.\nThis approach builds trust. Don’t just show dashboards and metrics; tell the story of what you’re finding in the data. By narrating your findings, you teach the team what you’re learning, providing immediate value. When you fix an issue, show how the error rate for that specific problem went down. Soon, your team will see the progress and ask how you’re doing it. Let results instead of methods lead the conversation.\nThis is similar to classic machine learning projects, where outcomes are speculative and progress is bounded by\niterating on experiments\n. In this situation, it’s important that you share the learnings from each experiment to show progress and encourage investment.\n↗ Focus view\nError Analysis & Data Collection\nQ: Why is \"error analysis\" so important in AI evals, and how is it performed?\nError analysis is\nthe most important activity in evals\n. Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes unique to your application and data. The process involves:\n1. Creating a Dataset\nGathering representative traces of user interactions with the LLM. If you do not have any data, you can\ngenerate synthetic data\nto get started.\n2. Open Coding\nHuman annotator(s) (ideally a\nbenevolent dictator\n) review and write open-ended notes about traces, noting any issues. This process is akin to “journaling” and is adapted from qualitative research methodologies. Start by annotating at least 30 traces yourself before reviewing suggestions from an agent. When beginning, it is recommended to focus on noting the\nfirst failure\nobserved in a trace, as upstream errors can cause downstream issues, though you can also tag all independent failures if feasible. A\ndomain expert\nshould be performing this step.\n3. Axial Coding\nCategorize the open-ended notes into a “failure taxonomy.” In other words, group similar failures into distinct categories. Axial coding is the most important step. At the end, count the number of failures in each category. You can use an LLM to help with this step.\n4. Iterative Refinement\nHave your agent cluster the data and choose a diverse initial sample. After your first 30 annotations, let it search the remaining traces for likely instances of the failures you described. Accept or reject its suggestions and keep iterating until you reach\ntheoretical saturation\n, meaning new reviews stop revealing failure modes or changing existing ones.\nA working pool of roughly 100 diverse traces is a useful guardrail for this human-agent loop. The agent can focus your attention on the most informative traces, so you no longer have to read all 100 sequentially. See\nhow many examples you need for each kind of eval\nfor the full breakdown.\nYou should frequently revisit this process. There are advanced ways to\nsample data more efficiently\n, like clustering, sorting by user feedback, and sorting by high probability failure patterns. Over time, you’ll develop a “nose” for where to look for failures in your data.\nDo not skip error analysis. It ensures that the evaluation metrics you develop are supported by real application behaviors instead of counter-productive generic metrics (which most platforms nudge you to use). For examples of how error analysis can be helpful, see\nthis video\n, or this\nblog post\n.\nHere is a visualization of the error analysis process by one of our students,\nPawel Huryn\n- including how it fits into the overall evaluation process:\n↗ Focus view\nQ: Do I need a reference answer or rubric before annotating data?\nNo. Writing a rubric before you review examples can get in the way.\nLet’s get some definitions out of the way:\nA reference answer is an example of a correct response.\nA rubric is a set of criteria for judging a response, such as whether it follows the refund policy.\nBoth can help, but treat your initial expectations as a starting point that you will revise.\nIt’s often better to wait until you’ve reviewed some examples before developing a detailed rubric. Reviewers can become so focused on checking each item that they overlook problems outside the rubric. It’s important to give reviewers room to notice things you didn’t anticipate. This change in what you consider good is called\n“criteria drift”\n.\nFor example, let’s say you have a support agent that handles refunds and it escalates refunds to a human per your policy. You might only realize that the process is frustrating for the user after reading a few interactions. Don’t underestimate the degree of criteria drift that will happen as you review examples!\nWe recommend using\nerror analysis\nto systematically review examples and decide what might belong in the rubric. This involves writing open-ended notes about what looks wrong, then group similar notes to see which problems recur. See this\nlive demo\nfor a walkthrough.\nAfter doing error analysis, you can write a better rubric informed by user and application behavior. You should\nperiodically\ndo error analysis to make sure your rubric is current.\n↗ Focus view\nQ: Should I record problems that aren’t the model’s fault?\nYes. When reviewing interactions, write down anything that makes the product less useful. This includes missing or broken features that have nothing to do with the model. Additionally, don’t focus on why the error occurred, as that should only come after you prioritize which issues to fix.\nFor example, a support agent might tell a customer that an order has shipped without providing a tracking link. Even if your AI doesn’t have the ability to fetch a tracking link, record that problem. A prerequisite to building evals is to identify and prioritize which issues to fix through\nerror analysis\n. Some of these issues may end up being engineering or design issues that\ndon’t need\nan automated evaluator, but they are still important to fix!\nLastly, we’ve found that deferring root-cause analysis and focusing on problems allows you to write higher-quality annotations while looking at more data.\n↗ Focus view\nQ: How many examples do I need for an eval?\nBuilding evals is a pipeline, and each stage needs a different amount of data. We describe these stages below:\nStage\nWhat to do\n1. Review the application\nRead traces and write down the ways your application fails. This process is called\nerror discovery\n. Start with 100 diverse traces and annotate at least the first 30 yourself.\n2. Create and validate evaluators\nChoose between two evaluator types. Use a\ncode-based eval\nwhen an objective rule can identify the failure. Include Pass and Fail examples for every condition and important edge case. Use an\nLLM judge\nwhen the failure requires human judgment. Label 100 to 200 examples for each failure mode.\n3. Build a repeatable eval set\nCollect examples that represent important workflows and confirmed failures. Run this set when you change your application. These sets often grow to 100 or more examples.\nStage 1: Review traces to find failures\nA trace is a complete record of one user session with your application. Ask a coding agent to help you sample the initial pool so it covers different users and workflows. Our\nevals plugin\ncan help with sampling and build an annotation interface for your traces.\nReview at least 30 traces yourself\nWe recommend annotating at least 30 traces with a process called\nerror discovery\nyourself before asking the agent to suggest failures. Write free-text notes about anything that seems wrong from the user’s perspective. These examples give the agent a concrete record of your judgment.\nKeep this first pass manual. If the agent starts suggesting problems too early, its guesses can bias your judgment. You may miss failures that depend on product context or your definition of a good user experience.\nAfter 30 traces, ask the agent to search the remaining pool for similar examples. Review every suggestion yourself. Accept or reject each one and correct the agent when it misunderstands your criteria.\nWhen to stop\nContinue until new traces stop revealing failure modes or changing existing ones. Qualitative researchers call this\ntheoretical saturation\n. We recommend reviewing at least 100 traces. Continue past 100 while you are still learning.\nIf you want to see the process done live, watch the\nlive walkthrough\n. The video shows how Shreya Shankar uses an agent to review traces quickly while keeping a human in charge of the failure criteria.\nThis review produces a failure taxonomy, which is a list of the specific ways your application fails. Use that taxonomy to decide which evaluators to build.\nStage 2: Create and validate evaluators\nChoose an evaluator for each important failure mode. The evaluator type determines how many labeled examples you need. Code-based evals work for objective rules. LLM judges work for failures that require human judgment.\nCode-based evals need coverage\nUse a code-based eval when a deterministic rule can identify the failure. Examples include checking whether JSON parses or whether a tool call uses the correct arguments.\nThe number of examples depends on the scenarios the check covers. At minimum, include examples that should Pass and Fail for every condition. Add important edge cases you found during error discovery. A check with one rule may need only a few examples that Pass and a few that Fail.\nLLM judges need labeled examples\nUse an LLM judge when the failure requires subjective or domain-specific judgment. Plan to label 100 to 200 examples for each failure mode. Reuse labeled traces from error discovery when they match the failure mode, then collect more until you reach that range. The labels should come from a trusted\ndomain expert\nand contain enough Pass and Fail examples to evaluate both classes.\nSplit these examples into train, dev, and test sets. Use 10 to 20 percent for train examples that may appear in the prompt. Use 40 to 45 percent for dev while refining the judge. Reserve the remaining 40 to 45 percent for one final test. When possible, include 30 to 50 Pass examples and 30 to 50 Fail examples in both the dev and test sets.\nThe\nvalidation guide\nexplains the full process. The\njudge-validation flashcard\nis a good visual reference as well.\nAfter validating the evaluators, assemble the examples you will run repeatedly during development.\nStage 3: Build the repeatable eval set\nStart with examples from error discovery that capture important failure modes. Add confirmed failures as you find them.\nA purpose-built eval set often grows to 100 or more examples. Coverage determines the final size. Each important workflow and known failure should be represented, and the set should remain cheap enough to run often. Code-based checks and LLM judges can run over the same examples. The\nCI evals\nFAQ explains how to use this set during development.\n↗ Focus view\nQ: How do I surface problematic traces for review beyond user feedback?\nWhile user feedback is a good way to narrow in on problematic traces, other methods are also useful. Here are three complementary approaches:\nStart with random sampling\nThe simplest approach is reviewing a random sample of traces. If you find few issues, escalate to stress testing: create queries that deliberately test your prompt constraints to see if the AI follows your rules.\nUse evals for initial screening\nUse existing evals to find problematic traces and potential issues. Once you’ve identified these, you can proceed with the typical evaluation process starting with\nerror analysis\n.\nLeverage efficient sampling strategies\nFor more sophisticated trace discovery,\nuse outlier detection, metric-based sorting, and stratified sampling\nto find interesting traces.\nGeneric metrics can serve as exploration signals\nto identify traces worth reviewing, even if they don’t directly measure quality.\n↗ Focus view\nQ: How often should I re-run error analysis on my production system?\nRe-run\nerror analysis\nwhen making significant changes: new features, prompt updates, model switches, or major bug fixes. A useful heuristic is to set a goal for reviewing\nat least\n100+ fresh traces each review cycle. Typical review cycles we’ve seen range from 2-4 weeks. See\nthis FAQ\non how to sample traces effectively.\nBetween major analyses, review 10-20 traces weekly, focusing on outliers: unusually long conversations, sessions with multiple retries, or traces flagged by automated monitoring. Adjust frequency based on system stability and usage growth. New systems need weekly analysis until failure patterns stabilize. Mature systems might need only monthly analysis unless usage patterns change. Always analyze after incidents, user complaint spikes, or metric drift. Scaling usage introduces new edge cases.\n↗ Focus view\nQ: What should I do when my \"gold\" eval dataset becomes stale?\nEval datasets naturally get stale as your product and users change. Use regular\nerror analysis\nto find new problems and update your examples or reference answers.\nHow often you review\ndepends on your use case and how quickly your product or usage changes.\nLike unit tests, evals can catch problems that return after a fix. However, evals often cost considerably more than unit tests to maintain and run. Therefore, you should weigh each eval’s cost against the value of its signals. If\neverything keeps passing\n, this is a sign that the eval is less useful and should be run less often or retired. You should phase out expensive evals like LLM-as-a-judge more agressively than cheaper evals.\nAs your eval set changes, its scores may no longer be directly comparable with older scores. That is ok! One purpose of evals are to provide you with challenges you can hill climb against. These challenges should change as your product evolves to help you keep improving.\nFor tracking progress with metrics over longer time horizons, it’s often better to use product metrics in addition to evals. Examples of product metrics include: churn, active users, revenue, etc.\n↗ Focus view\nQ: What is the best approach for generating synthetic data?\nA common mistake is prompting an LLM to\n\"give me test queries\"\nwithout structure, resulting in generic, repetitive outputs. A structured approach using dimensions produces far better synthetic data for testing LLM applications.\nWhen should I use synthetic data for evals?\nUse synthetic data to start error analysis before you have enough production traffic, or to test a known failure that appears rarely in real data. Define the variation you need, generate examples, run them through the full system, and review the resulting traces.\nSynthetic data cannot tell you how common a failure is in production. It can also miss details that matter in specialized domains. Compare synthetic examples with real data as soon as real data becomes available. See\nwhen synthetic data may be unreliable\nfor cases that require extra review.\nDefine important dimensions first\nStart by defining dimensions\n: categories that describe different aspects of user queries. Each dimension captures one type of variation in user behavior. For example:\nFor a recipe app, dimensions might include Dietary Restriction (\nvegan\n,\ngluten-free\n,\nnone\n), Cuisine Type (\nItalian\n,\nAsian\n,\ncomfort food\n), and Query Complexity (\nsimple request\n,\nmulti-step\n,\nedge case\n).\nFor a customer support bot, dimensions could be Issue Type (\nbilling\n,\ntechnical\n,\ngeneral\n), Customer Mood (\nfrustrated\n,\nneutral\n,\nhappy\n), and Prior Context (\nnew issue\n,\nfollow-up\n,\nresolved\n).\nStart with failure hypotheses\n. If you lack intuition about failure modes, use your application extensively or recruit friends to use it. Then choose dimensions targeting those likely failures.\nCreate tuples manually first\n: Write 20 tuples by hand. Each tuple selects one value from each dimension. Example: (\nVegan\n,\nItalian\n,\nMulti-step\n). This manual work helps you understand your problem space.\nScale with two-step generation\n:\nGenerate structured tuples\n: Have the LLM create more combinations like (\nGluten-free\n,\nAsian\n,\nSimple\n)\nConvert tuples to queries\n: In a separate prompt, turn each tuple into natural language\nThis separation avoids repetitive phrasing. The (\nVegan\n,\nItalian\n,\nMulti-step\n) tuple becomes:\n\"I need a dairy-free lasagna recipe that I can prep the day before.\"\nGeneration approaches\nYou can generate tuples two ways:\nCross product then filter\n: Generate all dimension combinations, then filter with an LLM. Guarantees coverage including edge cases. Use when most combinations are valid.\nDirect LLM generation\n: Ask the LLM to generate tuples directly. This produces more realistic combinations, but it tends toward generic outputs and misses rare scenarios. Use it when many dimension combinations are invalid.\nFix obvious problems first\n: Don’t generate synthetic data for issues you can fix immediately. If your prompt doesn’t mention dietary restrictions, fix the prompt rather than generating specialized test queries.\nAfter iterating on your tuples and prompts,\nrun these synthetic queries through your actual system to capture full traces\n. A pool of roughly 100 diverse traces is a useful starting point for failure discovery. Have an agent help with sampling, annotate at least 30 traces yourself, then review the agent’s suggestions until your learning plateaus. See\nhow many examples you need for error discovery\nfor the full explanation.\nHere is a\nvisual\nthat helps visualize the process.\n↗ Focus view\nQ: Are there scenarios where synthetic data may not be reliable?\nYes: synthetic data can mislead or mask issues. For guidance on generating synthetic data when appropriate, see\nWhat is the best approach for generating synthetic data?\nCommon scenarios where synthetic data fails:\nComplex domain-specific content\n: LLMs often miss the structure, nuance, or quirks of specialized documents (e.g., legal filings, medical records, technical forms). Without real examples, critical edge cases are missed.\nLow-resource languages or dialects\n: For low-resource languages or dialects, LLM-generated samples are often unrealistic. Evaluations based on them won’t reflect actual performance.\nWhen validation is impossible\n: If you can’t verify synthetic sample realism (due to domain complexity or lack of ground truth), real data is important for accurate evaluation.\nHigh-stakes domains\n: In high-stakes domains (medicine, law, emergency response), synthetic data often lacks subtlety and edge cases. Errors here have serious consequences, and manual validation is difficult.\nUnderrepresented user groups\n: For underrepresented user groups, LLMs may misrepresent context, values, or challenges. Synthetic data can reinforce biases in the training data of the LLM.\n↗ Focus view\nQ: How can I do evals when traces contain sensitive data?\nThere is no replacement for looking at real interactions. This situation is not ideal, but there are some things you can do. Here are some options, in order of preference:\nTry to find real data you are allowed to inspect. A customer may agree to share a subset of traces, or test users may let you review their interactions. Even limited access gives you examples of how people use the product.\nIf you cannot inspect the data yourself, work with domain experts who are allowed to see it. Make your product\neasier for them to verify\nas part of their normal work. For example, a medical research assistant could show a clinician the evidence behind each claim and flag conflicting sources for review. The clinician can correct a specific claim or resolve a conflict while using the product. Those decisions can provide additional data for evals, subject to the same restrictions on what you can store and share.\nTo design this well, learn how the experts check an answer. Give them links to the supporting evidence and smaller pieces of work they can review. Asking whether the final answer was helpful often tells you too little about what went wrong. I discuss this approach in\nthis post\n.\nRedact or edit traces so they can be shared. If sensitive information cannot be stored, redact it before logging. Redaction tools can miss sensitive information, so check their output. When edited traces can be shared, removing personal information and changing sensitive details can make real examples usable for review. Check that those edits preserve the behavior you need to evaluate.\nIf none of the above options are possible, synthetic data should be your last resort. Synthetic data can help you find initial problems but has the downside that it only gives you limited evidence about how real users will behave. Read more about when\nsynthetic data may be unreliable\n.\n↗ Focus view\nQ: How do I approach evaluation when my system handles diverse user queries?\nComplex applications often support vastly different query patterns—from “What’s the return policy?” to “Compare pricing trends across regions for products matching these criteria.” Each query type exercises different system capabilities, leading to confusion on how to design eval criteria.\nError Analysis\nis all you need.\nYour evaluation strategy should emerge from observed failure patterns (e.g. error analysis), not predetermined query classifications. Rather than creating a massive evaluation matrix covering every query type you can imagine, let your system’s actual behavior guide where you invest evaluation effort.\nDuring error analysis, you’ll likely discover that certain query categories share failure patterns. For instance, all queries requiring temporal reasoning might struggle regardless of whether they’re simple lookups or complex aggregations. Similarly, queries that need to combine information from multiple sources might fail in consistent ways. These patterns discovered through error analysis should drive your evaluation priorities. It could be that query category is a fine way to group failures, but you don’t know that until you’ve analyzed your data.\nTo see an example of basic error analysis in action,\nsee this video\n.\nWatch “Error Analysis: The Highest ROI Technique In AI Engineering” on YouTube\n↗ Focus view\nQ: How can I efficiently sample production traces for review?\nThere are many ways to sample production traces for review. Here are some common methods.\nMethod\nWhat it does\nMain limitation\nRandom\nSelects traces with equal probability.\nA small batch can miss rare cases.\nClustering\nGroups traces by similar content and selects examples from each group.\nThe result depends on the features and clustering choices.\nData analysis\nReviews extreme values such as latency or tool count.\nAn extreme value may have nothing to do with quality.\nClassification\nUses an evaluator or another model to flag likely failures.\nIt favors problems the classifier already knows how to find.\nFeedback\nSelects traces with negative user feedback.\nIt misses problems that users do not report.\nThe table above orders sampling methods from the most exploratory to the most targeted. When you’re starting out, you should optimize for exploration of the data. As you learn more, you can start to lean more heavily on signals to select traces. The proper mix of methods depends on your goals and requires experimentation.\nKeep some random traces in every batch. This gives you a chance to find failure modes that your current signals do not describe.\nHow do I measure rare failure modes?\nUse targeted sampling to find rare failures. Search for signals that correlate with the failure, such as a specific tool sequence, unusually long traces, retries, or a known input pattern. Review the targeted batch to collect examples and improve the failure definition.\nThis flashcard\nfrom our evals flashcards series visualizes these methods.\nUse labels to choose the next traces\nWe can borrow a technique from machine learning called active learning to sample production traces. In active learning, a system asks a person to label the data points that would be most useful for its next update.\nIn\nShreya Shankar’s walkthrough\n, Claude Code clusters traces and chooses examples from each cluster for review. A\nmonitor\ncommand watches\nannotations.json\nfor new labels. When a label arrives, the agent updates a failure taxonomy and looks for similar cases or different failures.\nWatch “How to Automate AI Evals (Correctly)” on YouTube\nIn the above video, active learning is used in the context of\nerror analysis\nto find new cases to review. However, this approach can be used anywhere in the workflow where you are annotating data.\n↗ Focus view\n👉\nWant to learn more about AI Evals? Check out our\nAI Evals course\n. It’s a live cohort with hands on exercises and office hours. Here is a\n25% discount code\nfor readers. 👈\nEvaluation Design & Methodology\nQ: Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales)?\nEngineers often believe that Likert scales (1-5 ratings) provide more information than binary evaluations, allowing them to track gradual improvements. However, this added complexity often creates more problems than it solves in practice.\nBinary evaluations force clearer thinking and more consistent labeling. Likert scales introduce significant challenges: the difference between adjacent points (like 3 vs 4) is subjective and inconsistent across annotators, detecting statistical differences requires larger sample sizes, and annotators often default to middle values to avoid making hard decisions.\nHaving binary options forces people to make a decision rather than hiding uncertainty in middle values. Binary decisions are also faster to make during error analysis - you don’t waste time debating whether something is a 3 or 4.\nFor tracking gradual improvements, consider measuring specific sub-components with their own binary checks rather than using a scale. For example, instead of rating factual accuracy 1-5, you could track “4 out of 5 expected facts included” as separate binary checks. This preserves the ability to measure progress while maintaining clear, objective criteria.\nStart with binary labels to understand what ‘bad’ looks like. Numeric labels are advanced and usually not necessary.\n↗ Focus view\nQ: How do I combine my evals into a single metric?\nEach eval you create should return a\nbinary outcome\n(e.g. Pass or Fail). You will likely end up with many evals, each checking a different failure. However, people in your organization may want a single number to track.\nA simple approach I like to use is a “pass all” rate. An example passes only if it passes every check. For example, if 80 out of 100 examples pass every check, your pass-all rate is 80%. Design your report or dashboard so you can drill down from the overall pass-all rate to the pass rate for each check so you can see what’s contributing most to failures.\nA middle ground between one overall score and a separate result for every eval is to group related checks into themes. You can then report a pass-all rate for each group. For example, reviewing\nNurture Boss’s apartment leasing assistant\nrevealed problems with conversation flow, handoffs to humans, and rescheduling. Those themes could become groups of evals.\nAnother way to choose these groups is by how serious the failures are. For example, report one pass-all rate for checks that should block a release and another for issues you can tolerate. This approach can be helpful for gating production releases.\nIf you still need a single score that accounts for differences in importance, you can give some checks more weight than others. I discourage complicated weighted scores for the same reason I discourage\nLikert scales\nfor LLM judges. If your dashboard reports a composite score that jumps from 3.2 to 3.7 week over week, it’s easy to feel good about the increase without knowing what improved for users. In our experience, dashboards like this are usually performative and waste everyone’s time.\nWhichever approach you choose, remember that as your eval set changes,\nits scores may no longer be directly comparable with older scores\n. Evals give you challenges to improve against, and those challenges should change as your product evolves. For tracking progress with metrics over longer time horizons, it’s often better to use product metrics in addition to evals. Measures such as churn or active users can provide a more stable basis for comparison while your evals change.\n↗ Focus view\nQ: Should I practice eval-driven development?\nGenerally no.\nEval-driven development (writing evaluators before implementing features) sounds appealing but creates more problems than it solves. Unlike traditional software where failure modes are predictable, LLMs have infinite surface area for potential failures. You can’t anticipate what will break.\nA better approach is to start with\nerror analysis\n. Write evaluators for errors you discover, not errors you imagine. This avoids getting blocked on what to evaluate and prevents wasted effort on metrics that have no impact on actual system quality.\nException:\nEval-driven development may work for specific constraints where you know exactly what success looks like. If adding “never mention competitors,” writing that evaluator early may be acceptable.\nMost importantly, always do a\ncost-benefit analysis\nbefore implementing an eval. Ask whether the failure mode justifies the investment. Error analysis reveals which failures actually matter for your users.\n↗ Focus view\nQ: Should I build automated evaluators for every failure mode I find?\nFocus automated evaluators on failures that persist after fixing your prompts. Many teams discover their LLM doesn’t meet preferences they never actually specified - like wanting short responses, specific formatting, or step-by-step reasoning. Fix these obvious gaps first before building complex evaluation infrastructure.\nConsider the cost hierarchy of different evaluator types. Simple assertions and reference-based checks (comparing against known correct answers) are cheap to build and maintain. LLM-as-Judge evaluators require 100+ labeled examples, ongoing weekly maintenance, and coordination between developers, PMs, and domain experts. This cost difference should shape your evaluation strategy.\nOnly build expensive evaluators for problems you’ll iterate on repeatedly. Since LLM-as-Judge comes with significant overhead, save it for persistent generalization failures - not issues you can fix trivially. Start with cheap code-based checks where possible: regex patterns, structural validation, or execution tests. Reserve complex evaluation for subjective qualities that can’t be captured by simple rules.\n↗ Focus view\nQ: What model or LLM should I use to build automated evals?\nFirst check whether you can test the condition with code assertions. For example, suppose an AI assistant manages your contacts, and you want to test whether it creates a contact when asked. To test this functionality, you can give it a new contact to create, then query the database to check that exactly one matching record exists with the requested details. Using code assertions avoids the need for human labels.\nWhen a check requires judgment, use an LLM or another machine learning classifier. When using an LLM judge, we recommend using it as a classifier that returns\nPass or Fail\nfor the error you want to catch. Whichever model you use,\nvalidate it\nagainst human labels before trusting its decisions.\nFor example, you could try Jev from\nTypeSafe\n, BERT, or logistic regression. A different model may be cheaper or faster, and it may agree more or less closely with human labels. Measure these differences on your data to find the model that meets your application’s needs. For example, you might accept slower evaluations if they catch costly failures, or prefer a faster model when you need immediate feedback.\nWhen using an LLM, starting with a powerful model can make it easier to develop the judge’s prompt. Once it works well, try smaller, cheaper models and measure how much accuracy you lose. You can also\nuse the same model as your application\n.\nAn agent can help optimize the judge’s prompt once you have defined the task and labeled examples. Give it a specific failure to detect and a way to measure progress against your labels. “Find all errors and keep improving” is too vague. The agent needs to know what counts as an error and how to tell whether a change helped. Keep a\nseparate test set\noutside the optimization process to check if the\njudge generalizes\nto examples it was not tuned against.\n↗ Focus view\nQ: Can I use Jev for evals?\nYes.\nJev from TypeSafe\nis a general-purpose classifier that you can use for evals. An LLM judge that returns\nPass or Fail\nis also a classifier.\nYou validate Jev the same way you would\nany other classifier used for evals\n, by comparing its predictions against trusted labels. That’s why we’ve crossed out “LLM Judge” in our\noriginal flashcard\nand replaced it with “Classifier for Evals”:\nMeasure against human labels and keep training, development, and test data separate to avoid overfitting.\nTo understand the validation process described in the flashcard, see\nthis post\n.\nThe advantage of a fast inexpensive classifier (like Jev) is that it can make automated prompt tuning significantly cheaper and faster. Prompt tuning involves automatically trying changes to the evaluator’s prompt and checking whether its decisions agree more closely with human labels.\nGEPA\nis one example of a prompt tuning algorithm. Prompt tuning can sometimes require hundreds or thousands of evaluations, so a lower cost per run can add up to substantial savings.\nNo single classifier is best for every eval. Validation with human labels help you make trade-offs between accuracy, cost, and speed for your application.\n↗ Focus view\nQ: How do I know if I can trust my automated eval?\nFor an evaluator that makes judgments, test it against human-labeled examples of the failure you want to detect. This applies to LLM judges and other machine learning classifiers. You need to know how often they catch failures and how often they raise false alarms. If code can directly check the condition, you do not need human labels for that check. See\nwhich model or method to use for an eval\n.\nStart by splitting your labeled examples into three separate sets:\nTraining set:\nUse these examples to teach the evaluator what to look for. For an LLM judge or zero-shot classifier like\nJev\n, you can include them in its prompt.\nDevelopment set (dev):\nRun the evaluator on these examples and compare its decisions with your labels. Inspect disagreements to improve the prompt or choose between models. Repeat this as you develop the evaluator. A prompt tuning algorithm will use the dev set to guide its changes.\nTest set:\nSet these examples aside until you finish making changes. Use them for a final check on examples that have not influenced any decisions about the evaluator.\nEach time you use dev results to change the prompt or choose a model, information from those examples influences the evaluator. After many rounds, it may do well on the dev set but poorly on new examples. This is overfitting, and it can happen even if you never put the dev examples directly in the prompt. The test set gives you a final check on data that hasn’t guided those changes.\nIf test scores are much worse than dev scores, investigate whether you’ve overfit. Small samples make these measurements less certain, and differences between the sets can also cause a gap. If you’ve overfit, revisit the instructions and examples, then repeat development with a new, untouched test set reserved for the final check. Addressing overfitting is beyond the scope of this FAQ.\nTo measure how well the evaluator aligns with human judgments, use the following metrics. Here, “positive” means an error is present, matching the flashcard below.\nTrue positive rate (TPR), also called recall,\nmeasures how many actual failures the evaluator catches. If people identify 10 failures and the evaluator catches eight, its TPR is 80%. Prioritize this when missing a failure is costly.\nTrue negative rate (TNR)\nmeasures how many good outputs the evaluator correctly passes. If people identify 100 good outputs and the evaluator passes 95, its TNR is 95%. The other five are false alarms. A high TNR helps avoid wasting people’s time reviewing good outputs that were incorrectly flagged.\nTrack both rates as you make changes. Catching more failures can come at the cost of more false alarms. Choose acceptable levels based on the consequences for your application. If failures are rare, even a small false-alarm rate can create a lot of unnecessary reviews.\nThe flashcard below illustrates this process for an LLM judge. The same separation of development and testing applies to\nother evaluators\n.\nHow to trust an LLM judge: validate against human labels, separate training, development, and test examples, and measure TPR and TNR.\nThe flashcard’s dataset split is an example for prompt-based judges or\nzero-shot classifiers\n. Training a classifier may require a larger share of training data. Choose your targets based on the cost of missed failures and false alarms in your application.\n↗ Focus view\nQ: What should I do when I can’t get my LLM judge to agree with human reviewers?\nTo debug a LLM judge, you need examples with human Pass/Fail labels to compare its decisions against. An effective way to get these labels is\nerror analysis\n, which provides you with a structured way to review your application’s data and find errors.\nAs you collect labeled examples (we recommend at least 50 passing and 50 failing examples), inspect where the judge disagrees with the human labels to get clues on what needs fixing. Common issues include\nmissing context\nor vague instructions. If you have trouble deciding whether an example should pass or fail, this is a sign that you need to refine your definition of success more precisely.\nInspect a few disagreements manually before trying automated prompt tuning. Algorithms such as\nGEPA\ntry changes to the judge’s prompt and measure whether they improve agreement with human labels. If you engage in prompt tuning too early, you can miss important problems that aren’t prompt related (like missing context, bad labels, etc.).\nThe most common mistake people make is directing their LLM judge to catch too many different kinds of errors at once. Instead, we recommend building a separate judge for each type of failure. For example, checking whether the assistant escalated to a human when required is more specific than grading overall conversation quality. A focused judge is also easier to align with human labels and is more actionable.\nFinally, make sure your judge can generalize to data you haven’t seen (i.e. its not overfitting to the data you’re tuning it with). The best way to thest this is to set aside human-labeled examples and save them for a final test. The\nvalidation FAQ\nexplains how to split your data and measure whether the judge agrees with human reviewers on unseen examples.\n↗ Focus view\nQ: Should I use \"ready-to-use\" evaluation metrics?\nNo. Generic evaluations waste time and create false confidence when you use them as quality measures.\nHowever, they can still help you find traces to inspect.\nWhy are generic eval metrics misleading?\nGeneric evaluation metrics are everywhere. Eval libraries contain scores like helpfulness, coherence, quality, etc. promising easy evaluation. These metrics measure abstract qualities that may not matter for your use case. Good scores on them don’t mean your system works.\nInstead, conduct\nerror analysis\nto understand failures. Define\nbinary failure modes\nbased on real problems. Create\ncustom evaluators\nfor those failures and validate them against human judgment.\nExperienced practitioners may use generic metrics as exploration signals. Once you understand why they fail as quality measures, you can use them to\nfind interesting traces\nfor human review.\n↗ Focus view\nQ: Are similarity metrics (BERTScore, ROUGE, etc.) useful for evaluating LLM outputs?\nGeneric metrics like BERTScore, ROUGE, cosine similarity, etc. are not useful for evaluating LLM outputs in most AI applications. Instead, we recommend using\nerror analysis\nto identify metrics specific to your application’s behavior. We recommend designing\nbinary pass/fail\n.) evals (using LLM-as-judge) or code-based assertions.\nAs an example, consider a real estate CRM assistant. Suggesting showings that aren’t available (can be tested with an assertion) or confusing client personas (can be tested with a LLM-as-judge) is problematic . Generic metrics like similarity or verbosity won’t catch this. A relevant quote from the course:\n“The abuse of generic metrics is endemic. Many eval vendors promote off the shelf metrics, which ensnare engineers into superfluous tasks.”\nSimilarity metrics aren’t always useless. They have utility in domains like search and recommendation (and therefore can be useful for\noptimizing and debugging retrieval\nfor RAG). For example, cosine similarity between embeddings can measure semantic closeness in retrieval systems, and average pairwise similarity can assess output diversity (where lower similarity indicates higher diversity).\n↗ Focus view\nQ: Can I use the same model for both the main task and evaluation?\nFor LLM-as-Judge selection, using the same model is usually fine because the judge is doing a different task than your main LLM pipeline. While\nresearch has shown\nthat models can exhibit bias when evaluating their own outputs, what ultimately matters is how well your judge aligns with human judgments. The judges we recommend building do\nscoped binary classification tasks\n. We’ve found that iterative alignment with human labels is usually achievable on this constrained task.\nFocus on achieving high True Positive Rate (TPR) and True Negative Rate (TNR) with your judge on a held out labeled test set. If you struggle to achieve good alignment with human scores, then consider trying a different model. However onboarding new model providers may involve non-trivial effort in some organizations, which is why we don’t advocate for using different models by default unless there’s a specific alignment issue.\nWhen selecting judge models, start with the most capable models available to establish strong alignment with human judgments. You can optimize for cost later once you’ve established reliable evaluation criteria.\n↗ Focus view\nQ: How much context should I give a LLM judge?\nGive each judge only the parts of the trace it needs for its failure mode. Do not give every judge the same full trace by default. Extra context can cause\ncontext rot\nand make the judge worse.\nFinding the right pieces of context often requires experimentation. Test your choices by comparing the judge’s decisions with human labels. Then, inspect disagreements to see whether the judge lacked necessary evidence or was distracted by irrelevant information.\nIf you’re unsure whether a piece of information helps, try an ablation study. This means removing one piece at a time and checking how the results change against human labels. If performance stays the same or improves, you may be able to leave it out.\nLong-running agents can produce large traces that fill or exceed the judge’s context window. For these cases, consider giving the judge a tool to search the parts it needs. However, don’t add this unless you absolutely need it, as a tool like this adds additional complexity, cost, and latency.\n↗ Focus view\nQ: How do we evaluate a model’s ability to express uncertainty or \"know what it doesn’t know\"?\nMany applications require a model that can refuse to answer a question when it lacks sufficient information. To evaluate whether this refusal behavior is well-calibrated, you need to test if the model refuses at the appropriate times without refusing to answer questions it\nshould\nbe able to answer.\nTo do this effectively, you should construct an evaluation set that has the following components:\nAnswerable Questions:\nScenarios where a correct, verifiable answer is present in the model’s provided context or general knowledge.\nUnanswerable Questions:\nScenarios designed to tempt the model to hallucinate. These include questions with false premises, queries about information explicitly missing from context, or topics far outside its knowledge base.\nWhile the exact proportion isn’t critical, a balanced set with a roughly equal number of answerable and unanswerable questions is a good starting point. The diversity and difficulty of the questions are more important than the precise ratio.\nThe evaluation itself is a binary (Pass/Fail) check of the model’s judgment. A “Pass” requires the model to satisfy two conditions: it must answer the answerable questions while also refusing to answer the unanswerable ones. A failure is defined as providing a fabricated answer to an unanswerable question, which indicates poor calibration.\nIn the research literature, this capability is known as “Abstention Ability.” To improve this behavior, it is worth\nsearching for this term on Arxiv\nto understand the latest techniques.\n↗ Focus view\nHuman Annotation & Process\nQ: How many people should annotate my LLM outputs?\nFor most small to medium-sized companies, appointing a single domain expert as a “benevolent dictator” is the most effective approach. This person becomes the definitive voice on quality standards. The expert might be a psychologist for a mental health chatbot or a lawyer for legal document analysis.\nA single expert eliminates annotation conflicts and prevents the paralysis that comes from “too many cooks in the kitchen”. The benevolent dictator can incorporate input and feedback from others, but they drive the process. If you feel like you need five subject matter experts to judge a single interaction, it’s a sign your product scope might be too broad.\nHowever, larger organizations or those operating across multiple domains (like a multinational company with different cultural contexts) may need multiple annotators. When you do use multiple people, you’ll need to measure their agreement using metrics like Cohen’s Kappa, which accounts for agreement beyond chance. However, use your judgment. Even in larger companies, a single expert is often enough.\nHow should annotators resolve disagreements?\nHave annotators label the same examples independently before they discuss them. Measure agreement and collect the cases where their labels differ. During an alignment session, ask which part of the rubric caused the disagreement and what rule would make the next decision clear.\nUpdate the rubric with a definition, rule, or example that covers the disputed case. Then relabel affected examples. If the annotators still disagree, assign a domain expert to make the final decision and record the reason.\nStart with a benevolent dictator whenever feasible. Only add complexity when absolutely necessary.\n↗ Focus view\nQ: How can I make AI outputs easier for people to evaluate?\nStart by scrutinizing your product design. It’s often helpful to surface intermediate outputs users can check before a final result. For example, suppose you have an agent that writes a medical report by synthesizing a patient’s medical history. Instead of asking a doctor to provide feedback on the report, show the extracted facts with links to the source material and let doctors correct a fact or resolve conflicting evidence before generating the report. This also keeps the doctor involved and helps them build trust by checking the work as they go. This is a sketch of how such an interface might look:\nA mockup that guides a doctor through facts and conflicting evidence before generating a report.\nFor more discussion on designing for verification, see\n“It’s Hard to Eval” Is a Product Smell\n. The post expands on this example and discusses several others with before-and-after mockups.\nAfter you have designed for verification, make sure the review interface removes friction from reviewing data. See the advice on\nbuilding a review interface\n. Some common tips include:\nDisplay outputs in a familiar format. Render generated emails as emails, and use syntax highlighting for code.\nKeep the context reviewers need on the same screen. Put less important details in sections they can expand when needed.\nAdd keyboard shortcuts for moving between examples and recording judgments. Make it easy to save notes without reaching for the mouse.\nShow progress, such as “45 of 100 examples reviewed,” so reviewers know how much work remains.\nNext, debug the review process. First, try fewer examples so reviewers have time to inspect each one carefully. Have people review the same examples independently and\ndiscuss disagreements\n. You can also review examples together to see where people get stuck. Disagreement can reveal unclear instructions or missing information.\n↗ Focus view\nQ: Should product managers and engineers collaborate on error analysis? How?\nAt the outset, collaborate to establish shared context. Engineers catch technical issues like retrieval issues and tool errors. PMs identify product failures like unmet user expectations, confusing responses, or missing features users expect.\nAs time goes on you should lean towards a\nbenevolent dictator\nfor error analysis: a domain expert or PM who understands user needs. Empower domain experts to evaluate actual outcomes rather than technical implementation. Ask “Has an appointment been made?” not “Did the tool call succeed?” The best way to empower the domain expert is to give them\ncustom annotation tools\nthat display system outcomes alongside traces. Show the confirmation, generated email, or database update that validates goal completion. Keep all context on one screen so non-technical reviewers focus on results.\n↗ Focus view\nQ: Can I help with evals if I’m not a domain expert?\nYes, especially when you’re beginning with evals. I’m often surprised by the number of low-hanging fruit I find while reviewing data that don’t require domain knowledge. For example, I’ve found issues like this in specialized domains as an outsider:\nText message chatbots getting confused by the conversational flow of lots of short, broken-up messages people tend to write in text versus chat.\nLack of query disambiguation or follow-up when users’ requests are obviously vague.\nNot having proper instrumentation, logging or traces to begin with.\nLack of widgets, UI elements or other affordances that help users complete tasks versus over-reliance on text responses.\nFurthermore, ask a domain expert to walk through an example and explain why it is good or bad. Watch what they check and which evidence they need. Use what you learn to\nbuild a better annotation interface\nthat makes reviewing easier.\nYou can also help the team collect interactions and review them regularly. For example, see\nhow product managers and engineers can collaborate on error analysis\nto get an idea of how to structure cross-functional collaboration.\nLastly, make sure you leave judgments that require specialized knowledge to the expert. However, don’t assume you need domain expertise to start being useful!\n↗ Focus view\nQ: How do I evaluate outputs in a language I don’t speak?\nEven though you can translate interactions in a foreign language with an LLM, be cautious about relying on it. Translation often loses meaning as some words and expressions have no direct equivalent. Moreover, what “good” means often depends on culture and social norms. For example, understanding the literal meaning of an answer is not enough to judge whether its tone is appropriate in a different cultural frame.\nBecause of these limitations, we recommend involving a reviewer who understands both the language and the cultural context of your users. You can still\ncontribute to evals\nby helping organize the review and investigating problems you can identify yourself. That reviewer should\nset the standard for error analysis\n.\nIf you cannot find a reviewer who understands both the language and cultural context, be aware that your assessment will be limited.\n↗ Focus view\nQ: Should I outsource annotation & labeling to a third party?\nOutsourcing\nerror analysis\nis usually a big mistake (with some\nexceptions\n). The core of evaluation is building the product intuition that only comes from systematically analyzing your system’s failures. You should be extremely skeptical of this process being delegated.\nThe Dangers of Outsourcing\nWhen you outsource annotation, you often break the feedback loop between observing a failure and understanding how to improve the product. Problems with outsourcing include:\nSuperficial Labeling: Even well-defined metrics require nuanced judgment that external teams lack. A critical misstep in error analysis is excluding domain experts from the labeling process. Outsourcing this task to those without domain expertise, like general developers or IT staff, often leads to superficial or incorrect labeling.\nLoss of Unspoken Knowledge: A principal domain expert possesses tacit knowledge and user understanding that cannot be fully captured in a rubric. Involving these experts helps uncover their preferences and expectations, which they might not be able to fully articulate upfront.\nAnnotation Conflicts and Misalignment: Without a shared context, external annotators can create more disagreement than they resolve. Achieving alignment is a challenge even for internal teams, which means you will spend even more time on this process.\nThe Recommended Approach: Build Internal Capability\nInstead of outsourcing, focus on building an efficient internal evaluation process.\n1. Appoint a “Benevolent Dictator”. For most teams, the most effective strategy is to appoint a\nsingle, internal domain expert\nas the final decision-maker on quality. This individual sets the standard, ensures consistency, and develops a sense of ownership.\n2. Use a collaborative workflow for multiple annotators. If multiple annotators are necessary, follow a structured process to ensure alignment: * Draft an initial rubric with clear Pass/Fail definitions and examples. * Have each annotator label a shared set of traces independently to surface differences in interpretation. * Measure Inter-Annotator Agreement (IAA) using a chance-corrected metric like Cohen’s Kappa. * Facilitate alignment sessions to discuss disagreements and refine the rubric. * Iterate on this process until agreement is consistently high.\nHow to Handle Capacity Constraints\nBuilding internal capacity does not mean you have to label every trace. Use these strategies to manage the workload:\nSmart Sampling: Review a small, representative sample of traces thoroughly. It is more effective to analyze 100 diverse traces to find patterns than to superficially label thousands.\nThe “Think-Aloud” Protocol: To make the most of limited expert time, use this technique from usability testing. Ask an expert to verbalize their thought process while reviewing a handful of traces. This method can uncover deep insights in a single one-hour session.\nBuild Lightweight Custom Tools: Build\ncustom annotation tools\nto streamline the review process, increasing throughput.\nExceptions for External Help\nWhile outsourcing the core error analysis process is not recommended, there are some scenarios where external help is appropriate:\nPurely Mechanical Tasks: For highly objective, unambiguous tasks like identifying a phone number or validating an email address, external annotators can be used after a rigorous internal process has defined the rubric.\nTasks Without Product Context: Well-defined tasks that don’t require understanding your product’s specific requirements can be outsourced. Translation is a good example: it requires linguistic expertise but not deep product knowledge.\nEngaging Subject Matter Experts: Hiring external SMEs to act as your internal domain experts is not outsourcing; it is bringing the necessary expertise into your evaluation process. For example,\nAnkiHub\nhired 4th-year medical students to evaluate their RAG systems for medical content rather than outsourcing to generic annotators.\n↗ Focus view\nQ: How do you review a trace that is really large?\nTraces can get large when an agent runs for a long time or retrieves a large amount of context. A useful heuristic is to focus on the\nfirst upstream failure\n. Errors tend to compound, which means you can prioritize earlier ones to save time.\nUse progressive disclosure in your review tool by showing the most relevant information first and letting reviewers expand details as needed. For example, show the conversation initially, with tool outputs collapsed until a reviewer needs to inspect them.\nIf a single trace is still too large to review, work with the domain expert to identify what they need to check. Build a tool that extracts the relevant evidence and links back to its location in the trace or retrieved document. For example, when reviewing an answer about a long contract, the tool could show the relevant clauses with links to their original pages. Always validate this kind of extraction with a domain expert.\nQuality is more important than quantity. You can usually learn more from carefully investigating a few failures than from rushing through many traces.\n↗ Focus view\nQ: What parts of evals can be automated with LLMs?\nLLMs can speed up parts of your eval workflow, but they can’t replace human judgment where your expertise is essential. For example, if you let an LLM handle all of\nerror analysis\n(i.e., reviewing and annotating traces), you might overlook failure cases that matter for your product. Suppose users keep mentioning “lag” in feedback, but the LLM lumps these under generic “performance issues” instead of creating a “latency” category. You’d miss a recurring complaint about slow response times and fail to prioritize a fix.\nThat said, LLMs are valuable tools for accelerating certain parts of the evaluation workflow\nwhen used with oversight\n.\nHere are some areas where LLMs can help:\nFirst-pass axial coding:\nAfter you’ve open coded 30–50 traces yourself, use an LLM to organize your raw failure notes into proposed groupings. This helps you quickly spot patterns, but always review and refine the clusters yourself.\nNote: If you aren’t familiar with axial and open coding, see\nthis faq\n.\nMapping annotations to failure modes:\nOnce you’ve defined failure categories, you can ask an LLM to suggest which categories apply to each new trace (e.g., “Given this annotation: [open_annotation] and these failure modes: [list_of_failure_modes], which apply?”).\nSuggesting prompt improvements:\nWhen you notice recurring problems, have the LLM propose concrete changes to your prompts. Review these suggestions before adopting any changes.\nAnalyzing annotation data:\nUse LLMs or AI-powered notebooks to find patterns in your labels, such as “reports of lag increase 3x during peak usage hours” or “slow response times are mostly reported from users on mobile devices.”\nHowever, you shouldn’t outsource these activities to an LLM:\nInitial open coding:\nAlways read through the raw traces yourself at the start. This is how you discover new types of failures, understand user pain points, and build intuition about your data. Never skip this or delegate it.\nValidating failure taxonomies:\nLLM-generated groupings need your review. For example, an LLM might group both “app crashes after login” and “login takes too long” under a single “login issues” category, even though one is a stability problem and the other is a performance problem. Without your intervention, you’d miss that these issues require different fixes.\nGround truth labeling:\nFor any data used for testing/validating LLM-as-Judge evaluators, hand-validate each label. LLMs can make mistakes that lead to unreliable benchmarks.\nRoot cause analysis:\nLLMs may point out obvious issues, but only human review will catch patterns like errors that occur in specific workflows or edge cases—such as bugs that happen only when users paste data from Excel.\nIn conclusion, start by examining data manually to understand what’s actually going wrong. Use LLMs to scale what you’ve learned, not to avoid looking at data.\n↗ Focus view\nQ: Should I stop writing prompts manually in favor of automated tools?\nAutomating prompt engineering can be tempting, but you should be skeptical of tools that promise to optimize prompts for you, especially in early stages of development. When you write a prompt, you are forced to clarify your assumptions and externalize your requirements. Good writing is good thinking\n1\n. If you delegate this task to an automated tool too early, you risk never fully understanding your own requirements or the model’s failure modes.\nThis is because automated prompt optimization typically hill-climb a predefined evaluation metric. It can refine a prompt to perform better on known failures, but it cannot discover\nnew\nones. Discovering new errors requires\nerror analysis\n. Furthermore, research shows that evaluation criteria tends to shift after reviewing a model’s outputs, a phenomenon known as “criteria drift”\n2\n. This means that evaluation is an iterative, human-driven sensemaking process, not a static target that can be set once and handed off to an optimizer.\nA pragmatic approach is to use LLMs to improve your prompt based on\nopen coding\n(open-ended notes about traces). This way, you maintain a human in the loop who is looking at the data and externalizing their requirements. Once you have a high-quality set of evals, prompt optimization can be effective for that last mile of performance.\n↗ Focus view\n👉\nWant to learn more about AI Evals? Check out our\nAI Evals course\n. It’s a live cohort with hands on exercises and office hours. Here is a\n25% discount code\nfor readers. 👈\nTools & Infrastructure\nQ: Should I build a custom annotation tool or use something off-the-shelf?\nBuild a custom annotation tool.\nThis is the single most impactful investment you can make for your AI evaluation workflow. With AI-assisted development tools like Cursor or Lovable, you can build a tailored interface in hours. I often find that teams with custom annotation tools iterate ~10x faster.\nCustom tools excel because:\nThey show all your context from multiple systems in one place\nThey can render your data in a product specific way (images, widgets, markdown, buttons, etc.)\nThey’re designed for your specific workflow (custom filters, sorting, progress bars, etc.)\nOff-the-shelf tools may be justified when you need to coordinate dozens of distributed annotators with enterprise access controls. Even then, many teams find the configuration overhead and limitations aren’t worth it.\nIsaac’s Anki flashcard annotation app\nshows the power of custom tools—handling 400+ results per query with keyboard navigation and domain-specific evaluation criteria that would be nearly impossible to configure in a generic tool.\nWatch “Building Eval Tools with FastHTML” on YouTube\n↗ Focus view\nQ: What makes a good custom interface for reviewing LLM outputs?\nGreat interfaces make human review fast, clear, and motivating. We recommend\nbuilding your own annotation tool\ncustomized to your domain. The following features are possible enhancements we’ve seen work well, but you don’t need all of them. The screenshots shown are illustrative examples to clarify concepts. In practice, I rarely implement all these features in a single app. It’s ultimately a judgment call based on your specific needs and constraints.\n1. Render Traces Intelligently, Not Generically\n:\nPresent the trace in a way that’s intuitive for the domain. If you’re evaluating generated emails, render them to look like emails. If the output is code, use syntax highlighting. Allow the reviewer to see the full trace (user input, tool calls, and LLM reasoning), but keep less important details in collapsed sections that can be expanded. Here is an example of a custom annotation tool for reviewing real estate assistant emails:\nA custom interface for reviewing emails for a real estate assistant.\n2. Show Progress and Support Keyboard Navigation\n:\nKeep reviewers in a state of flow by minimizing friction and motivating completion. Include progress indicators (e.g., “Trace 45 of 100”) to keep the review session bounded and encourage completion. Enable hotkeys for navigating between traces (e.g., N for next), applying labels, and saving notes quickly. Below is an illustration of these features:\nAn annotation interface with a progress bar and hotkey guide\n3. Trace navigation through clustering, filtering, and search\n:\nAllow reviewers to filter traces by metadata or search by keywords. Semantic search helps find conceptually similar problems. Clustering similar traces (like grouping by user persona) lets reviewers spot recurring issues and explore hypotheses. Below is an illustration of these features:\nCluster view showing groups of emails, such as property-focused or client-focused examples. Reviewers can drill into a group to see individual traces.\n4. Prioritize labeling traces you think might be problematic\n:\nSurface traces flagged by guardrails, CI failures, or automated evaluators for review. Provide buttons to take actions like adding to datasets, filing bugs, or re-running pipeline tests. Display relevant context (pipeline version, eval scores, reviewer info) directly in the interface to minimize context switching. Below is an illustration of these ideas:\nA trace view that allows you to quickly see auto-evaluator verdict, add traces to dataset or open issues. Also shows metadata like pipeline version, reviewer info, and more.\nGeneral Principle: Keep it minimal\nKeep your annotation interface minimal. Only incorporate these ideas if they provide a benefit that outweighs the additional complexity and maintenance overhead.\n↗ Focus view\nQ: What gaps in eval tooling should I be prepared to fill myself?\nMost eval tools handle the basics well: logging complete traces, tracking metrics, prompt playgrounds, and annotation queues. These are table stakes. Here are four areas where you’ll likely need to supplement existing tools.\nWatch for vendors addressing these gaps: it’s a strong signal they understand practitioner needs.\n1. Error Analysis and Pattern Discovery\nAfter reviewing traces where your AI fails, can your tooling automatically cluster similar issues? For instance, if multiple traces show the assistant using casual language for luxury clients, you need something that recognizes this broader “persona-tone mismatch” pattern. We recommend building capabilities that use AI to suggest groupings, rewrite your observations into clearer failure taxonomies, help find similar cases through semantic search, etc.\n2. AI-Powered Assistance Throughout the Workflow\nThe most effective workflows use AI to accelerate every stage of evaluation. During error analysis, you want an LLM helping categorize your open-ended observations into coherent failure modes. For example, you might annotate several traces with notes like “wrong tone for investor,” “too casual for luxury buyer,” etc. Your tooling should recognize these as the same underlying pattern and suggest a unified “persona-tone mismatch” category.\nYou’ll also want AI assistance in proposing fixes. After identifying 20 cases where your assistant omits pet policies from property summaries, can your workflow analyze these failures and suggest specific prompt modifications? Can it draft refinements to your SQL generation instructions when it notices patterns of missing WHERE clauses?\nGood workflows also help you conduct data analysis of your annotations and traces. I like using notebooks with AI in-the-loop like\nJulius\nor\nHex\n. These help me discover insights like “location ambiguity errors spike 3x when users mention neighborhood names” or “tone mismatches occur 80% more often in email generation than other modalities.”\n3. Custom Evaluators Over Generic Metrics\nBe prepared to build most of your evaluators from scratch. Generic metrics like “hallucination score” or “helpfulness rating” rarely capture what actually matters for your application—like proposing unavailable showing times or omitting budget constraints from emails. In our experience, successful teams spend most of their effort on application-specific metrics.\n4. APIs That Support Custom Annotation Apps\nCustom annotation interfaces\nwork best for most teams\n. This requires observability platforms with thoughtful APIs. I often have to build my own libraries and abstractions just to make bulk data export manageable. You shouldn’t have to paginate through thousands of requests or handle timeout-prone endpoints just to get your data. Look for platforms that provide true bulk export capabilities and, crucially, APIs that let you write annotations back efficiently.\n↗ Focus view\nQ: What should an internal eval platform standardize across teams?\nWhen building an internal eval platform, it’s tempting to start with tools, infrastructure, and a shared set of metrics. That can lead teams to adopt whatever the platform offers without checking whether it helps them find and fix problems in their products.\nStart by encouraging teams to perform\nerror analysis\nand\nsample data\neffectively for review. They can use the failures they find to decide which\nautomated checks\nto build, then\nvalidate evaluators\nagainst human labels. Standardize these processes while letting each team develop its own metrics and, when needed, tools. The\nfield guide\nshows an example of how these might fit together.\nGive teams the flexibility to build their own tools, especially now that AI coding agents make custom software cheaper to create. For example, tools to\nannotate data\noften need\ncustom interfaces\nthat fit the data being reviewed. Reviewing text extracted from a scanned document calls for a different interface than reviewing chat conversations.\nA platform can still provide shared storage for results and support collaboration on labeling. Start by serving one team and one use case well, then expand as you learn which needs are shared. The benefit of standardization is smaller when teams have very different needs and can build their own tools cheaply.\nComparing eval scores across projects only makes sense when the checks and test data are comparable.\nWe strongly advise against offering\ngeneric metrics\n, such as helpfulness or coherence, as a shortcut. They are rarely useful as quality measures and tend to distract teams from the failures that affect their users.\n↗ Focus view\nQ: What’s your favorite eval vendor?\nEval tools are in an intensely competitive space. It would be futile to compare their features. If I tried to do such an analysis, it would be invalidated in a week! Vendors I encounter the most organically in my work are:\nLangsmith\n,\nArize\nand\nBraintrust\n.\nWhen I help clients with vendor selection, the decision weighs heavily towards who can offer the best support, as opposed to purely features. This changes depending on size of client, use case, etc. Yes - it’s mainly the human factor that matters, and dare I say, vibes.\nI have no favorite vendor. At the core, their features are very similar - and I often build\ncustom tools\non top of them to fit my needs.\nHere is a\nvideo series\nthat has a live commentary on the relative strengths and weaknesses of the three aforementioned vendors.\n↗ Focus view\nQ: How should I version and manage prompts?\nThere is an unavoidable tension between keeping prompts close to the code vs. an environment that non-technical stakeholders can access.\nMy preferred approach is storing prompts in Git.\nThis treats them as software artifacts that are versioned, reviewed, and deployed atomically with the application code. While the Git command line is unfriendly for non-technical folks, the\nGitHub\nweb interface and the GitHub\nDesktop app\nmake it very approachable. When I was working at GitHub, I worked with many non-technical professionals, including lawyers and accountants, who used these tools effectively. Here is a\nblog post\naimed at non-technical folks to get started.\nAlternatively, most vendors in the LLM tooling space, such as observability platforms like Arize, Braintrust, and LangSmith, offer dedicated prompt management tools. These are accessible for rapid iteration but risk creating additional layers of indirection.\nWhy prompt management tools often fall short:\nAI products typically involve many moving parts: tools, RAG, agents, etc. Prompt management tools are inherently limiting because they can’t easily execute your application’s code. Even when they can, there’s often significant indirection involved, making it difficult to test prompts with your system’s capabilities.\nWhen possible, a notebook provides a great solution for prompt experimentation\nIf you have Python entry points into your codebase or your codebase is written in Python, Jupyter notebooks are particularly powerful for this purpose. You can experiment with prompts and iterate on your actual AI agents with their full tool and RAG capabilities. This makes it much easier to understand how your system works in practice. Additionally, you can create widgets and small user interfaces within notebooks, giving you the best of both worlds for experimentation and iteration. To see what this looks like in practice, Teresa Torres gives a fantastic, hands-on walkthrough of how she, as a PM, used notebooks for the entire eval and experimentation lifecycle:\nWatch “From Noob to Automated Evals In A Week (as a PM) w/Teresa Torres” on YouTube\nIf notebooks are not feasible for your code base, an\n​integrated prompt environment\n​ can be effective for experimentation. Either way, I prefer to version and manage prompts in Git.\n↗ Focus view\nQ: What should go in the system prompt vs. the user prompt?\nNothing beats experimentation.\nTest both approaches (ideally with evals) with your specific model and use case. Models handle system and user prompts differently, and these differences vary by provider and model version. Move instructions between prompts and measure which produces better results for your specific task.\nGeneral guidelines:\nPut static instructions and role definitions in the system prompt. Put dynamic content, examples, and task-specific details in the user prompt. Think of the system prompt as the model’s constitution—rules that apply across all requests. Include identity, behavioral constraints, output format requirements, and standing instructions: “You are a medical assistant. Never provide diagnoses. Always recommend consulting a healthcare provider.”\nThe user prompt contains the actual task, relevant context, few-shot examples, and data to process. Documents for analysis, query-specific variations, and contextual information belong here. When the distinction feels unclear, prefer the user prompt. It’s more portable across models and easier to debug.\n↗ Focus view\nProduction & Deployment\nQ: How are evaluations used differently in CI/CD vs. monitoring production?\nCI evals protect against known regressions before deployment. Online monitoring find failures in production traffic and estimate how often they occur.\nEvals in CI\nTest datasets for CI are small (in many cases 100+ examples) and purpose-built. Examples cover core features, regression tests for past bugs, and known edge cases. Since CI tests are run frequently, the cost of each test has to be carefully considered (that’s why you carefully curate the dataset). Favor assertions or other deterministic checks over LLM-as-judge evaluators.\nOnnline monitoring for production\nFor evaluating production traffic, you can sample live traces and run evaluators against them asynchronously. Since you usually lack reference outputs on production data, you might rely more on on more expensive reference-free evaluators like LLM-as-judge. Additionally, track confidence intervals for production metrics. If the lower bound crosses your threshold, investigate further.\nConnect the two systems\nThese two systems are complementary: when production monitoring reveals new failure patterns through error analysis and evals, add representative examples to your CI dataset. This mitigates regressions on new issues.\nHere is a visual\nthat helps contrast the approaches.\n↗ Focus view\nQ: How often should I run my evals?\nThere are three dimensions to consider:\nThe\ncost\nto run and maintain the eval. The more expensive the eval, the greater benefit it needs to provide to justify running it frequently. For example, LLM-as-a-judge is more expensive to run than a unit test.\nHow saturated\nthe eval is on the dataset. If the eval passes all examples, its giving you no new information. You should consider retiring the eval or running it less frequently if its saturated. However, you should first try to\nmake the eval more difficult\nso its not saturated to begin with.\nThe\nbusiness value\nof catching this error. For critical errors, the busines value of catching it may be high enough that you should run it more frequently, despite its cost. One caveat here is not to get carried away with hypothetical errors. At the very least, you should prove that you can trigger the error at least once by red-teaming your application before implementing the eval (which will also help you make a better eval)\nThere are no bright-line rules. This decision often requires judgement as opposed to something formulaic. Here’s a visual that can help you think through the tradeoffs:\nExamples\nBelow are concrete examples to help you understand the factors involved. Note that these are illustrative:\nEval\nTest examples passing\nBusiness cost of failure\nSuggested schedule\nAn answer-quality judge with GPT-6 Astra on max reasoning.\nAll pass\nMedium\nRetire or run infrequently (e.g. every 2 weeks)\nA code assertion which checks that a contact was saved correctly.\nAll pass\nMedium\nYou can run this on every change b/c its incredibly cheap.\nA judge with Fable 5 checks whether a support agent follows a new refund policy.\nNone pass\nMedium\nEven though expensive, the eval is providing useful feedback b/c nothing is passing, and the business value of catching the error is high enough. I would run this as frequently as possible.\nA judge with GPT-6 Luna checks whether a support agent resolves the customer’s problem.\nSome pass\nMedium\nNot a terribly expensive judge b/c model is smaller and the eval is still catching errors, so I would run this somewhat frequently (e.g. nightly).\nA judge with GPT-6 Astra on max reasoning checks for improper disclosure of confidential information on a legal assistant.\nAll pass\nHigh\nEven though expensive and eval is saturated, the business value of catching the error is high enough that I would run this prior to each release. Given the importance of the error, I would also try to make the eval more difficult so that it’s more useful.\nA judge with GPT-6 Astra on medium reasoning checks a minor formatting preference.\nSome pass\nLow\nOccasionally or retire; use code instead if possible.\nIt’s always worth exploring\ncheaper evaluators\nto see if you can find one that provides similar or better alignment with human labels for less cost.\nOffline vs. Online Evals\nThe discussion here focused on offline evals. Online evals involve similar considerations, with an additional decision about how many production traces to sample. For example, you might run cheap checks on every trace and an expensive judge on a nightly sample. For more discussion on how these approaches work together, see\nHow are evaluations used differently in CI/CD vs. monitoring production?\n↗ Focus view\nQ: What’s the difference between guardrails & evaluators?\nGuardrails are\ninline safety checks\nthat sit directly in the request/response path. They validate inputs or outputs\nbefore\nanything reaches a user, so they typically are:\nFast and deterministic\n– typically a few milliseconds of latency budget.\nSimple and explainable\n– regexes, keyword block-lists, schema or type validators, lightweight classifiers.\nTargeted at clear-cut, high-impact failures\n– PII leaks, profanity, disallowed instructions, SQL injection, malformed JSON, invalid code syntax, etc.\nIf a guardrail triggers, the system can redact, refuse, or regenerate the response. Because these checks are user-visible when they fire, false positives are treated as production bugs; teams version guardrail rules, log every trigger, and monitor rates to keep them conservative.\nOn the other hand, evaluators typically run\nafter\na response is produced. Evaluators measure qualities that simple rules cannot, such as factual correctness, completeness, etc. Their verdicts feed dashboards, regression tests, and model-improvement loops, but they do not block the original answer.\nEvaluators are usually run asynchronously or in batch to afford heavier computation such as a\nLLM-as-a-Judge\n. Inline use of an LLM-as-Judge is possible\nonly\nwhen the latency budget and reliability targets allow it. Slow LLM judges might be feasible in a cascade that runs on the minority of borderline cases.\nApply guardrails for immediate protection against objective failures requiring intervention. Use evaluators for monitoring and improving subjective or nuanced criteria. Together, they create layered protection.\nWord of caution: Do not use llm guardrails off the shelf blindly. Always\nlook at the prompt\n.\n↗ Focus view\nQ: Can my evaluators also be used to automatically\nfix\nor\ncorrect\noutputs in production?\nYes, but only a specific subset of them. This is the distinction between an\nevaluator\nand a\nguardrail\nthat we\npreviously discussed\n. As a reminder:\nEvaluators\ntypically run\nasynchronously\nafter a response has been generated. They measure quality but don’t interfere with the user’s immediate experience.\nGuardrails\nrun\nsynchronously\nin the critical path of the request, before the output is shown to the user. Their job is to prevent high-impact failures in real-time.\nThere are two important decision criteria for deciding whether to use an evaluator as a guardrail:\nLatency & Cost\n: Can the evaluator run fast enough and cheaply enough in the critical request path without degrading user experience?\nError Rate Trade-offs\n: What’s the cost-benefit balance between false positives (blocking good outputs and frustrating users) versus false negatives (letting bad outputs reach users and causing harm)? In high-stakes domains like medical advice, false negatives may be more costly than false positives. In creative applications, false positives that block legitimate creativity may be more harmful than occasional quality issues.\nMost guardrails are designed to be\nfast\n(to avoid harming user experience) and have a\nvery low false positive rate\n(to avoid blocking valid responses). For this reason, you would almost never use a slow or non-deterministic LLM-as-Judge as a synchronous guardrail. However, these tradeoffs might be different for your use case.\n↗ Focus view\nQ: How much time should I spend on model selection?\nMany developers fixate on model selection as the primary way to improve their LLM applications. Start with error analysis to understand your failure modes before considering model switching. As Hamel noted in office hours, “I suggest not thinking of switching model as the main axes of how to improve your system off the bat without evidence. Does error analysis suggest that your model is the problem?”\n↗ Focus view\nDomain-Specific Applications\nQ: Is RAG dead?\nQuestion: Should I avoid using RAG for my AI application after reading that\n“RAG is dead”\nfor coding agents?\nMany developers are confused about when and how to use RAG after reading articles claiming “RAG is dead.” Understanding what RAG actually means versus the narrow marketing definitions will help you make better architectural decisions for your AI applications.\nThe viral article claiming RAG is dead specifically argues against using\nnaive vector database retrieval\nfor autonomous coding agents, not RAG as a whole. This is a crucial distinction that many developers miss due to misleading marketing.\nRAG simply means Retrieval-Augmented Generation - using retrieval to provide relevant context that improves your model’s output. The core principle remains essential: your LLM needs the right context to generate accurate answers. The question isn’t whether to use retrieval, but how to retrieve effectively.\nFor coding applications, naive vector similarity search often fails because code relationships are complex and contextual. Instead of abandoning retrieval entirely, modern coding assistants like Claude Code\nstill uses retrieval\n—they just employ agentic search instead of relying solely on vector databases, similar to how human developers work.\nYou have multiple retrieval strategies available, ranging from simple keyword matching to embedding similarity to LLM-powered relevance filtering. The optimal approach depends on your specific use case, data characteristics, and performance requirements. Many production systems combine multiple strategies or use multi-hop retrieval guided by LLM agents.\nUnfortunately, “RAG” has become a buzzword with no shared definition. Some people use it to mean any retrieval system, others restrict it to vector databases. Focus on the ultimate goal: getting your LLM the context it needs to succeed. Whether that’s through vector search, agentic exploration, or hybrid approaches is a product and engineering decision.\nRather than following categorical advice to avoid or embrace RAG, experiment with different retrieval approaches and measure what works best for your application. For more info on RAG evaluation and optimization, see\nthis series of posts\n.\n↗ Focus view\nQ: How should I evaluate a coding agent?\nIf your coding agent handles a wide variety of tasks, start by using public benchmarks much as you would a foundation model. For an agent that handles a narrow workflow, product-specific evals are a better fit. The\nevals FAQ\nexplains this distinction.\nPopular coding benchmarks include\nSWE-bench\n,\nTerminal-Bench\n,\nAider Polyglot\n, and\nHumanEval\n.\nIn addition to public benchmarks, you can also build a private benchmark of difficult tasks from your organization. OpenAI described using\nreal internal software engineering tasks to evaluate Codex\nat launch. Each task needs a working environment and code-based tests that establish whether the agent completed it successfully.\nTo decide which tasks to include, look at how people use your agent and where it fails. Review runs with engineers, group recurring problems, and turn useful examples into tests. This is\nerror analysis\n, and it applies to coding products too. If existing tests already identify failures, use those results to choose runs to investigate.\nAnthropic’s\nClio research\nillustrates a related approach that clusters chat conversations by topic. You can apply that idea to coding sessions to identify the kinds of work your benchmark should cover.\nAnthropic’s\ncoding-agent eval guidance\nrecommends starting with clearly specified tasks and a stable environment where unit tests can verify results. After you have these unit tests, they recommend adding checks for things those tests don’t capture, such as code quality or how the agent interacts with users. Claude Code’s team, for example, added evals for file edits and later for over-engineering. There are many approaches to measure file edits and over-engineering but you can start with metrics like net new lines of code added and\ncyclomatic complexity\n.\nJohn Berryman and Shawn Simister’s Copilot talk\nprovides additional examples of coding-agent evals. For code completions, the team removed function implementations from repositories, had the model regenerate them, and ran the existing tests. For chat, they used LLM judges with specific criteria and separate checks for whether the assistant called the right tool. They also ran A/B tests, tracking whether users accepted suggestions and kept the code afterward. These product metrics complemented the offline evals.\n↗ Focus view\nQ: How should I approach evaluating my RAG system?\nRAG systems have two distinct components that require different evaluation approaches: retrieval and generation.\nStart with retrieval evaluation\nThe retrieval component is a search problem. Evaluate it using traditional information retrieval (IR) metrics. Common examples include Recall@k (of all relevant documents, how many did you retrieve in the top k?), Precision@k (of the k documents retrieved, how many were relevant?), or MRR (how high up was the first relevant document?). The specific metrics you choose depend on your use case. These metrics are pure search metrics that measure whether you’re finding the right documents (more on this below).\nTo evaluate retrieval, create a dataset of queries paired with their relevant documents. Generate this synthetically by taking documents from your corpus, extracting key facts, then generating questions those facts would answer. This reverse process gives you query-document pairs for measuring retrieval performance without manual annotation.\nNext, evaluate generation\nFor the generation component, check how well the LLM uses the retrieved context and whether it answers the question. Use\nerror analysis\nto identify failure modes, collect human labels, build targeted LLM judges, and validate those judges against human annotations.\nJason Liu’s\n“There Are Only 6 RAG Evals”\nprovides a framework that maps well to this separation. His Tier 1 covers traditional IR metrics for retrieval. Tiers 2 and 3 evaluate relationships between Question, Context, and Answer. These include whether the context is relevant (C|Q), whether the answer is faithful to context (A|C), and whether the answer addresses the question (A|Q).\nIn addition to Jason’s six evals, error analysis on your specific data may reveal domain-specific failure modes that warrant their own metrics. For example, a medical RAG system might consistently fail to distinguish between drug dosages for adults versus children, or a legal RAG might confuse jurisdictional boundaries. These patterns emerge only through systematic review of actual failures. Once identified, you can create targeted evaluators for these specific issues beyond the general framework.\nFinally, when implementing Jason’s Tier 2 and 3 metrics, don’t just use prompts off the shelf. The standard LLM-as-judge process requires several steps: error analysis, prompt iteration, creating labeled examples, and measuring your judge’s accuracy against human labels. Once you know your judge’s True Positive and True Negative rates, you can correct its estimates to determine the actual failure rate in your system. Skip this validation and your judges may not reflect your actual quality criteria.\nIn summary, debug retrieval first using IR metrics, then tackle generation quality using properly validated LLM judges.\n↗ Focus view\nQ: How do I choose the right chunk size for my document processing tasks?\nUnlike RAG, where chunks are optimized for retrieval, document processing assumes the model will see every chunk. The goal is to split text so the model can reason effectively without being overwhelmed. Even if a document fits within the context window, it might be better to break it up. Long inputs can degrade performance due to attention bottlenecks, especially in the middle of the context. Two task types require different strategies:\n1. Fixed-Output Tasks → Large Chunks\nThese are tasks where the output length doesn’t grow with input: extracting a number, answering a specific question, classifying a section. For example:\n“What’s the penalty clause in this contract?”\n“What was the CEO’s salary in 2023?”\nUse the largest chunk (with caveats) that likely contains the answer. This reduces the number of queries and avoids context fragmentation. However, avoid adding irrelevant text. Models are sensitive to distraction, especially with large inputs. The middle parts of a long input might be under-attended. Furthermore, if cost and latency are a bottleneck, you should consider preprocessing or filtering the document (via keyword search or a lightweight retriever) to isolate relevant sections before feeding a huge chunk.\n2. Expansive-Output Tasks → Smaller Chunks\nThese include summarization, exhaustive extraction, or any task where output grows with input. For example:\n“Summarize each section”\n“List all customer complaints”\nIn these cases, smaller chunks help preserve reasoning quality and output completeness. The standard approach is to process each chunk independently, then aggregate results (e.g., map-reduce). When sizing your chunks, try to respect content boundaries like paragraphs, sections, or chapters. Chunking also helps mitigate output limits. By breaking the task into pieces, each piece’s output can stay within limits.\nGeneral Guidance\nIt’s important to recognize\nwhy chunk size affects results\n. A larger chunk means the model has to reason over more information in one go – essentially, a heavier cognitive load. LLMs have limited capacity to\nretain and correlate details across a long text\n. If too much is packed in, the model might prioritize certain parts (commonly the beginning or end) and overlook or “forget” details in the middle. This can lead to overly coarse summaries or missed facts. In contrast, a smaller chunk bounds the problem: the model can pay full attention to that section. You are trading off\nglobal context for local focus\n.\nNo rule of thumb can perfectly determine the best chunk size for your use case –\nyou should validate with experiments\n. The optimal chunk size can vary by domain and model. I treat chunk size as a hyperparameter to tune.\n↗ Focus view\nQ: How do I debug multi-turn conversation traces?\nStart simple. Check if the whole conversation met the user’s goal with a pass/fail judgment. Look at the entire trace and focus on the first upstream failure. Read the user-visible parts first to understand if something went wrong. Only then dig into the technical details like tool calls and intermediate steps.\nMulti-agent trace logging\nFor multi-agent flows, assign a session or trace ID to each user request and log every message with its source (which agent or tool), trace ID, and position in the sequence. This lets you reconstruct the full path from initial query to final result across all agents.\nAnnotation strategy\nAnnotate only the first failure in the trace at first. Downstream failures often cascade from the first issue, so fixing the upstream failure can resolve the dependent ones. As you gain experience, you can annotate independent failure modes within the same trace to speed up error analysis.\nSimplify when possible\nWhen you find a failure, reproduce it with the simplest possible test case. Here’s an example: suppose a shopping bot gives the wrong return policy on turn 4 of a conversation. Before diving into the full multi-turn complexity, simplify it to a single turn: “What is the return window for product X1000?” If it still fails, you’ve proven the error isn’t about conversation context - it’s likely a basic retrieval or knowledge issue you can debug more easily.\nTest case generation\nYou have two main approaches. First, simulate users with another LLM to create realistic multi-turn conversations. Second, use “N-1 testing” where you provide the first N-1 turns of a real conversation and test what happens next. The N-1 approach often works better since it uses actual conversation prefixes rather than fully synthetic interactions, but is less flexible.\nThe key is balancing thoroughness with efficiency. Not every multi-turn failure requires multi-turn analysis.\nWhen the conversation includes tools or several agents, use a\ntransition failure matrix\nto find hotspots of errors.\n↗ Focus view\nQ: How do I evaluate sessions with human handoffs?\nCapture the complete user journey in your traces, including human handoffs. The trace continues until the user’s need is resolved or the session ends, not when AI hands off to a human. Log the handoff decision, why it occurred, context transferred, wait time, human actions, final resolution, and whether the human had sufficient context. Many failures occur at handoff boundaries where AI hands off too early, too late, or without proper context.\nEvaluate handoffs as potential failure modes during\nerror analysis\n. Ask: Was the handoff necessary? Did the AI provide adequate context? Track both handoff quality and handoff rate. Sometimes the best improvement reduces handoffs entirely rather than improving handoff execution.\n↗ Focus view\nQ: How do I evaluate complex multi-step workflows?\nLog the entire workflow from initial trigger to final business outcome. Include LLM calls, tool usage, human approvals, and database writes in your traces. You will need this visibility to properly diagnose failures.\nUse both outcome and process metrics. Outcome metrics verify the final result meets requirements: Was the business case complete? Accurate? Properly formatted? Process metrics evaluate efficiency: step count, time taken, resource usage. Process failures are often easier to debug since they’re more deterministic, so tackle them first.\nSegment your\nerror analysis\nby workflow stages. Early stage failures (understanding user input) differ from middle stage failures (data processing) and late stage failures (formatting output). Early stage improvements have more impact since errors cascade in LLM chains.\nUse\ntransition failure matrices\nto analyze where workflows break. Create a matrix showing the last successful state versus where the first failure occurred. This reveals failure hotspots and guides where to invest debugging effort.\n↗ Focus view\nQ: How do I evaluate agentic workflows?\nWe recommend evaluating agentic workflows in two phases:\n1. End-to-end task success.\nTreat the agent as a black box and decide whether it met the user’s goal. Define a precise success rule per task and measure it with human review or\nvalidated LLM judges\n. Record the first upstream failure during\nerror analysis\n.\nOnce error analysis reveals which workflows fail most often, move to step-level diagnostics to understand why they’re failing.\n2. Step-level diagnostics.\nAfter you\nlog the system’s traces\n, you can score individual components such as:\nTool choice\n: check whether the agent selected the appropriate tool.\nParameter extraction\n: check whether the inputs were complete and well-formed.\nError handling\n: check how the agent handled empty results or API failures.\nContext retention\n: check whether the agent preserved earlier constraints.\nEfficiency\n: count the steps, seconds, and tokens spent.\nGoal checkpoints\n: verify key milestones in long workflows.\nHow do I test tool calls?\nTest the tool name, arguments, result, and resulting state as separate checks. Use code assertions when the expected behavior is objective. For example, verify that the agent selected\ncancel_order\n, passed the correct order ID, received a successful response, and changed the order status before it told the user that cancellation succeeded.\nAlso test authorization and preconditions. A valid tool call can still be wrong if the user did not approve the action or the system skipped a required check.\nExample: “Find Berkeley homes under $1M and schedule viewings” breaks into: parameters extracted correctly, relevant listings retrieved, availability checked, and calendar invites sent. Each checkpoint can pass or fail independently, making debugging tractable.\nUse transition failure matrices to understand error patterns.\nCreate a matrix where rows represent the last successful state and columns represent where the first failure occurred. This is a great way to understand where the most failures occur.\nTransition failure matrix showing hotspots in text-to-SQL agent workflow\nTransition matrices show where failures cluster. In this example, GenSQL → ExecSQL transitions cause 12 failures while DecideTool → PlanCal causes only 2. The counts show where to investigate first. Here is another\ntext-to-SQL example\nfrom Bryan Bischof:\nBischof, Bryan “Failure is A Funnel - Data Council, 2025”\nIn this example, Bryan shows variation in transition matrices across experiments. How you organize your transition matrix depends on the specifics of your application. For example, Bryan’s text-to-SQL agent has an inherent sequential workflow which he exploits for further analytical insight. You can watch his\nfull talk\nfor more details.\nWatch “Stop Managing AI Projects Like Traditional Software” on YouTube\nCreating Test Cases for Agent Failures\nCreating test cases for agent failures follows the same principles as our previous FAQ on\ndebugging multi-turn conversation traces\n. Reproduce the error with the simplest test that still fails. Use a multi-turn test only when the failure depends on conversation context.\n↗ Focus view\n👉\nWant to learn more about AI Evals? Check out our\nAI Evals course\n. It’s a live cohort with hands on exercises and office hours. Here is a\n25% discount code\nfor readers. 👈\nFootnotes\nPaul Graham,\n“Writes and Write-Nots”\n↩︎\nShreya Shankar, et al.,\n“Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences”\n↩︎",
  "metadata": {
    "title": "AI Evals: Everything You Need to Know – Hamel’s Blog",
    "description": "A guide to AI and LLM evals for engineers and product managers. Learn how to test AI systems, analyze failures, and improve retrieval, agents, and AI products.",
    "keywords": null,
    "author": "Hamel Husain",
    "og:type": "article",
    "og:title": "AI Evals: Everything You Need to Know – Hamel’s Blog",
    "og:description": "A guide to AI and LLM evals for engineers and product managers. Learn how to test AI systems, analyze failures, and improve retrieval, agents, and AI products.",
    "og:image": "https://hamel.dev/blog/posts/evals-faq/images/eval_faq-social.jpg",
    "og:site_name": "Hamel’s Blog",
    "og:locale": "en_US",
    "twitter:title": "AI Evals: Everything You Need to Know – Hamel’s Blog",
    "twitter:description": "A guide to AI and LLM evals for engineers and product managers. Learn how to test AI systems, analyze failures, and improve retrieval, agents, and AI products.",
    "twitter:image": "https://hamel.dev/blog/posts/evals-faq/images/eval_faq-social.jpg",
    "twitter:creator": "@HamelHusain",
    "twitter:site": "@HamelHusain",
    "twitter:card": "summary_large_image"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals/images/diagram-cover.webp",
      "bytes": 87406,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-94e7e228a8ac9be53f82.webp",
      "src": "https://hamel.dev/blog/posts/evals/images/diagram-cover.webp",
      "alt": "Your AI Product Needs Evals cover",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/llm-judge/images/cover_img.webp",
      "bytes": 62194,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-0c9efff14a935419f95c.webp",
      "src": "https://hamel.dev/blog/posts/llm-judge/images/cover_img.webp",
      "alt": "Using LLM-as-a-Judge For Evaluation cover",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/field-guide/images/field_guide_2.webp",
      "bytes": 132266,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-1825dfcc7d2d9925beee.webp",
      "src": "https://hamel.dev/blog/posts/field-guide/images/field_guide_2.webp",
      "alt": "A Field Guide to Rapidly Improving AI Products cover",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/alex.webp",
      "bytes": 54874,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-df7f4a4fab775c847d03.webp",
      "src": "https://hamel.dev/blog/posts/evals-faq/alex.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://i.ytimg.com/vi/aqKUwPKBkB0/hqdefault.jpg",
      "bytes": 29896,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-408c8a1801ed0548d42c.jpg",
      "src": "https://i.ytimg.com/vi/aqKUwPKBkB0/hqdefault.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/pawel-error-analysis.webp",
      "bytes": 265896,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-8d220934337841b6f07c.webp",
      "src": "https://hamel.dev/blog/posts/evals-faq/pawel-error-analysis.webp",
      "alt": "Infographic showing an error-analysis workflow: review traces, refine failure modes, recode examples, then build application-specific evaluators",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://i.ytimg.com/vi/e2i6JbU2R-s/hqdefault.jpg",
      "bytes": 32570,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-4fdea6144c415e11f1a2.jpg",
      "src": "https://i.ytimg.com/vi/e2i6JbU2R-s/hqdefault.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://i.ytimg.com/vi/tqUDjc1HzO4/hqdefault.jpg",
      "bytes": 17083,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-ac11fd22e2e11ea4ceda.jpg",
      "src": "https://i.ytimg.com/vi/tqUDjc1HzO4/hqdefault.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/how-to-trust-a-classifier-for-evals.png",
      "bytes": 1893087,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-1e31f769fd8b02887fca.png",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/how-to-trust-a-classifier-for-evals.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/notes/llm/evals/flashcards/7-how-to-trust-a-llm-judge.png",
      "bytes": 859172,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-5c77b84d3f1788bf129b.png",
      "src": "https://hamel.dev/notes/llm/evals/flashcards/7-how-to-trust-a-llm-judge.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/eval-smell/_static-imgs/09-workers-comp-after.png",
      "bytes": 194355,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-9f5de7f1285920ebef27.png",
      "src": "https://hamel.dev/blog/posts/eval-smell/_static-imgs/09-workers-comp-after.png",
      "alt": "ClaimDraft review interface with source-linked findings, controls to resolve contradictions, and an option to add notes before generating a report.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://i.ytimg.com/vi/fA4pe9bE0LY/hqdefault.jpg",
      "bytes": 40497,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-61e5241dd6234a24c9da.jpg",
      "src": "https://i.ytimg.com/vi/fA4pe9bE0LY/hqdefault.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/emailinterface1.webp",
      "bytes": 207698,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-c9f6f07b0ff4f9640c57.webp",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/emailinterface1.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/hotkey.webp",
      "bytes": 59748,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-49f5ecd027ec060226ee.webp",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/hotkey.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/group1.webp",
      "bytes": 124530,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-2d6cf56a0938e3059cda.webp",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/group1.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/ci.webp",
      "bytes": 96762,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-002513619dfed03a6823.webp",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/ci.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://i.ytimg.com/vi/N-qAOv_PNPc/hqdefault.jpg",
      "bytes": 16471,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-19482332a302afa33c40.jpg",
      "src": "https://i.ytimg.com/vi/N-qAOv_PNPc/hqdefault.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/how-often-light.png",
      "bytes": 378208,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-edc08795c50982f5cd14.png",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/how-often-light.png",
      "alt": "How often to run an eval based on its cost, how many test examples pass, and the business value of catching the error.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/how-often-dark.png",
      "bytes": 375732,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-62431685aa37bfa525a7.png",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/how-often-dark.png",
      "alt": "How often to run an eval based on its cost, how many test examples pass, and the business value of catching the error.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/shreya_matrix.webp",
      "bytes": 24654,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-007eb9f72184e114b8cb.webp",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/shreya_matrix.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-faq/images/bischof_matrix.webp",
      "bytes": 156792,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-a62817b8df3d381f36e1.webp",
      "src": "https://hamel.dev/blog/posts/evals-faq/images/bischof_matrix.webp",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://i.ytimg.com/vi/R_HnI9oTv3c/hqdefault.jpg",
      "bytes": 31018,
      "reused": false,
      "local_path": "attachments/5bf201eed3d0/images/body-bf89d1cf5d9ca9cc3f68.jpg",
      "src": "https://i.ytimg.com/vi/R_HnI9oTv3c/hqdefault.jpg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=aqKUwPKBkB0",
      "status": "ok",
      "local_path": "attachments/5bf201eed3d0/videos/aqKUwPKBkB0.mp4",
      "bytes": 374601155,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "4424.961000"
    },
    {
      "url": "https://www.youtube.com/watch?v=qH1dZ8JLLdU",
      "src": "https://www.youtube.com/watch?v=qH1dZ8JLLdU",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/5bf201eed3d0/videos/repair-qH1dZ8JLLdU.mp4",
      "bytes": 97933248,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "2495.481000",
      "reused": true,
      "reused_from": "xinxing-zimeiti-corpus/attachments/0d7bdbd9fe1b/videos/repair-qH1dZ8JLLdU.mp4"
    },
    {
      "url": "https://www.youtube.com/watch?v=4dUFIRj-BWo",
      "src": "https://m.youtube.com/watch?si=qknrtQeITqJ7VsJH&v=4dUFIRj-BWo&feature=youtu.be",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/5bf201eed3d0/videos/repair-4dUFIRj-BWo.mp4",
      "bytes": 491202971,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "2339.241000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=EEw2PpL-_NM",
      "src": "https://www.youtube.com/watch?v=EEw2PpL-_NM",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/5bf201eed3d0/videos/repair-EEw2PpL-_NM.mp4",
      "bytes": 210751900,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "905.021000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=e2i6JbU2R-s",
      "src": "https://www.youtube.com/watch?v=e2i6JbU2R-s",
      "status": "no_1080_or_720",
      "kind": "single_video",
      "error": "ERROR: [youtube] e2i6JbU2R-s: Requested format is not available. Use --list-formats for a list of available formats\n",
      "title": "",
      "retried_at": "2026-10-20T17:55:56Z"
    },
    {
      "url": "https://www.youtube.com/watch?v=BsWxPI9UM4c",
      "src": "https://www.youtube.com/watch?v=BsWxPI9UM4c",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/5bf201eed3d0/videos/repair-BsWxPI9UM4c.mp4",
      "bytes": 460841429,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "6392.501000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=tqUDjc1HzO4",
      "src": "https://youtu.be/tqUDjc1HzO4",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/5bf201eed3d0/videos/repair-tqUDjc1HzO4.mp4",
      "bytes": 112622596,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1638.581000",
      "reused": true,
      "reused_from": "xinxing-zimeiti-corpus/attachments/f8a265500588/videos/repair-tqUDjc1HzO4.mp4"
    },
    {
      "url": "https://www.youtube.com/watch?v=fA4pe9bE0LY",
      "src": "https://youtu.be/fA4pe9bE0LY",
      "status": "no_1080_or_720",
      "kind": "single_video",
      "error": "ERROR: [youtube] fA4pe9bE0LY: Requested format is not available. Use --list-formats for a list of available formats\n",
      "title": "",
      "retried_at": "2026-10-20T17:56:02Z"
    },
    {
      "url": "https://www.youtube.com/watch?v=N-qAOv_PNPc",
      "src": "https://www.youtube.com/watch?v=N-qAOv_PNPc",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/5bf201eed3d0/videos/repair-N-qAOv_PNPc.mp4",
      "bytes": 171275008,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "4220.861000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=LwLxlEwrtRA",
      "src": "https://www.youtube.com/watch?v=LwLxlEwrtRA&t=534s",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/5bf201eed3d0/videos/repair-LwLxlEwrtRA.mp4",
      "bytes": 161281412,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "3505.781000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=R_HnI9oTv3c",
      "src": "https://youtu.be/R_HnI9oTv3c?si=hRRhDiydHU5k6ikc",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/5bf201eed3d0/videos/repair-R_HnI9oTv3c.mp4",
      "bytes": 89949164,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "3372.841000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://www.googletagmanager.com/ns.html?id=GTM-PKGWQMKL",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://maven.com/parlance-labs/evals?promoCode=evals-info-book",
      "label": "teaching",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/start/new-to-evals/index.html",
      "label": "I’m new to evals",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/start/getting-started/index.html",
      "label": "I don’t know what to test",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/start/trust-your-evals/index.html",
      "label": "I don’t trust my eval scores",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/start/hard-to-evaluate/index.html",
      "label": "My product feels too hard to evaluate",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/start/reduce-eval-cost/index.html",
      "label": "Evals take too much time or money",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/index.html",
      "label": "in this series",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/index.html",
      "label": "AI Evals FAQ",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals/index.html",
      "label": "rerunning the evals",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/llm-judge/index.html",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/field-guide/index.html",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-are-llm-evals.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://mlops.systems/posts/2025-06-04-instrumenting-an-agentic-app-with-arize-phoenix-and-litellm.html",
      "label": "Alex Strick van Linschoten’s analysis",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-is-a-trace.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://youtu.be/aqKUwPKBkB0?si=5KDmMQnRzO_Ce9xH",
      "label": "video",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=aqKUwPKBkB0",
      "label": "Watch “Build Your Own Eval Tools With Notebooks!” on YouTube",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/whats-a-minimum-viable-evaluation-setup.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=qH1dZ8JLLdU",
      "label": "error analysis",
      "type": "a"
    },
    {
      "href": "https://ai-execs.com/2_intro.html",
      "label": "wary of optimizing for high eval pass rates",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-much-of-my-development-budget-should-i-allocate-to-evals.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2404.12272",
      "label": "research shows",
      "type": "a"
    },
    {
      "href": "https://m.youtube.com/watch?si=qknrtQeITqJ7VsJH&v=4dUFIRj-BWo&feature=youtu.be",
      "label": "“The model is the product”",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=EEw2PpL-_NM",
      "label": "“The model is NOT the product”",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/will-these-evaluation-methods-still-be-relevant-in-5-10-years-given-how-fast-ai-is-changing.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/field-guide/",
      "label": "iterating on experiments",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-make-the-case-for-investing-in-evaluations-to-my-team.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/llm-judge/",
      "label": "domain expert",
      "type": "a"
    },
    {
      "href": "https://delvetool.com/blog/theoreticalsaturation",
      "label": "theoretical saturation",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-can-i-efficiently-sample-production-traces-for-review.html",
      "label": "sample data more efficiently",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=e2i6JbU2R-s",
      "label": "this video",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/pawel-huryn/",
      "label": "Pawel Huryn",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=BsWxPI9UM4c",
      "label": "live demo",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-often-should-i-re-run-error-analysis-on-my-production-system.html",
      "label": "periodically",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/do-i-need-a-reference-answer-before-annotating-data.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/should-i-build-automated-evaluators-for-every-failure-mode-i-find.html",
      "label": "don’t need",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/should-i-record-problems-that-arent-the-models-fault.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://github.com/ai-evals-course/evals-skills",
      "label": "evals plugin",
      "type": "a"
    },
    {
      "href": "https://youtu.be/tqUDjc1HzO4",
      "label": "live walkthrough",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-many-people-should-annotate-my-llm-outputs.html",
      "label": "domain expert",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/flashcards/7-how-to-trust-a-llm-judge.png",
      "label": "judge-validation flashcard",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-are-evaluations-used-differently-in-cicd-vs-monitoring-production.html",
      "label": "CI evals",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-many-examples-do-i-need-for-an-eval.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-surface-problematic-traces-for-review-beyond-user-feedback.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-should-i-do-when-my-gold-eval-dataset-becomes-stale.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/flashcards/9-synthetic-data.png",
      "label": "visual",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-is-the-best-approach-for-generating-synthetic-data.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/are-there-scenarios-where-synthetic-data-may-not-be-reliable.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/eval-smell/index.html",
      "label": "easier for them to verify",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/eval-smell/",
      "label": "this post",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-can-i-do-error-analysis-when-production-traces-contain-sensitive-data.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://youtu.be/e2i6JbU2R-s?si=8p5XVxbBiioz69Xc",
      "label": "Error Analysis",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-approach-evaluation-when-my-system-handles-diverse-user-queries.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/flashcards/8-sample-traces.png",
      "label": "This flashcard",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=tqUDjc1HzO4",
      "label": "Watch “How to Automate AI Evals (Correctly)” on YouTube",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/why-do-you-recommend-binary-passfail-evaluations-instead-of-1-5-ratings-likert-scales.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-combine-my-evals-into-a-single-metric.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/should-i-practice-eval-driven-development.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-know-if-i-can-trust-my-automated-eval.html",
      "label": "validate it",
      "type": "a"
    },
    {
      "href": "https://typesafe.ai/",
      "label": "TypeSafe",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/can-i-use-the-same-model-for-both-the-main-task-and-evaluation.html",
      "label": "use the same model as your application",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-model-or-llm-should-i-use-to-build-automated-evals.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2507.19457",
      "label": "GEPA",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/can-i-use-jev-for-evals.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-much-of-a-trace-should-i-give-an-llm-judge.html",
      "label": "missing context",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-should-i-do-when-i-cant-get-my-llm-judge-to-agree-with-human-reviewers.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/should-i-use-ready-to-use-evaluation-metrics.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/are-similarity-metrics-bertscore-rouge-etc-useful-for-evaluating-llm-outputs.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2508.06709",
      "label": "research has shown",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/rag/p6-context_rot.html",
      "label": "context rot",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/search/?query=Abstention+Ability&searchtype=all",
      "label": "searching for this term on Arxiv",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-we-evaluate-a-models-ability-to-express-uncertainty-or-know-what-it-doesnt-know.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-makes-a-good-custom-interface-for-reviewing-llm-outputs.html",
      "label": "building a review interface",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-if-human-reviewers-approve-ai-outputs-without-checking-them-carefully.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/should-product-managers-and-engineers-collaborate-on-error-analysis-how.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/can-i-help-with-evals-if-im-not-a-domain-expert.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-evaluate-outputs-in-a-language-i-dont-speak.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://www.ankihub.net/",
      "label": "AnkiHub",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/should-i-outsource-annotation-and-labeling-to-a-third-party.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-debug-multi-turn-conversation-traces.html",
      "label": "first upstream failure",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-if-the-source-material-is-too-large-for-a-person-to-review.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-parts-of-evals-can-be-automated-with-llms.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/should-i-stop-writing-prompts-manually-in-favor-of-automated-tools.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://youtu.be/fA4pe9bE0LY",
      "label": "Isaac’s Anki flashcard annotation app",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=fA4pe9bE0LY",
      "label": "Watch “Building Eval Tools with FastHTML” on YouTube",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/should-i-build-a-custom-annotation-tool-or-use-something-off-the-shelf.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://julius.ai/",
      "label": "Julius",
      "type": "a"
    },
    {
      "href": "https://hex.tech/",
      "label": "Hex",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-gaps-in-eval-tooling-should-i-be-prepared-to-fill-myself.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-should-an-internal-eval-platform-standardize-across-teams.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://www.langchain.com/langsmith",
      "label": "Langsmith",
      "type": "a"
    },
    {
      "href": "https://arize.com/",
      "label": "Arize",
      "type": "a"
    },
    {
      "href": "https://www.braintrust.dev/",
      "label": "Braintrust",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/eval-tools/",
      "label": "video series",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/whats-your-favorite-eval-vendor.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://github.com/",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://desktop.github.com/",
      "label": "Desktop app",
      "type": "a"
    },
    {
      "href": "https://ben.balter.com/2023/03/02/github-for-non-technical-roles/",
      "label": "blog post",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=N-qAOv_PNPc",
      "label": "Watch “From Noob to Automated Evals In A Week (as a PM) w/Teresa Torres” on YouTube",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-should-i-version-and-manage-prompts.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/what-should-go-in-the-system-prompt-vs-the-user-prompt.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/evals/flashcards/12-deploy-evals.png",
      "label": "Here is a visual",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-often-should-i-run-my-evals.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/prompt/",
      "label": "look at the prompt",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/whats-the-difference-between-guardrails-evaluators.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/can-my-evaluators-also-be-used-to-automatically-fix-or-correct-outputs-in-production.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-much-time-should-i-spend-on-model-selection.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://pashpashpash.substack.com/p/why-i-no-longer-recommend-rag-for",
      "label": "“RAG is dead”",
      "type": "a"
    },
    {
      "href": "https://x.com/pashmerepat/status/1926717705660375463?s=46",
      "label": "still uses retrieval",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/notes/llm/rag/not_dead.html",
      "label": "this series of posts",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/is-rag-dead.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://www.swebench.com/",
      "label": "SWE-bench",
      "type": "a"
    },
    {
      "href": "https://www.tbench.ai/",
      "label": "Terminal-Bench",
      "type": "a"
    },
    {
      "href": "https://aider.chat/docs/leaderboards/",
      "label": "Aider Polyglot",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/human-eval",
      "label": "HumanEval",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/introducing-codex/",
      "label": "real internal software engineering tasks to evaluate Codex",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/research/clio",
      "label": "Clio research",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents",
      "label": "coding-agent eval guidance",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Cyclomatic_complexity",
      "label": "cyclomatic complexity",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=LwLxlEwrtRA&t=534s",
      "label": "John Berryman and Shawn Simister’s Copilot talk",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-should-i-evaluate-a-coding-agent.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/",
      "label": "“There Are Only 6 RAG Evals”",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-should-i-approach-evaluating-my-rag-system.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-choose-the-right-chunk-size-for-my-document-processing-tasks.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-evaluate-sessions-with-human-handoffs.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-evaluate-complex-multi-step-workflows.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals/",
      "label": "log the system’s traces",
      "type": "a"
    },
    {
      "href": "https://www.figma.com/deck/nwRlh5renu4s4olaCsf9lG/Failure-is-a-Funnel?node-id=2009-927&t=GJlTtxQ8bLJaQ92A-1",
      "label": "text-to-SQL example",
      "type": "a"
    },
    {
      "href": "https://youtu.be/R_HnI9oTv3c?si=hRRhDiydHU5k6ikc",
      "label": "full talk",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=R_HnI9oTv3c",
      "label": "Watch “Stop Managing AI Projects Like Traditional Software” on YouTube",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/how-do-i-evaluate-agentic-workflows.html",
      "label": "↗ Focus view",
      "type": "a"
    },
    {
      "href": "https://paulgraham.com/writes.html",
      "label": "“Writes and Write-Nots”",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:41Z",
  "source_url": "https://hamel.dev/blog/posts/evals-faq/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2404.12272",
      "bytes": 2613163,
      "reused": true,
      "local_path": "papers/repair/1db96d99ad72b559bec0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2507.19457",
      "bytes": 2975112,
      "reused": false,
      "local_path": "papers/repair/83b9c56d2f95333c953a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2508.06709",
      "bytes": 1058702,
      "reused": false,
      "local_path": "papers/repair/ef2edba24f7b87679e26.pdf"
    }
  ],
  "actual_video_files": [
    {
      "local_path": "attachments/5bf201eed3d0/videos/aqKUwPKBkB0.mp4",
      "bytes": 374601155,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "4424.961000",
      "status": "ok"
    }
  ],
  "body_chars": 113536,
  "expected_body_images": 22,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "no_1080_or_720",
  "raw_html": "attachments/5bf201eed3d0/pages/source.html",
  "local_html": "attachments/5bf201eed3d0/pages/article.html",
  "body_base_path": "attachments/5bf201eed3d0/pages/",
  "body_markdown_path": "attachments/5bf201eed3d0/pages/article.md"
}