{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Hamel Husain / Parlance Labs",
  "title_listed": "Evals Skills for Coding Agents",
  "keywords": "评估,编码Agent,技能",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://hamel.dev/blog/posts/evals-skills/",
  "publish_date": "2026-03",
  "corpus_stage": "推理部署",
  "notes": "编码Agent的评估技能",
  "id": "f8a265500588",
  "collected_at": "2026-10-02T14:28:58",
  "title": "Evals Skills for Coding Agents – Hamel’s Blog",
  "body_markdown": "Evals Skills for Coding Agents\nAI\nEvals\nTeach your coding agent how to run product evals.\nAuthors\nHamel Husain\nShreya Shankar\nPublished\nMarch 2, 2026\nModified\nAugust 31, 2026\nToday, Shreya Shankar and I are publishing\nevals skills\n, a set of skills for AI product evals\n1\n.\nEval tools often get in the way. They nudge you toward\ngeneric off-the-shelf metrics\nand\nfully automated evals\nbefore you’ve looked at your data. These skills help you avoid common mistakes we’ve seen helping 50+ companies and teaching students in our\nAI Evals course\n.\nWhy skills for evals\nThere are many\neasily avoidable footguns\nin evals. These skills help you avoid them.\nevals-start\nis the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two:\neval-audit\n, if you already have an eval pipeline. It inspects your setup and recommends next steps.\n2\nerror-discovery\n, if you have traces but haven’t analyzed them yet. It builds a customized annotation interface and helps you sample traces intelligently. Shreya does a live walkthrough of using this skill\nhere\n.\nThe skills\nInstall the skills:\nnpx\nskills add https://github.com/ai-evals-course/evals-skills\nThen give your agent this prompt:\nRun the evals-start skill from the evals plugin and follow the skill it picks. If it picks eval-audit, investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report.\nIf you’re experienced with evals, skip the router and pick the skill you need:\nSkill\nWhat it does\nevals-start\nEntry point. Routes to the skill that matches your situation\neval-audit\nAudit an eval pipeline and surface problems with prioritized severity\nerror-discovery\nBuild a review app, select diverse samples, and organize your notes into failure modes\ngenerate-synthetic-data\nCreate diverse synthetic test inputs using dimension-based tuple generation\nwrite-judge-prompt\nDesign LLM-as-Judge evaluators for subjective quality criteria\nvalidate-evaluator\nCalibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction\nevaluate-rag\nEvaluate retrieval and generation quality in RAG pipelines\nbuild-review-interface\nBuild custom annotation interfaces for human trace review\nThese skills are only a starting point. To make them better, tune them to be more specific to your data and domain.\nThe repo is\nai-evals-course/evals-skills\n.\nYou can find me on\nX\nor email me through my\nnewsletter\n.\nFootnotes\nNot foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether\nyour\npipeline works on\nyour\ntask with\nyour\ndata. If you aren’t familiar with product-specific AI evals, check out my\nAI Evals FAQ\n.\n↩︎\nThe audit isn’t a complete solution, but it will catch common problems.\n↩︎",
  "metadata": {
    "title": "Evals Skills for Coding Agents – Hamel’s Blog",
    "description": "Teach your coding agent how to run product evals.",
    "keywords": null,
    "author": "Hamel Husain",
    "og:type": "article",
    "og:title": "Evals Skills for Coding Agents – Hamel’s Blog",
    "og:description": "Teach your coding agent how to run product evals.",
    "og:image": "https://hamel.dev/blog/posts/evals-skills/cover-gradient.png",
    "og:site_name": "Hamel’s Blog",
    "og:locale": "en_US",
    "og:image:height": "720",
    "og:image:width": "1280",
    "twitter:title": "Evals Skills for Coding Agents – Hamel’s Blog",
    "twitter:description": "Teach your coding agent how to run product evals.",
    "twitter:image": "https://hamel.dev/blog/posts/evals-skills/cover-gradient.png",
    "twitter:creator": "@HamelHusain",
    "twitter:site": "@HamelHusain",
    "twitter:card": "summary_large_image",
    "twitter:image-height": "720",
    "twitter:image-width": "1280"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/f8a265500588/code/body-000.txt",
      "chars": 62,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://hamel.dev/blog/posts/evals-skills/cover-original.png",
      "bytes": 85024,
      "reused": false,
      "local_path": "attachments/f8a265500588/images/body-a99f990d51b81f7fa2df.png",
      "src": "https://hamel.dev/blog/posts/evals-skills/cover-original.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=tqUDjc1HzO4",
      "src": "https://youtu.be/tqUDjc1HzO4",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/f8a265500588/videos/repair-tqUDjc1HzO4.mp4",
      "bytes": 112622596,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1638.581000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://www.googletagmanager.com/ns.html?id=GTM-PKGWQMKL",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://github.com/ai-evals-course/evals-skills",
      "label": "evals skills",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/evals-faq/index.html",
      "label": "generic off-the-shelf metrics",
      "type": "a"
    },
    {
      "href": "https://parlance-labs.com/blog/posts/auto-evals.html",
      "label": "fully automated evals",
      "type": "a"
    },
    {
      "href": "https://maven.com/parlance-labs/evals",
      "label": "AI Evals course",
      "type": "a"
    },
    {
      "href": "https://hamel.dev/blog/posts/revenge/index.html",
      "label": "easily avoidable footguns",
      "type": "a"
    },
    {
      "href": "https://youtu.be/tqUDjc1HzO4",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://x.com/hamelhusain",
      "label": "X",
      "type": "a"
    },
    {
      "href": "https://ai.hamel.dev/",
      "label": "newsletter",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:43Z",
  "source_url": "https://hamel.dev/blog/posts/evals-skills/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 2822,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/f8a265500588/pages/source.html",
  "local_html": "attachments/f8a265500588/pages/article.html",
  "body_base_path": "attachments/f8a265500588/pages/",
  "body_markdown_path": "attachments/f8a265500588/pages/article.md"
}