{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Jayakrishnan Vijayaraghavan / AI First Data Scientist",
  "title_listed": "The Four Tenets of Model Evaluation I Wish I Knew Sooner",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.aifirstdatascientist.com/articles/the-four-tenets-of-model-evaluation-i-wish-i-knew-sooner/",
  "publish_date": "2025-08",
  "corpus_stage": "",
  "notes": "",
  "id": "5cb70af6c5ba",
  "collected_at": "2026-10-02T12:25:30",
  "title": "The Four Tenets of Model Evaluation I Wish I Knew Sooner",
  "body_markdown": "Model Evaluation\nThe Four Tenets of Model Evaluation I Wish I Knew Sooner\nJK Vijayaraghavan\n22 Aug 2025\n—\n5 min read\nShare\nSome of my most rewarding work has been navigating the intricate nuances of model evaluation with Dave Spiegel, my mentor and manager at Google. We’ve spent non-trivial amount of hours debating the merits of different metrics, questioning assumptions, and trying to build a framework for evaluation that is both theoretically sound and pragmatically useful. It’s in those deep dives that you move past the textbook definitions and into the real art of data science.\nThis journey has led me to a set of core tenets for model evaluation, especially in today's world where ground truth can be an expensive luxury. I try to follow these religiously.\nMy Foundational Tenets for Model Evaluation\nThese are the principles I learned from Dave and my own research that now form my evaluation toolkit.\nSensitivity & Specificity Characterize the Model; Precision Characterizes Its Application.\nThis was a huge realization.\nSensitivity\n(true positive rate) and\nSpecificity\n(true negative rate) are intrinsic properties of your model at a given threshold. They answer questions like, \"Given a person is sick, how likely is the test to be positive?\" This doesn't change no matter how many sick people are in the room.\nPrecision\n, however, is critically dependent on the base rate (prevalence). It answers, \"Given the test is positive, how likely is it the person is sick?\" The answer to that changes dramatically if you're testing in a specialist clinic (high prevalence) versus the general population (low prevalence).\nThe Holy Trinity: Sensitivity, Specificity, and Prevalence.\nWith these three numbers, you can derive almost every other metric in a confusion matrix for a given population. They are the fundamental building blocks. If you have a solid grasp of these three, you can reconstruct the expected number of True Positives, False Positives, and so on, giving you a complete picture of your model's real-world performance.\nCorrect for Your Sampling Bias.\nWe often work with disproportionate or stratified samples for evaluation, especially with rare classes. If your \"Negative\" class has a true prevalence of 1% (p=0.01), but you build a test set where it makes up 50% of the samples (r=0.5), your raw metrics will be misleading. You\nmust adjust\nyour confusion matrix counts to reflect reality. The raw counts of False Positives, for instance, must be re-weighted to account for the fact that you massively undersampled the true negative population.\nDemand Statistical Significance.\nAny metric you compute is just an estimate. To confidently say that model A is better than model B, you need to ensure you've evaluated on a large enough sample to estimate the quantity within an acceptable margin of error. Flying blind with a small test set is a recipe for making the wrong decision.\nPutting the Tenets to Work: A Custom Metric for Scarce Labels\nSo how do these tenets help us solve a real problem? Let's consider a multi-class model (Negative, Positive, Neutral) where \"Negative\" is our critical minority class and getting ground-truth labels is hard.\nOur first tenet warns us that a simple precision-based metric could be misleading and make our model too conservative on the \"Negative\" class it's trying to avoid misclassifying. We care about finding\nall\nthe negatives (Recall), not just being right when we predict one. But Recall is hard to measure.\nThis is where Tenet #2 comes in. We can build a bridge to estimate recall. As derived from Bayes' theorem, the relationship is:\nA Bayesian bridge from Precision to Recall\nRecall expressed in terms of Precision, Prediction share (easier to measure), Prevalence (harder to measure)\nThis is our key. We can invest once in a high-quality labeled set to get a solid estimate for\nPrevalence\n. Then, on an ongoing basis, we can measure\nPrecision\n(which is cheaper, requiring human review only of model outputs) and the model's\nPrediction Share\n(which is free) to get a reliable estimate of Recall.\nThis insight allows us to build a smarter, hybrid metric that addresses our specific business needs:\nWhy this design is so powerful\nF-beta for the Critical Class: For our \"Negative\" class, we use the Fβ score. This score is a weighted harmonic mean of precision and our newly estimated recall. By setting β<1 (like in an F2-score), we explicitly tell the optimizer that precision is more important than recall for this class, directly encouraging the model to be more comprehensive.\nPrecision for Other Classes:\nFor the less critical classes, we can stick with precision. We’re making a conscious trade-off, prioritizing correctness over completeness for them.\nGeometric Mean for Robustness:\nUsing a geometric mean ensures the model can't \"cheat.\" To get a high score, it must perform reasonably well on\nall\ncomponents. A score of zero on any single metric will tank the entire score, promoting a more balanced and robust model.\nWeights for Business Logic:\nThe weights (wneg,wpos,wneu) are the final knobs, allowing us to dial in the exact business importance of each class directly into the evaluation function.\nThe Payoff: Smarter Evaluation for the Modern Era\nThis approach, born from those core tenets, is perfectly suited for the challenges of evaluating complex models like LLMs. It is:\nLabel Efficient:\nIt smartly leverages a small, expensive golden dataset and supplements it with cheaper, ongoing measurements.\nBusiness-Aligned:\nIt provides direct levers (β, weights) to encode your specific priorities into the metric.\nResilient to Imbalance:\nIt actively prevents the model from ignoring your critical minority class.\nThe lessons I learned in this front completely reshaped my approach regarding model evals. Moving beyond the default metrics and building an evaluation framework grounded in these first principles is the difference between simply measuring a model and truly understanding it.",
  "metadata": {
    "title": "The Four Tenets of Model Evaluation I Wish I Knew Sooner",
    "description": null,
    "keywords": null,
    "author": null,
    "og:site_name": "AI First Data Scientist",
    "og:type": "article",
    "og:title": "The Four Tenets of Model Evaluation I Wish I Knew Sooner",
    "og:description": "Some of my most rewarding work has been navigating the intricate nuances of model evaluation with Dave Spiegel, my mentor and manager at Google. We’ve spent non-trivial amount of hours debating the merits of different metrics, questioning assumptions, and trying to build a framework for evaluation that is",
    "og:url": "https://www.aifirstdatascientist.com/articles/the-four-tenets-of-model-evaluation-i-wish-i-knew-sooner/",
    "og:image": "https://storage.ghost.io/c/47/2b/472b355d-9bc0-4634-9df5-89578b761c54/content/images/size/w1200/2026/06/4-Tenets.png",
    "og:image:width": "1200",
    "og:image:height": "655",
    "twitter:card": "summary_large_image",
    "twitter:title": "The Four Tenets of Model Evaluation I Wish I Knew Sooner",
    "twitter:description": "Some of my most rewarding work has been navigating the intricate nuances of model evaluation with Dave Spiegel, my mentor and manager at Google. We’ve spent non-trivial amount of hours debating the merits of different metrics, questioning assumptions, and trying to build a framework for evaluation that is",
    "twitter:url": "https://www.aifirstdatascientist.com/articles/the-four-tenets-of-model-evaluation-i-wish-i-knew-sooner/",
    "twitter:image": "https://storage.ghost.io/c/47/2b/472b355d-9bc0-4634-9df5-89578b761c54/content/images/size/w1200/2026/06/4-Tenets.png",
    "twitter:label1": "Written by",
    "twitter:data1": "JK Vijayaraghavan",
    "twitter:label2": "Filed under",
    "twitter:data2": "Model Evaluation",
    "article:published_time": "2025-08-23T00:00:00.000Z",
    "article:modified_time": "2026-06-07T03:28:21.000Z",
    "article:tag": "Model Evaluation"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/47/2b/472b355d-9bc0-4634-9df5-89578b761c54/content/images/size/w1200/2026/06/4-Tenets.png",
      "bytes": 1079131,
      "reused": true,
      "local_path": "attachments/5cb70af6c5ba/images/001.png",
      "src": "https://storage.ghost.io/c/47/2b/472b355d-9bc0-4634-9df5-89578b761c54/content/images/size/w1200/2026/06/4-Tenets.png",
      "alt": "The Four Tenets of Model Evaluation I Wish I Knew Sooner",
      "downloaded": true
    },
    {
      "status": "http_403",
      "url": "https://media.licdn.com/dms/image/v2/D5612AQEisLJhLUMPbA/article-inline_image-shrink_1500_2232/B56ZjYHqVuH8AU-/0/1755972544663?e=1781740800&v=beta&t=_9G_j-8tghAhTiKHgNm4oB9OmtXO-8ga8XBcfBSG8bA",
      "src": "https://media.licdn.com/dms/image/v2/D5612AQEisLJhLUMPbA/article-inline_image-shrink_1500_2232/B56ZjYHqVuH8AU-/0/1755972544663?e=1781740800&v=beta&t=_9G_j-8tghAhTiKHgNm4oB9OmtXO-8ga8XBcfBSG8bA",
      "alt": "",
      "downloaded": false,
      "retried_at": "2026-10-20T17:42:14Z"
    },
    {
      "status": "http_403",
      "url": "https://media.licdn.com/dms/image/v2/D5612AQGl-3WfpWcBAw/article-inline_image-shrink_1000_1488/B56ZjYIJqFHMAQ-/0/1755972672602?e=1781740800&v=beta&t=TfVfIJ1nlJ3vSW6LIm-UJxa-M0RH1Ll09O3pKh6q9So",
      "src": "https://media.licdn.com/dms/image/v2/D5612AQGl-3WfpWcBAw/article-inline_image-shrink_1000_1488/B56ZjYIJqFHMAQ-/0/1755972672602?e=1781740800&v=beta&t=TfVfIJ1nlJ3vSW6LIm-UJxa-M0RH1Ll09O3pKh6q9So",
      "alt": "",
      "downloaded": false,
      "retried_at": "2026-10-20T17:42:14Z"
    },
    {
      "status": "http_403",
      "url": "https://media.licdn.com/dms/image/v2/D5612AQFfnkI4obw6Mg/article-inline_image-shrink_1000_1488/B56ZjYITrEHQAQ-/0/1755972713872?e=1781740800&v=beta&t=wRcsGSRJIzct_SgI4xKfkuwNgx9j6rYhqn3vB8t2lSQ",
      "src": "https://media.licdn.com/dms/image/v2/D5612AQFfnkI4obw6Mg/article-inline_image-shrink_1000_1488/B56ZjYITrEHQAQ-/0/1755972713872?e=1781740800&v=beta&t=wRcsGSRJIzct_SgI4xKfkuwNgx9j6rYhqn3vB8t2lSQ",
      "alt": "",
      "downloaded": false,
      "retried_at": "2026-10-20T17:42:14Z"
    },
    {
      "status": "http_403",
      "url": "https://media.licdn.com/dms/image/v2/D5612AQGExBOqWl04Iw/article-inline_image-shrink_1000_1488/B56ZjYUN2gH8AQ-/0/1755975835588?e=1781740800&v=beta&t=LL3WwZrkEZpJlwZyfZYuBTbxJzZQ_AKl6XgDvZJbT4U",
      "src": "https://media.licdn.com/dms/image/v2/D5612AQGExBOqWl04Iw/article-inline_image-shrink_1000_1488/B56ZjYUN2gH8AQ-/0/1755975835588?e=1781740800&v=beta&t=LL3WwZrkEZpJlwZyfZYuBTbxJzZQ_AKl6XgDvZJbT4U",
      "alt": "",
      "downloaded": false,
      "retried_at": "2026-10-20T17:42:14Z"
    },
    {
      "status": "http_403",
      "url": "https://media.licdn.com/dms/image/v2/D5612AQHut07Nu3tbuQ/article-inline_image-shrink_1000_1488/B56ZjYUvl5H8AQ-/0/1755975974321?e=1781740800&v=beta&t=KJUaOeLZxhWt6B1NLTAAFD0RnQz_8rXmNorhLvsrPcY",
      "src": "https://media.licdn.com/dms/image/v2/D5612AQHut07Nu3tbuQ/article-inline_image-shrink_1000_1488/B56ZjYUvl5H8AQ-/0/1755975974321?e=1781740800&v=beta&t=KJUaOeLZxhWt6B1NLTAAFD0RnQz_8rXmNorhLvsrPcY",
      "alt": "",
      "downloaded": false,
      "retried_at": "2026-10-20T17:42:14Z"
    },
    {
      "status": "http_403",
      "url": "https://media.licdn.com/dms/image/v2/D5612AQFNXuLRbBqDSA/article-inline_image-shrink_1000_1488/B56ZjYU3weHMAQ-/0/1755976007244?e=1781740800&v=beta&t=FQ8Pcs6PBrZC75DIYlBgjB0RZ9jcqT8W9Wu7XwmD9Ec",
      "src": "https://media.licdn.com/dms/image/v2/D5612AQFNXuLRbBqDSA/article-inline_image-shrink_1000_1488/B56ZjYU3weHMAQ-/0/1755976007244?e=1781740800&v=beta&t=FQ8Pcs6PBrZC75DIYlBgjB0RZ9jcqT8W9Wu7XwmD9Ec",
      "alt": "",
      "downloaded": false,
      "retried_at": "2026-10-20T17:42:14Z"
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.aifirstdatascientist.com/tag/model-evaluation/",
      "label": "Model Evaluation",
      "type": "a"
    },
    {
      "href": "https://www.aifirstdatascientist.com/author/jkvijay/",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:44Z",
  "source_url": "https://www.aifirstdatascientist.com/articles/the-four-tenets-of-model-evaluation-i-wish-i-knew-sooner/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5970,
  "expected_body_images": 7,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "http_403",
  "raw_html": "attachments/5cb70af6c5ba/pages/source.html",
  "local_html": "attachments/5cb70af6c5ba/pages/article.html",
  "body_base_path": "attachments/5cb70af6c5ba/pages/",
  "body_markdown_path": "attachments/5cb70af6c5ba/pages/article.md"
}