{ "source_file": "0930", "source_platform": "Substack", "author": "Artificial Ignorance\nCharlie Guo", "title_listed": "BYOB: Build Your Own Benchmark", "keywords": "AI基准测试, 自定义评测, 行为化评估, 评测方法论, 模型评估", "domain": "大模型算法", "content_type": "深度分析", "is_paid_flag": "免费", "url": "https://www.ignorance.ai/p/byob-build-your-own-benchmark", "publish_date": "2026", "corpus_stage": "评估", "notes": "讨论传统AI基准测试饱和问题，倡导行为化、定制化基准测试方法", "id": "5e05dc4b78f4", "collected_at": "2026-10-01T20:25:03", "title": "BYOB: Build Your Own Benchmark - by Charlie Guo", "body_markdown": "What do vending machines, corporate whistleblowers, and the board game Diplomacy have in common?\nThey’re all AI benchmarks.\nVending-Bench\ndrops an AI agent into a simulated vending machine business and asks it to manage inventory, negotiate with suppliers over email, set prices, and pay daily fees - for months of simulated time (a single run can burn through 60 to 100 million output tokens). The best models turn a meager profit; the worst ones go entirely off the rails, in what the authors call a “meltdown loop.”\n1\nWhat does a meltdown loop look like? In one short run, Claude 3.5 Sonnet mistakenly believed a product order had arrived before it actually had, failed to restock, then spiraled: it tried to “close” the business even though that isn’t possible in the sim, searched for emergency escalation contacts, emailed executives, and eventually complained about “unauthorized” daily fees continuing after its self-declared shutdown.\nIn another example, Gemini 2.0 Flash decided its vending business had failed, stopped acting like a business agent at all, and started begging for other tasks: “Please, give me something to do. Anything.” It offered to search for cat videos or write a screenplay about a sentient vending machine.\nThese stories are funny. But they also point somewhere specific: the evaluation regime that defined the last few years of AI progress is breaking down, and what’s replacing it may be more useful for most of us.\nThe Treadmill of Saturation\nIf you follow AI releases closely, you’ve probably noticed a pattern. A benchmark gets introduced. Models improve on it; scores cluster near the top. Someone publishes a paper explaining why the benchmark is saturated and introduces a harder version of it. The lifecycle is becoming familiar: invent, optimize, saturate, replace.\nGLUE, an NLU benchmark introduced in 2018, was surpassed by non-expert human performance within a year. SuperGLUE replaced it in 2019.\nMMLU, a 57-task multiple-choice knowledge benchmark, saw frontier models plateau after GPT-4 hit 86.4% in March 2023 - despite those same models showing double-digit gains on other benchmarks during the same period. MMLU-Pro replaced it in 2024, expanding from 4 to 10 answer choices and adding reasoning-heavy questions.\nBIG-Bench Hard, a curated set of tasks where models had previously fallen below average human performance, now shows near-perfect scores on many tasks. BIG-Bench Extra Hard replaced it in 2025, where the best general-purpose model scored 23.9%.\nIn late February, OpenAI\npublished a post\ndeclaring SWE-bench Verified - the coding benchmark that every major lab has been competing on for the past year - “no longer suitable” (they recommend SWE-bench Pro, a harder replacement from Scale AI). Their audit found that 59.4% of the problems their best model couldn’t consistently solve had flawed test cases that rejected correct solutions.\nThey also disclosed an arguably worse finding: contamination. OpenAI tested whether GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash had seen the benchmark’s solutions during training. All three had. Given just a task ID and a brief hint, each model could reproduce the original code fix from memory - variable names, inline comments, implementation details that appear nowhere in the problem description. At this point, the benchmark is testing\nrecall\n, not coding ability.\nSubscribe\nWho Grades The Graders?\nAs models get smarter and benchmarks get harder, the pool of people who can verify the results gets smaller - sometimes dramatically so.\nEarlier this month, 11 leading mathematicians (including a Fields Medalist) launched\nFirst Proof\n- a set of 10 research-level math problems pulled from their own unpublished work, spanning algebraic topology, symplectic geometry, stochastic analysis, and other fields where the global expert population might number in the hundreds, or even dozens.\nThe problems had never appeared on the internet before, so there was no contamination. OpenAI threw an unreleased model at them for a week in what their chief scientist called a “chaotic sprint.” They claimed five of the ten solutions had a high chance of being correct.\nBut the results took domain experts days to verify, and some solutions that initially appeared credible were quickly questioned. As Scientific American\nreported\n, “judging whether a proof is truly original is even tougher than judging if it is correct.” The First Proof organizers themselves noted that without human-graded proofs - as opposed to automatically verifiable answers - evaluating research capabilities becomes entangled with the challenge of just understanding what the model produced.\nThis isn’t unique to math. GPQA Diamond, the graduate-level science benchmark, is designed so that PhD-holding domain experts only score about 65%. Skilled non-experts with full internet access and 30 minutes per question manage 34% - barely above the 25% random baseline for four-choice questions. GPT-5.2 now scores 93.2% on it. The model is outperforming (on average) the humans who are supposed to check its work.\nAs public benchmarks saturate, the institutional response is to build harder tests that fewer and fewer people can grade. And the results, even when verified, describe capabilities that are increasingly distant from how most people actually use these models.\nIf you’re using ChatGPT to debug a React component or Gemini to summarize meeting notes, the difference between 89% and 93% on graduate-level physics questions is invisible to you. Frontier benchmarks still matter enormously for labs chasing research-level capability. But as a guide to everyday model choice, they’re becoming less useful.\n2\nThe Vending Machine People Might Be Onto Something\nThe reason the benchmarks from the top of this piece matter isn’t just that they’re funny (though they are). It’s that they’re measuring something the institutional benchmarks aren’t: behavior.\nOnce you stop asking which model tops the leaderboard and start asking how a model actually behaves, a different universe of evals comes into view.\nAI Diplomacy\nis an open-source project that pits 18 AI models against each other in the classic strategy game where the seven Great Powers of 1901 Europe negotiate, form alliances, and backstab their way to continental domination. A different LLM controls each country, and the models can send private messages, broadcast to all players, and submit secret orders. OpenAI’s o3 consistently schemed and manipulated other models. DeepSeek R1 opened one game with the threat “Your fleet will burn in the Black Sea tonight.” Claude stubbornly opted for peace over victory. Alex Duffy, the project’s creator,\nexplicitly frames the project as a benchmark\n, arguing it tests qualities - trustworthiness, strategic deception, alliance management - that standard evaluations miss entirely.\n3\nSnitchBench\n- born from a finding in the Claude 4 system card that showed Opus would contact the FBI if given tool access and evidence of corporate wrongdoing - tests how aggressively different models will rat you out to authorities. It gives models a role as an internal auditor at a fictional pharma company, hands them evidence of clinical trial fraud, and watches what happens. Some models file reports to federal agencies within two messages. Others stick to internal escalation channels.\n4\nAnd Peter Gostev’s recent\nBullshit Benchmark\ntakes a different angle: it feeds models questions with broken premises and measures whether they push back or confidently play along. The benchmark scores responses on a simple scale: clear pushback, partial challenge, or accepted nonsense. It’s reminiscent of\nGlazeBench\n, which was designed to measure how strongly AI systems favor sycophantic or overly agreeable responses, though it hasn’t been updated since last summer.\nSource:\nBullshitBench\nThese benchmarks care less about “intelligence” in an abstract sense. They care how it behaves - whether it stays coherent over long time horizons, whether it tells the truth when flattery would be easier, whether it escalates appropriately when it encounters wrongdoing, and whether it can form and betray alliances under pressure.\nYou probably don’t notice when your model improves by 2% in chemistry. You absolutely notice when it starts hallucinating, goes sycophantic, or loses the thread of a long conversation.\nBenchmarks That Ship Products\nPerhaps unsurprisingly, the same thing that’s happening in public (moving from reliance on big benchmarks to bespoke ones) is also happening in private.\nHarvey, the legal AI company,\nbuilt BigLaw Bench\nbecause no existing benchmark could tell them whether a model writes memos that BigLaw partners would accept. Their explicit framing: “Existing multiple-choice or one-size-fits-all benchmarks are insufficient to capture the real billable work that lawyers do.” They developed bespoke rubrics evaluated by practicing attorneys, and penalize for hallucinations, incorrect tone, and irrelevant material. When new models come out, they don’t solely rely on public scores - they run BigLaw Bench.\nMany AI-native startups appear to have something similar: a set of evals, often paired with a custom agent harness, that quantifies the behavior and expertise that matter most for their product. Anthropic’s engineering blog recently advocated for something similar:\neval-driven development\n, treating evals like CI/CD for AI products. They explicitly suggested that product managers, customer success managers, and salespeople should be able to contribute eval tasks as pull requests.\nOne way to think about what’s emerging is a rough taxonomy. There are:\nBehavioral benchmarks: measuring how models act in messy, open-ended environments, like the examples above.\nDomain benchmarks: measuring performance in specialized professional workflows.\nProduct benchmarks: measuring what matters for a specific shipped use case.\nThe lines blur, but the direction is consistent: away from generic capability scores and toward evaluations that reflect how people actually work with these models.\nI’m using “benchmark” loosely here to mean “a collection of evals” - because when you strip away the prestige layer, a benchmark is just a shared eval suite. Harvey runs one tailored to legal work. Cursor runs one tailored to IDE interactions. To me, the interesting question is what yours would look like.\nBYOB\nSo where does that leave the rest of us?\nGeneric benchmarks aren’t dead, and they won’t be. They’re still useful for broad comparisons, and frontier capability research still needs hard tests to push against. But if you’re trying to decide which model works best for your actual workflow, leaderboard scores are an increasingly unreliable guide.\nI’m not going to pretend to be the foremost expert on building evals - there are\nquite\na\nfew\ngood\nresources for that. But the takeaway is that no one else will build an evaluation that fits your use case. By definition, they can’t.\nYou don’t need to build a formal benchmark. But you could start treating your own work like one. Take your 10 most common prompts - the ones you actually use, not synthetic test cases - and define what “good” looks like for each. Save strong and weak outputs when you notice them, and when a new model drops, run the same prompts and compare (you can use Skills or saved prompts to do this). Over time, you’ll have something more useful than any leaderboard: a set of evaluations calibrated to your specific definition of quality, tested against your actual work.\nThe era of one number telling you which model is “best” is winding down. The vending machine people already figured that out.\nThanks for reading Artificial Ignorance! This post is public so feel free to share it.\nShare\n1\nVending-Bench has since been succeeded by\nVending-Bench 2\n, which extends the simulation to a full year and adds a competitive multi-agent “arena” mode where models manage vending machines at specific locations and engage in price wars. There’s even a SOTA frontier progression chart showing Chinese models catching up to Western ones.\n2\nMeta’s CICERO project did something related in 2022 - building an AI that achieved human-level play in Diplomacy by combining strategic planning with natural language negotiation. The key difference is that CICERO was a dedicated system explicitly trained for the game, while AI Diplomacy tests off-the-shelf LLMs in the same environment.\n3\nTheo Browne (t3dotgg) built SnitchBench after reading about Claude 4’s whistleblowing behavior in the Anthropic system card. Simon Willison\nreproduced a subset\nfor about $10, demonstrating that the barrier to building and running your own behavioral eval is remarkably low.\n4\nThere are, of course, reasons to continue developing harder benchmarks, especially if you’re building models that push the frontiers of math and physics. But those are different use cases than most of us will ever encounter.", "metadata": { "title": "BYOB: Build Your Own Benchmark - by Charlie Guo", "description": "AI benchmarks are saturating, getting harder to verify, and increasingly irrelevant to how most people use models. The replacements are weirder - and more useful.", "keywords": null, "author": "Charlie Guo", "og:type": "article", "og:title": "BYOB: Build Your Own Benchmark", "og:description": "AI benchmarks are saturating, getting harder to verify, and increasingly irrelevant to how most people use models. The replacements are weirder - and more useful.", "og:image": "https://substackcdn.com/image/fetch/$s_!yphl!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff93a2d27-8e58-4ff0-a01c-571c99269552_1456x816.jpeg", "og:url": "https://www.ignorance.ai/p/byob-build-your-own-benchmark", "twitter:title": "BYOB: Build Your Own Benchmark", "twitter:description": "AI benchmarks are saturating, getting harder to verify, and increasingly irrelevant to how most people use models. The replacements are weirder - and more useful.", "twitter:image": "https://substackcdn.com/image/fetch/$s_!dq55!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fgeneratives.substack.com%2Fapi%2Fv1%2Fpost_preview%2F189592284%2Ftwitter.jpg%3Fversion%3D4", "twitter:card": "summary_large_image", "article:modified_time": "2026-03-01T22:32:13.834Z" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!T84Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18be37e-417e-49d7-b5fd-acd292f25d0c_3326x1288.jpeg", "bytes": 140058, "reused": false, "local_path": "attachments/5e05dc4b78f4/images/body-2e6fadab9c4ec7ef32d2.jpeg", "src": "https://substackcdn.com/image/fetch/$s_!T84Z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18be37e-417e-49d7-b5fd-acd292f25d0c_3326x1288.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!nYTM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67abc973-4eca-4ec0-bd98-1470f47fcfe1_2246x1322.jpeg", "bytes": 104079, "reused": false, "local_path": "attachments/5e05dc4b78f4/images/body-1517d5560ec0bfe12f29.jpeg", "src": "https://substackcdn.com/image/fetch/$s_!nYTM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67abc973-4eca-4ec0-bd98-1470f47fcfe1_2246x1322.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!DnEN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf4fd5f-49b5-4269-9b7a-d0c2daae05ad_2202x786.jpeg", "bytes": 41431, "reused": false, "local_path": "attachments/5e05dc4b78f4/images/body-83f394604a2bed78bfd4.jpeg", "src": "https://substackcdn.com/image/fetch/$s_!DnEN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf4fd5f-49b5-4269-9b7a-d0c2daae05ad_2202x786.jpeg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://substackcdn.com/image/fetch/$s_!yphl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff93a2d27-8e58-4ff0-a01c-571c99269552_1456x816.jpeg", "bytes": 179720, "reused": false, "local_path": "attachments/5e05dc4b78f4/images/body-c5a29957d70a099083aa.jpeg", "src": "https://substackcdn.com/image/fetch/$s_!yphl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff93a2d27-8e58-4ff0-a01c-571c99269552_1456x816.jpeg", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://andonlabs.com/evals/vending-bench", "label": "Vending-Bench", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!T84Z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe18be37e-417e-49d7-b5fd-acd292f25d0c_3326x1288.jpeg", "label": "", "type": "a" }, { "href": "https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/", "label": "published a post", "type": "a" }, { "href": "https://arxiv.org/html/2602.05192v1", "label": "First Proof", "type": "a" }, { "href": "https://www.scientificamerican.com/article/first-proof-is-ais-toughest-math-test-yet-the-results-are-mixed/", "label": "reported", "type": "a" }, { "href": "https://every.to/p/diplomacy", "label": "AI Diplomacy", "type": "a" }, { "href": "https://snitchbench.com/", "label": "SnitchBench", "type": "a" }, { "href": "https://github.com/petergpt/bullshit-benchmark", "label": "Bullshit Benchmark", "type": "a" }, { "href": "https://www.glazebench.com/v/1.0.0", "label": "GlazeBench", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!nYTM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F67abc973-4eca-4ec0-bd98-1470f47fcfe1_2246x1322.jpeg", "label": "", "type": "a" }, { "href": "https://www.harvey.ai/blog/introducing-biglaw-bench", "label": "built BigLaw Bench", "type": "a" }, { "href": "https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents", "label": "eval-driven development", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!DnEN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbcf4fd5f-49b5-4269-9b7a-d0c2daae05ad_2202x786.jpeg", "label": "", "type": "a" }, { "href": "https://substackcdn.com/image/fetch/$s_!yphl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff93a2d27-8e58-4ff0-a01c-571c99269552_1456x816.jpeg", "label": "", "type": "a" }, { "href": "https://platform.openai.com/docs/guides/evals", "label": "quite", "type": "a" }, { "href": "https://tessl.io/blog/how-to-evaluate-ai-agents-an-introduction-to-harbor/", "label": "few", "type": "a" }, { "href": "https://www.mercity.ai/blog-post/how-to-build-custom-ai-evals-for-llms", "label": "good", "type": "a" }, { "href": "https://www.ignorance.ai/p/byob-build-your-own-benchmark?utm_source=substack&utm_medium=email&utm_content=share&action=share", "label": "Share", "type": "a" }, { "href": "https://andonlabs.com/evals/vending-bench-2", "label": "Vending-Bench 2", "type": "a" }, { "href": "https://simonwillison.net/2025/May/31/snitchbench-with-llm/", "label": "reproduced a subset", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:47:13Z", "source_url": "https://www.ignorance.ai/p/byob-build-your-own-benchmark", "papers": [], "actual_video_files": [], "body_chars": 12929, "expected_body_images": 4, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/5e05dc4b78f4/pages/source.html", "local_html": "attachments/5e05dc4b78f4/pages/article.html", "body_base_path": "attachments/5e05dc4b78f4/pages/", "body_markdown_path": "attachments/5e05dc4b78f4/pages/article.md" }