{ "source_file": "0930", "source_platform": "Ghost", "author": "Jayakrishnan Vijayaraghavan / AI First Data Scientist", "title_listed": "When You Have Counts, Expected Value Is All You Need", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://www.aifirstdatascientist.com/articles/when-you-have-counts-expected-value-is-all-you-need/", "publish_date": "2023-08", "corpus_stage": "", "notes": "", "id": "e5d02da90f84", "collected_at": "2026-10-02T12:26:00", "title": "When You Have Counts, Expected Value Is All You Need", "body_markdown": "Experimentation & A/B Testing\nWhen You Have Counts, Expected Value Is All You Need\nJK Vijayaraghavan\n13 Aug 2023\n—\n5 min read\nShare\nIs 10 and 11 significantly different from each other? What about 10,000 & 11,000.\nJust comparing counts is unusual in the world of experimentation, since we are often focused on comparing differences in means or differences in proportions. In many cases, this nudges us to create metrics that conform to the latter format - either as an average or as a proportion. And we have some pretty good reasons for doing this. One reason is that normalizing a value to the user level (or any other randomization unit) is much easier to compare. e.g., Average Revenue per user (a mean-based metric) or the percentage of active users who had made a purchase (proportion based metric). The other reason is that the normal distribution and the z-statistic is well defined for this kind of comparison (thanks to the Central Limit Theorem).\nBut what if, we would like to compare raw counts, like, the number of users assigned to the different variants. How can we figure out if the difference we see is actually significant and not just due to random chance?\nWhen does comparing counts makes sense\nComparing counts makes sense in the following scenarios:\nThe metric we'd like to compare is a top-of-the-funnel metric (there's nothing to normalize against)\nWe are comparing invariant metrics\nTop-of-the-funnel metric\nTop-of-the-funnel metrics are metrics that we probably use as randomization units. e.g., number of users assigned to both the variants or the numbers of qualified users assigned to the variants. As the metric suggests, any comparison at this level is a validation of the randomization process itself.\nWe anticipate that the number of units assigned to each variant should be equal. In other words, we assume that the expected value for the units in any bucket is around 50% of the total units (assuming an even 50-50 split). Additionally, the units in one bucket are not expected to significantly differ from those in the other. Testing any difference for these kinds of metrics are known as Sample Ratio Mismatch (SRM) tests and the significance level of the SRM tests are kept at a very low value (very low tolerance for TYPE I error).\nInvariant metrics\nInvariant metrics are very similar to the top-of-the-funnel metric, in the fact the fact that we expect the numbers in both the variant to be almost the same. They just need not be top-of-the funnel metric. e.g., we expect the number of units assigned to any of the covariates (factors which we assume are not impacted by the variant or the new feature being tested) to be the same.\nTests to compare counts\nIn such cases, we can either use the binomial test or the Chi-squared test. But the underlying distribution of both are slightly different. So which to use when?\nBinomial test\nThe binomial test assess whether the observed\nnumber of successes\nin a fixed\nnumber of trials\nsignificantly deviates from what would be expected by chance.\nThe key elements are the number of trials and the probability of success. Typically, the probability of success is assumed to be 50% (assuming an equal split). The \"number of trials\" refers to the total count of both the control and treatment groups in your experiment.\nThe crux of the binomial test lies in comparing the observed number of successes in one of the groups (say, the treatment group) with what we'd expect if the probability of success was truly 50%. If the observed number of successes significantly differs from the expected count, it suggests that the 50% assumption may not hold true (or that some kind of Sample Ratio mismatch has occurred, warranting further investigation).\nThe p-value of the binom-test indicates that the counts are indeed significantly different\nThe Chi-squared test\nThe chi-squared tests whether the observed distribution significantly varies from an expected distribution.\nTo apply the chi-squared test, we need the\nobserved counts\n(the counts of treatment and control) and the\nexpected counts.\nIn our case, we are expecting treatment and control to have equal counts, hence the expected values can be calculated as the average of both values.\nBy comparing the observed and expected counts, the chi-squared test quantifies the extent of the discrepancy between the observed and expected distributions. If this discrepancy is large enough, the test will indicate that the variables are likely not independent and that their association is statistically significant.\nThe p-value of the chisqaured-test also indicates that the counts are significantly different\nWhen we are dealing with large numbers, both distributions output almost identical p-values, though the underlying distributions itself is different. In the above examples, we compared two numbers, 10,000 and 11,000, the resulting p-values almost close to 0. When the counts are the same, the resulting p-values are exaclty 1.0 as expected.\nFor equal counts, the p-value is 1.0 for both distributions\nThe biggest different is when the counts are very small. Let's test this for counts such as 10 & 11.\nIt seems that the binomial distribution is well suited for smaller counts (since 10 & 11 could feasibly arise from the same underlying distribution), although chi-squared distribution is technically applicable for expected counts greater than 5.\nSummary\nIn experimentation settings, we usually deal with large numbers. Under those contexts, we can either use binomial test or a chi-squared tests to detect differences in counts, with identical results. The tests are also easily implementable and scalable (think about running the tests for each day of the experiment to detect of the number of units assigned to different buckets or covariates). All you need to do is figure out is, what's number you'd expect to see in both the buckets; which is usually an average (of both the buckets).\nSo, remember this: when you're working with counts, all you need is that expected number to dive into these tests and understand the health of the tests.\nNote\n: Underlying test distributions, stats and core assumptions validated by our Stats expert Athulya Ganapathi Kandy 🙏", "metadata": { "title": "When You Have Counts, Expected Value Is All You Need", "description": null, "keywords": null, "author": null, "og:site_name": "AI First Data Scientist", "og:type": "article", "og:title": "When You Have Counts, Expected Value Is All You Need", "og:description": "Is 10 and 11 significantly different from each other? What about 10,000 & 11,000.\n\n\nJust comparing counts is unusual in the world of experimentation, since we are often focused on comparing differences in means or differences in proportions. In many cases, this nudges us to create metrics that conform", "og:url": "https://www.aifirstdatascientist.com/articles/when-you-have-counts-expected-value-is-all-you-need/", "og:image": "https://storage.ghost.io/c/47/2b/472b355d-9bc0-4634-9df5-89578b761c54/content/images/2026/06/Counts.png", "og:image:width": "1007", "og:image:height": "576", "twitter:card": "summary_large_image", "twitter:title": "When You Have Counts, Expected Value Is All You Need", "twitter:description": "Is 10 and 11 significantly different from each other? What about 10,000 & 11,000.\n\n\nJust comparing counts is unusual in the world of experimentation, since we are often focused on comparing differences in means or differences in proportions. In many cases, this nudges us to create metrics that conform", "twitter:url": "https://www.aifirstdatascientist.com/articles/when-you-have-counts-expected-value-is-all-you-need/", "twitter:image": "https://storage.ghost.io/c/47/2b/472b355d-9bc0-4634-9df5-89578b761c54/content/images/2026/06/Counts.png", "twitter:label1": "Written by", "twitter:data1": "JK Vijayaraghavan", "twitter:label2": "Filed under", "twitter:data2": "Experimentation & A/B Testing", "article:published_time": "2023-08-14T00:00:00.000Z", "article:modified_time": "2026-06-07T03:28:23.000Z", "article:tag": "Experimentation & A/B Testing" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://storage.ghost.io/c/47/2b/472b355d-9bc0-4634-9df5-89578b761c54/content/images/size/w1200/2026/06/Counts.png", "bytes": 625718, "reused": true, "local_path": "attachments/e5d02da90f84/images/001.png", "src": "https://storage.ghost.io/c/47/2b/472b355d-9bc0-4634-9df5-89578b761c54/content/images/size/w1200/2026/06/Counts.png", "alt": "When You Have Counts, Expected Value Is All You Need", "downloaded": true }, { "status": "http_403", "url": "https://media.licdn.com/dms/image/v2/D5612AQGjv48HseUPLw/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1692559527066?e=1781740800&v=beta&t=7HhKbcm03Pd96q7sSywD-5Gcb4k_E8uY8Uakth0UixE", "src": "https://media.licdn.com/dms/image/v2/D5612AQGjv48HseUPLw/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1692559527066?e=1781740800&v=beta&t=7HhKbcm03Pd96q7sSywD-5Gcb4k_E8uY8Uakth0UixE", "alt": "No alt text provided for this image", "downloaded": false, "retried_at": "2026-10-20T17:48:24Z" }, { "status": "http_403", "url": "https://media.licdn.com/dms/image/v2/D5612AQHPKEN5xFQFBg/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1692562600461?e=1781740800&v=beta&t=Fc9nZC1fURKAqMaKqUPX6eKrCxfMIVzCma4QPMTew6A", "src": "https://media.licdn.com/dms/image/v2/D5612AQHPKEN5xFQFBg/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1692562600461?e=1781740800&v=beta&t=Fc9nZC1fURKAqMaKqUPX6eKrCxfMIVzCma4QPMTew6A", "alt": "No alt text provided for this image", "downloaded": false, "retried_at": "2026-10-20T17:48:25Z" }, { "status": "http_403", "url": "https://media.licdn.com/dms/image/v2/D5612AQE7kSlPM5F4lQ/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1692563714413?e=1781740800&v=beta&t=uDE93uebq_f1f4nu79JZi50BkqBAeZgKW1ErIUWHpu0", "src": "https://media.licdn.com/dms/image/v2/D5612AQE7kSlPM5F4lQ/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1692563714413?e=1781740800&v=beta&t=uDE93uebq_f1f4nu79JZi50BkqBAeZgKW1ErIUWHpu0", "alt": "No alt text provided for this image", "downloaded": false, "retried_at": "2026-10-20T17:48:25Z" }, { "status": "http_403", "url": "https://media.licdn.com/dms/image/v2/D5612AQGWpVTomA6UBQ/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1692563843384?e=1781740800&v=beta&t=1xQ8nubNpBUEyOQT7mLMhc9PQ9M-sljVk-fLh457p9c", "src": "https://media.licdn.com/dms/image/v2/D5612AQGWpVTomA6UBQ/article-inline_image-shrink_1000_1488/article-inline_image-shrink_1000_1488/0/1692563843384?e=1781740800&v=beta&t=1xQ8nubNpBUEyOQT7mLMhc9PQ9M-sljVk-fLh457p9c", "alt": "No alt text provided for this image", "downloaded": false, "retried_at": "2026-10-20T17:48:25Z" } ], "audio": [], "videos": [], "links": [ { "href": "https://www.aifirstdatascientist.com/tag/experimentation-a-b-testing/", "label": "Experimentation & A/B Testing", "type": "a" }, { "href": "https://www.aifirstdatascientist.com/author/jkvijay/", "label": "", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:47:45Z", "source_url": "https://www.aifirstdatascientist.com/articles/when-you-have-counts-expected-value-is-all-you-need/", "papers": [], "actual_video_files": [], "body_chars": 6196, "expected_body_images": 5, "discovery_status": "ok", "delivery_status": "partial", "missing_reason": "http_403", "raw_html": "attachments/e5d02da90f84/pages/source.html", "local_html": "attachments/e5d02da90f84/pages/article.html", "body_base_path": "attachments/e5d02da90f84/pages/", "body_markdown_path": "attachments/e5d02da90f84/pages/article.md" }