{ "source_file": "0930", "source_platform": "Medium", "author": "Cai Parry-Jones", "title_listed": "The Statistical Significance Scam", "keywords": "", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/the-statistical-significance-scam-db904be36714/", "publish_date": "2024-11-09", "corpus_stage": "", "notes": "", "id": "27167390663c", "collected_at": "2026-10-02T05:21:28", "title": "The Statistical Significance Scam | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# The Statistical Significance Scam\nA detailed look into the flaws of science's favorite tool\n[Cai Parry-Jones](https://towardsdatascience.com/author/caiparryjones96/)\nNovember 9, 202413 min read\n![source: unsplash.com](../images/000.jpeg)\nsource: [unsplash.com](https://unsplash.com/)\nStatistical significance is like the drive-thru of the research world. Roll up to the study, grab your \"significance meal,\" and boom - you've got a tasty conclusion to share with all your friends. And it isn't just convenient for the reader, it makes researchers' lives easier too. Why make the hard sell when you can say two words instead?\nBut there's a catch.\nThose fancy equations and nitty-gritty details we've conveniently avoided? They're the real meat of the matter. And when researchers and readers rely too heavily on one statistical tool, we can end up making a whopper of a mistake, like the one that nearly broke the laws of physics.\nIn 2011, physicists at the renowned CERN laboratory announced a shocking discovery: [neutrinos could travel faster than the speed of light](https://www.science.org/content/article/neutrinos-travel-faster-light-according-one-experiment). The finding threatened to overturn Einstein's theory of relativity, a cornerstone of modern physics. The researchers were confident in their results, passing physics' rigorous statistical significance threshold of 99.9999998%. Case closed, right?\nNot quite. As other scientists scrutinized the experiment, they found flaws in the methodology and ultimately [could not replicate the results](https://www.nbcnews.com/id/wbna46760561). The original finding, despite its impressive \"statistical significance,\" turned out to be false.\nIn this article, we'll delve into four critical reasons why you shouldn't instinctively trust a statistically significant finding. Moreover, why you shouldn't habitually discard non-statistically significant results.\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) and [Data Analyst](https://roadmap.sh/data-analyst) roadmaps.\n## TL;DR\nThe four key flaws of statistical significance:\n 1. **It's made up** : The statistical significance/non-significance line is all too often plucked out of thin air, or lazily taken from the general line of 95% confidence.\n 2. **It doesn't mean what (most) people think it means** : Statistical significance does not mean 'There is Y% chance X is true'.\n 3. **It's easy to hack (and frequently is)** : Randomness is frequently labeled statistically significant due to mass experiments.\n 4. **It's nothing to do with how important the result is** : Statistical significance is not related to the significance of the difference.\n\n\n## Flaw 1: It's made up\nStatistical significance is simply a line in the sand humans have created with _zero_ mathematical support. Think about that for a second. Something that is generally thought of as an objective measure is, at its core, entirely subjective.\nThe mathematical part is provided one step before deciding on the significance, via a numerical measure of confidence. The most common form used in [hypothesis testing](https://www.scribbr.com/statistics/hypothesis-testing/) is called the [p-value](https://www.scribbr.com/statistics/p-value/). This provides the actual mathematical probability that the test data results were not simply due to randomness.\nFor example, a p-value of 0.05 means there's a 5% chance of seeing these data points (or more extreme) due to random chance, or that we are 95% confident the result wasn't due to chance. For example, suppose you believe a coin is unfair in favour of heads i.e. the probability of landing on heads is greater than 50%. You toss the coin 5 times and it lands on heads each time. There's a 1/2 x 1/2 x 1/2 x 1/2 x 1/2 = 3.1% chance that it happened simply because of chance, if the coin was fair.\nBut is this enough to say it's statistically significant? It depends who you ask.\nOften, whoever is in charge of determining where the line of significance will be drawn in the sand has more influence on whether a result is significant than the underlying data itself.\nGiven this subjective final step, often in my own analysis I'd provide the reader of the study with the level of confidence percentage, rather than the binary significance/non-significance result. The final step is simply too opinion-based.\n**Sceptic** : \"_But there are standards in place for determining statistical significance._ \"\nI hear the argument a lot in response to my argument above (I talk about this quite a bit - much to the delight of my academic researcher girlfriend). To which, I respond with something like:\n**Me** : \"_Of course, if there is a specific standard you must adhere to, such as for regulatory or academic journal publishing reasons, then you have no choice but to follow the standard. But if that isn't the case then there's no reason not to._ \"\n**Sceptic** : \"_But there is a general standard. It's 95% confidence._ \"\nAt that point in the conversation I try my best not to roll my eyes. Deciding your test's statistical significance point is 95%, simply because that is the norm, is frankly lazy. It doesn't take into account the context of what is being tested.\nIn my day job, if I see someone using the 95% significance threshold for an experiment without a contextual explanation, it raises a red flag. It suggests that the person either doesn't understand the implications of their choice or doesn't care about the specific business needs of the experiment.\nAn example can best explain why this is so important.\nSuppose you work as a data scientist for a tech company, and the UI team want to know, \"Should we use the color red or blue for our 'subscribe' button to maximise out Click Through Rate (CTR)?\". The UI team favour neither color, but must choose one by the end of the week. After some A/B testing and statistical analysis we have our results:\n![Image created by the author.](../images/001.png)\nImage created by the author.\nThe follow-the-standards data scientist may come back to the UI team announcing, \"_Unfortunately, the experiment found no statistically significant difference between the click-through rate of the red and blue button._ \"\nThis is a horrendous analysis, purely due to the final subjective step. Had the data scientist taken the initiative to understand the context, critically, that 'the UI team favour neither color, but must choose one by the end of the week', then you should have set the significance point at a very high p-value, arguably 1.0 i.e. the statistical analysis doesn't matter, the UI team are happy to pick whichever color had the highest CTR.\nGiven the risk that data scientists and the like may not have the full context to determine the best point of significance, it's better (and simpler) to give the responsibility to those who have the full business context - in this example, the UI team. In other words, the data scientist should have announced to the UI team, \"_The experiment resulted with the blue button receiving a higher click-through rate, with a confidence of 94% that this wasn't attributed to random chance._ \" The final step of determining significance should be made by the UI team. Of course, this doesn't mean the data scientist shouldn't educate the team on what \"c _onfidence of 94%\"_ means, as well as clearly explaining _why_ the statistical significance is best left to them.\n## Flaw 2: It doesn't mean what (most) people think it means\nLet's assume we live in a slightly more perfect world, where point one is no longer an issue. The line in the sand figure is always perfect, huzza! Say we want to run an experiment, with the the significance line set at 99% confidence. Some weeks pass and at last we have our results and the statistical analysis finds that it's statistically significant, huzza again!.. But what does that actually mean?\nCommon belief, in the case of hypothesis testing, is that there is a 99% chance that the hypothesis is correct. This is painfully wrong. All it means is there is a 1% chance of observing data this extreme or more extreme by randomness _for this experiment_.\nStatistical significance doesn't take into account whether the experiment itself is accurate. Here are some examples of things statistical significance can't capture:\n * Sampling quality: The population sampled could be biased or unrepresentative.\n * Data quality: Measurement errors, missing data, or other data quality issues aren't addressed.\n * Assumption validity: The statistical test's assumptions (like normality, independence) could be violated.\n * Study design quality: Poor experimental controls, not controlling for confounding variables, testing multiple outcomes without adjusting significance levels.\n\n\nComing back to the example mentioned in the introduction. After failures to independently replicate the initial finding, physicists of the original 2011 experiment announced they had found a bug in their measuring device's master clock i.e. data quality issue, which resulted in a full retraction of their initial study.\nThe next time you hear a statistically significant discovery that goes against common belief, don't be so quick to believe it.\n## Flaw 3: It's easy to hack (and frequently is)\nGiven statistical significance is all about how likely something may have occurred due to randomness, an experimenter who is more interested in achieving a statistical significant result than uncovering the truth can quite easily game the system.\nThe odds of rolling two ones from two dice is (1/6 × 1/6) = 1/36, or 2.8%; a result so rare it would be classified as statistically significant by many people. But what if I throw more than two dice? Naturally, the odds of at least two ones will rise:\n * 3 dice: ≈ 7.4%\n * 4 dice: ≈ 14.4%\n * 5 dice: ≈ 23%\n * 6 dice: ≈ 32.4%\n * 7 dice: ≈ 42%\n * 8 dice: ≈ 51%\n * 12 dice: ≈ 80%*\n\n\n> *At least two dice rolling a **one** is the equivalent of: 1 (i.e. 100%, certain), minus the probability of rolling zero **ones,** minus the probability of rolling only one **one**\n> P(zero **ones**) = (5/6)^n\n> P(exactly one **one**) = n _(1/6)_ (5/6)^(n-1)\n> n is the number of dice\n> So the complete formula is: **1 - (5/6)^n - n** _**(1/6)**_**(5/6)^(n-1)**\nLet's say I run a simple experiment, with an initial theory that **one** is more likely than other numbers to be rolled. I roll 12 dice of different colors and sizes. Here are my results:\n![Image created by the author.](../images/002.png)\nImage created by the author.\nUnfortunately, my (calculated) hopes of getting at least two **ones** have been dashed... Actually, now that I think of it, I didn't really want two ones. I was more interested in the odds of big red dice. I believe there is a high chance of getting sixes from them. Ah! Looks like my theory is correct, the two big red dice have rolled sixes! There is only a 2.8% chance of this happening by chance. Very interesting. I shall now write a paper on my findings and aim to publish it in an academic journal that accepts my result as statistically significant.\nThis story may sound far-fetched, but the reality isn't as distant from this as you'd expect, especially in the highly regarded field of academic research. In fact, this sort of thing happens frequently enough to make a name for itself, [p-hacking](https://statisticsbyjim.com/hypothesis-testing/p-hacking/#:~:text=P%20hacking%20is%20the%20manipulation,the%20integrity%20of%20scientific%20research.).\nIf you're surprised, delving into the academic system will clarify why practices that seem abominable to the scientific method occur so frequently within the realm of science.\nAcademia is exceptionally difficult to have a successful career in. For example, In STEM subjects only [0.45% of PhD students become professors](https://occamstypewriter.org/athenedonald/2014/02/08/thinking-about-the-pipeline/). Of course, some PhD students don't want an academic career, but the majority do (67% according to [this](https://www.hepi.ac.uk/wp-content/uploads/2020/07/HEPI-Policy-Note-25_PhD-students-careers_FINAL.pdf) survey). So, roughly speaking, you have a 1% chance of making it as a professor if you have completed a PhD and want to make academia your career. Given these odds you need think of yourself as quite exceptional, or rather, you need other people to think that, since you can't hire yourself. So, how is exceptional measured?\nPerhaps unsurprisingly, the most important measure of an academic's success is their [research impact](https://www.lib.ncsu.edu/measuring-research-impact/your-impact#:~:text=Research%20impact%20is%20often%20measured,index%2C%20and%20journal%20impact%20factors.). Common measures of author impact include the h-index, g-index and i10-index. What they all have in common is they're heavily focused on citations i.e. how many times has their published work been mentioned in other published work. Knowing this, if we want to do well in academia, we need to focus on publishing research that's likely to get citations.\nYou're far [more likely to be cited if you publish your work in a highly rated academic journal](https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24810#:~:text=There%20is%20strong%20evidence%20from,tend%20to%20be%20more%20cited.). And, since [88% of top journal papers are statistically significant](https://pubmed.ncbi.nlm.nih.gov/19892276/), you're far more likely to get accepted into these journals if your research is statistically significant. This pushes a lot of well-meaning, but career-driven, academics down a slippery slope. They start out with a scientific methodology for producing research papers like so:\n![Decision-making tree for experimental research publication using the scientific method. Created by the author using Mermaid.](../images/003.png)\nDecision-making tree for experimental research publication using the scientific method. Created by the author using [Mermaid](https://github.com/mermaid-js/mermaid).\nBut end up warping their methodology to look scientific on the surface - but really, they've thrown proper scientific methods out the window:\n![Decision-making tree to maximise publication success per experiment. Created by the author using Mermaid.](../images/004.png)\nDecision-making tree to maximise publication success per experiment. Created by the author using [Mermaid](https://github.com/mermaid-js/mermaid).\nGiven the decision diagrams have the researcher writing the paper _after_ discovering a significant result, there's no evidence for the journal reviewer to criticise the experiment for p-hacking.\nThat's the theory anyway. But does it really happen all that often in reality?\nThe answer is a resounding yes. In fact, the [majority of scientific research is unreproducible by fellow academics](https://www.bbc.co.uk/news/science-environment-39054778). Unreproducible means a research paper attempts to copy another research paper's experiment, but ends up with statistically unexpected results. Often finding a statistically significant result in the original paper was statistically insignificant in the replication, or in [some instances](https://journals.sagepub.com/doi/full/10.1177/1745691616652873) statistically significant in the opposite direction!\n## Flaw 4: It's nothing to do with how important the result is\nFinally, statistical significance doesn't care about the scale of the difference.\nThink about it this way - statistical significance basically just tells you \"hey, this difference probably isn't due to random chance\" but says nothing about whether the difference actually matters in the real world.\nLet's say you test a new medication and find it reduces headache pain by 0.0001% compared to a placebo. If you run this test on millions of people, that tiny difference might be statistically significant, since your sample size is massive. But... who cares about a 0.0001% reduction in pain? That's meaningless in practical terms!\nOn the other hand, you might find a drug that reduces pain by 5%, but there hasn't been a large experiment to demonstrate statistical significance. It's likely there are many examples of this in medicine because if the drug in question is cheap there is no incentive for pharmaceutical companies to run the experiment since large scale medical testing is expensive.\nThis is why it's important to look at effect size (how big the difference is) separately from statistical significance. In the real world, you want both - a difference that's unlikely to be random _and_ big enough to actually matter.\nAn example of this mistake happening time and time again is when there is a (statistically significant) discovery in carcinogens i.e. something that causes cancer. A 2015 Guardian [article](https://www.theguardian.com/society/2015/oct/26/bacon-ham-sausages-processed-meats-cancer-risk-smoking-says-who) said:\n> \"Bacon, ham and sausages rank alongside cigarettes as a major cause of cancer, the World Health Organisation has said, placing cured and processed meats in the same category as asbestos, alcohol, arsenic and tobacco.\"\nThis is straight up misinformation. Indeed, bacon, ham and sausages are in the same category as asbestos, alcohol, arsenic and tobacco. However, the categories do not denote the scale of the effect of the carcinogens, rather, how confident the World Health Organisation is that these items are carcinogens i.e. statistical significance.\nThe scale of the cancer cases caused by processed meat is questionable, since there haven't been any Randomized Controlled Trials (RCT). One of the most damning research in favour of processed meat causing cancer is a 2020 observational (think correlation, not causation) [study](https://academic.oup.com/ije/article/49/5/1540/5894731?login=false) in the UK. It found that people eating over 79 grams per day on average of red and processed meat had a 32% increased risk of bowel cancer compared to people eating less than 11 grams per day on average.\nHowever, to understand the true risk we need to understand the number of people who are at risk of bowel cancer. For every 10,000 people on the study who ate less than 11 grams of processed and red meat a day, 45 were diagnosed with bowel cancer, while it was 59 from those eating 79 grams of processed and red meat a day. That's an extra 14 cases of bowel cancer per 10,000 people, or 0.14%. The survivability in the UK of bowel cancer is [53%](https://www.cancerresearchuk.org/health-professional/cancer-statistics/statistics-by-cancer-type/bowel-cancer), so a rough estimate of carcinogens in processed meat killing you is 0.07%.\nCompare this to another substance The Guardian mention, tobacco. [Cancer Research](https://www.cancerresearchuk.org/health-professional/cancer-statistics/risk/tobacco#heading-Zero) say:\n> \"Tobacco is the largest preventable cause of cancer and death in the UK. And one of the largest preventable causes of illness and death in the world. Tobacco caused an estimated 75,800 deaths in the UK in 2021 - around a tenth (11%) of all deaths from all causes.\"\nFirst of all, wow. Don't smoke.\nSecondly, the death rate of cancer caused by tobacco is 11%/0.07% = **157** times greater than processed meat! Coming back to the quotation in the article, \"Bacon, ham and sausages rank alongside cigarettes as a major cause of cancer\". Simply, fake news.\n## Summary\nIn conclusion, while statistical significance has a place in validating quantitative research, it's crucial to understand its severe limitations.\nAs readers, we have a responsibility to approach claims of statistical significance with a critical eye. The next time you encounter a study or article touting a \"statistically significant\" finding, take a moment to ask yourself:\n 1. Is the significance threshold appropriate for the context?\n 2. How robust was the study design and data collection process?\n 3. Could the researchers have engaged in p-hacking or other questionable practices?\n 4. What is the practical significance of the effect size?\n\n\nBy asking these questions and demanding more nuanced discussions around statistical significance, we can help promote a more responsible and accurate use of the tool.\n## Over-time analysis\nI actually think the main reason statistical significance has gained such over prominence is because of the name. People associate \"statistical\" with mathematical and objective, and \"significance\" with, well, significant. I hope this article has persuaded you that these associations are merely fallacies.\nIf the scientific and wider community wanted to deal with the over prominence issue, they should seriously consider simply renaming \"statistical significance\". Perhaps \"chance-threshold test\" or \"Non-random confidence\". Then again, this would lose its Big Mac convenience.\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-be9b2abe4d1a1929d9b6.jpeg)\nWritten By\n## [Cai Parry-Jones](https://towardsdatascience.com/author/caiparryjones96/)\n[See all from Cai Parry-Jones](https://towardsdatascience.com/author/caiparryjones96/)\n[Data Quality](https://towardsdatascience.com/tag/data-quality/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Getting Started](https://towardsdatascience.com/tag/getting-started/)[Scientific Research](https://towardsdatascience.com/tag/scientific-research/)[Statistics](https://towardsdatascience.com/tag/statistics/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-statistical-significance-scam-db904be36714%2F&title=The%20Statistical%20Significance%20Scam)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-statistical-significance-scam-db904be36714%2F&title=The%20Statistical%20Significance%20Scam)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-statistical-significance-scam-db904be36714%2F&text=The%20Statistical%20Significance%20Scam)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/005.jpeg)](https://towardsdatascience.com/11-different-uses-of-dimensionality-reduction-4325d62b4fa6/)\n## [11 Different Uses of Dimensionality Reduction](https://towardsdatascience.com/11-different-uses-of-dimensionality-reduction-4325d62b4fa6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThe whole ML is full of dimensionality reduction and its applications. Let's see them in action!\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nDecember 8, 202110 min read\n * [![](../images/006.jpg)](https://towardsdatascience.com/how-to-use-python-and-missforest-algorithm-to-impute-missing-data-ed45eb47cb9a/)\n## [How to Use Python and MissForest Algorithm to Impute Missing Data](https://towardsdatascience.com/how-to-use-python-and-missforest-algorithm-to-impute-missing-data-ed45eb47cb9a/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nStep-by-step guide on using Random Forests to handle missing data\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nNovember 5, 20204 min read\n * [![](../images/007.jpeg)](https://towardsdatascience.com/are-you-a-pandas-user-who-is-curious-about-r-for-data-analysis-here-is-a-quick-start-1f88bb5d7001/)\n## [Are You a Pandas User Who Is Curious About R for Data Analysis? Here Is a Quick Start](https://towardsdatascience.com/are-you-a-pandas-user-who-is-curious-about-r-for-data-analysis-here-is-a-quick-start-1f88bb5d7001/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nPandas is great but it is not the only one in the game.\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nJanuary 20, 20224 min read\n * [![](../images/008.jpg)](https://towardsdatascience.com/from-insights-to-impact-leveraging-data-science-to-maximize-customer-value-9c9af354e192/)\n## [From insights to impact: leveraging data science to maximize customer value](https://towardsdatascience.com/from-insights-to-impact-leveraging-data-science-to-maximize-customer-value-9c9af354e192/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nUplift modeling: how causal machine learning transforms customer relationships and revenue\n[Arthur Cruiziat](https://towardsdatascience.com/author/arthur.cruiziat/)\nAugust 13, 202410 min read\n * [![](../images/009.jpg)](https://towardsdatascience.com/are-we-thinking-about-explainability-backwards-c9a719cb1250/)\n## [Are We Thinking about eXplAInability Backwards?](https://towardsdatascience.com/are-we-thinking-about-explainability-backwards-c9a719cb1250/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThree questions you should be able to answer before building an AI solution\n[Bryce Murray, PhD](https://towardsdatascience.com/author/bryce.murray/)\nSeptember 27, 20216 min read\n * [![](../images/010.jpg)](https://towardsdatascience.com/how-to-up-skill-in-data-science-3f71fafeaab7/)\n## [How To Up-Skill In Data Science](https://towardsdatascience.com/how-to-up-skill-in-data-science-3f71fafeaab7/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nMy framework for continually becoming a better data scientist\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nNovember 14, 20246 min read\n * [![](../images/011.png)](https://towardsdatascience.com/cyberpunking-your-matplotlib-figures-96f4d473185d/)\n## [Cyberpunking Your Matplotlib Figures](https://towardsdatascience.com/cyberpunking-your-matplotlib-figures-96f4d473185d/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nTaking Your Matplotlib Figures from Boring to Interesting with a Few Lines of Code\n[Andy McDonald](https://towardsdatascience.com/author/andymcdonaldgeo/)\nMay 21, 20238 min read\n * [![](../images/012.jpg)](https://towardsdatascience.com/bounding-the-sample-size-of-a-machine-learning-algorithm-d01d4775263c/)\n## [Bounding the Sample Size of a Machine Learning Algorithm](https://towardsdatascience.com/bounding-the-sample-size-of-a-machine-learning-algorithm-d01d4775263c/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nIt can be very helpful to bound the sample size of a machine learning algorithm. Read on to find out how.\n[Wilson Wang](https://towardsdatascience.com/author/wilson-wang/)\nAugust 18, 20215 min read\n * [![](../images/013.jpeg)](https://towardsdatascience.com/maximize-savings-on-your-unused-fabric-capacity-4c74b12505fd/)\n## [Maximize Savings on Your Unused Fabric Capacities](https://towardsdatascience.com/maximize-savings-on-your-unused-fabric-capacity-4c74b12505fd/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAutomate your Fabric capacity state with Azure Logic Apps\n[Christabelle Santos](https://towardsdatascience.com/author/chrystasantos/)\nJuly 29, 20248 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "metadata": { "title": "The Statistical Significance Scam | Towards Data Science", "description": "A detailed look into the flaws of science's favorite tool", "keywords": null, "author": "Cai Parry-Jones", "og:locale": "en_US", "og:type": "article", "og:title": "The Statistical Significance Scam", "og:description": "A detailed look into the flaws of science's favorite tool", "og:url": "https://towardsdatascience.com/the-statistical-significance-scam-db904be36714/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1fUigVfmuyNJLUFY6m-lZUw-scaled.jpeg", "og:image:width": "2560", "og:image:height": "1440", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Cai Parry-Jones", "twitter:label2": "Est. reading time", "twitter:data2": "13 min read", "article:published_time": "2024-11-09T12:02:00.000Z", "article:modified_time": "2024-11-09T12:02:00.000Z", "article:author": "https://towardsdatascience.com/author/caiparryjones96/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 202, "error": "" }, "code_blocks": [], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:09:24Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:09:24Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:09:24Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:09:24Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/caiparryjones96/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/data-quality/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/getting-started/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/scientific-research/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/statistics/" }, { "type": "internal", "href": "https://towardsdatascience.com/11-different-uses-of-dimensionality-reduction-4325d62b4fa6/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/rukshanpramoditha/" }, { "type": "internal", "href": "https://towardsdatascience.com/how-to-use-python-and-missforest-algorithm-to-impute-missing-data-ed45eb47cb9a/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/radecicdario/" }, { "type": "internal", "href": "https://towardsdatascience.com/are-you-a-pandas-user-who-is-curious-about-r-for-data-analysis-here-is-a-quick-start-1f88bb5d7001/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://unsplash.com/" }, { "type": "external", "href": "https://www.science.org/content/article/neutrinos-travel-faster-light-according-one-experiment" }, { "type": "external", "href": "https://www.nbcnews.com/id/wbna46760561" }, { "type": "external", "href": "https://roadmap.sh/ai-data-scientist" }, { "type": "external", "href": "https://roadmap.sh/data-analyst" }, { "type": "external", "href": "https://www.scribbr.com/statistics/hypothesis-testing/" }, { "type": "external", "href": "https://www.scribbr.com/statistics/p-value/" }, { "type": "external", "href": "https://statisticsbyjim.com/hypothesis-testing/p-hacking/" }, { "type": "external", "href": "https://occamstypewriter.org/athenedonald/2014/02/08/thinking-about-the-pipeline/" }, { "type": "external", "href": "https://www.hepi.ac.uk/wp-content/uploads/2020/07/HEPI-Policy-Note-25_PhD-students-careers_FINAL.pdf" }, { "type": "external", "href": "https://www.lib.ncsu.edu/measuring-research-impact/your-impact" }, { "type": "external", "href": "https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24810" }, { "type": "external", "href": "https://pubmed.ncbi.nlm.nih.gov/19892276/" }, { "type": "external", "href": "https://github.com/mermaid-js/mermaid" }, { "type": "external", "href": "https://www.bbc.co.uk/news/science-environment-39054778" }, { "type": "external", "href": "https://journals.sagepub.com/doi/full/10.1177/1745691616652873" }, { "type": "external", "href": "https://www.theguardian.com/society/2015/oct/26/bacon-ham-sausages-processed-meats-cancer-risk-smoking-says-who" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:34:00Z", "source_url": "https://towardsdatascience.com/the-statistical-significance-scam-db904be36714/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# The Statistical Significance Scam\nA detailed look into the flaws of science's favorite tool\n[Cai Parry-Jones](https://towardsdatascience.com/author/caiparryjones96/)\nNovember 9, 202413 min read\n![source: unsplash.com](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1fUigVfmuyNJLUFY6m-lZUw-scaled.jpeg)\nsource: [unsplash.com](https://unsplash.com/)\nStatistical significance is like the drive-thru of the research world. Roll up to the study, grab your \"significance meal,\" and boom - you've got a tasty conclusion to share with all your friends. And it isn't just convenient for the reader, it makes researchers' lives easier too. Why make the hard sell when you can say two words instead?\nBut there's a catch.\nThose fancy equations and nitty-gritty details we've conveniently avoided? They're the real meat of the matter. And when researchers and readers rely too heavily on one statistical tool, we can end up making a whopper of a mistake, like the one that nearly broke the laws of physics.\nIn 2011, physicists at the renowned CERN laboratory announced a shocking discovery: [neutrinos could travel faster than the speed of light](https://www.science.org/content/article/neutrinos-travel-faster-light-according-one-experiment). The finding threatened to overturn Einstein's theory of relativity, a cornerstone of modern physics. The researchers were confident in their results, passing physics' rigorous statistical significance threshold of 99.9999998%. Case closed, right?\nNot quite. As other scientists scrutinized the experiment, they found flaws in the methodology and ultimately [could not replicate the results](https://www.nbcnews.com/id/wbna46760561). The original finding, despite its impressive \"statistical significance,\" turned out to be false.\nIn this article, we'll delve into four critical reasons why you shouldn't instinctively trust a statistically significant finding. Moreover, why you shouldn't habitually discard non-statistically significant results.\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) and [Data Analyst](https://roadmap.sh/data-analyst) roadmaps.\n## TL;DR\nThe four key flaws of statistical significance:\n 1. **It's made up** : The statistical significance/non-significance line is all too often plucked out of thin air, or lazily taken from the general line of 95% confidence.\n 2. **It doesn't mean what (most) people think it means** : Statistical significance does not mean 'There is Y% chance X is true'.\n 3. **It's easy to hack (and frequently is)** : Randomness is frequently labeled statistically significant due to mass experiments.\n 4. **It's nothing to do with how important the result is** : Statistical significance is not related to the significance of the difference.\n\n\n## Flaw 1: It's made up\nStatistical significance is simply a line in the sand humans have created with _zero_ mathematical support. Think about that for a second. Something that is generally thought of as an objective measure is, at its core, entirely subjective.\nThe mathematical part is provided one step before deciding on the significance, via a numerical measure of confidence. The most common form used in [hypothesis testing](https://www.scribbr.com/statistics/hypothesis-testing/) is called the [p-value](https://www.scribbr.com/statistics/p-value/). This provides the actual mathematical probability that the test data results were not simply due to randomness.\nFor example, a p-value of 0.05 means there's a 5% chance of seeing these data points (or more extreme) due to random chance, or that we are 95% confident the result wasn't due to chance. For example, suppose you believe a coin is unfair in favour of heads i.e. the probability of landing on heads is greater than 50%. You toss the coin 5 times and it lands on heads each time. There's a 1/2 x 1/2 x 1/2 x 1/2 x 1/2 = 3.1% chance that it happened simply because of chance, if the coin was fair.\nBut is this enough to say it's statistically significant? It depends who you ask.\nOften, whoever is in charge of determining where the line of significance will be drawn in the sand has more influence on whether a result is significant than the underlying data itself.\nGiven this subjective final step, often in my own analysis I'd provide the reader of the study with the level of confidence percentage, rather than the binary significance/non-significance result. The final step is simply too opinion-based.\n**Sceptic** : \"_But there are standards in place for determining statistical significance._ \"\nI hear the argument a lot in response to my argument above (I talk about this quite a bit - much to the delight of my academic researcher girlfriend). To which, I respond with something like:\n**Me** : \"_Of course, if there is a specific standard you must adhere to, such as for regulatory or academic journal publishing reasons, then you have no choice but to follow the standard. But if that isn't the case then there's no reason not to._ \"\n**Sceptic** : \"_But there is a general standard. It's 95% confidence._ \"\nAt that point in the conversation I try my best not to roll my eyes. Deciding your test's statistical significance point is 95%, simply because that is the norm, is frankly lazy. It doesn't take into account the context of what is being tested.\nIn my day job, if I see someone using the 95% significance threshold for an experiment without a contextual explanation, it raises a red flag. It suggests that the person either doesn't understand the implications of their choice or doesn't care about the specific business needs of the experiment.\nAn example can best explain why this is so important.\nSuppose you work as a data scientist for a tech company, and the UI team want to know, \"Should we use the color red or blue for our 'subscribe' button to maximise out Click Through Rate (CTR)?\". The UI team favour neither color, but must choose one by the end of the week. After some A/B testing and statistical analysis we have our results:\n![Image created by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1A9Ww6nQvuY8eag5cfrBvZw.png)\nImage created by the author.\nThe follow-the-standards data scientist may come back to the UI team announcing, \"_Unfortunately, the experiment found no statistically significant difference between the click-through rate of the red and blue button._ \"\nThis is a horrendous analysis, purely due to the final subjective step. Had the data scientist taken the initiative to understand the context, critically, that 'the UI team favour neither color, but must choose one by the end of the week', then you should have set the significance point at a very high p-value, arguably 1.0 i.e. the statistical analysis doesn't matter, the UI team are happy to pick whichever color had the highest CTR.\nGiven the risk that data scientists and the like may not have the full context to determine the best point of significance, it's better (and simpler) to give the responsibility to those who have the full business context - in this example, the UI team. In other words, the data scientist should have announced to the UI team, \"_The experiment resulted with the blue button receiving a higher click-through rate, with a confidence of 94% that this wasn't attributed to random chance._ \" The final step of determining significance should be made by the UI team. Of course, this doesn't mean the data scientist shouldn't educate the team on what \"c _onfidence of 94%\"_ means, as well as clearly explaining _why_ the statistical significance is best left to them.\n## Flaw 2: It doesn't mean what (most) people think it means\nLet's assume we live in a slightly more perfect world, where point one is no longer an issue. The line in the sand figure is always perfect, huzza! Say we want to run an experiment, with the the significance line set at 99% confidence. Some weeks pass and at last we have our results and the statistical analysis finds that it's statistically significant, huzza again!.. But what does that actually mean?\nCommon belief, in the case of hypothesis testing, is that there is a 99% chance that the hypothesis is correct. This is painfully wrong. All it means is there is a 1% chance of observing data this extreme or more extreme by randomness _for this experiment_.\nStatistical significance doesn't take into account whether the experiment itself is accurate. Here are some examples of things statistical significance can't capture:\n * Sampling quality: The population sampled could be biased or unrepresentative.\n * Data quality: Measurement errors, missing data, or other data quality issues aren't addressed.\n * Assumption validity: The statistical test's assumptions (like normality, independence) could be violated.\n * Study design quality: Poor experimental controls, not controlling for confounding variables, testing multiple outcomes without adjusting significance levels.\n\n\nComing back to the example mentioned in the introduction. After failures to independently replicate the initial finding, physicists of the original 2011 experiment announced they had found a bug in their measuring device's master clock i.e. data quality issue, which resulted in a full retraction of their initial study.\nThe next time you hear a statistically significant discovery that goes against common belief, don't be so quick to believe it.\n## Flaw 3: It's easy to hack (and frequently is)\nGiven statistical significance is all about how likely something may have occurred due to randomness, an experimenter who is more interested in achieving a statistical significant result than uncovering the truth can quite easily game the system.\nThe odds of rolling two ones from two dice is (1/6 × 1/6) = 1/36, or 2.8%; a result so rare it would be classified as statistically significant by many people. But what if I throw more than two dice? Naturally, the odds of at least two ones will rise:\n * 3 dice: ≈ 7.4%\n * 4 dice: ≈ 14.4%\n * 5 dice: ≈ 23%\n * 6 dice: ≈ 32.4%\n * 7 dice: ≈ 42%\n * 8 dice: ≈ 51%\n * 12 dice: ≈ 80%*\n\n\n> *At least two dice rolling a **one** is the equivalent of: 1 (i.e. 100%, certain), minus the probability of rolling zero **ones,** minus the probability of rolling only one **one**\n> P(zero **ones**) = (5/6)^n\n> P(exactly one **one**) = n _(1/6)_ (5/6)^(n-1)\n> n is the number of dice\n> So the complete formula is: **1 - (5/6)^n - n** _**(1/6)**_**(5/6)^(n-1)**\nLet's say I run a simple experiment, with an initial theory that **one** is more likely than other numbers to be rolled. I roll 12 dice of different colors and sizes. Here are my results:\n![Image created by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1y8j26EpfI1iKs9uMJFZnew.png)\nImage created by the author.\nUnfortunately, my (calculated) hopes of getting at least two **ones** have been dashed... Actually, now that I think of it, I didn't really want two ones. I was more interested in the odds of big red dice. I believe there is a high chance of getting sixes from them. Ah! Looks like my theory is correct, the two big red dice have rolled sixes! There is only a 2.8% chance of this happening by chance. Very interesting. I shall now write a paper on my findings and aim to publish it in an academic journal that accepts my result as statistically significant.\nThis story may sound far-fetched, but the reality isn't as distant from this as you'd expect, especially in the highly regarded field of academic research. In fact, this sort of thing happens frequently enough to make a name for itself, [p-hacking](https://statisticsbyjim.com/hypothesis-testing/p-hacking/#:~:text=P%20hacking%20is%20the%20manipulation,the%20integrity%20of%20scientific%20research.).\nIf you're surprised, delving into the academic system will clarify why practices that seem abominable to the scientific method occur so frequently within the realm of science.\nAcademia is exceptionally difficult to have a successful career in. For example, In STEM subjects only [0.45% of PhD students become professors](https://occamstypewriter.org/athenedonald/2014/02/08/thinking-about-the-pipeline/). Of course, some PhD students don't want an academic career, but the majority do (67% according to [this](https://www.hepi.ac.uk/wp-content/uploads/2020/07/HEPI-Policy-Note-25_PhD-students-careers_FINAL.pdf) survey). So, roughly speaking, you have a 1% chance of making it as a professor if you have completed a PhD and want to make academia your career. Given these odds you need think of yourself as quite exceptional, or rather, you need other people to think that, since you can't hire yourself. So, how is exceptional measured?\nPerhaps unsurprisingly, the most important measure of an academic's success is their [research impact](https://www.lib.ncsu.edu/measuring-research-impact/your-impact#:~:text=Research%20impact%20is%20often%20measured,index%2C%20and%20journal%20impact%20factors.). Common measures of author impact include the h-index, g-index and i10-index. What they all have in common is they're heavily focused on citations i.e. how many times has their published work been mentioned in other published work. Knowing this, if we want to do well in academia, we need to focus on publishing research that's likely to get citations.\nYou're far [more likely to be cited if you publish your work in a highly rated academic journal](https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24810#:~:text=There%20is%20strong%20evidence%20from,tend%20to%20be%20more%20cited.). And, since [88% of top journal papers are statistically significant](https://pubmed.ncbi.nlm.nih.gov/19892276/), you're far more likely to get accepted into these journals if your research is statistically significant. This pushes a lot of well-meaning, but career-driven, academics down a slippery slope. They start out with a scientific methodology for producing research papers like so:\n![Decision-making tree for experimental research publication using the scientific method. Created by the author using Mermaid.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1Fu3Q_Dc0ULKvuha7wChW4A.png)\nDecision-making tree for experimental research publication using the scientific method. Created by the author using [Mermaid](https://github.com/mermaid-js/mermaid).\nBut end up warping their methodology to look scientific on the surface - but really, they've thrown proper scientific methods out the window:\n![Decision-making tree to maximise publication success per experiment. Created by the author using Mermaid.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1_ode5fVmw0cTZm4Q0dIaKA.png)\nDecision-making tree to maximise publication success per experiment. Created by the author using [Mermaid](https://github.com/mermaid-js/mermaid).\nGiven the decision diagrams have the researcher writing the paper _after_ discovering a significant result, there's no evidence for the journal reviewer to criticise the experiment for p-hacking.\nThat's the theory anyway. But does it really happen all that often in reality?\nThe answer is a resounding yes. In fact, the [majority of scientific research is unreproducible by fellow academics](https://www.bbc.co.uk/news/science-environment-39054778). Unreproducible means a research paper attempts to copy another research paper's experiment, but ends up with statistically unexpected results. Often finding a statistically significant result in the original paper was statistically insignificant in the replication, or in [some instances](https://journals.sagepub.com/doi/full/10.1177/1745691616652873) statistically significant in the opposite direction!\n## Flaw 4: It's nothing to do with how important the result is\nFinally, statistical significance doesn't care about the scale of the difference.\nThink about it this way - statistical significance basically just tells you \"hey, this difference probably isn't due to random chance\" but says nothing about whether the difference actually matters in the real world.\nLet's say you test a new medication and find it reduces headache pain by 0.0001% compared to a placebo. If you run this test on millions of people, that tiny difference might be statistically significant, since your sample size is massive. But... who cares about a 0.0001% reduction in pain? That's meaningless in practical terms!\nOn the other hand, you might find a drug that reduces pain by 5%, but there hasn't been a large experiment to demonstrate statistical significance. It's likely there are many examples of this in medicine because if the drug in question is cheap there is no incentive for pharmaceutical companies to run the experiment since large scale medical testing is expensive.\nThis is why it's important to look at effect size (how big the difference is) separately from statistical significance. In the real world, you want both - a difference that's unlikely to be random _and_ big enough to actually matter.\nAn example of this mistake happening time and time again is when there is a (statistically significant) discovery in carcinogens i.e. something that causes cancer. A 2015 Guardian [article](https://www.theguardian.com/society/2015/oct/26/bacon-ham-sausages-processed-meats-cancer-risk-smoking-says-who) said:\n> \"Bacon, ham and sausages rank alongside cigarettes as a major cause of cancer, the World Health Organisation has said, placing cured and processed meats in the same category as asbestos, alcohol, arsenic and tobacco.\"\nThis is straight up misinformation. Indeed, bacon, ham and sausages are in the same category as asbestos, alcohol, arsenic and tobacco. However, the categories do not denote the scale of the effect of the carcinogens, rather, how confident the World Health Organisation is that these items are carcinogens i.e. statistical significance.\nThe scale of the cancer cases caused by processed meat is questionable, since there haven't been any Randomized Controlled Trials (RCT). One of the most damning research in favour of processed meat causing cancer is a 2020 observational (think correlation, not causation) [study](https://academic.oup.com/ije/article/49/5/1540/5894731?login=false) in the UK. It found that people eating over 79 grams per day on average of red and processed meat had a 32% increased risk of bowel cancer compared to people eating less than 11 grams per day on average.\nHowever, to understand the true risk we need to understand the number of people who are at risk of bowel cancer. For every 10,000 people on the study who ate less than 11 grams of processed and red meat a day, 45 were diagnosed with bowel cancer, while it was 59 from those eating 79 grams of processed and red meat a day. That's an extra 14 cases of bowel cancer per 10,000 people, or 0.14%. The survivability in the UK of bowel cancer is [53%](https://www.cancerresearchuk.org/health-professional/cancer-statistics/statistics-by-cancer-type/bowel-cancer), so a rough estimate of carcinogens in processed meat killing you is 0.07%.\nCompare this to another substance The Guardian mention, tobacco. [Cancer Research](https://www.cancerresearchuk.org/health-professional/cancer-statistics/risk/tobacco#heading-Zero) say:\n> \"Tobacco is the largest preventable cause of cancer and death in the UK. And one of the largest preventable causes of illness and death in the world. Tobacco caused an estimated 75,800 deaths in the UK in 2021 - around a tenth (11%) of all deaths from all causes.\"\nFirst of all, wow. Don't smoke.\nSecondly, the death rate of cancer caused by tobacco is 11%/0.07% = **157** times greater than processed meat! Coming back to the quotation in the article, \"Bacon, ham and sausages rank alongside cigarettes as a major cause of cancer\". Simply, fake news.\n## Summary\nIn conclusion, while statistical significance has a place in validating quantitative research, it's crucial to understand its severe limitations.\nAs readers, we have a responsibility to approach claims of statistical significance with a critical eye. The next time you encounter a study or article touting a \"statistically significant\" finding, take a moment to ask yourself:\n 1. Is the significance threshold appropriate for the context?\n 2. How robust was the study design and data collection process?\n 3. Could the researchers have engaged in p-hacking or other questionable practices?\n 4. What is the practical significance of the effect size?\n\n\nBy asking these questions and demanding more nuanced discussions around statistical significance, we can help promote a more responsible and accurate use of the tool.\n## Over-time analysis\nI actually think the main reason statistical significance has gained such over prominence is because of the name. People associate \"statistical\" with mathematical and objective, and \"significance\" with, well, significant. I hope this article has persuaded you that these associations are merely fallacies.\nIf the scientific and wider community wanted to deal with the over prominence issue, they should seriously consider simply renaming \"statistical significance\". Perhaps \"chance-threshold test\" or \"Non-random confidence\". Then again, this would lose its Big Mac convenience.\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657617193_bio6ww.jpeg)\nWritten By\n## [Cai Parry-Jones](https://towardsdatascience.com/author/caiparryjones96/)\n[See all from Cai Parry-Jones](https://towardsdatascience.com/author/caiparryjones96/)\n[Data Quality](https://towardsdatascience.com/tag/data-quality/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Getting Started](https://towardsdatascience.com/tag/getting-started/)[Scientific Research](https://towardsdatascience.com/tag/scientific-research/)[Statistics](https://towardsdatascience.com/tag/statistics/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-statistical-significance-scam-db904be36714%2F&title=The%20Statistical%20Significance%20Scam)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-statistical-significance-scam-db904be36714%2F&title=The%20Statistical%20Significance%20Scam)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-statistical-significance-scam-db904be36714%2F&text=The%20Statistical%20Significance%20Scam)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/1GJ2rrsjBI9bKCKPfuI9tYA.jpeg)](https://towardsdatascience.com/11-different-uses-of-dimensionality-reduction-4325d62b4fa6/)\n## [11 Different Uses of Dimensionality Reduction](https://towardsdatascience.com/11-different-uses-of-dimensionality-reduction-4325d62b4fa6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThe whole ML is full of dimensionality reduction and its applications. Let's see them in action!\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nDecember 8, 202110 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/0Zk_bHXqhMjZv_COc.jpg)](https://towardsdatascience.com/how-to-use-python-and-missforest-algorithm-to-impute-missing-data-ed45eb47cb9a/)\n## [How to Use Python and MissForest Algorithm to Impute Missing Data](https://towardsdatascience.com/how-to-use-python-and-missforest-algorithm-to-impute-missing-data-ed45eb47cb9a/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nStep-by-step guide on using Random Forests to handle missing data\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nNovember 5, 20204 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1VZpKKhLeK5_r6DnRqYGkFg-scaled.jpeg)](https://towardsdatascience.com/are-you-a-pandas-user-who-is-curious-about-r-for-data-analysis-here-is-a-quick-start-1f88bb5d7001/)\n## [Are You a Pandas User Who Is Curious About R for Data Analysis? Here Is a Quick Start](https://towardsdatascience.com/are-you-a-pandas-user-who-is-curious-about-r-for-data-analysis-here-is-a-quick-start-1f88bb5d7001/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nPandas is great but it is not the only one in the game.\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nJanuary 20, 20224 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/08/0qKFh_-RmR0CLy_sV-scaled.jpg)](https://towardsdatascience.com/from-insights-to-impact-leveraging-data-science-to-maximize-customer-value-9c9af354e192/)\n## [From insights to impact: leveraging data science to maximize customer value](https://towardsdatascience.com/from-insights-to-impact-leveraging-data-science-to-maximize-customer-value-9c9af354e192/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nUplift modeling: how causal machine learning transforms customer relationships and revenue\n[Arthur Cruiziat](https://towardsdatascience.com/author/arthur.cruiziat/)\nAugust 13, 202410 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/0LtAkUbwJxTP85DDE-scaled.jpg)](https://towardsdatascience.com/are-we-thinking-about-explainability-backwards-c9a719cb1250/)\n## [Are We Thinking about eXplAInability Backwards?](https://towardsdatascience.com/are-we-thinking-about-explainability-backwards-c9a719cb1250/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThree questions you should be able to answer before building an AI solution\n[Bryce Murray, PhD](https://towardsdatascience.com/author/bryce.murray/)\nSeptember 27, 20216 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/0rhbgFdPTErrC8Ow4-scaled.jpg)](https://towardsdatascience.com/how-to-up-skill-in-data-science-3f71fafeaab7/)\n## [How To Up-Skill In Data Science](https://towardsdatascience.com/how-to-up-skill-in-data-science-3f71fafeaab7/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nMy framework for continually becoming a better data scientist\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nNovember 14, 20246 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/1tPIAiDV3kYtgS0RsW-3BgA.png)](https://towardsdatascience.com/cyberpunking-your-matplotlib-figures-96f4d473185d/)\n## [Cyberpunking Your Matplotlib Figures](https://towardsdatascience.com/cyberpunking-your-matplotlib-figures-96f4d473185d/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nTaking Your Matplotlib Figures from Boring to Interesting with a Few Lines of Code\n[Andy McDonald](https://towardsdatascience.com/author/andymcdonaldgeo/)\nMay 21, 20238 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/0ANyQM0LNCDg4vkDp-scaled.jpg)](https://towardsdatascience.com/bounding-the-sample-size-of-a-machine-learning-algorithm-d01d4775263c/)\n## [Bounding the Sample Size of a Machine Learning Algorithm](https://towardsdatascience.com/bounding-the-sample-size-of-a-machine-learning-algorithm-d01d4775263c/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nIt can be very helpful to bound the sample size of a machine learning algorithm. Read on to find out how.\n[Wilson Wang](https://towardsdatascience.com/author/wilson-wang/)\nAugust 18, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1kcU_kcYLnpMD33l-8DxYLg.jpeg)](https://towardsdatascience.com/maximize-savings-on-your-unused-fabric-capacity-4c74b12505fd/)\n## [Maximize Savings on Your Unused Fabric Capacities](https://towardsdatascience.com/maximize-savings-on-your-unused-fabric-capacity-4c74b12505fd/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAutomate your Fabric capacity state with Azure Logic Apps\n[Christabelle Santos](https://towardsdatascience.com/author/chrystasantos/)\nJuly 29, 20248 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "body_base_path": "attachments/27167390663c/pages/", "body_markdown_path": "attachments/27167390663c/pages/archived-body.md", "local_html": "attachments/27167390663c/pages/archived-body.html", "body_source": "prior_capture" }