{ "source_file": "0930", "source_platform": "Medium", "author": "Zijing Zhu, PhD", "title_listed": "Why Understanding the Data-Generation Process Is More Important Than the Data Itself", "keywords": "", "domain": "数据处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/why-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662/", "publish_date": "2023-12-07", "corpus_stage": "", "notes": "", "id": "5a20c0fbeb3c", "collected_at": "2026-10-02T08:30:45", "title": "Why Understanding the Data-Generation Process Is More Important Than the Data Itself | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Statistics](https://towardsdatascience.com/category/statistics/)\n# Why Understanding the Data-Generation Process Is More Important Than the Data Itself\n\"The Book of Why\" Chapters 5&6, a Read with Me series\n[Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/)\nDecember 7, 202316 min read\n![Photo by Ryoji Iwata on Unsplash](../images/000.jpg)\nPhoto by [Ryoji Iwata](https://unsplash.com/@ryoji__iwata?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\nDuring the early stages of infancy, our brains already learn to associate correlation with causation and try to find an explanation for everything happening around us. If a car behind us takes the same turns we do for a long time, we assume it's following us, which is a causal assumption. However, when we snap out of the movie mood, we then think we are properly just heading to the same destination - - a confounder. A common cause introduces a correlation between the two cars' movements. This vivid and relatable example that Pearl gives proves how the human brain works.\nWhat about the correlations that we couldn't fathom a reasonable explanation? Such as two diseases that are uncorrelated among the whole population but correlated among the hospitalized population. If you recall [my last article](https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818) that discussed different causal structures, it points out that conditioning the colliders (hospitalized) generates an explain-away effect that makes two uncorrelated variables spuriously correlated. In other words, the hospitalized population is not an accurate representation of the general population, and any observations made from this sample cannot be generalized.\n![Collider Bias, image by author based on "The Book of Why" Chapter 6](../images/001.png)\nCollider Bias, image by author based on \"The Book of Why\" Chapter 6\nCollider-induced correlations are not intuitive to human brains, thus generating these so-called paradoxes. In this article, I will explore more interesting paradoxes that create optical illusions in our brains like magic tricks but can be explained with causal diagrams. Understanding what's causing these paradoxes is meaningful and educational. It is the 4th article for the \"Read with Me\" series, and it's based on Chapters 5 and 6 from \"[The Book of Why](https://www.amazon.com/Book-Why-Science-Cause-Effect/dp/046509760X)\" by Judea Pearl. This should be a fun read based on all the examples Pearl gives in these two chapters! You can find the previous article here:\n * [**Kick-off: Start with A Cat Story**](https://medium.com/towards-data-science/read-with-me-a-causality-book-club-edd7085d6ae6)\n * [**Chapter1 &2: Data Tells Us 'What\" and We Always Seek for \"Why**](https://medium.com/towards-data-science/data-tells-us-what-and-we-always-seek-for-why-66da7dc3f24d)\n * [**Chapter3 &4: Causal Diagram: Confronting the Achilles' Heel in Observational Data**](https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818)\n\n\n* * *\n![Photo by Glenn Carstens-Peters on Unsplash](../images/002.jpg)\nPhoto by [Glenn Carstens-Peters](https://unsplash.com/@glenncarstenspeters?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\n### Let's make a Deal\nThe American television show \"Let's Make a Deal\" features a famous statistical problem known as the \"Monty Hall Problem,\" named after the show's host, Monty Hall. In this game, a participant is facing three doors, behind one of which is a highly sought-after prize such as a car, and behind the other two are less desirable prizes, often humorously referred to as goats. The rules are:\n * Step 1: The participant chooses one of the three doors as an initial selection, say door 1;\n * Step 2: The host, who knows what is behind each door, opens one of the other two doors to reveal the goat, say door 3;\n * Step 3: The participant has the option to stick with their original choice or switch to the remaining unopened door, i.e., the participant can stay with door 1 or choose door 2 instead.\n\n\nThe best strategy for winning the car is always to switch after the host reveals a goat behind one of the other doors. If you want to see the detailed calculation, you can find it in Question 9 of this article using Bayesian statistics.\n> [**12 Probability Practice Questions for Data Science Interviews**](https://towardsdatascience.com/12-probability-practice-questions-for-data-science-interviews-2ec5230304d9)\nOr, you can refer to the following table that shows three scenarios and the outcomes of switching or not switching, given the participant has chosen door 1. For first row shows that the car is in door 1, and the host can open either door 2 or door 3 to reveal the goat. If the participant switches, they will lose. In row 2, the host has to reveal door 3 for the goat, and similarly, the host can only open door 2 for the goat. Since the participants are equally likely to choose any door as the initial selection, comparing the columns of \"Switch\" and \"Not Switch,\" we can see switch door has a higher probability of winning at 2/3 compared to not switching at 1/3. Thus, switching doors is a preferred strategy.\n![Table by author based on "The Book of Why" Chapter 6](../images/003.png)\nTable by author based on \"The Book of Why\" Chapter 6\nThis may seem counterintuitive since it implies the show producer somehow read the participants' minds about their initial selection. The switching strategy is preferred compared to not switching, indicating the participants' initial choice is correlated with the location of the car, while intuitively speaking, these two variables should be independent of each other. What's the explanation here? We can solve this paradox through this causal diagram:\n![Image by author based on "The Book of Why" Chapter 6](../images/004.png)\nImage by author based on \"The Book of Why\" Chapter 6\nThis causal diagram shows _**collider bias**_. Step 2 of the show rule indicates that based on the car's location and the participant's first choice, the host will only choose to open the goat door. This rule actually **adds information**. The fact that the host opened door 3 instead of door 2 makes door 2 more likely to have a car behind. Conditioning on the collider, in this case, given that the host opened door 3, the participants' first choice, door 1, is less likely to be the car location since switching is a better strategy.\nWhat if we change the game rule that in Step 2, the host can open any door even if it has the car behind it? The corresponding outcome for participants who chose door 1 initially are presented in the following table with all possible scenarios:\n![Table by author based on "The Book of Why" Chapter 6](../images/005.png)\nTable by author based on \"The Book of Why\" Chapter 6\nNow, if we calculate the chances of winning from switching or not switching based on the outcomes in the last two columns, we can see that there is no difference. The winning probability for both strategies is at 1/3. Combining with the causal diagram:\n![Image by author based on "The Book of Why" Chapter 6](../images/006.png)\nImage by author based on \"The Book of Why\" Chapter 6\nModifying the rule in Step 2 removes the causal relationship between the door opened by the host and the car location. Thus, the spurious correlation between the participant's initial selection and where the show producer put the car has disappeared.\nIn this case, modifying the **data generation process** (the rules) points to different strategies even with the same **data** (the door the host opens).\n* * *\n### Revisit Simpson's Paradox\nSimpson's Paradox is also a well-known confusion widely discussed by statisticians. When we observe different relationships between two variables in the population and subpopulation, which result should we trust? For example, if a medicine is found harmful for the male group and female group separately, but beneficial for the whole population, should we still approve the medicine?\n![Photo by Brendan Church on Unsplash](../images/007.jpg)\nPhoto by [Brendan Church](https://unsplash.com/@bdchu614?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\nPearl argues\n> the confusion over Simpson's paradox is a result of **incorrect application of causal principles to statistical proportions**.\nBy using causal diagrams to illustrate different data generation processes, even when working with the same data, we can arrive at vastly different conclusions. Let's examine an example of a drug designed to reduce the risk of heart attack.\n### **Scenario #1: Trust the subgroups**\nThe table below compares heart attack rates between treatment and control groups for two gender subpopulations and the whole population with synthetic data. Note the study is not randomized.\n![Table by author based on "The Book of Why" Chapter 6](../images/008.png)\nTable by author based on \"The Book of Why\" Chapter 6\nBy comparing heart attack rates, we can see the drug harms the female and male subgroups but is helpful for the pooled. Which result should we trust? To answer this question, we need to use the following causal diagram:\n![Image by author based on "The Book of Why" Chapter 6](../images/009.png)\nImage by author based on \"The Book of Why\" Chapter 6\nThe diagram shows gender as a confounder for drug and heart attack that will bias the causal relationship without conditioning on it. In other words, in the \"Total\" row, the 22% and 18% show the biased estimations.\nTo calculate what is the drug's impact on the overall population, we need to estimate the drug's impacts on different gender groups and then take a weighted average.\nUsing the data above, we know in the control group, the female and male heart attack rate is 5% and 30%. Assuming females and males are equally frequent in the general population, the control group's heart attack rate is (5%+30%)*0.5 = 17.5%.\nSimilarly, we can get the treatment group's heart attack rate is (8%+40%)*0.5 = 24%. Overall, it is still higher than the control group, which is consistent with what we found in the subgroups. Thus, this medicine should not be approved.\n### Scenario #2: Trust the aggregation\nNow, let's move to another scenario with the same data. The only difference is that instead of separating by gender, we separated it by the patient's blood pressure levels.\n![Table by author based on "The Book of Why" Chapter 6](../images/010.png)\nTable by author based on \"The Book of Why\" Chapter 6\nHere, the data reveals the same contradictions between the separated and aggregated groups. However, the different causal diagram changes our conclusion. The idea is that the drug is designed to reduce the heart attack risk by reducing a patient's blood pressure:\n![Image by author based on "The Book of Why" Chapter 6r](../images/011.png)\nImage by author based on \"The Book of Why\" Chapter 6r\nThe causal diagram shows blood pressure as a mediator instead of a confounder. Recall what we discussed in [the last article](https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818): controlling the mediator or the proxy of the mediator will block the causal impact completely or partially.\nThus, pooling the data would be appropriate. We can conclude that the drug is beneficial as it reduces heart attack risk from 22% to 18% in the treatment group compared to the control group.\n* * *\n### Simpson's Paradox in Continuous Variables\nNow, we have seen the importance of examining beyond data to look at the data-generation process when deciding whether to aggregate or separate by groups. Let's see an example that applies the same concept in continuous variables.\nThe paradox was first presented by Frederic Lord in 1967 when studying the effects of the diet provided by different dining halls, specifically, how the diet affects girls and boys differently.\nThe student's weight is measured in September as the initial weight and again in the following June to calculate the weight gained. The result is shown in the following graph:\n![Image by author based on "The Book of Why" Chapter 6](../images/012.png)\nImage by author based on \"The Book of Why\" Chapter 6\nThe two ellipses represent the scatter plots of initial weight and final weight for girls and boys. The graph itself shows contradicting conclusions in terms of the diet's impact on girls and boys:\n * If we focus on the 45-degree line where W_f =W_i, we can see that, on average, the diet has no impact on both girls and boys because the weight distribution is symmetric at W_f =W_i;\n * If we argue that initial weight is influential to the final weight and focus on each initial weight observation, such as W0, we can see that the vertical line intersects the boys' ellipse higher up than the girls'. In other words, on average, boys starting with weight W0 will reach a greater final weight than girls starting with the same initial weight.\n\n\nThe graph shows that boys gain more weight than girls starting with every initial weight. Yet it's equally obvious that both boys and girls gained nothing at all. These two conclusions give different guidance to school dietitians. Which decision should they make? That's when the following causal diagram comes in handy:\n![Image by author based on "The Book of Why" Chapter 6](../images/013.png)\nImage by author based on \"The Book of Why\" Chapter 6\nWhen studying whether sex has impacted weight gain, the causal diagram shows there is no backdoor path. So, the observed, aggregated data provide the accurate conclusion that the diet has no impact on both girls and boys. If we control initial weight, we will control a mediator that actually blocks part of the causal impact from sex to weight gain through initial weight.\nThis conclusion also proves that the overall gain is not always the average of stratum-specific gains (**sure-thing principle**). If the number of girls and boys is the same for each strata, i.e., each initial weight, then what we observed at each initial weight can be generalized to the whole population. It is clearly not the case here since the distributions of girls and boys are different at each initial weight.\nWhat if we put different narratives in the same data? Instead of observing how sex impact weight gain, we can serve different diet at different dining halls and compare the weight gain for students eating in different dining halls. The graph now becomes:\n![Image by author based on "The Book of Why" Chapter 6](../images/014.png)\nImage by author based on \"The Book of Why\" Chapter 6\nWe can see that students who go to dining hall B have higher initial weights. The same contradicting conclusion can be drawn since the data is the same. However, the causal diagram has changed:\n![Image by author based on "The Book of Why" Chapter 6](../images/015.png)\nImage by author based on \"The Book of Why\" Chapter 6\nNow, there is a backdoor path to diet and weight gain. Initial weight has become a confounder that will bias the causal relationship between diet and weight gain if we fail to control it.\nThus, we should go with the conclusion that takes initial weights into consideration. From the graph, we can see on average, students going to dining hall B gained more weight than students going to dining hall A.\nFrom all these examples above, we can see that Simpson's Paradox is not just a reversal that we find in real-world data sets all the time. By reversal, I mean if we observe A/B > a/b, and C/D > c/d, it is not guaranteed that (A+C)/(B+D) > (a+c)/(b+d). It is merely rules in arithmetic inequality.\nHowever, Simpson's Paradox is beyond mathematic calculation. It is necessary to thoroughly analyze the causal model to comprehend the data-generating process rather than solely relying on the observed data when facing contradictory conclusions.\n* * *\n### The long debate between smoking and lung cancer\nBesides providing real-life examples in Chapter 6 to illustrate the importance of understanding the data-generation process, Pearl also explores and focuses on the causal relationship between smoking and lung cancer in Chapter 5 to prove the point.\n![Photo by Volodymyr Hryshchenko on Unsplash](../images/body-709197ba5648be8e7a52.jpg)\nPhoto by [Volodymyr Hryshchenko](https://unsplash.com/@lunarts?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\nFrom the lengthy debate on whether smoking is a causal factor, we can infer how the theory developed and causality was brought to the front stage over time.\nThe studies on whether smoking causes lung cancer are challenging due to at least these three factors:\n * It is unethical to run RCT that forces participants to smoke and expose them to potential harm to health;\n * Observational data are likely to have confounders since the distribution of the treatment group can be fundamentally different from the control group.\n * The potential obstacles and deliberate efforts the tobacco companies can put to deceive the public about the health risks.\n\n\nIn the past decades, statisticians have had multiple trials to find the correct methodology, even facing these obstacles.\n### A case-control study\nAs Pearl summarizes in his book, it began with a case-control study that compared individuals with lung cancer to a control group of healthy volunteers. The study then determined whether the \"cases\" in the treatment group had more smokers than in the control group, taking into consideration some confounders like age, sex, and exposure to environmental pollutants.\nHowever, this study faced several drawbacks:\n * **A backward logic** : instead of answering what is the probability that a smoker can get cancer, it is showing what is the probability that a cancer patient is a smoker;\n * **Recall bias** : Even though the researcher made sure not to let the patients know about their diagnosis prior to the survey, they can probably infer it based on their own health condition;\n * **Selection bias** : The study is only focused on hospitalized cancer patients, which is not a good representation of the whole population or even the smoking population.\n\n\n### Cornfield's inequality\nThe debate continues when questions arise regarding whether smokers are \"self-selecting\" that they are genetically different from nonsmokers. Some concerns Pearl mentions in his book are\"\n> They are more risk-taking or more likely to develop addictive behaviors that actually cause adverse health effects.\nIn other words, the debate goes to which of the following causal diagrams is correct. Is the \"smoking gene\" the only reason we observe relationships between \"smoking\" and \"lung cancer\"?\n![Image by author based on "The Book of Why" Chapter 5](../images/body-9aafaaddc1d1d2c258df.png)\nImage by author based on \"The Book of Why\" Chapter 5\nCornfield challenged the left causal diagram using simple math. If the left causal diagram is true and in the data, we observe smokers are **nine times** more likely to develop lung cancer, it must be because they are **nine times** more likely to have the smoking gene than non-smokers.\nAs illustrated in Pearl's book referring to Cornfield' argument:\n> If 11% of non-smokers have the smoking gene, then 99% of the smokers would have to have it. However, the percentage becomes mathematically impossible if we observe that 12% of non-smokers have the smoking gene.\nIn this case, the smoking gene cannot account fully for the relationship between smoking and lung cancer. Thus, the right causal diagram provides a more accurate explanation.\nNaming after its proposer, this argument is called Cornfield's inequality, which sheds light on drawing inferences by evaluating how strong alternative relationships must be in the absence of causality to explain the observed data.\n### The birth-weight paradox\nI hope we have already established a foundation well enough to claim that data is not enough to understand causality and explain real-world scenarios that present as paradoxes.\nAnother example that Pearl gives in his book is a study that focused on finding how mothers who smoke affect their babies' survival rates, challenging the argument that smoke causes mortality. The findings show that babies of smoking mothers indeed weigh less. However, the low-birth-weight babies of smoking mothers had a better survival rate than those of non-smokers.\nThe finding might look counterintuitive that smoking reduces birth weight and reduces child mortality rate. However, if we think further about what are the factors that contribute to babies' birth weights other than smoking, we will find the collider bias here.\nFor example, other life-threatening genetic abnormalities might also explain the lower birth rate. We can see it more clearly by drawing the causal diagram:\n![Image by author based on "The Book of Why" Chapter 5](../images/body-fcdeab7c6e89b5da52ba.png)\nImage by author based on \"The Book of Why\" Chapter 5\nHere, \"smoking -> birth weight <- birth defect\" forms a collider structure. Smoking has no correlation with birth defects unless we control birth weight. If we only look at low-birth-weight babies, we are opening up the backdoor path of smoking to mortality. The relationship will be biased by the birth defects to mortality causal path.\nIndeed, other factors can add to the causal diagram that will point to birth weight. For example, studies have found _**race**_ is another factor that affects both birth weight and child mortality.\nThe list can continue, only one thing we need to keep in mind in terms of what's the **action point** taken from the causal diagram. As Pearl puts in his book:\n> We can use this diagram and data to persuade a pregnant woman to stop smoking. But we can't tell a pregnant black woman to stop being black.\n* * *\n### Last remarks\nCollider bias is everywhere. Have you noticed from your friends' experience that among all the people they date, the attractive ones tend to be jerks? Instead of utilizing psychosocial theories involving traumas and personality disorders, Pearl shows that **collider bias** can be the answer. Your friends' choices of people to date depend on two factors: attractiveness and personality. In this case, you will date a mean, attractive person, a nice, unattractive person, and certainly a nice, attractive person. However, you will never date a mean, unattractive person. As illustrated by the following causal diagram:\n![Image by author based on "The Book of Why" Chapter 6](../images/body-73ce6306f60e49f70eb1.png)\nImage by author based on \"The Book of Why\" Chapter 6\nBy only looking at your friend's dating pool, you are controlling a collider that will create a spurious negative correlation between attractiveness and personality.\nJust as Pearl says, even though the sad fact is that unattractive people are as mean as attractive people in the whole population (Attractiveness and Personality are not correlated), your friends never realize it because the dating pool never has somebody who is mean and unattractive.\nThis concludes this article full of fun facts and explanations of paradoxes using causal diagrams. Paradoxes are data we observe that trick and confuse us. However, by digging into the data generation process and drawing the causal diagrams, we can provide explanations of these paradoxes, which not only clears the confusion but also gives us trustworthy guidance into decision-making.\n* * *\nThat's all I want to share from Chapters 5 and 6 of \"The Book of Why\" by Judea Pearl, which completes the 4th article in this \"Read with Me\" series. I hope this article is helpful to you. If you haven't read the first three articles, check them out here:\n> [**Read with Me: A Causality Book Club**](https://towardsdatascience.com/read-with-me-a-causality-book-club-edd7085d6ae6)\n> [**Data Tells Us \"What\" and We Always Seek for \"Why\"**](https://towardsdatascience.com/data-tells-us-what-and-we-always-seek-for-why-66da7dc3f24d)\n> [**Causal Diagram: Confronting the Achilles' Heel in Observational Data**](https://towardsdatascience.com/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818)\n> [**You Can't Step in the Same River Twice**](https://towardsdatascience.com/you-cant-step-in-the-same-river-twice-cfacf7cee305)\n> [**What Makes A Strong AI?**](https://towardsdatascience.com/what-makes-a-strong-ai-012315722793)\n> [**How is Causal Inference Different in Academia and Industry?**](https://towardsdatascience.com/how-is-causal-inference-different-in-academia-and-industry-fb5afd12e2e7)\nIf interested, [subscribe to my email list](https://zzhu17.medium.com/subscribe) to join the currently ongoing biweekly discussions. As always, I highly encourage you to read, think, and share your main takeaways here or on [your own blog](https://youtu.be/oFDl0-SKAL8?si=r1QiRIizhDBdvX-A).\nThanks for reading. If you like this article, don't forget to:\n * _**Check my recent articles about**_[ _**the 4Ds in data storytelling: making art out of science**_](https://medium.com/towards-data-science/the-4ds-in-data-storytelling-making-art-out-of-science-c4998ed7875e); _[**continuous learning in data science**](https://towardsdatascience.com/continuous-learning-a-data-scientists-odyssey-8d3006c2ce01?source=post_page-----ac6e01fe5cc3--------------------------------)**;**[**how I become a data scientist**](https://towardsdatascience.com/how-i-became-a-data-scientist-7f5b10606612?source=post_page-----ac6e01fe5cc3--------------------------------)**;** _\n * _**Check my**[**other articles**](https://zzhu17.medium.com/my-blog-posts-gallery-ac6e01fe5cc3)**on different topics like**[**data science interview preparation**](https://zzhu17.medium.com/list/data-science-interview-preparation-bfb0986a61a3)**;**[**causal inference**](https://towardsdatascience.com/causal-inference-what-why-and-how-d7336b0081ec?source=post_page-----ac6e01fe5cc3--------------------------------)**;** _\n * [_**Subscribe**_](https://zzhu17.medium.com/subscribe) _**to my email list;**_\n * [_**Sign up for medium membership**_](https://zzhu17.medium.com/membership) _**;**_\n * _**Or follow me on**_[ _**YouTube**_](https://youtube.com/channel/UCMs6go1pvY5OOy1DXVtMo5A) _**and watch my most recent YouTube video about other books I read:**_\n\n\nPlease accept cookies to access this content\nPlease accept cookies to access this content\n## Reference\n[_**The Book of Why**_](https://www.amazon.com/Book-Why-Science-Cause-Effect/dp/046509760X) _**by Judea Pearl**_\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-6091039d5533a6240825.png)\nWritten By\n## [Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/)\n[See all from Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Causal Data Science](https://towardsdatascience.com/tag/causal-data-science/)[Causal Inference](https://towardsdatascience.com/tag/causal-inference/)[Causality](https://towardsdatascience.com/tag/causality/)[Statistics](https://towardsdatascience.com/tag/statistics/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fwhy-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662%2F&title=Why%20Understanding%20the%20Data-Generation%20Process%20Is%20More%20Important%20Than%20the%20Data%20Itself)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fwhy-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662%2F&title=Why%20Understanding%20the%20Data-Generation%20Process%20Is%20More%20Important%20Than%20the%20Data%20Itself)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fwhy-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662%2F&text=Why%20Understanding%20the%20Data-Generation%20Process%20Is%20More%20Important%20Than%20the%20Data%20Itself)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/016.jpeg)](https://towardsdatascience.com/22-statistics-questions-to-prepare-for-data-science-interviews-d5651a8b3c56/)\n## [22 Statistics Questions to Prepare for Data Science Interviews](https://towardsdatascience.com/22-statistics-questions-to-prepare-for-data-science-interviews-d5651a8b3c56/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nNail data science interviews with confidence, part 2\n[Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/)\nOctober 21, 202010 min read\n * [![](../images/017.png)](https://towardsdatascience.com/the-art-of-the-phillips-curve/)\n## [The Art of the Phillips Curve](https://towardsdatascience.com/the-art-of-the-phillips-curve/)\n[Economics](https://towardsdatascience.com/category/society/economics/)\nThe subjective details holding together one of economics’ favourite models\n[Cai Parry-Jones](https://towardsdatascience.com/author/caiparryjones96/)\nMay 12, 202515 min read\n * [![](../images/018.jpeg)](https://towardsdatascience.com/famous-italian-solitaire-simulation-and-statistics-using-python-b10d60b34fc6/)\n## [Famous Italian solitaire simulation and statistics using Python](https://towardsdatascience.com/famous-italian-solitaire-simulation-and-statistics-using-python-b10d60b34fc6/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nImplementing a solitaire simulator and extracting the winning probabilities out of it, with few lines of (Python) codes\n[Piero Paialunga](https://towardsdatascience.com/author/piero-paialunga/)\nJanuary 24, 20215 min read\n * [![](../images/019.jpg)](https://towardsdatascience.com/poisson-distribution-and-poisson-process-in-python-statistics-a7aa573c03f4/)\n## [Poisson Distribution and Poisson Process in Python - Statistics](https://towardsdatascience.com/poisson-distribution-and-poisson-process-in-python-statistics-a7aa573c03f4/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIn this article we will explore Poisson distribution and Poisson process in Python.\n[Misha Sv](https://towardsdatascience.com/author/pyshark/)\nNovember 23, 20217 min read\n * [![](../images/020.jpeg)](https://towardsdatascience.com/partial-correlation-508353cd8b5/)\n## [Partial correlation-the idea and significance](https://towardsdatascience.com/partial-correlation-508353cd8b5/)\n[Statistics](https://towardsdatascience.com/category/statistics/)\nIs the correlation between two variables reliable?\n[Xichu Zhang](https://towardsdatascience.com/author/xichu.zhang/)\nAugust 14, 20216 min read\n * [![](../images/021.jpeg)](https://towardsdatascience.com/histograms-and-density-plots-in-python-f6bda88f5ac0/)\n## [Histograms and Density Plots in Python](https://towardsdatascience.com/histograms-and-density-plots-in-python-f6bda88f5ac0/)\n[Programming](https://towardsdatascience.com/category/programming/)\nVisualizing One-Dimensional Data in Python Plotting a single variable seems like it should be easy....\n[Will Koehrsen](https://towardsdatascience.com/author/will%20koehrsen/)\nMarch 23, 201811 min read\n * [![](../images/022.jpg)](https://towardsdatascience.com/where-hurricanes-hit-hardest-a-county-level-analysis-with-python/)\n## [Where Hurricanes Hit Hardest: A County-Level Analysis with Python](https://towardsdatascience.com/where-hurricanes-hit-hardest-a-county-level-analysis-with-python/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nFrom storm tracks to insurance risk\n[Lee Vaughan](https://towardsdatascience.com/author/lee_vaughan/)\nAugust 21, 202514 min read\n * [![](../images/023.png)](https://towardsdatascience.com/logic-and-implementation-of-a-spam-filter-machine-learning-algorithm-a508fb9547bd/)\n## [Logic and implementation of a spam filter machine learning algorithm](https://towardsdatascience.com/logic-and-implementation-of-a-spam-filter-machine-learning-algorithm-a508fb9547bd/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA beginner's guide to spam classification\n[Lily Chen](https://towardsdatascience.com/author/lilychencodes/)\nDecember 7, 20208 min read\n * [![](../images/024.jpeg)](https://towardsdatascience.com/probability-learning-ii-how-bayes-theorem-is-applied-in-machine-learning-bd747a960962/)\n## [Probability Learning II: How Bayes' Theorem is applied in Machine Learning](https://towardsdatascience.com/probability-learning-ii-how-bayes-theorem-is-applied-in-machine-learning-bd747a960962/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLearn how Bayes Theorem is in Machine Learning for classification and regression!\n[James Thorn](https://towardsdatascience.com/author/james-thorn/)\nAugust 14, 20199 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "metadata": { "title": "Why Understanding the Data-Generation Process Is More Important Than the Data Itself | Towards Data Science", "description": "\"The Book of Why\" Chapters 5&6, a Read with Me series", "keywords": null, "author": "Zijing Zhu, PhD", "og:locale": "en_US", "og:type": "article", "og:title": "Why Understanding the Data-Generation Process Is More Important Than the Data Itself", "og:description": "\"The Book of Why\" Chapters 5&6, a Read with Me series", "og:url": "https://towardsdatascience.com/why-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/04KRL6wDjLyr6Y67u.jpg", "og:image:width": "1709", "og:image:height": "2560", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Zijing Zhu, PhD", "twitter:label2": "Est. reading time", "twitter:data2": "16 min read", "article:published_time": "2023-12-07T07:19:50.000Z", "article:modified_time": "2023-12-07T07:19:50.000Z", "article:author": "https://towardsdatascience.com/author/zzhu17/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:09:22Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:09:22Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:09:22Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T18:09:22Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=oFDl0-SKAL8", "status": "ok", "local_path": "attachments/5a20c0fbeb3c/videos/repair-oFDl0-SKAL8.mp4", "bytes": 352331571, "valid": true, "height": 1080, "width": 1920, "duration": "1437.941000" }, { "url": "https://youtube.com/channel/UCMs6go1pvY5OOy1DXVtMo5A", "kind": "channel_reference", "status": "reference_only" }, { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/statistics/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/zzhu17/" }, { "type": "internal", "href": "https://towardsdatascience.com/12-probability-practice-questions-for-data-science-interviews-2ec5230304d9" }, { "type": "internal", "href": "https://towardsdatascience.com/read-with-me-a-causality-book-club-edd7085d6ae6" }, { "type": "internal", "href": "https://towardsdatascience.com/data-tells-us-what-and-we-always-seek-for-why-66da7dc3f24d" }, { "type": "internal", "href": "https://towardsdatascience.com/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818" }, { "type": "internal", "href": "https://towardsdatascience.com/you-cant-step-in-the-same-river-twice-cfacf7cee305" }, { "type": "internal", "href": "https://towardsdatascience.com/what-makes-a-strong-ai-012315722793" }, { "type": "internal", "href": "https://towardsdatascience.com/how-is-causal-inference-different-in-academia-and-industry-fb5afd12e2e7" }, { "type": "internal", "href": "https://towardsdatascience.com/continuous-learning-a-data-scientists-odyssey-8d3006c2ce01?source=post_page-----ac6e01fe5cc3--------------------------------" }, { "type": "internal", "href": "https://towardsdatascience.com/how-i-became-a-data-scientist-7f5b10606612?source=post_page-----ac6e01fe5cc3--------------------------------" }, { "type": "internal", "href": "https://towardsdatascience.com/causal-inference-what-why-and-how-d7336b0081ec?source=post_page-----ac6e01fe5cc3--------------------------------" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/deep-dives/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/causal-data-science/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://unsplash.com/@ryoji__iwata" }, { "type": "external", "href": "https://unsplash.com" }, { "type": "external", "href": "https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818" }, { "type": "external", "href": "https://www.amazon.com/Book-Why-Science-Cause-Effect/dp/046509760X" }, { "type": "external", "href": "https://medium.com/towards-data-science/read-with-me-a-causality-book-club-edd7085d6ae6" }, { "type": "external", "href": "https://medium.com/towards-data-science/data-tells-us-what-and-we-always-seek-for-why-66da7dc3f24d" }, { "type": "external", "href": "https://unsplash.com/@glenncarstenspeters" }, { "type": "external", "href": "https://unsplash.com/@bdchu614" }, { "type": "external", "href": "https://unsplash.com/@lunarts" }, { "type": "external", "href": "https://zzhu17.medium.com/subscribe" }, { "type": "external", "href": "https://youtu.be/oFDl0-SKAL8?si=r1QiRIizhDBdvX-A" }, { "type": "external", "href": "https://medium.com/towards-data-science/the-4ds-in-data-storytelling-making-art-out-of-science-c4998ed7875e" }, { "type": "external", "href": "https://zzhu17.medium.com/my-blog-posts-gallery-ac6e01fe5cc3" }, { "type": "external", "href": "https://zzhu17.medium.com/list/data-science-interview-preparation-bfb0986a61a3" }, { "type": "external", "href": "https://zzhu17.medium.com/membership" }, { "type": "external", "href": "https://youtube.com/channel/UCMs6go1pvY5OOy1DXVtMo5A" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=Why+Understanding+the+Data-Generation+Process+Is+More+Important+Than+the+Data+Itself&u=https%3A%2F%2Ftowardsdatascience.com%2Fwhy-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662%2F" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:33:58Z", "source_url": "https://towardsdatascience.com/why-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [ { "local_path": "attachments/5a20c0fbeb3c/videos/repair-oFDl0-SKAL8.mp4", "bytes": 352331571, "valid": true, "height": 1080, "width": 1920, "duration": "1437.941000", "status": "ok" } ], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Statistics](https://towardsdatascience.com/category/statistics/)\n# Why Understanding the Data-Generation Process Is More Important Than the Data Itself\n\"The Book of Why\" Chapters 5&6, a Read with Me series\n[Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/)\nDecember 7, 202316 min read\n![Photo by Ryoji Iwata on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/04KRL6wDjLyr6Y67u.jpg)\nPhoto by [Ryoji Iwata](https://unsplash.com/@ryoji__iwata?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\nDuring the early stages of infancy, our brains already learn to associate correlation with causation and try to find an explanation for everything happening around us. If a car behind us takes the same turns we do for a long time, we assume it's following us, which is a causal assumption. However, when we snap out of the movie mood, we then think we are properly just heading to the same destination - - a confounder. A common cause introduces a correlation between the two cars' movements. This vivid and relatable example that Pearl gives proves how the human brain works.\nWhat about the correlations that we couldn't fathom a reasonable explanation? Such as two diseases that are uncorrelated among the whole population but correlated among the hospitalized population. If you recall [my last article](https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818) that discussed different causal structures, it points out that conditioning the colliders (hospitalized) generates an explain-away effect that makes two uncorrelated variables spuriously correlated. In other words, the hospitalized population is not an accurate representation of the general population, and any observations made from this sample cannot be generalized.\n![Collider Bias, image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1ZEAk1UHSzM9OLNHZP1cGUQ.png)\nCollider Bias, image by author based on \"The Book of Why\" Chapter 6\nCollider-induced correlations are not intuitive to human brains, thus generating these so-called paradoxes. In this article, I will explore more interesting paradoxes that create optical illusions in our brains like magic tricks but can be explained with causal diagrams. Understanding what's causing these paradoxes is meaningful and educational. It is the 4th article for the \"Read with Me\" series, and it's based on Chapters 5 and 6 from \"[The Book of Why](https://www.amazon.com/Book-Why-Science-Cause-Effect/dp/046509760X)\" by Judea Pearl. This should be a fun read based on all the examples Pearl gives in these two chapters! You can find the previous article here:\n * [**Kick-off: Start with A Cat Story**](https://medium.com/towards-data-science/read-with-me-a-causality-book-club-edd7085d6ae6)\n * [**Chapter1 &2: Data Tells Us 'What\" and We Always Seek for \"Why**](https://medium.com/towards-data-science/data-tells-us-what-and-we-always-seek-for-why-66da7dc3f24d)\n * [**Chapter3 &4: Causal Diagram: Confronting the Achilles' Heel in Observational Data**](https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818)\n\n\n* * *\n![Photo by Glenn Carstens-Peters on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/0D54YlImk6GD2rbQ1-scaled.jpg)\nPhoto by [Glenn Carstens-Peters](https://unsplash.com/@glenncarstenspeters?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\n### Let's make a Deal\nThe American television show \"Let's Make a Deal\" features a famous statistical problem known as the \"Monty Hall Problem,\" named after the show's host, Monty Hall. In this game, a participant is facing three doors, behind one of which is a highly sought-after prize such as a car, and behind the other two are less desirable prizes, often humorously referred to as goats. The rules are:\n * Step 1: The participant chooses one of the three doors as an initial selection, say door 1;\n * Step 2: The host, who knows what is behind each door, opens one of the other two doors to reveal the goat, say door 3;\n * Step 3: The participant has the option to stick with their original choice or switch to the remaining unopened door, i.e., the participant can stay with door 1 or choose door 2 instead.\n\n\nThe best strategy for winning the car is always to switch after the host reveals a goat behind one of the other doors. If you want to see the detailed calculation, you can find it in Question 9 of this article using Bayesian statistics.\n> [**12 Probability Practice Questions for Data Science Interviews**](https://towardsdatascience.com/12-probability-practice-questions-for-data-science-interviews-2ec5230304d9)\nOr, you can refer to the following table that shows three scenarios and the outcomes of switching or not switching, given the participant has chosen door 1. For first row shows that the car is in door 1, and the host can open either door 2 or door 3 to reveal the goat. If the participant switches, they will lose. In row 2, the host has to reveal door 3 for the goat, and similarly, the host can only open door 2 for the goat. Since the participants are equally likely to choose any door as the initial selection, comparing the columns of \"Switch\" and \"Not Switch,\" we can see switch door has a higher probability of winning at 2/3 compared to not switching at 1/3. Thus, switching doors is a preferred strategy.\n![Table by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/17bEObceydJhoaIwdxeXOrw.png)\nTable by author based on \"The Book of Why\" Chapter 6\nThis may seem counterintuitive since it implies the show producer somehow read the participants' minds about their initial selection. The switching strategy is preferred compared to not switching, indicating the participants' initial choice is correlated with the location of the car, while intuitively speaking, these two variables should be independent of each other. What's the explanation here? We can solve this paradox through this causal diagram:\n![Image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1zITjNVOaNyY2jBhptYXj5w.png)\nImage by author based on \"The Book of Why\" Chapter 6\nThis causal diagram shows _**collider bias**_. Step 2 of the show rule indicates that based on the car's location and the participant's first choice, the host will only choose to open the goat door. This rule actually **adds information**. The fact that the host opened door 3 instead of door 2 makes door 2 more likely to have a car behind. Conditioning on the collider, in this case, given that the host opened door 3, the participants' first choice, door 1, is less likely to be the car location since switching is a better strategy.\nWhat if we change the game rule that in Step 2, the host can open any door even if it has the car behind it? The corresponding outcome for participants who chose door 1 initially are presented in the following table with all possible scenarios:\n![Table by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1C-HqgWKKWccX_XapEdK6DQ.png)\nTable by author based on \"The Book of Why\" Chapter 6\nNow, if we calculate the chances of winning from switching or not switching based on the outcomes in the last two columns, we can see that there is no difference. The winning probability for both strategies is at 1/3. Combining with the causal diagram:\n![Image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1GNEywDqXUCcEbLWJewM3eA.png)\nImage by author based on \"The Book of Why\" Chapter 6\nModifying the rule in Step 2 removes the causal relationship between the door opened by the host and the car location. Thus, the spurious correlation between the participant's initial selection and where the show producer put the car has disappeared.\nIn this case, modifying the **data generation process** (the rules) points to different strategies even with the same **data** (the door the host opens).\n* * *\n### Revisit Simpson's Paradox\nSimpson's Paradox is also a well-known confusion widely discussed by statisticians. When we observe different relationships between two variables in the population and subpopulation, which result should we trust? For example, if a medicine is found harmful for the male group and female group separately, but beneficial for the whole population, should we still approve the medicine?\n![Photo by Brendan Church on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/0Jt3GG83wlcOA96qW-scaled.jpg)\nPhoto by [Brendan Church](https://unsplash.com/@bdchu614?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\nPearl argues\n> the confusion over Simpson's paradox is a result of **incorrect application of causal principles to statistical proportions**.\nBy using causal diagrams to illustrate different data generation processes, even when working with the same data, we can arrive at vastly different conclusions. Let's examine an example of a drug designed to reduce the risk of heart attack.\n### **Scenario #1: Trust the subgroups**\nThe table below compares heart attack rates between treatment and control groups for two gender subpopulations and the whole population with synthetic data. Note the study is not randomized.\n![Table by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1a1K0-j_HL_RfUc25cRNEZg.png)\nTable by author based on \"The Book of Why\" Chapter 6\nBy comparing heart attack rates, we can see the drug harms the female and male subgroups but is helpful for the pooled. Which result should we trust? To answer this question, we need to use the following causal diagram:\n![Image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1AyirUHtI_BCj0XycEXCIkg.png)\nImage by author based on \"The Book of Why\" Chapter 6\nThe diagram shows gender as a confounder for drug and heart attack that will bias the causal relationship without conditioning on it. In other words, in the \"Total\" row, the 22% and 18% show the biased estimations.\nTo calculate what is the drug's impact on the overall population, we need to estimate the drug's impacts on different gender groups and then take a weighted average.\nUsing the data above, we know in the control group, the female and male heart attack rate is 5% and 30%. Assuming females and males are equally frequent in the general population, the control group's heart attack rate is (5%+30%)*0.5 = 17.5%.\nSimilarly, we can get the treatment group's heart attack rate is (8%+40%)*0.5 = 24%. Overall, it is still higher than the control group, which is consistent with what we found in the subgroups. Thus, this medicine should not be approved.\n### Scenario #2: Trust the aggregation\nNow, let's move to another scenario with the same data. The only difference is that instead of separating by gender, we separated it by the patient's blood pressure levels.\n![Table by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1YdN5F7d8OO-DcnJsUxSRAQ.png)\nTable by author based on \"The Book of Why\" Chapter 6\nHere, the data reveals the same contradictions between the separated and aggregated groups. However, the different causal diagram changes our conclusion. The idea is that the drug is designed to reduce the heart attack risk by reducing a patient's blood pressure:\n![Image by author based on "The Book of Why" Chapter 6r](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1SUjIEjZsPWOj0lutuOBl1A.png)\nImage by author based on \"The Book of Why\" Chapter 6r\nThe causal diagram shows blood pressure as a mediator instead of a confounder. Recall what we discussed in [the last article](https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818): controlling the mediator or the proxy of the mediator will block the causal impact completely or partially.\nThus, pooling the data would be appropriate. We can conclude that the drug is beneficial as it reduces heart attack risk from 22% to 18% in the treatment group compared to the control group.\n* * *\n### Simpson's Paradox in Continuous Variables\nNow, we have seen the importance of examining beyond data to look at the data-generation process when deciding whether to aggregate or separate by groups. Let's see an example that applies the same concept in continuous variables.\nThe paradox was first presented by Frederic Lord in 1967 when studying the effects of the diet provided by different dining halls, specifically, how the diet affects girls and boys differently.\nThe student's weight is measured in September as the initial weight and again in the following June to calculate the weight gained. The result is shown in the following graph:\n![Image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1EysZtVcwkGC6hMWLPDqcSA.png)\nImage by author based on \"The Book of Why\" Chapter 6\nThe two ellipses represent the scatter plots of initial weight and final weight for girls and boys. The graph itself shows contradicting conclusions in terms of the diet's impact on girls and boys:\n * If we focus on the 45-degree line where W_f =W_i, we can see that, on average, the diet has no impact on both girls and boys because the weight distribution is symmetric at W_f =W_i;\n * If we argue that initial weight is influential to the final weight and focus on each initial weight observation, such as W0, we can see that the vertical line intersects the boys' ellipse higher up than the girls'. In other words, on average, boys starting with weight W0 will reach a greater final weight than girls starting with the same initial weight.\n\n\nThe graph shows that boys gain more weight than girls starting with every initial weight. Yet it's equally obvious that both boys and girls gained nothing at all. These two conclusions give different guidance to school dietitians. Which decision should they make? That's when the following causal diagram comes in handy:\n![Image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1i9l8xV3BTnI6Jlwkuo4znw.png)\nImage by author based on \"The Book of Why\" Chapter 6\nWhen studying whether sex has impacted weight gain, the causal diagram shows there is no backdoor path. So, the observed, aggregated data provide the accurate conclusion that the diet has no impact on both girls and boys. If we control initial weight, we will control a mediator that actually blocks part of the causal impact from sex to weight gain through initial weight.\nThis conclusion also proves that the overall gain is not always the average of stratum-specific gains (**sure-thing principle**). If the number of girls and boys is the same for each strata, i.e., each initial weight, then what we observed at each initial weight can be generalized to the whole population. It is clearly not the case here since the distributions of girls and boys are different at each initial weight.\nWhat if we put different narratives in the same data? Instead of observing how sex impact weight gain, we can serve different diet at different dining halls and compare the weight gain for students eating in different dining halls. The graph now becomes:\n![Image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1oBxSw4cmbJn4YVNbD8Og2A.png)\nImage by author based on \"The Book of Why\" Chapter 6\nWe can see that students who go to dining hall B have higher initial weights. The same contradicting conclusion can be drawn since the data is the same. However, the causal diagram has changed:\n![Image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1y61At2KBzlV0IpMlvxfL1g.png)\nImage by author based on \"The Book of Why\" Chapter 6\nNow, there is a backdoor path to diet and weight gain. Initial weight has become a confounder that will bias the causal relationship between diet and weight gain if we fail to control it.\nThus, we should go with the conclusion that takes initial weights into consideration. From the graph, we can see on average, students going to dining hall B gained more weight than students going to dining hall A.\nFrom all these examples above, we can see that Simpson's Paradox is not just a reversal that we find in real-world data sets all the time. By reversal, I mean if we observe A/B > a/b, and C/D > c/d, it is not guaranteed that (A+C)/(B+D) > (a+c)/(b+d). It is merely rules in arithmetic inequality.\nHowever, Simpson's Paradox is beyond mathematic calculation. It is necessary to thoroughly analyze the causal model to comprehend the data-generating process rather than solely relying on the observed data when facing contradictory conclusions.\n* * *\n### The long debate between smoking and lung cancer\nBesides providing real-life examples in Chapter 6 to illustrate the importance of understanding the data-generation process, Pearl also explores and focuses on the causal relationship between smoking and lung cancer in Chapter 5 to prove the point.\n![Photo by Volodymyr Hryshchenko on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/0cNnCeBD6mNXYeiXW-scaled.jpg)\nPhoto by [Volodymyr Hryshchenko](https://unsplash.com/@lunarts?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\nFrom the lengthy debate on whether smoking is a causal factor, we can infer how the theory developed and causality was brought to the front stage over time.\nThe studies on whether smoking causes lung cancer are challenging due to at least these three factors:\n * It is unethical to run RCT that forces participants to smoke and expose them to potential harm to health;\n * Observational data are likely to have confounders since the distribution of the treatment group can be fundamentally different from the control group.\n * The potential obstacles and deliberate efforts the tobacco companies can put to deceive the public about the health risks.\n\n\nIn the past decades, statisticians have had multiple trials to find the correct methodology, even facing these obstacles.\n### A case-control study\nAs Pearl summarizes in his book, it began with a case-control study that compared individuals with lung cancer to a control group of healthy volunteers. The study then determined whether the \"cases\" in the treatment group had more smokers than in the control group, taking into consideration some confounders like age, sex, and exposure to environmental pollutants.\nHowever, this study faced several drawbacks:\n * **A backward logic** : instead of answering what is the probability that a smoker can get cancer, it is showing what is the probability that a cancer patient is a smoker;\n * **Recall bias** : Even though the researcher made sure not to let the patients know about their diagnosis prior to the survey, they can probably infer it based on their own health condition;\n * **Selection bias** : The study is only focused on hospitalized cancer patients, which is not a good representation of the whole population or even the smoking population.\n\n\n### Cornfield's inequality\nThe debate continues when questions arise regarding whether smokers are \"self-selecting\" that they are genetically different from nonsmokers. Some concerns Pearl mentions in his book are\"\n> They are more risk-taking or more likely to develop addictive behaviors that actually cause adverse health effects.\nIn other words, the debate goes to which of the following causal diagrams is correct. Is the \"smoking gene\" the only reason we observe relationships between \"smoking\" and \"lung cancer\"?\n![Image by author based on "The Book of Why" Chapter 5](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1pAXcUYsMDl_MAGnqe4oh_Q.png)\nImage by author based on \"The Book of Why\" Chapter 5\nCornfield challenged the left causal diagram using simple math. If the left causal diagram is true and in the data, we observe smokers are **nine times** more likely to develop lung cancer, it must be because they are **nine times** more likely to have the smoking gene than non-smokers.\nAs illustrated in Pearl's book referring to Cornfield' argument:\n> If 11% of non-smokers have the smoking gene, then 99% of the smokers would have to have it. However, the percentage becomes mathematically impossible if we observe that 12% of non-smokers have the smoking gene.\nIn this case, the smoking gene cannot account fully for the relationship between smoking and lung cancer. Thus, the right causal diagram provides a more accurate explanation.\nNaming after its proposer, this argument is called Cornfield's inequality, which sheds light on drawing inferences by evaluating how strong alternative relationships must be in the absence of causality to explain the observed data.\n### The birth-weight paradox\nI hope we have already established a foundation well enough to claim that data is not enough to understand causality and explain real-world scenarios that present as paradoxes.\nAnother example that Pearl gives in his book is a study that focused on finding how mothers who smoke affect their babies' survival rates, challenging the argument that smoke causes mortality. The findings show that babies of smoking mothers indeed weigh less. However, the low-birth-weight babies of smoking mothers had a better survival rate than those of non-smokers.\nThe finding might look counterintuitive that smoking reduces birth weight and reduces child mortality rate. However, if we think further about what are the factors that contribute to babies' birth weights other than smoking, we will find the collider bias here.\nFor example, other life-threatening genetic abnormalities might also explain the lower birth rate. We can see it more clearly by drawing the causal diagram:\n![Image by author based on "The Book of Why" Chapter 5](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/12oD2AdzpFKUzcEz0nPpKcw.png)\nImage by author based on \"The Book of Why\" Chapter 5\nHere, \"smoking -> birth weight <- birth defect\" forms a collider structure. Smoking has no correlation with birth defects unless we control birth weight. If we only look at low-birth-weight babies, we are opening up the backdoor path of smoking to mortality. The relationship will be biased by the birth defects to mortality causal path.\nIndeed, other factors can add to the causal diagram that will point to birth weight. For example, studies have found _**race**_ is another factor that affects both birth weight and child mortality.\nThe list can continue, only one thing we need to keep in mind in terms of what's the **action point** taken from the causal diagram. As Pearl puts in his book:\n> We can use this diagram and data to persuade a pregnant woman to stop smoking. But we can't tell a pregnant black woman to stop being black.\n* * *\n### Last remarks\nCollider bias is everywhere. Have you noticed from your friends' experience that among all the people they date, the attractive ones tend to be jerks? Instead of utilizing psychosocial theories involving traumas and personality disorders, Pearl shows that **collider bias** can be the answer. Your friends' choices of people to date depend on two factors: attractiveness and personality. In this case, you will date a mean, attractive person, a nice, unattractive person, and certainly a nice, attractive person. However, you will never date a mean, unattractive person. As illustrated by the following causal diagram:\n![Image by author based on "The Book of Why" Chapter 6](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/12/1ucEWMNJRRmhni_F7qZCJew.png)\nImage by author based on \"The Book of Why\" Chapter 6\nBy only looking at your friend's dating pool, you are controlling a collider that will create a spurious negative correlation between attractiveness and personality.\nJust as Pearl says, even though the sad fact is that unattractive people are as mean as attractive people in the whole population (Attractiveness and Personality are not correlated), your friends never realize it because the dating pool never has somebody who is mean and unattractive.\nThis concludes this article full of fun facts and explanations of paradoxes using causal diagrams. Paradoxes are data we observe that trick and confuse us. However, by digging into the data generation process and drawing the causal diagrams, we can provide explanations of these paradoxes, which not only clears the confusion but also gives us trustworthy guidance into decision-making.\n* * *\nThat's all I want to share from Chapters 5 and 6 of \"The Book of Why\" by Judea Pearl, which completes the 4th article in this \"Read with Me\" series. I hope this article is helpful to you. If you haven't read the first three articles, check them out here:\n> [**Read with Me: A Causality Book Club**](https://towardsdatascience.com/read-with-me-a-causality-book-club-edd7085d6ae6)\n> [**Data Tells Us \"What\" and We Always Seek for \"Why\"**](https://towardsdatascience.com/data-tells-us-what-and-we-always-seek-for-why-66da7dc3f24d)\n> [**Causal Diagram: Confronting the Achilles' Heel in Observational Data**](https://towardsdatascience.com/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818)\n> [**You Can't Step in the Same River Twice**](https://towardsdatascience.com/you-cant-step-in-the-same-river-twice-cfacf7cee305)\n> [**What Makes A Strong AI?**](https://towardsdatascience.com/what-makes-a-strong-ai-012315722793)\n> [**How is Causal Inference Different in Academia and Industry?**](https://towardsdatascience.com/how-is-causal-inference-different-in-academia-and-industry-fb5afd12e2e7)\nIf interested, [subscribe to my email list](https://zzhu17.medium.com/subscribe) to join the currently ongoing biweekly discussions. As always, I highly encourage you to read, think, and share your main takeaways here or on [your own blog](https://youtu.be/oFDl0-SKAL8?si=r1QiRIizhDBdvX-A).\nThanks for reading. If you like this article, don't forget to:\n * _**Check my recent articles about**_[ _**the 4Ds in data storytelling: making art out of science**_](https://medium.com/towards-data-science/the-4ds-in-data-storytelling-making-art-out-of-science-c4998ed7875e); _[**continuous learning in data science**](https://towardsdatascience.com/continuous-learning-a-data-scientists-odyssey-8d3006c2ce01?source=post_page-----ac6e01fe5cc3--------------------------------)**;**[**how I become a data scientist**](https://towardsdatascience.com/how-i-became-a-data-scientist-7f5b10606612?source=post_page-----ac6e01fe5cc3--------------------------------)**;** _\n * _**Check my**[**other articles**](https://zzhu17.medium.com/my-blog-posts-gallery-ac6e01fe5cc3)**on different topics like**[**data science interview preparation**](https://zzhu17.medium.com/list/data-science-interview-preparation-bfb0986a61a3)**;**[**causal inference**](https://towardsdatascience.com/causal-inference-what-why-and-how-d7336b0081ec?source=post_page-----ac6e01fe5cc3--------------------------------)**;** _\n * [_**Subscribe**_](https://zzhu17.medium.com/subscribe) _**to my email list;**_\n * [_**Sign up for medium membership**_](https://zzhu17.medium.com/membership) _**;**_\n * _**Or follow me on**_[ _**YouTube**_](https://youtube.com/channel/UCMs6go1pvY5OOy1DXVtMo5A) _**and watch my most recent YouTube video about other books I read:**_\n\n\nPlease accept cookies to access this content\nPlease accept cookies to access this content\n## Reference\n[_**The Book of Why**_](https://www.amazon.com/Book-Why-Science-Cause-Effect/dp/046509760X) _**by Judea Pearl**_\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657091723_tdcxmv.png)\nWritten By\n## [Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/)\n[See all from Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Causal Data Science](https://towardsdatascience.com/tag/causal-data-science/)[Causal Inference](https://towardsdatascience.com/tag/causal-inference/)[Causality](https://towardsdatascience.com/tag/causality/)[Statistics](https://towardsdatascience.com/tag/statistics/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fwhy-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662%2F&title=Why%20Understanding%20the%20Data-Generation%20Process%20Is%20More%20Important%20Than%20the%20Data%20Itself)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fwhy-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662%2F&title=Why%20Understanding%20the%20Data-Generation%20Process%20Is%20More%20Important%20Than%20the%20Data%20Itself)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fwhy-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662%2F&text=Why%20Understanding%20the%20Data-Generation%20Process%20Is%20More%20Important%20Than%20the%20Data%20Itself)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/1GbhoZe-G1552CxXtz7b2ZA-scaled.jpeg)](https://towardsdatascience.com/22-statistics-questions-to-prepare-for-data-science-interviews-d5651a8b3c56/)\n## [22 Statistics Questions to Prepare for Data Science Interviews](https://towardsdatascience.com/22-statistics-questions-to-prepare-for-data-science-interviews-d5651a8b3c56/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nNail data science interviews with confidence, part 2\n[Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/)\nOctober 21, 202010 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/Screenshot-2025-05-08-at-16.38.31.png)](https://towardsdatascience.com/the-art-of-the-phillips-curve/)\n## [The Art of the Phillips Curve](https://towardsdatascience.com/the-art-of-the-phillips-curve/)\n[Economics](https://towardsdatascience.com/category/society/economics/)\nThe subjective details holding together one of economics’ favourite models\n[Cai Parry-Jones](https://towardsdatascience.com/author/caiparryjones96/)\nMay 12, 202515 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1mDkAUphgrNd3SXwNr1fZ-A-scaled.jpeg)](https://towardsdatascience.com/famous-italian-solitaire-simulation-and-statistics-using-python-b10d60b34fc6/)\n## [Famous Italian solitaire simulation and statistics using Python](https://towardsdatascience.com/famous-italian-solitaire-simulation-and-statistics-using-python-b10d60b34fc6/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nImplementing a solitaire simulator and extracting the winning probabilities out of it, with few lines of (Python) codes\n[Piero Paialunga](https://towardsdatascience.com/author/piero-paialunga/)\nJanuary 24, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/0KgVaJBqiDK8Zzmj9.jpg)](https://towardsdatascience.com/poisson-distribution-and-poisson-process-in-python-statistics-a7aa573c03f4/)\n## [Poisson Distribution and Poisson Process in Python - Statistics](https://towardsdatascience.com/poisson-distribution-and-poisson-process-in-python-statistics-a7aa573c03f4/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIn this article we will explore Poisson distribution and Poisson process in Python.\n[Misha Sv](https://towardsdatascience.com/author/pyshark/)\nNovember 23, 20217 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1SP2UAUALquEi45iPBQPjxw-scaled.jpeg)](https://towardsdatascience.com/partial-correlation-508353cd8b5/)\n## [Partial correlation-the idea and significance](https://towardsdatascience.com/partial-correlation-508353cd8b5/)\n[Statistics](https://towardsdatascience.com/category/statistics/)\nIs the correlation between two variables reliable?\n[Xichu Zhang](https://towardsdatascience.com/author/xichu.zhang/)\nAugust 14, 20216 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/image-2.jpeg)](https://towardsdatascience.com/histograms-and-density-plots-in-python-f6bda88f5ac0/)\n## [Histograms and Density Plots in Python](https://towardsdatascience.com/histograms-and-density-plots-in-python-f6bda88f5ac0/)\n[Programming](https://towardsdatascience.com/category/programming/)\nVisualizing One-Dimensional Data in Python Plotting a single variable seems like it should be easy....\n[Will Koehrsen](https://towardsdatascience.com/author/will%20koehrsen/)\nMarch 23, 201811 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/08/counties_Copilot_20250820_183417_cropped.jpg)](https://towardsdatascience.com/where-hurricanes-hit-hardest-a-county-level-analysis-with-python/)\n## [Where Hurricanes Hit Hardest: A County-Level Analysis with Python](https://towardsdatascience.com/where-hurricanes-hit-hardest-a-county-level-analysis-with-python/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nFrom storm tracks to insurance risk\n[Lee Vaughan](https://towardsdatascience.com/author/lee_vaughan/)\nAugust 21, 202514 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1_igArwmR7Pj_Mu_KUGD1SQ.png)](https://towardsdatascience.com/logic-and-implementation-of-a-spam-filter-machine-learning-algorithm-a508fb9547bd/)\n## [Logic and implementation of a spam filter machine learning algorithm](https://towardsdatascience.com/logic-and-implementation-of-a-spam-filter-machine-learning-algorithm-a508fb9547bd/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA beginner's guide to spam classification\n[Lily Chen](https://towardsdatascience.com/author/lilychencodes/)\nDecember 7, 20208 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/08/1g3Ht6gKYF_1pUSqXFUkBAA.jpeg)](https://towardsdatascience.com/probability-learning-ii-how-bayes-theorem-is-applied-in-machine-learning-bd747a960962/)\n## [Probability Learning II: How Bayes' Theorem is applied in Machine Learning](https://towardsdatascience.com/probability-learning-ii-how-bayes-theorem-is-applied-in-machine-learning-bd747a960962/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLearn how Bayes Theorem is in Machine Learning for classification and regression!\n[James Thorn](https://towardsdatascience.com/author/james-thorn/)\nAugust 14, 20199 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "body_base_path": "attachments/5a20c0fbeb3c/pages/", "body_markdown_path": "attachments/5a20c0fbeb3c/pages/archived-body.md", "local_html": "attachments/5a20c0fbeb3c/pages/archived-body.html", "body_source": "prior_capture" }