![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Data Science](https://towardsdatascience.com/category/data-science/) # You Can't Step in the Same River Twice "The Book of Why" Chapters 7&8, a Read with Me series [Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/) December 20, 202318 min read ![](../images/000.jpg) In my [previous articles](https://zzhu17.medium.com/list/causality-book-club-def5f1fd00fe), we learned about confounders and colliders in observational data that hinder establishing reliable causal relationships. The solution Pearl provided is to draw causal diagrams and use [the backdoor criterion](https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818) to find the sets of confounders to block and leave the colliders and mediators along. However, when dealing with confounding variables that cannot be observed or measured, it becomes difficult to estimate causality from the observational data. Coping with this issue, in Chapter 7 of "The Book of Why," Judea Pearl introduced the **do-calculus rules**. These rules are particularly useful for the **front-door criterion** and **instrumental variables**. They can be used to establish causality even when unobservable confounding variables are present. In Chapter 8, we will explore the amazing world of **counterfactuals**. Opening with Poet Robert Frost's famous lines: > "And sorry I could not travel both > And be one traveler, long I stood..." Pearl states although it's impossible to travel both paths or step into the same rivers twice, our brains can imagine what would have happened if we had taken the other path. In order to specify and pass down the wisdom to robots, Pearl introduces the distinctions between **necessary cause** and **sufficient cause** and how to utilize **Structure Causal Models** to conduct counterfactual analysis systematically. As we pass the midpoint, the chapters are getting more technical and information-intensive. In the following section, I will first discuss how to deal with unobserved confounders, unfortunately, with some math, for Rung 2 interventions. Then, I will discuss counterfactuals, a Rung 3 application. * * * Learn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) roadmap. ### The front-door criterion Starting with a causal diagram that we use to understand the causal impact of X on Y: Here, X is impacting Y through a mediator, M. However, we cannot estimate causality directly from data without controlling the confounder U. The backdoor path from **"U - > X"** generates a spurious correlation between X and Y. The backdoor criterion tells us to control U, but what if U is unobservable? For example, in analyzing the causal relationship between smoke (X) and cancer (Y), we might see this causal diagram: Here, smoking causes cancer through the accumulation of tar deposits, and there is a confounder, the smoking gene, which, as argued by some researchers, can influence both one's smoking behaviors and the chances of getting lung cancer. We couldn't collect data on such genes because we don't know whether they exist. Thus, back-door adjustments cannot work in this case. To get the causal impact, we can use the **front-door criterion** instead. The front door here is the mediating process, where smoking increases tar deposits and increases the chances of getting cancer. If we cannot estimate smoke to cancer directly, can we estimate how smoke causally impacts tar with how tar causally impacts cancer instead? Here are the steps: **Step 1: Smoking - > Tar** There is only one back door path from smoking to tar: > Smoking <- Smoking Gene -> Cancer <- Tar And it's blocked by the Collider cancer. So, we can estimate the causal impact of smoking to tar directly from data by calculating the conditional probabilities: text ``` The causal impact of smoking on tar is: P(tar|smoking) - P(tar|no smoking) ``` **Step 2: Tar - > Cancer** There is one back door path from tar to cancer: > Tar <- Smoking <- Smoking Gene -> Cancer Here, both smoking and the smoking gene are the confounders, but we can control one of them to block the path. Since we don't have data about the smoking gene, we can control smoking instead: text ``` The causal impact of tar on cancer is: P(cancer|do(tar)) - P(cancer|do(no tar)) where, P(cancer|do(tar)) = P(cancer|tar,smoking) * P(smoking) + P(cancer|tar,no smoking) * P(no smoking) P(cancer|do(no tar)) = P(cancer|no tar,smoking) * P(smoking) + P(cancer|no tar,no smoking) * P(no smoking) ``` To estimate how tar impacts cancer causally, we measure the following four probabilities from the data: * The probability of getting cancer through accumulated tar in the smoking population: > P(cancer|tar, smoking) * P(smoking) * The probability of getting cancer through accumulating tar in the non-smoking population: > P(cancer|tar, no smoking) * P(no smoking) * The probability of getting cancer without enough tar in the smoking population: > P(cancer|no tar, smoking) * P(smoking) * The probability of getting cancer without enough tar in the non-smoking population: > P(cancer|no tar, no smoking) * P(no smoking) **Step 3: Smoking - > Tar -> Cancer** Once we know the causal impact of smoking on tar and the causal impact of tar on cancer, we can derive the unbiased causal impact of smoking on cancer through the front door adjustment: text ``` The causal impact of smoking on cancer is: P(cancer|do(smoking)) = P(cancer|do(tar)) * P(tar|do(smoking)) + P(cancer|do(no tar)) * P(no tar|do(smoking)) ######Math Alert######################## Since no backdoor between smoking and tar: P(tar|do(smoking)) = P(tar|smoking) & P(no tar|do(smoking)) = P(no Tar|smoking) And from back door adjustment for tar and cancer: P(cancer|do(tar)) = P(cancer|tar,smoking) * P(smoking) + P(cancer|tar,no smoking) * P(no smoking) & P(cancer|do(no tar)) = P(cancer|no tar,smoking) * P(smoking) + P(cancer|no tar,no smoking) * P(no smoking) Finally, P(cancer|do(smoking)) = (P(cancer|tar,smoking) * P(smoking) + P(cancer|tar,no smoking) * P(no smoking)) * P(tar|smoking) + (P(cancer|no tar,smoking) * P(smoking) + P(cancer|no tar,no smoking) * P(no smoking)) * P(no tar|smoking) ``` Or, in more general terms, for any causal diagram that resembles the previous one in these ways: * X affects Y through only a mediator M, which is also the front door; * There is an unobservable confounder U, correlated with X and Y but not with the mediator M; The following formula using the front door adjustment works: Compared to the back door adjustment: Both formulas estimate the causal impact of X on Y, and they all remove the _**do**_ operator successfully. This means we can estimate the causal impact from data, i.e., **drawing Rung 2 &3 conclusions from Rung 1 data**. In the front door adjustment formula, we also removed the unobserved confounders U successfully. In the smoking to cancer case, we can estimate the impact of smoking without including the smoking gene. The front door criterion opens the door for squeezing a bit more juice out of observational data, even facing unmeasurable confounders. However, there are challenges regarding its applications since the real world usually functions messier and more complicated than the textbook causal diagrams. For example, the unobserved confounder could also impact the mediator: Even with this complication, as long as the relationship between U and M is weak, using the "gold standard" randomized control trial (RCT) estimation of P(Y|do(X)) as a benchmark, the empirical research shows that using the front door adjustment provides a better estimation than the back door adjustment without blocking all necessary back doors due to unobserved confounders. * * * ### Do-calculus rules Using the example above that illustrates the front door adjustment, Pearl also summaries three rules that provide general guidance in the process of removing _**do** _s : * Rule 1 shows if variable W is irrelevant to Y, or its direct path to Y is blocked by variable Z, then removing or adding W will not change the following probabilities: * Rule 2 shows if a variable set Z blocks all back-door paths from X to Y, then conditional on Z will remove the _**do**_ operation: * Rule 3 shows if there are no causal paths from X to Y, then we can remove the _**do**_ operation entirely: The above three rules set the foundation for _**do-** _calculus that allows us to derive Rung 2 and 3 causal impacts out of observational data. Rule 1 shows what variables are useful to collect from the data; Rule 2 shows how to derive intervention, a Rung 2 conclusion, out of observational data; Rule 3 shows whether an intervention will be effective. ### More math alert: skip if math gives you a headache Using the three rules, we can revisit the smoking-to-cancer causal diagram: And see how we utilize Rules 2 and 3 even in more general cases. Here are the steps: Here, to understand how smoking causally impacts cancer, we are using the front door path smoking -> tar -> cancer in 7 steps: * Step 1 is based on probability theories where we introduce the mediator t, tar, to estimate the causal impact; * Step 2 is based on Rule 2. Since tar -> cancer's back door is blocked by s, smoking, we can replace (do(s), t) with (do(s), do(t)); * Step 3 is based on Rule 2 again. Since smoking -> tar doesn't have back doors, we can replace do(s) with s; * Step 4 is based on Rule 3. Since smoking is only causally impacting cancer through tar, we can replace P(c|do(s), do(t)) with P(c|do(t)); * Step 5 is based on probability theories, where we introduce different spectra of S into the equation. In this case, we have smoking vs non-smoking; * Step 6 is based on Rule 2. Again, tar -> cancer's back door is blocked by s, so we replace (do(t), s') with (t, s'); * Step 7 is based on Rule 3. Since tar doesn't have a causal impact on smoking, we replace P(s'|do(t)) with P(s'). In the last equation, we can see that we have removed the do operation completely, and there are no unobservable variables. The next step will be using data to calculate the causal impact. * * * ### Instrument variable Another way to deal with the unobserved confounder is to find instrument variables. The definition of instrument variables is better illustrated in a causal diagram: Supposedly, when estimating the causal impact of X on Y, there is an unobservable confounder U that prevents us from getting the correct estimation. If there exists a variable Z that satisfies: * Z and U are not correlated. In the diagram, there is no arrow between Z and U; * Z is the direct cause of X; * Z only affects Y through X. In other words, there is no direct or indirect arrow from Z to Y except Z->X->Y. Z will be a very good instrument variable if all conditions are satisfied. Instrument variables are very useful in many scientific fields. In the book, Pearl provides an example of using instrument variables to study a medicine's treatment effect during clinical trials. The medicine was invented to lower patients' cholesterol levels. Although clinical trials are usually randomized, they still face the challenge of noncompliance, where subjects receive the medicine but choose not to take it. Their decisions on not taking the medicine can depend on multiple factors, like how sick they are, and are usually unobservable or hard to measure. The existence of noncompliances will reduce the estimation of drug effectiveness, and we don't really have a good way to predict how many noncompliances there will be in a clinical trial. In this case, researchers introduce an instrument variable, "Assigned," into the RCT design. "Assigned" takes the value one if the patient is randomly assigned to receive the drug and zero if they receive a placebo. We will have the following causal diagram: "Assigned" is an instrument variable because: * The assignment of receiving pills is randomized among patients. Thus, it is not correlated with any confounders U; * Which groups patients are assigned to will determine which treatment they receive, either a drug or a placebo. Thus, "Assigned" is the direct cause of "Received"; * Whether a patient is put into a placebo group or not doesn't affect his or her cholesterol level directly. Thus, "Assigned" only affects Cholesterol through Received. When we find or establish an instrument variable, we can estimate three relationships in the observational data: * The causal impact of Assigned -> Received; * The causal impact of Assigned -> Cholesterol; * The causal impact of Received -> Cholesterol by removing the "Assigned -> Received" impact from "Assigned -> Cholesterol". That's the definition of an instrument variable, along with an example. Pearl's book contains more examples of how instrument variables can be used to solve the unobserved confounders' problems. * * * ### Counterfactuals: What could have been? Moving from Rung 2 to Rung 3 of the causality ladder, we are now facing the problem of finding what could have been the outcome without the treatment. This is different from Rung 2 interventions in two aspects: 1. **From average causal effect to individual causal effect:** So far, the causal impact we have discussed is focused on a population or subpopulation. For example, does smoke cause cancer or not for everyone? However, the more applicable question, especially in solving real-world problems, is the **individual causal effect**. For example, if I start smoking, does it cause me to have cancer? If I give this customer a discount, will he or she buy more products? The personalized causation can be inferred through counterfactual analysis. **2. There are two types of counterfactuals:** In binary settings, the outcome and potential outcome have two options. In the example of smoke vs cancer, the outcome is either getting cancer or no cancer. Correspondingly, there would be two types of potential outcomes and two sets of causal factors: * **Necessary causation:** If one person gets cancer, the potential outcome is no cancer, and we determine whether smoking is what the lawyers called "_**but-for causation**_ ": the cancer would not have developed _**but for**_ the smoking behavior. * **Sufficient causation:** If one person does not get cancer, the potential outcome is cancer, and we determine whether having a smoking behavior would have developed cancer for this person. Distinguishing necessary causation and sufficient causation not only helps robot think more like a human in determining causality but us find better action points given different goals and scenarios—more on this in the next section. * * * ### Matching vs Structure Causal Models (SCM) When posed with the question of what could have been, we may approach it as a missing data problem. Take the example of figuring out how increasing education can improve income for different people. Here is the data summarized in a table: Here, we have various data entries from different employees and their current experience level (EX) and their highest education level (ED). To simplify, we assume three levels of education: 0 = high school degree, 1 = college degree, and 2 = graduate degree. Their salaries are also reported as S0, S1, S2. Note that since each employee can only have one highest-level education degree at a given time point, we will have two missing entries for S0, S1, and S2 for all employees. If we approach filling the question marks in the above table as a missing data problem, we have two ways: 1. **Matching:** We find similar employees and match salary levels at different education levels. For example, in the table, we only have one extra feature - Experience, and we see Bert and Caroline both have nine years of experience, then we can have S2(Bert) = S2(Caroline) = $97,000, and S1(Caroline) = S1(Bert) = $92,500. Two missing data filled! **2. Linear Regression or more complicated models** Under the strong assumption that all data came from some unknown random sources, standard statistical models find models that best fit the data. In a linear regression model, we might find an equation like this for this particular problem: S = $65,000 + 2,500 _EX + 5,000_ ED Looking at the coefficients, the equation tells us on average, one level increase in education increases salary by $5,000. We can use more complicated models when feature space increases. What's wrong with these methods? Fundamentally, they are both approaches that are data-driven rather than model-driven. We are still trying to solve a Rung 3 problem with Rung 1 approaches. Thus, no matter how complicated our models get and how many more features we gain to predict the outcome variable, we still face the fundamental flaw of missing the causal mechanisms. In this simple example, one problem we have is Experience and Education are not independent of each other. Generally, more education may reduce the years of experience for an individual. If Bert were to have a graduate degree rather than the college degree he has right now, his experience level would be lower than nine years. Thus, he would not be a good match with the nine-year experienced graduate degree holder Caroline anymore. In summarize, we will have a causal graph that looks like this: The causal diagram indicates that Education not only has a direct causal impact on Salary, but it also affects Salary through the mediator Experience. Thus, we will need two equations: text ``` S = f_s(EX, ED, U_s) where EX = f_ex(ED,U_ex) ``` There are several points to make regarding these equations, which constitute the Structure Causal Models (SCM) for this problem: * Salary is a function of Experience, Education, and some unobservable variable that affects Salary U_s. Note the unobservable variable is exogenous, meaning they are not correlated with Education and Experience; * Experience is a function of Education and some unobservable variables that affect Experience U_ex. * The fact that there is no function shows Education as a function of Experience means that there are no causal effects from Experience to Education. This is the assumption we make. * These two equations assume causalities between the right-hand side (the outcome) and the left-hand side(the treatment). * The unobservable variables U_s and U_ex quantify the uncertainties at the individual level, which differs from Bayesian networks that use probabilities at causal links to quantify uncertainties. They are independent of Experience or Education and can be customized by individuals. * Function f represents the relationship between the features and the outcome variable. It can be both linear and non-linear, depending on the assumptions. If we still assume a linear relationship, we will have the following equations based on the data and our understanding of how education affects experience: text ``` S = $65,000 + 2,500*EX + 5,000*ED + U_s and EX= 10 – 4*ED + U_ex ``` With these equations, we can calculate every one's idiosyncratic factors U_s and U_ex to predict counterfactuals. Take Alice as an example. We know S(Alice) is 81000, EX(Alice) is six, and ED(Alice) is 0. First, plug these into the second equation to get U_ex. We want to plug in the second equation because it only contains ED, the only causal factor of our interest: text ``` 6 = 10–4*0 + U_ex(Alice) -> U_ex(Alice) = -4 Thus, EX(Alice) = 10 – 4*ED(Alice) - 4 ``` Here, rather than simply plugging EX(Alice) into the equations, we are using this value indirectly. Knowing EX(Alice) equals six helps us calculate U_ex(Alice). Then we plug S(Alice), ED(Alice) and U_ex(Alice) into the first equation to get U_s(Alice): text ``` 81000 = 65000 + 2500*(10 – 4*ED + U_ex) + 5000*ED + U_s(Alice) -> U_s(Alice) = -5000*ED - 2500*U_ex(Alice) -9000 -> U_s(Alice) = 0 - 2500*(-4) - 9000 = 1000 #Note DO NOT calculate by plug EX=6 directly: 81000 = 65000 + 2500*6 + 5000*0 + U_s(Alice) ``` Here we have the functions for Alice: text ``` S(Alice) = $65,000 + 2,500*EX + 5,000*ED + U_s(Alice) and EX= 10 – 4*ED + U_ex(Alice) ``` Once the SCMs are ready, we can do a counterfactual analysis for Alice. We can calculate what would be her salary had she attended college. If ED(Alice) is 1 instead of 0, we will first calculate: text ``` EX(Alice) = 10–4+(-4) = 2 ``` Then calculate S_1(Alice): text ``` S_1(Alice) = $65,000 + 2,500*2 + 5,000*1 + 1000 = $76,000 ``` Note the difference we might get from the regression model where we plug in ED(Alice) = 1 and EX(Alice)=6: text ``` S = 65000 + 2500*6 + 5000*1 = $85,000 -> Biased estimation # If Alice has six years of experience and a high school degree now, # She couldn't get six years of experience if she goes to college. ``` This is a simple example of utilizing SCMs to understand the causal impact and calculate the counterfactuals at the individual level. In summary, Pearl calls "the first law of causal inference": The equation shows the potential outcome Y_x(u) can be imputed by Model M_x if we can remove all back-door paths to X. Here, the model takes much more flexibility than linear regression models as long as the causality is embedded based on the causal diagram. * * * ### Necessary Cause (PN) vs. Sufficient Cause (PS) To understand counterfactuals, we have two different measurements: the probability of necessity (PN) and the probability of sufficiency (PS). To see the difference, let's use an example: the house is set on fire because someone struck the match, and there is oxygen in the air: Both match and oxygen are causal factors in a house on fire. However, they differ in PN and PS. The probability of necessity is: text ``` PN = P(Y_x=0 = 0|x=1, Y=1) ``` In this case, the house was on fire, and a match had struck (Y=1, x=1). This probability asks if the house would not have been on fire had the match not struck (Y_(x=0) = 0). This probability is very high because even though we have enough oxygen, with fire from the match, the house will not be on fire. The same logic applies when we calculate the PN for x=oxygen. The fire would not occur if there were not enough oxygen in the house, even if we struck the match. The probability of necessity shows what would happen to the outcome variable if the treatment _**hadn't**_ happened. If the probability is high, it means this treatment is a necessary cause. In a court setting, proving the victim wouldn't have died if the accused hadn't done something is enough to convict. On the other hand, the probability of sufficient is: text ``` PS = P(Y_x=1 = 1|x=0, Y=0) ``` In this case, the house was not on fire, and a match hadn't struck (Y=0, x=0). This probability asks if the house would have been on fire had the match struck (Y_(x=0) = 0). This probability is also very high because usually, oxygen is everywhere, and the house is very likely to be on fire with oxygen and fire from striking the match. The PS for oxygen, however, is very low. The fire would not likely occur just because the house has oxygen. Oxygen is not sufficient to set the house on fire. We need the action of bringing other fire sources like striking the match. Thus, the probability of sufficient tells us what would happen to the outcome variable if the treatment _**had**_ happened. If the probability is high, it means this treatment is a sufficient cause. In this case, striking the match qualifies as a sufficient cause for a house fire, but oxygen doesn't. ### Why distinguish PS and PN _**Why bother making these distinctions?**_ In a nutshell, although multiple variables can be the causal factors for an outcome, human brains automatically "rank" these factors based on some conditions. Psychologists found that humans imagining actions could have been done to turn around an undesired outcome. They are more likely to: * Imagine undoing a rare event than a common one. For example, striking a match is a more rare event compared to having oxygen in the house. (Hopefully!) * Human is more likely to blame their own actions, i.e., striking a match, than events not under their control. As Pearl emphasizes, embedding both PN and PS into models would suggest a systematic way for teaching robots to produce meaningful explanations of the observed outcomes. In addition, understanding the distinctions between PS and PN guides us to the action points. Climate scientists who study and find the cause of the extreme heat waves may have two different statements: * PN: There is a 90% probability that man-made climate change was a necessary cause of a heat wave; * PS: There is an 80% probability that climate change will be sufficient to produce a heat wave this strong at least once every 50 years. The PN statement tells us about attribution: who was responsible for the heat wave - man-made climate change. It finds the cause. The PS statement tells us about policy. It says we are better prepared for the heat waves because more is happening. Who caused it? Not specified. Both statements are informative, just in different ways. * * * **What a long read!** This completes the 5th article in this "Read with Me" series for Judea Pearl's "The Book of Why." Thanks for hanging to the end. Chapters 7 and 8 are definitely information-dense. I hope this article is helpful to you. If you haven't read the first four articles, check them out here: * [**Kick-off: Start with A Cat Story**](https://medium.com/towards-data-science/read-with-me-a-causality-book-club-edd7085d6ae6) * [**Chapter1 &2: Data Tells Us 'What" and We Always Seek for "Why"**](https://medium.com/towards-data-science/data-tells-us-what-and-we-always-seek-for-why-66da7dc3f24d) * [**Chapter3 &4: Causal Diagram: Confronting the Achilles' Heel in Observational Data**](https://medium.com/towards-data-science/causal-diagram-confronting-the-achilles-heel-in-observational-data-a69cdb1c4818) * [**Chapter5 &6:Why Understanding the Data-Generation Process Is More Important Than the Data Itself**](https://zzhu17.medium.com/towards-data-science/why-understanding-the-data-generation-process-is-more-important-than-the-data-itself-f1b3b847e662) If interested, [subscribe to my email list](https://zzhu17.medium.com/subscribe) to join the currently ongoing biweekly discussions. We will have one more article to come that will focus on mediators and conclude the series: > [**What Makes A Strong AI?**](https://towardsdatascience.com/what-makes-a-strong-ai-012315722793) And a bonus article: > [**How is Causal Inference Different in Academia and Industry?**](https://towardsdatascience.com/how-is-causal-inference-different-in-academia-and-industry-fb5afd12e2e7) As always, I highly encourage you to read, think, and share your main takeaways here or on [your own blog](https://youtu.be/oFDl0-SKAL8?si=r1QiRIizhDBdvX-A). Thanks for reading. If you like this article, don't forget to: * _**Check my recent articles about**_[ _**the 4Ds in data storytelling: making art out of science**_](https://medium.com/towards-data-science/the-4ds-in-data-storytelling-making-art-out-of-science-c4998ed7875e); [_**continuous learning in data science**_](https://towardsdatascience.com/continuous-learning-a-data-scientists-odyssey-8d3006c2ce01) _**;**_[_**how I become a data scientist**_](https://towardsdatascience.com/how-i-became-a-data-scientist-7f5b10606612) _**;**_ * _**Check my**_[ _**other articles**_](https://zzhu17.medium.com/my-blog-posts-gallery-ac6e01fe5cc3) _**on different topics like**_[ _**data science interview preparation**_](https://zzhu17.medium.com/list/data-science-interview-preparation-bfb0986a61a3) _**;**_[_**causal inference**_](https://towardsdatascience.com/causal-inference-what-why-and-how-d7336b0081ec) _**;**_ * [_**Subscribe**_](https://zzhu17.medium.com/subscribe) _**to my email list;**_ * [_**Sign up for medium membership**_](https://zzhu17.medium.com/membership) _**;**_ * _**Or follow me on**_[ _**YouTube**_](https://youtube.com/channel/UCMs6go1pvY5OOy1DXVtMo5A) _**and watch my most recent YouTube videos:**_ Please accept cookies to access this content ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-6091039d5533a6240825.png) Written By ## [Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/) [See all from Zijing Zhu, PhD](https://towardsdatascience.com/author/zzhu17/) [Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Causal Inference](https://towardsdatascience.com/tag/causal-inference/)[Causality](https://towardsdatascience.com/tag/causality/)[Counterfactual](https://towardsdatascience.com/tag/counterfactual/)[Data Science](https://towardsdatascience.com/tag/data-science/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fyou-cant-step-in-the-same-river-twice-cfacf7cee305%2F&title=You%20Can't%20Step%20in%20the%20Same%20River%20Twice) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fyou-cant-step-in-the-same-river-twice-cfacf7cee305%2F&title=You%20Can't%20Step%20in%20the%20Same%20River%20Twice) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fyou-cant-step-in-the-same-river-twice-cfacf7cee305%2F&text=You%20Can't%20Step%20in%20the%20Same%20River%20Twice) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/001.jpeg)](https://towardsdatascience.com/introduction-to-sports-analytics-with-pandas-ad6303db9e11/) ## [Introduction to Sports Analytics with Pandas](https://towardsdatascience.com/introduction-to-sports-analytics-with-pandas-ad6303db9e11/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Analysis of German Bundesliga football matches. [Soner Yıldırım](https://towardsdatascience.com/author/sonery/) September 26, 20205 min read * [![](../images/002.jpg)](https://towardsdatascience.com/the-last-matplotlib-tweaking-guide-youll-ever-need-dbd4374a1c1e/) ## [The Last Matplotlib Tweaking Guide You'll Ever Need](https://towardsdatascience.com/the-last-matplotlib-tweaking-guide-youll-ever-need-dbd4374a1c1e/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) The default Matplotlib stylings are out of style - but here's what to do about it [Dario Radečić](https://towardsdatascience.com/author/radecicdario/) December 17, 20194 min read * [![](../images/003.png)](https://towardsdatascience.com/understanding-point-process-model-with-r-983553ca2a86/) ## [Understanding Point Process Model with R](https://towardsdatascience.com/understanding-point-process-model-with-r-983553ca2a86/) [Data Science](https://towardsdatascience.com/category/data-science/) I have recently been playing with spatial data as a learning exercise and I have been particularly interested in point pattern analysis... [Benjamin Cretois](https://towardsdatascience.com/author/bencretois/) January 26, 20218 min read * [![](../images/004.jpg)](https://towardsdatascience.com/ive-tried-r-for-the-first-time-how-bad-was-it-ba344f22e90b/) ## [I've Tried R for the First Time - How Bad Was It?](https://towardsdatascience.com/ive-tried-r-for-the-first-time-how-bad-was-it-ba344f22e90b/) [Programming](https://towardsdatascience.com/category/programming/) All-time Pythonista tries out R - comparisons with Python included. [Dario Radečić](https://towardsdatascience.com/author/radecicdario/) September 10, 20206 min read * [![](../images/005.png)](https://towardsdatascience.com/do-not-over-think-about-outliers-use-a-student-t-distribution-instead-b6c584b91d5c/) ## [Do not over-think about 'outliers', use a student-t distribution instead](https://towardsdatascience.com/do-not-over-think-about-outliers-use-a-student-t-distribution-instead-b6c584b91d5c/) [Data Visualization](https://towardsdatascience.com/category/data-science/data-visualization/) The Student's t-distribution provides a robust alternative to acknowledge that our data may be populated by extreme observations... [Daniel Manrique Castano](https://towardsdatascience.com/author/damanrique/) March 30, 202415 min read * [![](../images/006.png)](https://towardsdatascience.com/mastering-websockets-with-go-c30d0ac48081/) ## [Mastering WebSockets With Go](https://towardsdatascience.com/mastering-websockets-with-go-c30d0ac48081/) [Programming](https://towardsdatascience.com/category/programming/) Tutorial on how to use WebSockets to build real-time APIs in Go [Percy Bolmér](https://towardsdatascience.com/author/programmingpercy/) November 22, 202227 min read * [![](../images/007.jpg)](https://towardsdatascience.com/how-to-become-a-kaggle-competitions-grandmaster-9d77431c5b7d/) ## [How to become a Kaggle Competitions Grandmaster](https://towardsdatascience.com/how-to-become-a-kaggle-competitions-grandmaster-9d77431c5b7d/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) It won't be easy, but you can do it and it's worth it [Agnis Liukis](https://towardsdatascience.com/author/agnis-liukis/) September 1, 20218 min read * [![](../images/008.jpeg)](https://towardsdatascience.com/anticipating-wildfires-with-machine-learning-tools-5eb43b450d36/) ## [Anticipating Wildfires with Machine Learning Tools](https://towardsdatascience.com/anticipating-wildfires-with-machine-learning-tools-5eb43b450d36/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Using data to uncover ideal wildfire suppression resource placement [Patrick Norman](https://towardsdatascience.com/author/patrickisnorman/) June 9, 20215 min read * [![](../images/009.jpeg)](https://towardsdatascience.com/unlock-the-power-of-causal-inference-front-door-adjustment-an-in-depth-guide-for-data-scientists-8e7b8ba33421/) ## [Unlock the Power of Causal Inference & Front-door Adjustment: An In-depth Guide for Data Scientists](https://towardsdatascience.com/unlock-the-power-of-causal-inference-front-door-adjustment-an-in-depth-guide-for-data-scientists-8e7b8ba33421/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) A full explanation of causal inference front-door adjustment with examples including all the Python source code [Graham Harrison](https://towardsdatascience.com/author/grahamharrison-86487/) February 14, 202310 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) © Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)