{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "James Andrew Godwin",
  "title_listed": "Regularization and Linear Regression",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/regularization-and-linear-regression-bcaeba547c46/",
  "publish_date": "2021-03-25",
  "corpus_stage": "",
  "notes": "",
  "id": "0dac0d1e8663",
  "collected_at": "2026-10-02T08:38:56",
  "title": "Regularization and Linear Regression | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# Regularization and Linear Regression\nThe one where we correct overfitting\n[James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\nMarch 25, 202123 min read\n![Regularization and Linear Regression | Photo by Jr Korpa](../images/000.jpeg)\nRegularization and Linear Regression | Photo by [Jr Korpa](https://unsplash.com/@korpa)\nThis article is a continuation of my series on [linear regression and bootstrap](https://towardsdatascience.com/linear-regression-with-bootstrapping-4924c05d2a9) and [Bayesian statistics](https://towardsdatascience.com/bayesian-statistics-11f225174d5a).\nPreviously I talked at length about linear regression, and now I am going to continue that topic. As I hinted at previously, I am going to bring up the topic of regularization. And what regularization does, is it simplifies the model. And the reason why you want to have a simpler model is that, usually, a simpler will model will perform better in most if not pretty much all of the tests that can be run.\nN.B: It is not kosher to use training data as the test data. You only clearly see the full effects of regularization when you test your models with test data, which is data that the model has not yet seen.\nMy writing is primarily coming from the statistical modeling of things with less in the machine learning effectiveness. For simplicity, many of the following examples break this rule and we evaluate our models using training data and not test data.\nAs such, **the coefficients of determinations in our models are not truly representative of the models' effectiveness, rather they are only correlation coefficients**.\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) roadmap.\n### Finding the best model\nModel selection - sometimes referred to as feature selection, is when we select the best model for the data we have using different features/attributes/columns of available data. Choosing different features for your model essentially creates a different model.\nEither term is fine. After choosing our model we will then regularize it. Regularize is to simplify it: **Regularization = Simplification**.\nThe reason why we want a simpler model is that we want to avoid a phenomenon called overfitting or over-parameterization. Overfitting or over parameterization is a phenomenon that occurs very often, especially if you have a lot of parameters.\nThere are a variety of regularization techniques to use, including:\n  * Stepwise\n  * Ridge regression\n  * Lasso\n  * PCR - PCA + regression (principal component analysis and regularization)\n\n\nPCR is the most important technique to use for regularization.\n### Model fitness\nBroadly speaking, a good model \"fits the data well.\" However, there is nuance to this as there a**re different uses for different models**. If you have a data set and you want to know if this data set can be modeled; if you can use a mathematical formulation to explain the model, the data that you have - and this is all you want to do with it- then it is very good and legitimate to use that same data and evaluate your model using that data, or evaluating our model with our training data.\nHowever, if you want to be able to do any kind of prediction and determine how powerful this model is at predicting something - and that is usually what people want to do with a model: they want to actually make use of it - then **we need to test the model with test data**.\nA good model needs to generalize well. For example, it needs to generalize from your training data to your test data. Consider the following examples:\n![](../images/006.jpeg)\n  * **Overfitting** : model works great for training data but does not generalize to real world\n  * **Underfitting** : model is too simple, generalizable but not meaningful. Less variance but more bias towards the wrong answers\n\n\nWhy do I call the left-most image a great fit, the one that is just right? Well, because when we eyeball this, the black line seems to explain those blue dots pretty well. If we were to do a simple linear regression, we might end up with something like the line on the center image. And we would be calling that underfitting. We would have a lack of variance in our model, and we would have too much bias in our model; that is what we would say.\nNow, on the right-hand side, we are overfitting. Here, we have too much variance in the model. So the model variance is pretty clear because it's going up and down, jumping all around. That variance is helping with the fitting of this particular data set. But we call this overfitting, we say it is fitting the data too well. You might ask well, how/why is that a bad thing? Well, it is a bad thing because if I were to come along and put in test data, the test data probably would fit worse than either of the other two models. Because the test data would not be exactly at these peaks or troughs of these maximas or minimas of this black line.\n### Overfitting\nOver-parameterization leads often to overfitting and it also carries some other problems with it:\n  * Number of parameters > dimensionality of data\n  * Linear dependency of input features\n  * Works great on the training data set but has poor (if not abysmal) performance in the real world (test data)\n\n\nThere is a problem with what we call linear dependency of input features. So if one of your input features can be explained by other input features in a linear manner, then we call that a linear dependency. Translated that means if you can linearly combine (that means to add some set of columns to come up with or determine another column that already is there, just through addition, subtraction, and multiplication) then that means that you have a linear dependency; that means one of the columns, one or more of the columns is just a combination of the other columns. And that leads to problems when you do calculations and determine your model.\n### Prevent overfitting\nThe tool to prevent overfitting is regularization. There are different ways of doing regularization. From earlier, stepwise regression is one form of doing regularization. What is stepwise regression? Removing/adding features in a stepwise fashion. There are two flavors, forward and backward. Forward stepwise regression pretty much works like this: you choose what you think is your best feature. Say you have input features x_1, x_2, x_3, x_4, and so on; you choose the one that you think is best (there are a variety of ways that you could choose it.) And then you come, you will create a linear regression, based off of that feature, say it was x_2, which now is the input and y is the output. Then you select your next best, say x_4, and you try that one out with x_2. And if the model gets better, by some amount, then you keep the second x value. If it was not good, if the model performance does not improve, then you do not keep it and then you are back to having only x_2. Then you try out another one of these features. This is called forward stepwise regression.\nBackward stepwise regression is the more common way of doing things. You take all of your values like x_1, x_2, x_3, x_4, and create a linear regression from those four inputs to come up with a function that explains y; that calculates/ predicts y. And then you remove one of the columns and see if the model has improved. And typically, you will fit things less well. But your model might improve based on criteria that are not just purely fitting, they might fit better, because you also put in a penalty for parameters or you use a measure called information criterion that says you need to gain so and so much information from any given input parameter. Continue removing features until the model performance stops improving. This is called backward stepwise regression.\nOther regularization techniques are you put a penalty in for every column you have; a penalty for the coefficient that is associated with that column. Those are other regularization techniques.\n### Measuring the model\nHow do we measure the model? There is the Akaike information criterion (AIC):\n![AIC formula](../images/017.png)\nAIC formula\n  * k: number of estimated parameters\n  * L: likelihood\n\n\nThis is the general favorite metric to determine if a model has improved. As the AIC becomes smaller and smaller for comparable models you say that the AIC is improving and that your model is also improving. So by adding more columns, or by removing more columns, if your AIC is improving, then you know you are going in the right direction. If your AIC does not improve, that means non-decreasing, then you know that you took a step in the wrong direction; meaning the last addition of a column or the last removal of a column should be undone, you go back and try something else.\nAnother model metric is the Bayesian information criterion (BIC), but it is rarely used in this context - it is very similar to AIC:\n![BIC metric](../images/003.png)\nBIC metric\n  * k: number of estimated parameters\n  * L: likelihood\n  * N: number of recorded measurements\n\n\nRegardless of the criterion you choose, your model of best fit is the one that has the minimal AIC or BIC.\n### Cost\nWhen you determine any function, any machine learning algorithm has a so-called cost function. And this is a very fundamental concept. A cost function basically says, \"how bad is your model?\"\nI discussed previously on linear regression, badness would be the square of the residuals. As a reminder, a residual is the difference between the predicted value and the actual value of y, of the output; then you square that, and then you sum up all those squares. This is a cost function but was not explicitly called one before.\n![RSS is also referred to as SSR, but they are not always the same thing!](../images/004.png)\nRSS is also referred to as SSR, but they are not always the same thing!\nN.B: In some literature, SSR is not the residual sum of squares, but the sum of squares regression - a different quantity. For the rest of the article, I stick with RSS to avoid confusion about the above quantity.\nThere are a few bells and whistles on this that we change here and there, we can divide this by the number of cases, which means the number of values that are being used - the number of data points. We can put in a constant here, but overall we want to minimize this:\n![Minimize the cost function](../images/005.png)\nMinimize the cost function\nThat is probably one of the most common cost functions. There are others, especially for logistic regression. However, for the most part, this is our primary cost function - it might be modified in some minor ways.\nA cost function calculates the penalty. So the penalty here would be the sum of those squares: calculating how bad things are. This is a penalty for a given trained modeling data set. And the idea is to minimize this penalty or to minimize this cost. And so you need a few things to watch out for:\n  * models need to be compared to each other\n  * a cost function determines the variance\n  * depending on the dataset, the target variable, and purpose (business question to be answered), selecting the right cost function is critical and essential\n\n\nIt is important that similar models have to be compared to each other. So you cannot compare the root mean square error (RMSE), which is very closely related to the cost function above, you cannot really compare that from one model to the next unless they are very, very similar models - unless there are a lot of things the same.\nIf you have just two different models from different sources doing different having different purposes, **comparing the RMSE will not mean anything**. These root mean squared errors, or these costs, only are understandable in terms of when they are compared to another cost of a very similar model.\n### Variance and bias\nA good cost function tries to balance variance and bias to make it the best combination of variance and bias.\nWhat are variance and bias?\n![Demonstrating variance vs bias](../images/006.jpeg)\nDemonstrating variance vs bias\nIn the overfitting example on the right-hand side, there is a strong variance, there is too much variance; while in the center underfitting example there is a strong bias, there is too much bias.\n### Cost in regression\nThe measure of fit is another way of saying minimizing the cost function. So far we have been minimizing the residual sum of squares. Very confusingly this can be known as SSR and RSS, I will use RSS. Another component of the total cost is the weight of our coefficients, just another way of saying coefficients.\nTo remind you, a coefficient in this particular situation where we, let us say we have two columns: column x_1and x_2. Well, then we would have:\n![b and c are the coefficients](../images/007.png)\nb and c are the coefficients\nSince the coefficients are part of our regression (we treated them as 1 earlier) we should include them in our total cost function. Actually, we add the square of those in, and we get another penalty for just having that coefficient.\nNow if that coefficient goes to zero, it basically wipes out the column. So that way, we can actually get rid of columns. Or get rid of this penalty, or part of this penalty. Naturally, if we get rid of all the columns, and we have nothing left, then all we have is the mean of y itself.\nSo we need, when we add a column, we are going to say, \"okay, we can add a column and we can use that information. But we are going to accept a penalty for adding that column.\" **Whatever coefficient that column gets, is going to be our penalty**. That means if it is a pretty big, important column, it is going to have a big coefficient, so it had better be worth it. Because that coefficient is also going to count as a part of this so-called penalty, it will add to the total cost. And by putting this into our calculations what happens is we no longer actually look for the best fit anymore, **we look for a balance between the best fit and the smaller coefficients**. And a very, very small coefficient would be zero. Negative coefficients are also considered large.\n**Total cost** = measure of fit + measure of the magnitude of coefficients:\n![Total cost, not &quot;Total subtract quantity cost&quot;](../images/008.png)\nTotal cost, not \"Total subtract quantity cost\"\n  * When measure of fit (RSS)is small = good fit\n  * When the magnitude of the coefficients (sum of squared error) is small = good fit\n\n\nThis weight (w²)is actually called the L_2 norm if you know your linear algebra. All that matters for this story is just thinking about that as the square of the coefficients, sometimes called the square of the weights, but it is a coefficient.\nFrom a [previous article](https://towardsdatascience.com/an-introduction-to-linear-regression-9cbb64b52d23), we introduced epsilon, the error term. The terms m and b are coefficients (slope and y_intercept)\n![epsilon for error, i for which data point it is \\(first, second, etc.\\)](../images/009.png)\nepsilon for error, i for which data point it is (first, second, etc.)\nIt looks sort of like we have only one input column, namely x and one, coefficient b, but actually, depending on how we think about this, x could be a whole vector of columns, and could be a whole vector of coefficients.\n### Feature selection\nThe process of selecting a subset of features that are good predictors of the target is known as feature selection. As mentioned above, it is another term for model selection. The reason the two are the same is if you have a different set of features you actually have a different model.\nIt is useful for:\n  * controlling complexity of the model\n  * improving model performance\n  * improving generalization capability\n  * speeding up model learning w/o reducing accuracy\n\n\nMy personal preference is to use all the features, as much as I can, and then once I have all the features I can whittle down and find only the important features. And so I want to get rid of as many features as possible. Why do I want to do this? I want to have a simpler model. Why do I want a simpler model? Well, because a simpler model generalizes better. What does generalization mean? **It means that you can actually use it in the real world**. The model does not just explain the training data but it also has a better chance of explaining data that we have not seen before. First your test data and then the data that you've never seen before, namely your operational data.\nAs mentioned earlier stepwise regression is an important regularization tool. Backward is the most common, as in starting with all the features and removing columns that explain the least variance until model performance peaks. Forward is starting from just the mean and adding features until model performance peaks. Then there is \"both\" - at each step check whether add a feature or remove a feature. Say you are going forward, you added a column. When you go backward. Now from that point, it may not be that the best column to remove would be the one you just added. It could be another one. The reason for that is, is that there are complex interactions between the columns that often cannot be predicted. This may not be the case in the kind of linear regressions that I am doing. But in other types of regressions: trees and neural nets, that certainly can be the case.\nI venture to say everyone uses stepwise regression. Now, this does not mean you only use stepwise regression. Other types of regularization methods include ridge, lasso, and something called elastic net, which is a combination of ridge and lasso. And then I on top of that, I do the stepwise regression. So I think using a combination of these is the typical way that one progresses, that one works the best.\n## Stepwise regression: Python example\nThe following section walks through an example of stepwise regression using [Galton's height dataset](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/T0HSJ1), the same dataset I used in my [bootstrap resampling](https://towardsdatascience.com/bootstrap-resampling-2b453bb036ec) article.\ntext\n\n```\n# imports\n```\n\ntext\n\n```\nimport pandas as pd\nimport numpy as np\nimport statsmodels.formula.api as sm\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n```\n\ntext\n\n```\n%matplotlib inline\n```\n\ntext\n\n```\n# load data\n```\n\ntext\n\n```\n# I previously saved the Galton data as data\nfamily_data = pd.read_csv(data, delimiter='t')\n```\n\ntext\n\n```\n# column labels\nfamily_data.columns = [\"family\",\"father\",\"mother\",\"gender\",\"childHeight\", \"kids\"]\n```\n\ntext\n\n```\n# check data is well-imported\nfamily_data.head()\n```\n\n![First 5 rows, data looks good](../images/010.jpeg)\nFirst 5 rows, data looks good\ntext\n\n```\n# check data types\nfamily_data.dtypes\n```\n\n![](../images/body-0586f194248813a1b286.jpeg)\ntext\n\n```\n# subset the data with a Boolean flag, capture male children\nisMale = family_data.loc[:,\"gender\"] == \"M\"\n```\n\ntext\n\n```\n# create just the male children df\nmale_only = family_data[isMale].copy()\nmale_only.head()\n```\n\n![only \\(M\\)ale children captured in this df](../images/011.jpeg)\nonly (M)ale children captured in this df\ntext\n\n```\nprint('Number of rows: {}, Number of Males: {}'.format(len(family_data), len(male_only)))\n```\n\n![Data is well balanced, !M = F](../images/012.jpeg)\nData is well balanced, !M = F\ntext\n\n```\n# create new df for new feature\nmale_df = male_only.copy()\n```\n\ntext\n\n```\n# add in squares of mother and father heights\nmale_df['father_sqr'] = male_df['father'] **2\nmale_df['mother_sqr'] = male_df['mother'] **2\n```\n\ntext\n\n```\n# drop columns family, gender, kids\nDrop = [\"family\", \"gender\", \"kids\"]\nfor x in Drop:\n male_df = male_df.drop(x, axis=1)\n\n# reset index\nmale_df=male_df.reset_index(drop=True)\nmale_df.head()\n```\n\n![Successfully added new features](../images/013.jpeg)\nSuccessfully added new features\nN.B: The childHeight feature is just for sons, we copied and captured just the sons earlier. We do PCR analysis with daughters later.\nNext, we will z-score standardize the data. Why do we want to standardize the data? Larger values, like father_sqr, will have small coefficients, while smaller values will have large coefficients. This is not \"fair\". To make it fair we scale the data to make them on par with each other - and one way to do that is z-score standardization whereby we subtract from the height the mean of childHeight and divide by the standard deviation of childHeight. This is the z-score:\ntext\n\n```\n# scale all columns but the individual height (childHeight)\n# normalization function for a column in a pandas df\ndef scale(col):\n mean_col = np.mean(col)\n sd_col = np.std(col)\n std = (col - mean_col) / sd_col\n return std\n```\n\ntext\n\n```\n# add scaled x to data frame.\nmale_df['father'] = scale(male_df['father'])\nmale_df['mother'] = scale(male_df['mother'])\nmale_df['father_sqr'] = scale(male_df['father_sqr'])\nmale_df['mother_sqr'] = scale(male_df['mother_sqr'])\nmale_df.head()\n```\n\n![Values are scaled by std dev from column mean](../images/014.jpeg)\nValues are scaled by std dev from column mean\nNote that the first father's height is four. Four what? Four standard deviations away from the mean, which is pretty tall.\n### Compute model with all features\nThe following code computes the height of all male children using all available features. Up to this point, this is all I have shown, but what is special about today is that now we can find features that we can toss out. Why do we want to find features we can toss out? We want to make the model simpler. Why do we want to make the model simpler? It turns out that simpler models are better models.\nN.B: the '`+ 1`' in `ols_model` means that there is going to be an offset. `sklearn` does this automatically. There is always an intercept, another term for the offset. Since everything is scaled the intercept, in this case, is the mean height of the children.\ntext\n\n```\nols_model = sm.ols(formula = 'childHeight ~ father + mother + father_sqr + mother_sqr + 1', data=male_df)\n```\n\ntext\n\n```\nresults = ols_model.fit()\nn_points = male_df.shape[0]\ny_output = male_df['childHeight'].values.reshape(n_points, 1)\nprint('Intercept, Slopes : n{}'.format(results.params))\n```\n\ntext\n\n```\n# print  hypothesis test stats\nprint('Intercept t-value, Slope t-values: n{}'.format(results.tvalues))\nprint('nHypothesis test summary for each coefficient if they differ from zero:')\nprint(results.pvalues)\n```\n\ntext\n\n```\nprint('nSSE, SST, SSR, and RMSE:')\nmean_y = np.mean(y_output)\nsst = np.sum((y_output - mean_y)**2)\nsse = sst - results.ssr\nprint('SSE: {}'.format(sse))\nprint('SST: {}'.format(sst))\nprint('SSR: {}'.format(results.ssr))\nprint('RMSE: {}'.format(np.sqrt(results.mse_resid))) # ESHx\n```\n\ntext\n\n```\n# print summary stats\nprint(results.summary())\n```\n\ntext\n\n```\n# plot a histogram of the residuals\nsns.distplot(results.resid, hist=True)\nplt.xlabel('Residual')\nplt.ylabel('Frequency')\nplt.title('Residual Histogram')\nplt.show()\n```\n\n![](../images/body-653dbbde6fdea01ed779.jpeg)\n![Baseline model](../images/015.jpeg)\nBaseline model\n![Residuals appear normally distributed](../images/016.jpeg)\nResiduals appear normally distributed\nNow we have a baseline model, can we improve it with regularization techniques?\n### Apply stepwise regression\nFrom earlier, I defined the metric to use to define the fit of our model: AIC\n![AIC: model log-likelihood adjusted for the number of parameters](../images/017.png)\nAIC: model log-likelihood adjusted for the number of parameters\n  * k: number of estimated parameters\n  * L: likelihood\n\n\nWe can apply stepwise regression with this function, using backward feature selection:\ntext\n\n```\ndef backward_selected(data, response):\n \"\"\"Linear model designed by backward selection. Feature selection based on AIC\n```\n\ntext\n\n```\nParameters:\n - - - - - -\n data : pandas dfwith all possible predictors and response\n```\n\ntext\n\n```\nresponse: string, name of response column in data\n```\n\ntext\n\n```\nReturns:\n - - - - \n model: an \"optimal\" fitted statsmodels linear model\n with an intercept\n selected by backward selection\n \"\"\"\n # begin with all factors and intercept\n possible_factors = set(data.columns)\n possible_factors.remove(response)\n formula = \"{} ~ {} + 1\".format(response, ' + '.join(possible_factors))\n best_aic = sm.ols(formula, data).fit().aic\n current_aic = best_aic\n\n # create a non-empty set of columns that will be labeled as \"to remove and try\"\n to_try_remove = possible_factors\n\n # check which features remain\n while to_try_remove and current_aic == best_aic:\n aic_candidates = []\n for candidate in to_try_remove:\n\n columns = possible_factors - set([candidate])\n # removing the candidate column\n formula = \"{} ~ {} + 1\".format(response, ' + '.join(columns))\n # print AIC\n aic = sm.ols(formula, data).fit().aic\n\n # append tuple of the form (aic, response)\n aic_candidates.append((aic, candidate))\n\n # sort all the pairs by the first entry of tuple \n aic_candidates.sort()\n # change sort/pop order!\n best_new_aic, best_candidate = aic_candidates.pop(0)\n\n # check if we have something better:\n if best_new_aic < current_aic:\n # Remove the best candidate's name from possible_factors\n\n possible_factors.remove(best_candidate)\n current_aic = best_new_aic\n\n # repeat the process with all the remaining candidate columns\n```\n\ntext\n\n```\n# final formula\n formula = \"{} ~ {} + 1\".format(response, ' + '.join(possible_factors))\n # model object\n model = sm.ols(formula, data).fit()\n return model\n```\n\ntext\n\n```\nbackwards_model = backward_selected(male_df, 'childHeight')\n```\n\ntext\n\n```\nprint(backwards_model.model.formula)\n```\n\ntext\n\n```\nprint('Adjusted R-Squared: {}'.format(backwards_model.rsquared_adj))\nprint('AIC: {}'.format(backwards_model.aic))\n```\n\n![Backward stepwise regression, the AIC is better](../images/018.jpeg)\nBackward stepwise regression, the AIC is better\nThe backward selected formula `childHeight ~ father + mother + mother_sqr + 1` is now fed into a standard `sm` template:\ntext\n\n```\nols_model_forward = sm.ols(formula = 'childHeight ~ father + mother + mother_sqr + 1', data=male_df)\nresults = ols_model_forward.fit()\nn_points = male_df.shape[0]\ny_output = male_df['childHeight'].values.reshape(n_points, 1)\n```\n\ntext\n\n```\n# slope (m) and y-intercept (b)\nprint('Intercept, Slopes : n{}'.format(results.params))\n```\n\ntext\n\n```\n# t-values (hypothesis test statistics) \nprint('Intercept t-value, Slope t-values: n{}'.format(results.tvalues))\n```\n\ntext\n\n```\n# p-values for above t-value statistics\nprint('nHypothesis test summary for each coefficient if they differ from zero:')\nprint(results.pvalues)\n```\n\ntext\n\n```\nprint('nSSE, SST, SSR, and RMSE:')\nmean_y = np.mean(y_output)\nsst = np.sum((y_output - mean_y)**2)\nsse = sst - results.ssr\nprint('SSE: {}'.format(sse))\nprint('SST: {}'.format(sst))\nprint('SSR: {}'.format(results.ssr))\nprint('RMSE: {}'.format(np.sqrt(results.mse_resid))) # ESHx\nprint(results.summary())\n```\n\ntext\n\n```\n# plot a histogram of the residuals:\nsns.distplot(results.resid, hist=True)\nplt.xlabel('Residual')\nplt.ylabel('Frequency')\nplt.title('Residual Histogram')\nplt.show()\n```\n\n![The partial slopes are the coefficients](../images/019.jpeg)\nThe partial slopes are the coefficients\n![The R² is better](../images/020.jpeg)\nThe R² is better\n![Residuals appear normally distributed](../images/021.jpeg)\nResiduals appear normally distributed\nN.B: Stepwise regression appears to be a simple method for feature selection; be aware that **stepwise regression does not scale well**. As with any multiple comparison method, stepwise regression suffers from a high probability of false-positive results. In this case, a feature that should be dropped might not be, because of a low p-value or AIC.\n## Principle component regression: Python example\nThere is a lot of linear algebra that underlies PCR that I have omitted for brevity. The trick is to do a PCA, a principal component analysis. The PCA will help you determine which of the principal components are the best. This is done by creating linear combinations of the initial variables and creating a new one that explains the most amount of variance. Then another one is created that explains slightly less variance, and so on and so forth.\nFrom the models point of view, we have regularized it. No information has been thrown away, every PC contains linear combinations of all features. But the model sees new features that make it easy to determine which features should be thrown away.\nN.B: This relies on SVD behind the scenes.\nThe transformations in PCA create linearly independent features with an optimal amount of independence between them. PC's with small amounts of variance are likely unimportant to our model and thus we have made our model simpler by removing features with minuscule variance.\nFirst I will demonstrate PCR before applying it to the Galton dataset:\ntext\n\n```\nfrom sklearn.decomposition import PCA\nfrom sklearn.preprocessing import StandardScaler\n```\n\ntext\n\n```\nnp.random.seed(21)\n```\n\ntext\n\n```\n# right-multiply a 2x2 matrix to a 2x100 matrix to transform the points\nX = np.dot(np.random.randn(100, 2), np.random.rand(2, 2))\n# scale data\nX = StandardScaler().fit_transform(X)\n```\n\ntext\n\n```\nplt.scatter(X[:, 0], X[:, 1])\nplt.grid()\nplt.xlabel('x')\nplt.ylabel('y')\nplt.title('Generated x-y data with dependence')\nplt.show()\n```\n\n![](../images/body-58a7190dbe0c8fa3c8f2.jpeg)\nNow we will create linear combinations of variables to create PC1 and PC2 - principal component one and principal component two. PC1 will have the most amount of variance. Remember that there is a 1:1 mapping of input features to created principal components in PCA\ntext\n\n```\npca = PCA(n_components=2)\npca_result = pca.fit_transform(X)\npca_df = pd.DataFrame(data = pca_result , columns = ['pc1', 'pc2'])\n```\n\ntext\n\n```\nprint(pca_df.head())\nprint(pca_df.shape)\n```\n\n![100 pairs](../images/body-ce98415b077c3c14bb1c.jpeg)\n100 pairs\nPlot the output\ntext\n\n```\nplt.axis('equal') \nplt.scatter(pca_df.loc[:, 'pc1'], pca_df.loc[:, 'pc2'])\nplt.ylabel('Component 2')\nplt.xlabel('Component 1')\nplt.title('Two Dimensional PCA')\nplt.grid()\nplt.show()\n```\n\n![The data replotted in our new coordinate system.](../images/body-022436d0a9fe4909980a.jpeg)\nThe data replotted in our new coordinate system.\nPC1 and PC2 are linear combinations of the original features with some weightings attached in our new coordinate system (PC1 and PC2) as opposed to x and y.\ntext\n\n```\nprint(pca.explained_variance_)\nprint(pca.components_)\n```\n\n![These PC&#039;s explain 100% of the variance!](../images/body-390b744749ac776a2da5.jpeg)\nThese PC's explain 100% of the variance!\ntext\n\n```\nexp_var = pca.explained_variance_\ncomponents = pca.components_\n```\n\ntext\n\n```\n# plot vector\n# = plot scale * component direction * component length\nv1 = 2 * components[0] * np.sqrt(exp_var[0])\nv2 = 2 * components[1] * np.sqrt(exp_var[1])\nc = (0, 0) # Center is at 0,0 because of our standardization\n```\n\ntext\n\n```\nplt.scatter(X[:, 0], X[:, 1], alpha=0.5)\nplt.annotate('', c + v1, c, arrowprops={'arrowstyle': '->', 'shrinkA': 0,\n 'shrinkB': 0, 'linewidth': 3})\nplt.annotate('', c + v2, c, arrowprops={'arrowstyle': '->', 'shrinkA': 0,\n 'shrinkB': 0, 'linewidth': 3})\nplt.axis('equal')\nplt.grid()\nplt.title('F-vector functions (Principal Components)')\nplt.xlabel('x')\nplt.ylabel('y')\nplt.show()\n```\n\n![Vectors are indeed perpendicular in the new coordinate system](../images/body-9cdc20b5235d83873f9e.png)\nVectors are indeed perpendicular in the new coordinate system\nFrom PCR we have created two linearly independent features, as shown in the graph above. Because PC1 explains so much more variance than PC2, this model can likely be simplified by removing PC2.\nThis can be extended to higher-dimensional datasets. By calculating the principal components we have removed any multi-collinearity issues in the data.\nNow, let us use PCR on the Galton dataset - looking at daughters. Since there are four features in the df, there will be four principal components after PCA.\ntext\n\n```\n# subset the data with a Boolean flag to capture daughters\nisFemale = family_data.loc[:,\"gender\"] == \"F\"\n```\n\ntext\n\n```\n# create just the female df\nfemale_data = family_data[isFemale].copy()\n```\n\ntext\n\n```\n# create new df for new feature set\nfemale_df = female_data.copy()\n```\n\ntext\n\n```\n# feature engineer squares of mother and father heights\nfemale_df['father_sqr'] = female_df['father'] **2\nfemale_df['mother_sqr'] = female_df['mother'] **2\n```\n\ntext\n\n```\n# drop columns for family, gender, kids\nDrop= [\"family\", \"gender\", \"kids\"]\nfor x in Drop:\n female_df = female_df.drop(x, axis=1)\n\n# reset the index\nfemale_df=female_df.reset_index(drop=True)\n```\n\ntext\n\n```\n# add scaled x to df\nfemale_df['father'] = scale(female_df['father'])\nfemale_df['mother'] = scale(female_df['mother'])\nfemale_df['father_sqr'] = scale(female_df['father_sqr'])\nfemale_df['mother_sqr'] = scale(female_df['mother_sqr'])\nfemale_df.head()\n```\n\n![Daughter only dataset looks fine](../images/body-79f3fe7b03885d5dfa9c.jpeg)\nDaughter only dataset looks fine\ntext\n\n```\n# calculate all the principal components (4)\nX = female_df[['father', 'mother', 'father_sqr', 'mother_sqr']].values\ny = female_df['childHeight']\npca = PCA()\npca_result = pca.fit_transform(X)\npca_df = pd.DataFrame(data = pca_result, columns=['pc1', 'pc2', 'pc3', 'pc4'])\n```\n\ntext\n\n```\n# our data projected onto the four principal components.\nprint(pca_df.head())\nprint(pca_df.shape)\n```\n\ntext\n\n```\npca_df['childHeight'] = female_df['childHeight']\n```\n\n![](../images/body-66ff4f27ddba3b3736db.png)\nScree plot of variance:\ntext\n\n```\nplt.plot([i + 1 for i in range(4)], pca.explained_variance_)\nplt.title('Scree plot for PCA decoposition')\nplt.xlabel('Component')\nplt.ylabel('Variance explained')\nplt.show()\n```\n\n![The first two components explain the most variance](../images/body-1f03db42219ef2312202.png)\nThe first two components explain the most variance\ntext\n\n```\npca.explained_variance_\n```\n\n![Explained variance by four PCs](../images/body-9aa609604c006f56492b.jpeg)\nExplained variance by four PCs\nNow that we have all our data projected on our 4 total principal components, we can look at the explained variance. The first two PC's explain the most variance and thus we can likely drop PC3 and PC4 since their explained variance is so low.\ntext\n\n```\npcr_model = sm.ols(formula = 'childHeight ~ pc1 + pc2 + pc3 + pc4', data=pca_df)\n```\n\ntext\n\n```\nresults = pcr_model.fit()\nn_points = pca_df.shape[0]\ny_output = pca_df['childHeight'].values.reshape(n_points, 1)\nprint(results.summary())\n```\n\ntext\n\n```\n# plot a histogram of the residuals\nsns.distplot(results.resid, hist=True)\nplt.xlabel('Residual')\nplt.ylabel('Frequency')\nplt.title('Residual Histogram')\nplt.show()\n```\n\n![](../images/body-72020d115a8e714460ed.jpeg)\nHere, you may see that the last two principal components are not needed for a significant fit. You could also have noticed that from the 'explained variance' for each component- and how the last two components explained nearly 4 orders of magnitude less variance.\nNow, we can try the PCA regression with only two components; the components that explain most of the variance:\ntext\n\n```\npcr_model = sm.ols(formula = 'childHeight ~ pc1 + pc2', data=pca_df)\n```\n\ntext\n\n```\nresults = pcr_model.fit()\nn_points = pca_df.shape[0]\ny_output = pca_df['childHeight'].values.reshape(n_points, 1)\nprint(results.summary())\n```\n\ntext\n\n```\n# plot a histogram of the residuals\nsns.distplot(results.resid, hist=True)\nplt.xlabel('Residual')\nplt.ylabel('Frequency')\nplt.title('Residual Histogram')\nplt.show()\n```\n\n![Notice that all of the model coefficients are now significant](../images/body-1fc5efd07804fe38f0a1.png)\nNotice that all of the model coefficients are now significant\nThe p-values of PC3 and PC4 are large and thus not significant. Also, their respective 95% confidence intervals straddle zero. Thus they do not help the model explain daughter heights, which we suspected from the low variance they explained. If we had not performed PCR we likely could not have thrown out any of the initial features (father, mother, father_sqr, mother_sqr).\nIf you notice that the R² is not better on the improved model, remember that we committed a non-kosher act: we tested our model on training data. A big no-no. Almost certainly, if we had fed previously unseen test data into the last model, the one with only two PC's, the R² would have increased. This is because this is a simpler model, one less prone to overfitting. If the data were train/tune/test split then the model improvements would be demonstrable.\nThe reduction in dimensionality has worked as expected! The adjusted R² and AIC are the lowest for this model. Regularization has helped us create better, simpler, more generalizable models.\nIf you are curious as to why parent heights are rather poor indicators of child heights, this is where the phrase \"regression to the mean\" comes from. Contrary to popular belief in the 18th century, Galton's family analysis showed that tall parents tend to have children whose heights regress to the population mean.\nNext, I will talk about regularization with [Ridge, LASSO, and Elasticnet regressions](https://james-andrew.medium.com/ridge-lasso-and-elasticnet-regression-b1f9c00ea3a3)!\nFind me on [Linkedin](https://www.linkedin.com/in/james-a-w-godwin/)\n_Physicist cum Data Scientist- Available for new opportunity | SaaS | Sports | Start-ups | Scale-ups_\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-92fdfcb3140ff02db487.jpeg)\nWritten By\n## [James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\n[See all from James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\n[Data Science](https://towardsdatascience.com/tag/data-science/)[Linear Regression](https://towardsdatascience.com/tag/linear-regression/)[Python](https://towardsdatascience.com/tag/python/)[Regularization](https://towardsdatascience.com/tag/regularization/)[Towards Data Science](https://towardsdatascience.com/tag/towards-data-science/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F&title=Regularization%20and%20Linear%20Regression)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F&title=Regularization%20and%20Linear%20Regression)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F&text=Regularization%20and%20Linear%20Regression)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/022.png)](https://towardsdatascience.com/an-overview-of-feature-selection-1c50965551dd/)\n## [An Overview of Feature Selection](https://towardsdatascience.com/an-overview-of-feature-selection-1c50965551dd/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWith a presentation of a rarely used, highly effective method for establishing the predictive power of a given set of features\n[W Brett Kennedy](https://towardsdatascience.com/author/wkennedy934/)\nJanuary 7, 202523 min read\n  * [![](../images/023.jpg)](https://towardsdatascience.com/college-basketballs-net-rankings-explained-25faa0ce71ed/)\n## [College Basketball's NET Rankings Explained](https://towardsdatascience.com/college-basketballs-net-rankings-explained-25faa0ce71ed/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nHow data science influences March Madness\n[Giovanni Malloy](https://towardsdatascience.com/author/gspmalloy/)\nMarch 8, 202313 min read\n  * [![](../images/024.jpeg)](https://towardsdatascience.com/my-experience-as-a-product-data-analyst-3d01748bc6ea/)\n## [My Experience as a Product Data Analyst](https://towardsdatascience.com/my-experience-as-a-product-data-analyst-3d01748bc6ea/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow supporting product made me a better data analyst\n[Vicky Yu](https://towardsdatascience.com/author/madfordata/)\nMarch 8, 20215 min read\n  * [![](../images/025.png)](https://towardsdatascience.com/where-did-the-least-square-come-from-3f1abc7f7caf/)\n## [Where did the least-square come from?](https://towardsdatascience.com/where-did-the-least-square-come-from-3f1abc7f7caf/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat would you say in a machine learning interview, if asked about the mathematical basis of the least-square loss function?\n[Tirthajyoti Sarkar](https://towardsdatascience.com/author/tirthajyoti/)\nSeptember 13, 20187 min read\n  * [![](../images/026.png)](https://towardsdatascience.com/the-a-z-of-transformers-everything-you-need-to-know-c9f214c619ac/)\n## [The A-Z of Transformers: Everything You Need to Know](https://towardsdatascience.com/the-a-z-of-transformers-everything-you-need-to-know-c9f214c619ac/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nEverything you need to know about Transformers, and how to implement them\n[François Porcher](https://towardsdatascience.com/author/francoisporcher/)\nOctober 25, 202316 min read\n  * [![](../images/027.jpeg)](https://towardsdatascience.com/use-pandas-like-jedi-ab24b6519c22/)\n## [Use Pandas Like Jedi](https://towardsdatascience.com/use-pandas-like-jedi-ab24b6519c22/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nSmall yet functional tips for Pandas\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nDecember 7, 20204 min read\n  * [![](../images/028.jpg)](https://towardsdatascience.com/learn-to-implement-edit-distance-from-scratch-7a6f34412d07/)\n## [Learn to implement Edit Distance from Scratch](https://towardsdatascience.com/learn-to-implement-edit-distance-from-scratch-7a6f34412d07/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nEdit Distance or Levenstein distance (the most common) is a metric to calculate the similarity between a pair of sequences.\n[Prateek Jain](https://towardsdatascience.com/author/prateekkrjain/)\nFebruary 9, 202112 min read\n  * [![](../images/029.jpeg)](https://towardsdatascience.com/how-t-sne-outperforms-pca-in-dimensionality-reduction-7a3975e8cbdb/)\n## [How t-SNE Outperforms PCA in Dimensionality Reduction](https://towardsdatascience.com/how-t-sne-outperforms-pca-in-dimensionality-reduction-7a3975e8cbdb/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nPCA vs t-SNE for visualizing high-dimensional data in a lower-dimensional space\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nMay 23, 202315 min read\n  * [![](../images/030.jpg)](https://towardsdatascience.com/algorithms-from-scratch-linear-regression-c654353d1e7c/)\n## [Algorithms From Scratch: Linear Regression](https://towardsdatascience.com/algorithms-from-scratch-linear-regression-c654353d1e7c/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDetailing and Building a Linear Regression model from scratch\n[Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/)\nJuly 8, 20209 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "metadata": {
    "title": "Regularization and Linear Regression | Towards Data Science",
    "description": "The one where we correct overfitting",
    "keywords": null,
    "author": "James Andrew Godwin",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Regularization and Linear Regression",
    "og:description": "The one where we correct overfitting",
    "og:url": "https://towardsdatascience.com/regularization-and-linear-regression-bcaeba547c46/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1sNx2S1HbkqJRJoQ4o8rRfQ.jpeg",
    "og:image:width": "640",
    "og:image:height": "427",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "James Andrew Godwin",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "23 min read",
    "article:published_time": "2021-03-25T16:34:09.000Z",
    "article:modified_time": "2021-03-25T16:34:09.000Z",
    "article:author": "https://towardsdatascience.com/author/james-andrew/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 202,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_000.txt",
      "chars": 10
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_001.txt",
      "chars": 130
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_002.txt",
      "chars": 19
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_003.txt",
      "chars": 12
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_004.txt",
      "chars": 92
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_005.txt",
      "chars": 98
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_006.txt",
      "chars": 49
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_007.txt",
      "chars": 38
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_008.txt",
      "chars": 105
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_009.txt",
      "chars": 91
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_010.txt",
      "chars": 90
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_011.txt",
      "chars": 59
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_012.txt",
      "chars": 138
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_013.txt",
      "chars": 191
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_014.txt",
      "chars": 221
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_015.txt",
      "chars": 241
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_016.txt",
      "chars": 106
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_017.txt",
      "chars": 172
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_018.txt",
      "chars": 207
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_019.txt",
      "chars": 278
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_020.txt",
      "chars": 47
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_021.txt",
      "chars": 165
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_022.txt",
      "chars": 119
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_023.txt",
      "chars": 84
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_024.txt",
      "chars": 50
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_025.txt",
      "chars": 1333
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_026.txt",
      "chars": 155
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_027.txt",
      "chars": 60
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_028.txt",
      "chars": 37
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_029.txt",
      "chars": 114
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_030.txt",
      "chars": 225
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_031.txt",
      "chars": 88
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_032.txt",
      "chars": 113
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_033.txt",
      "chars": 144
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_034.txt",
      "chars": 303
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_035.txt",
      "chars": 166
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_036.txt",
      "chars": 87
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_037.txt",
      "chars": 19
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_038.txt",
      "chars": 181
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_039.txt",
      "chars": 132
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_040.txt",
      "chars": 128
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_041.txt",
      "chars": 41
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_042.txt",
      "chars": 182
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_043.txt",
      "chars": 54
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_044.txt",
      "chars": 63
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_045.txt",
      "chars": 221
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_046.txt",
      "chars": 380
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_047.txt",
      "chars": 105
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_048.txt",
      "chars": 71
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_049.txt",
      "chars": 67
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_050.txt",
      "chars": 156
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_051.txt",
      "chars": 191
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_052.txt",
      "chars": 250
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_053.txt",
      "chars": 270
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_054.txt",
      "chars": 98
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_055.txt",
      "chars": 49
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_056.txt",
      "chars": 174
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_057.txt",
      "chars": 24
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_058.txt",
      "chars": 81
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_059.txt",
      "chars": 139
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_060.txt",
      "chars": 165
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_061.txt",
      "chars": 69
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_062.txt",
      "chars": 139
    },
    {
      "language": "txt",
      "file": "attachments/0dac0d1e8663/code/block_063.txt",
      "chars": 165
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:40:54Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:40:54Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:40:55Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:40:55Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/james-andrew/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/linear-regression-with-bootstrapping-4924c05d2a9"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/bayesian-statistics-11f225174d5a"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/an-introduction-to-linear-regression-9cbb64b52d23"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/bootstrap-resampling-2b453bb036ec"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/linear-regression/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/python/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/regularization/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/towards-data-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/an-overview-of-feature-selection-1c50965551dd/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/wkennedy934/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/@korpa"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-data-scientist"
    },
    {
      "type": "external",
      "href": "https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910%2FDVN%2FT0HSJ1"
    },
    {
      "type": "external",
      "href": "https://james-andrew.medium.com/ridge-lasso-and-elasticnet-regression-b1f9c00ea3a3"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/in/james-a-w-godwin/"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Regularization+and+Linear+Regression&u=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Regularization+and+Linear+Regression&url=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Regularization+and+Linear+Regression&url=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh"
    },
    {
      "type": "external",
      "href": "https://insightmediagroup.io/creator-network"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:24:48Z",
  "source_url": "https://towardsdatascience.com/regularization-and-linear-regression-bcaeba547c46/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# Regularization and Linear Regression\nThe one where we correct overfitting\n[James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\nMarch 25, 202123 min read\n![Regularization and Linear Regression | Photo by Jr Korpa](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1sNx2S1HbkqJRJoQ4o8rRfQ.jpeg)\nRegularization and Linear Regression | Photo by [Jr Korpa](https://unsplash.com/@korpa)\nThis article is a continuation of my series on [linear regression and bootstrap](https://towardsdatascience.com/linear-regression-with-bootstrapping-4924c05d2a9) and [Bayesian statistics](https://towardsdatascience.com/bayesian-statistics-11f225174d5a).\nPreviously I talked at length about linear regression, and now I am going to continue that topic. As I hinted at previously, I am going to bring up the topic of regularization. And what regularization does, is it simplifies the model. And the reason why you want to have a simpler model is that, usually, a simpler will model will perform better in most if not pretty much all of the tests that can be run.\nN.B: It is not kosher to use training data as the test data. You only clearly see the full effects of regularization when you test your models with test data, which is data that the model has not yet seen.\nMy writing is primarily coming from the statistical modeling of things with less in the machine learning effectiveness. For simplicity, many of the following examples break this rule and we evaluate our models using training data and not test data.\nAs such, **the coefficients of determinations in our models are not truly representative of the models' effectiveness, rather they are only correlation coefficients**.\nLearn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) roadmap.\n### Finding the best model\nModel selection - sometimes referred to as feature selection, is when we select the best model for the data we have using different features/attributes/columns of available data. Choosing different features for your model essentially creates a different model.\nEither term is fine. After choosing our model we will then regularize it. Regularize is to simplify it: **Regularization = Simplification**.\nThe reason why we want a simpler model is that we want to avoid a phenomenon called overfitting or over-parameterization. Overfitting or over parameterization is a phenomenon that occurs very often, especially if you have a lot of parameters.\nThere are a variety of regularization techniques to use, including:\n  * Stepwise\n  * Ridge regression\n  * Lasso\n  * PCR - PCA + regression (principal component analysis and regularization)\n\n\nPCR is the most important technique to use for regularization.\n### Model fitness\nBroadly speaking, a good model \"fits the data well.\" However, there is nuance to this as there a**re different uses for different models**. If you have a data set and you want to know if this data set can be modeled; if you can use a mathematical formulation to explain the model, the data that you have - and this is all you want to do with it- then it is very good and legitimate to use that same data and evaluate your model using that data, or evaluating our model with our training data.\nHowever, if you want to be able to do any kind of prediction and determine how powerful this model is at predicting something - and that is usually what people want to do with a model: they want to actually make use of it - then **we need to test the model with test data**.\nA good model needs to generalize well. For example, it needs to generalize from your training data to your test data. Consider the following examples:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1ZYO4utfl2FcmDG-acgNvpg.jpeg)\n  * **Overfitting** : model works great for training data but does not generalize to real world\n  * **Underfitting** : model is too simple, generalizable but not meaningful. Less variance but more bias towards the wrong answers\n\n\nWhy do I call the left-most image a great fit, the one that is just right? Well, because when we eyeball this, the black line seems to explain those blue dots pretty well. If we were to do a simple linear regression, we might end up with something like the line on the center image. And we would be calling that underfitting. We would have a lack of variance in our model, and we would have too much bias in our model; that is what we would say.\nNow, on the right-hand side, we are overfitting. Here, we have too much variance in the model. So the model variance is pretty clear because it's going up and down, jumping all around. That variance is helping with the fitting of this particular data set. But we call this overfitting, we say it is fitting the data too well. You might ask well, how/why is that a bad thing? Well, it is a bad thing because if I were to come along and put in test data, the test data probably would fit worse than either of the other two models. Because the test data would not be exactly at these peaks or troughs of these maximas or minimas of this black line.\n### Overfitting\nOver-parameterization leads often to overfitting and it also carries some other problems with it:\n  * Number of parameters > dimensionality of data\n  * Linear dependency of input features\n  * Works great on the training data set but has poor (if not abysmal) performance in the real world (test data)\n\n\nThere is a problem with what we call linear dependency of input features. So if one of your input features can be explained by other input features in a linear manner, then we call that a linear dependency. Translated that means if you can linearly combine (that means to add some set of columns to come up with or determine another column that already is there, just through addition, subtraction, and multiplication) then that means that you have a linear dependency; that means one of the columns, one or more of the columns is just a combination of the other columns. And that leads to problems when you do calculations and determine your model.\n### Prevent overfitting\nThe tool to prevent overfitting is regularization. There are different ways of doing regularization. From earlier, stepwise regression is one form of doing regularization. What is stepwise regression? Removing/adding features in a stepwise fashion. There are two flavors, forward and backward. Forward stepwise regression pretty much works like this: you choose what you think is your best feature. Say you have input features x_1, x_2, x_3, x_4, and so on; you choose the one that you think is best (there are a variety of ways that you could choose it.) And then you come, you will create a linear regression, based off of that feature, say it was x_2, which now is the input and y is the output. Then you select your next best, say x_4, and you try that one out with x_2. And if the model gets better, by some amount, then you keep the second x value. If it was not good, if the model performance does not improve, then you do not keep it and then you are back to having only x_2. Then you try out another one of these features. This is called forward stepwise regression.\nBackward stepwise regression is the more common way of doing things. You take all of your values like x_1, x_2, x_3, x_4, and create a linear regression from those four inputs to come up with a function that explains y; that calculates/ predicts y. And then you remove one of the columns and see if the model has improved. And typically, you will fit things less well. But your model might improve based on criteria that are not just purely fitting, they might fit better, because you also put in a penalty for parameters or you use a measure called information criterion that says you need to gain so and so much information from any given input parameter. Continue removing features until the model performance stops improving. This is called backward stepwise regression.\nOther regularization techniques are you put a penalty in for every column you have; a penalty for the coefficient that is associated with that column. Those are other regularization techniques.\n### Measuring the model\nHow do we measure the model? There is the Akaike information criterion (AIC):\n![AIC formula](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/09Sk_TbNABYgz2ZLO.png)\nAIC formula\n  * k: number of estimated parameters\n  * L: likelihood\n\n\nThis is the general favorite metric to determine if a model has improved. As the AIC becomes smaller and smaller for comparable models you say that the AIC is improving and that your model is also improving. So by adding more columns, or by removing more columns, if your AIC is improving, then you know you are going in the right direction. If your AIC does not improve, that means non-decreasing, then you know that you took a step in the wrong direction; meaning the last addition of a column or the last removal of a column should be undone, you go back and try something else.\nAnother model metric is the Bayesian information criterion (BIC), but it is rarely used in this context - it is very similar to AIC:\n![BIC metric](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/0IS9Bl9YtageWSsf3.png)\nBIC metric\n  * k: number of estimated parameters\n  * L: likelihood\n  * N: number of recorded measurements\n\n\nRegardless of the criterion you choose, your model of best fit is the one that has the minimal AIC or BIC.\n### Cost\nWhen you determine any function, any machine learning algorithm has a so-called cost function. And this is a very fundamental concept. A cost function basically says, \"how bad is your model?\"\nI discussed previously on linear regression, badness would be the square of the residuals. As a reminder, a residual is the difference between the predicted value and the actual value of y, of the output; then you square that, and then you sum up all those squares. This is a cost function but was not explicitly called one before.\n![RSS is also referred to as SSR, but they are not always the same thing!](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/0akcxQuKHkl-xDhwK.png)\nRSS is also referred to as SSR, but they are not always the same thing!\nN.B: In some literature, SSR is not the residual sum of squares, but the sum of squares regression - a different quantity. For the rest of the article, I stick with RSS to avoid confusion about the above quantity.\nThere are a few bells and whistles on this that we change here and there, we can divide this by the number of cases, which means the number of values that are being used - the number of data points. We can put in a constant here, but overall we want to minimize this:\n![Minimize the cost function](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/0GOlW2DeKjr7Dk88Z.png)\nMinimize the cost function\nThat is probably one of the most common cost functions. There are others, especially for logistic regression. However, for the most part, this is our primary cost function - it might be modified in some minor ways.\nA cost function calculates the penalty. So the penalty here would be the sum of those squares: calculating how bad things are. This is a penalty for a given trained modeling data set. And the idea is to minimize this penalty or to minimize this cost. And so you need a few things to watch out for:\n  * models need to be compared to each other\n  * a cost function determines the variance\n  * depending on the dataset, the target variable, and purpose (business question to be answered), selecting the right cost function is critical and essential\n\n\nIt is important that similar models have to be compared to each other. So you cannot compare the root mean square error (RMSE), which is very closely related to the cost function above, you cannot really compare that from one model to the next unless they are very, very similar models - unless there are a lot of things the same.\nIf you have just two different models from different sources doing different having different purposes, **comparing the RMSE will not mean anything**. These root mean squared errors, or these costs, only are understandable in terms of when they are compared to another cost of a very similar model.\n### Variance and bias\nA good cost function tries to balance variance and bias to make it the best combination of variance and bias.\nWhat are variance and bias?\n![Demonstrating variance vs bias](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1ZYO4utfl2FcmDG-acgNvpg.jpeg)\nDemonstrating variance vs bias\nIn the overfitting example on the right-hand side, there is a strong variance, there is too much variance; while in the center underfitting example there is a strong bias, there is too much bias.\n### Cost in regression\nThe measure of fit is another way of saying minimizing the cost function. So far we have been minimizing the residual sum of squares. Very confusingly this can be known as SSR and RSS, I will use RSS. Another component of the total cost is the weight of our coefficients, just another way of saying coefficients.\nTo remind you, a coefficient in this particular situation where we, let us say we have two columns: column x_1and x_2. Well, then we would have:\n![b and c are the coefficients](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/0BHYbeG6Bv6c7WE_h.png)\nb and c are the coefficients\nSince the coefficients are part of our regression (we treated them as 1 earlier) we should include them in our total cost function. Actually, we add the square of those in, and we get another penalty for just having that coefficient.\nNow if that coefficient goes to zero, it basically wipes out the column. So that way, we can actually get rid of columns. Or get rid of this penalty, or part of this penalty. Naturally, if we get rid of all the columns, and we have nothing left, then all we have is the mean of y itself.\nSo we need, when we add a column, we are going to say, \"okay, we can add a column and we can use that information. But we are going to accept a penalty for adding that column.\" **Whatever coefficient that column gets, is going to be our penalty**. That means if it is a pretty big, important column, it is going to have a big coefficient, so it had better be worth it. Because that coefficient is also going to count as a part of this so-called penalty, it will add to the total cost. And by putting this into our calculations what happens is we no longer actually look for the best fit anymore, **we look for a balance between the best fit and the smaller coefficients**. And a very, very small coefficient would be zero. Negative coefficients are also considered large.\n**Total cost** = measure of fit + measure of the magnitude of coefficients:\n![Total cost, not &quot;Total subtract quantity cost&quot;](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/0Q_hvMX_GIufdILyK.png)\nTotal cost, not \"Total subtract quantity cost\"\n  * When measure of fit (RSS)is small = good fit\n  * When the magnitude of the coefficients (sum of squared error) is small = good fit\n\n\nThis weight (w²)is actually called the L_2 norm if you know your linear algebra. All that matters for this story is just thinking about that as the square of the coefficients, sometimes called the square of the weights, but it is a coefficient.\nFrom a [previous article](https://towardsdatascience.com/an-introduction-to-linear-regression-9cbb64b52d23), we introduced epsilon, the error term. The terms m and b are coefficients (slope and y_intercept)\n![epsilon for error, i for which data point it is \\(first, second, etc.\\)](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/0wrv1rlRy7l88NetW.png)\nepsilon for error, i for which data point it is (first, second, etc.)\nIt looks sort of like we have only one input column, namely x and one, coefficient b, but actually, depending on how we think about this, x could be a whole vector of columns, and could be a whole vector of coefficients.\n### Feature selection\nThe process of selecting a subset of features that are good predictors of the target is known as feature selection. As mentioned above, it is another term for model selection. The reason the two are the same is if you have a different set of features you actually have a different model.\nIt is useful for:\n  * controlling complexity of the model\n  * improving model performance\n  * improving generalization capability\n  * speeding up model learning w/o reducing accuracy\n\n\nMy personal preference is to use all the features, as much as I can, and then once I have all the features I can whittle down and find only the important features. And so I want to get rid of as many features as possible. Why do I want to do this? I want to have a simpler model. Why do I want a simpler model? Well, because a simpler model generalizes better. What does generalization mean? **It means that you can actually use it in the real world**. The model does not just explain the training data but it also has a better chance of explaining data that we have not seen before. First your test data and then the data that you've never seen before, namely your operational data.\nAs mentioned earlier stepwise regression is an important regularization tool. Backward is the most common, as in starting with all the features and removing columns that explain the least variance until model performance peaks. Forward is starting from just the mean and adding features until model performance peaks. Then there is \"both\" - at each step check whether add a feature or remove a feature. Say you are going forward, you added a column. When you go backward. Now from that point, it may not be that the best column to remove would be the one you just added. It could be another one. The reason for that is, is that there are complex interactions between the columns that often cannot be predicted. This may not be the case in the kind of linear regressions that I am doing. But in other types of regressions: trees and neural nets, that certainly can be the case.\nI venture to say everyone uses stepwise regression. Now, this does not mean you only use stepwise regression. Other types of regularization methods include ridge, lasso, and something called elastic net, which is a combination of ridge and lasso. And then I on top of that, I do the stepwise regression. So I think using a combination of these is the typical way that one progresses, that one works the best.\n## Stepwise regression: Python example\nThe following section walks through an example of stepwise regression using [Galton's height dataset](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/T0HSJ1), the same dataset I used in my [bootstrap resampling](https://towardsdatascience.com/bootstrap-resampling-2b453bb036ec) article.\ntext\n\n```\n# imports\n```\n\ntext\n\n```\nimport pandas as pd\nimport numpy as np\nimport statsmodels.formula.api as sm\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n```\n\ntext\n\n```\n%matplotlib inline\n```\n\ntext\n\n```\n# load data\n```\n\ntext\n\n```\n# I previously saved the Galton data as data\nfamily_data = pd.read_csv(data, delimiter='t')\n```\n\ntext\n\n```\n# column labels\nfamily_data.columns = [\"family\",\"father\",\"mother\",\"gender\",\"childHeight\", \"kids\"]\n```\n\ntext\n\n```\n# check data is well-imported\nfamily_data.head()\n```\n\n![First 5 rows, data looks good](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1-5NQJDaKfnkRlM08soAwPw.jpeg)\nFirst 5 rows, data looks good\ntext\n\n```\n# check data types\nfamily_data.dtypes\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/18EdIPaIWohrNzh-QOsz-ow.jpeg)\ntext\n\n```\n# subset the data with a Boolean flag, capture male children\nisMale = family_data.loc[:,\"gender\"] == \"M\"\n```\n\ntext\n\n```\n# create just the male children df\nmale_only = family_data[isMale].copy()\nmale_only.head()\n```\n\n![only \\(M\\)ale children captured in this df](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1wT0cR1MgeGgjoRD9Cw2F5g.jpeg)\nonly (M)ale children captured in this df\ntext\n\n```\nprint('Number of rows: {}, Number of Males: {}'.format(len(family_data), len(male_only)))\n```\n\n![Data is well balanced, !M = F](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1I5aQv_etNTZb0Xv51t8NsQ.jpeg)\nData is well balanced, !M = F\ntext\n\n```\n# create new df for new feature\nmale_df = male_only.copy()\n```\n\ntext\n\n```\n# add in squares of mother and father heights\nmale_df['father_sqr'] = male_df['father'] **2\nmale_df['mother_sqr'] = male_df['mother'] **2\n```\n\ntext\n\n```\n# drop columns family, gender, kids\nDrop = [\"family\", \"gender\", \"kids\"]\nfor x in Drop:\n male_df = male_df.drop(x, axis=1)\n\n# reset index\nmale_df=male_df.reset_index(drop=True)\nmale_df.head()\n```\n\n![Successfully added new features](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1CwsIeu0V1dLgNcx_rP_cig.jpeg)\nSuccessfully added new features\nN.B: The childHeight feature is just for sons, we copied and captured just the sons earlier. We do PCR analysis with daughters later.\nNext, we will z-score standardize the data. Why do we want to standardize the data? Larger values, like father_sqr, will have small coefficients, while smaller values will have large coefficients. This is not \"fair\". To make it fair we scale the data to make them on par with each other - and one way to do that is z-score standardization whereby we subtract from the height the mean of childHeight and divide by the standard deviation of childHeight. This is the z-score:\ntext\n\n```\n# scale all columns but the individual height (childHeight)\n# normalization function for a column in a pandas df\ndef scale(col):\n mean_col = np.mean(col)\n sd_col = np.std(col)\n std = (col - mean_col) / sd_col\n return std\n```\n\ntext\n\n```\n# add scaled x to data frame.\nmale_df['father'] = scale(male_df['father'])\nmale_df['mother'] = scale(male_df['mother'])\nmale_df['father_sqr'] = scale(male_df['father_sqr'])\nmale_df['mother_sqr'] = scale(male_df['mother_sqr'])\nmale_df.head()\n```\n\n![Values are scaled by std dev from column mean](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1xBLgvMI6ZnmSx9MaRb_C2A.jpeg)\nValues are scaled by std dev from column mean\nNote that the first father's height is four. Four what? Four standard deviations away from the mean, which is pretty tall.\n### Compute model with all features\nThe following code computes the height of all male children using all available features. Up to this point, this is all I have shown, but what is special about today is that now we can find features that we can toss out. Why do we want to find features we can toss out? We want to make the model simpler. Why do we want to make the model simpler? It turns out that simpler models are better models.\nN.B: the '`+ 1`' in `ols_model` means that there is going to be an offset. `sklearn` does this automatically. There is always an intercept, another term for the offset. Since everything is scaled the intercept, in this case, is the mean height of the children.\ntext\n\n```\nols_model = sm.ols(formula = 'childHeight ~ father + mother + father_sqr + mother_sqr + 1', data=male_df)\n```\n\ntext\n\n```\nresults = ols_model.fit()\nn_points = male_df.shape[0]\ny_output = male_df['childHeight'].values.reshape(n_points, 1)\nprint('Intercept, Slopes : n{}'.format(results.params))\n```\n\ntext\n\n```\n# print  hypothesis test stats\nprint('Intercept t-value, Slope t-values: n{}'.format(results.tvalues))\nprint('nHypothesis test summary for each coefficient if they differ from zero:')\nprint(results.pvalues)\n```\n\ntext\n\n```\nprint('nSSE, SST, SSR, and RMSE:')\nmean_y = np.mean(y_output)\nsst = np.sum((y_output - mean_y)**2)\nsse = sst - results.ssr\nprint('SSE: {}'.format(sse))\nprint('SST: {}'.format(sst))\nprint('SSR: {}'.format(results.ssr))\nprint('RMSE: {}'.format(np.sqrt(results.mse_resid))) # ESHx\n```\n\ntext\n\n```\n# print summary stats\nprint(results.summary())\n```\n\ntext\n\n```\n# plot a histogram of the residuals\nsns.distplot(results.resid, hist=True)\nplt.xlabel('Residual')\nplt.ylabel('Frequency')\nplt.title('Residual Histogram')\nplt.show()\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1QO_NFeGUiIt9X-yDs-OlYA.jpeg)\n![Baseline model](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/14-QiUd1VOQ-s0zvpUYtI5g.jpeg)\nBaseline model\n![Residuals appear normally distributed](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1UELULmiMy8O4Zz-AYdbwCQ.jpeg)\nResiduals appear normally distributed\nNow we have a baseline model, can we improve it with regularization techniques?\n### Apply stepwise regression\nFrom earlier, I defined the metric to use to define the fit of our model: AIC\n![AIC: model log-likelihood adjusted for the number of parameters](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/09Sk_TbNABYgz2ZLO.png)\nAIC: model log-likelihood adjusted for the number of parameters\n  * k: number of estimated parameters\n  * L: likelihood\n\n\nWe can apply stepwise regression with this function, using backward feature selection:\ntext\n\n```\ndef backward_selected(data, response):\n \"\"\"Linear model designed by backward selection. Feature selection based on AIC\n```\n\ntext\n\n```\nParameters:\n - - - - - -\n data : pandas dfwith all possible predictors and response\n```\n\ntext\n\n```\nresponse: string, name of response column in data\n```\n\ntext\n\n```\nReturns:\n - - - - \n model: an \"optimal\" fitted statsmodels linear model\n with an intercept\n selected by backward selection\n \"\"\"\n # begin with all factors and intercept\n possible_factors = set(data.columns)\n possible_factors.remove(response)\n formula = \"{} ~ {} + 1\".format(response, ' + '.join(possible_factors))\n best_aic = sm.ols(formula, data).fit().aic\n current_aic = best_aic\n\n # create a non-empty set of columns that will be labeled as \"to remove and try\"\n to_try_remove = possible_factors\n\n # check which features remain\n while to_try_remove and current_aic == best_aic:\n aic_candidates = []\n for candidate in to_try_remove:\n\n columns = possible_factors - set([candidate])\n # removing the candidate column\n formula = \"{} ~ {} + 1\".format(response, ' + '.join(columns))\n # print AIC\n aic = sm.ols(formula, data).fit().aic\n\n # append tuple of the form (aic, response)\n aic_candidates.append((aic, candidate))\n\n # sort all the pairs by the first entry of tuple \n aic_candidates.sort()\n # change sort/pop order!\n best_new_aic, best_candidate = aic_candidates.pop(0)\n\n # check if we have something better:\n if best_new_aic < current_aic:\n # Remove the best candidate's name from possible_factors\n\n possible_factors.remove(best_candidate)\n current_aic = best_new_aic\n\n # repeat the process with all the remaining candidate columns\n```\n\ntext\n\n```\n# final formula\n formula = \"{} ~ {} + 1\".format(response, ' + '.join(possible_factors))\n # model object\n model = sm.ols(formula, data).fit()\n return model\n```\n\ntext\n\n```\nbackwards_model = backward_selected(male_df, 'childHeight')\n```\n\ntext\n\n```\nprint(backwards_model.model.formula)\n```\n\ntext\n\n```\nprint('Adjusted R-Squared: {}'.format(backwards_model.rsquared_adj))\nprint('AIC: {}'.format(backwards_model.aic))\n```\n\n![Backward stepwise regression, the AIC is better](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/178yKxli-f6-JaxErB1A0Xw.jpeg)\nBackward stepwise regression, the AIC is better\nThe backward selected formula `childHeight ~ father + mother + mother_sqr + 1` is now fed into a standard `sm` template:\ntext\n\n```\nols_model_forward = sm.ols(formula = 'childHeight ~ father + mother + mother_sqr + 1', data=male_df)\nresults = ols_model_forward.fit()\nn_points = male_df.shape[0]\ny_output = male_df['childHeight'].values.reshape(n_points, 1)\n```\n\ntext\n\n```\n# slope (m) and y-intercept (b)\nprint('Intercept, Slopes : n{}'.format(results.params))\n```\n\ntext\n\n```\n# t-values (hypothesis test statistics) \nprint('Intercept t-value, Slope t-values: n{}'.format(results.tvalues))\n```\n\ntext\n\n```\n# p-values for above t-value statistics\nprint('nHypothesis test summary for each coefficient if they differ from zero:')\nprint(results.pvalues)\n```\n\ntext\n\n```\nprint('nSSE, SST, SSR, and RMSE:')\nmean_y = np.mean(y_output)\nsst = np.sum((y_output - mean_y)**2)\nsse = sst - results.ssr\nprint('SSE: {}'.format(sse))\nprint('SST: {}'.format(sst))\nprint('SSR: {}'.format(results.ssr))\nprint('RMSE: {}'.format(np.sqrt(results.mse_resid))) # ESHx\nprint(results.summary())\n```\n\ntext\n\n```\n# plot a histogram of the residuals:\nsns.distplot(results.resid, hist=True)\nplt.xlabel('Residual')\nplt.ylabel('Frequency')\nplt.title('Residual Histogram')\nplt.show()\n```\n\n![The partial slopes are the coefficients](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1UAIFfIIwmqxKeBs4VmnUiw.jpeg)\nThe partial slopes are the coefficients\n![The R² is better](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1j01CKgY6kYF_pj6k9M40AA.jpeg)\nThe R² is better\n![Residuals appear normally distributed](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1gW9ye8PVq7n1GpgENzJ_nQ.jpeg)\nResiduals appear normally distributed\nN.B: Stepwise regression appears to be a simple method for feature selection; be aware that **stepwise regression does not scale well**. As with any multiple comparison method, stepwise regression suffers from a high probability of false-positive results. In this case, a feature that should be dropped might not be, because of a low p-value or AIC.\n## Principle component regression: Python example\nThere is a lot of linear algebra that underlies PCR that I have omitted for brevity. The trick is to do a PCA, a principal component analysis. The PCA will help you determine which of the principal components are the best. This is done by creating linear combinations of the initial variables and creating a new one that explains the most amount of variance. Then another one is created that explains slightly less variance, and so on and so forth.\nFrom the models point of view, we have regularized it. No information has been thrown away, every PC contains linear combinations of all features. But the model sees new features that make it easy to determine which features should be thrown away.\nN.B: This relies on SVD behind the scenes.\nThe transformations in PCA create linearly independent features with an optimal amount of independence between them. PC's with small amounts of variance are likely unimportant to our model and thus we have made our model simpler by removing features with minuscule variance.\nFirst I will demonstrate PCR before applying it to the Galton dataset:\ntext\n\n```\nfrom sklearn.decomposition import PCA\nfrom sklearn.preprocessing import StandardScaler\n```\n\ntext\n\n```\nnp.random.seed(21)\n```\n\ntext\n\n```\n# right-multiply a 2x2 matrix to a 2x100 matrix to transform the points\nX = np.dot(np.random.randn(100, 2), np.random.rand(2, 2))\n# scale data\nX = StandardScaler().fit_transform(X)\n```\n\ntext\n\n```\nplt.scatter(X[:, 0], X[:, 1])\nplt.grid()\nplt.xlabel('x')\nplt.ylabel('y')\nplt.title('Generated x-y data with dependence')\nplt.show()\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1Rukv8ccwG-QRdhRJmJsO7A.jpeg)\nNow we will create linear combinations of variables to create PC1 and PC2 - principal component one and principal component two. PC1 will have the most amount of variance. Remember that there is a 1:1 mapping of input features to created principal components in PCA\ntext\n\n```\npca = PCA(n_components=2)\npca_result = pca.fit_transform(X)\npca_df = pd.DataFrame(data = pca_result , columns = ['pc1', 'pc2'])\n```\n\ntext\n\n```\nprint(pca_df.head())\nprint(pca_df.shape)\n```\n\n![100 pairs](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1y8t5KMM15XdK3TWP695P6A.jpeg)\n100 pairs\nPlot the output\ntext\n\n```\nplt.axis('equal') \nplt.scatter(pca_df.loc[:, 'pc1'], pca_df.loc[:, 'pc2'])\nplt.ylabel('Component 2')\nplt.xlabel('Component 1')\nplt.title('Two Dimensional PCA')\nplt.grid()\nplt.show()\n```\n\n![The data replotted in our new coordinate system.](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/16c6Hjum490bBcVRyIxooPw.jpeg)\nThe data replotted in our new coordinate system.\nPC1 and PC2 are linear combinations of the original features with some weightings attached in our new coordinate system (PC1 and PC2) as opposed to x and y.\ntext\n\n```\nprint(pca.explained_variance_)\nprint(pca.components_)\n```\n\n![These PC&#039;s explain 100% of the variance!](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1hZKjf5zsYqdGM1mYNz1MiA.jpeg)\nThese PC's explain 100% of the variance!\ntext\n\n```\nexp_var = pca.explained_variance_\ncomponents = pca.components_\n```\n\ntext\n\n```\n# plot vector\n# = plot scale * component direction * component length\nv1 = 2 * components[0] * np.sqrt(exp_var[0])\nv2 = 2 * components[1] * np.sqrt(exp_var[1])\nc = (0, 0) # Center is at 0,0 because of our standardization\n```\n\ntext\n\n```\nplt.scatter(X[:, 0], X[:, 1], alpha=0.5)\nplt.annotate('', c + v1, c, arrowprops={'arrowstyle': '->', 'shrinkA': 0,\n 'shrinkB': 0, 'linewidth': 3})\nplt.annotate('', c + v2, c, arrowprops={'arrowstyle': '->', 'shrinkA': 0,\n 'shrinkB': 0, 'linewidth': 3})\nplt.axis('equal')\nplt.grid()\nplt.title('F-vector functions (Principal Components)')\nplt.xlabel('x')\nplt.ylabel('y')\nplt.show()\n```\n\n![Vectors are indeed perpendicular in the new coordinate system](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/10MhrlWw9L9-Qpg_FrVyDkQ.png)\nVectors are indeed perpendicular in the new coordinate system\nFrom PCR we have created two linearly independent features, as shown in the graph above. Because PC1 explains so much more variance than PC2, this model can likely be simplified by removing PC2.\nThis can be extended to higher-dimensional datasets. By calculating the principal components we have removed any multi-collinearity issues in the data.\nNow, let us use PCR on the Galton dataset - looking at daughters. Since there are four features in the df, there will be four principal components after PCA.\ntext\n\n```\n# subset the data with a Boolean flag to capture daughters\nisFemale = family_data.loc[:,\"gender\"] == \"F\"\n```\n\ntext\n\n```\n# create just the female df\nfemale_data = family_data[isFemale].copy()\n```\n\ntext\n\n```\n# create new df for new feature set\nfemale_df = female_data.copy()\n```\n\ntext\n\n```\n# feature engineer squares of mother and father heights\nfemale_df['father_sqr'] = female_df['father'] **2\nfemale_df['mother_sqr'] = female_df['mother'] **2\n```\n\ntext\n\n```\n# drop columns for family, gender, kids\nDrop= [\"family\", \"gender\", \"kids\"]\nfor x in Drop:\n female_df = female_df.drop(x, axis=1)\n\n# reset the index\nfemale_df=female_df.reset_index(drop=True)\n```\n\ntext\n\n```\n# add scaled x to df\nfemale_df['father'] = scale(female_df['father'])\nfemale_df['mother'] = scale(female_df['mother'])\nfemale_df['father_sqr'] = scale(female_df['father_sqr'])\nfemale_df['mother_sqr'] = scale(female_df['mother_sqr'])\nfemale_df.head()\n```\n\n![Daughter only dataset looks fine](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/12N8QnNoNjN2-QmIo2-jf9Q.jpeg)\nDaughter only dataset looks fine\ntext\n\n```\n# calculate all the principal components (4)\nX = female_df[['father', 'mother', 'father_sqr', 'mother_sqr']].values\ny = female_df['childHeight']\npca = PCA()\npca_result = pca.fit_transform(X)\npca_df = pd.DataFrame(data = pca_result, columns=['pc1', 'pc2', 'pc3', 'pc4'])\n```\n\ntext\n\n```\n# our data projected onto the four principal components.\nprint(pca_df.head())\nprint(pca_df.shape)\n```\n\ntext\n\n```\npca_df['childHeight'] = female_df['childHeight']\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1t_kQqgEyttDeaJlHI-rBEQ.png)\nScree plot of variance:\ntext\n\n```\nplt.plot([i + 1 for i in range(4)], pca.explained_variance_)\nplt.title('Scree plot for PCA decoposition')\nplt.xlabel('Component')\nplt.ylabel('Variance explained')\nplt.show()\n```\n\n![The first two components explain the most variance](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1tcqLGOPPznVR1buBWSkmDQ.png)\nThe first two components explain the most variance\ntext\n\n```\npca.explained_variance_\n```\n\n![Explained variance by four PCs](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/15-8WJl37_UaKIzWsfnzfZg.jpeg)\nExplained variance by four PCs\nNow that we have all our data projected on our 4 total principal components, we can look at the explained variance. The first two PC's explain the most variance and thus we can likely drop PC3 and PC4 since their explained variance is so low.\ntext\n\n```\npcr_model = sm.ols(formula = 'childHeight ~ pc1 + pc2 + pc3 + pc4', data=pca_df)\n```\n\ntext\n\n```\nresults = pcr_model.fit()\nn_points = pca_df.shape[0]\ny_output = pca_df['childHeight'].values.reshape(n_points, 1)\nprint(results.summary())\n```\n\ntext\n\n```\n# plot a histogram of the residuals\nsns.distplot(results.resid, hist=True)\nplt.xlabel('Residual')\nplt.ylabel('Frequency')\nplt.title('Residual Histogram')\nplt.show()\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/10PpRvq1D7DpGClmmWMrxig.jpeg)\nHere, you may see that the last two principal components are not needed for a significant fit. You could also have noticed that from the 'explained variance' for each component- and how the last two components explained nearly 4 orders of magnitude less variance.\nNow, we can try the PCA regression with only two components; the components that explain most of the variance:\ntext\n\n```\npcr_model = sm.ols(formula = 'childHeight ~ pc1 + pc2', data=pca_df)\n```\n\ntext\n\n```\nresults = pcr_model.fit()\nn_points = pca_df.shape[0]\ny_output = pca_df['childHeight'].values.reshape(n_points, 1)\nprint(results.summary())\n```\n\ntext\n\n```\n# plot a histogram of the residuals\nsns.distplot(results.resid, hist=True)\nplt.xlabel('Residual')\nplt.ylabel('Frequency')\nplt.title('Residual Histogram')\nplt.show()\n```\n\n![Notice that all of the model coefficients are now significant](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1EW1KKoQoKUG_LEhYirxocA.png)\nNotice that all of the model coefficients are now significant\nThe p-values of PC3 and PC4 are large and thus not significant. Also, their respective 95% confidence intervals straddle zero. Thus they do not help the model explain daughter heights, which we suspected from the low variance they explained. If we had not performed PCR we likely could not have thrown out any of the initial features (father, mother, father_sqr, mother_sqr).\nIf you notice that the R² is not better on the improved model, remember that we committed a non-kosher act: we tested our model on training data. A big no-no. Almost certainly, if we had fed previously unseen test data into the last model, the one with only two PC's, the R² would have increased. This is because this is a simpler model, one less prone to overfitting. If the data were train/tune/test split then the model improvements would be demonstrable.\nThe reduction in dimensionality has worked as expected! The adjusted R² and AIC are the lowest for this model. Regularization has helped us create better, simpler, more generalizable models.\nIf you are curious as to why parent heights are rather poor indicators of child heights, this is where the phrase \"regression to the mean\" comes from. Contrary to popular belief in the 18th century, Galton's family analysis showed that tall parents tend to have children whose heights regress to the population mean.\nNext, I will talk about regularization with [Ridge, LASSO, and Elasticnet regressions](https://james-andrew.medium.com/ridge-lasso-and-elasticnet-regression-b1f9c00ea3a3)!\nFind me on [Linkedin](https://www.linkedin.com/in/james-a-w-godwin/)\n_Physicist cum Data Scientist- Available for new opportunity | SaaS | Sports | Start-ups | Scale-ups_\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657234707_1wi608.jpeg)\nWritten By\n## [James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\n[See all from James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\n[Data Science](https://towardsdatascience.com/tag/data-science/)[Linear Regression](https://towardsdatascience.com/tag/linear-regression/)[Python](https://towardsdatascience.com/tag/python/)[Regularization](https://towardsdatascience.com/tag/regularization/)[Towards Data Science](https://towardsdatascience.com/tag/towards-data-science/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F&title=Regularization%20and%20Linear%20Regression)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F&title=Regularization%20and%20Linear%20Regression)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fregularization-and-linear-regression-bcaeba547c46%2F&text=Regularization%20and%20Linear%20Regression)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/1VzibOvv_bKTOshoL4yKjtQ.png)](https://towardsdatascience.com/an-overview-of-feature-selection-1c50965551dd/)\n## [An Overview of Feature Selection](https://towardsdatascience.com/an-overview-of-feature-selection-1c50965551dd/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWith a presentation of a rarely used, highly effective method for establishing the predictive power of a given set of features\n[W Brett Kennedy](https://towardsdatascience.com/author/wkennedy934/)\nJanuary 7, 202523 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/03/09ZMFgMJndIua_tQX-scaled.jpg)](https://towardsdatascience.com/college-basketballs-net-rankings-explained-25faa0ce71ed/)\n## [College Basketball's NET Rankings Explained](https://towardsdatascience.com/college-basketballs-net-rankings-explained-25faa0ce71ed/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nHow data science influences March Madness\n[Giovanni Malloy](https://towardsdatascience.com/author/gspmalloy/)\nMarch 8, 202313 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1RKjnDQ6T7Me-xeYHuydSwA-scaled.jpeg)](https://towardsdatascience.com/my-experience-as-a-product-data-analyst-3d01748bc6ea/)\n## [My Experience as a Product Data Analyst](https://towardsdatascience.com/my-experience-as-a-product-data-analyst-3d01748bc6ea/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow supporting product made me a better data analyst\n[Vicky Yu](https://towardsdatascience.com/author/madfordata/)\nMarch 8, 20215 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2018/09/1DxfQbIRcWBDuy1O5cGkEaw.png)](https://towardsdatascience.com/where-did-the-least-square-come-from-3f1abc7f7caf/)\n## [Where did the least-square come from?](https://towardsdatascience.com/where-did-the-least-square-come-from-3f1abc7f7caf/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat would you say in a machine learning interview, if asked about the mathematical basis of the least-square loss function?\n[Tirthajyoti Sarkar](https://towardsdatascience.com/author/tirthajyoti/)\nSeptember 13, 20187 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1vCqM2C1uL3JpkKcr3IolbA.png)](https://towardsdatascience.com/the-a-z-of-transformers-everything-you-need-to-know-c9f214c619ac/)\n## [The A-Z of Transformers: Everything You Need to Know](https://towardsdatascience.com/the-a-z-of-transformers-everything-you-need-to-know-c9f214c619ac/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nEverything you need to know about Transformers, and how to implement them\n[François Porcher](https://towardsdatascience.com/author/francoisporcher/)\nOctober 25, 202316 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1wNFtnAmiYa0RPOj_3cNgTg-scaled.jpeg)](https://towardsdatascience.com/use-pandas-like-jedi-ab24b6519c22/)\n## [Use Pandas Like Jedi](https://towardsdatascience.com/use-pandas-like-jedi-ab24b6519c22/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nSmall yet functional tips for Pandas\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nDecember 7, 20204 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/0szve6Stkbo4U-qIR-scaled.jpg)](https://towardsdatascience.com/learn-to-implement-edit-distance-from-scratch-7a6f34412d07/)\n## [Learn to implement Edit Distance from Scratch](https://towardsdatascience.com/learn-to-implement-edit-distance-from-scratch-7a6f34412d07/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nEdit Distance or Levenstein distance (the most common) is a metric to calculate the similarity between a pair of sequences.\n[Prateek Jain](https://towardsdatascience.com/author/prateekkrjain/)\nFebruary 9, 202112 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/1G5Ibbu86lRgTUfkI-DZ-Hg.jpeg)](https://towardsdatascience.com/how-t-sne-outperforms-pca-in-dimensionality-reduction-7a3975e8cbdb/)\n## [How t-SNE Outperforms PCA in Dimensionality Reduction](https://towardsdatascience.com/how-t-sne-outperforms-pca-in-dimensionality-reduction-7a3975e8cbdb/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nPCA vs t-SNE for visualizing high-dimensional data in a lower-dimensional space\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nMay 23, 202315 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/07/0uOMo-OrOXNaCzWif-scaled.jpg)](https://towardsdatascience.com/algorithms-from-scratch-linear-regression-c654353d1e7c/)\n## [Algorithms From Scratch: Linear Regression](https://towardsdatascience.com/algorithms-from-scratch-linear-regression-c654353d1e7c/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDetailing and Building a Linear Regression model from scratch\n[Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/)\nJuly 8, 20209 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "body_base_path": "attachments/0dac0d1e8663/pages/",
  "body_markdown_path": "attachments/0dac0d1e8663/pages/archived-body.md",
  "local_html": "attachments/0dac0d1e8663/pages/archived-body.html",
  "body_source": "prior_capture"
}