{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "James Andrew Godwin",
  "title_listed": "An Introduction to Supervised Learning",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/supervised-learning-61256f2aebeb/",
  "publish_date": "2022-02-26",
  "corpus_stage": "",
  "notes": "",
  "id": "fbf7561e44ff",
  "collected_at": "2026-10-02T08:33:58",
  "title": "An Introduction to Supervised Learning | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# An Introduction to Supervised Learning\nThe one with an example of the Naive Bayes classification algorithm\n[James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\nFebruary 26, 202223 min read\n![caption | Photo by Anni Roenkae](../images/000.jpeg)\ncaption | Photo by [Anni Roenkae](https://www.pexels.com/@anniroenkae)\nThis continues a series of articles that build on each other. Please read my last article on [Time Series Analysis](https://towardsdatascience.com/time-series-analysis-7138ec68754a) and [Logistic Regression](https://towardsdatascience.com/logistic-regression-cebee0728cbf).\nAlright, let us start with supervised learning. I covered some of this in my earlier articles but let me go over some specifics for this article.\n  * Supervised learning: learns from existing data which are categorized and labeled with predefined classes. Test data are labeled into these classes as well\n\n\nWell, this is true of all types of learning. But, specifically, it has **labeled predefined classes**. That is the most important thing - **supervised learning has something that is called an expert label**. That's a fancy word for meaning that it is labeled for an outcome; or for any given case, there is a known, desired outcome.\nUnsupervised learning (clustering) does not assume that it knows the answer. So clustering, for instance, will take a look at the data and partition the data, according to how the data are.\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\nSupervised learning will partition the data according to the label. This is a big difference. An example of unsupervised learning is clustering. An example of supervised learning is regression, as I have [written](https://towardsdatascience.com/an-introduction-to-linear-regression-9cbb64b52d23) about before.\nFor instance, in a regression model, we have X and Y, and we draw a best fit line through that. In this example, X are our inputs, and Y is the output. We can say this is supervised learning because Y is the expert label; the model is learning how to take X and come up with Y given these expert labels.\n## Examples:\n  * Fraud transaction: we know which transactions in the training data were fraud (1), and which were not (0)\n  * Readmission: we know which patients were readmitted to a hospital within a certain time window after discharge\n  * Recommendation: we know which items were presented to customers, and which items were clicked, added to the cart, or purchased\n  * A classic example is a spam filter\n\n\nMachine learning tasks have a variety of purposes. Here are a few examples of classic machine learning tasks: identifying fraud, identifying readmission of patients, identifying recommendations like your typical Netflix machine learning case that is when you go on to Netflix, you get recommendations of movies that you will like; those movies are based on previous movies that you looked at and considering what other people liked when they had watched similar movies to the movies you watched, and what movies they watched, in addition to that. This is called a recommendation engine.\n> As an aside, this recommendation engine is sort of a gray area, that might not always be described as supervised learning, but I digress.\nA classic example of supervised learning is a spam filter. What does a spam filter do? It classifies between spam and no spam on your email. The way the model learned this is by someone, an \"expert\" - a human, who came by and marked various emails as spam or not spam. Then the emails and those labels were given to an algorithm that was expected to figure out how to get from the email to that label on its own. This whole process is called supervised learning.\n## Typical machine learning tasks:\n  * **Classification:** can we label the input?\n  * **Regression** : can we make a prediction?\n  * **Recommendations** : can we predict something based on user preference?\n\n\nSupervised learning has two important steps: first, you train a model, and then you test the model. It is very important to use unseen test data for testing your model. In previous articles, I wrote about supervised learning, like with a regression, I did not use test data, and used training data to test the model. This is is bad, not a good thing to do. People do it on occasion, there are different reasons for doing it. There are very few legitimate reasons, but there are some. If you use training data to determine the accuracy of your model, then you are at best fooling yourself; and at worst, you are lying.\nSo to test the model, you need to use so-called \"unseen test data.\" Now, how do we create unseen test data? Well, that is pretty simple. You just take your modeling data and split it into training data and test data: the test data is used for the testing of the model after the model has been trained on the training segment.\nWhat do we mean by learning? You use the data together with the algorithm to come up with a performance measure or to minimize a cost function - that is the other way of saying it. When you do that, then you will come up with a model with new operational data and you will be able to get an answer.\n## Fundamental assumption of learning\nA fundamental assumption of learning is that the distribution of training examples is identical to the distribution of test examples, including future unseen examples. More importantly, the distribution of training examples is similar enough to future examples. Given that they are future examples, they are unseen.\nIf you want to have an accuracy measure, then that distribution should also be similar to the test examples. This assumption is not a requirement. What is a requirement is that your test examples, for you to trust your accuracy, have to come from the same distribution as your future examples.\nWhere your training examples came from is not important if you create a usable model. If your training examples come from a different distribution, more power to you. To be a valid measure, the test examples have to be similar enough to the examples that will be used in practice when it comes to operational data - when it comes to operationalizing. Now strong violations to the above assumption will result in poor classification accuracy.\n### Machine learning framework\n![Image by author](../images/001.jpeg)\nImage by author\n**Training:** given a _training_ set of labeled examples {(x_1, y_1, ... (x_n, y_n)}, estimate the prediction function (f) and parameters (Theta) which minimizes the prediction error on the training set\n![Training function | Image by author](../images/002.jpeg)\nTraining function | Image by author\n**Testing:** apply (f) to a never before seen _test_ example (x) and output the predicted value (y = f(x))\nWith the overall machine learning framework, you give an X and you get back a Y. And there is going to be somewhat of an error because there is some random noise associated with everything. That means that the actual Y that you get is not going to be exactly the above function.\n## **Classification Workflow**\n![Classification workflow diagram | Image by author](../images/003.jpeg)\nClassification workflow diagram | Image by author\nThe most important thing is the data have to be split into a training set and test set. The model is developed in the training set, the model is evaluated on the test set. Lastly, you get performance measures, also called accuracy measures. Here there are three performance measures: of which the performance measure called accuracy is probably the worst performance measure. Precision and recall are pretty good. There is a better one that is a hybrid of precision and recall, the [F1 score](https://en.wikipedia.org/wiki/F-score).\nThe reason I like it more is that it is sort of a hybrid between the recall and the precision. And you will want to have both good recall and precision. If you have very good recall and very lousy precision or vice versa, your model is not worth much. There is an even better accuracy measure for binary classifications - the ROC AUC.\nSo the AUC, area under the curve, of a ROC curve, is for classifications. The reason why it is better than the previous performance measures is that it is valid for all thresholds; whereas accuracy or precision presumes a threshold - typically it presumes a threshold of 0.5. Likewise, the F1 score also presumes a threshold of 0.5. You could choose a different threshold, but most people would not assume that you would have chosen a different threshold.\nThe reason why you choose a different threshold is that you may have a different purpose in mind, where you have false positives might be worse for you than false negatives, or vice versa. By being able to tune your threshold, you can trade those off.\n### Many classifiers to choose from\n  * Linear Regression\n  * Logistic Regression\n  * Support Vector Machine (SVM)\n  * Neural networks (ANN, CNN, RNN, DNN)\n  * Naive Bayes\n  * Bayesian network\n  * Randomized Forests (Random forests)\n  * Boosted Decision Trees (Gradient boosted trees)\n  * K-nearest neighbor (KNN)\n\n\nHere are so-called classifiers, algorithms used in supervised learning. KNN is generally considered your simplest concept and is often taught in classes like these because the algorithm is relatively simple: you can write out the algorithm from scratch.\nBoosted decision trees and randomized forests used to be your generally favored machine learning classification algorithms. Today the \"newest\" types of algorithms are the neural nets. And when I say the newest, my statement is about five to 10 years old, depending on how you look at it. But up to about five years ago, Kaggle competitions [were won](https://www.kaggle.com/msjgriffiths/r-what-algorithms-are-most-successful-on-kaggle/data?scriptVersionId=0) by boosted decision trees and randomized forests. Now Kaggle competitions are being won by deep neural nets. So not neural nets, as such, but a more sophisticated version of neural nets called deep neural nets (DNN).\nLogistic regression is everyone's favorite - whenever you do a classification, you first do a logistic regression, always! Naive Bayes is a favorite too in that same sense. But logistic regression maybe even a little bit more so. So you always do logistic regression; not because you want to show off the results of the logistic regression. It is just that it is so robust and you know if you cannot get the logistic regression to work, you will not be able to get anything else to work either. So logistic regression is the first thing you try to get to work. When that works, then you start becoming more daring and venture into some of these other algorithms listed above.\n## Generalization\n**Underfitting** : model is too \"simple\" to represent all the relevant class characteristics - high training error and high validation error.\n**Overfitting** : model is too \"complex\" and fits irrelevant characteristics (noise) in the data - low training error and high validation error (big gap between training and validation performance)\nGeneralization means how well it works. As a good heuristic **whenever you say generalization, you think of regularization**. Why do this? Because the way you generalize a model is by regularizing the model. If you have a well-regularized model, it is presumably generalized. What does generalization mean? It means it will work under more than just minimal circumstances. Thus no overfitting has occurred. An underfit model is generally too simple and more work needs to be done to extract information from it. But an overfit model is not generalized well enough and requires regularization.\n## Common pitfalls in machine learning\n  * **Overfitting**\n  * **Underfitting**\n  * **Data leakage** : the creation of unexpected additional information in the training data, creating unrealistically good predictions. Leaking test data into training data. The inclusion of data not present in the model's operational environment\n\n\nRarely is underfitting a pitfall, maybe the usage of algorithms which are too primitive and that results in sort of a way of underfitting because people might have stopped with logistic regression and they should have used a gradient boosted tree if you want to consider that a kind of underfitting. Overfitting is one of your big issues, as mentioned earlier.\nNow the next big issue is data leakage or proxy columns or proxies of any sort. These issues happen a lot. This concept is hard to understand, it is hard to explain, hard to teach why such things happen often unless you have done this yourself.\nFor example, where you have been looking at your data for a month, working with it, and only once your test results come back, and you see your test results for your machine learning algorithm like .97 on your accuracy. Only then do you realize, \"Oh, I must have data leakage.\" Simply put - it is vanishingly rare that you get a model that good. Then you have to dissect your model, you have to find out why your model is too good to be true. And the reason your model is too good to be true is usually because of data leakage.\n> A real-life example: a prostate cancer model included a variable stating whether or not patients had prostate cancer surgery. A model cannot see the future, and the data for whether or not the patient had surgery should have been removed from the model for training.\n## Confusion Matrix\n![Encoding for the confusion matrix](../images/004.jpeg)\nEncoding for the confusion matrix\nHere is an example of a confusion matrix. One important note about these confusion matrices is that they typically have the true negative in the upper left and the true positive in the lower right. The reason why this is the case, nowadays, is because zero is less than one. Since true negatives are usually modeled as zeros, it makes numeric sense to list them first. When you have your actual versus your predicted classes reduced to the easiest format for a computer to understand them, then you have reduced them to zero and one. Since zero is smaller than one zero goes first and one goes second. Since zeros are smaller than one, zeros go first also in the vertical dimension and then ones well. Zeros are typically associated with the absence of something and they are typically associated with a negative.\nHere a negative is not a bad thing - if I am COVID-19 negative, this means I do not have COVID, or at least my test came back negative for COVID. If I do not have COVID then this is what I call a true negative or 0 0. Likewise, the encoding of 1 1 would mean I do have COVID and I tested positive for it.\nThis is why true negatives and true positives are aligned this way: true negatives are in the upper left, true positives are in the lower right. This was not always the case - it used to be the other way around! But this is what you see most of the time these days. And that just has to do with the ordering of zero and one!\nPrecision and recall are important concepts to understand but are beyond the scope of this article. As I mentioned above, I prefer the F1 score, the harmonic mean (as opposed to the arithmetic mean) of precision and recall as my \"accuracy\" metric over choosing either one. But the best performance measure for classification is the ROC curve.\nThe Receiver Operating Characteristic (ROC) curve is great for classifications. I believe it is the most important accuracy measure but sometimes it cannot be used. There are reasons for that, but generally, the ROC curve is what you want to use. The reason why you want to use the ROC curve is that it considers all thresholds. To illustrate why this is important, consider this hypothetical interview question: give people numbers for false positive, for true positive, false negative, and true negative - there are there 10 in each of those. So that would give you about the worst accuracy score possible: 0.5\n> as an aside consider that if your model is .01 accurate, then just switch the labels for your 0 and 1 classes and suddenly your model is .99 accurate\nBut if you look at the probabilities, the probabilities for the true positives and the true negatives: are all-around zero and one - meaning they are good. The probabilities of the false positives and false negatives are all around 0.5, which means they are really bad. **This means the errors are shaky, and the correct results are very good or very secure;** thus you will get a higher ROC AUC score for that! You will never see anything like that with any of these other accuracy measures because **they do not consider probabilities. They only consider the boolean outcomes** : is it a true positive or is it not a true positive?\n## Naive Bayes\nSome properties of Naive Bayes models are\n  * Do not require a prior\n  * Computational complexity is linear in the number of parameters/features\n  * Requires minimal data to produce models that generalize well\n  * Have a simple and inherent regularization\n\n\nSome applications:\n  * Document classification\n  * SPAM detection\n  * Image classification\n\n\nYes, the Bayes in Naive Bayes does refer to Bayes theorem, which I have previously [written ](https://towardsdatascience.com/bayesian-statistics-11f225174d5a)about. There is also a very clear link between Bayes theorem and Naive Bayes model. However, the presumption with the Naive Bayes model is that the individual dimensions are independent, or naive. That means the important part about Bayes theorem, where you can relate the probability of A to the probability of B or the intersection is not needed. What we have here in the naive consideration is that we say that these probabilities are independent of each other. Thus the conditional probability basically goes away. This is the \"naive\" in Naive Bayes - acting as if your data dimensions are independent.\nIf your data dimensions are not independent, then your Naive Bayes will not work well. A surprising result, or at least surprising if you focus too much on this independence requirement, is that the Naive Bayes model will still work pretty well if your dimensions are not fully independent. Regardless my point is that some of the most difficult parts in considering Bayes theorem become very simple with Naive Bayes; because you presume that the probabilities are independent of each other - thus no conditional probabilities need to be calculated or presumed. The dimensional independence presumption allows you to multiply probabilities with each other, which greatly simplifies the computations needed to train and test the model.\nLike how a logistic regression is used as a classifier, you would not be wrong to call it, instead, a logistic classifier. Thus a better name for Naive Bayes might be the Naive Probability Model, to distance itself from the conditional probabilities inherent in Bayes theorem.\n## Pitfalls of Naive Bayes\n  * Multiplication of small probabilities leads to floating-point underflow. This problem is corrected by computing with the log probabilities, ln(p)\n  * If there are no samples/data then p(xj | Ck) = 0, leading to the product of the probabilities to be 0. A Laplace smoother is used to ensure that all p(xj | Ck) > 0\n  * Collinear features do not exhibit independence. Ideally, such features should be removed from the dataset to prevent problems with the model\n  * Regularization is generally a minor issue with Naive Bayes models, as uninformative features tend to a uniform distribution, which does not affect the outcome\n\n\n## Types of Naive Bayes Models\nNow that we have looked into the basics of a Naive Bayes model, let us look at some specific formulations. It is important to keep in mind that a specific naive Bayes model is required for each class of problems.\nThe **multinomial naive Bayes classifier** is a widely used form of the model. The multinomial classifier finds the most likely class from multiple possibilities. To prevent numerical underflow we write this classifier taking the logarithms of both sides of the equation as follows:\n![Image by author](../images/005.png)\nImage by author\nThe most likely class 𝑦̂ is then:\n![Image by author](../images/006.png)\nImage by author\nThe multinomial classifier can be simplified for the Bernoulli or binary case as:\n![Image by author](../images/007.png)\nImage by author\n## Document Classification Example\n  * The Naive Bayes topic model computes the probability that document D has topic C based on the occurrence of the words {w1, w2,..., wn}, using the following relationship:\n\n\n![Credit: Wikipedia](../images/008.jpeg)\nCredit: [Wikipedia](https://en.wikipedia.org/wiki/Naive_Bayes_classifier)\nIn this document classification example, you have a variety of documents, a so-called corpus of documents. That means you have a collection of documents and these documents are labeled in some way. First, a document is given an \"expert\" label (spam or not spam). After applying labels what you do is you create a column out of every word, which means every word becomes a boolean column, like with one-hot encoding.\nYou repeat this process for every word in that document. There are ways of reducing the number of columns but my point is that Naive Bayes is very useful for document classification. Now, what are some examples of this? For instance, is this document plagiarized or not? Is this document fake news or not? Is this spam or not? The reason why Naive Bayes is a favored method is that the model does not mind the huge number of dimensions. There can be some performance degradation or fidelity degradation resulting in the classifier not having as good accuracy with huge numbers of data dimensions. But, there are ways of removing superfluous inputs. Computationally speaking, this is very quick hence why it is a favorite choice for document classification.\n## Python Code Example\nNow that the theory has been laid let me walk you through a practical example. What follows is a simple example of a Naive Bayes model. Consider the `HouseVotes84`which contains political party and votes on 16 important bills for 435 members of the US House of Representatives in 1984.\nI will use this data set to build and test a classifier to predict the political party of the congresspeople. Let us begin with the imports:\ntext\n\n```\nimport pandas\nimport numpy\nimport seaborn\nfrom matplotlib import pyplot\n```\n\ntext\n\n```\n%matplotlib inline\n```\n\ntext\n\n```\nvotes = pandas.read_csv('https://library.startlearninglabs.uw.edu/DATASCI410/Datasets/house-votes-84.csv', header=None, \nnames=['class', 'V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9','V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16'])\nprint(votes.shape)\nvotes.head()\n```\n\n![Shape and first 5 rows of dataset | Image by author](../images/009.png)\nShape and first 5 rows of dataset | Image by author\n## Assigning Topics\nEach of the vote columns in the above data frame corresponds to the following topics, so let us replace the column names with something easier to remember:\ntext\n\n```\nvote_names = [\n    'handicapped_infants',\n    'water_project_cost_sharing',\n    'adoption_of_the_budget_resolution',\n    'physician_fee_freeze',\n    'el_salvador_aid',\n    'religious_groups_in_schools',\n    'anti_satellite_test_ban',\n    'aid_to_nicaraguan_contras',\n    'mx_missile',\n    'immigration',\n    'synfuels_corporation_cutback',\n    'education_spending',\n    'superfund_right_to_sue',\n    'crime',\n    'duty_free_exports',\n    'export_administration_act_south_africa']\n```\n\ntext\n\n```\nvotes.columns = ['class'] + vote_names\nvotes.head()\n```\n\n![Relabled columns | Image by author](../images/010.jpeg)\nRelabled columns | Image by author\n## Visual exploration - Bar Plots\nTo further understand this data, let's make some plots of the first 5 votes. The code in the cell below creates bar plots for these votes by `Class` political parties. Execute this code and examine the results:\ntext\n\n```\n# Convert dataframe columns to \"category\" type to take advantage\n# categorical utilities like facet-based plotting in seaborn\n```\n\ntext\n\n```\nvotes['class'] = pandas.Categorical(votes['class'].astype('object')\n    , categories=['republican', 'democrat'])\n```\n\ntext\n\n```\nvalues = ['y', 'n', '?']\nvote_dtype = pandas.api.types.CategoricalDtype(categories=values)\nfor c in votes.columns[1:]:\n    votes[c] = votes[c].astype('object').astype(\n        vote_dtype)\ndemocrats = votes[votes['class'] == 'democrat']\nrepublicans = votes[votes['class'] == 'republican']\nfor vote_col in votes.columns[1:6]:\n    pyplot.subplot(1, 2, 1)\n    pyplot.hist([1 if x == 'y' else 0 for x in democrats[vote_col]])\n    pyplot.title('Democrats: n' + vote_col)\n    pyplot.ylabel('count')\n    pyplot.subplot(1, 2, 2)\n    pyplot.hist([1 if x == 'y' else 0 for x in republicans[vote_col]])\n    pyplot.title('Republicans: n' + vote_col)\n    pyplot.show()\n```\n\n![Image by author](../images/011.jpeg)\nImage by author\n![Image by author](../images/012.jpeg)\nImage by author\n![Image by author](../images/013.jpeg)\nImage by author\nWhen examining these charts, keep in mind that there are more Democrats than Republicans in this data set - thus it is not perfectly balanced. What is important is the probability of a Yes vote or No vote for the members of each party. Some votes are quite skewed by party affiliation, such as 'adoption of the budget resolution'. Whereas, some votes have similar probabilities by party, such as 'water project cost sharing'. These probabilities of votes by political party are used to train the Naive Bayes model.\n## Train and Test the Model\nNow that we understand a bit about the characteristics of the data, it's time to train and test a naive Bayes model. The python `sklearn.naive_bayes` package provides a library that trains a naive Bayes model and produces a model object that can make predictions on new instances:\ntext\n\n```\nimport sklearn.model_selection as m_s\nVotes_Train, Votes_Test = m_s.train_test_split(votes)\nvotes = Votes_Test\n```\n\ntext\n\n```\nlabel_col = 'class'\n```\n\ntext\n\n```\n# Convert our categorical values to numeric feature vectors\nfeature_vecs = numpy.array([\n        votes[c].cat.codes \n        for c in votes.columns \n        if c != label_col]).T\nprint(feature_vecs.shape)\n```\n\ntext\n\n```\nfeature_Train = numpy.array([\n        Votes_Train[c].cat.codes \n        for c in Votes_Train.columns \n        if c != label_col]).T\nprint(feature_Train.shape)\n```\n\n![](../images/body-0e1d46ebf63e6b693b56.jpeg)\ntext\n\n```\nfeature_vecs[:5]\n```\n\n![Image by author](../images/014.png)\nImage by author\ntext\n\n```\n# Convert our label (democrat vs republican) to numeric values\nlabels = votes[label_col].cat.codes\nlabels_Train = Votes_Train[label_col].cat.codes\n# Look at the mapping for the first 5 values\nlist(zip(votes[label_col][:5], labels[:5]))\n```\n\n![Image by author](../images/015.jpeg)\nImage by author\ntext\n\n```\nimport sklearn.naive_bayes\n```\n\ntext\n\n```\n# Define model\nmodel = sklearn.naive_bayes.MultinomialNB(alpha=1e-7)\n# Train model with votes dataset\nmodel.fit(feature_Train, labels_Train)\n```\n\n![Image by author](../images/016.jpeg)\nImage by author\n## Evaluate Model Performance\nWith the model computed, let us now evaluate the performance. We get a quick overview of the model's effectiveness by printing the first 10 rows of the result:\ntext\n\n```\npredicted_party = model.predict(feature_vecs[:10])\nparty_probabilities = model.predict_proba(feature_vecs[:10])\n```\n\ntext\n\n```\nresults = pandas.DataFrame({\n        'party': votes['class'][:10],\n        'predicted': pandas.Categorical.from_codes(\n            predicted_party, votes['class'][:10].cat.categories),\n        'proba(Republican)': party_probabilities[:, 0],\n        'proba(Democrat)': party_probabilities[:, 1],\n    })\nresults\n```\n\n![Image by author](../images/017.jpeg)\nImage by author\nAs you examine these results notice:\n  1. There is 1 clear classification error, 1 classification that is undecided, and 8 cases correctly classified.\n  2. In most cases, the probability of the class predicted (score) is much larger than for the other class, including for the cases with classification errors.\n  3. One case has nearly identical probabilities for the classes.\n\n\nAs a next step, we compute the confusion matrix and performance metrics for this model:\ntext\n\n```\nimport sklearn.metrics\n```\n\ntext\n\n```\ndef confusion_matrix(labels, predicted_labels, label_classes):\n    return pandas.DataFrame(\n        sklearn.metrics.confusion_matrix(labels, predicted_labels),\n        index=[label_classes], \n        columns=label_classes)\n```\n\ntext\n\n```\ndef performance(results):\n    accuracy = sklearn.metrics.accuracy_score(\n        results['party'].cat.codes, results['predicted'].cat.codes)\n    precision = sklearn.metrics.precision_score(\n            results['party'].cat.codes, results['predicted'].cat.codes)\n    recall = sklearn.metrics.recall_score(\n            results['party'].cat.codes, results['predicted'].cat.codes)\n```\n\ntext\n\n```\nprint('Accuracy = %.3f, Precision = %.3f, Recall = %.3f' % (accuracy, precision, recall))\n\n    return confusion_matrix(\n        results['party'], \n        results['predicted'], \n        results.party.cat.categories)\n```\n\ntext\n\n```\nperformance(results)\n```\n\n![Image by author](../images/018.jpeg)\nImage by author\nThe results are reasonably good looking at only the first 10 of 435 congresspeople.\nExecute the code in the cell below to compute and print an evaluation of the model using all the data and compare the results to the first model:\ntext\n\n```\npredicted_party = model.predict(feature_vecs)\nparty_probabilities = model.predict_proba(feature_vecs)\n```\n\ntext\n\n```\nresults_all = pandas.DataFrame({\n        'party': votes['class'],\n        'predicted': pandas.Categorical.from_codes(\n            predicted_party, votes['class'].cat.categories),\n        'proba(democrat)': party_probabilities[:, 0],\n        'proba(republican)': party_probabilities[:, 1],\n    })\nperformance(results_all)\n```\n\n![Image by author](../images/019.png)\nImage by author\n## Laplace Smoothing\nThe Laplace smoothing method is an effective way to deal with data sets that do not have sufficient samples to compute probabilities. This method avoids the case where 𝑝(𝑥𝑗|𝐶𝑘)=0p(xj|Ck)=0.\nThe code in the cell below computes a naive Bayes model using the same congressional vote data, but with a Laplace smoother with a span of 3 data points. Execute this code and examine the results:\ntext\n\n```\npredicted_party = model.predict(feature_vecs)\nparty_probabilities = model.predict_proba(feature_vecs)\n```\n\ntext\n\n```\nresults_all = pandas.DataFrame({\n        'party': votes['class'],\n        'predicted': pandas.Categorical.from_codes(\n            predicted_party, votes['class'].cat.categories),\n        'proba(democrat)': party_probabilities[:, 0],\n        'proba(republican)': party_probabilities[:, 1],\n    })\nperformance(results_all)\n```\n\n![Image by author](../images/020.png)\nImage by author\nThe results are similar to the model computed without Laplace smoothing. This result is expected as all the cases in the data set have sufficient data.\n## Another Example - Census Income\nNow we can try another binary classification example. The code in the cell below loads some sample US Census data. We want to build and evaluate a naive Bayes model to classify people by high and low income using $50,000 as the cut-off. Execute this code and examine the features in the data set.\ntext\n\n```\n# income is a pd read_csv function from a source URL that no longer works\nincome.head()\n```\n\n![Image by author](../images/021.png)\nImage by author\ntext\n\n```\nincome.shape\n```\n\n![Image by author](../images/body-a8a88b33429f90652785.png)\nImage by author\ntext\n\n```\nincome.info()\n```\n\n![Image by author](../images/body-600384f19f3d96ad3555.png)\nImage by author\ntext\n\n```\nincome.describe()\n```\n\n![Image by author](../images/body-9746114f366727148285.png)\nImage by author\n## Reducing Features\nWe can see some features which are likely to be collinear. There is also one feature, `fnlwgt`which is not useful in classifying these people. The code in the cell below removes these columns. Execute this code to create a data set with reduced features:\ntext\n\n```\nincome = income.drop(['workclass', 'fnlwgt', 'education-num', 'relationship'], axis=1)\nincome.head(10)\n```\n\n![Image by author](../images/body-a593939a1cac6365e2e6.png)\nImage by author\nNow I compute a Naive Bayes model to classify `income` using the features in the Income data set:\ntext\n\n```\nfor c in income.columns:\n    if income[c].dtype == 'object':\n        income[c] = income[c].astype('category')\n```\n\ntext\n\n```\nincome_label_col = 'income'\n```\n\ntext\n\n```\nincome_labels = income[income_label_col].cat.codes\n```\n\ntext\n\n```\nfeatures = []\nfor c in income.columns:\n    if c != income_label_col:\n        if str(income[c].dtype) == 'category':\n            features.append(income[c].cat.codes)\n        else:\n            features.append(income[c])\nincome_feature_vecs = numpy.array(features).T\n```\n\ntext\n\n```\n# Create model\nmodel = sklearn.naive_bayes.MultinomialNB(alpha=3)\nmodel.fit(income_feature_vecs, income_labels)\n```\n\n![Image by author](../images/body-979bff60110e215a5aa5.png)\nImage by author\ntext\n\n```\nfeatures[8]\n```\n\n![Image by author](../images/body-927b9d7d093d54d99dbf.png)\nImage by author\ntext\n\n```\nconditional_probas = pandas.DataFrame(model.feature_log_prob_, columns=income.columns[:-1])\nconditional_probas\n```\n\n![Image by author](../images/body-304a7bc816f07d528428.png)\nImage by author\ntext\n\n```\n(-conditional_probas).plot(kind='bar')\npyplot.show()\n```\n\n![Image by author](../images/body-61e63d7cddb5196b3740.png)\nImage by author\n## Changing Number of Features\nLet us now investigate the effect of adding more data samples to the naive Bayes model. The code in the cell below computes and evaluates naive Bayes models using 2, 3, 4, 5, and 6 votes. Execute this code and compare the results to those obtained using the full data set:\ntext\n\n```\nlabel_col = 'class'\n```\n\ntext\n\n```\n# We need to convert our categorical values to numeric feature vectors\nfeature_vecs = numpy.array([\n        votes[c].cat.codes \n        for c in votes.columns \n        if c != label_col]).T\nprint(feature_vecs.shape)\n```\n\ntext\n\n```\n# we also need to convert our label (democrat vs republican) to numeric values\nlabels = votes[label_col].cat.codes\n# take a look at the mapping for the first 5 values like so\nlist(zip(votes[label_col][:5], labels[:5]))\n```\n\n![Image by author](../images/body-d43918d2bf2004f1af8e.png)\nImage by author\ntext\n\n```\nfor num_features in [2, 3, 4, 5, 6]:\n    model = sklearn.naive_bayes.MultinomialNB()\n    model.fit(feature_vecs[:, :num_features], labels)\n```\n\ntext\n\n```\npredicted_votes = model.predict(feature_vecs[:, :num_features])\n```\n\ntext\n\n```\nprint('Number of features = %d' % num_features)\n    accuracy = sklearn.metrics.accuracy_score(labels, predicted_votes)\n    precision = sklearn.metrics.precision_score(labels, predicted_votes)\n    recall = sklearn.metrics.recall_score(labels, predicted_votes)\n    print('Accuracy = %.3f, Precision = %.3f, Recall = %.3f' % (accuracy, precision, recall))\n    print('Confusion matrix:')\n    print(confusion_matrix(labels, predicted_votes, votes['class'].cat.categories))\n    print('')\n```\n\n![Image by author](../images/body-cd61f1e38460d572921a.png)\nImage by author\nYou can see from these results, that the model gains accuracy rapidly with just a few features. In fact, 5 or 6 of the 16 features provide equivalent results.\nCompute a Naive Bayes model, compute and print the performance statistics for 100, 500, 1000, 2000, 8000, and 32561 rows of the `income`data set:\ntext\n\n```\nNumberOfRowsTests = [100, 500, 1000, 2000, 8000, 32561]\nfor num_rows in (NumberOfRowsTests):\n    model = sklearn.naive_bayes.MultinomialNB()\n    model.fit(income_feature_vecs[:num_rows, :], income_labels[:num_rows])\n```\n\ntext\n\n```\npredicted_income = model.predict(income_feature_vecs[:num_rows, :])\n    income_probabilities = model.predict_proba(income_feature_vecs[:num_rows, :])\n\n    accuracy = sklearn.metrics.accuracy_score(income_labels[:num_rows], predicted_income)\n    precision = sklearn.metrics.precision_score(income_labels[:num_rows], predicted_income)\n    recall = sklearn.metrics.recall_score(income_labels[:num_rows], predicted_income)\n    print('%6.0f rows:  Accuracy = %.3f, Precision = %.3f, Recall = %.3f' % (num_rows, accuracy, precision, recall))\n    print('Confusion matrix:')\n    print(confusion_matrix(\n            income_labels[:num_rows], predicted_income, income.income.cat.categories), 'n')\n```\n\n![Image by author](../images/body-80f72db11c6de4b601c8.png)\nImage by author\n## Summary\nIn this notebook I have worked through the following:\n  1. Theory of Naive Bayes models\n  2. Pitfalls of Naive Bayes\n  3. Good regularization properties\n  4. Computationally efficiency\n  5. Examples of computing and evaluating Naive Bayes models\n  6. Examine the effects of data set size on the results of Naive Bayes models\n  7. An explanation of Supervised Learning\n\n\nRemember that while Naive Bayes is a useful and powerful classifier - this model should always be compared against a [logistic regression](https://towardsdatascience.com/logistic-regression-cebee0728cbf) model. This heuristic allows for a useful sanity check and it provides an important ground truth for evaluating your model.\nI hope you have enjoyed this article! Whether you are new to the concept or simply are in need of a refresher I want my readers to have a deep, and complete intuition and understanding of these complex yet highly important topics in data science and machine learning.\nPlease subscribe if you want me to continue writing on similar topics!\nFind me on [Linkedin](https://www.linkedin.com/in/james-a-w-godwin/)\n_Physicist cum Data Scientist- Available for new opportunity | SaaS | Sports | Start-ups | Scale-ups_\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-92fdfcb3140ff02db487.jpeg)\nWritten By\n## [James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\n[See all from James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Naive Bayes](https://towardsdatascience.com/tag/naive-bayes/)[Python](https://towardsdatascience.com/tag/python/)[Supervised Learning](https://towardsdatascience.com/tag/supervised-learning/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F&title=An%20Introduction%20to%20Supervised%20Learning)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F&title=An%20Introduction%20to%20Supervised%20Learning)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F&text=An%20Introduction%20to%20Supervised%20Learning)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/022.jpeg)](https://towardsdatascience.com/common-mistakes-in-hyper-parameters-tuning-ff5951e6c2d/)\n## [Common Mistakes in Hyper-Parameters Tuning](https://towardsdatascience.com/common-mistakes-in-hyper-parameters-tuning-ff5951e6c2d/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHyper-parameter tuning for machine learning models is a trial and error game. To succeed, it is best to avoid the following mistakes.\n[Georgia Deaconu](https://towardsdatascience.com/author/georgiadeaconu/)\nJanuary 28, 20224 min read\n  * [![](../images/023.png)](https://towardsdatascience.com/automl-for-time-series-advanced-approaches-with-fedot-framework-4f9d8ea3382c/)\n## [AutoML for time series: advanced approaches with FEDOT framework](https://towardsdatascience.com/automl-for-time-series-advanced-approaches-with-fedot-framework-4f9d8ea3382c/)\n[](https://towardsdatascience.com/supervised-learning-61256f2aebeb/)\nAn example of using FEDOT and other AutoML libraries on real-world data with gaps and non-stationarity\n[Mikhail Sarafanov](https://towardsdatascience.com/author/mik.sarafanov/)\nJuly 2, 202113 min read\n  * [![](../images/024.jpg)](https://towardsdatascience.com/setting-up-python-projects-part-iv-82059eba4ca4/)\n## [Setting up Python Projects: Part IV](https://towardsdatascience.com/setting-up-python-projects-part-iv-82059eba4ca4/)\n[](https://towardsdatascience.com/supervised-learning-61256f2aebeb/)\nMastering the Art of Python Project Setup: A Step-by-Step Guide\n[Johannes Schmidt](https://towardsdatascience.com/author/johschmidt42/)\nOctober 21, 202212 min read\n  * [![](../images/025.jpeg)](https://towardsdatascience.com/convolutional-neural-networks-for-binary-classification-of-solar-flares-b97de0852037/)\n## [Convolutional Neural Networks for binary classification of Solar Flares](https://towardsdatascience.com/convolutional-neural-networks-for-binary-classification-of-solar-flares-b97de0852037/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nConvolutional neural networks can help us in performing extremely important classifications for our communications, satellites, people...\n[Piero Paialunga](https://towardsdatascience.com/author/piero-paialunga/)\nDecember 21, 20206 min read\n  * [![](../images/026.jpeg)](https://towardsdatascience.com/how-to-pick-the-best-graphics-card-for-machine-learning-32ce9679e23b/)\n## [How to Pick the Best Graphics Card for Machine Learning](https://towardsdatascience.com/how-to-pick-the-best-graphics-card-for-machine-learning-32ce9679e23b/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nSpeed up your training, and iterate faster\n[Michael Clayton](https://towardsdatascience.com/author/maclayton/)\nSeptember 19, 202211 min read\n  * [![](../images/027.jpg)](https://towardsdatascience.com/properties-of-markov-chains-79380dc0358f/)\n## [Properties of Markov Chains](https://towardsdatascience.com/properties-of-markov-chains-79380dc0358f/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nSimply explaining the most common characteristics of Markov Chains\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nFebruary 21, 20223 min read\n  * [![](../images/028.jpeg)](https://towardsdatascience.com/10-most-frequently-asked-pandas-questions-on-stack-overflow-b9d7d94cd83e/)\n## [10 Most Frequently Asked Pandas Questions on Stack Overflow](https://towardsdatascience.com/10-most-frequently-asked-pandas-questions-on-stack-overflow-b9d7d94cd83e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWith the ans\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nNovember 25, 20226 min read\n  * [![](../images/029.jpeg)](https://towardsdatascience.com/non-negative-matrix-factorization-nmf-for-dimensionality-reduction-in-image-data-8450f4cae8fa/)\n## [Non-Negative Matrix Factorization (NMF) for Dimensionality Reduction in Image Data](https://towardsdatascience.com/non-negative-matrix-factorization-nmf-for-dimensionality-reduction-in-image-data-8450f4cae8fa/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDiscussing theory and implementation with Python and Scikit-learn\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nMay 6, 20239 min read\n  * [![](../images/030.jpg)](https://towardsdatascience.com/how-to-write-python-without-code-cbd536046fa9/)\n## [How To Write Python without Code](https://towardsdatascience.com/how-to-write-python-without-code-cbd536046fa9/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA brief intro to a python package that generates the code while you analyze your data\n[Roman Orac](https://towardsdatascience.com/author/romanorac/)\nMay 31, 20223 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "metadata": {
    "title": "An Introduction to Supervised Learning | Towards Data Science",
    "description": "The one with an example of the Naive Bayes classification algorithm",
    "keywords": null,
    "author": "James Andrew Godwin",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "An Introduction to Supervised Learning",
    "og:description": "The one with an example of the Naive Bayes classification algorithm",
    "og:url": "https://towardsdatascience.com/supervised-learning-61256f2aebeb/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1BE9Q5-9elTlkQjgc-pi-6g.jpeg",
    "og:image:width": "640",
    "og:image:height": "480",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "James Andrew Godwin",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "23 min read",
    "article:published_time": "2022-02-26T05:38:26.000Z",
    "article:modified_time": "2022-02-26T05:38:26.000Z",
    "article:author": "https://towardsdatascience.com/author/james-andrew/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_000.txt",
      "chars": 72
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_001.txt",
      "chars": 19
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_002.txt",
      "chars": 272
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_003.txt",
      "chars": 480
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_004.txt",
      "chars": 52
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_005.txt",
      "chars": 126
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_006.txt",
      "chars": 113
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_007.txt",
      "chars": 655
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_008.txt",
      "chars": 111
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_009.txt",
      "chars": 20
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_010.txt",
      "chars": 205
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_011.txt",
      "chars": 159
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_012.txt",
      "chars": 17
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_013.txt",
      "chars": 236
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_014.txt",
      "chars": 27
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_015.txt",
      "chars": 141
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_016.txt",
      "chars": 112
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_017.txt",
      "chars": 310
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_018.txt",
      "chars": 23
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_019.txt",
      "chars": 223
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_020.txt",
      "chars": 377
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_021.txt",
      "chars": 216
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_022.txt",
      "chars": 21
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_023.txt",
      "chars": 102
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_024.txt",
      "chars": 321
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_025.txt",
      "chars": 102
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_026.txt",
      "chars": 321
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_027.txt",
      "chars": 88
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_028.txt",
      "chars": 13
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_029.txt",
      "chars": 14
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_030.txt",
      "chars": 18
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_031.txt",
      "chars": 103
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_032.txt",
      "chars": 110
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_033.txt",
      "chars": 28
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_034.txt",
      "chars": 51
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_035.txt",
      "chars": 264
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_036.txt",
      "chars": 112
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_037.txt",
      "chars": 12
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_038.txt",
      "chars": 111
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_039.txt",
      "chars": 53
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_040.txt",
      "chars": 20
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_041.txt",
      "chars": 216
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_042.txt",
      "chars": 219
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_043.txt",
      "chars": 139
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_044.txt",
      "chars": 64
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_045.txt",
      "chars": 482
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_046.txt",
      "chars": 216
    },
    {
      "language": "txt",
      "file": "attachments/fbf7561e44ff/code/block_047.txt",
      "chars": 687
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:47:34Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:47:34Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:47:34Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T17:47:34Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/james-andrew/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/time-series-analysis-7138ec68754a"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/logistic-regression-cebee0728cbf"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/an-introduction-to-linear-regression-9cbb64b52d23"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/bayesian-statistics-11f225174d5a"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/naive-bayes/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/python/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/supervised-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/common-mistakes-in-hyper-parameters-tuning-ff5951e6c2d/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/georgiadeaconu/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/automl-for-time-series-advanced-approaches-with-fedot-framework-4f9d8ea3382c/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://www.pexels.com/@anniroenkae"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://en.wikipedia.org/wiki/F-score"
    },
    {
      "type": "external",
      "href": "https://www.kaggle.com/msjgriffiths/r-what-algorithms-are-most-successful-on-kaggle/data?scriptVersionId=0"
    },
    {
      "type": "external",
      "href": "https://en.wikipedia.org/wiki/Naive_Bayes_classifier"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/in/james-a-w-godwin/"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=An+Introduction+to+Supervised+Learning&u=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=An+Introduction+to+Supervised+Learning&url=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=An+Introduction+to+Supervised+Learning&url=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh"
    },
    {
      "type": "external",
      "href": "https://insightmediagroup.io/creator-network"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:26:59Z",
  "source_url": "https://towardsdatascience.com/supervised-learning-61256f2aebeb/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# An Introduction to Supervised Learning\nThe one with an example of the Naive Bayes classification algorithm\n[James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\nFebruary 26, 202223 min read\n![caption | Photo by Anni Roenkae](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1BE9Q5-9elTlkQjgc-pi-6g.jpeg)\ncaption | Photo by [Anni Roenkae](https://www.pexels.com/@anniroenkae)\nThis continues a series of articles that build on each other. Please read my last article on [Time Series Analysis](https://towardsdatascience.com/time-series-analysis-7138ec68754a) and [Logistic Regression](https://towardsdatascience.com/logistic-regression-cebee0728cbf).\nAlright, let us start with supervised learning. I covered some of this in my earlier articles but let me go over some specifics for this article.\n  * Supervised learning: learns from existing data which are categorized and labeled with predefined classes. Test data are labeled into these classes as well\n\n\nWell, this is true of all types of learning. But, specifically, it has **labeled predefined classes**. That is the most important thing - **supervised learning has something that is called an expert label**. That's a fancy word for meaning that it is labeled for an outcome; or for any given case, there is a known, desired outcome.\nUnsupervised learning (clustering) does not assume that it knows the answer. So clustering, for instance, will take a look at the data and partition the data, according to how the data are.\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\nSupervised learning will partition the data according to the label. This is a big difference. An example of unsupervised learning is clustering. An example of supervised learning is regression, as I have [written](https://towardsdatascience.com/an-introduction-to-linear-regression-9cbb64b52d23) about before.\nFor instance, in a regression model, we have X and Y, and we draw a best fit line through that. In this example, X are our inputs, and Y is the output. We can say this is supervised learning because Y is the expert label; the model is learning how to take X and come up with Y given these expert labels.\n## Examples:\n  * Fraud transaction: we know which transactions in the training data were fraud (1), and which were not (0)\n  * Readmission: we know which patients were readmitted to a hospital within a certain time window after discharge\n  * Recommendation: we know which items were presented to customers, and which items were clicked, added to the cart, or purchased\n  * A classic example is a spam filter\n\n\nMachine learning tasks have a variety of purposes. Here are a few examples of classic machine learning tasks: identifying fraud, identifying readmission of patients, identifying recommendations like your typical Netflix machine learning case that is when you go on to Netflix, you get recommendations of movies that you will like; those movies are based on previous movies that you looked at and considering what other people liked when they had watched similar movies to the movies you watched, and what movies they watched, in addition to that. This is called a recommendation engine.\n> As an aside, this recommendation engine is sort of a gray area, that might not always be described as supervised learning, but I digress.\nA classic example of supervised learning is a spam filter. What does a spam filter do? It classifies between spam and no spam on your email. The way the model learned this is by someone, an \"expert\" - a human, who came by and marked various emails as spam or not spam. Then the emails and those labels were given to an algorithm that was expected to figure out how to get from the email to that label on its own. This whole process is called supervised learning.\n## Typical machine learning tasks:\n  * **Classification:** can we label the input?\n  * **Regression** : can we make a prediction?\n  * **Recommendations** : can we predict something based on user preference?\n\n\nSupervised learning has two important steps: first, you train a model, and then you test the model. It is very important to use unseen test data for testing your model. In previous articles, I wrote about supervised learning, like with a regression, I did not use test data, and used training data to test the model. This is is bad, not a good thing to do. People do it on occasion, there are different reasons for doing it. There are very few legitimate reasons, but there are some. If you use training data to determine the accuracy of your model, then you are at best fooling yourself; and at worst, you are lying.\nSo to test the model, you need to use so-called \"unseen test data.\" Now, how do we create unseen test data? Well, that is pretty simple. You just take your modeling data and split it into training data and test data: the test data is used for the testing of the model after the model has been trained on the training segment.\nWhat do we mean by learning? You use the data together with the algorithm to come up with a performance measure or to minimize a cost function - that is the other way of saying it. When you do that, then you will come up with a model with new operational data and you will be able to get an answer.\n## Fundamental assumption of learning\nA fundamental assumption of learning is that the distribution of training examples is identical to the distribution of test examples, including future unseen examples. More importantly, the distribution of training examples is similar enough to future examples. Given that they are future examples, they are unseen.\nIf you want to have an accuracy measure, then that distribution should also be similar to the test examples. This assumption is not a requirement. What is a requirement is that your test examples, for you to trust your accuracy, have to come from the same distribution as your future examples.\nWhere your training examples came from is not important if you create a usable model. If your training examples come from a different distribution, more power to you. To be a valid measure, the test examples have to be similar enough to the examples that will be used in practice when it comes to operational data - when it comes to operationalizing. Now strong violations to the above assumption will result in poor classification accuracy.\n### Machine learning framework\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1eSzY2JvbjdMIsVrkRxUZGQ.jpeg)\nImage by author\n**Training:** given a _training_ set of labeled examples {(x_1, y_1, ... (x_n, y_n)}, estimate the prediction function (f) and parameters (Theta) which minimizes the prediction error on the training set\n![Training function | Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1h0CDa5x-LeQCen98-9meaA.jpeg)\nTraining function | Image by author\n**Testing:** apply (f) to a never before seen _test_ example (x) and output the predicted value (y = f(x))\nWith the overall machine learning framework, you give an X and you get back a Y. And there is going to be somewhat of an error because there is some random noise associated with everything. That means that the actual Y that you get is not going to be exactly the above function.\n## **Classification Workflow**\n![Classification workflow diagram | Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1IAsxwJYW7GKp-LVpgksdmA.jpeg)\nClassification workflow diagram | Image by author\nThe most important thing is the data have to be split into a training set and test set. The model is developed in the training set, the model is evaluated on the test set. Lastly, you get performance measures, also called accuracy measures. Here there are three performance measures: of which the performance measure called accuracy is probably the worst performance measure. Precision and recall are pretty good. There is a better one that is a hybrid of precision and recall, the [F1 score](https://en.wikipedia.org/wiki/F-score).\nThe reason I like it more is that it is sort of a hybrid between the recall and the precision. And you will want to have both good recall and precision. If you have very good recall and very lousy precision or vice versa, your model is not worth much. There is an even better accuracy measure for binary classifications - the ROC AUC.\nSo the AUC, area under the curve, of a ROC curve, is for classifications. The reason why it is better than the previous performance measures is that it is valid for all thresholds; whereas accuracy or precision presumes a threshold - typically it presumes a threshold of 0.5. Likewise, the F1 score also presumes a threshold of 0.5. You could choose a different threshold, but most people would not assume that you would have chosen a different threshold.\nThe reason why you choose a different threshold is that you may have a different purpose in mind, where you have false positives might be worse for you than false negatives, or vice versa. By being able to tune your threshold, you can trade those off.\n### Many classifiers to choose from\n  * Linear Regression\n  * Logistic Regression\n  * Support Vector Machine (SVM)\n  * Neural networks (ANN, CNN, RNN, DNN)\n  * Naive Bayes\n  * Bayesian network\n  * Randomized Forests (Random forests)\n  * Boosted Decision Trees (Gradient boosted trees)\n  * K-nearest neighbor (KNN)\n\n\nHere are so-called classifiers, algorithms used in supervised learning. KNN is generally considered your simplest concept and is often taught in classes like these because the algorithm is relatively simple: you can write out the algorithm from scratch.\nBoosted decision trees and randomized forests used to be your generally favored machine learning classification algorithms. Today the \"newest\" types of algorithms are the neural nets. And when I say the newest, my statement is about five to 10 years old, depending on how you look at it. But up to about five years ago, Kaggle competitions [were won](https://www.kaggle.com/msjgriffiths/r-what-algorithms-are-most-successful-on-kaggle/data?scriptVersionId=0) by boosted decision trees and randomized forests. Now Kaggle competitions are being won by deep neural nets. So not neural nets, as such, but a more sophisticated version of neural nets called deep neural nets (DNN).\nLogistic regression is everyone's favorite - whenever you do a classification, you first do a logistic regression, always! Naive Bayes is a favorite too in that same sense. But logistic regression maybe even a little bit more so. So you always do logistic regression; not because you want to show off the results of the logistic regression. It is just that it is so robust and you know if you cannot get the logistic regression to work, you will not be able to get anything else to work either. So logistic regression is the first thing you try to get to work. When that works, then you start becoming more daring and venture into some of these other algorithms listed above.\n## Generalization\n**Underfitting** : model is too \"simple\" to represent all the relevant class characteristics - high training error and high validation error.\n**Overfitting** : model is too \"complex\" and fits irrelevant characteristics (noise) in the data - low training error and high validation error (big gap between training and validation performance)\nGeneralization means how well it works. As a good heuristic **whenever you say generalization, you think of regularization**. Why do this? Because the way you generalize a model is by regularizing the model. If you have a well-regularized model, it is presumably generalized. What does generalization mean? It means it will work under more than just minimal circumstances. Thus no overfitting has occurred. An underfit model is generally too simple and more work needs to be done to extract information from it. But an overfit model is not generalized well enough and requires regularization.\n## Common pitfalls in machine learning\n  * **Overfitting**\n  * **Underfitting**\n  * **Data leakage** : the creation of unexpected additional information in the training data, creating unrealistically good predictions. Leaking test data into training data. The inclusion of data not present in the model's operational environment\n\n\nRarely is underfitting a pitfall, maybe the usage of algorithms which are too primitive and that results in sort of a way of underfitting because people might have stopped with logistic regression and they should have used a gradient boosted tree if you want to consider that a kind of underfitting. Overfitting is one of your big issues, as mentioned earlier.\nNow the next big issue is data leakage or proxy columns or proxies of any sort. These issues happen a lot. This concept is hard to understand, it is hard to explain, hard to teach why such things happen often unless you have done this yourself.\nFor example, where you have been looking at your data for a month, working with it, and only once your test results come back, and you see your test results for your machine learning algorithm like .97 on your accuracy. Only then do you realize, \"Oh, I must have data leakage.\" Simply put - it is vanishingly rare that you get a model that good. Then you have to dissect your model, you have to find out why your model is too good to be true. And the reason your model is too good to be true is usually because of data leakage.\n> A real-life example: a prostate cancer model included a variable stating whether or not patients had prostate cancer surgery. A model cannot see the future, and the data for whether or not the patient had surgery should have been removed from the model for training.\n## Confusion Matrix\n![Encoding for the confusion matrix](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1O8ODG1d__f3IMLpJuFOjjw.jpeg)\nEncoding for the confusion matrix\nHere is an example of a confusion matrix. One important note about these confusion matrices is that they typically have the true negative in the upper left and the true positive in the lower right. The reason why this is the case, nowadays, is because zero is less than one. Since true negatives are usually modeled as zeros, it makes numeric sense to list them first. When you have your actual versus your predicted classes reduced to the easiest format for a computer to understand them, then you have reduced them to zero and one. Since zero is smaller than one zero goes first and one goes second. Since zeros are smaller than one, zeros go first also in the vertical dimension and then ones well. Zeros are typically associated with the absence of something and they are typically associated with a negative.\nHere a negative is not a bad thing - if I am COVID-19 negative, this means I do not have COVID, or at least my test came back negative for COVID. If I do not have COVID then this is what I call a true negative or 0 0. Likewise, the encoding of 1 1 would mean I do have COVID and I tested positive for it.\nThis is why true negatives and true positives are aligned this way: true negatives are in the upper left, true positives are in the lower right. This was not always the case - it used to be the other way around! But this is what you see most of the time these days. And that just has to do with the ordering of zero and one!\nPrecision and recall are important concepts to understand but are beyond the scope of this article. As I mentioned above, I prefer the F1 score, the harmonic mean (as opposed to the arithmetic mean) of precision and recall as my \"accuracy\" metric over choosing either one. But the best performance measure for classification is the ROC curve.\nThe Receiver Operating Characteristic (ROC) curve is great for classifications. I believe it is the most important accuracy measure but sometimes it cannot be used. There are reasons for that, but generally, the ROC curve is what you want to use. The reason why you want to use the ROC curve is that it considers all thresholds. To illustrate why this is important, consider this hypothetical interview question: give people numbers for false positive, for true positive, false negative, and true negative - there are there 10 in each of those. So that would give you about the worst accuracy score possible: 0.5\n> as an aside consider that if your model is .01 accurate, then just switch the labels for your 0 and 1 classes and suddenly your model is .99 accurate\nBut if you look at the probabilities, the probabilities for the true positives and the true negatives: are all-around zero and one - meaning they are good. The probabilities of the false positives and false negatives are all around 0.5, which means they are really bad. **This means the errors are shaky, and the correct results are very good or very secure;** thus you will get a higher ROC AUC score for that! You will never see anything like that with any of these other accuracy measures because **they do not consider probabilities. They only consider the boolean outcomes** : is it a true positive or is it not a true positive?\n## Naive Bayes\nSome properties of Naive Bayes models are\n  * Do not require a prior\n  * Computational complexity is linear in the number of parameters/features\n  * Requires minimal data to produce models that generalize well\n  * Have a simple and inherent regularization\n\n\nSome applications:\n  * Document classification\n  * SPAM detection\n  * Image classification\n\n\nYes, the Bayes in Naive Bayes does refer to Bayes theorem, which I have previously [written ](https://towardsdatascience.com/bayesian-statistics-11f225174d5a)about. There is also a very clear link between Bayes theorem and Naive Bayes model. However, the presumption with the Naive Bayes model is that the individual dimensions are independent, or naive. That means the important part about Bayes theorem, where you can relate the probability of A to the probability of B or the intersection is not needed. What we have here in the naive consideration is that we say that these probabilities are independent of each other. Thus the conditional probability basically goes away. This is the \"naive\" in Naive Bayes - acting as if your data dimensions are independent.\nIf your data dimensions are not independent, then your Naive Bayes will not work well. A surprising result, or at least surprising if you focus too much on this independence requirement, is that the Naive Bayes model will still work pretty well if your dimensions are not fully independent. Regardless my point is that some of the most difficult parts in considering Bayes theorem become very simple with Naive Bayes; because you presume that the probabilities are independent of each other - thus no conditional probabilities need to be calculated or presumed. The dimensional independence presumption allows you to multiply probabilities with each other, which greatly simplifies the computations needed to train and test the model.\nLike how a logistic regression is used as a classifier, you would not be wrong to call it, instead, a logistic classifier. Thus a better name for Naive Bayes might be the Naive Probability Model, to distance itself from the conditional probabilities inherent in Bayes theorem.\n## Pitfalls of Naive Bayes\n  * Multiplication of small probabilities leads to floating-point underflow. This problem is corrected by computing with the log probabilities, ln(p)\n  * If there are no samples/data then p(xj | Ck) = 0, leading to the product of the probabilities to be 0. A Laplace smoother is used to ensure that all p(xj | Ck) > 0\n  * Collinear features do not exhibit independence. Ideally, such features should be removed from the dataset to prevent problems with the model\n  * Regularization is generally a minor issue with Naive Bayes models, as uninformative features tend to a uniform distribution, which does not affect the outcome\n\n\n## Types of Naive Bayes Models\nNow that we have looked into the basics of a Naive Bayes model, let us look at some specific formulations. It is important to keep in mind that a specific naive Bayes model is required for each class of problems.\nThe **multinomial naive Bayes classifier** is a widely used form of the model. The multinomial classifier finds the most likely class from multiple possibilities. To prevent numerical underflow we write this classifier taking the logarithms of both sides of the equation as follows:\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1171fccbPxHOTAbVCl6C5wA.png)\nImage by author\nThe most likely class 𝑦̂ is then:\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1vz2v52QqhxEBbIr9gwzEaw.png)\nImage by author\nThe multinomial classifier can be simplified for the Bernoulli or binary case as:\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/10IUbUYpqpo7DFChelufGyQ.png)\nImage by author\n## Document Classification Example\n  * The Naive Bayes topic model computes the probability that document D has topic C based on the occurrence of the words {w1, w2,..., wn}, using the following relationship:\n\n\n![Credit: Wikipedia](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1wQZOpv71IOiftu0rstEtew.jpeg)\nCredit: [Wikipedia](https://en.wikipedia.org/wiki/Naive_Bayes_classifier)\nIn this document classification example, you have a variety of documents, a so-called corpus of documents. That means you have a collection of documents and these documents are labeled in some way. First, a document is given an \"expert\" label (spam or not spam). After applying labels what you do is you create a column out of every word, which means every word becomes a boolean column, like with one-hot encoding.\nYou repeat this process for every word in that document. There are ways of reducing the number of columns but my point is that Naive Bayes is very useful for document classification. Now, what are some examples of this? For instance, is this document plagiarized or not? Is this document fake news or not? Is this spam or not? The reason why Naive Bayes is a favored method is that the model does not mind the huge number of dimensions. There can be some performance degradation or fidelity degradation resulting in the classifier not having as good accuracy with huge numbers of data dimensions. But, there are ways of removing superfluous inputs. Computationally speaking, this is very quick hence why it is a favorite choice for document classification.\n## Python Code Example\nNow that the theory has been laid let me walk you through a practical example. What follows is a simple example of a Naive Bayes model. Consider the `HouseVotes84`which contains political party and votes on 16 important bills for 435 members of the US House of Representatives in 1984.\nI will use this data set to build and test a classifier to predict the political party of the congresspeople. Let us begin with the imports:\ntext\n\n```\nimport pandas\nimport numpy\nimport seaborn\nfrom matplotlib import pyplot\n```\n\ntext\n\n```\n%matplotlib inline\n```\n\ntext\n\n```\nvotes = pandas.read_csv('https://library.startlearninglabs.uw.edu/DATASCI410/Datasets/house-votes-84.csv', header=None, \nnames=['class', 'V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9','V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16'])\nprint(votes.shape)\nvotes.head()\n```\n\n![Shape and first 5 rows of dataset | Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1TTuBaF1h8vfAJnvteSD9zQ.png)\nShape and first 5 rows of dataset | Image by author\n## Assigning Topics\nEach of the vote columns in the above data frame corresponds to the following topics, so let us replace the column names with something easier to remember:\ntext\n\n```\nvote_names = [\n    'handicapped_infants',\n    'water_project_cost_sharing',\n    'adoption_of_the_budget_resolution',\n    'physician_fee_freeze',\n    'el_salvador_aid',\n    'religious_groups_in_schools',\n    'anti_satellite_test_ban',\n    'aid_to_nicaraguan_contras',\n    'mx_missile',\n    'immigration',\n    'synfuels_corporation_cutback',\n    'education_spending',\n    'superfund_right_to_sue',\n    'crime',\n    'duty_free_exports',\n    'export_administration_act_south_africa']\n```\n\ntext\n\n```\nvotes.columns = ['class'] + vote_names\nvotes.head()\n```\n\n![Relabled columns | Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/14RvalkAuwVwhrVFiYLpp4A.jpeg)\nRelabled columns | Image by author\n## Visual exploration - Bar Plots\nTo further understand this data, let's make some plots of the first 5 votes. The code in the cell below creates bar plots for these votes by `Class` political parties. Execute this code and examine the results:\ntext\n\n```\n# Convert dataframe columns to \"category\" type to take advantage\n# categorical utilities like facet-based plotting in seaborn\n```\n\ntext\n\n```\nvotes['class'] = pandas.Categorical(votes['class'].astype('object')\n    , categories=['republican', 'democrat'])\n```\n\ntext\n\n```\nvalues = ['y', 'n', '?']\nvote_dtype = pandas.api.types.CategoricalDtype(categories=values)\nfor c in votes.columns[1:]:\n    votes[c] = votes[c].astype('object').astype(\n        vote_dtype)\ndemocrats = votes[votes['class'] == 'democrat']\nrepublicans = votes[votes['class'] == 'republican']\nfor vote_col in votes.columns[1:6]:\n    pyplot.subplot(1, 2, 1)\n    pyplot.hist([1 if x == 'y' else 0 for x in democrats[vote_col]])\n    pyplot.title('Democrats: n' + vote_col)\n    pyplot.ylabel('count')\n    pyplot.subplot(1, 2, 2)\n    pyplot.hist([1 if x == 'y' else 0 for x in republicans[vote_col]])\n    pyplot.title('Republicans: n' + vote_col)\n    pyplot.show()\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1-FpuJ4_lzZXF_wIYXz23ZA.jpeg)\nImage by author\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1EKt7PlxgfScQCvJoCdqh0Q.jpeg)\nImage by author\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1JfmughvF75cE88SSulrZSg.jpeg)\nImage by author\nWhen examining these charts, keep in mind that there are more Democrats than Republicans in this data set - thus it is not perfectly balanced. What is important is the probability of a Yes vote or No vote for the members of each party. Some votes are quite skewed by party affiliation, such as 'adoption of the budget resolution'. Whereas, some votes have similar probabilities by party, such as 'water project cost sharing'. These probabilities of votes by political party are used to train the Naive Bayes model.\n## Train and Test the Model\nNow that we understand a bit about the characteristics of the data, it's time to train and test a naive Bayes model. The python `sklearn.naive_bayes` package provides a library that trains a naive Bayes model and produces a model object that can make predictions on new instances:\ntext\n\n```\nimport sklearn.model_selection as m_s\nVotes_Train, Votes_Test = m_s.train_test_split(votes)\nvotes = Votes_Test\n```\n\ntext\n\n```\nlabel_col = 'class'\n```\n\ntext\n\n```\n# Convert our categorical values to numeric feature vectors\nfeature_vecs = numpy.array([\n        votes[c].cat.codes \n        for c in votes.columns \n        if c != label_col]).T\nprint(feature_vecs.shape)\n```\n\ntext\n\n```\nfeature_Train = numpy.array([\n        Votes_Train[c].cat.codes \n        for c in Votes_Train.columns \n        if c != label_col]).T\nprint(feature_Train.shape)\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1wcD3gLKSnbFvQezKaV057Q.jpeg)\ntext\n\n```\nfeature_vecs[:5]\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1D5V7uWSDU8AfkRjQI2vHYQ.png)\nImage by author\ntext\n\n```\n# Convert our label (democrat vs republican) to numeric values\nlabels = votes[label_col].cat.codes\nlabels_Train = Votes_Train[label_col].cat.codes\n# Look at the mapping for the first 5 values\nlist(zip(votes[label_col][:5], labels[:5]))\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1C0qoyKQznqYqcEuAhIKnCg.jpeg)\nImage by author\ntext\n\n```\nimport sklearn.naive_bayes\n```\n\ntext\n\n```\n# Define model\nmodel = sklearn.naive_bayes.MultinomialNB(alpha=1e-7)\n# Train model with votes dataset\nmodel.fit(feature_Train, labels_Train)\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1WpiX3DTzGWhCi9q9aZk4MA.jpeg)\nImage by author\n## Evaluate Model Performance\nWith the model computed, let us now evaluate the performance. We get a quick overview of the model's effectiveness by printing the first 10 rows of the result:\ntext\n\n```\npredicted_party = model.predict(feature_vecs[:10])\nparty_probabilities = model.predict_proba(feature_vecs[:10])\n```\n\ntext\n\n```\nresults = pandas.DataFrame({\n        'party': votes['class'][:10],\n        'predicted': pandas.Categorical.from_codes(\n            predicted_party, votes['class'][:10].cat.categories),\n        'proba(Republican)': party_probabilities[:, 0],\n        'proba(Democrat)': party_probabilities[:, 1],\n    })\nresults\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1z9t5CRo6jDNFC65UYTQVYg.jpeg)\nImage by author\nAs you examine these results notice:\n  1. There is 1 clear classification error, 1 classification that is undecided, and 8 cases correctly classified.\n  2. In most cases, the probability of the class predicted (score) is much larger than for the other class, including for the cases with classification errors.\n  3. One case has nearly identical probabilities for the classes.\n\n\nAs a next step, we compute the confusion matrix and performance metrics for this model:\ntext\n\n```\nimport sklearn.metrics\n```\n\ntext\n\n```\ndef confusion_matrix(labels, predicted_labels, label_classes):\n    return pandas.DataFrame(\n        sklearn.metrics.confusion_matrix(labels, predicted_labels),\n        index=[label_classes], \n        columns=label_classes)\n```\n\ntext\n\n```\ndef performance(results):\n    accuracy = sklearn.metrics.accuracy_score(\n        results['party'].cat.codes, results['predicted'].cat.codes)\n    precision = sklearn.metrics.precision_score(\n            results['party'].cat.codes, results['predicted'].cat.codes)\n    recall = sklearn.metrics.recall_score(\n            results['party'].cat.codes, results['predicted'].cat.codes)\n```\n\ntext\n\n```\nprint('Accuracy = %.3f, Precision = %.3f, Recall = %.3f' % (accuracy, precision, recall))\n\n    return confusion_matrix(\n        results['party'], \n        results['predicted'], \n        results.party.cat.categories)\n```\n\ntext\n\n```\nperformance(results)\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1oxj7rPlFQ4SGIk3bmaxYiw.jpeg)\nImage by author\nThe results are reasonably good looking at only the first 10 of 435 congresspeople.\nExecute the code in the cell below to compute and print an evaluation of the model using all the data and compare the results to the first model:\ntext\n\n```\npredicted_party = model.predict(feature_vecs)\nparty_probabilities = model.predict_proba(feature_vecs)\n```\n\ntext\n\n```\nresults_all = pandas.DataFrame({\n        'party': votes['class'],\n        'predicted': pandas.Categorical.from_codes(\n            predicted_party, votes['class'].cat.categories),\n        'proba(democrat)': party_probabilities[:, 0],\n        'proba(republican)': party_probabilities[:, 1],\n    })\nperformance(results_all)\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1aqCctXwBkNXvkEh3Q8M7dQ.png)\nImage by author\n## Laplace Smoothing\nThe Laplace smoothing method is an effective way to deal with data sets that do not have sufficient samples to compute probabilities. This method avoids the case where 𝑝(𝑥𝑗|𝐶𝑘)=0p(xj|Ck)=0.\nThe code in the cell below computes a naive Bayes model using the same congressional vote data, but with a Laplace smoother with a span of 3 data points. Execute this code and examine the results:\ntext\n\n```\npredicted_party = model.predict(feature_vecs)\nparty_probabilities = model.predict_proba(feature_vecs)\n```\n\ntext\n\n```\nresults_all = pandas.DataFrame({\n        'party': votes['class'],\n        'predicted': pandas.Categorical.from_codes(\n            predicted_party, votes['class'].cat.categories),\n        'proba(democrat)': party_probabilities[:, 0],\n        'proba(republican)': party_probabilities[:, 1],\n    })\nperformance(results_all)\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1H86wGeXZNsgaFuFOshACqw.png)\nImage by author\nThe results are similar to the model computed without Laplace smoothing. This result is expected as all the cases in the data set have sufficient data.\n## Another Example - Census Income\nNow we can try another binary classification example. The code in the cell below loads some sample US Census data. We want to build and evaluate a naive Bayes model to classify people by high and low income using $50,000 as the cut-off. Execute this code and examine the features in the data set.\ntext\n\n```\n# income is a pd read_csv function from a source URL that no longer works\nincome.head()\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1RTGer3Ad6rDtrJzxCLgoAg.png)\nImage by author\ntext\n\n```\nincome.shape\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/18gUw6f-u3HAFOHowaGhmQ.png)\nImage by author\ntext\n\n```\nincome.info()\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1O585feVqdF4HzdV-nuxs_Q.png)\nImage by author\ntext\n\n```\nincome.describe()\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1DHXqvtshLF6Wj85rE23PQQ.png)\nImage by author\n## Reducing Features\nWe can see some features which are likely to be collinear. There is also one feature, `fnlwgt`which is not useful in classifying these people. The code in the cell below removes these columns. Execute this code to create a data set with reduced features:\ntext\n\n```\nincome = income.drop(['workclass', 'fnlwgt', 'education-num', 'relationship'], axis=1)\nincome.head(10)\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1MeRSrgtAdpYtmtpLVs25fQ.png)\nImage by author\nNow I compute a Naive Bayes model to classify `income` using the features in the Income data set:\ntext\n\n```\nfor c in income.columns:\n    if income[c].dtype == 'object':\n        income[c] = income[c].astype('category')\n```\n\ntext\n\n```\nincome_label_col = 'income'\n```\n\ntext\n\n```\nincome_labels = income[income_label_col].cat.codes\n```\n\ntext\n\n```\nfeatures = []\nfor c in income.columns:\n    if c != income_label_col:\n        if str(income[c].dtype) == 'category':\n            features.append(income[c].cat.codes)\n        else:\n            features.append(income[c])\nincome_feature_vecs = numpy.array(features).T\n```\n\ntext\n\n```\n# Create model\nmodel = sklearn.naive_bayes.MultinomialNB(alpha=3)\nmodel.fit(income_feature_vecs, income_labels)\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1iYl0_I6A8YqlfsnZgxwo5g.png)\nImage by author\ntext\n\n```\nfeatures[8]\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1C8w1Po3-QSQXlT8SK0nw_w.png)\nImage by author\ntext\n\n```\nconditional_probas = pandas.DataFrame(model.feature_log_prob_, columns=income.columns[:-1])\nconditional_probas\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1FwlIvbF2PgM1Q7LlsVkGlA.png)\nImage by author\ntext\n\n```\n(-conditional_probas).plot(kind='bar')\npyplot.show()\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1b6SLYi3MJd_4kht57cCi4w.png)\nImage by author\n## Changing Number of Features\nLet us now investigate the effect of adding more data samples to the naive Bayes model. The code in the cell below computes and evaluates naive Bayes models using 2, 3, 4, 5, and 6 votes. Execute this code and compare the results to those obtained using the full data set:\ntext\n\n```\nlabel_col = 'class'\n```\n\ntext\n\n```\n# We need to convert our categorical values to numeric feature vectors\nfeature_vecs = numpy.array([\n        votes[c].cat.codes \n        for c in votes.columns \n        if c != label_col]).T\nprint(feature_vecs.shape)\n```\n\ntext\n\n```\n# we also need to convert our label (democrat vs republican) to numeric values\nlabels = votes[label_col].cat.codes\n# take a look at the mapping for the first 5 values like so\nlist(zip(votes[label_col][:5], labels[:5]))\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1pO1WQUF7PNjqzi6tVrGbYA.png)\nImage by author\ntext\n\n```\nfor num_features in [2, 3, 4, 5, 6]:\n    model = sklearn.naive_bayes.MultinomialNB()\n    model.fit(feature_vecs[:, :num_features], labels)\n```\n\ntext\n\n```\npredicted_votes = model.predict(feature_vecs[:, :num_features])\n```\n\ntext\n\n```\nprint('Number of features = %d' % num_features)\n    accuracy = sklearn.metrics.accuracy_score(labels, predicted_votes)\n    precision = sklearn.metrics.precision_score(labels, predicted_votes)\n    recall = sklearn.metrics.recall_score(labels, predicted_votes)\n    print('Accuracy = %.3f, Precision = %.3f, Recall = %.3f' % (accuracy, precision, recall))\n    print('Confusion matrix:')\n    print(confusion_matrix(labels, predicted_votes, votes['class'].cat.categories))\n    print('')\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/11xwYpAXM5HFs3uo58YEHLw.png)\nImage by author\nYou can see from these results, that the model gains accuracy rapidly with just a few features. In fact, 5 or 6 of the 16 features provide equivalent results.\nCompute a Naive Bayes model, compute and print the performance statistics for 100, 500, 1000, 2000, 8000, and 32561 rows of the `income`data set:\ntext\n\n```\nNumberOfRowsTests = [100, 500, 1000, 2000, 8000, 32561]\nfor num_rows in (NumberOfRowsTests):\n    model = sklearn.naive_bayes.MultinomialNB()\n    model.fit(income_feature_vecs[:num_rows, :], income_labels[:num_rows])\n```\n\ntext\n\n```\npredicted_income = model.predict(income_feature_vecs[:num_rows, :])\n    income_probabilities = model.predict_proba(income_feature_vecs[:num_rows, :])\n\n    accuracy = sklearn.metrics.accuracy_score(income_labels[:num_rows], predicted_income)\n    precision = sklearn.metrics.precision_score(income_labels[:num_rows], predicted_income)\n    recall = sklearn.metrics.recall_score(income_labels[:num_rows], predicted_income)\n    print('%6.0f rows:  Accuracy = %.3f, Precision = %.3f, Recall = %.3f' % (num_rows, accuracy, precision, recall))\n    print('Confusion matrix:')\n    print(confusion_matrix(\n            income_labels[:num_rows], predicted_income, income.income.cat.categories), 'n')\n```\n\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1UCv_msZPuDsABsFMYQx0Lw.png)\nImage by author\n## Summary\nIn this notebook I have worked through the following:\n  1. Theory of Naive Bayes models\n  2. Pitfalls of Naive Bayes\n  3. Good regularization properties\n  4. Computationally efficiency\n  5. Examples of computing and evaluating Naive Bayes models\n  6. Examine the effects of data set size on the results of Naive Bayes models\n  7. An explanation of Supervised Learning\n\n\nRemember that while Naive Bayes is a useful and powerful classifier - this model should always be compared against a [logistic regression](https://towardsdatascience.com/logistic-regression-cebee0728cbf) model. This heuristic allows for a useful sanity check and it provides an important ground truth for evaluating your model.\nI hope you have enjoyed this article! Whether you are new to the concept or simply are in need of a refresher I want my readers to have a deep, and complete intuition and understanding of these complex yet highly important topics in data science and machine learning.\nPlease subscribe if you want me to continue writing on similar topics!\nFind me on [Linkedin](https://www.linkedin.com/in/james-a-w-godwin/)\n_Physicist cum Data Scientist- Available for new opportunity | SaaS | Sports | Start-ups | Scale-ups_\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657234707_1wi608.jpeg)\nWritten By\n## [James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\n[See all from James Andrew Godwin](https://towardsdatascience.com/author/james-andrew/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Naive Bayes](https://towardsdatascience.com/tag/naive-bayes/)[Python](https://towardsdatascience.com/tag/python/)[Supervised Learning](https://towardsdatascience.com/tag/supervised-learning/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F&title=An%20Introduction%20to%20Supervised%20Learning)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F&title=An%20Introduction%20to%20Supervised%20Learning)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fsupervised-learning-61256f2aebeb%2F&text=An%20Introduction%20to%20Supervised%20Learning)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1swAEakUcYMxj-EiJFtj8_Q-scaled.jpeg)](https://towardsdatascience.com/common-mistakes-in-hyper-parameters-tuning-ff5951e6c2d/)\n## [Common Mistakes in Hyper-Parameters Tuning](https://towardsdatascience.com/common-mistakes-in-hyper-parameters-tuning-ff5951e6c2d/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHyper-parameter tuning for machine learning models is a trial and error game. To succeed, it is best to avoid the following mistakes.\n[Georgia Deaconu](https://towardsdatascience.com/author/georgiadeaconu/)\nJanuary 28, 20224 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/1F4YApqG-OqDJNcSthafXaw.png)](https://towardsdatascience.com/automl-for-time-series-advanced-approaches-with-fedot-framework-4f9d8ea3382c/)\n## [AutoML for time series: advanced approaches with FEDOT framework](https://towardsdatascience.com/automl-for-time-series-advanced-approaches-with-fedot-framework-4f9d8ea3382c/)\n[](https://towardsdatascience.com/supervised-learning-61256f2aebeb/)\nAn example of using FEDOT and other AutoML libraries on real-world data with gaps and non-stationarity\n[Mikhail Sarafanov](https://towardsdatascience.com/author/mik.sarafanov/)\nJuly 2, 202113 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/10/0ErL9B6U4LE5-aFd--scaled.jpg)](https://towardsdatascience.com/setting-up-python-projects-part-iv-82059eba4ca4/)\n## [Setting up Python Projects: Part IV](https://towardsdatascience.com/setting-up-python-projects-part-iv-82059eba4ca4/)\n[](https://towardsdatascience.com/supervised-learning-61256f2aebeb/)\nMastering the Art of Python Project Setup: A Step-by-Step Guide\n[Johannes Schmidt](https://towardsdatascience.com/author/johschmidt42/)\nOctober 21, 202212 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1NS0JPcIWkOSoMgz18MiOJw-scaled.jpeg)](https://towardsdatascience.com/convolutional-neural-networks-for-binary-classification-of-solar-flares-b97de0852037/)\n## [Convolutional Neural Networks for binary classification of Solar Flares](https://towardsdatascience.com/convolutional-neural-networks-for-binary-classification-of-solar-flares-b97de0852037/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nConvolutional neural networks can help us in performing extremely important classifications for our communications, satellites, people...\n[Piero Paialunga](https://towardsdatascience.com/author/piero-paialunga/)\nDecember 21, 20206 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/09/1Xoe-zZtyqG7f0bv0wsm3-Q-scaled.jpeg)](https://towardsdatascience.com/how-to-pick-the-best-graphics-card-for-machine-learning-32ce9679e23b/)\n## [How to Pick the Best Graphics Card for Machine Learning](https://towardsdatascience.com/how-to-pick-the-best-graphics-card-for-machine-learning-32ce9679e23b/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nSpeed up your training, and iterate faster\n[Michael Clayton](https://towardsdatascience.com/author/maclayton/)\nSeptember 19, 202211 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/0x_9F3aLms3Fn1Kbb-scaled.jpg)](https://towardsdatascience.com/properties-of-markov-chains-79380dc0358f/)\n## [Properties of Markov Chains](https://towardsdatascience.com/properties-of-markov-chains-79380dc0358f/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nSimply explaining the most common characteristics of Markov Chains\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nFebruary 21, 20223 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/12/12ajoTeyhXC9LSOxB9VI33g-scaled.jpeg)](https://towardsdatascience.com/10-most-frequently-asked-pandas-questions-on-stack-overflow-b9d7d94cd83e/)\n## [10 Most Frequently Asked Pandas Questions on Stack Overflow](https://towardsdatascience.com/10-most-frequently-asked-pandas-questions-on-stack-overflow-b9d7d94cd83e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWith the ans\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nNovember 25, 20226 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/1MbpueR-MP41LKn-O4udCmA.jpeg)](https://towardsdatascience.com/non-negative-matrix-factorization-nmf-for-dimensionality-reduction-in-image-data-8450f4cae8fa/)\n## [Non-Negative Matrix Factorization (NMF) for Dimensionality Reduction in Image Data](https://towardsdatascience.com/non-negative-matrix-factorization-nmf-for-dimensionality-reduction-in-image-data-8450f4cae8fa/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDiscussing theory and implementation with Python and Scikit-learn\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nMay 6, 20239 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/05/03jxV6RycTdf_B2Qr-scaled.jpg)](https://towardsdatascience.com/how-to-write-python-without-code-cbd536046fa9/)\n## [How To Write Python without Code](https://towardsdatascience.com/how-to-write-python-without-code-cbd536046fa9/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA brief intro to a python package that generates the code while you analyze your data\n[Roman Orac](https://towardsdatascience.com/author/romanorac/)\nMay 31, 20223 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "body_base_path": "attachments/fbf7561e44ff/pages/",
  "body_markdown_path": "attachments/fbf7561e44ff/pages/archived-body.md",
  "local_html": "attachments/fbf7561e44ff/pages/archived-body.html",
  "body_source": "prior_capture"
}