保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Artificial Intelligence

Courage to Learn ML: A Deeper Dive into F1, Recall, Precision, and ROC Curves

F1 Score: Your Key Metric for Imbalanced Data - But Do You Really Know Why? Amy Ma December 17, 202312 min read Well use the analogy of sorting laundry to illustrate the core concepts of recall and precision; Photo by Ace Maxwell on Unsplash We'll use the analogy of sorting laundry to illustrate the core concepts of recall and precision; Photo by Ace Maxwell on Unsplash Welcome back to our journey with the 'Courage to Learn ML' series. In this session, we're exploring the nuanced world of metrics. Many resources introduce these metrics or delve into their mathematical aspects, yet the logic behind these 'simple' maths can sometimes remain opaque. For those new to this topic, I recommend checking out Shervin's thorough post along with the comprehensive guide from neptune.ai. In typical data science interview preparations, when addressing how to handle imbalanced data, the go-to metric is often the F1 score, known as the harmonic mean of recall and precision. However, the rationale behind why the F1 score is particularly suitable for such cases is frequently left unexplained. This post is dedicated to unraveling these reasons, helping you understand the choice of specific metrics in various scenarios. As usual, this post will outline all the questions we're tackling. If you've been pondering these same queries, you're in the right place:

With a fundamental understanding of these metrics, our learner approaches the mentor, who is busy doing laundry, with the first question: Learn this step by step with the interactive AI Engineer, AI and Data Scientist, Computer Science and Machine Learning roadmaps.

I'm working on a game recommendation system. It's designed to suggest video games based on users' preferences and lifestyles. But I've noticed that it mostly recommends popular games, like this year's TGA game - Baldur's Gate, and users are missing out on niche and cult classic games they're searching for. How can I tackle this issue? Should I change my algorithm or maybe use LLM, given its power?

Photo by Nick Hamze on Unsplash Photo by Nick Hamze on Unsplash Let's not rush to the conclusion that you need the most advanced algorithm just yet. Instead, let's explore why your model isn't performing as expected. It seems your model scores well in Precision@k but get a low Recall@k. To understand this better, let's break down these metrics:

From this, it seems users often find relevant games in your recommendations, but not all relevant games are making it to your top k list. It's important to note that the items recommended are those your model predicts to be relevant, which can be considered as 'the number of items predicted to be relevant'.

Hold on, are you suggesting I should use both recall and precision to evaluate my model? But aren't recall and precision mainly used for imbalanced data, similar to their harmonic mean, the F1 score?

You've grasped an essential aspect of precision and recall, and you understand why accuracy isn't always reliable. However, your perspective on recall and precision seems a bit limited, and you shouldn't restrict them to just one scenario, handling imbalanced data. Let's dissect this into smaller parts, starting with:

What are precision and recall?

Precision measures the accuracy of the model's positive predictions , calculated as Precision = # of samples correctly predicted as positive / total # of samples predicted as positive = true positive / (true positive + false positive) On the other hand, recall assesses how well the model identifies all positive cases , calculated as recall = # of samples correctly predicted as positive / total # of actual positive samples = true positive / (true positive + false negative)

A quick tip for remembering TP, TN: The first letter (True/False) indicates if your prediction is correct or not, while the second (Positive/Negative) refers to the predicted label. So, true positive means 'correctly predicted as positive,' and false negative is 'incorrectly predicted as negative...it's actually positive!' The total number of predicted positives is the sum of true positives (TP) and false positives (FP). source: https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/800px-Precisionrecall.svg.png source: https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/800px-Precisionrecall.svg.png Let me offer an intuitive example to understand those two terms: Consider I'm sorting laundry, my aim is to select all the dirty clothes from the pile for washing. In this scenario, precision is about how accurately I can identify a piece of clothing as dirty. Meanwhile, recall measures how many of the actual dirty clothes I can correctly identify Then, our next question is:

Why do we care about precision and recall, and why do they often seem to conflict? Is it possible to have both high precision and recall?

The importance of precision and recall lies in their complementary nature. Let's use the laundry sorting analogy again. My objectives are twofold: first, to ensure all dirty clothes are picked up, and second, to avoid unnecessary washing. Linking this back to metrics, precision is like my aim to correctly identify dirty clothes and save effort by not washing clean ones. Recall, meanwhile, assesses how well I manage to gather all the dirty clothes. To clarify, let's look at two extreme scenarios:

No matter what kind of laundry types you are, you can see the choice of metrics does impact our (the model's ) behaviors. Recall and precision measure different aspects and optimizing both simultaneously is challenging. That's why in classification, we talk about the trade-off between them. They work together to ensure our model accurately predicts while capturing all positive cases. Next, let's dive into:

What is the F1 score, and why is it calculated as the harmonic mean of recall and precision?

Most data science interview guides suggest handling imbalanced data by using the F1 score, which is the harmonic mean of recall and precision. But often, they don't explain why the F1 score is effective in these situations. Then, why use the F1 score? In model evaluations, we're concerned with balancing precision and recall - we want correct predictions and comprehensive coverage. Monitoring both metrics separately can be tedious, so a single measure that reflects the balance is preferred. A simple average doesn't reveal much about the balance; a high score could still mask an imbalance. However, the harmonic mean, used in the F1 score, penalizes extreme values more severely. If either recall or precision is low, it significantly lowers the F1 score. Consider two hypothetical cases to understand why we use the harmonic mean instead of a simple average:

Simple average calculation :

Harmonic mean calculation (F1 Score):

While both scenarios have the same average, the simple average hides Scenario A's poor recall. The harmonic mean, on the other hand, provides a more accurate reflection of the balance between precision and recall. A higher F1 score indicates a better balance.

Then... why is the F1 Score often used for imbalanced data, and is its use limited to these scenarios?

Let's explore the challenges of imbalanced data, which is common in binary classification problems. Here, one class often has far fewer samples and represents rare but significant cases (like customer churn or cancer diagnosis). These rare cases usually have higher consequences, and accurately identifying them is crucial. We need a model that not only makes accurate predictions but also effectively identifies these rare cases. This requirement leads us to seek a balance between precision and recall, and the F1 score becomes a handy tool. It provides a single number that reflects this balance, making it a preferred metric in imbalanced datasets. The F1 score's value lies in its ability to accurately portray a model's efficacy in spotting the minority class. However, the F1 score's usefulness isn't confined to just imbalanced datasets. It's also relevant wherever balancing precision and recall is essential, even in balanced datasets. The F1 score remains a vital metric for balancing precision and recall, and it simplifies model comparisons. Beyond the F1 score, other metrics are also useful for assessing model performance in cases of imbalanced data.

I've heard that F1, precision, and recall are asymmetric metrics, meaning they depend on which class is labeled as positive. How does the F1 score's interpretation change when the positive class is actually the majority?

Good question. To answer that, let's think about how recall and precision would shift if the majority class is considered positive. Achieving a high recall becomes easier because most samples will be predicted as positive. But here's the catch: high precision might be misleading in this scenario. With a larger majority class, it's easy to get high precision just by predicting the majority class all the time. By switching the majority class to positive, we lose sight of how the model handles the rare class, especially in imbalanced data situations. So, the balance between precision and recall doesn't guarantee the model's effectiveness anymore since its focus has shifted. This means a model might show a high F1 score even if it's not great at identifying the minority class. When the positive class forms the majority, a high F1 score might not truly reflect the model's ability to identify the minority class. It could simply mean the model often predicts the majority class. In such cases, it's wise to include other metrics less biased towards the majority, like the recall of the negative (minority) class, to get a fuller picture of the model's performance.

What are the limitations of the F1 score, and what other metrics can we use to evaluate model performance on imbalanced data?

You know, we often approach classification problems in a regression-like manner. What I mean is, some algorithms predict probabilities, not just classes. For these, you need to set a threshold. But the F1 score doesn't really show how the model performs at different thresholds. That's where ROC curves or precision-recall curves come in, helping us assess performance across various thresholds. It's quite common that we do classification in regression way. What I mean is some algorithm will predict probabilities instead of the class. for those models, we need to choose a threshold. However, F1 score cannot give us an illustration on model performance related to different threshold. So we can use ROC curve or precision and recall curve to consider the model performance with various threshold. Additionally, the Area Under the Curve (AUC) metric serves as a single-number summary, facilitating comparison between multiple models. The AUC scale ranges from 1, indicating a perfect classifier, to 0, the mark of the poorest classifier. Notably, an AUC of 0.5 signifies performance equivalent to random guessing, where True Positive Rate (TPR) and the False Positive Rate (FPR) are equal at all thresholds.

An interesting question to explore is How an AUC (Area Under the Curve) value of 0.5 equates to a classifier making random guesses? When the AUC equals 0.5, the ROC curve is represented by a diagonal line connecting the points (0,0) and (1,1) on the plot. This diagonal line signifies that with this classifier, we have TPR = FPR at all thresholds. In simpler terms, the likelihood of correctly identifying a positive case is exactly the same as the likelihood of incorrectly labeling it as negative. Therefore, for any given positive instance, the classifier has a 50% chance of making a correct prediction, akin to the randomness of a coin flip. This equivalence to chance highlights why an AUC of 0.5 is considered indicative of a model performing no better than random guessing. When it comes to imbalanced data, we can use precision-recall curve to observe the model's performance on balancing precision and recall under different threshold. To sum it up, for some scenarios, we'd love the model to balance recall and precision well. That's why we use the F1 score, the harmonic mean of recall and precision, as a single-number metric. But with imbalanced data, where the focus is more on the minority class's performance, the F1 score becomes particularly valuable since achieving balance is tougher. Other handy tools are ROC and PR (Precision-Recall) curves. So, for your game recommendation system, consider using multiple metrics. This way, you can better evaluate how well the model retrieves relevant items (recall) and ensures those items are indeed relevant (precision). You could evaluate precision@k and recall@k together, calculate f1@k, or draw PR curves. In practice, it's crucial to select model metrics based on the actual cost of errors , like whether recall matters more than precision to you. Using multiple metrics gives a fuller picture of your model's performance. And remember, the key is to align your metrics with the model's business or application goals. Before wrapping up this post, there's one more topic I'd like to touch on:

What's the difference between the PR curve and the ROC curve, and when should you choose one over the other?

Most DS interview guides state using the PR curve instead of the ROC curve for imbalanced data, but often don't explain when to opt for the ROC curve. While I won't delve into how to draw these curves here (for that, check out the excellent explanation by StatQuest with Josh Starmer here), let's understand that these curves are drawn by varying the threshold and calculating two metrics (precision and recall for PR, or TPR and FPR for ROC). Both curves represent different balances in binary classification: A sample calculation of ROC curve vs. PR curve. source: https://modtools.files.wordpress.com/2020/01/roc_pr-1.png?w=946 A sample calculation of ROC curve vs. PR curve. source: https://modtools.files.wordpress.com/2020/01/roc_pr-1.png?w=946 The ROC curve focuses on TPR and FPR; the PR curve on precision and recall:

While precision and recall focus solely on the model's performance for the positive class, TPR and FPR provide a broader view of predictability (correct positives vs. misclassified samples). ROC curves are less sensitive to data distribution, as FPR uses the size of the negative class. If the negative class is the majority, the FPR value can still remain low, even with lots of negative predictions, due to the larger size of this class. This means ROC is less affected by data imbalance. On the other hand, PR curves, with precision calculated using predicted positives, are more sensitive to the positive class. What does this imply? It means when comparing model performance across different datasets, ROC curves offer more stability than PR curves and can better reflect a model's performance. So, rather than just remembering PR curves as preferable for imbalanced data, it's important to recognize that ROC curves provide a consistent measure less influenced by data distribution.

In our upcoming session, the mentor-learner duo will delve into the common loss functions, exploring cross-entropy through the lenses of information theory and MLE. If you're enjoying this series, remember that your interactions - claps, comments, and follows - do more than just support; they're the driving force that keeps this series going and inspires my continued sharing.

Other posts in this series:

If you liked the article, you can find me on LinkedIn, and please don't hesitate to connect or reach out with your questions and suggestions!

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Amy Ma

See all from Amy Ma Artificial IntelligenceCourage To Learn MlData ScienceMachine LearningTechnology Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Generating Mock Sales Data with Python Pandas

Artificial Intelligence Everything in data science starts from data Soner Yıldırım July 15, 20215 min read

ML Engineering with DynamoDB

Machine Learning How to leverage this powerhouse NoSQL database for online inference Avi Chad-Friedman February 7, 20235 min read

Rethinking how we teach the tidyverse

Data Science An attempt at an "unbiased" perspective from a tidyverse fanboy Lathan Liou January 16, 20216 min read

Blazing Hot Python AsyncIO Pipelines

Data Science Handle very many pipelined web, database, and micro-service requests concurrently and efficiently Dewald Abrie November 11, 20204 min read

Let's Play Blackjack (with Python)

Artificial Intelligence We Implement a Blackjack Simulator in Python to Better Understand the Risks of Going to Vegas Tony Yiu September 22, 201915 min read

Demystifying Dependence and Why it is Important in Causal Inference and Causal Validation

Machine Learning A step-by-step guide in understanding the concept of dependence and how to apply it to validate directed acyclic graphs in causal inference Graham Harrison November 11, 202316 min read

Create synthetic time-series with anomaly signatures in Python

Machine Learning A simple and intuitive way to create synthetic (artificial) time-series data with customized anomalies - suited to industrial applications. Tirthajyoti Sarkar August 9, 20217 min read

Aliasing in Audio, Easily Explained: From Wagon Wheels to Waveforms

Machine Learning Understanding the foundational distortion of digital audio from first principles, with worked examples and visual intuition Aman Agrawal February 25, 202618 min read

Are Slots and Entities the same?

Machine Learning Building Chatbots with Rasa - part II Aniruddha Karajgi October 4, 20217 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network