![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Data Science](https://towardsdatascience.com/category/data-science/) # Courage to Learn ML: An In-Depth Guide to the Most Common Loss Functions MSE, Log Loss, Cross Entropy, RMSE, and the Foundational Principles of Popular Loss Functions [Amy Ma](https://towardsdatascience.com/author/amyma101/) December 28, 202314 min read ![Photo by William Warby on Unsplash](../images/000.jpg) Photo by [William Warby](https://unsplash.com/@wwarby?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral) Welcome back! In the '[Courage to Learn ML](http://towardsdatascience.com/tagged/courage-to-learn-ml)' series, where we conquer machine learning fears one challenge at a time. Today, we're diving headfirst into the world of loss functions: the silent superheroes guiding our models to learn from mistakes. In this post, we'd cover the following topics: * What is a loss function? * Difference between loss functions and metrics * Explaining MSE and MAE from two perspectives * Three basic ideas when designing loss functions * Using those three basic ideas to interpret MSE, log loss, and cross-entropy loss * Connection between log loss and cross-entropy loss * How to handle multiple loss functions (objectives) in practice * Difference between MSE and RMSE * * * Learn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps. ## What are loss functions, and why are they important in machine learning models? Loss functions are crucial in evaluating a model's effectiveness during its learning process, akin to an exam or a set of criteria. They serve as indicators of how far the model's predictions deviate from the true labels ( the 'correct' answers). Typically, loss functions assess performance by measuring the discrepancy between the predictions made by the model and the actual labels. This evaluation of the gap informs the model about the extent of adjustments needed in its parameters, such as weights or coefficients, to more accurately capture the underlying patterns in the data. There are different loss functions in machine learning. These factors include the nature of the predictive task at hand, whether it's regression or classification, the distribution of the target variable, as illustrated by the use of Focal Loss for handling imbalanced datasets, and the specific learning methodology of the algorithm, such as the application of hinge loss in SVMs. Understanding and selecting the appropriate loss function is quite important, since it directly influences how a model learns from the data. To learn machine learning, one should know the most popular ones. For example, (Mean Squared Error) MSE and (Mean Absolute Error) MAE are commonly used in regression problems, while cross entropy is the most common loss function for classification tasks. ## How do loss functions differ from metrics, and in what ways can a loss function also serve as a metric? Your statement about loss function can also be metrics is misleading. Loss functions and metrics both assess model performance, but in different stages and for different purpose: * **Loss Functions** : These are **used during the model's learning process** to guide its adjustments. They need to be **differentiable** to facilitate optimization. For instance, Mean Squared Error (MSE) and Mean Absolute Error (MAE) are common loss functions in regression models. * **Metrics** : These **evaluate the model's performance after training**. Metrics should be **interpretable** and provide clear insights into model effectiveness. While some metrics, like accuracy, can be straightforward, others like F1 score involve threshold decisions and are non-differentiable, making them less suitable for guiding learning. Notably, some measures, such as MSE and MAE, can serve both as loss functions and metrics due to their differentiability and interpretability. However, **not all metrics are suitable as loss functions** , primarily due to the need for differentiability in loss functions for optimization purposes. In practice, one should always carefully choose the loss function and metrics **together** for learning, and ensure that the learning and evaluation are **aligned in the same direction**. This alignment ensures that the model is optimized and evaluated **based on the same criteria** that reflect the end goals of the application. > **Author's Note:** It's important to clarify that using the F1 score as a loss function in machine learning models isn't entirely infeasible. In my ongoing study, I've encountered innovative methods that address the non-differentiability issue commonly associated with the F1 score. For instance, Ashref Maiza's [post](https://medium.com/r?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d) introduces a differentiable approximation of the F1 score. This approach involves "softening" precision and recall using likelihood concepts, rather than setting arbitrary thresholds. Additionally, some online discussions like [the one](https://datascience.stackexchange.com/questions/66581/is-it-possible-to-make-f1-score-differentiable-and-use-it-directly-as-a-loss-fun) explore similar themes. > The challenge lies in the inherent nature of the F1 score. While it's a highly informative metric, selecting an appropriate loss function to effectively optimize the model under the same criteria can be complex. Moreover, tuning thresholds adds another layer of complexity. I'm really interested into this topic. If you have insights or experiences to share, please feel free to connect with me. I'm eager to expand my understanding and engage in further discussions. ## You said MSE and MAE as typical metrics in regression problems. What are they and when to use them? In regression problems, where the predictions are continuous values, the goal is to minimize the difference between the model's predictions and the actual values. To assess the model's effectiveness in grasping the underlying pattern, we use metrics like Mean Squared Error (MSE) and Mean Absolute Error (MAE). Both these metrics quantify the gap between predictions and actual values, but they do so using different evaluation approaches. MSE is defined as: ![Here, y_i is the actual value, y_hat_i is the predicted value, and n is the number of observations.](../images/001.png) Here, y_i is the actual value, y_hat_i is the predicted value, and n is the number of observations. MSE calculates the average of the squared differences between predictions and actual values, which is the Euclidean distance (l2 norm) of predictions and the true labels. On the other hand, MAE is defined as: ![](../images/body-7f485f12a381a62105ab.png) Here, the absolute differences between the actual and predicted values are averaged, corresponding to the Manhattan distance (l1 norm). In the other words, MAE calculates the average distance between the estimated values and the actual value **without considering the direction** (positive or negative). > We talked about Lp norm and different distances in our discussion on l1, l2 regularizations , The primary distinction between MSE and MAE is their response to **outliers**. MSE, by squaring the errors, amplifies and gives more weight to larger errors, making it sensitive to outliers. **This is useful if larger errors are more significant in your problem context.** However, MAE assigns equal weight to all errors, making it more robust to outliers and non-normal error distributions. The choice between MSE and MAE should based on the properties of the training data and the implications of larger errors in the model. MSE is preferable when we want to heavily penalize larger errors, while MAE is better when we want to treat all errors equally. ## I get that squaring the differences in MSE amplifies the errors, leading to a greater emphasis on outliers. Are there other perspectives or aspects that help differentiate between these two metrics? Certainly, there's another perspective to understand the differences between Mean Squared Error (MSE) and Mean Absolute Error (MAE) beyond their handling of outliers. Imagine you're tasked with predicting a value 'y' without any additional features (no 'Xs'). In this scenario, the simplest model would predict a constant value for all inputs. When using MSE as the loss function, t**he constant that minimizes the MSE is the mean of the target values**. This is because the mean is the central point that minimizes the sum of squared differences from all other points. On the other hand, if you use MAE, the median of the target values is the minimizing constant. The median, unlike the mean, is less influenced by extreme values or outliers. ![In the universe of Douglas Adams' 'The Hitchhiker's Guide to the Galaxy,' 42 is the ultimate answer to life, the universe, and everything. Who knows, maybe 42 is also the magic number to shrink your loss function - but hey, it all depends on what your loss function is! Image created by ChatGPT](../images/002.jpg) In the universe of Douglas Adams' 'The Hitchhiker's Guide to the Galaxy,' 42 is the ultimate answer to life, the universe, and everything. Who knows, maybe 42 is also the magic number to shrink your loss function - but hey, it all depends on what your loss function is! Image created by ChatGPT This difference in sensitivity to outliers stems from how the mean and median are calculated. The mean takes into account the magnitude of each value, making it easier to being skewed by outliers. The median, however, is only concerned with the order of the values, thus maintaining its position regardless of the extremities in the dataset. This intrinsic property of the median contributes to MAE's robustness to outliers, providing an alternative interpretation of the distinct behaviors of MSE and MAE in modeling contexts. > You can find an explanation of why the mean minimizes MSE and the median minimizes MAE in Shubham Dhingra's [post](https://towardsdatascience.com/evaluation-metrics-ii-e6f09ded4981). We've talked about how MSE and MAE measure errors, but there's more to the story. Different tasks need different ways to measure how good our models are doing. This is where loss functions come in, and there are three basic ideas behind them. Understanding these ideas will help you pick the right loss function for any job. So, let's get started with the most important question: ## What are the 3 basic ideas that guide the design of any loss function? In designing loss functions, three basic ideas generally guide the process: 1. **Minimizing Residuals** : The key is to reduce the residuals, which are the differences between predicted and actual values. To address both negative and positive discrepancies, we often square these residuals, as seen in the **least squares method**. This approach, which sums the squared residuals, is a staple in regression problems for its simplicity and effectiveness. 2. **Maximizing Likelihood (MLE)** : Here, the goal is to adjust the model parameters to maximize the likelihood of the observed data, making the model as representative of the underlying process as possible. This **probabilistic approach** is fundamental in models like logistic regression and neural networks, where fitting the model to the data distribution is crucial. 3. **Distinguishing Signal from Noise** : This principle, rooted in **information theory** , involves separating valuable data (signal) from irrelevant data (noise). Methods based on this idea, focusing on entropy and impurity, are essential in classification tasks and form the basis for algorithms like decision trees. Additionally, it's important to recognize that **some loss functions are tailored to specific algorithms** , such as the hinge loss for SVM, indicating that the nature of the algorithm also plays a role in loss function design. Additionally, **the nature of the data impacts the selection of a loss function**. For instance, in cases of imbalanced training data, we might adjust our loss function to a class-balanced loss or opt for focal loss. Now, equipped with these fundamental concepts, let's apply them for interpretive analysis to enhance our comprehension. With this approach, we can attempt to address the following question: ## How might we apply MLE and the least squares method to enhance our comprehension of MSE? First, let's break down MSE with the least squares method. The LSE approach finds the best model fit by **minimizing the sum of the squares of the residuals**. In linear regression (which deals with continuous outputs), a residual is the difference between the predicted value and the actual label. MSE, or Mean Squared Error, is essentially the average of these squared differences. Therefore, the least squares method aims to minimize MSE (factoring in this averaging step), making MSE an appropriate loss function for this method. Next, looking at MSE from a Maximum Likelihood Estimation (MLE) perspective, **under the assumption of linear regression, we typically assume that residuals follow a normal distribution**. This allows us to model the likelihood of observing our data as a product of individual probability density functions (PDFs). For simplification, we take the natural logarithm of this likelihood, transforming it into a sum of the logs of individual PDFs. It's important to note that we use density functions for continuous variables, as opposed to probability mass functions for discrete variables. ![Note: Likelihood calculations differ for discrete and continuous variables. Discrete variables use a probability mass function, while continuous variables employ a probability density function. For more on MLE, refer to my previous post.](../images/003.png) Note: Likelihood calculations differ for discrete and continuous variables. Discrete variables use a probability mass function, while continuous variables employ a probability density function. For more on MLE, refer to my [previous post](https://medium.com/p/65218b2c2b99). When we examine the log likelihood, it comprises two parts: a constant component and a variable component that calculates the squared differences between the true labels and predictions. **To maximize this log likelihood, we focus on minimizing the variable component, which is essentially the sum of squared residuals.** In the context of linear regression, this minimization equates to minimizing MSE, especially when we consider the scaling factor 1/2_σ_² that arises from the normal distribution assumption. In summary, MSE can be derived and understood from both the perspectives of the Least Squares Estimation (LSE) and MLE, with each approach providing a unique lens into the significance and application of MSE in regression analysis. ## So MSE is a common loss function for regression problem. But can I use it for classification problem? Such as logistic regression? MSE, while common in regression, isn't ideal for classification tasks, such as logistic regression. **The primary reason is the mismatch in the nature of outputs** : logistic regression predicts probabilities, whereas MSE assumes continuous numerical values. This misalignment leads to theoretical and practical challenges. Practically, **MSE creates a non-convex loss surface when combined with logistic regression, which often uses a sigmoid activation function**. This non-convexity means the error surface has multiple local minima, making it difficult for optimization algorithms like gradient descent to find the global minimum. Essentially, the algorithm might get 'stuck' in a local minimum, leading to suboptimal model performance. Moreover, **combining MSE with the sigmoid function can cause the gradients to become very small,** particularly for extreme input values. This leads to the 'gradient vanishing' problem, where the model stops learning or learns very slowly because the updates to the model parameters become insignificantly small. Therefore, for classification problems, especially binary ones like logistic regression, MSE is not a idea loss function. ## So what is a good loss function for logistic regression or more general classification problem? Alright, diving into the world of loss functions for logistic regression, let's see how we can apply some basic design ideas to understand them better. First off, let's look at the **least squares method**. The core idea here is to minimize the gap between our model's output and the true labels. A straightforward approach is setting a threshold to convert logistic regression's probability outputs into binary labels, and then comparing these with the true labels. If we choose, say, a 0.5 threshold for classifying donuts and bagels, we label predictions above 0.5 as donuts and below as bagels, then tally up the mismatches. This approach, known as the **0–1 loss** , is directly corresponds to accuracy but isn't used as a loss function for training due to its non-differentiability and non-convex nature, making it impractical for optimization methods like gradient descent. It's more of a conceptual approach than a practical loss function. ![When I first visited America, I couldn't tell the difference between a donut and a bagel. A classifier to distinguish between donuts and bagels could be useful. Image created by ChatGPT](../images/004.png) When I first visited America, I couldn't tell the difference between a donut and a bagel. A classifier to distinguish between donuts and bagels could be useful. Image created by ChatGPT Moving on, let's use the **MLE (Maximum Likelihood Estimation)** idea. In logistic regression, MLE tries to find the weights and bias that maximize the probability of seeing the actual observed data. Imagine our goal is to find a set of weights and bias that maximize the log likelihood, where the likelihood _L_ is the product of individual probabilities of observing each outcome. We're **assuming our data points are independent and each follows a Bernoulli distribution**. ![](../images/body-b36896f4bbf373ef3394.png) So we'd have the log loss as: ![](../images/body-e3f61646850553482949.png) Finally, let's bring in some **information theory** , treating logistic regression as a signal capture machine. In this approach, we employ concepts like entropy and cross-entropy to assess the information our model captures. Entropy measures the amount of uncertainty or surprise in an event. Cross-entropy gauge how well our model's predicted probability distribution lines up with the actual, true distribution. The goal here is to minimize cross-entropy, which is kind of like minimizing the KL divergence. Though not exactly a 'distance' in the strict sense, KL divergence represents how far off our model's predictions are from the actual labels. ![Softmax is another topic on my writing list. Source: https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e](../images/005.png) Softmax is another topic on my writing list. Source: [_https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e_](https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e) So, through the application of three distinct design principles for loss functions, we've crafted various types of loss functions suitable for logistic regression and broader classification challenges. It's particularly fascinating to observe that, despite originating from different perspectives**, log loss and cross-entropy loss are essentially the same in the context of binary classification**. This convergence occurs in situations where only two possible outcomes exist; under these conditions, cross-entropy effortlessly simplifies and transforms into log loss. Comprehending this shift is vital for understanding the interplay and practical application of these theoretical concepts: ![Derive log loss from cross-entropy loss. Source: https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e](../images/006.png) Derive log loss from cross-entropy loss. Source: [_https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e_](https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e) > **Author's Note:** In the future, I'm considering delving into the fascinating world of information theory - a topic that, surprisingly, is both intuitive and practical in real-world applications. Until then, I highly recommend [Kiprono Elijah Koech's post](https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e) as an excellent resource on the subject. Stay tuned for more! ## In practical scenarios, how should one approach the situation where multiple loss functions need to be minimized? When managing multiple loss functions in a model, balancing them can be challenging, as they may conflict. One common approach is to create a weighted sum of these loss functions, assigning specific weights to each. However, this introduces new hyperparameters (the weights), necessitating careful tuning. Adjusting these weights means retraining the model, which can be time-consuming and may affect interpretability and performance. Alternatively, a constraint-based approach can be effective. For instance, in SVM, we aim to maximize the margin (reducing variance) while minimizing classification error (reducing bias). This can be achieved by treating the margin maximization as a constraint, using techniques like Lagrange multipliers, and focusing on minimizing the classification error. This method requires a strong mathematical foundation and thoughtful formulation of constraints. A third option is to decouple the objectives, building separate models for each and then combining their results. This approach simplifies model development and maintenance, as each model can be independently monitored and retrained. It also offers flexibility in responding to changes in objectives or business goals. However, it's important to consider how the combined results of these models align with the overall objective. However, it's important to understand that adversarial loss in GANs isn't just a combination of the discriminator's and generator's losses. This is because the two networks are engaged in a responsive interaction, learning and adapting in response to each other, rather than optimizing their losses independently. Before we conclude, I'd like to address a straightforward yet practical query: ## Why do we sometimes prefer using RMSE (Root Mean Squared Error) instead of MSE ? MSE (Root Mean Squared Error) is often preferred over MSE (Mean Squared Error) in certain situations due to its interpretability. By taking the square root of MSE, RMSE converts the error units back to the original units of the data. This makes RMSE more intuitive and directly comparable to the scale of the data being analyzed. For instance, if you're predicting housing prices, RMSE provides an error metric in the same unit as the prices themselves, making it easier to understand the magnitude of the errors. Additionally, **RMSE is more sensitive to larger errors than RMAE (Root Mean Absolute Error) due to the square root transformation** , emphasizing significant deviations more than RMAE. This can be particularly useful in scenarios where larger errors are more undesirable. _(Unless otherwise noted, all images are by the author)_ * * * If you're enjoying t[his series](http://towardsdatascience.com/tagged/courage-to-learn-ml), remember that your interactions - claps, comments, and follows - do more than just support; they're the driving force that keeps this series going and inspires my continued sharing. Other posts in this series: * [Courage to Learn ML: Demystifying L1 & L2 Regularization (part 1)](https://towardsdatascience.com/understanding-l1-l2-regularization-part-1-9c7affe6f920) * [Courage to Learn ML: Decoding Likelihood, MLE, and MAP](https://towardsdatascience.com/courage-to-learn-ml-decoding-likelihood-mle-and-map-65218b2c2b99) * [Courage to Learn ML: A Deeper Dive into F1, Recall, Precision, and ROC Curves](https://medium.com/p/d5c0a46e5eb7) * * * _**If you liked the article, you can find me on**_[ _**LinkedIn**_](https://www.linkedin.com/in/amyma101/) _**.**_ ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-bf30693e05bb856f0cb2.png) Written By ## [Amy Ma](https://towardsdatascience.com/author/amyma101/) [See all from Amy Ma](https://towardsdatascience.com/author/amyma101/) [AI](https://towardsdatascience.com/tag/ai/)[Courage To Learn Ml](https://towardsdatascience.com/tag/courage-to-learn-ml/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fcourage-to-learn-ml-an-in-depth-guide-to-the-most-common-loss-functions-84a6b07cca17%2F&title=Courage%20to%20Learn%20ML%3A%20An%20In-Depth%20Guide%20to%20the%20Most%20Common%20Loss%20Functions) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fcourage-to-learn-ml-an-in-depth-guide-to-the-most-common-loss-functions-84a6b07cca17%2F&title=Courage%20to%20Learn%20ML%3A%20An%20In-Depth%20Guide%20to%20the%20Most%20Common%20Loss%20Functions) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fcourage-to-learn-ml-an-in-depth-guide-to-the-most-common-loss-functions-84a6b07cca17%2F&text=Courage%20to%20Learn%20ML%3A%20An%20In-Depth%20Guide%20to%20the%20Most%20Common%20Loss%20Functions) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/007.jpg)](https://towardsdatascience.com/deploying-a-multistage-multimodal-recommender-system-on-amazon-eks-featuring-bloom-filters-feature-caching-and-contextual-recommendations/) ## [Deploying a Multistage Multimodal Recommender System on Amazon Elastic Kubernetes Service](https://towardsdatascience.com/deploying-a-multistage-multimodal-recommender-system-on-amazon-eks-featuring-bloom-filters-feature-caching-and-contextual-recommendations/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Featuring Bloom filters, feature caching, contextual ranking, and an end‑to‑end pipeline from data preparation to model serving. [Mustapha Unubi Momoh](https://towardsdatascience.com/author/mustapha.momoh/) May 19, 202619 min read * [![](../images/008.jpg)](https://towardsdatascience.com/overfitting-is-not-the-only-problem-regularisation-can-help-with-6fcdbfdb9384/) ## [Overfitting is not the only problem Regularisation can help with](https://towardsdatascience.com/overfitting-is-not-the-only-problem-regularisation-can-help-with-6fcdbfdb9384/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Understanding mathematically how Ridge Regression helps in cases where the number of features exceeds data points [Raj Sangani](https://towardsdatascience.com/author/rajbsangani/) September 8, 20215 min read * [![](../images/009.jpeg)](https://towardsdatascience.com/efficient-generalized-spherical-cnns-1493426362ca/) ## [Efficient Generalized Spherical CNNs](https://towardsdatascience.com/efficient-generalized-spherical-cnns-1493426362ca/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Hybrid rotationally equivariant spherical CNNs [Jason McEwen](https://towardsdatascience.com/author/jasonmcewen/) May 3, 202210 min read * [![](../images/010.png)](https://towardsdatascience.com/how-to-train-your-clip-45a451dcd303/) ## [How to Train your CLIP](https://towardsdatascience.com/how-to-train-your-clip-45a451dcd303/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Introduction to CLIP and to how we fine-tuned it for the Italian Language during the HuggingFace Communtiy Week. [Federico Bianchi](https://towardsdatascience.com/author/fede-bianchi/) February 1, 20228 min read * [![](../images/011.png)](https://towardsdatascience.com/temporal-graph-learning-in-2024-feaa9371b8e2/) ## [Temporal Graph Learning in 2024](https://towardsdatascience.com/temporal-graph-learning-in-2024-feaa9371b8e2/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) Continue the journey for evolving networks [Shenyang(Andy) Huang](https://towardsdatascience.com/author/shenyanghuang1996/) January 18, 202421 min read * [![](../images/012.jpeg)](https://towardsdatascience.com/how-to-effectively-define-and-validate-use-cases-932c284837fe/) ## [How to effectively define and validate AI/ML Use Cases](https://towardsdatascience.com/how-to-effectively-define-and-validate-use-cases-932c284837fe/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Most data professionals struggle to define Use Cases that transform into AI/ ML solutions that bring real value. [Przemek Pospieszny](https://towardsdatascience.com/author/przemek_pospieszny/) September 9, 20209 min read * [![](../images/013.png)](https://towardsdatascience.com/the-art-and-science-of-building-ai-48f320d66da6/) ## [The Art and Science of Building AI](https://towardsdatascience.com/the-art-and-science-of-building-ai-48f320d66da6/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) For the AI enthusiasts of the world, it is always a wonder as to what makes an AI model great. Does it take a scientific mind or an... [Amit Gupta](https://towardsdatascience.com/author/amitgupta01/) August 19, 20215 min read * [![](../images/014.jpeg)](https://towardsdatascience.com/donuts-coffee-meet-the-city-economy-12a540faf83b/) ## [Donuts & Coffee Meet The City Economy](https://towardsdatascience.com/donuts-coffee-meet-the-city-economy-12a540faf83b/) [Data Science](https://towardsdatascience.com/category/data-science/) Mathematical Scaling in Canadian Fast Food Service [Anders Ohrn](https://towardsdatascience.com/author/ajohrn/) June 6, 20187 min read * [![](../images/015.jpeg)](https://towardsdatascience.com/building-data-science-gui-apps-with-pysimplegui-179db54a9a15/) ## [Building Data Science GUI Apps with PySimpleGUI](https://towardsdatascience.com/building-data-science-gui-apps-with-pysimplegui-179db54a9a15/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) How to build simple Data Science Windows/GUI apps following a minimal learning curve? [Tirthajyoti Sarkar](https://towardsdatascience.com/author/tirthajyoti/) January 14, 20217 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) © Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network) #### Recommended Articles Close * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ "How Relevance Models Foreshadowed Transformers for NLP") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ "Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ "The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ "Does Your Company Have a Data Strategy?")