![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Data Science](https://towardsdatascience.com/category/data-science/) # Courage to Learn ML: A Detailed Exploration of Gradient Descent and Popular Optimizers Are You Truly Mastering Gradient Descent? Use This Post as Your Ultimate Checkpoint [Amy Ma](https://towardsdatascience.com/author/amyma101/) January 9, 202423 min read ![We will use an RPG game as an analogy today. Created By ChatGPT](../images/000.png) We will use an RPG game as an analogy today. Created By ChatGPT Welcome back to a new chapter of '[Courage to Learn ML](https://towardsdatascience.com/tagged/courage-to-learn-ml). For those new to this series, this series aims to make these complex topics accessible and engaging, much like a casual conversation between a mentor and a learner, inspired by the writing style of "[The Courage to Be Disliked](https://www.goodreads.com/book/show/43306206-the-courage-to-be-disliked)," with a specific focus on machine learning. In our previous discussions, our mentor and learner discussed about some common loss functions and the three fundamental principles of designing loss functions. Today, they'll explore another key concept: gradient descent. As always, here's a list of the topics we'll be exploring today: * What exactly is a gradient, and why is the technique called 'gradient descent'? * Why doesn't vanilla gradient descent perform well in Deep Neural Networks (DNNs), and what are the improvements? * A review of various optimizers and their relationships (Newton's method, Adagrad, Momentum, RMSprop, and Adam) * Practical insights on selecting the right optimizer based on my personal experience * * * Learn this step by step with the interactive [AI and Data Scientist](https://roadmap.sh/ai-data-scientist) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps. ## So, we've set up the loss function to measure how different our predictions are from actual results. To close this gap, we adjust the model's parameters. Why do most algorithms use gradient descent for their learning and updating process? To address this question, let's imagine developing a unique update theory, assuming we're unfamiliar with gradient descent. We start by using a loss function to quantify the discrepancy, which encompasses both the signal (the divergence of the current model from the underlying pattern) and noise (such as data irregularities). The subsequent step involves conveying this data and utilizing it to adjust various parameters. The challenge then becomes determining the extent of modification needed for each parameter. A basic approach might involve calculating the contribution of each parameter and updating it proportionally. For instance, in a linear model like W*x + b = y, if the prediction is 50 for x = 1 but the actual value is 100, the gap is 50. We could then compute the contribution of w and b, adjusting them to align the prediction with the actual value of 100. However, two significant issues arise: 1. Calculating the Contribution: With many potential combinations of w and b that could yield 100 when x = 1, how do we decide which combination is better? 2. Computational Demand in Complex Models: Updating a Deep Neural Network (DNN) with millions of parameters could be computationally demanding. How can we efficiently manage this? These difficulties highlight the complexity of the problem. It's nearly impossible to accurately determine each parameter's contribution to the final prediction, especially in non-linear and intricate models. Therefore, to update model parameters effectively based on the loss, we need a method that can precisely dictate the adjustments for each parameter without being computationally costly. Thus, rather than focusing on how to allocate loss across each parameter, we could consider it as a strategy to traverse the loss surface. The objective is to locate a set of parameters that guide us to the global minimum - the closest approximation achievable by the model. This adjustment process is akin to playing an RPG game, where the player seeks the lowest point in a map. This is the foundational idea behind gradient descent. ![We have a set of basic movements to guide our hero towards finding the treasure, which is located at the lowest point in the map. Created by ChatGPT](../images/001.png) We have a set of basic movements to guide our hero towards finding the treasure, which is located at the lowest point in the map. Created by ChatGPT ## So what is exactly gradient descent? Why it call gradient descent? Let's break it down. The loss surface, central to optimization, is shaped by the loss function and model parameters, varying with different parameter combinations. Imagine a 3D loss surface plot: the vertical axis represents the loss function value, and the other two axes are the parameters. At the global minimum, we find the parameter set with the lowest loss, our ultimate target to minimize the gap between actual results and our predictions. ![Source: https://miro.medium.com/v2/resize:fit:1400/format:webp/1*DDjCOEPSHLsU7tff7LmYUQ.png](../images/002.png) Source: But how do we navigate towards this global minimum? That's where the gradient comes in. It guides us in the direction to move. You might wonder, why calculate the gradient? Ideally, we'd see the entire loss surface and head straight for the minimum. But in reality, especially with complex models and numerous parameters, we can't visualize the entire landscape - it's more complex than a simple valley. We can only see what's immediately around us, like being in a foggy landscape in an RPG game. So, we use **the gradient, which points towards the steepest ascent, and then head in the opposite direction, towards the steepest descent**. By following the gradient, we gradually descend to the global minimum on the loss surface. This journey is what we call gradient descent. ![The foggy landscape. Created By ChatGPT](../images/003.png) The foggy landscape. Created By ChatGPT ## How exactly does gradient descent decide the adjustments needed for each parameter, and why is it more effective than our initially proposed simple method? Our objective is to minimize the loss function with a particular set of parameters, achievable only by adjusting these parameters. We indirectly influence the loss function through these changes. Let's revisit our RPG analogy. The hero, with only basic movements (left/right, forward/backward) and limited visibility, aims to find the lowest point on an uncharted map to unearth a legendary weapon. We know the gradient indicates the direction to move, but it's more than just a pointer. It also decomposes into fundamental movements. The gradient, a vector of partial derivatives with respect to each parameter, signifies how much and in which basic direction (left, right, forward, backward) to move. It's like a magical guide not only pointing to the hill's left side will lead us to the legendary weapon but also instructing specific turns and steps to take. However, it's crucial to understand what the gradient actually is. Often tutorials suggest imagining you're on a hill and looking around to choose the steepest direction to descend. But this can be misleading. The gradient isn't a direction on the loss surface itself but a projection of that direction onto the parameter dimensions (in the graph, the x,y coordinates), guiding us in the loss function's minimal direction. This distinction is crucial - the gradient isn't on the loss surface but a directional guide within the parameter space. ![The vector v represents the gradient and can be expressed in the coordinate form determined by our parameters x and y. Source: https://eli.thegreenplace.net/images/2016/plot-3d-parabola.png](../images/004.png) The vector v represents the gradient and can be expressed in the coordinate form determined by our parameters x and y. Source: That's why most visualizations use parameter contours, not the loss function to illustrate gradient descent processes. The movement is about adjusting parameters, with changes on the loss function being a consequence. ![Source: https://machinelearningmastery.com/wp-content/uploads/2021/07/gradientDescent1.png](../images/005.png) Source: Gradient is formed by partial derivatives. A partial derivative, on the other hand, helps us understand how a function changes in relation to a specific parameter, holding others constant. This is how gradients quantify each parameter's influence on the loss function's directional change. Gradient descent naturally and efficiently resolves the parameter tuning dilemma we initially faced. It's especially adept at locating global minima in convex problems and local minima in nonconvex scenarios. Modern implementations benefit from parallelization and acceleration via GPUs or TPUs. Variations like mini-batch gradient descent and Adam optimize its efficiency across different contexts. In summary, gradient descent is stable, capable of handling large datasets and numerous parameters, making it a superior choice for our learning purposes. ## In the gradient descent formula πœƒ = πœƒ - πœ‚β‹…βˆ‡πœƒπ½(πœƒ), we use the learning rate (πœ‚) multiplied by the gradient (βˆ‡πœƒπ½(πœƒ)) to update each parameter. Why is the learning rate necessary? Wouldn't adjusting the parameter directly with the gradient be faster? At first glance, using only the gradient to reach the global minimum seems straightforward. However, this overlooks the gradient's nature. The gradient, a vector of partial derivatives of the loss function with respect to parameters, indicates **the direction of steepest ascent and its steepness**. Essentially, it guides us in the most promising direction given our limited perspective, but it's just that - a direction. We must decide how to act on this information. The learning rate, or step size, is crucial here. It controls the pace of our learning process. Imagine the character at the top of a mountain in our video game example. The game indicates that moving left and downhill is the quickest way to the treasure (representing the global minimum). It also informs you about the hill's steepness: a steepness of 5 to the left and 10 forward. Now, you decide how far to move your hero. If you're cautious, you might choose a small step, say a learning rate of 0.001. This means your hero moves 0.001_5 = 0.005 units to the left and 0.001_10 = 0.01 units forward. As a result, your hero moves towards the goal, aligned with the gradient's direction but at a controlled pace without overshooting. To sum up, it's important not to confuse the gradient's magnitude with the learning pace. The magnitude indicates the steepness of the ascent direction, which can vary depending on the loss surface. **The learning rate, on the other hand, is a choice you make, independent of the dataset. It's a hyperparameter, signifying how cautiously or aggressively we want to proceed in our learning journey.** ## If the learning rate isn't dependent on the dataset, how do we determine it? What's the typical range for a learning rate, and what are the issues with setting it too high or too low? Since the learning rate is a hyperparameter, we typically choose it through experimentation, like cross-validation, or based on previous experience. The common range for learning rates is between 0.001 and 0.1. This range is based on empirical observations from the data science community, who have found that learning rates within this range tend to converge faster and more effectively. Theoretically, we prefer a learning rate no higher than 0.1 because larger rates can alter our parameters too drastically at each step, leading to risks like overshooting. On the practical side, we avoid rates lower than 0.001 as they can slow down the learning process, making it computationally expensive and time-consuming. The common range gives us insights into the problems with extreme learning rates. When the rate is too high, the large step size might cause the algorithm to move too fast, leading to overshooting and never reaching the goal. Conversely, a very low rate results in tiny steps, potentially taking an excessive amount of time to reach the global minimum, thus wasting computational resources and time. Here's a visual representation to help understand the impact of different learning rates on the learning process: ![Source: https://www.jeremyjordan.me/content/images/2018/02/Screen-Shot-2018-02-24-at-11.47.09-AM.png](../images/006.png) Source: ## What are the limitations of vanilla gradient descent, especially when applied to DNN models? Even though gradient descent works well in theory, it faces significant challenges in practical applications, particularly with DNN models. The key limitations include: * **Vanilla gradient descent is computationally intensive.** The vanilla gradient descent formula πœƒ = πœƒ - πœ‚β‹…βˆ‡πœƒπ½(πœƒ) requires calculating the average loss 𝐽(πœƒ) across the entire dataset. This means comparing all predicted values with their true labels and averaging these differences. For a typical DNN model, which often involves millions of parameters and large datasets, this process becomes computationally intense. This complexity is one of the reasons why practical applications of DNNs weren't feasible until the advent of models like AlexNet in 2012, despite gradient descent being a much older concept. * **The loss surface in DNNs, usually non-convex, contains multiple local minimums.** The simplistic criterion of using gradient descent to find points where 𝐽(πœƒ) = 0 is inadequate in such scenarios. In our video game analogy, this is like our hero being unsure of where to move next, as the steepest descent direction becomes indiscernible. Particularly problematic are plateau and saddle points, where 𝐽(πœƒ) is zero. The areas that are close to saddle and local minimums can be even harmful, since in those areas 𝐽(πœƒ) is very small and close to zero. The small value of 𝐽(πœƒ) can significantly slow down the learning process, consuming time and computational resources unnecessarily. ![Keep in mind that in the foggy landscape scenario, we have no idea whether we're on a plateau, at a global minimum, or a local minimum. Source: https://miro.medium.com/v2/resize:fit:1400/1*vx235JUNzgv0fGzONlNIYg.png](../images/007.png) Keep in mind that in the foggy landscape scenario, we have no idea whether we're on a plateau, at a global minimum, or a local minimum. Source: ## Why does gradient descent work, and why must the learning rate be small? > **Author's Note:** I debated whether to include this section in my post, knowing that mathematical formulas can be intimidating. Yet, while exploring gradient descent, I found myself questioning why the direction of the steepest descent is believed to lead to the global minimum and why tutorials emphasize setting a small learning rate to prevent overshooting. My attempts to rationalize these concepts through RPG game analogies were helpful but not entirely satisfying. It was the straightforward mathematical proof behind gradient descent that finally put my doubts to rest. If you've had similar questions, I encourage you to read this section. It might just offer the clarity you need to grasp these concepts fully. To grasp gradient descent, we need to discuss approximating the loss function using Taylor series. The Taylor series is an incredibly powerful tool for estimating the value of a complex function at a specific point. It's like trying to describe a complex situation using simpler, individual elements. Instead of using a broad statement like "I had a car accident," you break it down into specific events: driving the dog to the vet, the dog popping up in the backseat, the phone ringing, and then the crash. The Taylor series does something similar. Instead of trying to describe a complex function f(x) with a single general term, it uses a series of terms to describe the specific value of f(x) at x = a. Taylor series breaks down a function into a sum of polynomial terms based on the function's derivatives at a specific point (x = a). Each term of the series adds more detail to the approximation. ![](../images/body-b32fdf7fc8a06162bcc5.png) > For an engaging explanation of the Taylor series and how the first and second derivatives contribute to its expansion, check out [3Blue1Brown's video on Taylor series](https://www.youtube.com/watch?v=3d6DsjIBzJ4&t=1020s). It effectively demonstrates why these derivatives are often sufficient for a solid approximation around the point of interest. Now, returning to our objective of minimizing the loss function, 𝐽(πœƒ), we can apply the Taylor series, primarily relying on the first derivative for an effective approximation. In this context, 𝐽(πœƒ) is a multivariable function, where πœƒ is a vector representing a set of parameter variables, and a is the current value of these parameters. ![](../images/body-624097ee950c2959bcd9.png) To minimize 𝐽(πœƒ), we can only adjust (πœƒ - a) to be really small, meaning our next set of πœƒ values must be very close to our current position, represented by vector a. This necessity for proximity is why a small learning rate is crucial. If πœƒ deviates significantly from the current parameter values a, minimizing our loss function becomes challenging. Additionally, since the gradient 𝐽'(a) indicates a direction, we choose (πœƒ - a) to move in the opposite direction. This approach explains why following the direction of the steepest descent (opposite to the gradient) steers us towards the global minimum, where 𝐽(πœƒ) = 0. ## You particularly mentioned that vanilla gradient descent isn't ideal for DNNs. Why? The core issue with applying vanilla gradient descent to DNNs lies in the nature of their loss functions, which are typically non-convex. Gradient descent struggles in these scenarios. Imagine navigating a complex map that isn't just a simple hill or valley but a terrain filled with unpredictable elevations and depressions. In such a landscape, gradient descent may fail to guide us to the global minimum and might even increase the loss function's value after parameter updates. This is partly because first-order partial derivatives provide limited information, showing how the loss changes with respect to one parameter while holding others constant. But this **doesn't account for interactions between multiple parameters** , which are crucial in complex models. ![Source: https://www.researchgate.net/publication/334986617/figure/fig1/AS:789459528335363@1565233227897/Illustration-of-the-non-convex-mean-squared-error-cost-function-of-a-two-dimensional.png](../images/008.png) Source: Revisiting the Taylor series concept from our previous discussion, we see that when the objective or loss function becomes highly complex, using only the first derivative doesn't provide an accurate approximation. Hence, vanilla gradient descent is less effective in navigating complex, non-convex loss functions. It's important to note that gradient descent can still reduce the loss in non-convex scenarios. It's a general method for continuous optimization and can be applied to nonconvex functions, where it will converge to a stationary point. However, this point might not be the global minimum but could be a local minimum or even a saddle point, depending on the function's convexity. ## How can we improve upon vanilla gradient descent to accelerate reaching the global minimum? Great question! Our previous discussions and the video game analogy reveal several ways to refine gradient descent. Let's explore these enhancements: ### **Making the map more player-friendly.** Just like navigating a more accommodating map in a game makes reaching the treasure easier, certain modifications can make the path to the global minimum smoother in gradient descent. One of the most obvious method to modify your map (loss surface) is by **selecting the appropriate loss function**. For example, in classification problems, cross-entropy is commonly used (or log loss for binary classification). Opting for Mean Squared Error (MSE) in classification tasks can result in a loss surface with multiple local minima, increasing the likelihood of the learning process getting stuck in one of these minima. Some other methods include using techniques like feature selection, regularization, feature scaling and batch normalization. Feature selection not only reduces computational costs but also simplifies the loss surface, as the loss function is influenced by the number of parameters. **L1/L2 regularization** can help in this process: L1 for feature selection and L2 for smoothing the loss surface. **Feature scaling** is crucial because features on different scales can cause uneven step sizes, potentially slowing down convergence or preventing it altogether. By scaling features to similar ranges, we ensure more uniform steps, facilitating faster and more consistent convergence. Like feature scaling, **batch normalization** aims to normalize the inputs between layers. This method introduces additional hyperparameters for tuning. Batch normalization standardizes a layer's output before it becomes the input for the subsequent layer, establishing a dependency among samples in a batch through the calculation of mean and variance of the outputs from the whole batch. Therefore, selecting an appropriate batch size is crucial, as a batch size of 1 is typically ineffective for batch normalization. The paper "How Does Batch Normalization Help Optimization?" suggests that this technique will make the loss surface smoother while minimally altering the position of the global minimum. One thing to note is that, during prediction phrase, batch normalization uses the moving average and variance computed during the training phase, instead of calculating batch-wise statistics from test data. This is the same idea as any data transformation between training and test phrase. It's important to note that during the prediction phase, batch normalization uses the moving average and variance calculated during training to normalize batches, rather than computing batch-specific statistics from the test data.This approach mirrors aligns with the standard practice of preventing data leakage between the training and testing data during data transformation phrase. ![Source: https://qph.cf2.quoracdn.net/main-qimg-591ce5575f71110585d9246bf8c2ca0b-pjlq](../images/009.jpg) Source: ### **Move faster.** There's a Chinese saying, "ε€©δΈ‹ζ­¦εŠŸζ— εΏ«δΈη ΄" (no martial art is indestructible, except for speed). This concept applies to gradient descent as well. In vanilla gradient descent, calculating the loss use the whole training data in each iteration is time-consuming and resource-intensive. However, we can achieve similar results with less data. The idea is that the average loss from a smaller sample is not significantly different from the entire dataset. By employing methods like mini-batch gradient descent (using a subset of data) or Stochastic Gradient Descent (SGD, selecting a random sample each time), we can speed up the process significantly. These approaches enable quicker computations and updates, making them particularly effective in DNNs. ![Source: https://editor.analyticsvidhya.com/uploads/58182variations_comparison.png](../images/010.png) Source: > In the world of machine learning, the term 'SGD' (Stochastic Gradient Descent) has evolved to generally refer to any variant of gradient descent that uses a small subset of data for each run. This includes both the original form of SGD and mini-batch gradient descent. Conversely, 'batch gradient descent' denotes the approach where the entire training dataset is used for each gradient descent iteration. > Here are some common terms you'll encounter in discussions about gradient descent: > **Batch Size** : This is the number of observations (or training data points) used for one iteration of gradient descent. > **Iterations per epoch** : This indicates how many gradient descent iterations are required to go through the entire dataset once, given the current batch size. > **Epoch** : An epoch represents one complete pass through the entire training dataset. The number of epochs is a choice made by the modeler and is independent from both the batch size and the number of iterations. > To illustrate, consider a training dataset with 1000 observations. If you opt for a batch size of 10, you'll need 1000/10 = 100 iterations to cover the entire dataset once. If you set your epoch count to 5, the model will go through the entire dataset 5 times. This means a total of 5 * 100 = 500 gradient descent iterations will be performed. ### **Move smarter by gathering more information.** Moving smarter in gradient descent involves looking beyond just the steepest descent. In our previous session on the mathematics behind gradient descent, we learned that using the first derivative of f(x) at x = a provides a solid approximation of f(x). However, to enhance this approximation, incorporating additional derivative terms is beneficial. In our video game analogy, this equates to not just finding the steepest descent direction but also moving our camera around to gain a comprehensive understanding of the landscape. This approach is particularly useful for identifying whether we're at a local/global minimum (like being at the bottom of a valley), or a maximum (at the top of a hill), or a saddle point (surrounded by hills and valleys). However, our limited view means we can't always distinguish between a local and a global one. Expanding on this concept, we can apply **Newton's Method**. This technique calculates both the first and second derivatives of the objective or loss function. The second derivatives create the Hessian matrix, offering a more detailed view of the loss function's landscape. With information from both the first and second derivatives, we gain a closer approximation of the loss function. Unlike traditional gradient descent, Newton's Method doesn't use a learning rate. However, variations such as Newton's method with line search do include a learning rate, adding a level of adjustability. While Newton's Method might seem more efficient than vanilla gradient descent, it's not commonly used in practice due to its higher computational demands. For more insights into why Newton's Method is less prevalent in machine learning, you can refer to the discussion [here](https://stats.stackexchange.com/questions/253632/why-is-newtons-method-not-widely-used-in-machine-learning). ### **Adjust step size while moving around.** This strategy can make our journey more efficient. The simplest approach is to reduce the step size as we approach the global minimum. This is particularly relevant in gradient descent variants like SGD, where the learning rate decreases with each epoch (via learning rate schedules). However, this method uniformly adjusts the learning rate across all parameters, which might not be ideal given the complexity of the loss landscape. In the realm of video games, this is akin to varying the intensity of different movements to navigate more effectively. That's where methods like **Adagrad** come in, offering an adaptive gradient approach. It does so by accumulating the history of squared gradients for each parameter and using the square root of this accumulation to adjust the learning rate individually. This method is like fine-tuning our actions in the game based on past experiences, especially when unusual updates occur. However, while intuitive, **Adagrad can slow down due to its aggressive rate decay**. Variations of this method seek to balance this aspect. ![](../images/body-d663fddf586c9a5ae79e.png) > There are two ways to understand why Adagrad use the sum of squared gradients. Firstly, it allows more cautious learning when the current gradient is significantly larger than historical ones. If we encounter a substantially larger gradient at a given time, adding its squared value to the decay term increases the term significantly, leading to a smaller learning rate and more cautious updates. Secondly, this approach approximates the magnitude of the function's second derivatives. A larger sum of squared first derivatives suggests a steeper surface, indicating the need for smaller steps. ### **Choosing better basic movements.** Optimizing our moves in gradient descent involves refining our approach based on the landscape. In vanilla gradient descent, our path often zigzags, similar to uncertain steps in a video game. This can be visualized with the contour of two parameters showing more vertical movements and fewer horizontal ones, even though a horizontal path might be more efficient. ![](../images/body-360eeab5ee13d995226f.png) Feature scaling can help, but with the complex surfaces in DNNs, a more directed approach is needed. This is where **Momentum Gradient Descent** comes into play. It identifies if movements in certain directions are counterproductive. Instead of directly updating parameters with the current gradient, it calculates an exponential moving average of past gradients. This 'momentum' helps accelerate progress in consistent directions and dampens movements in unproductive ones. ![](../images/body-eeb62ead1701d635f9b9.png) ![](../images/body-11586c2aaa871c53b60c.png) Notice here, unlike Adagrad which uses squared gradients, we accumulate the historical gradients directly. This approach is crucial because it allows for the positive and negative movements to cancel out each other. Think of this as building momentum in a specific direction, accelerating the learning process if we consistently move in a similar direction. Conversely, a small accumulated value suggests little progress in that direction, hinting that it may not be the most promising movement toward the minimum. To enhance this smoothing effect and capitalize on past movements, we assign more weight to the accumulated gradient history. This is done by setting a higher value for the momentum coefficient, typically denoted as _Ξ²_. A common choice for _Ξ²_ is 0.9, which strikes a balance between giving importance to historical movements while still being responsive to new gradient information. This method ensures a smoother journey by favoring directions with consistent progress and dampening oscillations that do not contribute effectively towards reaching the goal. ### **Combine those strategies!** Merging the principles of Adagrad and Momentum Gradient Descent offers an innovative way to enhance gradient descent. Both these methods rely on historical gradients, but with a key difference in their approach. Momentum Gradient Descent uses the exponential moving average of gradients instead of a simple average. The advantage here is that by adjusting the momentum coefficient Ξ², we can strike a balance between the influence of historical gradient trends and the current gradient. * Inspired by this, we can apply a similar logic to Adagrad, leading to the development of **RMSprop (Root Mean Square Propagation)**. RMSprop is essentially an evolved version of Adagrad, utilizing the exponential moving average of historical gradients rather than a simple average. This modification places more weight on historical gradients, reducing the impact of exceptionally large current gradients. Consequently, it leads to a less aggressive decrease in the learning rate, addressing the issue of slow learning rates that Adagrad often faces. ![](../images/body-57913581c25741e94a5d.png) * Building further on this idea, why not combine the learning rate adjustment of Adagrad/RMSprop with the gradient adjustment strategy of Momentum? This thought led to the creation of Adam (**Adaptive Moment Estimation** , I remember it as the baby of Adaptive learning rate and Momentum). Adam essentially combines these two methods by using historical gradients in two ways: one for adjusting the exponential moving average (momentum), and the other for managing the scale of historical gradients (RMSprop). This dual application makes Adam a highly effective and stable optimizer. Adam is a popular choice of optimizer, even though it introduces two additional hyperparameters for fine-tuning. ![](../images/body-a33cb3aa283b5b1d4bb1.png) ## With various optimizers available, how should one choose the right optimizer in practice? In practice, choosing the right optimizer depends on the specifics of your data and learning objectives. Here are some of general guidelines from my observations and experience: * **SGD for Online Learning** : Online learning involves processing a continuous stream of incoming data, which requires frequent updates to the model with new, small data batches. Stochastic Gradient Descent (SGD) is particularly well-suited for this scenario as it efficiently uses small batches for more frequent model updates compared to other optimizers. Additionally, SGD is effective in environments where data is unstable or experiencing minor shifts. Using an appropriate learning rate to avoid overshooting, SGD can quickly adapt the model to subtle changes in data. However, it's important to note that while SGD is capable of handling non-stationary environments, it may not be as effective in capturing major shifts in data patterns. * **Adam and sparse data.** When dealing with data that has a high number of zero entries, it's described as being sparse. **The challenge with sparse data lies in the limited information available for certain features.** Adam optimizer is particularly effective in this context as it integrates both momentum and adaptive learning rate mechanisms. This combination allows Adam to tailor the learning rate for each parameter individually. Consequently, it provides more frequent updates for features that are underrepresented or have less information due to the data's sparsity, ensuring a more balanced and effective learning process. * **Don't combine optimizers, but use them together wisely.** While it's possible to use multiple optimizers in the learning process, **they shouldn't be applied simultaneously within a single learning phase** , as this can lead to confusion in the model and complicate the learning path. Instead, there are two strategic approaches to effectively utilizing multiple optimizers: * **Switching between optimizers at different stages.** For instance, when training DNNs, you might begin with Adam for its rapid progress capabilities in the initial epochs. Later, as the training progresses, switching to SGD in the subsequent epochs can offer more precise control over the learning process, aiding the model in converging to a more optimal local minimum. * **Using different optimizers to train different parts of a model.** In scenarios where new layers are added to an existing, pre-trained model, a nuanced approach can be beneficial. For the pre-trained layers, a stable and less aggressive optimizer is ideal for maintaining the integrity of the already learned features. In contrast, for the newly added layers, an optimizer that's more aggressive and facilitates faster learning adjustments would be more suitable. This method ensures that each part of the model receives the most appropriate optimization technique based on its specific learning requirements. ## Wait, why can't we use Adam and SGD together? And why is SGD often preferred in later epochs for finer optimization, even though Adam is considered more advanced? Adam and SGD differ significantly, especially in how they manage the learning rate. While Adam uses an adaptive learning rate, SGD typically employs a fixed rate or a scheduled adjustment. This distinction makes them fundamentally different optimizers. Adam's ability to adjust the learning rate doesn't always make it "more advanced". In some cases, simplicity is better. SGD's simple learning rate and steady approach are more effective, particularly in later epochs. Adam may reduce the learning rate too aggressively, leading to instability, whereas SGD's consistent rate can more reliably approach a better local minimum and enhance stability. Furthermore, SGD's slower convergence can help prevent overfitting. Its fine-grained and controlled adjustments allow for a more precise fit to training data, potentially improving generalization to unseen data. The fixed or scheduled learning rate of SGD also offers researchers better control over the model's learning process, emphasizing preferences on precision and stability over speed, especially in the final phases of model tuning. * * * If you're enjoying t[his series](http://towardsdatascience.com/tagged/courage-to-learn-ml), remember that your interactions - claps, comments, and follows - do more than just support; they're the driving force that keeps this series going and inspires my continued sharing. Other posts in this series: * [Courage to Learn ML: Demystifying L1 & L2 Regularization (part 1)](https://towardsdatascience.com/understanding-l1-l2-regularization-part-1-9c7affe6f920) * [Courage to Learn ML: Decoding Likelihood, MLE, and MAP](https://towardsdatascience.com/courage-to-learn-ml-decoding-likelihood-mle-and-map-65218b2c2b99) * [Courage to Learn ML: A Deeper Dive into F1, Recall, Precision, and ROC Curves](https://medium.com/p/d5c0a46e5eb7) * [Courage to Learn ML: An In-Depth Guide to the Most Common Loss Functions](https://medium.com/p/84a6b07cca17) * * * _**If you liked the article, you can find me on**_[ _**LinkedIn**_](https://www.linkedin.com/in/amyma101/) _**.**_ ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-bf30693e05bb856f0cb2.png) Written By ## [Amy Ma](https://towardsdatascience.com/author/amyma101/) [See all from Amy Ma](https://towardsdatascience.com/author/amyma101/) [Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Courage To Learn Ml](https://towardsdatascience.com/tag/courage-to-learn-ml/)[Data Science](https://towardsdatascience.com/tag/data-science/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fcourage-to-learn-ml-a-detailed-exploration-of-gradient-descent-and-popular-optimizers-022ecf97be7d%2F&title=Courage%20to%20Learn%20ML%3A%20A%20Detailed%20Exploration%20of%20Gradient%20Descent%20and%20Popular%20Optimizers) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fcourage-to-learn-ml-a-detailed-exploration-of-gradient-descent-and-popular-optimizers-022ecf97be7d%2F&title=Courage%20to%20Learn%20ML%3A%20A%20Detailed%20Exploration%20of%20Gradient%20Descent%20and%20Popular%20Optimizers) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fcourage-to-learn-ml-a-detailed-exploration-of-gradient-descent-and-popular-optimizers-022ecf97be7d%2F&text=Courage%20to%20Learn%20ML%3A%20A%20Detailed%20Exploration%20of%20Gradient%20Descent%20and%20Popular%20Optimizers) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/011.jpg)](https://towardsdatascience.com/a-case-for-the-t-statistic/) ## [A Case for the T-statistic](https://towardsdatascience.com/a-case-for-the-t-statistic/) [Data Science](https://towardsdatascience.com/category/data-science/) And how it compares to the run-of-the-mill z-score [Aniruddha Karajgi](https://towardsdatascience.com/author/polaris000/) January 21, 202612 min read * [![](../images/012.png)](https://towardsdatascience.com/rgb-color-image-compression-using-principal-component-analysis-fce3f48dfdd0/) ## [RGB Color Image Compression Using Principal Component Analysis](https://towardsdatascience.com/rgb-color-image-compression-using-principal-component-analysis-fce3f48dfdd0/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) PCA in Action for Dimensionality Reduction [Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/) March 29, 20229 min read * [![](../images/013.jpg)](https://towardsdatascience.com/debugging-sagemaker-endpoints-quickly-with-local-mode-2975bd55f6f7/) ## [Debugging SageMaker Endpoints Quickly With Local Mode](https://towardsdatascience.com/debugging-sagemaker-endpoints-quickly-with-local-mode-2975bd55f6f7/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Stop Waiting For Your Endpoints To Create [Ram Vegiraju](https://towardsdatascience.com/author/ram-vegiraju/) August 8, 20224 min read * [![](../images/014.jpg)](https://towardsdatascience.com/beginners-baseline-model-for-machine-learning-project-f835ebd53ee0/) ## [Beginners Baseline Model for Machine Learning Project](https://towardsdatascience.com/beginners-baseline-model-for-machine-learning-project-f835ebd53ee0/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Learn to standardize your machine learning development [Cornellius Wijaya](https://towardsdatascience.com/author/cornelliusyudhawijaya/) May 5, 20227 min read * [![](../images/015.jpg)](https://towardsdatascience.com/pytorch-elmo-844d2391a0b2/) ## [Pytorch ELMo, trained from scratch](https://towardsdatascience.com/pytorch-elmo-844d2391a0b2/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) Learn powerful contextual word embeddings on your own corpus using AllenNLP [Yilong Qin](https://towardsdatascience.com/author/louixp/) February 17, 20218 min read * [![](../images/016.jpg)](https://towardsdatascience.com/avoiding-entity-key-drift-in-a-data-lake-step-1-normalization/) ## [Avoiding Entity Key Drift in a Data Lake: Step 1, Normalization](https://towardsdatascience.com/avoiding-entity-key-drift-in-a-data-lake-step-1-normalization/) [Data Engineering](https://towardsdatascience.com/category/data-science/data-engineering/) Tackling the bedrock problem of maintaining identity in a high-frequency data lake [Rahul Saha](https://towardsdatascience.com/author/rahul.saha/) July 29, 202614 min read * [![](../images/017.png)](https://towardsdatascience.com/cleaning-a-messy-car-dataset-with-python-pandas-700fe10a7180/) ## [Cleaning a Messy Car Dataset with Python Pandas](https://towardsdatascience.com/cleaning-a-messy-car-dataset-with-python-pandas-700fe10a7180/) [Programming](https://towardsdatascience.com/category/programming/) Whether you are performing exploratory data analysis or building a complex ML system, you need to make sure the data is cleaned [Soner YΔ±ldΔ±rΔ±m](https://towardsdatascience.com/author/sonery/) October 17, 20236 min read * [![](../images/018.jpeg)](https://towardsdatascience.com/top-3-reasons-why-im-not-buying-the-new-m1-pro-max-macbook-as-a-data-scientist-3249e39ff30/) ## [Top 3 Reasons Why I'm NOT Buying The New M1 Pro/Max MacBook as a Data Scientist](https://towardsdatascience.com/top-3-reasons-why-im-not-buying-the-new-m1-pro-max-macbook-as-a-data-scientist-3249e39ff30/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) I'm waiting for the M2 instead - and here's why you should too. [Dario RadečiΔ‡](https://towardsdatascience.com/author/radecicdario/) October 28, 20216 min read * [![](../images/019.jpg)](https://towardsdatascience.com/how-to-maximize-the-value-of-your-data-analysis-309059dcd243/) ## [How to Maximize the Value of Your Data Analysis](https://towardsdatascience.com/how-to-maximize-the-value-of-your-data-analysis-309059dcd243/) [Data Science](https://towardsdatascience.com/category/data-science/) Our weekly selection of must-read Editors' Picks and original features [TDS Editors](https://towardsdatascience.com/author/towardsdatascience/) June 9, 20223 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)β€’[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)β€’[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)β€’[Privacy Policy](https://towardsdatascience.com/privacy-policy/)β€’[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) Β© Insight Media Group, LLC 2026β€’[The New Stack](https://thenewstack.io)β€’[roadmap.sh](https://roadmap.sh)β€’[Creator Network](https://insightmediagroup.io/creator-network) #### Recommended Articles Close * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ "How Relevance Models Foreshadowed Transformers for NLP") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT β€” Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ "Javascript Fatigue: HTMX Is All You Need to Build ChatGPT β€” Part 1") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ "The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ "Does Your Company Have a Data Strategy?")