{ "source_file": "0930", "source_platform": "Medium", "author": "Riccardo Andreoni", "title_listed": "The Best Optimization Algorithm for Your Neural Network", "keywords": "", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/the-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb/", "publish_date": "2023-10-14", "corpus_stage": "", "notes": "", "id": "5d522e7f2a42", "collected_at": "2026-10-02T03:20:04", "title": "The Best Optimization Algorithm for Your Neural Network | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# The Best Optimization Algorithm for Your Neural Network\nHow to choose it and minimize your neural network training time.\n[Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/)\nOctober 14, 202312 min read\n![Image source: unsplash.com.](../images/000.jpeg)\nImage source: [unsplash.com](https://unsplash.com/photos/FfbVFLAVscw).\nDeveloping any machine learning model involves a rigorous experimental process that follows the [**idea-experiment-evaluation cycle**](https://towardsdatascience.com/ai-ml-practicalities-the-cycle-of-experimentation-fd46fc1f3835).\n![Image by the author.](../images/001.png)\nImage by the author.\nThe above cycle is repeated multiple times until satisfactory performance levels are achieved. The \"experiment\" phase involves both the coding and the training steps of the machine learning model. As **models become more complex** and are trained over much **larger datasets** , training time inevitably expands. As a consequence, training a large deep neural network can be painfully slow.\nFortunately for data science practitioners, there exist several techniques to accelerate the training process, including:\n * **Transfer Learning**.\n * **Weight Initialization** , as Glorot or He initialization.\n * **Batch Normalization** for training data.\n * Picking a **reliable activation function**.\n * Use a **faster optimizer**.\n\n\nWhile all the techniques I pointed out are important, in this post I will focus deeply on the last point. I will describe multiple algorithm for neural network parameters optimization, highlighting both their advantages and limitations.\nIn the last section of this post, I will present a visualization displaying the comparison between the discussed optimization algorithms.\nFor **practical implementation** , all the code used in this article can be accessed in this **GitHub repository** :\n> [**articles/NN-optimizer at main · andreoniriccardo/articles**](https://github.com/andreoniriccardo/articles/tree/main/NN-optimizer)\n## Batch Gradient Descent\nTraditonally, Batch Gradient Descent is considered the **default choice** for the optimizer method in neural networks.\nAfter the neural network generates predictions for the entire training set _X_ , we compare the network's predictions to the actual labels of each training point. This is done to compute a cost function _J(W,b)_ , which is a scalar representation of the capacity of the model to yield accurate predictions. The Gradient Descent optimization algorithm uses the cost function as a guide to tune every network's parameter. This iterative process continues until the cost function is close to zero, or it can't be reduced further. What GD does is to adjust each weight and bias of the network in a certain direction. The chosen direction is the one that reduces the cost function the most.\n![Image by the author.](../images/002.png)\nImage by the author.\nIn visual terms, take the image above for reference. Gradient descent starts at the leftmost blue point. By analyzing the gradient of the cost function around the starting point, it adjusts the parameter (x-axis value) to the right. This process is repeated multiple times until the algorithm ends up with a very good approximation of the optimum.\nIf the cost function has 2 input parameters, the function is not a line but instead, it creates a three-dimensional surface. The batch gradient descent steps can be displayed considering the level curves of the surface:\n![Image by the author.](../images/003.png)\nImage by the author.\nA neural network consists of thousands or millions of parameters that affect the cost functions. While visualizing million-dimensional representations is infeasible, mathematical equations are useful to understand the gradient descent process.\nAt first, we compute the cost function _J(W,b)_ , which is a function of the network's weights _W_ and biases _b_. Then, the backpropagation algorithm computes the derivative of the cost function with respect to each weight and bias of each layer of the network:\n![](../images/body-2a8314583a19366a2c78.png)\n![](../images/body-1f7faf9556391d730954.png)\nKnowing the direction towards which to adjust the parameters, we update them. The magnitude of each parameter's update is regulated by the gradient itself and by the learning rate alpha. Alpha is a hyperparameter of the optimization algorithm and its value is typically maintained constant.\n![](../images/body-a310acfe6e10feeffdde.png)\nBatch Gradient Descent offers several advantages:\n * **Simplicity** : It's a straightforward and easy-to-understand method.\n * **Hyperparameter Management** : It requires tuning only one hyperparameter, the learning rate.\n * **Optimality** : For convex cost functions, it reliably reaches the global optimum with a reasonable learning rate.\n\n\nThe disadvantages of gradient descent are:\n * **Computational Intensity** : It can be slow, particularly for large training sets, as it updates parameters after evaluating all examples.\n * **Local Optima and Saddle Points** : It might get stuck in local optima or saddle points, slowing convergence.\n * **Gradient Issues** : It is prone to vanishing gradient issues.\n * **Memory Usage** : It requires storing the entire training data in CPU/GPU memory.\n\n\n## Mini-Batch Gradient Descent\nIn batch gradient descent algorithm all training examples are used to calculate a single improvement step. The step will be the most accurate, as it considers all the available information. However, this approach often is impractically slow in real-world applications, suggesting the implementation of faster alternatives.\nThe most common solution is called [**Mini-Batch Gradient Descent**](https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/), and it uses only a **small subset of the training data** in order to update the weights' values.\nConsider the whole training set _X_ , and the associated labels _Y_ :\n![](../images/body-e343b6b36936e4a37c27.png)\nwhere _m_ represents the number of training examples.\nRather than feeding the whole batch to the optimization algorithm, we process only a small portion of the training set. Let's say we feed the algorithm the subsets _X^t_ and _Y^t_ , each containing 512 training examples:\n![](../images/body-42654bd8713ba0bba8bf.png)\nFor instance, if the total training set contains 5,120,000 points, it can be divided into 10,000 mini-batches, each containing 512 examples.\nFor each mini-batch, we perform the classical gradient descent operations:\n * **Compute the cost** relative to the mini-batch t, _J^t(W,b)_.\n * **Perform backpropagation** to compute the gradients of _J^t(W,b)_ with respect to each weight and bias.\n * **Update the parameters**.\n\n\nThe green line in the image below shows a typical optimization path of the mini-batch gradient descent. While batch gradient descent follows a more direct path to the optimal point, mini-batch gradient descent appears to take several unnecessary steps, due to its limited dataset at each iteration.\n![Image by the author.](../images/004.png)\nImage by the author.\nThe reason for that is that mini-batch gradient descent, at any time _t,_ has only a small portion of the training set to compute its decision on. With that limitation on the available information, it's clear that the route followed is not the most direct.\nHowever, the significant advantage of mini-batch gradient descent, is that each step is extremely fast to be computed since the algorithm needs to evaluate only a small portion of the data instead of the whole training set. In our example, each step requires the evaluation of only 512 data points instead of 5 million. This is also the reason why almost no real application, requiring a large amount of data, uses batch gradient descent.\nAs the image shows, with mini-batch gradient descent there is no guarantee that the cost at iteration _t+1_ is lower than the cost at iteration t, but, if the problem is well defined, the optimization algorithm reaches an area very close to the optimal point really quickly.\nThe choice of the mini-batch size represents an additional hyperparameter of the network's training process. A batch size equal to the total number of examples (_m_) corresponds to the batch gradient descent optimization. Instead, if _m=1_ we are performing Stochastic Gradient Descent, where each training example is a mini-batch.\nIf the training set is small, batch gradient descent can be a valid option, otherwise, popular mini-batch sizes like 32, 64, 128, 256, and 512 are commonly considered. For some reason, a batch size equal to a power of 2 seems to perform better.\nWhile stochastic gradient descent with a batch size of 1 is an option, it tends to be extremely noisy, and it can bounce further away from the minima. Moreover, stochastic gradient descent is very inefficient if we consider the computation time per example, as it cannot exploit the benefits of vectorization.\n## Gradient Descent with Momentum\nRecall that in both Batch and Mini-Batch Gradient Descent, the parameter update follows a defined formula:\n![](../images/body-0c9942b779d6877bee52.png)\nTherefore, in this equation, the size of each optimization step is determined by the learning rate, a fixed quantity, and the gradient, which is computed at a specific point of the cost function.\nWhen the gradient is computed in a nearly flat section of the cost function, it will be very small, leading to a proportionally small step of gradient descent. Consider the difference the gradient difference at points A and B of the image below.\n![Image by the author.](../images/005.png)\nImage by the author.\n[**Momentum Gradient Descent**](https://towardsdatascience.com/gradient-descent-with-momentum-59420f626c8f) overcomes this issue. We can imagine momentum gradient descent as a bowling ball rolling down a hill, which shape is defined by the cost function. If the ball begins its descent from a steep portion of the slope, its movement begins slowly but it will quickly gain velocity and momentum. Due to its momentum, the ball maintains great velocity even through a -flat area of the slope.\nThis is the core concept of momentum gradient descent: the algorithm will consider previous gradients, and not only the one computed at iteration _t_. Similar to the bowling ball analogy, the gradient computed at iteration _t_ defines the acceleration and not the speed of the ball.\nBoth weights and biases' velocities are computed at each step using the previous velocity and the current iteration's gradient:\n![](../images/body-c672af39ae53d46db6b8.png)\nThe parameter _beta_ , known as momentum, regulates how much the new velocity value is determined from the current slope or from the past velocity value.\nFinally, parameters are updated using the computed velocity:\n![](../images/body-3eb099dfa0c975064737.png)\nMomentum Gradient Descent has proven superior performance compared to Mini-Batch Gradient Descent in most applications. The main drawback of Momentum Gradient Descent, with respect to standard Gradient Descent, is that it requires an additional parameter to tune. However, practice shows how a value of _beta_ equal to 0.9 works effectively.\n## RMS Prop\nConsider a cost function shaped similarly to an **elongated bowl** , where the minimum point is located in the narrowest part. The function's contour is described by the level curves in the image below.\n![Image by the author.](../images/006.png)\nImage by the author.\nIn cases where the starting point is far away from the minimum, Gradient Descent, also with the momentum variation, starts by following the steepest slope, which, in the picture above, is not the optimal path towards the minimum. The key idea of [**RMS Prop**](https://keras.io/api/optimizers/rmsprop) optimization algorithm is to correct the direction early and to aim for the global minimum more promptly.\nSimilarly to Momentum Gradient Descent, RMS Prop requires fine-tuning through an additional hyperparameter known as the decay rate. Through practical experience, it has been proven that setting the decay rate to 0.9 is a good choice for most problems.\n## Adam\n[**Adam**](https://keras.io/api/optimizers/adam) and its variations are probably the most employed optimization algorithms for neural networks. Adam, which stands for **Adaptive Moment Estimation** , is derived from the combinations of Momentum Gradient Descent and RMS Prop together.\nAs a mix of the two optimization methods, Adam needs two additional hyperparameters to be tuned (besides the learning rate _alpha_). We call them _beta _1_ and _beta _2_ and they are the hyperparameters used respectively in Momentum and in RMS Prop. As happened for the other discussed algorithms, effective default choices for _beta _1_ and _beta _2_ exist: _beta _1_ is usually set to 0.9 and _beta _2_ to 0.999. In Adam appears also a parameter _epsilon_ that works as a smoothing term and it is nearly always set to a small value like e-7 or e-8.\nAdam optimization algorithm outperforms all the above-mentioned methods in most cases. The only exceptions can be very simple problems where simpler methods work faster. The efficiency of Adam does come with a trade-off: the need to fine-tune two additional parameters. However, in my opinion it is a small price to pay for its efficiency.\n### Nadam and AdaMax\nIt is worth citing two algorithms born as modifications of the widely used Adam optimization: **Nadam** and **AdaMax**.\n[Nadam](https://keras.io/api/optimizers/Nadam), short of Nesterov-accelerated Adaptive Moment Estimation, enhances Adam by incorporating Nesterov Accelerated Gradient (NAG) into its framework. This means that Nadam not only benefits from the adaptive learning rates and momentum of Adam, but also from the NAG component, a technique that helps the algorithm predict the next step more accurately. Especially in high-dimensional spaces, Nadam converges faster and more effectively.\n[AdaMax](https://keras.io/api/optimizers/adamax), instead, adopts a slightly different approach. While Adam calculates adaptive learning rates based on both the first and second moments of the gradients, AdaMax focuses only on the maximum norm of the gradients. AdaMax's simplicity and efficiency in handling sparse gradients make it an appealing choice for training deep neural networks, especially in tasks involving sparse data.\n## Optimization Algorithms Comparison\nIn order to practically test and visualize the performance of each discussed optimization algorithm, I trained a simple deep neural network with each one of the above optimizers. The job of the network is to classify fashion items displayed in 28x28 pixels images. The dataset is called [Fashion MNIST](https://github.com/zalandoresearch/fashion-mnist) (MIT licensed) and is composed of 70,000 small grey-scale images of clothing items.\nThe algorithm I used to test different optimizers is presented in the following snippet and, more completely in the GitHub repository linked below.\npython\n\n```\n# Load data\n(X_train, y_train), (X_test, y_test) = load_my_data()\n# Print an example\n#print_image(X_train, 42)\n\n# Normalize the inputs\nX_train = X_train / 255.\nX_test = X_test / 255.\n\n# Define the optimizers to test\nmy_optimizers = {\"Mini-batch GD\":tf.keras.optimizers.SGD(learning_rate = 0.001, momentum = 0.0),\n \"Momentum GD\":tf.keras.optimizers.SGD(learning_rate = 0.001, momentum = 0.9),\n \"RMS Prop\":tf.keras.optimizers.RMSprop(learning_rate = 0.001, rho = 0.9),\n \"Adam\":tf.keras.optimizers.Adam(learning_rate = 0.001, beta_1 = 0.9, beta_2 = 0.999)\n }\n\nhistories = {}\nfor optimizer_name, optimizer in my_optimizers.items():\n # Define a neural network\n\n my_network = tf.keras.models.Sequential([\n tf.keras.layers.Flatten(input_shape=(28, 28)), \n tf.keras.layers.Dense(128, activation='relu'),\n tf.keras.layers.Dense(64, activation='relu'),\n tf.keras.layers.Dense(10, activation='softmax')\n ])\n # Compile the model\n my_network.compile(optimizer=optimizer,\n loss='sparse_categorical_crossentropy', # since labels are more than 2 and not one-hot-encoded\n metrics=['accuracy'])\n\n # Train the model\n print('Training the model with optimizer {}'.format(optimizer_name))\n histories[optimizer_name] = my_network.fit(X_train, y_train, epochs=50, validation_split=0.1, verbose=1)\n\n# Plot learning curves\nfor optimizer_name, history in histories.items():\n loss = history.history['loss']\n epochs = range(1,len(loss)+1)\n plt.plot(epochs, loss, label=optimizer_name)\n plt.legend(loc=\"upper right\")\n plt.xlabel(\"Epoch\")\n plt.ylabel(\"Loss\")\n```\n\n> [**articles/NN-optimizer at main · andreoniriccardo/articles**](https://github.com/andreoniriccardo/articles/tree/main/NN-optimizer)\nThe resulting plot is visualized in this line chart:\n![Image by the author.](../images/007.png)\nImage by the author.\nWe can immediately see how the Momentum Gradient Descent (yellow line) is considerably faster than the standard Gradient Descent (blue line). Instead, RMS Prop (green line) seems to obtain similar results as Momentum Gradient Descent. This may be due to several reasons such as not perfectly tuned hyperparameters or a too-simple neural network. Finally, Adam optimizer (red line) looks superior, by a margin, to all the other methods.\n## Learning Rate Decay\nI want to include learning rate decay in this post because, even though it is not strictly an optimization algorithm, it is a powerful technique to **accelerate the network's learning process**.\n![Image source: unsplash.com.](../images/008.jpeg)\nImage source: [unsplash.com](https://unsplash.com/photos/OyCl7Y4y0Bk).\nLearning rate decay consists of the **reduction of the learning rate** hyperparameter _alpha_ over the epochs. Theis adjustment is essential because at the initial stages of the optimization, the algorithm can afford to take bigger steps but, as it approaches the minimum point, we prefer to make smaller steps so that it will bounce in an area closer to the minimum.\nThere exist several methods to decrease the learning rate over iterations. One approach is described by the following formula:\n![](../images/body-4a76ac26fcf99c649c21.png)\nwhere the decay rate is an additional parameter to tune.\nAlternatively, other possibilities are:\n![](../images/body-cfe05c431016182dde22.png)\n![](../images/body-5568855499862c28d2d2.png)\nThe first one is called Exponential Learning Rate Decay, and in the second one, parameter _k_ is a constant.\nFinally, it is also possible to apply a discrete learning rate decay, such as halving it after every _t_ iterations, or decreasing it manually.\n![Image by the author.](../images/body-4eeaade2d370bb628339.png)\nImage by the author.\n## Conclusion\nTime is a valuable, but also limited, resource for every data science practitioner. For this reason, mastering the tools to speed up a learning algorithm training can make a difference.\nIn this article, we have seen how the standard Gradient Descent optimizer is an outdated tool, and there exist several alternatives that provide better solutions in less time.\nChoosing the right optimizer for a given Machine Learning application is not always easy. It depends on the task and there is not a clear consensus yet on which one is the best. However, as we saw above, Adam optimizer with the proposed hyperparameters' values is a valid choice most of the time, and knowing the functioning principle of the most popular ones is an excellent starting point.\n* * *\nIf you liked this story, consider following me to be notified of my upcoming projects and articles!\nHere are some of my past projects:\n> [**Social Network Analysis with NetworkX: A Gentle Introduction**](https://towardsdatascience.com/social-network-analysis-with-networkx-a-gentle-introduction-6123eddced3)\n> [**Use Deep Learning to Generate Fantasy Names: Build a Language Model from Scratch**](https://towardsdatascience.com/use-deep-learning-to-generate-fantasy-character-names-build-a-language-model-from-scratch-792b13629efa)\n> [**Support Vector Machine with Scikit-Learn: A Friendly Introduction**](https://towardsdatascience.com/support-vector-machine-with-scikit-learn-a-friendly-introduction-a2969f2ff00d)\n## References\n * [Gradient descent - Wikipedia.org](https://en.wikipedia.org/wiki/Gradient_descent)\n * [Dive into Deep Learning - Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola](https://d2l.ai/)\n * [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification - Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun](https://arxiv.org/abs/1502.01852v1)\n * [Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization - Andrew Ng](https://www.coursera.org/learn/deep-neural-network)\n * [Understanding the difficulty of training deep feedforward neural networks - Xavier Glorot, Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n * [Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition - Aurélien Géron](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)\n * [Keras Python library](https://keras.io/)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-34347b17fbf8612c4a0c.jpeg)\nWritten By\n## [Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/)\n[See all from Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/)\n[Data Science](https://towardsdatascience.com/tag/data-science/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Optimization Algorithms](https://towardsdatascience.com/tag/optimization-algorithms/)[Tips And Tricks](https://towardsdatascience.com/tag/tips-and-tricks/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&title=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&title=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&text=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/009.jpeg)](https://towardsdatascience.com/logistic-regression-as-a-nonlinear-classifier-bdc6746db734/)\n## [Logistic Regression as a Nonlinear Classifier](https://towardsdatascience.com/logistic-regression-as-a-nonlinear-classifier-bdc6746db734/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLogistic Regression has traditionally been used as a linear classifier, i.e. when the classes can be separated in the feature space by...\n[Ashok Chilakapati](https://towardsdatascience.com/author/ashok.chilakapati/)\nMarch 13, 20198 min read\n * [![](../images/010.jpeg)](https://towardsdatascience.com/how-i-achieved-90-accuracy-on-a-text-classification-problem-with-zero-preprocessing-6acfa96e8d2e/)\n## [How I achieved 90% accuracy on a text classification problem with ZERO preprocessing](https://towardsdatascience.com/how-i-achieved-90-accuracy-on-a-text-classification-problem-with-zero-preprocessing-6acfa96e8d2e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIn a previous post, I demonstrated how different word embeddings (GloVe, ELMo, BERT) could be used for a text classification task. We saw...\n[Ryan Burke](https://towardsdatascience.com/author/ryancburke8/)\nMarch 27, 20213 min read\n * [![](../images/011.jpeg)](https://towardsdatascience.com/mlops-vs-devops-5c9a4d5a60ba/)\n## [MLOps vs DevOps](https://towardsdatascience.com/mlops-vs-devops-5c9a4d5a60ba/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThis article goes through the similarities and differences between DevOps and MLOps as well as platforms that help enable MLOps.\n[Marco Susilo](https://towardsdatascience.com/author/marco-susilo/)\nApril 20, 20213 min read\n * [![](../images/012.png)](https://towardsdatascience.com/taking-on-the-ml-pipeline-challenge-3dfeb7b24cc6/)\n## [Taking on the ML pipeline challenge](https://towardsdatascience.com/taking-on-the-ml-pipeline-challenge-3dfeb7b24cc6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWhy data scientists need to own their ML workflows in production.\n[Hamza Tahir](https://towardsdatascience.com/author/htahir1/)\nOctober 27, 20217 min read\n * [![](../images/013.jpeg)](https://towardsdatascience.com/highlights-from-data-ai-summit-2021-3abfd9aaccaa/)\n## [Highlights from DATA+AI Summit 2021 💥](https://towardsdatascience.com/highlights-from-data-ai-summit-2021-3abfd9aaccaa/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTakeaways from one of the biggest DATA/AI conference\n[mehdio](https://towardsdatascience.com/author/mehdio/)\nJune 27, 20215 min read\n * [![](../images/014.jpeg)](https://towardsdatascience.com/matplotlib-vs-ggplot2-which-to-choose-for-2020-and-beyond-ced5e294bfdc/)\n## [Matplotlib vs. ggplot2: Which to Choose for 2020 and Beyond?](https://towardsdatascience.com/matplotlib-vs-ggplot2-which-to-choose-for-2020-and-beyond-ced5e294bfdc/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIn-depth comparison of the two most popular visualization libraries\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nSeptember 29, 20205 min read\n * [![](../images/015.jpg)](https://towardsdatascience.com/debugging-made-easy-use-pytest-to-track-down-and-fix-python-code-ecbad62057b8/)\n## [Pytest Tutorial: An Introduction To Unit Testing](https://towardsdatascience.com/debugging-made-easy-use-pytest-to-track-down-and-fix-python-code-ecbad62057b8/)\n[Programming](https://towardsdatascience.com/category/programming/)\nHow to use Pytest fixtures and mock for unit testing\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nApril 18, 20236 min read\n * [![](../images/016.jpeg)](https://towardsdatascience.com/top-3-books-for-every-data-science-engineer-e1180ab041f1/)\n## [Top 3 Books for Every Data Science Engineer](https://towardsdatascience.com/top-3-books-for-every-data-science-engineer-e1180ab041f1/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nGreat resources I keep on my shelf and love to introduce to software engineers and data scientists.\n[Rose Day](https://towardsdatascience.com/author/rjday/)\nOctober 10, 20204 min read\n * [![](../images/017.jpeg)](https://towardsdatascience.com/aws-sagemaker-ais-next-game-changer-480d79e252a8/)\n## [AWS SageMaker: AI's Next Game Changer](https://towardsdatascience.com/aws-sagemaker-ais-next-game-changer-480d79e252a8/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nToday AWS SageMaker was released, and it is awesome. I have mentioned in previous articles that we do mostly AWS deployments for our...\n[Daniel Shapiro, PhD](https://towardsdatascience.com/author/lemaysolutions/)\nNovember 30, 20174 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "metadata": { "title": "The Best Optimization Algorithm for Your Neural Network | Towards Data Science", "description": "How to choose it and minimize your neural network training time.", "keywords": null, "author": "Riccardo Andreoni", "og:locale": "en_US", "og:type": "article", "og:title": "The Best Optimization Algorithm for Your Neural Network", "og:description": "How to choose it and minimize your neural network training time.", "og:url": "https://towardsdatascience.com/the-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1oGPTczEfBjtNFu4MnGbOLQ-scaled.jpeg", "og:image:width": "2560", "og:image:height": "1698", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Riccardo Andreoni", "twitter:label2": "Est. reading time", "twitter:data2": "12 min read", "article:published_time": "2023-10-14T04:06:42.000Z", "article:modified_time": "2023-10-14T04:06:42.000Z", "article:author": "https://towardsdatascience.com/author/riccardo.andreoni/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/5d522e7f2a42/code/block_000.txt", "chars": 1756 } ], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:55:02Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:55:02Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:55:02Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:55:02Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/riccardo.andreoni/" }, { "type": "internal", "href": "https://towardsdatascience.com/ai-ml-practicalities-the-cycle-of-experimentation-fd46fc1f3835" }, { "type": "internal", "href": "https://towardsdatascience.com/gradient-descent-with-momentum-59420f626c8f" }, { "type": "internal", "href": "https://towardsdatascience.com/social-network-analysis-with-networkx-a-gentle-introduction-6123eddced3" }, { "type": "internal", "href": "https://towardsdatascience.com/use-deep-learning-to-generate-fantasy-character-names-build-a-language-model-from-scratch-792b13629efa" }, { "type": "internal", "href": "https://towardsdatascience.com/support-vector-machine-with-scikit-learn-a-friendly-introduction-a2969f2ff00d" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/data-science/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/deep-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/optimization-algorithms/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/tips-and-tricks/" }, { "type": "internal", "href": "https://towardsdatascience.com/logistic-regression-as-a-nonlinear-classifier-bdc6746db734/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://unsplash.com/photos/FfbVFLAVscw" }, { "type": "external", "href": "https://github.com/andreoniriccardo/articles/tree/main/NN-optimizer" }, { "type": "external", "href": "https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/" }, { "type": "external", "href": "https://keras.io/api/optimizers/rmsprop" }, { "type": "external", "href": "https://keras.io/api/optimizers/adam" }, { "type": "external", "href": "https://keras.io/api/optimizers/Nadam" }, { "type": "external", "href": "https://keras.io/api/optimizers/adamax" }, { "type": "external", "href": "https://github.com/zalandoresearch/fashion-mnist" }, { "type": "external", "href": "https://unsplash.com/photos/OyCl7Y4y0Bk" }, { "type": "external", "href": "https://en.wikipedia.org/wiki/Gradient_descent" }, { "type": "external", "href": "https://d2l.ai/" }, { "type": "external", "href": "https://arxiv.org/abs/1502.01852v1" }, { "type": "external", "href": "https://www.coursera.org/learn/deep-neural-network" }, { "type": "external", "href": "http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" }, { "type": "external", "href": "https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/" }, { "type": "external", "href": "https://keras.io/" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=The+Best+Optimization+Algorithm+for+Your+Neural+Network&u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:28:31Z", "source_url": "https://towardsdatascience.com/the-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Data Science](https://towardsdatascience.com/category/data-science/)\n# The Best Optimization Algorithm for Your Neural Network\nHow to choose it and minimize your neural network training time.\n[Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/)\nOctober 14, 202312 min read\n![Image source: unsplash.com.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1oGPTczEfBjtNFu4MnGbOLQ-scaled.jpeg)\nImage source: [unsplash.com](https://unsplash.com/photos/FfbVFLAVscw).\nDeveloping any machine learning model involves a rigorous experimental process that follows the [**idea-experiment-evaluation cycle**](https://towardsdatascience.com/ai-ml-practicalities-the-cycle-of-experimentation-fd46fc1f3835).\n![Image by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1VWCNnabEsMOFvma_-Zrp0A.png)\nImage by the author.\nThe above cycle is repeated multiple times until satisfactory performance levels are achieved. The \"experiment\" phase involves both the coding and the training steps of the machine learning model. As **models become more complex** and are trained over much **larger datasets** , training time inevitably expands. As a consequence, training a large deep neural network can be painfully slow.\nFortunately for data science practitioners, there exist several techniques to accelerate the training process, including:\n * **Transfer Learning**.\n * **Weight Initialization** , as Glorot or He initialization.\n * **Batch Normalization** for training data.\n * Picking a **reliable activation function**.\n * Use a **faster optimizer**.\n\n\nWhile all the techniques I pointed out are important, in this post I will focus deeply on the last point. I will describe multiple algorithm for neural network parameters optimization, highlighting both their advantages and limitations.\nIn the last section of this post, I will present a visualization displaying the comparison between the discussed optimization algorithms.\nFor **practical implementation** , all the code used in this article can be accessed in this **GitHub repository** :\n> [**articles/NN-optimizer at main · andreoniriccardo/articles**](https://github.com/andreoniriccardo/articles/tree/main/NN-optimizer)\n## Batch Gradient Descent\nTraditonally, Batch Gradient Descent is considered the **default choice** for the optimizer method in neural networks.\nAfter the neural network generates predictions for the entire training set _X_ , we compare the network's predictions to the actual labels of each training point. This is done to compute a cost function _J(W,b)_ , which is a scalar representation of the capacity of the model to yield accurate predictions. The Gradient Descent optimization algorithm uses the cost function as a guide to tune every network's parameter. This iterative process continues until the cost function is close to zero, or it can't be reduced further. What GD does is to adjust each weight and bias of the network in a certain direction. The chosen direction is the one that reduces the cost function the most.\n![Image by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0YsMBBdnKmzDGMifZ.png)\nImage by the author.\nIn visual terms, take the image above for reference. Gradient descent starts at the leftmost blue point. By analyzing the gradient of the cost function around the starting point, it adjusts the parameter (x-axis value) to the right. This process is repeated multiple times until the algorithm ends up with a very good approximation of the optimum.\nIf the cost function has 2 input parameters, the function is not a line but instead, it creates a three-dimensional surface. The batch gradient descent steps can be displayed considering the level curves of the surface:\n![Image by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/10dU0bLWgz0OVKgx_Xg83UQ.png)\nImage by the author.\nA neural network consists of thousands or millions of parameters that affect the cost functions. While visualizing million-dimensional representations is infeasible, mathematical equations are useful to understand the gradient descent process.\nAt first, we compute the cost function _J(W,b)_ , which is a function of the network's weights _W_ and biases _b_. Then, the backpropagation algorithm computes the derivative of the cost function with respect to each weight and bias of each layer of the network:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0txSZoah_MVmmPz0X.png)\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0o41MTTN5yN0DPcMU.png)\nKnowing the direction towards which to adjust the parameters, we update them. The magnitude of each parameter's update is regulated by the gradient itself and by the learning rate alpha. Alpha is a hyperparameter of the optimization algorithm and its value is typically maintained constant.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0FEIidrHTv3JENcMk.png)\nBatch Gradient Descent offers several advantages:\n * **Simplicity** : It's a straightforward and easy-to-understand method.\n * **Hyperparameter Management** : It requires tuning only one hyperparameter, the learning rate.\n * **Optimality** : For convex cost functions, it reliably reaches the global optimum with a reasonable learning rate.\n\n\nThe disadvantages of gradient descent are:\n * **Computational Intensity** : It can be slow, particularly for large training sets, as it updates parameters after evaluating all examples.\n * **Local Optima and Saddle Points** : It might get stuck in local optima or saddle points, slowing convergence.\n * **Gradient Issues** : It is prone to vanishing gradient issues.\n * **Memory Usage** : It requires storing the entire training data in CPU/GPU memory.\n\n\n## Mini-Batch Gradient Descent\nIn batch gradient descent algorithm all training examples are used to calculate a single improvement step. The step will be the most accurate, as it considers all the available information. However, this approach often is impractically slow in real-world applications, suggesting the implementation of faster alternatives.\nThe most common solution is called [**Mini-Batch Gradient Descent**](https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/), and it uses only a **small subset of the training data** in order to update the weights' values.\nConsider the whole training set _X_ , and the associated labels _Y_ :\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0EolEpZFs_vW9_i4o.png)\nwhere _m_ represents the number of training examples.\nRather than feeding the whole batch to the optimization algorithm, we process only a small portion of the training set. Let's say we feed the algorithm the subsets _X^t_ and _Y^t_ , each containing 512 training examples:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0cKI9Y5sE9mcwzoY4.png)\nFor instance, if the total training set contains 5,120,000 points, it can be divided into 10,000 mini-batches, each containing 512 examples.\nFor each mini-batch, we perform the classical gradient descent operations:\n * **Compute the cost** relative to the mini-batch t, _J^t(W,b)_.\n * **Perform backpropagation** to compute the gradients of _J^t(W,b)_ with respect to each weight and bias.\n * **Update the parameters**.\n\n\nThe green line in the image below shows a typical optimization path of the mini-batch gradient descent. While batch gradient descent follows a more direct path to the optimal point, mini-batch gradient descent appears to take several unnecessary steps, due to its limited dataset at each iteration.\n![Image by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1fDHTg_RSZVRe0rrIiE-vjA.png)\nImage by the author.\nThe reason for that is that mini-batch gradient descent, at any time _t,_ has only a small portion of the training set to compute its decision on. With that limitation on the available information, it's clear that the route followed is not the most direct.\nHowever, the significant advantage of mini-batch gradient descent, is that each step is extremely fast to be computed since the algorithm needs to evaluate only a small portion of the data instead of the whole training set. In our example, each step requires the evaluation of only 512 data points instead of 5 million. This is also the reason why almost no real application, requiring a large amount of data, uses batch gradient descent.\nAs the image shows, with mini-batch gradient descent there is no guarantee that the cost at iteration _t+1_ is lower than the cost at iteration t, but, if the problem is well defined, the optimization algorithm reaches an area very close to the optimal point really quickly.\nThe choice of the mini-batch size represents an additional hyperparameter of the network's training process. A batch size equal to the total number of examples (_m_) corresponds to the batch gradient descent optimization. Instead, if _m=1_ we are performing Stochastic Gradient Descent, where each training example is a mini-batch.\nIf the training set is small, batch gradient descent can be a valid option, otherwise, popular mini-batch sizes like 32, 64, 128, 256, and 512 are commonly considered. For some reason, a batch size equal to a power of 2 seems to perform better.\nWhile stochastic gradient descent with a batch size of 1 is an option, it tends to be extremely noisy, and it can bounce further away from the minima. Moreover, stochastic gradient descent is very inefficient if we consider the computation time per example, as it cannot exploit the benefits of vectorization.\n## Gradient Descent with Momentum\nRecall that in both Batch and Mini-Batch Gradient Descent, the parameter update follows a defined formula:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0n3nDUYcUytPRxfHv.png)\nTherefore, in this equation, the size of each optimization step is determined by the learning rate, a fixed quantity, and the gradient, which is computed at a specific point of the cost function.\nWhen the gradient is computed in a nearly flat section of the cost function, it will be very small, leading to a proportionally small step of gradient descent. Consider the difference the gradient difference at points A and B of the image below.\n![Image by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0nCiFVj192QTmfW-K.png)\nImage by the author.\n[**Momentum Gradient Descent**](https://towardsdatascience.com/gradient-descent-with-momentum-59420f626c8f) overcomes this issue. We can imagine momentum gradient descent as a bowling ball rolling down a hill, which shape is defined by the cost function. If the ball begins its descent from a steep portion of the slope, its movement begins slowly but it will quickly gain velocity and momentum. Due to its momentum, the ball maintains great velocity even through a -flat area of the slope.\nThis is the core concept of momentum gradient descent: the algorithm will consider previous gradients, and not only the one computed at iteration _t_. Similar to the bowling ball analogy, the gradient computed at iteration _t_ defines the acceleration and not the speed of the ball.\nBoth weights and biases' velocities are computed at each step using the previous velocity and the current iteration's gradient:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0oSo6paA7i9xLi278.png)\nThe parameter _beta_ , known as momentum, regulates how much the new velocity value is determined from the current slope or from the past velocity value.\nFinally, parameters are updated using the computed velocity:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0eSK-UR7SvWrixq7F.png)\nMomentum Gradient Descent has proven superior performance compared to Mini-Batch Gradient Descent in most applications. The main drawback of Momentum Gradient Descent, with respect to standard Gradient Descent, is that it requires an additional parameter to tune. However, practice shows how a value of _beta_ equal to 0.9 works effectively.\n## RMS Prop\nConsider a cost function shaped similarly to an **elongated bowl** , where the minimum point is located in the narrowest part. The function's contour is described by the level curves in the image below.\n![Image by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/02fN3OjzLC3pHSmcw.png)\nImage by the author.\nIn cases where the starting point is far away from the minimum, Gradient Descent, also with the momentum variation, starts by following the steepest slope, which, in the picture above, is not the optimal path towards the minimum. The key idea of [**RMS Prop**](https://keras.io/api/optimizers/rmsprop) optimization algorithm is to correct the direction early and to aim for the global minimum more promptly.\nSimilarly to Momentum Gradient Descent, RMS Prop requires fine-tuning through an additional hyperparameter known as the decay rate. Through practical experience, it has been proven that setting the decay rate to 0.9 is a good choice for most problems.\n## Adam\n[**Adam**](https://keras.io/api/optimizers/adam) and its variations are probably the most employed optimization algorithms for neural networks. Adam, which stands for **Adaptive Moment Estimation** , is derived from the combinations of Momentum Gradient Descent and RMS Prop together.\nAs a mix of the two optimization methods, Adam needs two additional hyperparameters to be tuned (besides the learning rate _alpha_). We call them _beta _1_ and _beta _2_ and they are the hyperparameters used respectively in Momentum and in RMS Prop. As happened for the other discussed algorithms, effective default choices for _beta _1_ and _beta _2_ exist: _beta _1_ is usually set to 0.9 and _beta _2_ to 0.999. In Adam appears also a parameter _epsilon_ that works as a smoothing term and it is nearly always set to a small value like e-7 or e-8.\nAdam optimization algorithm outperforms all the above-mentioned methods in most cases. The only exceptions can be very simple problems where simpler methods work faster. The efficiency of Adam does come with a trade-off: the need to fine-tune two additional parameters. However, in my opinion it is a small price to pay for its efficiency.\n### Nadam and AdaMax\nIt is worth citing two algorithms born as modifications of the widely used Adam optimization: **Nadam** and **AdaMax**.\n[Nadam](https://keras.io/api/optimizers/Nadam), short of Nesterov-accelerated Adaptive Moment Estimation, enhances Adam by incorporating Nesterov Accelerated Gradient (NAG) into its framework. This means that Nadam not only benefits from the adaptive learning rates and momentum of Adam, but also from the NAG component, a technique that helps the algorithm predict the next step more accurately. Especially in high-dimensional spaces, Nadam converges faster and more effectively.\n[AdaMax](https://keras.io/api/optimizers/adamax), instead, adopts a slightly different approach. While Adam calculates adaptive learning rates based on both the first and second moments of the gradients, AdaMax focuses only on the maximum norm of the gradients. AdaMax's simplicity and efficiency in handling sparse gradients make it an appealing choice for training deep neural networks, especially in tasks involving sparse data.\n## Optimization Algorithms Comparison\nIn order to practically test and visualize the performance of each discussed optimization algorithm, I trained a simple deep neural network with each one of the above optimizers. The job of the network is to classify fashion items displayed in 28x28 pixels images. The dataset is called [Fashion MNIST](https://github.com/zalandoresearch/fashion-mnist) (MIT licensed) and is composed of 70,000 small grey-scale images of clothing items.\nThe algorithm I used to test different optimizers is presented in the following snippet and, more completely in the GitHub repository linked below.\npython\n\n```\n# Load data\n(X_train, y_train), (X_test, y_test) = load_my_data()\n# Print an example\n#print_image(X_train, 42)\n\n# Normalize the inputs\nX_train = X_train / 255.\nX_test = X_test / 255.\n\n# Define the optimizers to test\nmy_optimizers = {\"Mini-batch GD\":tf.keras.optimizers.SGD(learning_rate = 0.001, momentum = 0.0),\n \"Momentum GD\":tf.keras.optimizers.SGD(learning_rate = 0.001, momentum = 0.9),\n \"RMS Prop\":tf.keras.optimizers.RMSprop(learning_rate = 0.001, rho = 0.9),\n \"Adam\":tf.keras.optimizers.Adam(learning_rate = 0.001, beta_1 = 0.9, beta_2 = 0.999)\n }\n\nhistories = {}\nfor optimizer_name, optimizer in my_optimizers.items():\n # Define a neural network\n\n my_network = tf.keras.models.Sequential([\n tf.keras.layers.Flatten(input_shape=(28, 28)), \n tf.keras.layers.Dense(128, activation='relu'),\n tf.keras.layers.Dense(64, activation='relu'),\n tf.keras.layers.Dense(10, activation='softmax')\n ])\n # Compile the model\n my_network.compile(optimizer=optimizer,\n loss='sparse_categorical_crossentropy', # since labels are more than 2 and not one-hot-encoded\n metrics=['accuracy'])\n\n # Train the model\n print('Training the model with optimizer {}'.format(optimizer_name))\n histories[optimizer_name] = my_network.fit(X_train, y_train, epochs=50, validation_split=0.1, verbose=1)\n\n# Plot learning curves\nfor optimizer_name, history in histories.items():\n loss = history.history['loss']\n epochs = range(1,len(loss)+1)\n plt.plot(epochs, loss, label=optimizer_name)\n plt.legend(loc=\"upper right\")\n plt.xlabel(\"Epoch\")\n plt.ylabel(\"Loss\")\n```\n\n> [**articles/NN-optimizer at main · andreoniriccardo/articles**](https://github.com/andreoniriccardo/articles/tree/main/NN-optimizer)\nThe resulting plot is visualized in this line chart:\n![Image by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0Egbl0e6Fpv0ePGAn.png)\nImage by the author.\nWe can immediately see how the Momentum Gradient Descent (yellow line) is considerably faster than the standard Gradient Descent (blue line). Instead, RMS Prop (green line) seems to obtain similar results as Momentum Gradient Descent. This may be due to several reasons such as not perfectly tuned hyperparameters or a too-simple neural network. Finally, Adam optimizer (red line) looks superior, by a margin, to all the other methods.\n## Learning Rate Decay\nI want to include learning rate decay in this post because, even though it is not strictly an optimization algorithm, it is a powerful technique to **accelerate the network's learning process**.\n![Image source: unsplash.com.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1S2aP0eRCGydj7teghygVDw.jpeg)\nImage source: [unsplash.com](https://unsplash.com/photos/OyCl7Y4y0Bk).\nLearning rate decay consists of the **reduction of the learning rate** hyperparameter _alpha_ over the epochs. Theis adjustment is essential because at the initial stages of the optimization, the algorithm can afford to take bigger steps but, as it approaches the minimum point, we prefer to make smaller steps so that it will bounce in an area closer to the minimum.\nThere exist several methods to decrease the learning rate over iterations. One approach is described by the following formula:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/01bfhcNVm-7PIGFmq.png)\nwhere the decay rate is an additional parameter to tune.\nAlternatively, other possibilities are:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/08fmggT2rgeixQcdW.png)\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0rH_6ft-zur-vLBF_.png)\nThe first one is called Exponential Learning Rate Decay, and in the second one, parameter _k_ is a constant.\nFinally, it is also possible to apply a discrete learning rate decay, such as halving it after every _t_ iterations, or decreasing it manually.\n![Image by the author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/0ttUf-NQR6b1J3j47.png)\nImage by the author.\n## Conclusion\nTime is a valuable, but also limited, resource for every data science practitioner. For this reason, mastering the tools to speed up a learning algorithm training can make a difference.\nIn this article, we have seen how the standard Gradient Descent optimizer is an outdated tool, and there exist several alternatives that provide better solutions in less time.\nChoosing the right optimizer for a given Machine Learning application is not always easy. It depends on the task and there is not a clear consensus yet on which one is the best. However, as we saw above, Adam optimizer with the proposed hyperparameters' values is a valid choice most of the time, and knowing the functioning principle of the most popular ones is an excellent starting point.\n* * *\nIf you liked this story, consider following me to be notified of my upcoming projects and articles!\nHere are some of my past projects:\n> [**Social Network Analysis with NetworkX: A Gentle Introduction**](https://towardsdatascience.com/social-network-analysis-with-networkx-a-gentle-introduction-6123eddced3)\n> [**Use Deep Learning to Generate Fantasy Names: Build a Language Model from Scratch**](https://towardsdatascience.com/use-deep-learning-to-generate-fantasy-character-names-build-a-language-model-from-scratch-792b13629efa)\n> [**Support Vector Machine with Scikit-Learn: A Friendly Introduction**](https://towardsdatascience.com/support-vector-machine-with-scikit-learn-a-friendly-introduction-a2969f2ff00d)\n## References\n * [Gradient descent - Wikipedia.org](https://en.wikipedia.org/wiki/Gradient_descent)\n * [Dive into Deep Learning - Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola](https://d2l.ai/)\n * [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification - Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun](https://arxiv.org/abs/1502.01852v1)\n * [Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization - Andrew Ng](https://www.coursera.org/learn/deep-neural-network)\n * [Understanding the difficulty of training deep feedforward neural networks - Xavier Glorot, Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n * [Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition - Aurélien Géron](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/)\n * [Keras Python library](https://keras.io/)\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657095428_xaezyc.jpeg)\nWritten By\n## [Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/)\n[See all from Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/)\n[Data Science](https://towardsdatascience.com/tag/data-science/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Optimization Algorithms](https://towardsdatascience.com/tag/optimization-algorithms/)[Tips And Tricks](https://towardsdatascience.com/tag/tips-and-tricks/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&title=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&title=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&text=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/03/1ULgx5OUC-7NuMUfA7kAg0w.jpeg)](https://towardsdatascience.com/logistic-regression-as-a-nonlinear-classifier-bdc6746db734/)\n## [Logistic Regression as a Nonlinear Classifier](https://towardsdatascience.com/logistic-regression-as-a-nonlinear-classifier-bdc6746db734/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nLogistic Regression has traditionally been used as a linear classifier, i.e. when the classes can be separated in the feature space by...\n[Ashok Chilakapati](https://towardsdatascience.com/author/ashok.chilakapati/)\nMarch 13, 20198 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/1rSS5LYxOox9RFs20xLieOw-scaled.jpeg)](https://towardsdatascience.com/how-i-achieved-90-accuracy-on-a-text-classification-problem-with-zero-preprocessing-6acfa96e8d2e/)\n## [How I achieved 90% accuracy on a text classification problem with ZERO preprocessing](https://towardsdatascience.com/how-i-achieved-90-accuracy-on-a-text-classification-problem-with-zero-preprocessing-6acfa96e8d2e/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIn a previous post, I demonstrated how different word embeddings (GloVe, ELMo, BERT) could be used for a text classification task. We saw...\n[Ryan Burke](https://towardsdatascience.com/author/ryancburke8/)\nMarch 27, 20213 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1fVniO2FZsuP4wwfJXdpf5g-scaled.jpeg)](https://towardsdatascience.com/mlops-vs-devops-5c9a4d5a60ba/)\n## [MLOps vs DevOps](https://towardsdatascience.com/mlops-vs-devops-5c9a4d5a60ba/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nThis article goes through the similarities and differences between DevOps and MLOps as well as platforms that help enable MLOps.\n[Marco Susilo](https://towardsdatascience.com/author/marco-susilo/)\nApril 20, 20213 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1qWJBz5cI6X6asHL9oSldjQ.png)](https://towardsdatascience.com/taking-on-the-ml-pipeline-challenge-3dfeb7b24cc6/)\n## [Taking on the ML pipeline challenge](https://towardsdatascience.com/taking-on-the-ml-pipeline-challenge-3dfeb7b24cc6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nWhy data scientists need to own their ML workflows in production.\n[Hamza Tahir](https://towardsdatascience.com/author/htahir1/)\nOctober 27, 20217 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/06/1-oieOVGxUSgO-i6GXCr59w.jpeg)](https://towardsdatascience.com/highlights-from-data-ai-summit-2021-3abfd9aaccaa/)\n## [Highlights from DATA+AI Summit 2021 💥](https://towardsdatascience.com/highlights-from-data-ai-summit-2021-3abfd9aaccaa/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTakeaways from one of the biggest DATA/AI conference\n[mehdio](https://towardsdatascience.com/author/mehdio/)\nJune 27, 20215 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/1_7xF3K4vW8QHhMPCqybAtA-scaled.jpeg)](https://towardsdatascience.com/matplotlib-vs-ggplot2-which-to-choose-for-2020-and-beyond-ced5e294bfdc/)\n## [Matplotlib vs. ggplot2: Which to Choose for 2020 and Beyond?](https://towardsdatascience.com/matplotlib-vs-ggplot2-which-to-choose-for-2020-and-beyond-ced5e294bfdc/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nIn-depth comparison of the two most popular visualization libraries\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nSeptember 29, 20205 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/04/0eZ1Ga95SYAX9zCcX-scaled.jpg)](https://towardsdatascience.com/debugging-made-easy-use-pytest-to-track-down-and-fix-python-code-ecbad62057b8/)\n## [Pytest Tutorial: An Introduction To Unit Testing](https://towardsdatascience.com/debugging-made-easy-use-pytest-to-track-down-and-fix-python-code-ecbad62057b8/)\n[Programming](https://towardsdatascience.com/category/programming/)\nHow to use Pytest fixtures and mock for unit testing\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nApril 18, 20236 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/1M1CC8F4GU-IG5uV5KNPeJA-scaled.jpeg)](https://towardsdatascience.com/top-3-books-for-every-data-science-engineer-e1180ab041f1/)\n## [Top 3 Books for Every Data Science Engineer](https://towardsdatascience.com/top-3-books-for-every-data-science-engineer-e1180ab041f1/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nGreat resources I keep on my shelf and love to introduce to software engineers and data scientists.\n[Rose Day](https://towardsdatascience.com/author/rjday/)\nOctober 10, 20204 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2017/11/12bEiEPBk1SoVk1Pl0d_r-w.jpeg)](https://towardsdatascience.com/aws-sagemaker-ais-next-game-changer-480d79e252a8/)\n## [AWS SageMaker: AI's Next Game Changer](https://towardsdatascience.com/aws-sagemaker-ais-next-game-changer-480d79e252a8/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nToday AWS SageMaker was released, and it is awesome. I have mentioned in previous articles that we do mostly AWS deployments for our...\n[Daniel Shapiro, PhD](https://towardsdatascience.com/author/lemaysolutions/)\nNovember 30, 20174 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "body_base_path": "attachments/5d522e7f2a42/pages/", "body_markdown_path": "attachments/5d522e7f2a42/pages/archived-body.md", "local_html": "attachments/5d522e7f2a42/pages/archived-body.html", "body_source": "prior_capture" }