![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Data Science](https://towardsdatascience.com/category/data-science/) # The Best Optimization Algorithm for Your Neural Network How to choose it and minimize your neural network training time. [Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/) October 14, 202312 min read ![Image source: unsplash.com.](../images/000.jpeg) Image source: [unsplash.com](https://unsplash.com/photos/FfbVFLAVscw). Developing any machine learning model involves a rigorous experimental process that follows the [**idea-experiment-evaluation cycle**](https://towardsdatascience.com/ai-ml-practicalities-the-cycle-of-experimentation-fd46fc1f3835). ![Image by the author.](../images/001.png) Image by the author. The above cycle is repeated multiple times until satisfactory performance levels are achieved. The "experiment" phase involves both the coding and the training steps of the machine learning model. As **models become more complex** and are trained over much **larger datasets** , training time inevitably expands. As a consequence, training a large deep neural network can be painfully slow. Fortunately for data science practitioners, there exist several techniques to accelerate the training process, including: * **Transfer Learning**. * **Weight Initialization** , as Glorot or He initialization. * **Batch Normalization** for training data. * Picking a **reliable activation function**. * Use a **faster optimizer**. While all the techniques I pointed out are important, in this post I will focus deeply on the last point. I will describe multiple algorithm for neural network parameters optimization, highlighting both their advantages and limitations. In the last section of this post, I will present a visualization displaying the comparison between the discussed optimization algorithms. For **practical implementation** , all the code used in this article can be accessed in this **GitHub repository** : > [**articles/NN-optimizer at main · andreoniriccardo/articles**](https://github.com/andreoniriccardo/articles/tree/main/NN-optimizer) ## Batch Gradient Descent Traditonally, Batch Gradient Descent is considered the **default choice** for the optimizer method in neural networks. After the neural network generates predictions for the entire training set _X_ , we compare the network's predictions to the actual labels of each training point. This is done to compute a cost function _J(W,b)_ , which is a scalar representation of the capacity of the model to yield accurate predictions. The Gradient Descent optimization algorithm uses the cost function as a guide to tune every network's parameter. This iterative process continues until the cost function is close to zero, or it can't be reduced further. What GD does is to adjust each weight and bias of the network in a certain direction. The chosen direction is the one that reduces the cost function the most. ![Image by the author.](../images/002.png) Image by the author. In visual terms, take the image above for reference. Gradient descent starts at the leftmost blue point. By analyzing the gradient of the cost function around the starting point, it adjusts the parameter (x-axis value) to the right. This process is repeated multiple times until the algorithm ends up with a very good approximation of the optimum. If the cost function has 2 input parameters, the function is not a line but instead, it creates a three-dimensional surface. The batch gradient descent steps can be displayed considering the level curves of the surface: ![Image by the author.](../images/003.png) Image by the author. A neural network consists of thousands or millions of parameters that affect the cost functions. While visualizing million-dimensional representations is infeasible, mathematical equations are useful to understand the gradient descent process. At first, we compute the cost function _J(W,b)_ , which is a function of the network's weights _W_ and biases _b_. Then, the backpropagation algorithm computes the derivative of the cost function with respect to each weight and bias of each layer of the network: ![](../images/body-2a8314583a19366a2c78.png) ![](../images/body-1f7faf9556391d730954.png) Knowing the direction towards which to adjust the parameters, we update them. The magnitude of each parameter's update is regulated by the gradient itself and by the learning rate alpha. Alpha is a hyperparameter of the optimization algorithm and its value is typically maintained constant. ![](../images/body-a310acfe6e10feeffdde.png) Batch Gradient Descent offers several advantages: * **Simplicity** : It's a straightforward and easy-to-understand method. * **Hyperparameter Management** : It requires tuning only one hyperparameter, the learning rate. * **Optimality** : For convex cost functions, it reliably reaches the global optimum with a reasonable learning rate. The disadvantages of gradient descent are: * **Computational Intensity** : It can be slow, particularly for large training sets, as it updates parameters after evaluating all examples. * **Local Optima and Saddle Points** : It might get stuck in local optima or saddle points, slowing convergence. * **Gradient Issues** : It is prone to vanishing gradient issues. * **Memory Usage** : It requires storing the entire training data in CPU/GPU memory. ## Mini-Batch Gradient Descent In batch gradient descent algorithm all training examples are used to calculate a single improvement step. The step will be the most accurate, as it considers all the available information. However, this approach often is impractically slow in real-world applications, suggesting the implementation of faster alternatives. The most common solution is called [**Mini-Batch Gradient Descent**](https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/), and it uses only a **small subset of the training data** in order to update the weights' values. Consider the whole training set _X_ , and the associated labels _Y_ : ![](../images/body-e343b6b36936e4a37c27.png) where _m_ represents the number of training examples. Rather than feeding the whole batch to the optimization algorithm, we process only a small portion of the training set. Let's say we feed the algorithm the subsets _X^t_ and _Y^t_ , each containing 512 training examples: ![](../images/body-42654bd8713ba0bba8bf.png) For instance, if the total training set contains 5,120,000 points, it can be divided into 10,000 mini-batches, each containing 512 examples. For each mini-batch, we perform the classical gradient descent operations: * **Compute the cost** relative to the mini-batch t, _J^t(W,b)_. * **Perform backpropagation** to compute the gradients of _J^t(W,b)_ with respect to each weight and bias. * **Update the parameters**. The green line in the image below shows a typical optimization path of the mini-batch gradient descent. While batch gradient descent follows a more direct path to the optimal point, mini-batch gradient descent appears to take several unnecessary steps, due to its limited dataset at each iteration. ![Image by the author.](../images/004.png) Image by the author. The reason for that is that mini-batch gradient descent, at any time _t,_ has only a small portion of the training set to compute its decision on. With that limitation on the available information, it's clear that the route followed is not the most direct. However, the significant advantage of mini-batch gradient descent, is that each step is extremely fast to be computed since the algorithm needs to evaluate only a small portion of the data instead of the whole training set. In our example, each step requires the evaluation of only 512 data points instead of 5 million. This is also the reason why almost no real application, requiring a large amount of data, uses batch gradient descent. As the image shows, with mini-batch gradient descent there is no guarantee that the cost at iteration _t+1_ is lower than the cost at iteration t, but, if the problem is well defined, the optimization algorithm reaches an area very close to the optimal point really quickly. The choice of the mini-batch size represents an additional hyperparameter of the network's training process. A batch size equal to the total number of examples (_m_) corresponds to the batch gradient descent optimization. Instead, if _m=1_ we are performing Stochastic Gradient Descent, where each training example is a mini-batch. If the training set is small, batch gradient descent can be a valid option, otherwise, popular mini-batch sizes like 32, 64, 128, 256, and 512 are commonly considered. For some reason, a batch size equal to a power of 2 seems to perform better. While stochastic gradient descent with a batch size of 1 is an option, it tends to be extremely noisy, and it can bounce further away from the minima. Moreover, stochastic gradient descent is very inefficient if we consider the computation time per example, as it cannot exploit the benefits of vectorization. ## Gradient Descent with Momentum Recall that in both Batch and Mini-Batch Gradient Descent, the parameter update follows a defined formula: ![](../images/body-0c9942b779d6877bee52.png) Therefore, in this equation, the size of each optimization step is determined by the learning rate, a fixed quantity, and the gradient, which is computed at a specific point of the cost function. When the gradient is computed in a nearly flat section of the cost function, it will be very small, leading to a proportionally small step of gradient descent. Consider the difference the gradient difference at points A and B of the image below. ![Image by the author.](../images/005.png) Image by the author. [**Momentum Gradient Descent**](https://towardsdatascience.com/gradient-descent-with-momentum-59420f626c8f) overcomes this issue. We can imagine momentum gradient descent as a bowling ball rolling down a hill, which shape is defined by the cost function. If the ball begins its descent from a steep portion of the slope, its movement begins slowly but it will quickly gain velocity and momentum. Due to its momentum, the ball maintains great velocity even through a -flat area of the slope. This is the core concept of momentum gradient descent: the algorithm will consider previous gradients, and not only the one computed at iteration _t_. Similar to the bowling ball analogy, the gradient computed at iteration _t_ defines the acceleration and not the speed of the ball. Both weights and biases' velocities are computed at each step using the previous velocity and the current iteration's gradient: ![](../images/body-c672af39ae53d46db6b8.png) The parameter _beta_ , known as momentum, regulates how much the new velocity value is determined from the current slope or from the past velocity value. Finally, parameters are updated using the computed velocity: ![](../images/body-3eb099dfa0c975064737.png) Momentum Gradient Descent has proven superior performance compared to Mini-Batch Gradient Descent in most applications. The main drawback of Momentum Gradient Descent, with respect to standard Gradient Descent, is that it requires an additional parameter to tune. However, practice shows how a value of _beta_ equal to 0.9 works effectively. ## RMS Prop Consider a cost function shaped similarly to an **elongated bowl** , where the minimum point is located in the narrowest part. The function's contour is described by the level curves in the image below. ![Image by the author.](../images/006.png) Image by the author. In cases where the starting point is far away from the minimum, Gradient Descent, also with the momentum variation, starts by following the steepest slope, which, in the picture above, is not the optimal path towards the minimum. The key idea of [**RMS Prop**](https://keras.io/api/optimizers/rmsprop) optimization algorithm is to correct the direction early and to aim for the global minimum more promptly. Similarly to Momentum Gradient Descent, RMS Prop requires fine-tuning through an additional hyperparameter known as the decay rate. Through practical experience, it has been proven that setting the decay rate to 0.9 is a good choice for most problems. ## Adam [**Adam**](https://keras.io/api/optimizers/adam) and its variations are probably the most employed optimization algorithms for neural networks. Adam, which stands for **Adaptive Moment Estimation** , is derived from the combinations of Momentum Gradient Descent and RMS Prop together. As a mix of the two optimization methods, Adam needs two additional hyperparameters to be tuned (besides the learning rate _alpha_). We call them _beta _1_ and _beta _2_ and they are the hyperparameters used respectively in Momentum and in RMS Prop. As happened for the other discussed algorithms, effective default choices for _beta _1_ and _beta _2_ exist: _beta _1_ is usually set to 0.9 and _beta _2_ to 0.999. In Adam appears also a parameter _epsilon_ that works as a smoothing term and it is nearly always set to a small value like e-7 or e-8. Adam optimization algorithm outperforms all the above-mentioned methods in most cases. The only exceptions can be very simple problems where simpler methods work faster. The efficiency of Adam does come with a trade-off: the need to fine-tune two additional parameters. However, in my opinion it is a small price to pay for its efficiency. ### Nadam and AdaMax It is worth citing two algorithms born as modifications of the widely used Adam optimization: **Nadam** and **AdaMax**. [Nadam](https://keras.io/api/optimizers/Nadam), short of Nesterov-accelerated Adaptive Moment Estimation, enhances Adam by incorporating Nesterov Accelerated Gradient (NAG) into its framework. This means that Nadam not only benefits from the adaptive learning rates and momentum of Adam, but also from the NAG component, a technique that helps the algorithm predict the next step more accurately. Especially in high-dimensional spaces, Nadam converges faster and more effectively. [AdaMax](https://keras.io/api/optimizers/adamax), instead, adopts a slightly different approach. While Adam calculates adaptive learning rates based on both the first and second moments of the gradients, AdaMax focuses only on the maximum norm of the gradients. AdaMax's simplicity and efficiency in handling sparse gradients make it an appealing choice for training deep neural networks, especially in tasks involving sparse data. ## Optimization Algorithms Comparison In order to practically test and visualize the performance of each discussed optimization algorithm, I trained a simple deep neural network with each one of the above optimizers. The job of the network is to classify fashion items displayed in 28x28 pixels images. The dataset is called [Fashion MNIST](https://github.com/zalandoresearch/fashion-mnist) (MIT licensed) and is composed of 70,000 small grey-scale images of clothing items. The algorithm I used to test different optimizers is presented in the following snippet and, more completely in the GitHub repository linked below. python ``` # Load data (X_train, y_train), (X_test, y_test) = load_my_data() # Print an example #print_image(X_train, 42) # Normalize the inputs X_train = X_train / 255. X_test = X_test / 255. # Define the optimizers to test my_optimizers = {"Mini-batch GD":tf.keras.optimizers.SGD(learning_rate = 0.001, momentum = 0.0), "Momentum GD":tf.keras.optimizers.SGD(learning_rate = 0.001, momentum = 0.9), "RMS Prop":tf.keras.optimizers.RMSprop(learning_rate = 0.001, rho = 0.9), "Adam":tf.keras.optimizers.Adam(learning_rate = 0.001, beta_1 = 0.9, beta_2 = 0.999) } histories = {} for optimizer_name, optimizer in my_optimizers.items(): # Define a neural network my_network = tf.keras.models.Sequential([ tf.keras.layers.Flatten(input_shape=(28, 28)), tf.keras.layers.Dense(128, activation='relu'), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dense(10, activation='softmax') ]) # Compile the model my_network.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', # since labels are more than 2 and not one-hot-encoded metrics=['accuracy']) # Train the model print('Training the model with optimizer {}'.format(optimizer_name)) histories[optimizer_name] = my_network.fit(X_train, y_train, epochs=50, validation_split=0.1, verbose=1) # Plot learning curves for optimizer_name, history in histories.items(): loss = history.history['loss'] epochs = range(1,len(loss)+1) plt.plot(epochs, loss, label=optimizer_name) plt.legend(loc="upper right") plt.xlabel("Epoch") plt.ylabel("Loss") ``` > [**articles/NN-optimizer at main · andreoniriccardo/articles**](https://github.com/andreoniriccardo/articles/tree/main/NN-optimizer) The resulting plot is visualized in this line chart: ![Image by the author.](../images/007.png) Image by the author. We can immediately see how the Momentum Gradient Descent (yellow line) is considerably faster than the standard Gradient Descent (blue line). Instead, RMS Prop (green line) seems to obtain similar results as Momentum Gradient Descent. This may be due to several reasons such as not perfectly tuned hyperparameters or a too-simple neural network. Finally, Adam optimizer (red line) looks superior, by a margin, to all the other methods. ## Learning Rate Decay I want to include learning rate decay in this post because, even though it is not strictly an optimization algorithm, it is a powerful technique to **accelerate the network's learning process**. ![Image source: unsplash.com.](../images/008.jpeg) Image source: [unsplash.com](https://unsplash.com/photos/OyCl7Y4y0Bk). Learning rate decay consists of the **reduction of the learning rate** hyperparameter _alpha_ over the epochs. Theis adjustment is essential because at the initial stages of the optimization, the algorithm can afford to take bigger steps but, as it approaches the minimum point, we prefer to make smaller steps so that it will bounce in an area closer to the minimum. There exist several methods to decrease the learning rate over iterations. One approach is described by the following formula: ![](../images/body-4a76ac26fcf99c649c21.png) where the decay rate is an additional parameter to tune. Alternatively, other possibilities are: ![](../images/body-cfe05c431016182dde22.png) ![](../images/body-5568855499862c28d2d2.png) The first one is called Exponential Learning Rate Decay, and in the second one, parameter _k_ is a constant. Finally, it is also possible to apply a discrete learning rate decay, such as halving it after every _t_ iterations, or decreasing it manually. ![Image by the author.](../images/body-4eeaade2d370bb628339.png) Image by the author. ## Conclusion Time is a valuable, but also limited, resource for every data science practitioner. For this reason, mastering the tools to speed up a learning algorithm training can make a difference. In this article, we have seen how the standard Gradient Descent optimizer is an outdated tool, and there exist several alternatives that provide better solutions in less time. Choosing the right optimizer for a given Machine Learning application is not always easy. It depends on the task and there is not a clear consensus yet on which one is the best. However, as we saw above, Adam optimizer with the proposed hyperparameters' values is a valid choice most of the time, and knowing the functioning principle of the most popular ones is an excellent starting point. * * * If you liked this story, consider following me to be notified of my upcoming projects and articles! Here are some of my past projects: > [**Social Network Analysis with NetworkX: A Gentle Introduction**](https://towardsdatascience.com/social-network-analysis-with-networkx-a-gentle-introduction-6123eddced3) > [**Use Deep Learning to Generate Fantasy Names: Build a Language Model from Scratch**](https://towardsdatascience.com/use-deep-learning-to-generate-fantasy-character-names-build-a-language-model-from-scratch-792b13629efa) > [**Support Vector Machine with Scikit-Learn: A Friendly Introduction**](https://towardsdatascience.com/support-vector-machine-with-scikit-learn-a-friendly-introduction-a2969f2ff00d) ## References * [Gradient descent - Wikipedia.org](https://en.wikipedia.org/wiki/Gradient_descent) * [Dive into Deep Learning - Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola](https://d2l.ai/) * [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification - Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun](https://arxiv.org/abs/1502.01852v1) * [Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization - Andrew Ng](https://www.coursera.org/learn/deep-neural-network) * [Understanding the difficulty of training deep feedforward neural networks - Xavier Glorot, Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) * [Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition - Aurélien Géron](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/) * [Keras Python library](https://keras.io/) ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-34347b17fbf8612c4a0c.jpeg) Written By ## [Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/) [See all from Riccardo Andreoni](https://towardsdatascience.com/author/riccardo.andreoni/) [Data Science](https://towardsdatascience.com/tag/data-science/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Optimization Algorithms](https://towardsdatascience.com/tag/optimization-algorithms/)[Tips And Tricks](https://towardsdatascience.com/tag/tips-and-tricks/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&title=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&title=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-best-optimization-algorithm-for-your-neural-network-d16d87ef15cb%2F&text=The%20Best%20Optimization%20Algorithm%20for%20Your%20Neural%20Network) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/009.jpeg)](https://towardsdatascience.com/logistic-regression-as-a-nonlinear-classifier-bdc6746db734/) ## [Logistic Regression as a Nonlinear Classifier](https://towardsdatascience.com/logistic-regression-as-a-nonlinear-classifier-bdc6746db734/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Logistic Regression has traditionally been used as a linear classifier, i.e. when the classes can be separated in the feature space by... [Ashok Chilakapati](https://towardsdatascience.com/author/ashok.chilakapati/) March 13, 20198 min read * [![](../images/010.jpeg)](https://towardsdatascience.com/how-i-achieved-90-accuracy-on-a-text-classification-problem-with-zero-preprocessing-6acfa96e8d2e/) ## [How I achieved 90% accuracy on a text classification problem with ZERO preprocessing](https://towardsdatascience.com/how-i-achieved-90-accuracy-on-a-text-classification-problem-with-zero-preprocessing-6acfa96e8d2e/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) In a previous post, I demonstrated how different word embeddings (GloVe, ELMo, BERT) could be used for a text classification task. We saw... [Ryan Burke](https://towardsdatascience.com/author/ryancburke8/) March 27, 20213 min read * [![](../images/011.jpeg)](https://towardsdatascience.com/mlops-vs-devops-5c9a4d5a60ba/) ## [MLOps vs DevOps](https://towardsdatascience.com/mlops-vs-devops-5c9a4d5a60ba/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) This article goes through the similarities and differences between DevOps and MLOps as well as platforms that help enable MLOps. [Marco Susilo](https://towardsdatascience.com/author/marco-susilo/) April 20, 20213 min read * [![](../images/012.png)](https://towardsdatascience.com/taking-on-the-ml-pipeline-challenge-3dfeb7b24cc6/) ## [Taking on the ML pipeline challenge](https://towardsdatascience.com/taking-on-the-ml-pipeline-challenge-3dfeb7b24cc6/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Why data scientists need to own their ML workflows in production. [Hamza Tahir](https://towardsdatascience.com/author/htahir1/) October 27, 20217 min read * [![](../images/013.jpeg)](https://towardsdatascience.com/highlights-from-data-ai-summit-2021-3abfd9aaccaa/) ## [Highlights from DATA+AI Summit 2021 💥](https://towardsdatascience.com/highlights-from-data-ai-summit-2021-3abfd9aaccaa/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Takeaways from one of the biggest DATA/AI conference [mehdio](https://towardsdatascience.com/author/mehdio/) June 27, 20215 min read * [![](../images/014.jpeg)](https://towardsdatascience.com/matplotlib-vs-ggplot2-which-to-choose-for-2020-and-beyond-ced5e294bfdc/) ## [Matplotlib vs. ggplot2: Which to Choose for 2020 and Beyond?](https://towardsdatascience.com/matplotlib-vs-ggplot2-which-to-choose-for-2020-and-beyond-ced5e294bfdc/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) In-depth comparison of the two most popular visualization libraries [Dario Radečić](https://towardsdatascience.com/author/radecicdario/) September 29, 20205 min read * [![](../images/015.jpg)](https://towardsdatascience.com/debugging-made-easy-use-pytest-to-track-down-and-fix-python-code-ecbad62057b8/) ## [Pytest Tutorial: An Introduction To Unit Testing](https://towardsdatascience.com/debugging-made-easy-use-pytest-to-track-down-and-fix-python-code-ecbad62057b8/) [Programming](https://towardsdatascience.com/category/programming/) How to use Pytest fixtures and mock for unit testing [Egor Howell](https://towardsdatascience.com/author/egorhowell/) April 18, 20236 min read * [![](../images/016.jpeg)](https://towardsdatascience.com/top-3-books-for-every-data-science-engineer-e1180ab041f1/) ## [Top 3 Books for Every Data Science Engineer](https://towardsdatascience.com/top-3-books-for-every-data-science-engineer-e1180ab041f1/) [Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/) Great resources I keep on my shelf and love to introduce to software engineers and data scientists. [Rose Day](https://towardsdatascience.com/author/rjday/) October 10, 20204 min read * [![](../images/017.jpeg)](https://towardsdatascience.com/aws-sagemaker-ais-next-game-changer-480d79e252a8/) ## [AWS SageMaker: AI's Next Game Changer](https://towardsdatascience.com/aws-sagemaker-ais-next-game-changer-480d79e252a8/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Today AWS SageMaker was released, and it is awesome. I have mentioned in previous articles that we do mostly AWS deployments for our... [Daniel Shapiro, PhD](https://towardsdatascience.com/author/lemaysolutions/) November 30, 20174 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) © Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network) #### Recommended Articles Close * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ "How Relevance Models Foreshadowed Transformers for NLP") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ "Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ "The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ "Does Your Company Have a Data Strategy?")