保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Deep Learning

Why Batch Normalization Matters for Deep Learning

Discover the role of batch normalization in streamlining neural network training and improving model performance Niklas Lang November 25, 202411 min read Photo by Petra Reid on Unsplash Photo by Petra Reid on Unsplash Batch normalization has become a very important technique for training neural networks in recent years. It makes training much more efficient and stable, which is a crucial factor, especially for large and deep networks. It was originally introduced to solve the problem of internal covariance shift. This article will examine the problems involved in training neural networks and how batch normalization can solve them. We will describe the process in detail and show how batch normalization can be implemented in Python and integrated into existing models. We will also consider the advantages and disadvantages of this method to determine whether it makes sense to use it.

What Problems arise when training Deep Neural Networks?

When training a deep neural network, backpropagation occurs after each run. The prediction error runs through the network layer by layer from behind. During this process, the weights of the individual neurons are then changed so that the error is reduced as quickly as possible. This changes the weights assuming that all other layers remain the same. In practice, however, these conditions only apply to a limited extent, as all layers are changed quickly during backpropagation. The paper "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" describes this core problem in more detail. The problem with this fact is that the statistical key figures of the value distribution also change with each change in the weightings. This means that after each run, the weights in a layer have a new distribution with a different mean value and a new standard deviation. This problem is known as an internal covariance shift and leads to more difficult training, as the subsequent layers cannot optimally process the changed distribution. To still have successful training, lower learning rates must be used to minimize the fluctuations in the weight changes. However, this means that the model takes significantly longer to converge and training is slowed down overall. Learn this step by step with the interactive Machine Learning roadmap.

What is Normalization?

The normalization of data is a process that is often used in the preparation of data sets for machine learning. The aim is to bring the numerical values of different attributes onto a common scale. For example, you can divide all numerical values in a data series by the maximum value of the data series to obtain a new data series that lies in the range from 0 to 1. Suppose you want to train a model to learn different marketing activities and their effect on sales and the quantity sold. You could simply calculate the sum of the quantity sold and the turnover as the dependent variable. However, this can quickly lead to distorted results, for example, if you have a product series in which many products are sold, but these have a relatively low unit price. In a second series, it can be the other way around, i.e. the products are not sold as often, but have a high unit price. A marketing campaign that leads to 100,000 additional products being sold, for example, is rated worse in the product series with low unit prices than in the product series with high unit prices. Similar problems also arise in other fields, for example when looking at the private expenditure of individuals. For two different people, food expenditure of €200 can be very different when set concerning monthly income. Normalization therefore helps to bring the data onto a neutral and comparable basis.

What is Batch Normalization?

Batch normalization was introduced to mitigate the problem of internal covariance shifts. Batches are used for this purpose, i.e. certain subsets of the data set with a fixed size, which contain a random selection of the data set in a training run. The main idea is that the activations of each layer within a batch are normalized so that they have a constant mean value of 0 and a constant standard deviation of 1. This additional step reduces the shift of the activation distributions so that the model learns faster and converges better.

How does Batch Normalization work?

When training neural networks, the entire data set is divided into so-called batches. These contain a random selection of data of a certain size and are used for a training run. In most cases, a so-called batch size of 32 or 64 is used, i.e. there are 32 or 64 individual data points in a batch. The data that arrives in the input layer of the network is already normalized during regular data preprocessing. This means that all numerical values have been brought to a uniform scale and a common distribution and are therefore comparable. In most cases, the data then has a mean value of 0 and a standard deviation of 1. Single Layer Perceptron | Source: Author Single Layer Perceptron | Source: Author After the first layer, however, the values have already passed through the so-called activation function, which leads to a shift in the distribution and thus denormalizes the values again. An activation function is run through again with each layer in the network and the numbers are no longer normalized in the output layer. This process is also called "internal covariate shift " in the technical literature (Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift). For this reason, the values are normalized again before each activation function during batch normalization. The following steps are carried out for this purpose. 1. Calculation of the Mean Value and the Variance per Mini-Batch First, the mean value and the variance for the entire mini-batch are calculated for each activation in a specific layer. If 𝑥ᵢ stands for any activation value of a neuron for a data point i , then the mean μ __ and the variance σ² are calculated as follows: Here, m is the number of data points in a mini-batch. As these key figures are recalculated for each layer, this ensures that the batch normalization can react flexibly to the different distributions. 2. Normalization of the Activations In this step, the activations are normalized so that they have a mean value of 0 and a standard deviation of 1. This is done by subtracting the mean of the batch from each activation and dividing it by the standard deviation. To avoid dividing by 0, a small value is added to the variance. This then results in the following formula: Here x̂ᵢ is the normalized activation value for the data point i. 3 Scaling and Shifting the Normalized Activations After the normalization has taken place, a further step is carried out to give the model more flexibility. Two learnable parameters γ and β are added to scale and shift the normalized values. This allows the model to reverse the normalization if the original distribution of the data was reasonable. The final transformed activation parameters are then calculated as follows: Here, γ is the scaling parameter that determines how strongly the normalized values are scaled, and β is the shift parameter that can shift the values along the y-axis. The process of batch normalization ensures that the activations in the different layers of the neural network are optimally scaled and that the training is therefore stable. These calculations take place thousands or even millions of times during training, which is why it is very good that we can use simple functionalities in Python for the implementation and do not have to perform these calculations manually. In the next section, we will therefore take a closer look at how this procedure can be implemented in Python.

How can Batch Normalization be implemented in Python?

To implement batch normalization in Python, we can use the TensorFlow library, which already has a built-in function tf.keras.layers.BatchNormalization and can be built directly into a model. As an example, we take a simple Convolutional Neural Network, which we train on the MNIST dataset. We apply the batch normalization after each convolutional or dense layer to stabilize the activations: As you can see, batch normalization is simply inserted into the model after each activation layer. No hyperparameters need to be determined, as the function calculates the mean and variance automatically and then adjusts the learnable parameters automatically.

What are the Advantages and Disadvantages of Batch Normalization?

Batch normalization has many advantages, especially when training deep neural networks, as it speeds up training and makes it more stable. However, it is not always the optimal choice for every architecture. Therefore, it is important to know the advantages and disadvantages, which we will explain in more detail in this section. Advantages:

Overfitting Example | Source: Author Overfitting Example | Source: Author

Disadvantages:

How should the model be structured?

There are a few things to consider when building a model with a batch normalization layer. Among other things, the learning rate should be increased by including the normalization layer. Normalization makes the model more stable, which is why it can change the weightings more quickly and still converge. At the same time, the use of a dropout layer should be avoided. On the one hand, normalization already provides an additional degree of generalization, which is why the dropout layer may not even be necessary. On the other hand, it can even worsen the result, as noise is generated by the normalization and the simultaneous omission of neurons. Finally, it may be useful to vary the position of the batch normalization and to test both the normalization before and after the activation function. Depending on the structure of the model, this can lead to better results.

What Variants and Extensions of Batch Normalization are there?

For most neural network architectures, batch normalization has proven to be a useful tool for circumventing the internal covariance shift and achieving faster convergence. However, problems can also arise with recurrent neural networks or small batch sizes. For this reason, different variants of the model have been developed over the years to overcome these disadvantages. 1. Layer Normalization This method normalizes the activations within a layer and not within a batch. For each input example, the neurons in a layer are normalized separately and must be recalculated in each step. This property makes layer normalization particularly suitable for recurrent neural networks, as it is not only well suited for sequential data but also acts independently of the batch size. Due to these properties, it is often used in the field of natural language processing or for sequential tasks. 2. Group Normalization With this method, the activations are divided into different groups and then normalized within the groups. It is therefore particularly suitable for image processing, where only small batch sizes are possible because the image files often require a lot of storage space. Group Normalization strikes a balance between dependency on batch size and flexibility in handling different amounts of data. 3. Batch Renormalization This approach is an extension of Batch Normalization, which was developed to better handle small batch sizes. It introduces additional parameters that strengthen the estimation of the mean and variance, even if the batch statistics are unstable. This enables stable training with fast convergence even for small batch sizes. These variants enable the use of batch normalization in different application scenarios, so that the disadvantages can be overcome even with demanding models and stable training can be carried out.

This is what you should take with you

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Niklas Lang

See all from Niklas Lang Batch NormalizationDeep LearningDropout LayerNeural NetworksRecurrent Neural Network Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Evaluating Multi-label Classifiers

Deep Learning And how metrics change in different scenarios Aniruddha Karajgi November 1, 202112 min read

ML & Neuroscience: March 2022 must-reads

Deep Learning This month: PDE and deep learning to build up brain's 🧠 connectome🔗, brain-computer 💻 interface for hand prosthesis and finally a new... Stefano Bosisio April 19, 20228 min read

Are The New M1 Macbooks Any Good for Deep Learning? Let's Find Out

Deep Learning M1 Macs vs. Google Colab for basic deep learning tasks. Dario Radečić January 25, 20215 min read

YOLO V3 Explained

Artificial Intelligence In this post we'll discuss the YOLO detection network and its versions 1, 2 and especially 3. Uri Almog October 9, 202013 min read

Neural network from TENET exploiting time inversion

Machine Learning Modelling dynamical systems with recurrent neural networks Kirill Tsyganov April 3, 20216 min read

Exploring the Helium Network with Graph Theory

Deep Learning Using blockchain data to extract insights about network coverage and "suspicious" hotspots Evan Diewald October 6, 20218 min read

How to Become an AI Engineer Fast (Skills, Projects, Salary)

Artificial Intelligence Spoiler, it will take longer than 3 months Egor Howell March 29, 202610 min read

A Simple CI/CD Setup for ML Projects

Artificial Intelligence Apply best practices and learn to use GitHub Actions to build robust code Marcello Politi December 20, 20237 min read

Image Colorization using Convolutional Autoencoders

Deep Learning A case study of colorizing images coming from an old-school video game using Deep Learning in Python Eryk Lewinson April 17, 202016 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close