保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Deep Learning

CBAM Paper Walkthrough: The Double-Attention Mechanism

Understanding and implementing CBAM (Convolutional Block Attention Module) from scratch with PyTorch Muhammad Ardi Putra September 20, 202623 min read Photo by Waldemar Brandt on Unsplash

Introduction

In this article, I am going to review and implement the deep learning paper titled “ CBAM: Convolutional Block Attention Module ” by Woo et al. [1]. As the name suggests, this is essentially a block we can attach to a CNN-based model to enhance feature quality by performing an attention mechanism. Despite the name attention , it is completely different from the one in the ViT (Vision Transformer) architecture. Keep in mind that CBAM was first released in 2018, while ViT was only introduced in 2020. So, we can simply say that CBAM is the older approach to apply an attention mechanism to image data. Despite being older, we should not worry about its relevance since CBAM is a lot more lightweight as compared to ViT, which makes it suitable to be used as a starting point for deployment on low-power devices.

Better Than SENet

If we were to talk about the history, CBAM was actually proposed as the improvement of SENet (Squeeze-and-Excitation Network), which was introduced a year before CBAM. If you remember the SENet architecture, it essentially works by performing attention across the channel dimension. By doing so, the channels that seem unimportant would be weighted less than that of the more important ones. — I actually got a separate article talking more thoroughly about SENet, which you can access through the link given in reference number [2]. Instead of just assigning weights to each channel, CBAM also gives weights to every single pixel in the spatial dimension of the image. So, we can say that CBAM has two attention mechanisms, which the authors refer to as the CAM (Channel Attention Module) and the SAM (Spatial Attention Module). So, based solely on this theory, CBAM should perform better than SENet. Learn this step by step with the interactive Machine Learning roadmap. ···

CBAM Architecture

Let’s talk more specifically about the CBAM architecture which I display in Figure 1 below. As I’ve mentioned earlier, CBAM consists of CAM and SAM. These two sub-blocks are responsible for creating attention weights, which will then be applied to the original tensor by multiplication. The output tensor of this block (the one referred to as Refined Features) has the exact same dimension as the input (Input Feature), meaning that we can easily plug CBAM to any backbone model without needing to worry about altering the tensor shapes. Figure 1. The high-level view of the CBAM architecture [1].

Channel Attention Module (CAM)

Now let’s take a closer look at the channel attention module in Figure 2 below. This component is actually very similar to the SENet block, except that CAM also uses global maxpooling layer in addition to the global average-pooling layer. It is explained in the paper that the two operations capture different kind of information, allowing the tensor produced by CAM to be more informative as compared to that of the SENet block. Figure 2. The structure of the channel attention module [1]. Remember that the spatial dimension of the tensor collapses to 1×1 when we apply global pooling operation to it. This essentially means that the input tensor, which has the original shape of C × H × W , now becomes C ×1×1, allowing us to process it further easily with the MLP in the subsequent step. There are two linear layers in this MLP, where the first one is used to shrink the number of features according to the reduction ratio parameter, whereas the second one works by expanding the feature vector back to the original length (i.e., the number of channels C). These two linear layers in the MLP are together responsible to learn the importance of each channel. Also, keep in mind that this MLP is shared for the tensor produced by the maxpooling and the average-pooling operations, meaning that these two tensors will be processed by the exact same MLP. As these two tensors have been processed, we then combine them by element-wise summation and pass it through a sigmoid function. Since this function shrinks any number to the range of 0 to 1, we can now perceive the resulting tensor as the channel attention weight. The elements that correspond to the more important channels will have the value close to 1, indicating that these channels will be weighted more than the others. According to the paper, this kind of mechanism helps the model to understand what kind of features to attend. You can see the formal mathematical expression of the channel attention module in Figure 3 below, where F is an arbitrary intermediate tensor within a network. One thing you need to keep in mind here is that there should be a ReLU activation function placed between the two linear layers (i.e., W₀ and W₁) yet is somehow not written in this equation. Figure 3. The formal mathematical expression of the channel attention module [1].

Spatial Attention Module (SAM)

The spatial attention module is conceptually similar to the channel attention module. Take a look at the illustration of this sub-block in Figure 4 below. Figure 4. The structure of the spatial attention module [1]. What essentially differentiates SAM from CAM is the axis where the pooling operation is performed. Previously in CAM the pooling is done across the spatial dimension, allowing each channel to have a single value representing that channel. Meanwhile, here in SAM the pooling is done across the channel dimension for each spatial pixel location. Thus, every pixel now contains a single value that represents all channels at once. By doing so, the input tensor that initially has the shape of C × H × W will collapse to 1× H × W. Remember that since we use maximum and average-pooling operations, we will thus have two tensors of that size. These two tensors are then concatenated, forming a new tensor of shape 2× H × W. This tensor is then processed with a 7×7 convolution layer of a single kernel, which effectively combines the information from the two channels into one. So, at this point the tensor becomes 1× H × W again and is then forwarded to the sigmoid function. Similar to CAM, the tensor produced by this sigmoid acts as the spatial attention weight. By using this weight tensor, we can essentially let the model know where it should pay more attention to. Below is what the formal mathematical definition of the spatial attention module looks like. Figure 5. The mathematical definition of the spatial attention module [1].

Integrating CBAM to Any Backbone Model

Previously I mentioned that the output shape of CBAM is exactly the same as the input, allowing it to be integrated to any backbone model easily. In fact, the authors also provide an illustration regarding how we can do that, which I show you in Figure 6 below. In this example, they illustrate how to plug CBAM into a ResNet building block. Once we have successfully integrated them like this, we can just stack these blocks as usual. Later in the coding part I will demonstrate how to implement CBAM from scratch and how to plug it into a ResNeXt model. Figure 6. How to integrate CBAM into any backbone model [1]. ···

Experimental Results

The design of the CBAM architecture itself was not chosen arbitrarily. Instead, it was built based on empirical results on their ablation studies, in which they proved that their final model is indeed the most optimal one.

Ablation Study on the Channel Attention Module

The first ablation study they conducted was related to the pooling layers in the CAM. The results of this experimental set are shown in Figure 7. We can see in this table that the errors when we use either of the two poolings are significantly lower than the plain backbone ResNet50 model. This essentially indicates that the information extracted by the maximum-pooling and the average-pooling are both important. Furthermore, when we utilize both pooling mechanisms simultaneously, the top-1 error goes even lower to 22.80%, which I believe this proves that the two tensors contain information that are not only important but also complementary (i.e., completing each other). Theoretically speaking, the features produced by maxpooling and average-pooling should indeed be complementary since the former captures the most prominent pixel value within a channel while the latter extracts the general information of the channel. So, this is essentially the reason why in Figure 2 the authors ended up using both pooling operations. Figure 7. Ablation study on the use of average and maximum-pooling operations in the CAM [1].

Ablation Study on the Spatial Attention Module

Regarding the spatial attention module, it is explained in the paper that the authors also used varying configurations as displayed in Figure 8 below. You can see here that the configuration in the last row produces the best result, where it utilizes the two pooling operations followed by a convolution layer with 7×7 kernel. In the case of SAM, both the maximum and the average-pooling operations are technically replaceable by a 1×1 convolution (which will combine information across channel dimension with learnable parameters instead of using a “fixed” average and max operations), yet the classification performance appears to be suboptimal. Figure 8. Ablation study on the configuration of the SAM [1].

Ablation Study on the Placement of CAM and SAM

The last ablation study the authors conducted was related to how the CAM and SAM are arranged within the CBAM. It is shown in Figure 9 below that using sequential method, especially CAM followed by SAM, allows the model to perform best with the top-1 error of only 22.66%. You can see in the next row that they tried to swap the sequence of the two modules, but they found that the error increases instead. Furthermore, the classification performance was getting even worse when they tried to parallelize CAM and SAM, even though this approach is still better than the ResNet50 with SE module only. Figure 9. Ablation study on how the CAM and SAM are arranged [1].

Comparison with Other Models

In the subsequent experiment the authors compared the performance of a plain model, the model with SE module, and the model with CBAM on different backbones. You can see in Figure 10 below that the model that uses CBAM almost always performs better than the other two as highlighted in green. Moreover, in ResNeXt50, although the model with SE module is better than the same model with CBAM, the error gap is only 0.01%, which I think is negligible. I also found in this figure that the error of ResNet50 with CBAM is lower than the plain ResNet101 as highlighted in orange. Interestingly, it is seen here that the number of params and the GFLOPs of ResNet50 with CBAM are much smaller. These facts show that CBAM enables a shallower network to outperform the deeper one while significantly conserving computational resources, which is ideal for deployment on low-end devices. Figure 10. How CBAM performs on different backbone CNN models [1][3].

Attention Heatmap

In addition to the quantitative results explained above, the authors also used Grad-CAM to perform qualitative evaluation. If you’re not yet familiar with Grad-CAM, it is essentially a method we can use to find out the specific area that contributes more to the predicted class. Figure 11 below displays several examples of the attention heatmap produced using Grad-CAM, where the area highlighted in red indicates that it gives more contribution to the predicted class. Figure 11. Attention heatmap obtained by Grad-CAM [1]. The results are pretty interesting. Let’s now take a look at the Croquet ball class. With the plain ResNet50, it looks like the model focuses on both the ball and the person. As the SE module is applied (i.e., channel-wise attention only), the attention map becomes more refined toward the ball. And then, after we replace the SE module with CBAM, the model achieves an even sharper focus on the target object. A similar thing can also be observed in the other classes. In Eskimo dog and Snow leopard , for example, we can see that the model only pays attention to the eyes. If I were to say, this is basically not wrong as long as the predicted class is correct. However, if we were to predict something (as a human), it would make more sense to see the entire object whenever possible, right? And so, this is exactly what the attention modules do. You can see that when CBAM is used, the red area in the attention heatmap covers the entire face, indicating the model now take that facial region into consideration to make predictions. Additionally, it is also seen in the figure that by using CBAM we can make the model more confident when making predictions. Take a look at the School bus image in the above figure. You can see here that the bus is not centered at the middle of the image. This basically causes the plain ResNet50 to have a confidence score of only 0.07 in predicting the bus (which I believe this should have been misclassified). Meanwhile, SE module allows the model to correctly classify it with the confidence of 0.92, and then CBAM improves it even further to 0.98. ···

CBAM Implementation

As we have understood all the theories behind CBAM, let’s now roll our sleeves and get our hands dirty with some code! As I’ve mentioned earlier, here I am going to implement CBAM and try to integrate it into the ResNeXt backbone. Although I am implementing both from scratch, I’ll focus the discussion primarily on the CBAM module. So if you’re not yet familiar with ResNeXt, I do encourage you read my previous article about that model beforehand, which you can access through the link at reference [4]. As usual, the very first thing we need to do is to import the required modules, i.e., the base torch module and its nn submodule. python

# Codeblock 1
import torch
import torch.nn as nn

Next, in Codeblock 2 below I initialize the configurable variables. The reduction ratio R is used to adjust the width of the MLP layer inside the CAM. I set the value for this to 16 as suggested in the paper. Meanwhile, CARDINALITY, NUM_CHANNELS, and NUM_BLOCKS are the ones belong to ResNeXt. python

# Codeblock 2
R            = 16
CARDINALITY  = 32
NUM_CHANNELS = [3, 64, 256, 512, 1024, 2048]
NUM_BLOCKS   = [3, 4, 6, 3]
NUM_CLASSES  = 1000

···

CAM Implementation

Let’s start with the CAM first. If you go back to Figure 2, you can see that we have two pooling operations. In Codeblock 3 below, the two layers that correspond to them are initialized at lines #(1) and #(2). Don’t forget to set the output_size parameter to (1,1) since we want each channel to be represented as a single number. What we do next inside the __init__() method is initializing the MLP that consists of two linear layers. The first linear layer is responsible to reduce the number of features according to the R parameter (#(3)), whereas the second one is used to expand it back to the original number of features (#(5)). Also, don’t forget to place the ReLU activation function in between (#(4)). In fact, the structure of this MLP layer is exactly the same as the one used in SENet. — You can read more about the underlying idea behind this structure in my previous article about that module at reference [2]. — The last thing we do inside the __init__() method is to initialize the sigmoid activation function (#(6)), which is responsible to rescale the tensor such that the values will always be between 0 and 1, suitable to be used as an attention weight. python

# Codeblock 3
class CAM(nn.Module):
    def __init__(self, num_channels, r=16):
        super().__init__()
        
        self.maxpool = nn.AdaptiveMaxPool2d(output_size=(1,1))  #(1)
        self.avgpool = nn.AdaptiveAvgPool2d(output_size=(1,1))  #(2)
        
        self.mlp = nn.Sequential(
            nn.Linear(in_features=num_channels,
                      out_features=num_channels//r,   #(3)
                      bias=False),
            
            nn.ReLU(inplace=True),                    #(4)
            
            nn.Linear(in_features=num_channels//r,    #(5)
                      out_features=num_channels, 
                      bias=False)
        )
        
        self.sigmoid = nn.Sigmoid()                   #(6)
        
    def forward(self, x):         #(7)
        original = x
        print(f'original\t\t: {x.size()}\n')
        
        
        x_max = self.maxpool(x)   #(8)
        print(f'x after maxpool (x_max)\t: {x_max.size()}')
        
        x_avg = self.avgpool(x)   #(9)
        print(f'x after avgpool (x_avg)\t: {x_avg.size()}\n')
        
        
        x_max = torch.flatten(x_max, start_dim=1)    #(10)
        print(f'x_max after flatten\t: {x_max.size()}')
        
        x_avg = torch.flatten(x_avg, start_dim=1)    #(11)
        print(f'x_avg after flatten\t: {x_avg.size()}\n')
        
        
        x_max = self.mlp(x_max)   #(12)
        print(f'x_max after mlp\t\t: {x_max.size()}')
        
        x_avg = self.mlp(x_avg)   #(13)
        print(f'x_avg after mlp\t\t: {x_avg.size()}\n')
        
        
        x = x_max + x_avg         #(14)
        print(f'after sum\t\t: {x.size()}')
        
        x = self.sigmoid(x)       #(15)
        print(f'after sigmoid\t\t: {x.size()}')
        
        x = x[:, :, None, None]   #(16)
        print(f'after reshape\t\t: {x.size()}')
        
        x = x * original          #(17)           
        print(f'after multiply\t\t: {x.size()}')
        
        return x

Now let’s move on to the forward() method where the execution would happen. You can see at line #(7) in the above codeblock that we take a single tensor x as the input. This input tensor will then be saved in the original variable, which is done because we will later multiply it with the resulting channel attention weight tensor (#(17)). The x tensor itself will be processed by maxpooling and average-pooling operations in parallel (#(8–9)). Both x_max and x_avg are forwarded to the same MLP, which is the reason why this MLP is called “shared” (#(12–13)). Then at line #(14), we combine x_max and x_avg through element-wise summation before forwarding the resulting tensor to the sigmoid function (#(15)). There is a little technical thing we do at line #(16), which is used to reintroduce the spatial dimension we previously dropped at lines #(10) and #(11). Finally, as the weight tensor is ready, we can then actually weight the original tensor by multiplying them (#(17)). At this point we already got our CAM class completed. What we are going to do next is to test it with the following code. Here I initialize a CAM instance that accepts a 512-channel image and pass a dummy tensor of size 512×28×28 through it, simulating an arbitrary intermediate tensor within a network. python

# Codeblock 4
cam = CAM(num_channels=512, r=16)
x = torch.randn(1, 512, 28, 28)

out = cam(x)

If you run the above code, you should get the following output. Notice that previously in Codeblock 3 I wrote lots of print functions, which is the reason why here you can clearly see the detailed flow of the network. http

# Codeblock 4 Output
original                : torch.Size([1, 512, 28, 28])

x after maxpool (x_max) : torch.Size([1, 512, 1, 1])
x after avgpool (x_avg) : torch.Size([1, 512, 1, 1])

x_max after flatten     : torch.Size([1, 512])
x_avg after flatten     : torch.Size([1, 512])

x_max after mlp         : torch.Size([1, 512])    #(1)
x_avg after mlp         : torch.Size([1, 512])    #(2)

after sum               : torch.Size([1, 512])
after sigmoid           : torch.Size([1, 512])
after reshape           : torch.Size([1, 512, 1, 1])    #(3)
after multiply          : torch.Size([1, 512, 28, 28])  #(4)

It is necessary to understand that although the MLP looks like it doesn’t change the tensor dimension at all (#(1–2)), you need to know that the feature vector length is internally reduced to 32 by the first linear layer before eventually expanded back to 512 by the second one. Next, it might also be worth noting that the channel attention weight tensor originally has the shape of 512×1×1 (#(3)), indicating that every single channel in the original tensor has a single weighting number associated with it. This attention weight is then applied to the original tensor by using a simple multiplication, which technically speaking, this weight tensor is broadcasted along the spatial dimension of the original tensor (#(4)). At this point our tensor is now ready to be forwarded to the SAM, which we are going to build very soon. ···

SAM Implementation

The implementation of the spatial attention module is displayed in Codeblock 5 below. What we need to initialize inside the __init__() method is only a single 7×7 convolution layer (#(1)) and a sigmoid activation function (#(2)) python

# Codeblock 5
class SAM(nn.Module):
    def __init__(self):
        super().__init__()
        
        self.conv = nn.Conv2d(in_channels=2,     #(1)
                              out_channels=1, 
                              kernel_size=7, 
                              padding=3, 
                              bias=False)
        self.sigmoid = nn.Sigmoid()              #(2)
    
    def forward(self, x):
        original = x      #(3)
        print(f'original\t\t: {x.size()}\n')
        
        
        x_max, _ = torch.max(x,  dim=1, keepdim=True)    #(4)
        print(f'x after maxpool (x_max)\t: {x_max.size()}')
        
        x_avg    = torch.mean(x, dim=1, keepdim=True)    #(5)
        print(f'x after avgpool (x_avg)\t: {x_avg.size()}\n')
        
        
        x = torch.cat([x_max, x_avg], dim=1)             #(6)
        print(f'after concatenate\t: {x.size()}')
        
        x = self.conv(x)                                 #(7)
        print(f'after conv\t\t: {x.size()}')
        
        x = self.sigmoid(x)                              #(8)
        print(f'after sigmoid\t\t: {x.size()}')
        
        x = x * original                                 #(9)
        print(f'after multiply\t\t: {x.size()}')
        
        return x

In the forward() method, the first thing we do is to store the original input into a separate variable (#(3)), which is exactly the same as what we did in the CAM. The pooling mechanism in the SAM is a bit unique since here we want to do that across the channel dimension. This is essentially the reason that I didn’t initialize any pooling layers in the __init__() method since nn.AdaptiveMaxPool2d() and nn.AdaptiveAvgPool2d() operate on spatial dimension, which is irrelevant for this case. Instead, here we use a simple torch.max() and torch.mean() functions to do the maxpooling (#(4)) and average-pooling (#(5)) operations, respectively. Just don’t forget to set the dim parameter to 1 so that they really do the operations across the channel dimension. Despite taking different values, the tensor shape produced by the two poolings are exactly the same, which is the reason that we can just concatenate them as shown at line #(6). Keep in mind that tensor concatenation does not really combine information as it only stacks the two without blending the numbers. Thus, in the subsequent step we apply the convolution layer we initialized earlier to actually do that (#(7)). This convolution only consists of a single kernel, which implies that the resulting tensor will have a single channel as well. This idea is conceptually different from the one in the CAM, where in that module we combine the information by element-wise summation. Technically speaking, we can essentially use summation for the SAM too as it will produce the exact same tensor dimension. However, I do believe that the authors might also intended to capture the correlation between neighboring pixels instead of independently giving weight to each pixel, which is the reason why they decided to use convolution over summation. As the two tensors have been combined, the next thing we do is to pass the resulting tensor through the sigmoid activation function to actually obtain the spatial attention weight (#(8)). And finally, we will multiply this weight tensor with the original SAM input as shown at line #(9). Now let’s run the Codeblock 6 below to test if our spatial attention module works properly. python

# Codeblock 6
sam = SAM()
x = torch.randn(1, 512, 28, 28)

out = sam(x)

And below is what the flow of the SAM looks like. We can see here that as the pooling operations are applied to the input tensor, the channel dimension collapses to 1 (#(1–2)). This essentially means that every pixel is now represented as a single number aggregated from all channels in that spatial location. Then at line #(3), the tensor becomes 2×28×28 as we concatenate the two before eventually reducing it again to 1×28×28 using the convolution layer (#(4)). After being processed by the sigmoid function, the resulting spatial attention weight is then multiplied with the original tensor, in which the former is broadcasted along the channel dimension of the latter, allowing the final output tensor to have the exact same shape as the input (#(5)). http

# Codeblock 6 Output
original                : torch.Size([1, 512, 28, 28])

x after maxpool (x_max) : torch.Size([1, 1, 28, 28])    #(1)
x after avgpool (x_avg) : torch.Size([1, 1, 28, 28])    #(2)

after concatenate       : torch.Size([1, 2, 28, 28])    #(3)
after conv              : torch.Size([1, 1, 28, 28])    #(4)
after sigmoid           : torch.Size([1, 1, 28, 28])
after multiply          : torch.Size([1, 512, 28, 28])  #(5)

···

The Complete CBAM Block

Now as the CAM and SAM are done, we will now put them together in the CBAM class. See the details in Codeblock 7 below. You can see here that this class is very simple as what we need to do is just to initialize the two attention modules and place them sequentially. Keep in mind that we need to pass the num_channels parameter every time we want to initialize a CBAM instance (#(1)) since we will later integrate this module into ResNeXt, in which every single one of its building blocks accepts different number of channels, and so we need to make this CBAM block flexible as well. python

# Codeblock 7
class CBAM(nn.Module):
    def __init__(self, num_channels):    #(1)
        super().__init__()
        
        self.cam = CAM(num_channels=num_channels)
        self.sam = SAM()
        
    def forward(self, x):
        print(f'original\t\t: {x.size()}')
        
        x = self.cam(x)
        print(f'after cam\t\t: {x.size()}')
        
        x = self.sam(x)
        print(f'after sam\t\t: {x.size()}')
        
        return x

Again, just to ensure that this class works properly, let’s pass a dummy tensor through it using the Codeblock 8 below. python

# Codeblock 8
cbam = CBAM(num_channels=512)
x = torch.randn(1, 512, 28, 28)

out = cbam(x)

http

# Codeblock 8 Output
original   : torch.Size([1, 512, 28, 28])
after cam  : torch.Size([1, 512, 28, 28])
after sam  : torch.Size([1, 512, 28, 28])

···

Implementing CBAM on ResNeXt Building Block

Alright, so at this point our CBAM is ready and in this section I am going to actually demonstrate how we can attach this module to a ResNeXt building block. The code I write in Codeblock 9 onwards are basically the same as the one used when I demonstrated how to integrate SENet on ResNeXt. I do encourage you to read that article [2] and my explanation on the pure ResNeXt backbone [4] because it would be too long if I explain everything here. The only thing I want to emphasize in Codeblock 9 is that the CBAM module itself is initialized at line #(1) which is then attached to the flow at line #(2). python

# Codeblock 9
class Block(nn.Module):
    def __init__(self, 
                 in_channels,
                 add_channel=False,
                 channel_multiplier=2,
                 downsample=False):
        super().__init__()

        self.add_channel = add_channel
        self.channel_multiplier = channel_multiplier
        self.downsample = downsample
        
        
        if self.add_channel:
            out_channels = in_channels*self.channel_multiplier
        else:
            out_channels = in_channels
        
        mid_channels = out_channels//2
        
        
        if self.downsample:
            stride = 2
        else:
            stride = 1


        if self.add_channel or self.downsample:
            self.projection = nn.Conv2d(in_channels=in_channels,
                                        out_channels=out_channels, 
                                        kernel_size=1, 
                                        stride=stride, 
                                        padding=0, 
                                        bias=False)
            nn.init.kaiming_normal_(self.projection.weight, nonlinearity='relu')
            self.bn_proj = nn.BatchNorm2d(num_features=out_channels)

        self.conv0 = nn.Conv2d(in_channels=in_channels,
                               out_channels=mid_channels,
                               kernel_size=1, 
                               stride=1, 
                               padding=0, 
                               bias=False)
        nn.init.kaiming_normal_(self.conv0.weight, nonlinearity='relu')
        self.bn0 = nn.BatchNorm2d(num_features=mid_channels)

        self.conv1 = nn.Conv2d(in_channels=mid_channels,
                               out_channels=mid_channels, 
                               kernel_size=3, 
                               stride=stride,
                               padding=1, 
                               bias=False, 
                               groups=CARDINALITY)
        nn.init.kaiming_normal_(self.conv1.weight, nonlinearity='relu')
        self.bn1 = nn.BatchNorm2d(num_features=mid_channels)

        self.conv2 = nn.Conv2d(in_channels=mid_channels,
                               out_channels=out_channels,
                               kernel_size=1, 
                               stride=1, 
                               padding=0, 
                               bias=False)
        nn.init.kaiming_normal_(self.conv2.weight, nonlinearity='relu')
        self.bn2 = nn.BatchNorm2d(num_features=out_channels)
        
        self.relu = nn.ReLU()
        
        self.cbam = CBAM(num_channels=out_channels)               #(1)
        
    def forward(self, x):
        print(f'original\t\t: {x.size()}')
        
        if self.add_channel or self.downsample:
            residual = self.bn_proj(self.projection(x))
            print(f'after projection\t: {residual.size()}')
        else:
            residual = x
            print(f'no projection\t\t: {residual.size()}')
        
        x = self.conv0(x)
        x = self.bn0(x)
        x = self.relu(x)
        print(f'after conv0-bn0-relu\t: {x.size()}')

        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        print(f'after conv1-bn1-relu\t: {x.size()}')
        
        x = self.conv2(x)
        x = self.bn2(x)
        print(f'after conv2-bn2\t\t: {x.size()}')
        
        x = self.cbam(x)                                          #(2)
        print(f'after cbam\t\t: {x.size()}')
        
        x = x + residual
        x = self.relu(x)
        print(f'after summation\t\t: {x.size()}')
        
        return x

And now we can test the Block class above by running the Codeblock 10 below. You can see in the following output that the tensor successfully passes through the entire network, including the CBAM block we attached at the end of the main flow (#(1)). python

# Codeblock 10
block = Block(in_channels=512, add_channel=False, downsample=False)
x = torch.randn(1, 512, 28, 28)

out = block(x)

http

# Codeblock 10 Output
original             : torch.Size([1, 512, 28, 28])
no projection        : torch.Size([1, 512, 28, 28])
after conv0-bn0-relu : torch.Size([1, 256, 28, 28])
after conv1-bn1-relu : torch.Size([1, 256, 28, 28])
after conv2-bn2      : torch.Size([1, 512, 28, 28])
after cbam           : torch.Size([1, 512, 28, 28])    #(1)
after summation      : torch.Size([1, 512, 28, 28])

···

The Final CBAM-ized ResNeXt

As the CBAM module has been attached to the main ResNeXt building block, we can just stack these blocks according to the structure given in the ResNeXt paper. The CBAMResNeXt class in Codeblock 11 is literally copy-pasted from my SENet article [2] since the only thing we need to do to attach CBAM module to ResNeXt is modifying the Block class back in Codeblock 9. python

# Codeblock 11
class CBAMResNeXt(nn.Module):
    def __init__(self):
        super().__init__()

        # conv1 stage
        self.resnext_conv1 = nn.Conv2d(in_channels=NUM_CHANNELS[0],
                                       out_channels=NUM_CHANNELS[1],
                                       kernel_size=7,
                                       stride=2,
                                       padding=3, 
                                       bias=False)
        nn.init.kaiming_normal_(self.resnext_conv1.weight, 
                                nonlinearity='relu')
        self.resnext_bn1 = nn.BatchNorm2d(num_features=NUM_CHANNELS[1])
        self.relu = nn.ReLU()
        self.resnext_maxpool1 = nn.MaxPool2d(kernel_size=3,
                                             stride=2, 
                                             padding=1)

        # conv2 stage
        self.resnext_conv2 = nn.ModuleList([
            Block(in_channels=NUM_CHANNELS[1],
                  add_channel=True,
                  channel_multiplier=4,
                  downsample=False)
        ])
        for _ in range(NUM_BLOCKS[0]-1):
            self.resnext_conv2.append(Block(in_channels=NUM_CHANNELS[2]))

        # conv3 stage
        self.resnext_conv3 = nn.ModuleList([Block(in_channels=NUM_CHANNELS[2],
                                                  add_channel=True, 
                                                  downsample=True)])
        for _ in range(NUM_BLOCKS[1]-1):
            self.resnext_conv3.append(Block(in_channels=NUM_CHANNELS[3]))
            
            
        # conv4 stage
        self.resnext_conv4 = nn.ModuleList([Block(in_channels=NUM_CHANNELS[3],
                                                  add_channel=True, 
                                                  downsample=True)])
        
        for _ in range(NUM_BLOCKS[2]-1):
            self.resnext_conv4.append(Block(in_channels=NUM_CHANNELS[4]))
            
            
        # conv5 stage
        self.resnext_conv5 = nn.ModuleList([Block(in_channels=NUM_CHANNELS[4],
                                                  add_channel=True, 
                                                  downsample=True)])
        
        for _ in range(NUM_BLOCKS[3]-1):
            self.resnext_conv5.append(Block(in_channels=NUM_CHANNELS[5]))
 
       
        self.avgpool = nn.AdaptiveAvgPool2d(output_size=(1,1))

        self.fc = nn.Linear(in_features=NUM_CHANNELS[5],
                            out_features=NUM_CLASSES)

    def forward(self, x):
        print(f'original\t\t: {x.size()}')
        
        x = self.relu(self.resnext_bn1(self.resnext_conv1(x)))
        print(f'after resnext_conv1\t: {x.size()}')
        
        x = self.resnext_maxpool1(x)
        print(f'after resnext_maxpool1\t: {x.size()}')
        
        for i, block in enumerate(self.resnext_conv2):
            x = block(x)
            print(f'after resnext_conv2 #{i}\t: {x.size()}')
            
        for i, block in enumerate(self.resnext_conv3):
            x = block(x)
            print(f'after resnext_conv3 #{i}\t: {x.size()}')
            
        for i, block in enumerate(self.resnext_conv4):
            x = block(x)
            print(f'after resnext_conv4 #{i}\t: {x.size()}')
            
        for i, block in enumerate(self.resnext_conv5):
            x = block(x)
            print(f'after resnext_conv5 #{i}\t: {x.size()}')
        
        x = self.avgpool(x)
        print(f'after avgpool\t\t: {x.size()}')
        
        x = torch.flatten(x, start_dim=1)
        print(f'after flatten\t\t: {x.size()}')
        
        x = self.fc(x)
        print(f'after fc\t\t: {x.size()}')
        
        return x

And now we can check if the entire network works properly by running the following testing code. Here I initialize a CBAMResNeXt instance and pass a dummy RGB image of size 224×224 through it. python

# Codeblock 12
cbamresnext = CBAMResNeXt()
x = torch.randn(1, 3, 224, 224)

out = cbamresnext(x)

Below is what the resulting output looks like. Here we can see that the model successfully passes the original image through the entire network up until the classification head. This essentially indicates that our CBAM is properly attached, and thus this model is ready to be trained for image classification. http

# Codeblock 12 Output
original               : torch.Size([1, 3, 224, 224])
after resnext_conv1    : torch.Size([1, 64, 112, 112])
after resnext_maxpool1 : torch.Size([1, 64, 56, 56])
after resnext_conv2 #0 : torch.Size([1, 256, 56, 56])
after resnext_conv2 #1 : torch.Size([1, 256, 56, 56])
after resnext_conv2 #2 : torch.Size([1, 256, 56, 56])
after resnext_conv3 #0 : torch.Size([1, 512, 28, 28])
after resnext_conv3 #1 : torch.Size([1, 512, 28, 28])
after resnext_conv3 #2 : torch.Size([1, 512, 28, 28])
after resnext_conv3 #3 : torch.Size([1, 512, 28, 28])
after resnext_conv4 #0 : torch.Size([1, 1024, 14, 14])
after resnext_conv4 #1 : torch.Size([1, 1024, 14, 14])
after resnext_conv4 #2 : torch.Size([1, 1024, 14, 14])
after resnext_conv4 #3 : torch.Size([1, 1024, 14, 14])
after resnext_conv4 #4 : torch.Size([1, 1024, 14, 14])
after resnext_conv4 #5 : torch.Size([1, 1024, 14, 14])
after resnext_conv5 #0 : torch.Size([1, 2048, 7, 7])
after resnext_conv5 #1 : torch.Size([1, 2048, 7, 7])
after resnext_conv5 #2 : torch.Size([1, 2048, 7, 7])
after avgpool          : torch.Size([1, 2048, 1, 1])
after flatten          : torch.Size([1, 2048])
after fc               : torch.Size([1, 1000])

···

Ending

And well I think that’s pretty much everything about CBAM and how to implement it from scratch. You can also find the code used in this article in my GitHub repository [6]. Please let me know if you find any mistakes in the discussion or in the code. Thanks for reading, I hope you learn something new today. See ya in my next article! ···

References

[1] Sanghyun Woo et al. CBAM: Convolutional Block Attention Module. Arxiv. https://arxiv.org/abs/1807.06521 [Accessed November 12, 2025]. [2] Muhammad Ardi Putra. SENet Paper Walkthrough: The Channel-Wise Attention. Towards Data Science. https://towardsdatascience.com/the-channel-wise-attention/ [Accessed November 12, 2025]. Also available at https://medium.com/ai-advances/senet-paper-walkthrough-the-channel-wise-attention-8ac72b9cc252. [3] Image originally created by author. [4] Muhammad Ardi Putra. ResNeXt Paper Walkthrough: Taking ResNet to the Next Level. Towards Data Science. https://towardsdatascience.com/taking-resnet-to-the-next-level/ [Accessed November 12, 2025]. Also available at https://medium.com/ai-advances/taking-resnet-to-the-next-level-resnext-77088c245698. [5] Saining Xie et al. Aggregated Residual Transformations for Deep Neural Networks. Arxiv. https://arxiv.org/abs/1611.05431 [Accessed November 12, 2025]. [6] MuhammadArdiPutra. CBAM. GitHub. https://github.com/MuhammadArdiPutra/medium_articles/blob/main/Deep Learning From Scratch/CBAM.ipynb [Accessed November 12, 2025].

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Muhammad Ardi Putra

Deep learning storyteller See all from Muhammad Ardi Putra Editor's PicksComputer VisionDeep LearningNeural NetworkPytorch Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Visualizing What Batch Normalization Is and Its Advantages

Deep Learning Optimizing your neural network training with batch normalization Peng Qian February 3, 20244 min read

What I Am Doing to Stay Relevant as a Senior Analytics Consultant in 2026

Data analysis Learn how to work with AI, while strengthening your unique human skills that technology cannot replace Rashi Desai February 7, 20266 min read

DeepMind, AlphaFold & the Protein Folding Problem

Deep Learning

Prem Oommen January 25, 20215 min read

🏐 Ultimate Volleyball: A multi-agent reinforcement learning environment built using Unity...

Artificial Intelligence Train reinforcement learning agents to play Volleyball Joy Zhang August 11, 20215 min read

Why a Data Science Mentor Can Help You in 2022

Data Science Lessons learned from two years with a data science mentor Rose Day December 10, 20216 min read

Finding A Mentor In Data Science

Artificial Intelligence How to climb on to the shoulders of a Giant Kurtis Pykes November 2, 20205 min read

Memory-Efficient Embeddings

Deep Learning Creating smaller models with a new kind of embedding layer Robert Kübler January 1, 202414 min read

Signal Processing on Cell Complexes with Cell Complex Networks (CXNs)

Deep learning supported on simplicial/cell complexes and beyond Mustafa Hajij November 19, 20218 min read

What can flatness teach us: understanding generalisation in Deep Neural Networks

Artificial Intelligence This is the third post in a series summarising work that seeks to provide a theory of generalisation in Deep Neural Networks (DNNs)... Chris Mingard March 29, 202110 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close