{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Muhammad Ardi Putra",
  "title_listed": "ConvNeXt Paper Walkthrough: The CNN That Challenges ViT",
  "keywords": "",
  "domain": "图像处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/the-cnn-that-challenges-vit/",
  "publish_date": "2025-05-05",
  "corpus_stage": "",
  "notes": "",
  "id": "ac847f5a52fe",
  "collected_at": "2026-10-02T02:53:20",
  "title": "ConvNeXt Paper Walkthrough: The CNN That Challenges ViT | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# ConvNeXt Paper Walkthrough: The CNN That Challenges ViT\nA PyTorch implementation on the ConvNeXt architecture\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nMay 5, 202523 min read\n![](../images/000.jpg)\n[Image](https://unsplash.com/photos/a-very-long-line-of-yellow-lines-on-a-black-background-YeUVDKZWSZ4) by Rene Böhmer via Unsplash\n## Introduction\nThe invention of ViT (Vision Transformer) causes us to think that CNNs are obsolete. But is this really true?\nIt is widely believed that the impressive performance of ViT comes primarily from its transformer-based architecture. However, researchers from Meta argued that it’s not entirely true. If we take a closer look at the architectural design, ViT introduced radical changes not only to the structure of the network but also to the model configurations. Meta’s researchers thought that perhaps it is not the structure that makes ViT superior, but its configuration. In order to prove this, they tried to apply the ViT configuration parameters to the ResNet architecture from 2015. \n— And they found their thesis true.\n···\nIn this article I am going to talk about ConvNeXt which was first proposed in the paper titled “ _A ConvNet for the 2020s_ ” written by Liu _et al._ [1] back in 2022. Here I’ll also try to implement it myself from scratch with PyTorch so that you can get better understanding of the changes made from the original ResNet. In fact, the actual ConvNeXt implementation is available in their GitHub repository [2], but I find it too complex to explain line by line. Thus, I decided to write it down on my own so that I can explain it with my style, which I believe is more beginner-friendly. Disclaimer on, my implementation might not perfectly replicate the original one, but I think it’s still good to consider my code as a resource to learn. So, after reading my article I recommend you check the original code especially if you’re planning to use ConvNeXt for your project.\n···\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## The Hyperparameter Tuning\nWhat the authors essentially did in the research was hyperparameter tuning on the ResNet model. Generally speaking, there were five aspects they experimented with: _macro design_ , _ResNeXt_ , _inverted bottleneck_ , _large kernel_ , and _micro design_. We can see the experimental results on these aspects in the following figure.\n![](../images/001.png)\nFigure 1. The hyperparameter tuning results done on the original ResNet architecture [1].\nThere were two ResNet variants used in their experiments: ResNet-50 and ResNet-200 (shown in purple and gray, respectively). Let’s now focus on the results obtained from tuning the ResNet-50 architecture. Based on the figure, we can see that this model initially obtained 78.8% accuracy on ImageNet dataset. They tuned this model until eventually it reached 82.0%, surpassing the state-of-the-art Swin-T architecture which only achieved 81.3% (the orange bar). This tuned version of the ResNet model is the one so-called ConvNeXt proposed in the paper. Their experiments on ResNet-200 confirm that the previous results are valid since its tuned version, i.e., ConvNeXt-B, also successfully surpasses the performance of Swin-B (the larger variant of Swin-T).\n### Macro Design\nThe first change made on the original ResNet was the _macro design_. If we take a closer look at Figure 2 below, we can see that a ResNet model essentially consists of four main stages, namely _conv2_x_ , _conv3_x_ , _conv4_x_ and _conv5_x_ , which each of them also comprises multiple _bottleneck_ blocks. Talking more specifically about ResNet-50, the bottleneck blocks in each stage is repeated 3, 4, 6 and 3 times, respectively. Later on, I’ll refer to these numbers as _stage ratio_.\n![](../images/002.png)\nFigure 2. The ResNet architecture variants [3].\nThe authors of the ConvNeXt paper tried to change this stage ratio according to the Swin-T architecture, i.e., 1:1:3:1. Well, it’s actually 2:2:6:2 if you see the architectural details from the original Swin Transformer paper in Figure 3, but it’s basically just a derivation from the same ratio. By applying this configuration, authors obtained 0.6% improvement (from 78.8% to 79.4%). Thus, they decided to use 1:1:3:1 stage ratio for the upcoming experiments.\n![](../images/body-23415fc680e2aa7e02cd.png)\nFigure 3. The Swin Transformer architecture variants [4].\nStill related to macro design, changes were also made to the first convolution layer of ResNet. If you go back to Figure 2 (the _conv1_ row), you’ll see that it originally uses 7×7 kernel with stride 2, which reduces the image size from 224×224 to 112×112. Being inspired by Swin Transformer, authors also wanted to treat the input image as non-overlapping patches. Thus, they changed the kernel size to 4×4 and the stride to 4. This idea was actually adopted from the original ViT, where it uses 16×16 kernel with stride 16. One thing you need to know in ConvNeXt is that the resulting patches are treated as a standard image rather than a sequence. With this modification, the accuracy slightly improved from 79.4% to 79.5%. Hence, authors used this configuration for the first convolution layer in the next experiments.\n### ResNeXt-ification\nAs the macro design is done, the next thing authors did was to adopt the ResNeXt architecture, which was first proposed in a paper titled “ _Aggregated Residual Transformations for Deep Neural Networks_ ” [5]. The idea of ResNeXt is that it basically applies group convolution to the bottleneck blocks of the ResNet architecture. In case you’re not yet familiar with group convolution, it essentially works by separating input channels into groups and performing convolution operations within each group independently, allowing faster computation as the number of groups increases. ConvNeXt adopts this idea by setting the number of groups to be the same as the number of kernels. This approach, which is commonly known as _depthwise convolution_ , enables the network to obtain the lowest possible computational complexity. However, it is important to note that increasing the number of convolution groups like this leads to a reduction in accuracy as it lowers the model capacity to learn. Thus, the drop in accuracy to 78.3% was expected.\nThat wasn’t the end of the ResNeXt-ification section, though. In fact, the ResNeXt paper gives us a guidance that if we increase the number of groups, we also need to expand the width of the network, i.e., add more channels. Thus, ConvNeXt authors readjusted the number of kernels based on the one used in Swin-T. You can see in Figure 2 and 3 that ResNet originally uses 64, 128, 256 and 512 kernels in each stage, whereas Swin-T uses 96, 192, 384, and 768. Such an increase in the model width allows the network to significantly push the accuracy to 80.5%.\n### Inverted Bottleneck\nStill with Figure 2, it is also seen that ResNet-50, ResNet-101, and ResNet-152 share the exact same bottleneck structure. For instance, the block at stage _conv5_x_ consists of 3 convolution layers with 512, 512, and 2048 kernels, where the input of the first convolution is either 1024 (coming from the _conv4_x_ stage) or 2048 (from the previous block in the _conv5_x_ stage itself). These ResNet variations essentially follow the _wide → narrow → wide_ structure, which is the reason that this block is called _bottleneck_. Instead of using a structure like this, ConvNeXt employs the inverted version of bottleneck, where it follows the _narrow → wide → narrow_ structure adopted from the feed-forward layer of the Transformer architecture. In Figure 4 below (a) is the _bottleneck_ block used in ResNet and (b) is the so-called _inverted bottleneck_ block. By using this structure, the model accuracy increased from 80.5% to 80.6%.\n![](../images/body-fdbfff671ec187455162.png)\nFigure 4. The bottleneck block in ResNeXt (a), the inverted bottleneck block (b), and the ConvNeXt block (c) [1].\n### Kernel Size\nThe next exploration was done on the kernel size inside the inverted bottleneck block. Before experimenting with different kernel sizes, further modification was done to the structure of the block, where authors swapped the order of the first and second layer such that the depthwise convolution is now placed at the beginning of the block as seen in Figure 4 (c). Thanks to this modification, the block is now called _ConvNeXt block_ as it no longer completely resembles the original inverted bottleneck structure. This idea was actually adopted from Transformer, where the MSA (Multihead Self-Attention) layer is placed before the MLP layers. In the case of ConvNeXt, the depthwise convolution acts as the replacement of MSA, while the linear layers in MLP Transformers are replaced by pointwise convolutions. Simply moving up the depthwise convolution like this reduced the accuracy from 80.6% to 79.9%. However, this is acceptable because the current experiment set is still ongoing.\nExperiments on the kernel size was then applied only on the depthwise convolution layer, leaving the remaining pointwise convolutions unchanged. Here authors tried to use different kernel sizes, where they found that 7×7 worked best as it successfully recovered the accuracy back to 80.6% with lower computational complexity (4.6 vs 4.2 GFLOPS). Interestingly, this kernel size matches the window dimensions in the Swin Transformer architecture, which corresponds to the patch size used in the self-attention mechanism. You can actually see this in Figure 3 where the window sizes in Swin Transformer variants are all 7×7.\n### Micro Design\nThe final aspect tuned in the paper is the so-called _micro design_ , which essentially refers to the things related to the intricate details of the network. Similar to the previous ones, the parameters used here are mainly also adopted from Transformers. Authors initially replaced ReLU with GELU. Even though with this replacement the accuracy remained the same (80.6%), but they decided to go with this activation function for the subsequent experiments. The accuracy finally increased after the number of activation functions was reduced. Instead of applying GELU after each convolution layer in the ConvNeXt block, this activation function was placed only between the two pointwise convolutions. This modification allowed the network to boost the accuracy up to 81.3%, at which point this score was already on par with the Swin-T architecture while still having lower GFLOPS (4.2 vs 4.5).\nNext, it is a common practice to use _Conv-BN-ReLU_ structure in CNN-based architecture, which is exactly what ResNet implements as well. Instead of following this convention, authors decided to implement only a single batch normalization layer, which is placed before the first pointwise convolution layer. This change improved the accuracy to 81.4%, surpassing the accuracy of Swin-T by a little bit. Despite this achievement, parameter tuning was still continued by replacing batch norm with layer norm, which again raised the accuracy by 0.1% to 81.5%. All the modifications related to micro design resulted in the architecture shown in Figure 5 (the rightmost image). Here you can see how a ConvNeXt block differs from Swin Transformer and ResNet blocks.\n![](../images/body-5535382ca9c8d055883e.png)\nFigure 5. What the Swin-T, ResNet-50 and ConvNeXt-T blocks look like at the initial stage [1].\nThe last thing the authors did related to the micro design was applying separate downsampling layers. In the original ResNet architecture, the spatial dimension of a tensor reduces by half when we move from one stage to another. You can see in Figure 2 that initially ResNet accepts input of size 224×224 which then shrinks to 112×112, 56×56, 28×28, 14×14, and 7×7 at stage _conv1_ , _conv2_x_ , _conv3_x_ , _conv4_x_ and _conv5_x_ , respectively. Especially in _conv2_x_ and the subsequent ones, the spatial dimension reduction is done by changing the stride parameter of the pointwise convolution to 2. Instead of doing so, ConvNeXt performs downsampling by placing another convolution layer right before the element-wise summation operation within the block. The kernel size and stride of this layer are set to 2, simulating a non-overlapping sliding window. In fact, it is mentioned in the paper that using this separate downsampling layer caused the accuracy to degrade instead. Nevertheless, authors managed to solve this issue by applying additional layer normalization layers at several parts of the network, i.e., before each downsampling layer, after the _stem_ stage and after the global average pooling layer (right before the final output layer). With this tuning, authors successfully boosted the accuracy to 82.0%, which is much higher than Swin-T (81.3%) while still having the exact same GFLOPS (4.5).\nAnd that’s basically all the modifications made on the original ResNet to create the ConvNeXt architecture. Don’t worry if it still feels a bit unclear for now — I believe things will become clearer as we get into the code.\n···\n## ConvNeXt Implementation\nFigure 6 below displays the details of the entire ConvNeXt-T architecture which we will later implement every single of its components one by one. Here you can also see how it differs from ResNet-50 and Swin-T, the two models that are comparable to ConvNeXt-T.\n![](../images/body-f82c8c506ac7a3bb4f84.png)\nFigure 6. The details of the ResNet-50, ConvNeXt-T, and Swin-T architectures [1].\nWhen it comes to the implementation, the first thing we need to do is to import the required modules. The only two we import here are the base `torch` module and its `nn` submodule for loading neural network layers.\npython\n\n```\n# Codeblock 1\nimport torch\nimport torch.nn as nn\n```\n\n### ConvNeXt Block\nNow let’s start with the ConvNeXt block. You can see in Figure 6 that the block structures in _res2_ , _res3_ , _res4_ , and _res5_ stages are basically the same, in which all of those correspond to the rightmost illustration in Figure 5. Thanks to these identical structures, we can implement them in a single class and use it repeatedly. Look at the Codeblock 2a and 2b below to see how I do that.\npython\n\n```\n# Codeblock 2a\nclass ConvNeXtBlock(nn.Module):\n    def __init__(self, num_channels):         #(1)\n        super().__init__()\n        hidden_channels = num_channels * 4    #(2)\n\n        \n        self.conv0 = nn.Conv2d(in_channels=num_channels,         #(3) \n                               out_channels=num_channels,        #(4)\n                               kernel_size=7,    #(5)\n                               stride=1,\n                               padding=3,        #(6)\n                               groups=num_channels)              #(7)\n        \n        self.norm = nn.LayerNorm(normalized_shape=num_channels)  #(8)\n        \n        self.conv1 = nn.Conv2d(in_channels=num_channels,         #(9)\n                               out_channels=hidden_channels, \n                               kernel_size=1, \n                               stride=1, \n                               padding=0)\n        \n        self.gelu = nn.GELU()  #(10)\n        \n        self.conv2 = nn.Conv2d(in_channels=hidden_channels,      #(11)\n                               out_channels=num_channels, \n                               kernel_size=1, \n                               stride=1, \n                               padding=0)\n```\n\nI decided to name this class `ConvNeXtBlock`. You can see at line `#(1)` in the above codeblock that this class accepts `num_channels` as the only parameter, in which it denotes both the number of input and output channels. Remember that a ConvNeXt block follows the pattern of the inverted bottleneck structure, i.e., _narrow_ → _wide_ → _narrow_. If you take a closer look at Figure 6, you’ll notice that the _wide_ part is 4 times larger than the _narrow_ part. Thus, we set the value of the `hidden_channels` variable accordingly (`#(2)`). \nNext, we initialize 3 convolution layers which I refer to them as `conv0`, `conv1` and `conv2`. Every single of these convolution layers has their own specifications. For `conv0`, we set the number of input and output channels to be the same, which is the reason that both its `in_channels` and `out_channels` parameters are set to `num_channels` (`#(3–4)`). We set the kernel size of this layer to 7×7 (`#(5)`). Given this specification, we need to set the padding size to 3 in order to retain the spatial dimension (`#(6)`). Don’t forget to set the `groups` parameter to `num_channels` because we want this to be a depthwise convolution layer (`#(7)`). On the other hand, the `conv1` layer (`#(9)`) is responsible to increase the number of image channels, whereas the subsequent `conv2` layer (`#(11)`) is employed to shrink the tensor back to the original channel count. It is important to note that `conv1` and `conv2` are both using 1×1 kernel size, which essentially means that it only works by combining information along the channel dimension. Additionally, here we also need to initialize layer norm (`#(8)`) and GELU activation function (`#(10)`) as the replacement for batch norm and ReLU.\nAs all layers required in the ConvNeXtBlock have been initialized, what we need to do next is to define the flow of the tensor in the `forward()` method below.\npython\n\n```\n# Codeblock 2b\n    def forward(self, x):\n        residual = x                 #(1)\n        print(f'x & residual\\t: {x.size()}')\n        \n        x = self.conv0(x)\n        print(f'after conv0\\t: {x.size()}')\n        \n        x = x.permute(0, 2, 3, 1)    #(2)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = self.norm(x)\n        print(f'after norm\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)    #(3)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = self.conv1(x)\n        print(f'after conv1\\t: {x.size()}')\n        \n        x = self.gelu(x)\n        print(f'after gelu\\t: {x.size()}')\n        \n        x = self.conv2(x)\n        print(f'after conv2\\t: {x.size()}')\n        \n        x = x + residual             #(4)\n        print(f'after summation\\t: {x.size()}')\n        \n        return x\n```\n\nWhat we basically do in the above code is just passing the tensor to each layer we defined earlier sequentially. However, there are two things I need to highlight here. First, we need to store the original input tensor to the `residual` variable (`#(1)`), in which it will skip over all operations within the ConvNeXt block. Secondly, remember that layer norm is commonly used for sequential data, where it typically has a different shape from that of image data. Due to this reason, we need to adjust the tensor dimension such that the shape becomes _(N, H, W, C)_ (`#(2)`) before we actually perform the layer normalization operation. Afterwards, don’t forget to permute this tensor back to _(N, C, H, W)_ (`#(3)`). The resulting tensor is then passed through the remaining layers before being summed with the residual connection (`#(4)`).\nTo check if our `ConvNeXtBlock` class works properly, we can test it using the Codeblock 3 below. Here we are going to simulate the block used in _res2_ stage. So, we set the `num_channels` parameter to 96 (`#(1)`) and create a dummy tensor which we assume as a batch of single image of size 56×56 (`#(2)`).\npython\n\n```\n# Codeblock 3\nconvnext_block_test = ConvNeXtBlock(num_channels=96)  #(1)\nx_test = torch.rand(1, 96, 56, 56)  #(2)\n\nout_test = convnext_block_test(x_test)\n```\n\nBelow is what the resulting output looks like. Talking about the internal flow, it seems like all layers we stacked earlier work properly. At line `#(1)` in the output below we can see that the tensor dimension changed to 1×56×56×96 _(N, H, W, C)_ after being permuted. This tensor size then changed back to 1×96×56×56 _(N, C, H, W)_ after the second permute operation (`#(2)`). Next, the _conv1_ layer successfully expanded the number of channels to be 4 times greater than the input (`#(3)`) which was then reduced back to the original channel count (`#(4)`). Here you can see that the tensor shape at the first and the last layer are exactly the same, allowing us to stack multiple ConvNeXt blocks as many as we want.\npython\n\n```\n# Codeblock 3 Output\nx & residual    : torch.Size([1, 96, 56, 56])\nafter conv0     : torch.Size([1, 96, 56, 56])  \nafter permute   : torch.Size([1, 56, 56, 96])    #(1)\nafter norm      : torch.Size([1, 56, 56, 96])\nafter permute   : torch.Size([1, 96, 56, 56])    #(2)\nafter conv1     : torch.Size([1, 384, 56, 56])   #(3)\nafter gelu      : torch.Size([1, 384, 56, 56])\nafter conv2     : torch.Size([1, 96, 56, 56])    #(4)\nafter summation : torch.Size([1, 96, 56, 56])\n```\n\n···\n### ConvNeXt Block Transition\nThe next component I want to implement is the one I refer to as the _ConvNeXt block transition_. The idea of this block is actually similar to the ConvNeXt block we implemented earlier, except that this transition block is used when we are about to move from a stage to the subsequent one. More specifically, this block will later be employed as the first ConvNeXt block in each stage (except _res2_). The reason I implement it in separate class is that there are some intricate details that differ from the ConvNeXt block. Additionally, it is worth noting that the term _transition_ is not officially used in the paper. Rather, it’s just the word I use on my own to describe this idea. — I actually also used this technique back when I write about the smaller ResNet version, i.e., ResNet-18 and ResNet-34. Click on the link at reference number [6] at the end of this article if you’re interested to read that one.\npython\n\n```\n# Codeblock 4a\nclass ConvNeXtBlockTransition(nn.Module):\n    def __init__(self, in_channels, out_channels):  #(1)\n        super().__init__()\n        hidden_channels = out_channels * 4\n        \n        self.projection = nn.Conv2d(in_channels=in_channels,      #(2) \n                                    out_channels=out_channels, \n                                    kernel_size=1, \n                                    stride=2,\n                                    padding=0)\n        \n        self.conv0 = nn.Conv2d(in_channels=in_channels, \n                               out_channels=out_channels, \n                               kernel_size=7,\n                               stride=1,\n                               padding=3,\n                               groups=in_channels)\n        \n        self.norm0 = nn.LayerNorm(normalized_shape=out_channels)\n        \n        self.conv1 = nn.Conv2d(in_channels=out_channels, \n                               out_channels=hidden_channels, \n                               kernel_size=1, \n                               stride=1, \n                               padding=0)\n        \n        self.gelu = nn.GELU()\n        \n        self.conv2 = nn.Conv2d(in_channels=hidden_channels, \n                               out_channels=out_channels, \n                               kernel_size=1, \n                               stride=1,\n                               padding=0)\n        \n        self.norm1 = nn.LayerNorm(normalized_shape=out_channels)  #(3)\n\n        self.downsample = nn.Conv2d(in_channels=out_channels,     #(4)\n                                    out_channels=out_channels, \n                                    kernel_size=2, \n                                    stride=2)\n```\n\nThe first difference you might notice here is the input of the `__init__()` method, which in this case we separate the number of input and output channels into two parameters as seen at line `#(1)` in Codeblock 4a. This is essentially done because we need this block to take the output tensor from the previous stage which has different number of channels from that of the one to be generated in the subsequent stage. Referring to Figure 6, for example, if we were to create the first ConvNeXt block in _res3_ stage, we need to configure it such that it accepts a tensor of 96 channels from _res2_ and returns another tensor with 192 channels.\nSecondly, here we implement the _separate downsample layer_ I explained earlier (`#(4)`) alongside the corresponding layer norm to be placed before it (`#(3)`). As the name suggests, this layer is employed to reduce the spatial dimension of the image by half.\nThird, we initialize the so-called _projection layer_ at line `#(2)`. In the ConvNeXtBlock we created earlier, this layer is not necessary because the input and output tensor is exactly the same. In the case of _transition_ block, the image spatial dimension is reduced by half, while at the same time the number of output channels is doubled. This _projection_ layer is responsible to adjust the dimension of the residual connection in order to match it with the one from the main flow, allowing element-wise operation to be performed.\nThe `forward()` method in the Codeblock 4b below is also similar to the one belongs to the `ConvNeXtBlock` class, except that here the residual connection needs to be processed with the projection layer (`#(1)`) while the main tensor requires to be downsampled (`#(2)`) before the summation is done at line `#(3)`.\npython\n\n```\n# Codeblock 4b\n    def forward(self, x):\n        print(f'original\\t\\t: {x.size()}')\n\n        residual = self.projection(x)  #(1)\n        print(f'residual after proj\\t: {residual.size()}')\n        \n        x = self.conv0(x)\n        print(f'after conv0\\t\\t: {x.size()}')\n        \n        x = x.permute(0, 2, 3, 1)\n        print(f'after permute\\t\\t: {x.size()}')\n        \n        x = self.norm0(x)\n        print(f'after norm1\\t\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)\n        print(f'after permute\\t\\t: {x.size()}')\n        \n        x = self.conv1(x)\n        print(f'after conv1\\t\\t: {x.size()}')\n        \n        x = self.gelu(x)\n        print(f'after gelu\\t\\t: {x.size()}')\n        \n        x = self.conv2(x)\n        print(f'after conv2\\t\\t: {x.size()}')\n\n        x = x.permute(0, 2, 3, 1)\n        print(f'after permute\\t\\t: {x.size()}')\n        \n        x = self.norm1(x)\n        print(f'after norm1\\t\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)\n        print(f'after permute\\t\\t: {x.size()}')\n        \n        x = self.downsample(x)  #(2)\n        print(f'after downsample\\t: {x.size()}')\n        \n        x = x + residual  #(3)\n        print(f'after summation\\t\\t: {x.size()}')\n        \n        return x\n```\n\nNow let’s test the `ConvNeXtBlockTransition` class above using the following codeblock. Suppose we are about to implement the first ConvNeXt block in stage _res3_. To do so, we can simply instantiate the transition block with `in_channels=96` and `out_channels=192` before eventually passing a dummy tensor of size 1×96×56×56 through it.\npython\n\n```\n# Codeblock 5\nconvnext_block_transition_test = ConvNeXtBlockTransition(in_channels=96, \n                                                         out_channels=192)\nx_test = torch.rand(1, 96, 56, 56)\n\nout_test = convnext_block_transition_test(x_test)\n```\n\nmarkup\n\n```\n# Codeblock 5 Outputoriginal            : torch.Size([1, 96, 56, 56])residual after proj : torch.Size([1, 192, 28, 28])  #(1)after conv0         : torch.Size([1, 192, 56, 56])  #(2)after permute       : torch.Size([1, 56, 56, 192])after norm0         : torch.Size([1, 56, 56, 192])after permute       : torch.Size([1, 192, 56, 56])after conv1         : torch.Size([1, 768, 56, 56])after gelu          : torch.Size([1, 768, 56, 56])after conv2         : torch.Size([1, 192, 56, 56])  #(3)after permute       : torch.Size([1, 56, 56, 192])after norm1         : torch.Size([1, 56, 56, 192])after permute       : torch.Size([1, 192, 56, 56])after downsample    : torch.Size([1, 192, 28, 28])  #(4)after summation     : torch.Size([1, 192, 28, 28])  #(5)\n```\n\nYou can see in the resulting output that our projection layer directly maps the 1×96×56×56 residual tensor to 1×192×28×28 as shown at line `#(1)`. Meanwhile, the main tensor `x` needs to be processed by the other layers we initialized earlier to achieve this shape. The steps we performed from line `#(2)` to `#(3)` on the `x` tensor are basically the same as those in the `ConvNeXtBlock` class. At this point we already got the number of channels matches our need (192). The spatial dimension is then reduced after the tensor being processed by the `downsample` layer (`#(4)`). As the tensor dimensions of `x` and `residual` have matched, we can finally perform the element-wise summation (`#(5)`).\n···\n### The Entire ConvNeXt Architecture\nAs we got `ConvNeXtBlock` and `ConvNeXtBlockTransition` classes ready to use, we can now start to construct the entire ConvNeXt architecture. Before we do that, I would like to introduce some config parameters first. See the Codeblock 6 below.\npython\n\n```\n# Codeblock 6\nIN_CHANNELS  = 3     #(1)\nIMAGE_SIZE   = 224   #(2)\n\nNUM_BLOCKS   = [3, 3, 9, 3]         #(3)\nOUT_CHANNELS = [96, 192, 384, 768]  #(4)\nNUM_CLASSES  = 1000  #(5)\n```\n\nThe first one is the dimension of the input image. As shown at line `#(1)` and `#(2)`, here we set `in_channels` to 3 and `image_size` to 224 since by default ConvNeXt accepts a batch of RGB images of that size. The next ones are related to the model configuration. In this case, I set the number of ConvNeXt blocks of each stage to `[3, 3, 9, 3]` (`#(3)`) and the corresponding number of output channels to `[96, 192, 384, 768]` (`#(4)`) since I want to implement the ConvNeXt-T variant. You can actually change these numbers according to the configuration provided by the original paper shown in Figure 7. Finally, we set the number of neurons of the output channel to 1000, which corresponds to the number of classes in the dataset we train the model on (`#(5)`).\n![](../images/body-f8a8deffe3486c2f4cb9.png)\nFigure 7. The ConvNeXt variants [1].\nWe will now implement the entire architecture in the `ConvNeXt` class shown in Codeblock 7a and 7b below. The following `__init__()` method might seem a bit complicated at glance, but don’t worry as I’ll explain it thoroughly.\npython\n\n```\n# Codeblock 7a\nclass ConvNeXt(nn.Module):\n    def __init__(self):\n        super().__init__()\n        \n        self.stem = nn.Conv2d(in_channels=IN_CHANNELS,    #(1)\n                              out_channels=OUT_CHANNELS[0],\n                              kernel_size=4,\n                              stride=4,\n                             )\n\n        self.normstem = nn.LayerNorm(normalized_shape=OUT_CHANNELS[0])  #(2)\n        \n        #(3)\n        self.res2 = nn.ModuleList()\n        for _ in range(NUM_BLOCKS[0]):\n            self.res2.append(ConvNeXtBlock(num_channels=OUT_CHANNELS[0]))\n        \n        #(4)\n        self.res3 = nn.ModuleList([ConvNeXtBlockTransition(in_channels=OUT_CHANNELS[0], \n                                                           out_channels=OUT_CHANNELS[1])])\n        for _ in range(NUM_BLOCKS[1]-1):\n            self.res3.append(ConvNeXtBlock(num_channels=OUT_CHANNELS[1]))\n\n        #(5)\n        self.res4 = nn.ModuleList([ConvNeXtBlockTransition(in_channels=OUT_CHANNELS[1], \n                                                           out_channels=OUT_CHANNELS[2])])\n        for _ in range(NUM_BLOCKS[2]-1):\n            self.res4.append(ConvNeXtBlock(num_channels=OUT_CHANNELS[2]))\n\n        #(6)\n        self.res5 = nn.ModuleList([ConvNeXtBlockTransition(in_channels=OUT_CHANNELS[2], \n                                                           out_channels=OUT_CHANNELS[3])])\n        for _ in range(NUM_BLOCKS[3]-1):\n            self.res5.append(ConvNeXtBlock(num_channels=OUT_CHANNELS[3]))\n\n                \n        self.avgpool = nn.AdaptiveAvgPool2d(output_size=(1,1))  #(7)\n        self.normpool = nn.LayerNorm(normalized_shape=OUT_CHANNELS[3])  #(8)\n        self.fc = nn.Linear(in_features=OUT_CHANNELS[3],        #(9)\n                            out_features=NUM_CLASSES)\n        \n        self.relu = nn.ReLU()\n```\n\nThe first thing we do here is initializing the _stem_ stage (`#(1)`), which is essentially just a convolution layer with 4×4 kernel size and stride 4. This configuration will effectively reduce the image size to be 4 times smaller, where every single pixel in the output tensor represents a 4×4 patch in the input tensor. For the subsequent stages, we need to wrap the corresponding ConvNeXt blocks with `nn.ModuleList()`. For stage _res3_ (`#(4)`), _res4_ (`#(5)`) and _res5_ (`#(6)`) we place `ConvNeXtBlockTransition` at the beginning of each list as a “bridge” between stages. We don’t do this for stage _res2_ since the tensor produced by the _stem_ stage is already compatible with it (`#(3)`). Next, we initialize an `nn.AdaptiveAvgPool2d` layer, which will be used to reduce the spatial dimensions of the tensor to 1x1 by computing the mean across each channel (`#(7)`). In fact, this is the exact same process used by ResNet to prepare the tensor from the last convolution layer so that it matches the shape required by the subsequent output layer (`#(9)`). Additionally, don’t forget to initialize two layer normalization layers which I refer to as `normstem` (`#(2)`) and `normpool` (`#(8)`), in which these two layers will then be placed right after the `stem` stage and the `avgpool` layer.\nThe `forward()` method is pretty straightforward. All we need to do in the following code is just to place the layers one after another. Keep in mind that since the ConvNeXt blocks are stored in lists, we need to call them iteratively with loops as seen at line `#(1–4)`. Additionally, don’t forget to reshape the tensor produced by the `nn.AdaptiveAvgPool2d` layer (`#(5)`) so that it will be compatible with the subsequent fully-connected layer (`#(6)`).\npython\n\n```\n# Codeblock 7b\n    def forward(self, x):\n        print(f'original\\t: {x.size()}')\n        \n        x = self.relu(self.stem(x))\n        print(f'after stem\\t: {x.size()}')\n\n        x = x.permute(0, 2, 3, 1)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = self.normstem(x)\n        print(f'after normstem\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)\n        print(f'after permute\\t: {x.size()}')\n        \n        print()\n        for i, block in enumerate(self.res2):    #(1)\n            x = block(x)\n            print(f'after res2 #{i}\\t: {x.size()}')\n        \n        print()\n        for i, block in enumerate(self.res3):    #(2)\n            x = block(x)\n            print(f'after res3 #{i}\\t: {x.size()}')\n        \n        print()\n        for i, block in enumerate(self.res4):    #(3)\n            x = block(x)\n            print(f'after res4 #{i}\\t: {x.size()}')\n        \n        print()\n        for i, block in enumerate(self.res5):    #(4)\n            x = block(x)\n            print(f'after res5 #{i}\\t: {x.size()}')\n        \n        print()\n        x = self.avgpool(x)\n        print(f'after avgpool\\t: {x.size()}')\n\n        x = x.permute(0, 2, 3, 1)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = self.normpool(x)\n        print(f'after normpool\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = x.reshape(x.shape[0], -1)             #(5)\n        print(f'after reshape\\t: {x.size()}')\n        \n        x = self.fc(x)\n        print(f'after fc\\t: {x.size()}')          #(6)\n        \n        return x\n```\n\nNow for the moment of truth, let’s see if we have correctly implemented the entire ConvNeXt model by running the following code. Here I try to pass a tensor of size 1×3×224×224 to the network, simulating a batch of a single RGB image of size 224×224.\npython\n\n```\n# Codeblock 8\nconvnext_test = ConvNeXt()\n\nx_test   = torch.rand(1, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nout_test = convnext_test(x_test)\n```\n\nYou can see in the following output that it looks like our implementation is correct as the behavior of the network aligns with the architectural design shown in Figure 6. The spatial dimension of the image gradually gets smaller as we get deeper into the network, and at the same time the number of channels increases instead thanks to the `ConvNeXtBlockTransition` blocks we placed at the beginning of stage _res3_ (`#(1)`), _res4_ (`#(2)`), and _res5_ (`#(3)`). The `avgpool` layer then correctly downsampled the spatial dimension to 1×1 (`#(4)`), allowing it to be connected to the output layer (`#(5)`).\nmarkup\n\n```\n# Codeblock 8 Outputoriginal       : torch.Size([1, 3, 224, 224])after stem     : torch.Size([1, 96, 56, 56])after permute  : torch.Size([1, 56, 56, 96])after normstem : torch.Size([1, 56, 56, 96])after permute  : torch.Size([1, 96, 56, 56])after res2 #0  : torch.Size([1, 96, 56, 56])after res2 #1  : torch.Size([1, 96, 56, 56])after res2 #2  : torch.Size([1, 96, 56, 56])after res3 #0  : torch.Size([1, 192, 28, 28])  #(1)after res3 #1  : torch.Size([1, 192, 28, 28])after res3 #2  : torch.Size([1, 192, 28, 28])after res4 #0  : torch.Size([1, 384, 14, 14])  #(2)after res4 #1  : torch.Size([1, 384, 14, 14])after res4 #2  : torch.Size([1, 384, 14, 14])after res4 #3  : torch.Size([1, 384, 14, 14])after res4 #4  : torch.Size([1, 384, 14, 14])after res4 #5  : torch.Size([1, 384, 14, 14])after res4 #6  : torch.Size([1, 384, 14, 14])after res4 #7  : torch.Size([1, 384, 14, 14])after res4 #8  : torch.Size([1, 384, 14, 14])after res5 #0  : torch.Size([1, 768, 7, 7])    #(3)after res5 #1  : torch.Size([1, 768, 7, 7])after res5 #2  : torch.Size([1, 768, 7, 7])after avgpool  : torch.Size([1, 768, 1, 1])    #(4)after permute  : torch.Size([1, 1, 1, 768])after normpool : torch.Size([1, 1, 1, 768])after permute  : torch.Size([1, 768, 1, 1])after reshape  : torch.Size([1, 768])after fc       : torch.Size([1, 1000])         #(5)\n```\n\n···\n## Ending\nWell, that was pretty much everything about the theory and the implementation of the ConvNeXt architecture. Again, I do acknowledge that the code I demonstrate above might not fully capture everything since this article is intended to cover the general idea of the model. So, I highly recommend you read the original implementation by Meta’s researchers [2] if you want to know more about the intricate details.\nI hope you find this article useful. Thanks for reading!\n_P.S. the notebook used in this article is available on my GitHub repo. See the link at reference number [7]._\n···\n## References\n[1] Zhuang Liu _et al_. A ConvNet for the 2020s. Arxiv. <https://arxiv.org/pdf/2201.03545> [Accessed January 18, 2025].\n[2] facebookresearch. ConvNeXt. GitHub. <https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py> [Accessed January 18, 2025].\n[3] Kaiming He _et al_. Deep Residual Learning for Image Recognition. Arxiv. <https://arxiv.org/pdf/1512.03385> [Accessed January 18, 2025].\n[4] Ze Liu _et al_. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Arxiv. <https://arxiv.org/pdf/2103.14030> [Accessed January 18, 2025].\n[5] Saining Xie _et al_. Aggregated Residual Transformations for Deep Neural Networks. Arxiv. <https://arxiv.org/pdf/1611.05431> [Accessed January 18, 2025].\n[6] [Muhammad Ardi](https://medium.com/u/9801a58700ac). Paper Walkthrough: Residual Network (ResNet). Python in Plain English. <https://python.plainenglish.io/paper-walkthrough-residual-network-resnet-62af58d1c521> [Accessed January 19, 2025].\n[7] MuhammadArdiPutra. The CNN That Challenges ViT — ConvNeXt. GitHub. <https://github.com/MuhammadArdiPutra/medium_articles/blob/main/The%20CNN%20That%20Challenges%20ViT%20-%20ConvNeXt.ipynb> [Accessed January 24, 2025].\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-59e1c17daab1fcb582c5.jpeg)\nWritten By\n## [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nDeep learning storyteller\n[See all from Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Computer Vision](https://towardsdatascience.com/tag/computer-vision/)[Image Processing](https://towardsdatascience.com/tag/image-processing/)[Neural Network](https://towardsdatascience.com/tag/neural-network/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F&title=ConvNeXt%20Paper%20Walkthrough%3A%20The%20CNN%20That%20Challenges%20ViT)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F&title=ConvNeXt%20Paper%20Walkthrough%3A%20The%20CNN%20That%20Challenges%20ViT)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F&text=ConvNeXt%20Paper%20Walkthrough%3A%20The%20CNN%20That%20Challenges%20ViT)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/003.jpg)](https://towardsdatascience.com/machine-learning-an-ethics-review-5f4c35112be5/)\n## [Machine Learning - An Ethics Review](https://towardsdatascience.com/machine-learning-an-ethics-review-5f4c35112be5/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nI'm looking back at my projects from my Data Science program at Flatiron and reviewing the ethical concerns that each has...\n[Mike Erb](https://towardsdatascience.com/author/merb92/)\nNovember 3, 20209 min read\n  * [![](../images/004.jpeg)](https://towardsdatascience.com/a-brief-history-of-embedded-ai-6109be99199d/)\n## [A Brief History of Embedded AI](https://towardsdatascience.com/a-brief-history-of-embedded-ai-6109be99199d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow sensor hubs on smartphones helped bring artificial intelligence to the edge\n[Kenneth Joel](https://towardsdatascience.com/author/kennethjoel/)\nSeptember 28, 20224 min read\n  * [![](../images/005.jpeg)](https://towardsdatascience.com/how-to-control-a-robot-with-python/)\n## [How to Control a Robot with Python](https://towardsdatascience.com/how-to-control-a-robot-with-python/)\n[Robotics](https://towardsdatascience.com/category/robotics/)\n3D simulations and movement control with PyBullet\n[Mauro Di Pietro](https://towardsdatascience.com/author/maurodp/)\nOctober 23, 20259 min read\n  * [![](../images/006.png)](https://towardsdatascience.com/image-processing-with-python-application-of-fourier-transformation-5a8584dc175b/)\n## [Image Processing with Python - Application of Fourier Transformation](https://towardsdatascience.com/image-processing-with-python-application-of-fourier-transformation-5a8584dc175b/)\n[Programming](https://towardsdatascience.com/category/programming/)\nHow to make use of the Fourier Transformation to remove image elements\n[Tonichi Edeza](https://towardsdatascience.com/author/tonichi-edeza/)\nJanuary 28, 20213 min read\n  * [![](../images/007.jpg)](https://towardsdatascience.com/are-you-aware-how-difficult-your-regression-problem-is-b7dae830652b/)\n## [Are You Aware How Difficult Your Regression Problem Is?](https://towardsdatascience.com/are-you-aware-how-difficult-your-regression-problem-is-b7dae830652b/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nHalf of the solution is comprehending your problem, but half of the comprehension is knowing how complicated it is\n[Houssem Ben Braiek](https://towardsdatascience.com/author/houssem.benbraiek/)\nJanuary 31, 20226 min read\n  * [![](../images/008.jpeg)](https://towardsdatascience.com/the-evolution-of-big-data-compute-platforms-past-now-and-later-7c46697366d9/)\n## [The Evolution of Big Data Compute Platforms - Past, Now and Later](https://towardsdatascience.com/the-evolution-of-big-data-compute-platforms-past-now-and-later-7c46697366d9/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA journey into the evolution of Big Data Compute Platforms like Hadoop and Spark. Sharing my perspective on where we were, where we are...\n[Manoj Kukreja](https://towardsdatascience.com/author/mkukreja1/)\nDecember 27, 20207 min read\n  * [![](../images/009.jpeg)](https://towardsdatascience.com/10-amazing-machine-learning-visualizations-you-should-know-in-2023-528282940582/)\n## [10 Amazing Machine Learning Visualizations You Should Know in 2023](https://towardsdatascience.com/10-amazing-machine-learning-visualizations-you-should-know-in-2023-528282940582/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nYellowbrick for creating machine learning plots with less code\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nNovember 5, 202215 min read\n  * [![](../images/010.jpeg)](https://towardsdatascience.com/nested-class-imbalance-problem-within-machine-learning-based-multi-class-classification-a4b2c43ba832/)\n## [Nested Class Imbalance Problem Within Machine Learning-Based Multi-Class Classification](https://towardsdatascience.com/nested-class-imbalance-problem-within-machine-learning-based-multi-class-classification-a4b2c43ba832/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nShedding the light on real-life's non-human-centric data in which a hidden nested class imbalance problem exists, which is -up to the...\n[Ahmed Masri](https://towardsdatascience.com/author/ahmed-masri/)\nApril 21, 20216 min read\n  * [![](../images/011.jpg)](https://towardsdatascience.com/how-to-solve-optimisation-problems-using-linear-programming-912cc951afbb/)\n## [How To Solve Optimization Problems Using Linear Programming](https://towardsdatascience.com/how-to-solve-optimisation-problems-using-linear-programming-912cc951afbb/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn introduction to linear programming and how to solve it using the graphical and simplex algorithms\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nJuly 24, 20236 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "metadata": {
    "title": "ConvNeXt Paper Walkthrough: The CNN That Challenges ViT | Towards Data Science",
    "description": "A PyTorch implementation on the ConvNeXt architecture",
    "keywords": null,
    "author": "Muhammad Ardi Putra",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "ConvNeXt Paper Walkthrough: The CNN That Challenges ViT",
    "og:description": "A PyTorch implementation on the ConvNeXt architecture",
    "og:url": "https://towardsdatascience.com/the-cnn-that-challenges-vit/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/rene-bohmer-YeUVDKZWSZ4-unsplash-scaled-1.jpg",
    "og:image:width": "2560",
    "og:image:height": "1707",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Muhammad Ardi Putra",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "23 min read",
    "article:published_time": "2025-05-05T19:44:13.000Z",
    "article:modified_time": "2025-05-05T19:44:13.000Z",
    "article:author": "https://towardsdatascience.com/author/muhammad_ardi/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 202,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_000.txt",
      "chars": 49
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_001.txt",
      "chars": 1213
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_002.txt",
      "chars": 838
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_003.txt",
      "chars": 154
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_004.txt",
      "chars": 470
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_005.txt",
      "chars": 1725
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_006.txt",
      "chars": 1239
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_007.txt",
      "chars": 249
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_008.txt",
      "chars": 750
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_009.txt",
      "chars": 175
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_010.txt",
      "chars": 1851
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_011.txt",
      "chars": 1626
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_012.txt",
      "chars": 137
    },
    {
      "language": "txt",
      "file": "attachments/ac847f5a52fe/code/block_013.txt",
      "chars": 1331
    }
  ],
  "images": [
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:04:55Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:04:55Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:04:55Z"
    },
    {
      "status": "http_404",
      "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500",
      "alt": "",
      "downloaded": false,
      "discovery": "saved_markdown",
      "retried_at": "2026-10-20T18:04:55Z"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/muhammad_ardi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/computer-vision/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/image-processing/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/neural-network/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/machine-learning-an-ethics-review-5f4c35112be5/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/merb92/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/a-brief-history-of-embedded-ai-6109be99199d/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/kennethjoel/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/how-to-control-a-robot-with-python/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/robotics/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/photos/a-very-long-line-of-yellow-lines-on-a-black-background-YeUVDKZWSZ4"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/2201.03545"
    },
    {
      "type": "external",
      "href": "https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/1512.03385"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/2103.14030"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/1611.05431"
    },
    {
      "type": "external",
      "href": "https://medium.com/u/9801a58700ac"
    },
    {
      "type": "external",
      "href": "https://python.plainenglish.io/paper-walkthrough-residual-network-resnet-62af58d1c521"
    },
    {
      "type": "external",
      "href": "https://github.com/MuhammadArdiPutra/medium_articles/blob/main/The%20CNN%20That%20Challenges%20ViT%20-%20ConvNeXt.ipynb"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=ConvNeXt+Paper+Walkthrough%3A+The+CNN+That+Challenges+ViT&u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=ConvNeXt+Paper+Walkthrough%3A+The+CNN+That+Challenges+ViT&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=ConvNeXt+Paper+Walkthrough%3A+The+CNN+That+Challenges+ViT&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:31:53Z",
  "source_url": "https://towardsdatascience.com/the-cnn-that-challenges-vit/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 4,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# ConvNeXt Paper Walkthrough: The CNN That Challenges ViT\nA PyTorch implementation on the ConvNeXt architecture\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nMay 5, 202523 min read\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/rene-bohmer-YeUVDKZWSZ4-unsplash-scaled-1.jpg)\n[Image](https://unsplash.com/photos/a-very-long-line-of-yellow-lines-on-a-black-background-YeUVDKZWSZ4) by Rene Böhmer via Unsplash\n## Introduction\nThe invention of ViT (Vision Transformer) causes us to think that CNNs are obsolete. But is this really true?\nIt is widely believed that the impressive performance of ViT comes primarily from its transformer-based architecture. However, researchers from Meta argued that it’s not entirely true. If we take a closer look at the architectural design, ViT introduced radical changes not only to the structure of the network but also to the model configurations. Meta’s researchers thought that perhaps it is not the structure that makes ViT superior, but its configuration. In order to prove this, they tried to apply the ViT configuration parameters to the ResNet architecture from 2015. \n— And they found their thesis true.\n···\nIn this article I am going to talk about ConvNeXt which was first proposed in the paper titled “ _A ConvNet for the 2020s_ ” written by Liu _et al._ [1] back in 2022. Here I’ll also try to implement it myself from scratch with PyTorch so that you can get better understanding of the changes made from the original ResNet. In fact, the actual ConvNeXt implementation is available in their GitHub repository [2], but I find it too complex to explain line by line. Thus, I decided to write it down on my own so that I can explain it with my style, which I believe is more beginner-friendly. Disclaimer on, my implementation might not perfectly replicate the original one, but I think it’s still good to consider my code as a resource to learn. So, after reading my article I recommend you check the original code especially if you’re planning to use ConvNeXt for your project.\n···\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## The Hyperparameter Tuning\nWhat the authors essentially did in the research was hyperparameter tuning on the ResNet model. Generally speaking, there were five aspects they experimented with: _macro design_ , _ResNeXt_ , _inverted bottleneck_ , _large kernel_ , and _micro design_. We can see the experimental results on these aspects in the following figure.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/1jVLy6bFa6ldR3NNfpodS8g.png)\nFigure 1. The hyperparameter tuning results done on the original ResNet architecture [1].\nThere were two ResNet variants used in their experiments: ResNet-50 and ResNet-200 (shown in purple and gray, respectively). Let’s now focus on the results obtained from tuning the ResNet-50 architecture. Based on the figure, we can see that this model initially obtained 78.8% accuracy on ImageNet dataset. They tuned this model until eventually it reached 82.0%, surpassing the state-of-the-art Swin-T architecture which only achieved 81.3% (the orange bar). This tuned version of the ResNet model is the one so-called ConvNeXt proposed in the paper. Their experiments on ResNet-200 confirm that the previous results are valid since its tuned version, i.e., ConvNeXt-B, also successfully surpasses the performance of Swin-B (the larger variant of Swin-T).\n### Macro Design\nThe first change made on the original ResNet was the _macro design_. If we take a closer look at Figure 2 below, we can see that a ResNet model essentially consists of four main stages, namely _conv2_x_ , _conv3_x_ , _conv4_x_ and _conv5_x_ , which each of them also comprises multiple _bottleneck_ blocks. Talking more specifically about ResNet-50, the bottleneck blocks in each stage is repeated 3, 4, 6 and 3 times, respectively. Later on, I’ll refer to these numbers as _stage ratio_.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/1XpgH0qrZ4uzUrzxbQw7nfA.png)\nFigure 2. The ResNet architecture variants [3].\nThe authors of the ConvNeXt paper tried to change this stage ratio according to the Swin-T architecture, i.e., 1:1:3:1. Well, it’s actually 2:2:6:2 if you see the architectural details from the original Swin Transformer paper in Figure 3, but it’s basically just a derivation from the same ratio. By applying this configuration, authors obtained 0.6% improvement (from 78.8% to 79.4%). Thus, they decided to use 1:1:3:1 stage ratio for the upcoming experiments.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/19LdnTmb2t9aRTDjWPelqxA.png)\nFigure 3. The Swin Transformer architecture variants [4].\nStill related to macro design, changes were also made to the first convolution layer of ResNet. If you go back to Figure 2 (the _conv1_ row), you’ll see that it originally uses 7×7 kernel with stride 2, which reduces the image size from 224×224 to 112×112. Being inspired by Swin Transformer, authors also wanted to treat the input image as non-overlapping patches. Thus, they changed the kernel size to 4×4 and the stride to 4. This idea was actually adopted from the original ViT, where it uses 16×16 kernel with stride 16. One thing you need to know in ConvNeXt is that the resulting patches are treated as a standard image rather than a sequence. With this modification, the accuracy slightly improved from 79.4% to 79.5%. Hence, authors used this configuration for the first convolution layer in the next experiments.\n### ResNeXt-ification\nAs the macro design is done, the next thing authors did was to adopt the ResNeXt architecture, which was first proposed in a paper titled “ _Aggregated Residual Transformations for Deep Neural Networks_ ” [5]. The idea of ResNeXt is that it basically applies group convolution to the bottleneck blocks of the ResNet architecture. In case you’re not yet familiar with group convolution, it essentially works by separating input channels into groups and performing convolution operations within each group independently, allowing faster computation as the number of groups increases. ConvNeXt adopts this idea by setting the number of groups to be the same as the number of kernels. This approach, which is commonly known as _depthwise convolution_ , enables the network to obtain the lowest possible computational complexity. However, it is important to note that increasing the number of convolution groups like this leads to a reduction in accuracy as it lowers the model capacity to learn. Thus, the drop in accuracy to 78.3% was expected.\nThat wasn’t the end of the ResNeXt-ification section, though. In fact, the ResNeXt paper gives us a guidance that if we increase the number of groups, we also need to expand the width of the network, i.e., add more channels. Thus, ConvNeXt authors readjusted the number of kernels based on the one used in Swin-T. You can see in Figure 2 and 3 that ResNet originally uses 64, 128, 256 and 512 kernels in each stage, whereas Swin-T uses 96, 192, 384, and 768. Such an increase in the model width allows the network to significantly push the accuracy to 80.5%.\n### Inverted Bottleneck\nStill with Figure 2, it is also seen that ResNet-50, ResNet-101, and ResNet-152 share the exact same bottleneck structure. For instance, the block at stage _conv5_x_ consists of 3 convolution layers with 512, 512, and 2048 kernels, where the input of the first convolution is either 1024 (coming from the _conv4_x_ stage) or 2048 (from the previous block in the _conv5_x_ stage itself). These ResNet variations essentially follow the _wide → narrow → wide_ structure, which is the reason that this block is called _bottleneck_. Instead of using a structure like this, ConvNeXt employs the inverted version of bottleneck, where it follows the _narrow → wide → narrow_ structure adopted from the feed-forward layer of the Transformer architecture. In Figure 4 below (a) is the _bottleneck_ block used in ResNet and (b) is the so-called _inverted bottleneck_ block. By using this structure, the model accuracy increased from 80.5% to 80.6%.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/1AX881phAfF4DmZk6UrTD6g.png)\nFigure 4. The bottleneck block in ResNeXt (a), the inverted bottleneck block (b), and the ConvNeXt block (c) [1].\n### Kernel Size\nThe next exploration was done on the kernel size inside the inverted bottleneck block. Before experimenting with different kernel sizes, further modification was done to the structure of the block, where authors swapped the order of the first and second layer such that the depthwise convolution is now placed at the beginning of the block as seen in Figure 4 (c). Thanks to this modification, the block is now called _ConvNeXt block_ as it no longer completely resembles the original inverted bottleneck structure. This idea was actually adopted from Transformer, where the MSA (Multihead Self-Attention) layer is placed before the MLP layers. In the case of ConvNeXt, the depthwise convolution acts as the replacement of MSA, while the linear layers in MLP Transformers are replaced by pointwise convolutions. Simply moving up the depthwise convolution like this reduced the accuracy from 80.6% to 79.9%. However, this is acceptable because the current experiment set is still ongoing.\nExperiments on the kernel size was then applied only on the depthwise convolution layer, leaving the remaining pointwise convolutions unchanged. Here authors tried to use different kernel sizes, where they found that 7×7 worked best as it successfully recovered the accuracy back to 80.6% with lower computational complexity (4.6 vs 4.2 GFLOPS). Interestingly, this kernel size matches the window dimensions in the Swin Transformer architecture, which corresponds to the patch size used in the self-attention mechanism. You can actually see this in Figure 3 where the window sizes in Swin Transformer variants are all 7×7.\n### Micro Design\nThe final aspect tuned in the paper is the so-called _micro design_ , which essentially refers to the things related to the intricate details of the network. Similar to the previous ones, the parameters used here are mainly also adopted from Transformers. Authors initially replaced ReLU with GELU. Even though with this replacement the accuracy remained the same (80.6%), but they decided to go with this activation function for the subsequent experiments. The accuracy finally increased after the number of activation functions was reduced. Instead of applying GELU after each convolution layer in the ConvNeXt block, this activation function was placed only between the two pointwise convolutions. This modification allowed the network to boost the accuracy up to 81.3%, at which point this score was already on par with the Swin-T architecture while still having lower GFLOPS (4.2 vs 4.5).\nNext, it is a common practice to use _Conv-BN-ReLU_ structure in CNN-based architecture, which is exactly what ResNet implements as well. Instead of following this convention, authors decided to implement only a single batch normalization layer, which is placed before the first pointwise convolution layer. This change improved the accuracy to 81.4%, surpassing the accuracy of Swin-T by a little bit. Despite this achievement, parameter tuning was still continued by replacing batch norm with layer norm, which again raised the accuracy by 0.1% to 81.5%. All the modifications related to micro design resulted in the architecture shown in Figure 5 (the rightmost image). Here you can see how a ConvNeXt block differs from Swin Transformer and ResNet blocks.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/1x_2FqNNvmAzS9uq_PubJeA.png)\nFigure 5. What the Swin-T, ResNet-50 and ConvNeXt-T blocks look like at the initial stage [1].\nThe last thing the authors did related to the micro design was applying separate downsampling layers. In the original ResNet architecture, the spatial dimension of a tensor reduces by half when we move from one stage to another. You can see in Figure 2 that initially ResNet accepts input of size 224×224 which then shrinks to 112×112, 56×56, 28×28, 14×14, and 7×7 at stage _conv1_ , _conv2_x_ , _conv3_x_ , _conv4_x_ and _conv5_x_ , respectively. Especially in _conv2_x_ and the subsequent ones, the spatial dimension reduction is done by changing the stride parameter of the pointwise convolution to 2. Instead of doing so, ConvNeXt performs downsampling by placing another convolution layer right before the element-wise summation operation within the block. The kernel size and stride of this layer are set to 2, simulating a non-overlapping sliding window. In fact, it is mentioned in the paper that using this separate downsampling layer caused the accuracy to degrade instead. Nevertheless, authors managed to solve this issue by applying additional layer normalization layers at several parts of the network, i.e., before each downsampling layer, after the _stem_ stage and after the global average pooling layer (right before the final output layer). With this tuning, authors successfully boosted the accuracy to 82.0%, which is much higher than Swin-T (81.3%) while still having the exact same GFLOPS (4.5).\nAnd that’s basically all the modifications made on the original ResNet to create the ConvNeXt architecture. Don’t worry if it still feels a bit unclear for now — I believe things will become clearer as we get into the code.\n···\n## ConvNeXt Implementation\nFigure 6 below displays the details of the entire ConvNeXt-T architecture which we will later implement every single of its components one by one. Here you can also see how it differs from ResNet-50 and Swin-T, the two models that are comparable to ConvNeXt-T.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/1Posc_QEd2WO7PjOtgFJuTw.png)\nFigure 6. The details of the ResNet-50, ConvNeXt-T, and Swin-T architectures [1].\nWhen it comes to the implementation, the first thing we need to do is to import the required modules. The only two we import here are the base `torch` module and its `nn` submodule for loading neural network layers.\npython\n\n```\n# Codeblock 1\nimport torch\nimport torch.nn as nn\n```\n\n### ConvNeXt Block\nNow let’s start with the ConvNeXt block. You can see in Figure 6 that the block structures in _res2_ , _res3_ , _res4_ , and _res5_ stages are basically the same, in which all of those correspond to the rightmost illustration in Figure 5. Thanks to these identical structures, we can implement them in a single class and use it repeatedly. Look at the Codeblock 2a and 2b below to see how I do that.\npython\n\n```\n# Codeblock 2a\nclass ConvNeXtBlock(nn.Module):\n    def __init__(self, num_channels):         #(1)\n        super().__init__()\n        hidden_channels = num_channels * 4    #(2)\n\n        \n        self.conv0 = nn.Conv2d(in_channels=num_channels,         #(3) \n                               out_channels=num_channels,        #(4)\n                               kernel_size=7,    #(5)\n                               stride=1,\n                               padding=3,        #(6)\n                               groups=num_channels)              #(7)\n        \n        self.norm = nn.LayerNorm(normalized_shape=num_channels)  #(8)\n        \n        self.conv1 = nn.Conv2d(in_channels=num_channels,         #(9)\n                               out_channels=hidden_channels, \n                               kernel_size=1, \n                               stride=1, \n                               padding=0)\n        \n        self.gelu = nn.GELU()  #(10)\n        \n        self.conv2 = nn.Conv2d(in_channels=hidden_channels,      #(11)\n                               out_channels=num_channels, \n                               kernel_size=1, \n                               stride=1, \n                               padding=0)\n```\n\nI decided to name this class `ConvNeXtBlock`. You can see at line `#(1)` in the above codeblock that this class accepts `num_channels` as the only parameter, in which it denotes both the number of input and output channels. Remember that a ConvNeXt block follows the pattern of the inverted bottleneck structure, i.e., _narrow_ → _wide_ → _narrow_. If you take a closer look at Figure 6, you’ll notice that the _wide_ part is 4 times larger than the _narrow_ part. Thus, we set the value of the `hidden_channels` variable accordingly (`#(2)`). \nNext, we initialize 3 convolution layers which I refer to them as `conv0`, `conv1` and `conv2`. Every single of these convolution layers has their own specifications. For `conv0`, we set the number of input and output channels to be the same, which is the reason that both its `in_channels` and `out_channels` parameters are set to `num_channels` (`#(3–4)`). We set the kernel size of this layer to 7×7 (`#(5)`). Given this specification, we need to set the padding size to 3 in order to retain the spatial dimension (`#(6)`). Don’t forget to set the `groups` parameter to `num_channels` because we want this to be a depthwise convolution layer (`#(7)`). On the other hand, the `conv1` layer (`#(9)`) is responsible to increase the number of image channels, whereas the subsequent `conv2` layer (`#(11)`) is employed to shrink the tensor back to the original channel count. It is important to note that `conv1` and `conv2` are both using 1×1 kernel size, which essentially means that it only works by combining information along the channel dimension. Additionally, here we also need to initialize layer norm (`#(8)`) and GELU activation function (`#(10)`) as the replacement for batch norm and ReLU.\nAs all layers required in the ConvNeXtBlock have been initialized, what we need to do next is to define the flow of the tensor in the `forward()` method below.\npython\n\n```\n# Codeblock 2b\n    def forward(self, x):\n        residual = x                 #(1)\n        print(f'x & residual\\t: {x.size()}')\n        \n        x = self.conv0(x)\n        print(f'after conv0\\t: {x.size()}')\n        \n        x = x.permute(0, 2, 3, 1)    #(2)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = self.norm(x)\n        print(f'after norm\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)    #(3)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = self.conv1(x)\n        print(f'after conv1\\t: {x.size()}')\n        \n        x = self.gelu(x)\n        print(f'after gelu\\t: {x.size()}')\n        \n        x = self.conv2(x)\n        print(f'after conv2\\t: {x.size()}')\n        \n        x = x + residual             #(4)\n        print(f'after summation\\t: {x.size()}')\n        \n        return x\n```\n\nWhat we basically do in the above code is just passing the tensor to each layer we defined earlier sequentially. However, there are two things I need to highlight here. First, we need to store the original input tensor to the `residual` variable (`#(1)`), in which it will skip over all operations within the ConvNeXt block. Secondly, remember that layer norm is commonly used for sequential data, where it typically has a different shape from that of image data. Due to this reason, we need to adjust the tensor dimension such that the shape becomes _(N, H, W, C)_ (`#(2)`) before we actually perform the layer normalization operation. Afterwards, don’t forget to permute this tensor back to _(N, C, H, W)_ (`#(3)`). The resulting tensor is then passed through the remaining layers before being summed with the residual connection (`#(4)`).\nTo check if our `ConvNeXtBlock` class works properly, we can test it using the Codeblock 3 below. Here we are going to simulate the block used in _res2_ stage. So, we set the `num_channels` parameter to 96 (`#(1)`) and create a dummy tensor which we assume as a batch of single image of size 56×56 (`#(2)`).\npython\n\n```\n# Codeblock 3\nconvnext_block_test = ConvNeXtBlock(num_channels=96)  #(1)\nx_test = torch.rand(1, 96, 56, 56)  #(2)\n\nout_test = convnext_block_test(x_test)\n```\n\nBelow is what the resulting output looks like. Talking about the internal flow, it seems like all layers we stacked earlier work properly. At line `#(1)` in the output below we can see that the tensor dimension changed to 1×56×56×96 _(N, H, W, C)_ after being permuted. This tensor size then changed back to 1×96×56×56 _(N, C, H, W)_ after the second permute operation (`#(2)`). Next, the _conv1_ layer successfully expanded the number of channels to be 4 times greater than the input (`#(3)`) which was then reduced back to the original channel count (`#(4)`). Here you can see that the tensor shape at the first and the last layer are exactly the same, allowing us to stack multiple ConvNeXt blocks as many as we want.\npython\n\n```\n# Codeblock 3 Output\nx & residual    : torch.Size([1, 96, 56, 56])\nafter conv0     : torch.Size([1, 96, 56, 56])  \nafter permute   : torch.Size([1, 56, 56, 96])    #(1)\nafter norm      : torch.Size([1, 56, 56, 96])\nafter permute   : torch.Size([1, 96, 56, 56])    #(2)\nafter conv1     : torch.Size([1, 384, 56, 56])   #(3)\nafter gelu      : torch.Size([1, 384, 56, 56])\nafter conv2     : torch.Size([1, 96, 56, 56])    #(4)\nafter summation : torch.Size([1, 96, 56, 56])\n```\n\n···\n### ConvNeXt Block Transition\nThe next component I want to implement is the one I refer to as the _ConvNeXt block transition_. The idea of this block is actually similar to the ConvNeXt block we implemented earlier, except that this transition block is used when we are about to move from a stage to the subsequent one. More specifically, this block will later be employed as the first ConvNeXt block in each stage (except _res2_). The reason I implement it in separate class is that there are some intricate details that differ from the ConvNeXt block. Additionally, it is worth noting that the term _transition_ is not officially used in the paper. Rather, it’s just the word I use on my own to describe this idea. — I actually also used this technique back when I write about the smaller ResNet version, i.e., ResNet-18 and ResNet-34. Click on the link at reference number [6] at the end of this article if you’re interested to read that one.\npython\n\n```\n# Codeblock 4a\nclass ConvNeXtBlockTransition(nn.Module):\n    def __init__(self, in_channels, out_channels):  #(1)\n        super().__init__()\n        hidden_channels = out_channels * 4\n        \n        self.projection = nn.Conv2d(in_channels=in_channels,      #(2) \n                                    out_channels=out_channels, \n                                    kernel_size=1, \n                                    stride=2,\n                                    padding=0)\n        \n        self.conv0 = nn.Conv2d(in_channels=in_channels, \n                               out_channels=out_channels, \n                               kernel_size=7,\n                               stride=1,\n                               padding=3,\n                               groups=in_channels)\n        \n        self.norm0 = nn.LayerNorm(normalized_shape=out_channels)\n        \n        self.conv1 = nn.Conv2d(in_channels=out_channels, \n                               out_channels=hidden_channels, \n                               kernel_size=1, \n                               stride=1, \n                               padding=0)\n        \n        self.gelu = nn.GELU()\n        \n        self.conv2 = nn.Conv2d(in_channels=hidden_channels, \n                               out_channels=out_channels, \n                               kernel_size=1, \n                               stride=1,\n                               padding=0)\n        \n        self.norm1 = nn.LayerNorm(normalized_shape=out_channels)  #(3)\n\n        self.downsample = nn.Conv2d(in_channels=out_channels,     #(4)\n                                    out_channels=out_channels, \n                                    kernel_size=2, \n                                    stride=2)\n```\n\nThe first difference you might notice here is the input of the `__init__()` method, which in this case we separate the number of input and output channels into two parameters as seen at line `#(1)` in Codeblock 4a. This is essentially done because we need this block to take the output tensor from the previous stage which has different number of channels from that of the one to be generated in the subsequent stage. Referring to Figure 6, for example, if we were to create the first ConvNeXt block in _res3_ stage, we need to configure it such that it accepts a tensor of 96 channels from _res2_ and returns another tensor with 192 channels.\nSecondly, here we implement the _separate downsample layer_ I explained earlier (`#(4)`) alongside the corresponding layer norm to be placed before it (`#(3)`). As the name suggests, this layer is employed to reduce the spatial dimension of the image by half.\nThird, we initialize the so-called _projection layer_ at line `#(2)`. In the ConvNeXtBlock we created earlier, this layer is not necessary because the input and output tensor is exactly the same. In the case of _transition_ block, the image spatial dimension is reduced by half, while at the same time the number of output channels is doubled. This _projection_ layer is responsible to adjust the dimension of the residual connection in order to match it with the one from the main flow, allowing element-wise operation to be performed.\nThe `forward()` method in the Codeblock 4b below is also similar to the one belongs to the `ConvNeXtBlock` class, except that here the residual connection needs to be processed with the projection layer (`#(1)`) while the main tensor requires to be downsampled (`#(2)`) before the summation is done at line `#(3)`.\npython\n\n```\n# Codeblock 4b\n    def forward(self, x):\n        print(f'original\\t\\t: {x.size()}')\n\n        residual = self.projection(x)  #(1)\n        print(f'residual after proj\\t: {residual.size()}')\n        \n        x = self.conv0(x)\n        print(f'after conv0\\t\\t: {x.size()}')\n        \n        x = x.permute(0, 2, 3, 1)\n        print(f'after permute\\t\\t: {x.size()}')\n        \n        x = self.norm0(x)\n        print(f'after norm1\\t\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)\n        print(f'after permute\\t\\t: {x.size()}')\n        \n        x = self.conv1(x)\n        print(f'after conv1\\t\\t: {x.size()}')\n        \n        x = self.gelu(x)\n        print(f'after gelu\\t\\t: {x.size()}')\n        \n        x = self.conv2(x)\n        print(f'after conv2\\t\\t: {x.size()}')\n\n        x = x.permute(0, 2, 3, 1)\n        print(f'after permute\\t\\t: {x.size()}')\n        \n        x = self.norm1(x)\n        print(f'after norm1\\t\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)\n        print(f'after permute\\t\\t: {x.size()}')\n        \n        x = self.downsample(x)  #(2)\n        print(f'after downsample\\t: {x.size()}')\n        \n        x = x + residual  #(3)\n        print(f'after summation\\t\\t: {x.size()}')\n        \n        return x\n```\n\nNow let’s test the `ConvNeXtBlockTransition` class above using the following codeblock. Suppose we are about to implement the first ConvNeXt block in stage _res3_. To do so, we can simply instantiate the transition block with `in_channels=96` and `out_channels=192` before eventually passing a dummy tensor of size 1×96×56×56 through it.\npython\n\n```\n# Codeblock 5\nconvnext_block_transition_test = ConvNeXtBlockTransition(in_channels=96, \n                                                         out_channels=192)\nx_test = torch.rand(1, 96, 56, 56)\n\nout_test = convnext_block_transition_test(x_test)\n```\n\nmarkup\n\n```\n# Codeblock 5 Outputoriginal            : torch.Size([1, 96, 56, 56])residual after proj : torch.Size([1, 192, 28, 28])  #(1)after conv0         : torch.Size([1, 192, 56, 56])  #(2)after permute       : torch.Size([1, 56, 56, 192])after norm0         : torch.Size([1, 56, 56, 192])after permute       : torch.Size([1, 192, 56, 56])after conv1         : torch.Size([1, 768, 56, 56])after gelu          : torch.Size([1, 768, 56, 56])after conv2         : torch.Size([1, 192, 56, 56])  #(3)after permute       : torch.Size([1, 56, 56, 192])after norm1         : torch.Size([1, 56, 56, 192])after permute       : torch.Size([1, 192, 56, 56])after downsample    : torch.Size([1, 192, 28, 28])  #(4)after summation     : torch.Size([1, 192, 28, 28])  #(5)\n```\n\nYou can see in the resulting output that our projection layer directly maps the 1×96×56×56 residual tensor to 1×192×28×28 as shown at line `#(1)`. Meanwhile, the main tensor `x` needs to be processed by the other layers we initialized earlier to achieve this shape. The steps we performed from line `#(2)` to `#(3)` on the `x` tensor are basically the same as those in the `ConvNeXtBlock` class. At this point we already got the number of channels matches our need (192). The spatial dimension is then reduced after the tensor being processed by the `downsample` layer (`#(4)`). As the tensor dimensions of `x` and `residual` have matched, we can finally perform the element-wise summation (`#(5)`).\n···\n### The Entire ConvNeXt Architecture\nAs we got `ConvNeXtBlock` and `ConvNeXtBlockTransition` classes ready to use, we can now start to construct the entire ConvNeXt architecture. Before we do that, I would like to introduce some config parameters first. See the Codeblock 6 below.\npython\n\n```\n# Codeblock 6\nIN_CHANNELS  = 3     #(1)\nIMAGE_SIZE   = 224   #(2)\n\nNUM_BLOCKS   = [3, 3, 9, 3]         #(3)\nOUT_CHANNELS = [96, 192, 384, 768]  #(4)\nNUM_CLASSES  = 1000  #(5)\n```\n\nThe first one is the dimension of the input image. As shown at line `#(1)` and `#(2)`, here we set `in_channels` to 3 and `image_size` to 224 since by default ConvNeXt accepts a batch of RGB images of that size. The next ones are related to the model configuration. In this case, I set the number of ConvNeXt blocks of each stage to `[3, 3, 9, 3]` (`#(3)`) and the corresponding number of output channels to `[96, 192, 384, 768]` (`#(4)`) since I want to implement the ConvNeXt-T variant. You can actually change these numbers according to the configuration provided by the original paper shown in Figure 7. Finally, we set the number of neurons of the output channel to 1000, which corresponds to the number of classes in the dataset we train the model on (`#(5)`).\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/1EXPODqjCzuvV3CYMPv1H1w.png)\nFigure 7. The ConvNeXt variants [1].\nWe will now implement the entire architecture in the `ConvNeXt` class shown in Codeblock 7a and 7b below. The following `__init__()` method might seem a bit complicated at glance, but don’t worry as I’ll explain it thoroughly.\npython\n\n```\n# Codeblock 7a\nclass ConvNeXt(nn.Module):\n    def __init__(self):\n        super().__init__()\n        \n        self.stem = nn.Conv2d(in_channels=IN_CHANNELS,    #(1)\n                              out_channels=OUT_CHANNELS[0],\n                              kernel_size=4,\n                              stride=4,\n                             )\n\n        self.normstem = nn.LayerNorm(normalized_shape=OUT_CHANNELS[0])  #(2)\n        \n        #(3)\n        self.res2 = nn.ModuleList()\n        for _ in range(NUM_BLOCKS[0]):\n            self.res2.append(ConvNeXtBlock(num_channels=OUT_CHANNELS[0]))\n        \n        #(4)\n        self.res3 = nn.ModuleList([ConvNeXtBlockTransition(in_channels=OUT_CHANNELS[0], \n                                                           out_channels=OUT_CHANNELS[1])])\n        for _ in range(NUM_BLOCKS[1]-1):\n            self.res3.append(ConvNeXtBlock(num_channels=OUT_CHANNELS[1]))\n\n        #(5)\n        self.res4 = nn.ModuleList([ConvNeXtBlockTransition(in_channels=OUT_CHANNELS[1], \n                                                           out_channels=OUT_CHANNELS[2])])\n        for _ in range(NUM_BLOCKS[2]-1):\n            self.res4.append(ConvNeXtBlock(num_channels=OUT_CHANNELS[2]))\n\n        #(6)\n        self.res5 = nn.ModuleList([ConvNeXtBlockTransition(in_channels=OUT_CHANNELS[2], \n                                                           out_channels=OUT_CHANNELS[3])])\n        for _ in range(NUM_BLOCKS[3]-1):\n            self.res5.append(ConvNeXtBlock(num_channels=OUT_CHANNELS[3]))\n\n                \n        self.avgpool = nn.AdaptiveAvgPool2d(output_size=(1,1))  #(7)\n        self.normpool = nn.LayerNorm(normalized_shape=OUT_CHANNELS[3])  #(8)\n        self.fc = nn.Linear(in_features=OUT_CHANNELS[3],        #(9)\n                            out_features=NUM_CLASSES)\n        \n        self.relu = nn.ReLU()\n```\n\nThe first thing we do here is initializing the _stem_ stage (`#(1)`), which is essentially just a convolution layer with 4×4 kernel size and stride 4. This configuration will effectively reduce the image size to be 4 times smaller, where every single pixel in the output tensor represents a 4×4 patch in the input tensor. For the subsequent stages, we need to wrap the corresponding ConvNeXt blocks with `nn.ModuleList()`. For stage _res3_ (`#(4)`), _res4_ (`#(5)`) and _res5_ (`#(6)`) we place `ConvNeXtBlockTransition` at the beginning of each list as a “bridge” between stages. We don’t do this for stage _res2_ since the tensor produced by the _stem_ stage is already compatible with it (`#(3)`). Next, we initialize an `nn.AdaptiveAvgPool2d` layer, which will be used to reduce the spatial dimensions of the tensor to 1x1 by computing the mean across each channel (`#(7)`). In fact, this is the exact same process used by ResNet to prepare the tensor from the last convolution layer so that it matches the shape required by the subsequent output layer (`#(9)`). Additionally, don’t forget to initialize two layer normalization layers which I refer to as `normstem` (`#(2)`) and `normpool` (`#(8)`), in which these two layers will then be placed right after the `stem` stage and the `avgpool` layer.\nThe `forward()` method is pretty straightforward. All we need to do in the following code is just to place the layers one after another. Keep in mind that since the ConvNeXt blocks are stored in lists, we need to call them iteratively with loops as seen at line `#(1–4)`. Additionally, don’t forget to reshape the tensor produced by the `nn.AdaptiveAvgPool2d` layer (`#(5)`) so that it will be compatible with the subsequent fully-connected layer (`#(6)`).\npython\n\n```\n# Codeblock 7b\n    def forward(self, x):\n        print(f'original\\t: {x.size()}')\n        \n        x = self.relu(self.stem(x))\n        print(f'after stem\\t: {x.size()}')\n\n        x = x.permute(0, 2, 3, 1)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = self.normstem(x)\n        print(f'after normstem\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)\n        print(f'after permute\\t: {x.size()}')\n        \n        print()\n        for i, block in enumerate(self.res2):    #(1)\n            x = block(x)\n            print(f'after res2 #{i}\\t: {x.size()}')\n        \n        print()\n        for i, block in enumerate(self.res3):    #(2)\n            x = block(x)\n            print(f'after res3 #{i}\\t: {x.size()}')\n        \n        print()\n        for i, block in enumerate(self.res4):    #(3)\n            x = block(x)\n            print(f'after res4 #{i}\\t: {x.size()}')\n        \n        print()\n        for i, block in enumerate(self.res5):    #(4)\n            x = block(x)\n            print(f'after res5 #{i}\\t: {x.size()}')\n        \n        print()\n        x = self.avgpool(x)\n        print(f'after avgpool\\t: {x.size()}')\n\n        x = x.permute(0, 2, 3, 1)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = self.normpool(x)\n        print(f'after normpool\\t: {x.size()}')\n        \n        x = x.permute(0, 3, 1, 2)\n        print(f'after permute\\t: {x.size()}')\n        \n        x = x.reshape(x.shape[0], -1)             #(5)\n        print(f'after reshape\\t: {x.size()}')\n        \n        x = self.fc(x)\n        print(f'after fc\\t: {x.size()}')          #(6)\n        \n        return x\n```\n\nNow for the moment of truth, let’s see if we have correctly implemented the entire ConvNeXt model by running the following code. Here I try to pass a tensor of size 1×3×224×224 to the network, simulating a batch of a single RGB image of size 224×224.\npython\n\n```\n# Codeblock 8\nconvnext_test = ConvNeXt()\n\nx_test   = torch.rand(1, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nout_test = convnext_test(x_test)\n```\n\nYou can see in the following output that it looks like our implementation is correct as the behavior of the network aligns with the architectural design shown in Figure 6. The spatial dimension of the image gradually gets smaller as we get deeper into the network, and at the same time the number of channels increases instead thanks to the `ConvNeXtBlockTransition` blocks we placed at the beginning of stage _res3_ (`#(1)`), _res4_ (`#(2)`), and _res5_ (`#(3)`). The `avgpool` layer then correctly downsampled the spatial dimension to 1×1 (`#(4)`), allowing it to be connected to the output layer (`#(5)`).\nmarkup\n\n```\n# Codeblock 8 Outputoriginal       : torch.Size([1, 3, 224, 224])after stem     : torch.Size([1, 96, 56, 56])after permute  : torch.Size([1, 56, 56, 96])after normstem : torch.Size([1, 56, 56, 96])after permute  : torch.Size([1, 96, 56, 56])after res2 #0  : torch.Size([1, 96, 56, 56])after res2 #1  : torch.Size([1, 96, 56, 56])after res2 #2  : torch.Size([1, 96, 56, 56])after res3 #0  : torch.Size([1, 192, 28, 28])  #(1)after res3 #1  : torch.Size([1, 192, 28, 28])after res3 #2  : torch.Size([1, 192, 28, 28])after res4 #0  : torch.Size([1, 384, 14, 14])  #(2)after res4 #1  : torch.Size([1, 384, 14, 14])after res4 #2  : torch.Size([1, 384, 14, 14])after res4 #3  : torch.Size([1, 384, 14, 14])after res4 #4  : torch.Size([1, 384, 14, 14])after res4 #5  : torch.Size([1, 384, 14, 14])after res4 #6  : torch.Size([1, 384, 14, 14])after res4 #7  : torch.Size([1, 384, 14, 14])after res4 #8  : torch.Size([1, 384, 14, 14])after res5 #0  : torch.Size([1, 768, 7, 7])    #(3)after res5 #1  : torch.Size([1, 768, 7, 7])after res5 #2  : torch.Size([1, 768, 7, 7])after avgpool  : torch.Size([1, 768, 1, 1])    #(4)after permute  : torch.Size([1, 1, 1, 768])after normpool : torch.Size([1, 1, 1, 768])after permute  : torch.Size([1, 768, 1, 1])after reshape  : torch.Size([1, 768])after fc       : torch.Size([1, 1000])         #(5)\n```\n\n···\n## Ending\nWell, that was pretty much everything about the theory and the implementation of the ConvNeXt architecture. Again, I do acknowledge that the code I demonstrate above might not fully capture everything since this article is intended to cover the general idea of the model. So, I highly recommend you read the original implementation by Meta’s researchers [2] if you want to know more about the intricate details.\nI hope you find this article useful. Thanks for reading!\n_P.S. the notebook used in this article is available on my GitHub repo. See the link at reference number [7]._\n···\n## References\n[1] Zhuang Liu _et al_. A ConvNet for the 2020s. Arxiv. <https://arxiv.org/pdf/2201.03545> [Accessed January 18, 2025].\n[2] facebookresearch. ConvNeXt. GitHub. <https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py> [Accessed January 18, 2025].\n[3] Kaiming He _et al_. Deep Residual Learning for Image Recognition. Arxiv. <https://arxiv.org/pdf/1512.03385> [Accessed January 18, 2025].\n[4] Ze Liu _et al_. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Arxiv. <https://arxiv.org/pdf/2103.14030> [Accessed January 18, 2025].\n[5] Saining Xie _et al_. Aggregated Residual Transformations for Deep Neural Networks. Arxiv. <https://arxiv.org/pdf/1611.05431> [Accessed January 18, 2025].\n[6] [Muhammad Ardi](https://medium.com/u/9801a58700ac). Paper Walkthrough: Residual Network (ResNet). Python in Plain English. <https://python.plainenglish.io/paper-walkthrough-residual-network-resnet-62af58d1c521> [Accessed January 19, 2025].\n[7] MuhammadArdiPutra. The CNN That Challenges ViT — ConvNeXt. GitHub. <https://github.com/MuhammadArdiPutra/medium_articles/blob/main/The%20CNN%20That%20Challenges%20ViT%20-%20ConvNeXt.ipynb> [Accessed January 24, 2025].\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657598287_taxhya.jpeg)\nWritten By\n## [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nDeep learning storyteller\n[See all from Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Computer Vision](https://towardsdatascience.com/tag/computer-vision/)[Image Processing](https://towardsdatascience.com/tag/image-processing/)[Neural Network](https://towardsdatascience.com/tag/neural-network/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F&title=ConvNeXt%20Paper%20Walkthrough%3A%20The%20CNN%20That%20Challenges%20ViT)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F&title=ConvNeXt%20Paper%20Walkthrough%3A%20The%20CNN%20That%20Challenges%20ViT)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-cnn-that-challenges-vit%2F&text=ConvNeXt%20Paper%20Walkthrough%3A%20The%20CNN%20That%20Challenges%20ViT)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/0HmU34ZoPDIPVeOED-scaled.jpg)](https://towardsdatascience.com/machine-learning-an-ethics-review-5f4c35112be5/)\n## [Machine Learning - An Ethics Review](https://towardsdatascience.com/machine-learning-an-ethics-review-5f4c35112be5/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nI'm looking back at my projects from my Data Science program at Flatiron and reviewing the ethical concerns that each has...\n[Mike Erb](https://towardsdatascience.com/author/merb92/)\nNovember 3, 20209 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/09/0lTQasT50agNl7PGY.jpeg)](https://towardsdatascience.com/a-brief-history-of-embedded-ai-6109be99199d/)\n## [A Brief History of Embedded AI](https://towardsdatascience.com/a-brief-history-of-embedded-ai-6109be99199d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow sensor hubs on smartphones helped bring artificial intelligence to the edge\n[Kenneth Joel](https://towardsdatascience.com/author/kennethjoel/)\nSeptember 28, 20224 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/10/image-2.jpeg)](https://towardsdatascience.com/how-to-control-a-robot-with-python/)\n## [How to Control a Robot with Python](https://towardsdatascience.com/how-to-control-a-robot-with-python/)\n[Robotics](https://towardsdatascience.com/category/robotics/)\n3D simulations and movement control with PyBullet\n[Mauro Di Pietro](https://towardsdatascience.com/author/maurodp/)\nOctober 23, 20259 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/1Z8_jCOCyTZJAPd6AXjR4AA.png)](https://towardsdatascience.com/image-processing-with-python-application-of-fourier-transformation-5a8584dc175b/)\n## [Image Processing with Python - Application of Fourier Transformation](https://towardsdatascience.com/image-processing-with-python-application-of-fourier-transformation-5a8584dc175b/)\n[Programming](https://towardsdatascience.com/category/programming/)\nHow to make use of the Fourier Transformation to remove image elements\n[Tonichi Edeza](https://towardsdatascience.com/author/tonichi-edeza/)\nJanuary 28, 20213 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/0RCLHdecd0L3IQzvS-scaled.jpg)](https://towardsdatascience.com/are-you-aware-how-difficult-your-regression-problem-is-b7dae830652b/)\n## [Are You Aware How Difficult Your Regression Problem Is?](https://towardsdatascience.com/are-you-aware-how-difficult-your-regression-problem-is-b7dae830652b/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nHalf of the solution is comprehending your problem, but half of the comprehension is knowing how complicated it is\n[Houssem Ben Braiek](https://towardsdatascience.com/author/houssem.benbraiek/)\nJanuary 31, 20226 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/1S9UMBnXntNzzgRENKNeinw.jpeg)](https://towardsdatascience.com/the-evolution-of-big-data-compute-platforms-past-now-and-later-7c46697366d9/)\n## [The Evolution of Big Data Compute Platforms - Past, Now and Later](https://towardsdatascience.com/the-evolution-of-big-data-compute-platforms-past-now-and-later-7c46697366d9/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA journey into the evolution of Big Data Compute Platforms like Hadoop and Spark. Sharing my perspective on where we were, where we are...\n[Manoj Kukreja](https://towardsdatascience.com/author/mkukreja1/)\nDecember 27, 20207 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/11/1nhYMydDVKn41s4vBWKr6kw.jpeg)](https://towardsdatascience.com/10-amazing-machine-learning-visualizations-you-should-know-in-2023-528282940582/)\n## [10 Amazing Machine Learning Visualizations You Should Know in 2023](https://towardsdatascience.com/10-amazing-machine-learning-visualizations-you-should-know-in-2023-528282940582/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nYellowbrick for creating machine learning plots with less code\n[Rukshan Pramoditha](https://towardsdatascience.com/author/rukshanpramoditha/)\nNovember 5, 202215 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1yEVKMhO1NT7Kg9Tf9QnBSA-scaled.jpeg)](https://towardsdatascience.com/nested-class-imbalance-problem-within-machine-learning-based-multi-class-classification-a4b2c43ba832/)\n## [Nested Class Imbalance Problem Within Machine Learning-Based Multi-Class Classification](https://towardsdatascience.com/nested-class-imbalance-problem-within-machine-learning-based-multi-class-classification-a4b2c43ba832/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nShedding the light on real-life's non-human-centric data in which a hidden nested class imbalance problem exists, which is -up to the...\n[Ahmed Masri](https://towardsdatascience.com/author/ahmed-masri/)\nApril 21, 20216 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/0wCeuEHjTNeomk9E2-scaled.jpg)](https://towardsdatascience.com/how-to-solve-optimisation-problems-using-linear-programming-912cc951afbb/)\n## [How To Solve Optimization Problems Using Linear Programming](https://towardsdatascience.com/how-to-solve-optimisation-problems-using-linear-programming-912cc951afbb/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn introduction to linear programming and how to solve it using the graphical and simplex algorithms\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nJuly 24, 20236 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n",
  "body_base_path": "attachments/ac847f5a52fe/pages/",
  "body_markdown_path": "attachments/ac847f5a52fe/pages/archived-body.md",
  "local_html": "attachments/ac847f5a52fe/pages/archived-body.html",
  "body_source": "prior_capture"
}