{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Muhammad Ardi Putra",
  "title_listed": "CPTR Paper Walkthrough: Image Captioning, Transformer Mode On",
  "keywords": "",
  "domain": "图像处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://towardsdatascience.com/image-captioning-transformer-mode-on/",
  "publish_date": "2025-03-07",
  "corpus_stage": "",
  "notes": "",
  "id": "3d5614bc7b74",
  "collected_at": "2026-10-02T02:58:33",
  "title": "CPTR Paper Walkthrough: Image Captioning, Transformer Mode On | Towards Data Science",
  "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# CPTR Paper Walkthrough: Image Captioning, Transformer Mode On\nImplementing CPTR (CaPtion TransformeR) from scratch with PyTorch\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nMarch 7, 202530 min read\n![Black and white photo of a person standing, looking at a wall of art](../images/000.jpg)\nPhoto by [Antenna](https://unsplash.com/@antenna%20) on [Unsplash](https://unsplash.com/)\n## Introduction\nIn my previous article, I discussed one of the earliest deep learning approaches for image captioning. If you’re interested in reading it, you can find the link to that article at the end of this one.\nToday, I would like to talk about image captioning again, but this time with the more advanced neural network architecture. The deep learning I am going to talk about is the one proposed in the paper titled “ _CPTR: Full Transformer Network for Image Captioning_ ,” written by Liu _et al._ back in 2021 [1]. Specifically, here I will reproduce the model proposed in the paper and explain the underlying theory behind the architecture. However, keep in mind that I won’t actually demonstrate the training process since I only want to focus on the model architecture.\n### The idea behind CPTR\nIn fact, the main idea of the CPTR architecture is exactly the same as the earlier image captioning model, as both use the encoder-decoder structure. Previously, in the paper titled “ _Show and Tell: A Neural Image Caption Generator_ ” [2], the models used are GoogLeNet (a.k.a. Inception V1) and LSTM for the two components, respectively. The illustration of the model proposed in the _Show and Tell_ paper is shown in the following figure.\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n![](../images/001.png)\nFigure 1. The neural network architecture for image captioning proposed in the Show and Tell paper [2].\nDespite having the same encoder-decoder structure, what makes CPTR different from the previous approach is the basis of the encoder and the decoder themselves. In CPTR, we combine the encoder part of the ViT (Vision Transformer) model with the decoder part of the original Transformer model. The use of transformer-based architecture for both components is essentially where the name CPTR comes from: CaPtion TransformeR.\nNote that the discussions in this article are going to be highly related to ViT and Transformer, so I highly recommend you read my previous article about these two topics if you’re not yet familiar with them. You can find the links at the end of this article.\nFigure 2 shows what the original ViT architecture looks like. Everything inside the green box is the encoder part of the architecture to be adopted as the CPTR encoder.\n![](../images/002.png)\nFigure 2. The Vision Transformer (ViT) architecture [3].\nNext, Figure 3 displays the original Transformer architecture. The components enclosed in the blue box are the layers that we are going to implement in the CPTR decoder.\n![](../images/body-a000d3ae44bd0cf84cc1.png)\nFigure 3. The original Transformer architecture [4].\nIf we combine the components inside the green and blue boxes above, we are going to obtain the architecture shown in Figure 4 below. This is exactly what the CPTR model we are going to implement looks like. The idea here is that the ViT Encoder (green) works by encoding the input image into a specific tensor representation which will then be used as the basis of the Transformer Decoder (blue) to generate the corresponding caption.\n![](../images/body-2b651976b51610bf75a3.png)\nFigure 4. The CPTR architecture [5].\nThat’s pretty much everything you need to know for now. I’ll explain more about the details as we go through the implementation.\n## Module imports & parameter configuration\nAs always, the first thing we need to do in the code is to import the required modules. In this case, we only import torch and torch.nn since we are about to implement the model from scratch.\npython\n\n```\n# Codeblock 1\nimport torch\nimport torch.nn as nn\n```\n\nNext, we are going to initialize some parameters in Codeblock 2. If you have read my previous article about image captioning with GoogLeNet and LSTM, you’ll notice that here, we got a lot more parameters to initialize. In this article, I want to reproduce the CPTR model as closely as possible to the original one, so the parameters mentioned in the paper will be used in this implementation.\npython\n\n```\n# Codeblock 2\nBATCH_SIZE         = 1              #(1)\n\nIMAGE_SIZE         = 384            #(2)\nIN_CHANNELS        = 3              #(3)\n\nSEQ_LENGTH         = 30             #(4)\nVOCAB_SIZE         = 10000          #(5)\n\nEMBED_DIM          = 768            #(6)\nPATCH_SIZE         = 16             #(7)\nNUM_PATCHES        = (IMAGE_SIZE//PATCH_SIZE) ** 2  #(8)\nNUM_ENCODER_BLOCKS = 12             #(9)\nNUM_DECODER_BLOCKS = 4              #(10)\nNUM_HEADS          = 12             #(11)\nHIDDEN_DIM         = EMBED_DIM * 4  #(12)\nDROP_PROB          = 0.1            #(13)\n```\n\nThe first parameter I want to explain is the `BATCH_SIZE`, which is written at the line marked with `#(1)`. The number assigned to this variable is not quite important in our case since we are not actually going to train this model. This parameter is set to 1 because, by default, PyTorch treats input tensors as a batch of samples. Here I assume that we only have a single sample in a batch. \nNext, remember that in the case of image captioning we are dealing with images and texts simultaneously. This essentially means that we need to set the parameters for the two. It is mentioned in the paper that the model accepts an RGB image of size 384×384 for the encoder input. Hence, we assign the values for `IMAGE_SIZE` and `IN_CHANNELS` variables based on this information (`#(2)` and `#(3)`). On the other hand, the paper does not mention the parameters for the captions. So, here I assume that the length of the caption is no more than 30 words (`#(4)`), with the vocabulary size estimated at 10000 unique words (`#(5)`).\nThe remaining parameters are related to the model configuration. Here we set the `EMBED_DIM` variable to 768 (`#(6)`). In the encoder side, this number indicates the length of the feature vector that represents each 16×16 image patch (`#(7)`). The same concept also applies to the decoder side, but in that case the feature vector will represent a single word in the caption. Talking more specifically about the `PATCH_SIZE` parameter, we are going to use the value to compute the total number of patches in the input image. Since the image has the size of 384×384, there will be 576 patches in total (`#(8)`).\nWhen it comes to using an encoder-decoder architecture, it is possible to specify the number of encoder and decoder blocks to be used. Using more blocks typically allows the model to perform better in terms of the accuracy, yet in return, it will require more computational power. The authors of this paper decided to stack 12 encoder blocks (`#(9)`) and 4 decoder blocks (`#(10)`). Next, since CPTR is a transformer-based model, it is necessary to specify the number of attention heads within the attention blocks inside the encoders and the decoders, which in this case authors use 12 attention heads (`#(11)`). The value for the `HIDDEN_DIM` parameter is not mentioned anywhere in the paper. However, according to the ViT and the Transformer paper, this parameter is configured to be 4 times larger than `EMBED_DIM` (`#(12)`). The dropout rate is not mentioned in the paper either. Hence, I arbitrarily set `DROP_PROB` to 0.1 (`#(13)`).\n## Encoder\nAs the modules and parameters have been set up, now that we will get into the encoder part of the network. In this section we are going to implement and explain every single component inside the green box in Figure 4 one by one.\n### Patch embedding\n![](../images/body-6b159a5730767cc33b10.png)\nFigure 5. Dividing the input image into patches and converting them into vectors [5].\nYou can see in Figure 5 above that the first step to be done is dividing the input image into patches. This is essentially done because instead of focusing on local patterns like CNNs, ViT captures global context by learning the relationships between these patches. We can model this process with the `Patcher` class shown in the Codeblock 3 below. For the sake of simplicity, here I also include the process inside the _patch embedding_ block within the same class.\npython\n\n```\n# Codeblock 3\nclass Patcher(nn.Module):\n   def __init__(self):\n       super().__init__()\n\n       #(1)\n       self.unfold = nn.Unfold(kernel_size=PATCH_SIZE, stride=PATCH_SIZE)\n\n       #(2)\n       self.linear_projection = nn.Linear(in_features=IN_CHANNELS*PATCH_SIZE*PATCH_SIZE,\n                                          out_features=EMBED_DIM)\n      \n   def forward(self, images):\n       print(f'images\\t\\t: {images.size()}')\n       images = self.unfold(images)  #(3)\n       print(f'after unfold\\t: {images.size()}')\n      \n       images = images.permute(0, 2, 1)  #(4)\n       print(f'after permute\\t: {images.size()}')\n      \n       features = self.linear_projection(images)  #(5)\n       print(f'after lin proj\\t: {features.size()}')\n      \n       return features\n```\n\nThe patching itself is done using the `nn.Unfold` layer (`#(1)`). Here we need to set both the `kernel_size` and `stride` parameters to `PATCH_SIZE (16)` so that the resulting patches do not overlap with each other. This layer also automatically flattens these patches once it is applied to the input image. Meanwhile, the `nn.Linear layer` (`#(2)`) is employed to perform linear projection, i.e., the process done by the _patch embedding_ block. By setting the `out_features` parameter to `EMBED_DIM`, this layer will map every single flattened patch into a feature vector of length 768.\nThe entire process should make more sense once you read the `forward()` method. You can see at line `#(3)` in the same codeblock that the input image is directly processed by the unfold layer. Next, we need to process the resulting tensor with the `permute()` method (`#(4)`) to swap the first and the second axis before feeding it to the `linear_projection` layer (`#(5)`). Additionally, here I also print out the tensor dimension after each layer so that you can better understand the transformation made at each step.\nIn order to check if our `Patcher` class works properly, we can just pass a dummy tensor through the network. Look at the Codeblock 4 below to see how I do it.\npython\n\n```\n# Codeblock 4\npatcher  = Patcher()\n\nimages   = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nfeatures = patcher(images)\n```\n\npython\n\n```\n# Codeblock 4 Output\nimages         : torch.Size([1, 3, 384, 384])\nafter unfold   : torch.Size([1, 768, 576])  #(1)\nafter permute  : torch.Size([1, 576, 768])  #(2)\nafter lin proj : torch.Size([1, 576, 768])  #(3)\n```\n\nThe tensor I passed above represents an RGB image of size 384×384. Here we can see that after the unfold operation is performed, the tensor dimension changed to 1×768×576 (`#(1)`), denoting the flattened 3×16×16 patch for each of the 576 patches. Unfortunately, this output shape does not match what we need. Remember that in ViT, we perceive image patches as a sequence, so we need to swap the 1st and 2nd axes because typically, the 1st dimension of a tensor represents the temporal axis, while the 2nd one represents the feature vector of each timestep. As the `permute()` operation is performed, our tensor is now having the dimension of 1×576×768 (`#(2)`). Lastly, we pass this tensor through the linear projection layer, which the resulting tensor shape remains the same since we set the `EMBED_DIM` parameter to the same size (768) (`#(3)`). Despite having the same dimension, the information contained in the final tensor should be richer thanks to the transformation applied by the trainable weights of the linear projection layer.\n### Learnable positional embedding\n![](../images/body-90652a40b2d322f3209d.png)\nFigure 6. Injecting the learnable positional embeddings into the embedded image patches [5].\nAfter the input image has successfully been converted into a sequence of patches, the next thing to do is to inject the so-called _positional embedding_ tensor. This is essentially done because a transformer without positional embedding is permutation-invariant, meaning that it treats the input sequence as if their order does not matter. Interestingly, since an image is not a literal sequence, we should set the positional embedding to be _learnable_ such that it will be able to somewhat reorder the patch sequence that it thinks works best in representing the spatial information. However, keep in mind that the term “reordering” here does not mean that we physically rearrange the sequence. Rather, it does so by adjusting the embedding weights.\nThe implementation is pretty simple. All we need to do is just to initialize a tensor using `nn.Parameter` which the dimension is set to match with the output from the `Patcher` model, i.e., 576×768. Also, don’t forget to write `requires_grad=True` just to ensure that the tensor is trainable. Look at the Codeblock 5 below for the details.\npython\n\n```\n# Codeblock 5\nclass LearnableEmbedding(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.learnable_embedding = nn.Parameter(torch.randn(size=(NUM_PATCHES, EMBED_DIM)),\n                                               requires_grad=True)\n      \n   def forward(self):\n       pos_embed = self.learnable_embedding\n       print(f'learnable embedding\\t: {pos_embed.size()}')\n      \n       return pos_embed\n```\n\nNow let’s run the following codeblock to see whether our `LearnableEmbedding` class works properly. You can see in the printed output that it successfully created the positional embedding tensor as expected.\npython\n\n```\n# Codeblock 6\nlearnable_embedding = LearnableEmbedding()\n\npos_embed = learnable_embedding()\n```\n\ntext\n\n```\n# Codeblock 6 Output\nlearnable embedding : torch.Size([576, 768])\n```\n\n### The main encoder block\n![](../images/body-fac850f893528cfd337d.png)\nFigure 7. The main encoder block [5].\nThe next thing we are going to do is to construct the main encoder block displayed in the Figure 7 above. Here you can see that this block consists of several sub-components, namely _self-attention_ , _layer norm_ , FFN (Feed-Forward Network), and another _layer norm_. The Codeblock 7a below shows how I initialize these layers inside the `__init__()` method of the `EncoderBlock` class.\npython\n\n```\n# Codeblock 7a\nclass EncoderBlock(nn.Module):\n   def __init__(self):\n       super().__init__()\n      \n       #(1)\n       self.self_attention = nn.MultiheadAttention(embed_dim=EMBED_DIM,\n                                                   num_heads=NUM_HEADS,\n                                                   batch_first=True,  #(2)\n                                                   dropout=DROP_PROB)\n      \n       self.layer_norm_0 = nn.LayerNorm(EMBED_DIM)  #(3)\n      \n       self.ffn = nn.Sequential(  #(4)\n           nn.Linear(in_features=EMBED_DIM, out_features=HIDDEN_DIM),\n           nn.GELU(),\n           nn.Dropout(p=DROP_PROB),\n           nn.Linear(in_features=HIDDEN_DIM, out_features=EMBED_DIM),\n       )\n      \n       self.layer_norm_1 = nn.LayerNorm(EMBED_DIM)  #(5)\n```\n\nI’ve previously mentioned that the idea of ViT is to capture the relationships between patches within an image. This process is done by the _multihead attention_ layer I initialize at line `#(1)` in the above codeblock. One thing to keep in mind here is that we need to set the batch_first parameter to `True` (`#(2)`). This is essentially done so that the attention layer will be compatible with our tensor shape, in which the batch dimension (`batch_size`) is at the 0th axis of the tensor. Next, the two layer normalization layers need to be initialized separately, as shown at line `#(3)` and `#(5)`. Lastly, we initialize the FFN block at line `#(4)`, which the layers stacked using `nn.Sequential` follows the structure defined in the following equation.\n![](../images/body-a2ec509a63a7a58620f9.png)\nFigure 8. The operations done inside the FFN block [1].\nAs the `__init__()` method is complete, we will now continue with the `forward()` method. Let’s take a look at the Codeblock 7b below.\npython\n\n```\n# Codeblock 7b\n   def forward(self, features):  #(1)\n      \n       residual = features  #(2)\n       print(f'features & residual\\t: {residual.size()}')\n      \n       #(3)\n       features, self_attn_weights = self.self_attention(query=features,\n                                                         key=features,\n                                                         value=features)\n       print(f'after self attention\\t: {features.size()}')\n       print(f\"self attn weights\\t: {self_attn_weights.shape}\")\n      \n       features = self.layer_norm_0(features + residual)  #(4)\n       print(f'after norm\\t\\t: {features.size()}')\n      \n\n       residual = features\n       print(f'\\nfeatures & residual\\t: {residual.size()}')\n      \n       features = self.ffn(features)  #(5)\n       print(f'after ffn\\t\\t: {features.size()}')\n      \n       features = self.layer_norm_1(features + residual)\n       print(f'after norm\\t\\t: {features.size()}')\n      \n       return features\n```\n\nHere you can see that the input tensor is named `features (#(1)`). I name it this way because the input of the `EncoderBlock` is the image that has already been processed with `Patcher` and `LearnableEmbedding`, instead of a raw image. Before doing anything, notice in the `encoder` block that there is a branch separated from the main flow which then returns back to the normalization layer. This branch is commonly known as a _residual connection_. To implement this, we need to store the original input tensor to the residual variable as I demonstrate at line `#(2)`. As the input tensor has been copied, now we are ready to process the original input with the multihead attention layer (`#(3)`). Since this is a _self_ -attention (not a _cross_ -attention), the `query`, `key`, and `value` inputs for this layer are all derived from the `features` tensor. Next, the layer normalization operation is then performed at line `#(4)`, which the input for this layer already contains information from the attention block as well as the residual connection. The remaining steps are basically the same as what I just explained, except that here we replace the self-attention block with FFN (`#(5)`).\nIn the following codeblock, I’ll test the `EncoderBlock` class by passing a dummy tensor of size 1×576×768, simulating an output tensor from the previous operations.\npython\n\n```\n# Codeblock 8\nencoder_block = EncoderBlock()\n\nfeatures = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM)\nfeatures = encoder_block(features)\n```\n\nBelow is what the tensor dimension looks like throughout the entire process inside the model.\npython\n\n```\n# Codeblock 8 Output\nfeatures & residual  : torch.Size([1, 576, 768])  #(1)\nafter self attention : torch.Size([1, 576, 768])\nself attn weights    : torch.Size([1, 576, 576])  #(2)\nafter norm           : torch.Size([1, 576, 768])\n\nfeatures & residual  : torch.Size([1, 576, 768])\nafter ffn            : torch.Size([1, 576, 768])  #(3)\nafter norm           : torch.Size([1, 576, 768])  #(4)\n```\n\nHere you can see that the final output tensor (`#(4)`) has the same size as the input (`#(1)`), allowing us to stack multiple encoder blocks without having to worry about messing up the tensor dimensions. Not only that, the size of the tensor also appears to be unchanged from the beginning all the way to the last layer. In fact, there are actually lots of transformations performed inside the attention block, but we just can’t see it since the entire process is done internally by the `nn.MultiheadAttention` layer. One of the tensors produced in the layer that we can observe is the attention weight (`#(2)`). This weight matrix, which has the size of 576×576, is responsible for storing information regarding the relationships between one patch and every other patch in the image. Furthermore, changes in tensor dimension actually also happened inside the FFN layer. The feature vector of each patch which has the initial length of 768 changed to 3072 and immediately shrunk back to 768 again (`#(3)`). However, this transformation is not printed since the process is wrapped with `nn.Sequential` back at line #(4) in Codeblock 7a.\n### ViT encoder\n![](../images/body-23e7000a7a8f06980ada.png)\nFigure 9. The entire ViT Encoder in the CPTR architecture [5].\nAs we have finished implementing all encoder components, now that we will assemble them to construct the actual ViT Encoder. We are going to do it in the `Encoder` class in Codeblock 9.\npython\n\n```\n# Codeblock 9\nclass Encoder(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.patcher = Patcher()  #(1)\n       self.learnable_embedding = LearnableEmbedding()  #(2)\n\n       #(3)\n       self.encoder_blocks = nn.ModuleList(EncoderBlock() for _ in range(NUM_ENCODER_BLOCKS))\n  \n   def forward(self, images):  #(4)\n       print(f'images\\t\\t\\t: {images.size()}')\n      \n       features = self.patcher(images)  #(5)\n       print(f'after patcher\\t\\t: {features.size()}')\n      \n       features = features + self.learnable_embedding()  #(6)\n       print(f'after learn embed\\t: {features.size()}')\n      \n       for i, encoder_block in enumerate(self.encoder_blocks):\n           features = encoder_block(features)  #(7)\n           print(f\"after encoder block #{i}\\t: {features.shape}\")\n\n       return features\n```\n\nInside the` __init__()` method, what we need to do is to initialize all components we created earlier, i.e., `Patcher` (`#(1)`), `LearnableEmbedding` (`#(2)`), and `EncoderBlock` (`#(3)`). In this case, the `EncoderBlock` is initialized inside `nn.ModuleList` since we want to repeat it `NUM_ENCODER_BLOCKS` (12) times. To the `forward()` method, it initially works by accepting raw image as the input (`#(4)`). We then process it with the `patcher` layer (`#(5)`) to divide the image into small patches and transform them with the linear projection operation. The learnable positional embedding tensor is then injected into the resulting output by element-wise addition (`#(6)`). Lastly, we pass it into the 12 encoder blocks sequentially with a simple for loop (`#(7)`).\nNow, in Codeblock 10, I am going to pass a dummy image through the entire encoder. Note that since I want to focus on the flow of this Encoder class, I re-run the previous classes we created earlier with the `print()` functions commented out so that the outputs will look neat.\npython\n\n```\n# Codeblock 10\nencoder = Encoder()\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nfeatures = encoder(images)\n```\n\nAnd below is what the flow of the tensor looks like. Here, we can see that our dummy input image successfully passed through all layers in the network, including the encoder blocks that we repeat 12 times. The resulting output tensor is now context-aware, meaning that it already contains information about the relationships between patches within the image. Therefore, this tensor is now ready to be processed further with the decoder, which will later be discussed in the subsequent section.\npython\n\n```\n# Codeblock 10 Output\nimages                  : torch.Size([1, 3, 384, 384])\nafter patcher           : torch.Size([1, 576, 768])\nafter learn embed       : torch.Size([1, 576, 768])\nafter encoder block #0  : torch.Size([1, 576, 768])\nafter encoder block #1  : torch.Size([1, 576, 768])\nafter encoder block #2  : torch.Size([1, 576, 768])\nafter encoder block #3  : torch.Size([1, 576, 768])\nafter encoder block #4  : torch.Size([1, 576, 768])\nafter encoder block #5  : torch.Size([1, 576, 768])\nafter encoder block #6  : torch.Size([1, 576, 768])\nafter encoder block #7  : torch.Size([1, 576, 768])\nafter encoder block #8  : torch.Size([1, 576, 768])\nafter encoder block #9  : torch.Size([1, 576, 768])\nafter encoder block #10 : torch.Size([1, 576, 768])\nafter encoder block #11 : torch.Size([1, 576, 768])\n```\n\n### ViT encoder (alternative)\nI want to show you something before we talk about the decoder. If you think that our approach above is too complicated, it is actually possible for you to use `nn.TransformerEncoderLayer` from PyTorch so that you don’t need to implement the `EncoderBlock` class from scratch. To do so, I am going to reimplement the `Encoder` class, but this time I’ll name it `EncoderTorch`.\npython\n\n```\n# Codeblock 11\nclass EncoderTorch(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.patcher = Patcher()\n       self.learnable_embedding = LearnableEmbedding()\n      \n       #(1)\n       encoder_block = nn.TransformerEncoderLayer(d_model=EMBED_DIM,\n                                                  nhead=NUM_HEADS,\n                                                  dim_feedforward=HIDDEN_DIM,\n                                                  dropout=DROP_PROB,\n                                                  batch_first=True)\n      \n       #(2)\n       self.encoder_blocks = nn.TransformerEncoder(encoder_layer=encoder_block,\n                                                   num_layers=NUM_ENCODER_BLOCKS)\n  \n   def forward(self, images):\n       print(f'images\\t\\t\\t: {images.size()}')\n      \n       features = self.patcher(images)\n       print(f'after patcher\\t\\t: {features.size()}')\n      \n       features = features + self.learnable_embedding()\n       print(f'after learn embed\\t: {features.size()}')\n      \n       features = self.encoder_blocks(features)  #(3)\n       print(f'after encoder blocks\\t: {features.size()}')\n\n       return features\n```\n\nWhat we basically do in the above codeblock is that instead of using the EncoderBlock class, here we use `nn.TransformerEncoderLayer` (`#(1)`), which will automatically create a single encoder block based on the parameters we pass to it. To repeat it multiple times, we can just use nn.`TransformerEncoder` and pass a number to the `num_layers` parameter (`#(2)`). With this approach, we don’t necessarily need to write the forward pass in a loop like what we did earlier (`#(3)`).\nThe testing code in the Codeblock 12 below is exactly the same as the one in Codeblock 10, except that here I use the `EncoderTorch` class. You can also see here that the output is basically the same as the previous one.\npython\n\n```\n# Codeblock 12\nencoder_torch = EncoderTorch()\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nfeatures = encoder_torch(images)\n```\n\npython\n\n```\n# Codeblock 12 Output\nimages               : torch.Size([1, 3, 384, 384])\nafter patcher        : torch.Size([1, 576, 768])\nafter learn embed    : torch.Size([1, 576, 768])\nafter encoder blocks : torch.Size([1, 576, 768])\n```\n\n## Decoder\nAs we have successfully created the encoder part of the CPTR architecture, now that we will talk about the decoder. In this section I am going to implement every single component inside the blue box in Figure 4. Based on the figure, we can see that the decoder accepts two inputs, i.e., the image caption ground truth (the lower part of the blue box) and the sequence of embedded patches produced by the encoder (the arrow coming from the green box). It is important to know that the architecture drawn in Figure 4 is intended to illustrate the training phase, where the entire caption ground truth is fed into the decoder. Later in the inference phase, we only provide a <BOS> (_Beginning of Sentence_) token for the caption input. The decoder will then predict each word sequentially based on the given image and the previously generated words. This process is commonly known as an _autoregressive_ mechanism.\n### Sinusoidal positional embedding\n![](../images/body-d327aa2f936ca3b4de94.png)\nFigure 10. Where the sinusoidal positional embedding component is located in the decoder [5].\nIf you take a look at the CPTR model, you’ll see that the first step in the decoder is to convert each word into the corresponding feature vector representation using the _word embedding_ block. However, since this step is very easy, we are going to implement it later. Now let’s assume that this word vectorization process is already done, so we can move to the positional embedding part.\nAs I’ve mentioned earlier, since transformer is permutation-invariant by nature, we need to apply positional embedding to the input sequence. Different from the previous one, here we use the so-called _sinusoidal positional embedding_. We can think of it like a method to label each word vector by assigning numbers obtained from a sinusoidal wave. By doing so, we can expect our model to understand word orders thanks to the information given by the wave patterns.\nIf you go back to Codeblock 6 Output, you’ll see that the positional embedding tensor in the encoder has the size of `NUM_PATCHES` × `EMBED_DIM` (576×768). What we basically want to do in the decoder is to create a tensor having the size of `SEQ_LENGTH` × `EMBED_DIM` (30×768), which the values are computed based on the equation shown in Figure 11. This tensor is then set to be non-trainable because a sequence of words must maintain a fixed order to preserve its meaning.\n![](../images/body-cac59b5e5c3619142e68.png)\nFigure 11. The equation for creating sinusoidal positional encoding proposed in the Transformer paper [6].\nHere I want to explain the following code quickly because I actually have discussed this more thoroughly in my previous article about Transformer. Generally speaking, what we basically do here is to create the sine and cosine wave using `torch.sin()` (`#(1)`) and `torch.cos()` (`#(2)`). The resulting two tensors are then merged using the code at line `#(3)` and `#(4)`.\npython\n\n```\n# Codeblock 13\nclass SinusoidalEmbedding(nn.Module):\n   def forward(self):\n       pos = torch.arange(SEQ_LENGTH).reshape(SEQ_LENGTH, 1)\n       print(f\"pos\\t\\t: {pos.shape}\")\n      \n       i = torch.arange(0, EMBED_DIM, 2)\n       denominator = torch.pow(10000, i/EMBED_DIM)\n       print(f\"denominator\\t: {denominator.shape}\")\n      \n       even_pos_embed = torch.sin(pos/denominator)  #(1)\n       odd_pos_embed  = torch.cos(pos/denominator)  #(2)\n       print(f\"even_pos_embed\\t: {even_pos_embed.shape}\")\n      \n       stacked = torch.stack([even_pos_embed, odd_pos_embed], dim=2)  #(3)\n       print(f\"stacked\\t\\t: {stacked.shape}\")\n\n       pos_embed = torch.flatten(stacked, start_dim=1, end_dim=2)  #(4)\n       print(f\"pos_embed\\t: {pos_embed.shape}\")\n      \n       return pos_embed\n```\n\nNow we can check if the `SinusoidalEmbedding` class above works properly by running the Codeblock 14 below. As expected earlier, here you can see that the resulting tensor has the size of 30×768. This dimension matches with the tensor obtained by the process done in the _word embedding_ block, allowing them to be summed in an element-wise manner.\npython\n\n```\n# Codeblock 14\nsinusoidal_embedding = SinusoidalEmbedding()\npos_embed = sinusoidal_embedding()\n```\n\npython\n\n```\n# Codeblock 14 Output\npos            : torch.Size([30, 1])\ndenominator    : torch.Size([384])\neven_pos_embed : torch.Size([30, 384])\nstacked        : torch.Size([30, 384, 2])\npos_embed      : torch.Size([30, 768])\n```\n\n### Look-ahead mask\n![](../images/body-3a02494252b3e0d4a353.png)\nFigure 12. A look-ahead mask needs to be applied to the masked-self attention layer [5].\nThe next thing I am going to talk about in the decoder is the _masked self-attention_ layer highlighted in the above figure. I am not going to code the attention mechanism from scratch. Rather, I’ll only implement the so-called _look-ahead mask_ , which will be useful for the self-attention layer so that it doesn’t attend to the subsequent words in the caption during the training phase.\nThe way to do it is pretty easy, what we need to do is just to create a triangular matrix which the size is set to match with the attention weight matrix, i.e., `SEQ_LENGTH` × `SEQ_LENGTH` (30×30). Look at the `create_mask()`function below for the details.\npython\n\n```\n# Codeblock 15\ndef create_mask(seq_length):\n   mask = torch.tril(torch.ones((seq_length, seq_length)))  #(1)\n   mask[mask == 0] = -float('inf')  #(2)\n   mask[mask == 1] = 0  #(3)\n   return mask\n```\n\nEven though creating a triangular matrix can simply be done with `torch.tril()` and `torch.ones()` (`#(1)`), but here we need to make a little modification by changing the 0 values to _-inf_ (`#(2)`) and the 1s to 0 (`#(3)`). This is essentially done because the `nn.MultiheadAttention` layer applies the mask by element-wise addition. By assigning _-inf_ to the subsequent words, the attention mechanism will completely ignore them. Again, the internal process inside an attention layer has also been discussed in detail in[ my previous article about transformer](https://contributor.insightmediagroup.io/paper-walkthrough-attention-is-all-you-need-80399cdc59e1).\nNow I am going to run the function with `seq_length=7` so that you can see what the mask actually looks like. Later in the complete flow, we need to set the `seq_length` parameter to `SEQ_LENGTH` (30) so that it matches with the actual caption length.\npython\n\n```\n# Codeblock 16\nmask_example = create_mask(seq_length=7)\nmask_example\n```\n\npython\n\n```\n# Codeblock 16 Output\ntensor([[0., -inf, -inf, -inf, -inf, -inf, -inf],\n       [0., 0., -inf, -inf, -inf, -inf, -inf],\n       [0., 0., 0., -inf, -inf, -inf, -inf],\n       [0., 0., 0., 0., -inf, -inf, -inf],\n       [0., 0., 0., 0., 0., -inf, -inf],\n       [0., 0., 0., 0., 0., 0., -inf],\n       [0., 0., 0., 0., 0., 0., 0.]])\n```\n\n### The main decoder block\n![](../images/body-5425d3ce16026d48365c.png)\nFigure 13. The main decoder block [5].\nWe can see in the above figure that the structure of the decoder block is a bit longer than that of the encoder block. It seems like everything is nearly the same, except that the decoder part has a _cross-attention_ mechanism and an additional layer normalization step placed after it. This cross-attention layer can actually be perceived as the bridge between the encoder and the decoder, as it is employed to capture the relationships between each word in the caption and every single patch in the input image. The two arrows coming from the encoder are the _key_ and _value_ inputs for the attention layer, whereas the _query_ is derived from the previous layer in the decoder itself. Look at the Codeblock 17a and 17b below to see the implementation of the entire decoder block.\npython\n\n```\n# Codeblock 17a\nclass DecoderBlock(nn.Module):\n   def __init__(self):\n       super().__init__()\n      \n       #(1)\n       self.self_attention = nn.MultiheadAttention(embed_dim=EMBED_DIM,\n                                                   num_heads=NUM_HEADS,\n                                                   batch_first=True,\n                                                   dropout=DROP_PROB)\n       #(2)\n       self.layer_norm_0 = nn.LayerNorm(EMBED_DIM)\n       #(3)\n       self.cross_attention = nn.MultiheadAttention(embed_dim=EMBED_DIM,\n                                                    num_heads=NUM_HEADS,\n                                                    batch_first=True,\n                                                    dropout=DROP_PROB)\n\n       #(4)\n       self.layer_norm_1 = nn.LayerNorm(EMBED_DIM)\n      \n       #(5)      \n       self.ffn = nn.Sequential(\n           nn.Linear(in_features=EMBED_DIM, out_features=HIDDEN_DIM),\n           nn.GELU(),\n           nn.Dropout(p=DROP_PROB),\n           nn.Linear(in_features=HIDDEN_DIM, out_features=EMBED_DIM),\n       )\n      \n       #(6)\n       self.layer_norm_2 = nn.LayerNorm(EMBED_DIM)\n```\n\nIn the `__init__()` method, we first initialize both self-attention (`#(1)`) and cross-attention (`#(3)`) layers with `nn.MultiheadAttention`. These two layers appear to be exactly the same now, but later you’ll see the difference in the `forward()` method. The three layer normalization operations are initialized separately as shown at line `#(2)`, `#(4)` and `#(6)`, since each of them will contain different normalization parameters. Lastly, the `ffn` layer (`#(5)`) is exactly the same as the one in the encoder, which basically follows the equation back in Figure 8.\nTalking about the `forward()` method below, it initially works by accepting three inputs: `features`, `captions`, and `attn_mask`, which each of them denotes the tensor coming from the encoder, the tensor from the decoder itself, and a look-ahead mask, respectively (`#(1)`). The remaining steps are somewhat similar to that of the `EncoderBlock`, except that here we repeat the multihead attention block twice. The first attention mechanism takes captions as the `query`, `key`, and `value` parameters (`#(2)`). This is essentially done because we want the layer to capture the context within the captions tensor itself — hence the name _self-attention_. Here we also need to pass the attn_mask parameter to this layer so that it cannot see the subsequent words during the training phase. The second attention mechanism is different (`#(3)`). Since we want to combine the information from the encoder and the decoder, we need to pass the `captions` tensor as the `query`, whereas the `features` tensor will be passed as the `key` and `value` — hence the name _cross-attention_. A look-ahead mask is not necessary in the cross-attention layer since later in the inference phase the model will be able to see the entire input image at once rather than looking at the patches one by one. As the tensor has been processed by the two attention layers, we will then pass it through the feed forward network (`#(4)`). Lastly, don’t forget to create the residual connections and apply the layer normalization steps after each sub-component.\npython\n\n```\n# Codeblock 17b\n   def forward(self, features, captions, attn_mask):  #(1)\n       print(f\"attn_mask\\t\\t: {attn_mask.shape}\")\n       residual = captions\n       print(f\"captions & residual\\t: {captions.shape}\")\n      \n       #(2)\n       captions, self_attn_weights = self.self_attention(query=captions,\n                                                         key=captions,\n                                                         value=captions,\n                                                         attn_mask=attn_mask)\n       print(f\"after self attention\\t: {captions.shape}\")\n       print(f\"self attn weights\\t: {self_attn_weights.shape}\")\n      \n       captions = self.layer_norm_0(captions + residual)\n       print(f\"after norm\\t\\t: {captions.shape}\")\n      \n      \n       print(f\"\\nfeatures\\t\\t: {features.shape}\")\n       residual = captions\n       print(f\"captions & residual\\t: {captions.shape}\")\n      \n       #(3)\n       captions, cross_attn_weights = self.cross_attention(query=captions,\n                                                           key=features,\n                                                           value=features)\n       print(f\"after cross attention\\t: {captions.shape}\")\n       print(f\"cross attn weights\\t: {cross_attn_weights.shape}\")\n      \n       captions = self.layer_norm_1(captions + residual)\n       print(f\"after norm\\t\\t: {captions.shape}\")\n      \n       residual = captions\n       print(f\"\\ncaptions & residual\\t: {captions.shape}\")\n      \n       captions = self.ffn(captions)  #(4)\n       print(f\"after ffn\\t\\t: {captions.shape}\")\n      \n       captions = self.layer_norm_2(captions + residual)\n       print(f\"after norm\\t\\t: {captions.shape}\")\n      \n       return captions\n\n\n```\n\nAs the `DecoderBlock` class is completed, we can now test it with the following code.\npython\n\n```\n# Codeblock 18\ndecoder_block = DecoderBlock()\n\nfeatures = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM)  #(1)\ncaptions = torch.randn(BATCH_SIZE, SEQ_LENGTH, EMBED_DIM)   #(2)\nlook_ahead_mask = create_mask(seq_length=SEQ_LENGTH)  #(3)\n\ncaptions = decoder_block(features, captions, look_ahead_mask)\n```\n\nHere we assume that features is a tensor containing a sequence of patch embeddings produced by the `encoder` (`#(1)`), while captions is a sequence of embedded words (`#(2)`). The `seq_length` parameter of the look-ahead mask is set to `SEQ_LENGTH` (30) to match it to the number of words in the caption (`#(3)`). The tensor dimensions after each step are displayed in the following output.\npython\n\n```\n# Codeblock 18 Output\nattn_mask             : torch.Size([30, 30])\ncaptions & residual   : torch.Size([1, 30, 768])\nafter self attention  : torch.Size([1, 30, 768])\nself attn weights     : torch.Size([1, 30, 30])    #(1)\nafter norm            : torch.Size([1, 30, 768])\n\nfeatures              : torch.Size([1, 576, 768])\ncaptions & residual   : torch.Size([1, 30, 768])\nafter cross attention : torch.Size([1, 30, 768])\ncross attn weights    : torch.Size([1, 30, 576])   #(2)\nafter norm            : torch.Size([1, 30, 768])\n\ncaptions & residual   : torch.Size([1, 30, 768])\nafter ffn             : torch.Size([1, 30, 768])\nafter norm            : torch.Size([1, 30, 768])\n```\n\nHere we can see that our `DecoderBlock` class works properly as it successfully processed the input tensors all the way to the last layer in the network. Here I want you to take a closer look at the attention weights at lines `#(1)` and `#(2)`. Based on these two lines, we can confirm that our decoder implementation is correct since the attention weight produced by the self-attention layer has the size of 30×30 (`#(1)`), which basically means that this layer really captured the context within the input caption. Meanwhile, the attention weight matrix generated by the cross-attention layer has the size of 30×576 (`#(2)`), indicating that it successfully captured the relationships between the words and the patches. This essentially implies that after cross-attention operation is performed, the resulting captions tensor has been enriched with the information from the image.\n### Transformer decoder\n![](../images/body-36e9660848f4d4da495e.png)\nFigure 14. The entire Transformer Decoder in the CPTR architecture [5].\nNow that we have successfully created all components for the entire decoder, what I am going to do next is to put them together into a single class. Look at the Codeblock 19a and 19b below to see how I do that.\npython\n\n```\n# Codeblock 19a\nclass Decoder(nn.Module):\n   def __init__(self):\n       super().__init__()\n\n       #(1)\n       self.embedding = nn.Embedding(num_embeddings=VOCAB_SIZE,\n                                     embedding_dim=EMBED_DIM)\n\n       #(2)\n       self.sinusoidal_embedding = SinusoidalEmbedding()\n\n       #(3)\n       self.decoder_blocks = nn.ModuleList(DecoderBlock() for _ in range(NUM_DECODER_BLOCKS))\n\n       #(4)\n       self.linear = nn.Linear(in_features=EMBED_DIM,\n                               out_features=VOCAB_SIZE)\n```\n\nIf you compare this `Decoder` class with the `Encoder` class from codeblock 9, you’ll notice that they are somewhat similar in terms of the structure. In the encoder, we convert image patches into vectors using `Patcher`, while in the decoder we convert every single word in the caption into a vector using the `nn.Embedding layer` (`#(1)`), which I haven’t explained earlier. Afterward, we initialize the positional embedding layer, where for the decoder we use the _sinusoidal_ rather than the _trainable_ one (`#(2)`). Next, we stack multiple decoder blocks using `nn.ModuleList` (`#(3)`). The linear layer written at line #(4), which doesn’t exist in the encoder, is necessary to be implemented here since it will be responsible to map each of the embedded words into a vector of length `VOCAB_SIZE` (10000). Later on, this vector will contain the logit of every word in the dictionary, and what we need to do afterward is just to take the index containing the highest value, i.e., the most likely word to be predicted.\nThe flow of the tensors within the `forward()` method itself is also pretty similar to the one in the `Encoder` class. In the Codeblock 19b below we pass features, captions, and `attn_mask` as the input (`#(1)`). Keep in mind that in this case the captions tensor contains the raw word sequence, so we need to vectorize these words with the embedding layer beforehand (`#(2)`). Next, we inject the sinusoidal positional embedding tensor using the code at line `#(3)` before eventually passing it through the four decoder blocks sequentially (`#(4)`). Finally, we pass the resulting tensor through the last linear layer to obtain the `prediction` logits (`#(5)`).\npython\n\n```\n# Codeblock 19b\n   def forward(self, features, captions, attn_mask):  #(1)\n       print(f\"features\\t\\t: {features.shape}\")\n       print(f\"captions\\t\\t: {captions.shape}\")\n      \n       captions = self.embedding(captions)  #(2)\n       print(f\"after embedding\\t\\t: {captions.shape}\")\n      \n       captions = captions + self.sinusoidal_embedding()  #(3)\n       print(f\"after sin embed\\t\\t: {captions.shape}\")\n      \n       for i, decoder_block in enumerate(self.decoder_blocks):\n           captions = decoder_block(features, captions, attn_mask)  #(4)\n           print(f\"after decoder block #{i}\\t: {captions.shape}\")\n      \n       captions = self.linear(captions)  #(5)\n       print(f\"after linear\\t\\t: {captions.shape}\")\n      \n       return captions\n```\n\nAt this point you might be wondering why we don’t implement the softmax activation function as drawn in the illustration. This is essentially because during the training phase, softmax is typically included within the loss function, whereas in the inference phase, the index of the largest value will remain the same regardless of whether softmax is applied.\nNow let’s run the following testing code to check whether there are errors in our implementation. Previously I mentioned that the captions input of the `Decoder` class is a raw word sequence. To simulate this, we can simply create a sequence of random integers ranging between 0 and `VOCAB_SIZE` (10000) with the length of `SEQ_LENGTH` (30) words (`#(1)`).\npython\n\n```\n# Codeblock 20\ndecoder = Decoder()\n\nfeatures = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH))  #(1)\n\ncaptions = decoder(features, captions, look_ahead_mask)\n```\n\nAnd below is what the resulting output looks like. Here you can see in the last line that the linear layer produced a tensor of size 30×10000, indicating that our decoder model is now capable of predicting the logit scores for each word in the vocabulary across all 30 sequence positions.\npython\n\n```\n# Codeblock 20 Output\nfeatures               : torch.Size([1, 576, 768])\ncaptions               : torch.Size([1, 30])\nafter embedding        : torch.Size([1, 30, 768])\nafter sin embed        : torch.Size([1, 30, 768])\nafter decoder block #0 : torch.Size([1, 30, 768])\nafter decoder block #1 : torch.Size([1, 30, 768])\nafter decoder block #2 : torch.Size([1, 30, 768])\nafter decoder block #3 : torch.Size([1, 30, 768])\nafter linear           : torch.Size([1, 30, 10000])\n```\n\n### Transformer decoder (alternative)\nIt is actually also possible to make the code simpler by replacing the `DecoderBlock` class with the `nn.TransformerDecoderLayer`, just like what we did in the ViT Encoder. Below is what the code looks like if we use this approach instead.\npython\n\n```\n# Codeblock 21\nclass DecoderTorch(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.embedding = nn.Embedding(num_embeddings=VOCAB_SIZE,\n                                     embedding_dim=EMBED_DIM)\n      \n       self.sinusoidal_embedding = SinusoidalEmbedding()\n      \n       #(1)\n       decoder_block = nn.TransformerDecoderLayer(d_model=EMBED_DIM,\n                                                  nhead=NUM_HEADS,\n                                                  dim_feedforward=HIDDEN_DIM,\n                                                  dropout=DROP_PROB,\n                                                  batch_first=True)\n      \n       #(2)\n       self.decoder_blocks = nn.TransformerDecoder(decoder_layer=decoder_block,\n                                                   num_layers=NUM_DECODER_BLOCKS)\n      \n       self.linear = nn.Linear(in_features=EMBED_DIM,\n                               out_features=VOCAB_SIZE)\n      \n   def forward(self, features, captions, tgt_mask):\n       print(f\"features\\t\\t: {features.shape}\")\n       print(f\"captions\\t\\t: {captions.shape}\")\n      \n       captions = self.embedding(captions)\n       print(f\"after embedding\\t\\t: {captions.shape}\")\n      \n       captions = captions + self.sinusoidal_embedding()\n       print(f\"after sin embed\\t\\t: {captions.shape}\")\n      \n       #(3)\n       captions = self.decoder_blocks(tgt=captions,\n                                      memory=features,\n                                      tgt_mask=tgt_mask)\n       print(f\"after decoder blocks\\t: {captions.shape}\")\n      \n       captions = self.linear(captions)\n       print(f\"after linear\\t\\t: {captions.shape}\")\n      \n       return captions\n```\n\nThe main difference you will see in the `__init__()` method is the use of `nn.TransformerDecoderLayer` and `nn.TransformerDecoder` at line `#(1)` and `#(2)`, where the former is used to initialize a single decoder block, and the latter is for repeating the block multiple times. Next, the `forward()` method is mostly similar to the one in the `Decoder` class, except that the forward propagation on the decoder blocks is automatically repeated four times without needing to be put inside a loop (`#(3)`). One thing that you need to pay attention to in the `decoder_blocks` layer is that the tensor coming from the encoder (features) must be passed as the argument for the `memory` parameter. Meanwhile, the tensor from the decoder itself (captions) has to be passed as the input to the `tgt` parameter.\nThe testing code for the `DecoderTorch` model below is basically the same as the one written in Codeblock 20. Here you can see that this model also generates the final output tensor of size 30×10000.\npython\n\n```\n# Codeblock 22\ndecoder_torch = DecoderTorch()\n\nfeatures = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH))\n\ncaptions = decoder_torch(features, captions, look_ahead_mask)\n```\n\npython\n\n```\n# Codeblock 22 Output\nfeatures             : torch.Size([1, 576, 768])\ncaptions             : torch.Size([1, 30])\nafter embedding      : torch.Size([1, 30, 768])\nafter sin embed      : torch.Size([1, 30, 768])\nafter decoder blocks : torch.Size([1, 30, 768])\nafter linear         : torch.Size([1, 30, 10000])\n```\n\n## The entire CPTR model\nFinally, it’s time to put the encoder and the decoder part we just created into a single class to actually construct the CPTR architecture. You can see in Codeblock 23 below that the implementation is very simple. All we need to do here is just to initialize the encoder (`#(1)`) and the decoder (`#(2)`) components, then pass the raw images and the corresponding caption ground truths as well as the look-ahead mask to the `forward()` method (#(3)). Additionally, it is also possible for you to replace the `Encoder` and the `Decoder` with `EncoderTorch` and `DecoderTorch`, respectively.\npython\n\n```\n# Codeblock 23\nclass EncoderDecoder(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.encoder = Encoder()  #EncoderTorch()  #(1)\n       self.decoder = Decoder()  #DecoderTorch()  #(2)\n      \n   def forward(self, images, captions, look_ahead_mask):  #(3)\n       print(f\"images\\t\\t\\t: {images.shape}\")\n       print(f\"captions\\t\\t: {captions.shape}\")\n      \n       features = self.encoder(images)\n       print(f\"after encoder\\t\\t: {features.shape}\")\n      \n       captions = self.decoder(features, captions, look_ahead_mask)\n       print(f\"after decoder\\t\\t: {captions.shape}\")\n      \n       return captions\n```\n\nWe can do the testing by passing dummy tensors through it. See the Codeblock 24 below for the details. In this case, images is basically just a tensor of random numbers having the dimension of 1×3×384×384 (`#(1)`), while captions is a tensor of size 1×30 containing random integers (`#(2)`).\npython\n\n```\n# Codeblock 24\nencoder_decoder = EncoderDecoder()\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)  #(1)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH))  #(2)\n\ncaptions = encoder_decoder(images, captions, look_ahead_mask)\n```\n\nBelow is what the output looks like. We can see here that our input images and captions successfully went through all layers in the network, which basically means that the CPTR model we created is now ready to actually be trained on image captioning datasets.\npython\n\n```\n# Codeblock 24 Output\nimages         : torch.Size([1, 3, 384, 384])\ncaptions       : torch.Size([1, 30])\nafter encoder  : torch.Size([1, 576, 768])\nafter decoder  : torch.Size([1, 30, 10000])\n```\n\n## Ending\nThat was pretty much everything about the theory and implementation of the CaPtion TransformeR architecture. Let me know what deep learning architecture I should implement next. Feel free to leave a comment if you spot any mistakes in this article!\n_The code used in this article is available in_[ _my GitHub repo_](https://github.com/MuhammadArdiPutra/medium_articles/blob/main/Image%20Captioning%2C%20Transformer%20Mode%20On.ipynb) _. Here’s the link to my previous article about_[ _image captioning_](https://contributor.insightmediagroup.io/show-and-tell-e1a1142456e2/) _,_[_Vision Transformer (ViT)_](https://contributor.insightmediagroup.io/paper-walkthrough-vision-transformer-vit-c5dcf76f1a7a)_, and the original_[ _Transformer_](https://contributor.insightmediagroup.io/paper-walkthrough-attention-is-all-you-need-80399cdc59e1) _._\n## References\n[1] Wei Liu _et al._ CPTR: Full Transformer Network for Image Captioning. Arxiv.[ https://arxiv.org/pdf/2101.10804](https://arxiv.org/pdf/2101.10804) [Accessed November 16, 2024].\n[2] Oriol Vinyals _et al._ Show and Tell: A Neural Image Caption Generator. Arxiv.[ https://arxiv.org/pdf/1411.4555](https://arxiv.org/pdf/1411.4555) [Accessed December 3, 2024].\n[3] Image originally created by author based on: Alexey Dosovitskiy _et al._ An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. Arxiv.[ https://arxiv.org/pdf/2010.11929](https://arxiv.org/pdf/2010.11929) [Accessed December 3, 2024].\n[4] Image originally created by author based on [6].\n[5] Image originally created by author based on [1].\n[6] Ashish Vaswani _et al._ Attention Is All You Need. Arxiv.[ https://arxiv.org/pdf/1706.03762](https://arxiv.org/pdf/1706.03762) [Accessed December 3, 2024].\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-59e1c17daab1fcb582c5.jpeg)\nWritten By\n## [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nDeep learning storyteller\n[See all from Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Image Captioning](https://towardsdatascience.com/tag/image-captioning/)[Transformer](https://towardsdatascience.com/tag/transformer/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F&title=CPTR%20Paper%20Walkthrough%3A%20Image%20Captioning%2C%20Transformer%20Mode%20On)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F&title=CPTR%20Paper%20Walkthrough%3A%20Image%20Captioning%2C%20Transformer%20Mode%20On)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F&text=CPTR%20Paper%20Walkthrough%3A%20Image%20Captioning%2C%20Transformer%20Mode%20On)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](../images/003.jpg)](https://towardsdatascience.com/you-dont-need-a-job-to-get-data-science-experience-26af4fd4c200/)\n## [You Don't Need A Job To Get Data Science Experience](https://towardsdatascience.com/you-dont-need-a-job-to-get-data-science-experience-26af4fd4c200/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nFlipping the coin on its head\n[Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/)\nOctober 4, 20205 min read\n  * [![](../images/004.png)](https://towardsdatascience.com/image-data-generators-in-keras-7c5fc6928400/)\n## [Image Data Generators in Keras](https://towardsdatascience.com/image-data-generators-in-keras-7c5fc6928400/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow to effectively and efficiently use data generators in Keras for Computer Vision applications of Deep Learning\n[Manpreet Singh Minhas](https://towardsdatascience.com/author/manpreetsinghminhas/)\nApril 24, 20194 min read\n  * [![](../images/005.jpg)](https://towardsdatascience.com/can-quantum-machine-learning-help-us-solve-climate-change-3500741d583b/)\n## [Can Quantum Machine Learning Help Us Solve Climate Change?](https://towardsdatascience.com/can-quantum-machine-learning-help-us-solve-climate-change-3500741d583b/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nThe history and potential use cases of quantum-derived technology\n[Daniel García Solla](https://towardsdatascience.com/author/cardstdani/)\nJuly 13, 202219 min read\n  * [![](../images/006.jpeg)](https://towardsdatascience.com/multi-label-land-cover-classification-with-deep-learning-d39ce2944a3d/)\n## [Multi-label Land Cover Classification with Deep Learning](https://towardsdatascience.com/multi-label-land-cover-classification-with-deep-learning-d39ce2944a3d/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA step by step guide on Classifying Multi-label Land cover classification using Deep Neural Networks\n[Abdishakur](https://towardsdatascience.com/author/shakasom/)\nNovember 26, 20195 min read\n  * [![](../images/007.jpg)](https://towardsdatascience.com/how-to-use-claude-code-to-build-a-minimum-viable-product/)\n## [How to Use Claude Code to Build a Minimum Viable Product](https://towardsdatascience.com/how-to-use-claude-code-to-build-a-minimum-viable-product/)\n[Agentic AI](https://towardsdatascience.com/category/artificial-intelligence/agentic-ai/)\nLearn how to effectively present product ideas by building MVPs with coding agents\n[Eivind Kjosbakken](https://towardsdatascience.com/author/oieivind/)\nApril 8, 20267 min read\n  * [![](../images/008.jpeg)](https://towardsdatascience.com/detecting-scams-using-ai-for-real-6c96e712cf09/)\n## [Detecting scams using AI, for real](https://towardsdatascience.com/detecting-scams-using-ai-for-real-6c96e712cf09/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA year-long journey to develop one of the best fraud prevention systems on the market\n[Massimo Belloni](https://towardsdatascience.com/author/massibelloni/)\nJuly 21, 20197 min read\n  * [![](../images/009.png)](https://towardsdatascience.com/understanding-cuped-a822523641af/)\n## [Understanding CUPED](https://towardsdatascience.com/understanding-cuped-a822523641af/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAn in-depth guide to the state-of-the-art variance reduction technique for A/B tests\n[Matteo Courthoud](https://towardsdatascience.com/author/matteo.courthoud/)\nJune 29, 20224 min read\n  * [![](../images/010.png)](https://towardsdatascience.com/i-tried-fine-tuning-a-robot-ai-model-on-colab-here-is-what-worked/)\n## [I Fine-Tuned OpenVLA-7B with LoRA on Google Colab: What Worked and What 100 Steps Proved](https://towardsdatascience.com/i-tried-fine-tuning-a-robot-ai-model-on-colab-here-is-what-worked/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA reproducible 100-step LoRA fine-tuning run for OpenVLA, with dataset checks, Colab setup, training metrics, and W&B evidence.\n[Abdullahi Dattijo](https://towardsdatascience.com/author/abdullahi.dattijo/)\nJuly 21, 202612 min read\n  * [![](../images/011.png)](https://towardsdatascience.com/multi-headed-self-attention-by-hand-d2ce1ae031db/)\n## [Multi-Headed Self Attention - By Hand](https://towardsdatascience.com/multi-headed-self-attention-by-hand-d2ce1ae031db/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHand computing the cornerstone of modern AI.\n[Daniel Warfield](https://towardsdatascience.com/author/danielwarfield1/)\nJuly 12, 20245 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "CPTR Paper Walkthrough: Image Captioning, Transformer Mode On | Towards Data Science",
    "description": "Implementing CPTR (CaPtion TransformeR) from scratch with PyTorch",
    "keywords": null,
    "author": "Muhammad Ardi Putra",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "CPTR Paper Walkthrough: Image Captioning, Transformer Mode On",
    "og:description": "Implementing CPTR (CaPtion TransformeR) from scratch with PyTorch",
    "og:url": "https://towardsdatascience.com/image-captioning-transformer-mode-on/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Caption-transformer.jpg",
    "og:image:width": "1080",
    "og:image:height": "786",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Muhammad Ardi Putra",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "30 min read",
    "article:published_time": "2025-03-07T13:47:02.000Z",
    "article:modified_time": "2025-03-07T13:47:02.000Z",
    "article:author": "https://towardsdatascience.com/author/muhammad_ardi/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 202,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_000.txt",
      "chars": 49
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_001.txt",
      "chars": 570
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_002.txt",
      "chars": 765
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_003.txt",
      "chars": 135
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_004.txt",
      "chars": 214
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_005.txt",
      "chars": 421
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_006.txt",
      "chars": 92
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_007.txt",
      "chars": 66
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_008.txt",
      "chars": 784
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_009.txt",
      "chars": 971
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_010.txt",
      "chars": 140
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_011.txt",
      "chars": 389
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_012.txt",
      "chars": 824
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_013.txt",
      "chars": 133
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_014.txt",
      "chars": 805
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_015.txt",
      "chars": 1176
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_016.txt",
      "chars": 150
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_017.txt",
      "chars": 221
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_018.txt",
      "chars": 784
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_019.txt",
      "chars": 95
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_020.txt",
      "chars": 214
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_021.txt",
      "chars": 194
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_022.txt",
      "chars": 69
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_023.txt",
      "chars": 325
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_024.txt",
      "chars": 1159
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_025.txt",
      "chars": 1726
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_026.txt",
      "chars": 299
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_027.txt",
      "chars": 672
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_028.txt",
      "chars": 530
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_029.txt",
      "chars": 751
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_030.txt",
      "chars": 224
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_031.txt",
      "chars": 470
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_032.txt",
      "chars": 1702
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_033.txt",
      "chars": 235
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_034.txt",
      "chars": 308
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_035.txt",
      "chars": 628
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_036.txt",
      "chars": 262
    },
    {
      "language": "txt",
      "file": "attachments/3d5614bc7b74/code/block_037.txt",
      "chars": 192
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn-cookieyes.com/assets/images/revisit.svg",
      "bytes": 2339,
      "reused": false,
      "content_type": "image/svg+xml",
      "local_path": "attachments/3d5614bc7b74/images/body-df29beacd7496220cc94.svg",
      "src": "https://cdn-cookieyes.com/assets/images/revisit.svg",
      "alt": "Revisit consent button",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://cdn-cookieyes.com/assets/images/close.svg",
      "bytes": 1345,
      "reused": false,
      "content_type": "image/svg+xml",
      "local_path": "attachments/3d5614bc7b74/images/body-2159bb67f3de8ad870eb.svg",
      "src": "https://cdn-cookieyes.com/assets/images/close.svg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Caption-transformer.jpg",
      "bytes": 136716,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/000.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Caption-transformer.jpg",
      "alt": "Black and white photo of a person standing, looking at a wall of art",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-1.png",
      "bytes": 88405,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/001.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-1.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-2.png",
      "bytes": 170472,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/002.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-2.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-3.png",
      "bytes": 106212,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-a000d3ae44bd0cf84cc1.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-3.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-4-1024x608.png",
      "bytes": 256521,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-2b651976b51610bf75a3.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-4-1024x608.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-5.png",
      "bytes": 182780,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-6b159a5730767cc33b10.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-5.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-6.png",
      "bytes": 142568,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-90652a40b2d322f3209d.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-6.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-7.png",
      "bytes": 150240,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-fac850f893528cfd337d.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-7.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-8.png",
      "bytes": 9613,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-a2ec509a63a7a58620f9.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-8.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-9.png",
      "bytes": 195853,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-23e7000a7a8f06980ada.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-9.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-10.png",
      "bytes": 141667,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-d327aa2f936ca3b4de94.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-10.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-11.png",
      "bytes": 14772,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-cac59b5e5c3619142e68.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-11.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-12.png",
      "bytes": 143188,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-3a02494252b3e0d4a353.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-12.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-13.png",
      "bytes": 155502,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-5425d3ce16026d48365c.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-13.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-14.png",
      "bytes": 170895,
      "reused": false,
      "content_type": "image/png",
      "local_path": "attachments/3d5614bc7b74/images/body-36e9660848f4d4da495e.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-14.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/1787657598287_taxhya.jpeg",
      "bytes": 183845,
      "reused": false,
      "content_type": "image/jpeg",
      "local_path": "attachments/3d5614bc7b74/images/body-59e1c17daab1fcb582c5.jpeg",
      "src": "https://assets.insightmediagroup.io/media/1787657598287_taxhya.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0RkS_9a39Nah1bXso-scaled.jpg",
      "bytes": 399661,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/003.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0RkS_9a39Nah1bXso-scaled.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/04/0ujxW0qAvKh1FzgJE.png",
      "bytes": 539869,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/004.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/04/0ujxW0qAvKh1FzgJE.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/0jxBva0jm4TGpWN__-scaled.jpg",
      "bytes": 1159811,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/005.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/0jxBva0jm4TGpWN__-scaled.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/11/1RScKhJWtCNHxJiEFcFrH_A.jpeg",
      "bytes": 53189,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/006.jpeg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/11/1RScKhJWtCNHxJiEFcFrH_A.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/image-24.jpg",
      "bytes": 429446,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/007.jpg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/image-24.jpg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/07/1TXjs1ZXXN7TEOeUBTn1dkw-scaled.jpeg",
      "bytes": 698455,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/008.jpeg",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/07/1TXjs1ZXXN7TEOeUBTn1dkw-scaled.jpeg",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/06/1hRXZK7oEAf2FA7cwpcqgfw.png",
      "bytes": 103575,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/009.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/06/1hRXZK7oEAf2FA7cwpcqgfw.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/openvla-colab-featured-image-clean.png",
      "bytes": 1685626,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/010.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/openvla-colab-featured-image-clean.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    },
    {
      "status": "ok",
      "url": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/10271c9WJID5z7I_Gj0KuLg.png",
      "bytes": 1515442,
      "reused": true,
      "local_path": "attachments/3d5614bc7b74/images/011.png",
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/10271c9WJID5z7I_Gj0KuLg.png",
      "alt": "",
      "downloaded": true,
      "discovery": "saved_markdown"
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/c/TowardsDataScience",
      "kind": "channel_reference",
      "status": "reference_only"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/muhammad_ardi/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-dives/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/image-captioning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/transformer/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/you-dont-need-a-job-to-get-data-science-experience-26af4fd4c200/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/kurtispykes/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/image-data-generators-in-keras-7c5fc6928400/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/manpreetsinghminhas/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/can-quantum-machine-learning-help-us-solve-climate-change-3500741d583b/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/cardstdani/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/@antenna"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://contributor.insightmediagroup.io/paper-walkthrough-attention-is-all-you-need-80399cdc59e1"
    },
    {
      "type": "external",
      "href": "https://github.com/MuhammadArdiPutra/medium_articles/blob/main/Image%20Captioning%2C%20Transformer%20Mode%20On.ipynb"
    },
    {
      "type": "external",
      "href": "https://contributor.insightmediagroup.io/show-and-tell-e1a1142456e2/"
    },
    {
      "type": "external",
      "href": "https://contributor.insightmediagroup.io/paper-walkthrough-vision-transformer-vit-c5dcf76f1a7a"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/2101.10804"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/1411.4555"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/2010.11929"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/1706.03762"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=CPTR+Paper+Walkthrough%3A+Image+Captioning%2C+Transformer+Mode+On&u=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=CPTR+Paper+Walkthrough%3A+Image+Captioning%2C+Transformer+Mode+On&url=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=CPTR+Paper+Walkthrough%3A+Image+Captioning%2C+Transformer+Mode+On&url=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-20T17:34:01Z",
  "source_url": "https://towardsdatascience.com/image-captioning-transformer-mode-on/",
  "delivery_status": "unverified",
  "discovery_status": "http_403",
  "missing_reason": "http_403",
  "repair_error": "",
  "actual_video_files": [],
  "expected_saved_body_images": 27,
  "image_discovery": "saved_markdown",
  "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\n# CPTR Paper Walkthrough: Image Captioning, Transformer Mode On\nImplementing CPTR (CaPtion TransformeR) from scratch with PyTorch\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nMarch 7, 202530 min read\n![Black and white photo of a person standing, looking at a wall of art](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/Caption-transformer.jpg)\nPhoto by [Antenna](https://unsplash.com/@antenna%20) on [Unsplash](https://unsplash.com/)\n## Introduction\nIn my previous article, I discussed one of the earliest deep learning approaches for image captioning. If you’re interested in reading it, you can find the link to that article at the end of this one.\nToday, I would like to talk about image captioning again, but this time with the more advanced neural network architecture. The deep learning I am going to talk about is the one proposed in the paper titled “ _CPTR: Full Transformer Network for Image Captioning_ ,” written by Liu _et al._ back in 2021 [1]. Specifically, here I will reproduce the model proposed in the paper and explain the underlying theory behind the architecture. However, keep in mind that I won’t actually demonstrate the training process since I only want to focus on the model architecture.\n### The idea behind CPTR\nIn fact, the main idea of the CPTR architecture is exactly the same as the earlier image captioning model, as both use the encoder-decoder structure. Previously, in the paper titled “ _Show and Tell: A Neural Image Caption Generator_ ” [2], the models used are GoogLeNet (a.k.a. Inception V1) and LSTM for the two components, respectively. The illustration of the model proposed in the _Show and Tell_ paper is shown in the following figure.\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-1.png)\nFigure 1. The neural network architecture for image captioning proposed in the Show and Tell paper [2].\nDespite having the same encoder-decoder structure, what makes CPTR different from the previous approach is the basis of the encoder and the decoder themselves. In CPTR, we combine the encoder part of the ViT (Vision Transformer) model with the decoder part of the original Transformer model. The use of transformer-based architecture for both components is essentially where the name CPTR comes from: CaPtion TransformeR.\nNote that the discussions in this article are going to be highly related to ViT and Transformer, so I highly recommend you read my previous article about these two topics if you’re not yet familiar with them. You can find the links at the end of this article.\nFigure 2 shows what the original ViT architecture looks like. Everything inside the green box is the encoder part of the architecture to be adopted as the CPTR encoder.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-2.png)\nFigure 2. The Vision Transformer (ViT) architecture [3].\nNext, Figure 3 displays the original Transformer architecture. The components enclosed in the blue box are the layers that we are going to implement in the CPTR decoder.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-3.png)\nFigure 3. The original Transformer architecture [4].\nIf we combine the components inside the green and blue boxes above, we are going to obtain the architecture shown in Figure 4 below. This is exactly what the CPTR model we are going to implement looks like. The idea here is that the ViT Encoder (green) works by encoding the input image into a specific tensor representation which will then be used as the basis of the Transformer Decoder (blue) to generate the corresponding caption.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-4-1024x608.png)\nFigure 4. The CPTR architecture [5].\nThat’s pretty much everything you need to know for now. I’ll explain more about the details as we go through the implementation.\n## Module imports & parameter configuration\nAs always, the first thing we need to do in the code is to import the required modules. In this case, we only import torch and torch.nn since we are about to implement the model from scratch.\npython\n\n```\n# Codeblock 1\nimport torch\nimport torch.nn as nn\n```\n\nNext, we are going to initialize some parameters in Codeblock 2. If you have read my previous article about image captioning with GoogLeNet and LSTM, you’ll notice that here, we got a lot more parameters to initialize. In this article, I want to reproduce the CPTR model as closely as possible to the original one, so the parameters mentioned in the paper will be used in this implementation.\npython\n\n```\n# Codeblock 2\nBATCH_SIZE         = 1              #(1)\n\nIMAGE_SIZE         = 384            #(2)\nIN_CHANNELS        = 3              #(3)\n\nSEQ_LENGTH         = 30             #(4)\nVOCAB_SIZE         = 10000          #(5)\n\nEMBED_DIM          = 768            #(6)\nPATCH_SIZE         = 16             #(7)\nNUM_PATCHES        = (IMAGE_SIZE//PATCH_SIZE) ** 2  #(8)\nNUM_ENCODER_BLOCKS = 12             #(9)\nNUM_DECODER_BLOCKS = 4              #(10)\nNUM_HEADS          = 12             #(11)\nHIDDEN_DIM         = EMBED_DIM * 4  #(12)\nDROP_PROB          = 0.1            #(13)\n```\n\nThe first parameter I want to explain is the `BATCH_SIZE`, which is written at the line marked with `#(1)`. The number assigned to this variable is not quite important in our case since we are not actually going to train this model. This parameter is set to 1 because, by default, PyTorch treats input tensors as a batch of samples. Here I assume that we only have a single sample in a batch. \nNext, remember that in the case of image captioning we are dealing with images and texts simultaneously. This essentially means that we need to set the parameters for the two. It is mentioned in the paper that the model accepts an RGB image of size 384×384 for the encoder input. Hence, we assign the values for `IMAGE_SIZE` and `IN_CHANNELS` variables based on this information (`#(2)` and `#(3)`). On the other hand, the paper does not mention the parameters for the captions. So, here I assume that the length of the caption is no more than 30 words (`#(4)`), with the vocabulary size estimated at 10000 unique words (`#(5)`).\nThe remaining parameters are related to the model configuration. Here we set the `EMBED_DIM` variable to 768 (`#(6)`). In the encoder side, this number indicates the length of the feature vector that represents each 16×16 image patch (`#(7)`). The same concept also applies to the decoder side, but in that case the feature vector will represent a single word in the caption. Talking more specifically about the `PATCH_SIZE` parameter, we are going to use the value to compute the total number of patches in the input image. Since the image has the size of 384×384, there will be 576 patches in total (`#(8)`).\nWhen it comes to using an encoder-decoder architecture, it is possible to specify the number of encoder and decoder blocks to be used. Using more blocks typically allows the model to perform better in terms of the accuracy, yet in return, it will require more computational power. The authors of this paper decided to stack 12 encoder blocks (`#(9)`) and 4 decoder blocks (`#(10)`). Next, since CPTR is a transformer-based model, it is necessary to specify the number of attention heads within the attention blocks inside the encoders and the decoders, which in this case authors use 12 attention heads (`#(11)`). The value for the `HIDDEN_DIM` parameter is not mentioned anywhere in the paper. However, according to the ViT and the Transformer paper, this parameter is configured to be 4 times larger than `EMBED_DIM` (`#(12)`). The dropout rate is not mentioned in the paper either. Hence, I arbitrarily set `DROP_PROB` to 0.1 (`#(13)`).\n## Encoder\nAs the modules and parameters have been set up, now that we will get into the encoder part of the network. In this section we are going to implement and explain every single component inside the green box in Figure 4 one by one.\n### Patch embedding\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-5.png)\nFigure 5. Dividing the input image into patches and converting them into vectors [5].\nYou can see in Figure 5 above that the first step to be done is dividing the input image into patches. This is essentially done because instead of focusing on local patterns like CNNs, ViT captures global context by learning the relationships between these patches. We can model this process with the `Patcher` class shown in the Codeblock 3 below. For the sake of simplicity, here I also include the process inside the _patch embedding_ block within the same class.\npython\n\n```\n# Codeblock 3\nclass Patcher(nn.Module):\n   def __init__(self):\n       super().__init__()\n\n       #(1)\n       self.unfold = nn.Unfold(kernel_size=PATCH_SIZE, stride=PATCH_SIZE)\n\n       #(2)\n       self.linear_projection = nn.Linear(in_features=IN_CHANNELS*PATCH_SIZE*PATCH_SIZE,\n                                          out_features=EMBED_DIM)\n      \n   def forward(self, images):\n       print(f'images\\t\\t: {images.size()}')\n       images = self.unfold(images)  #(3)\n       print(f'after unfold\\t: {images.size()}')\n      \n       images = images.permute(0, 2, 1)  #(4)\n       print(f'after permute\\t: {images.size()}')\n      \n       features = self.linear_projection(images)  #(5)\n       print(f'after lin proj\\t: {features.size()}')\n      \n       return features\n```\n\nThe patching itself is done using the `nn.Unfold` layer (`#(1)`). Here we need to set both the `kernel_size` and `stride` parameters to `PATCH_SIZE (16)` so that the resulting patches do not overlap with each other. This layer also automatically flattens these patches once it is applied to the input image. Meanwhile, the `nn.Linear layer` (`#(2)`) is employed to perform linear projection, i.e., the process done by the _patch embedding_ block. By setting the `out_features` parameter to `EMBED_DIM`, this layer will map every single flattened patch into a feature vector of length 768.\nThe entire process should make more sense once you read the `forward()` method. You can see at line `#(3)` in the same codeblock that the input image is directly processed by the unfold layer. Next, we need to process the resulting tensor with the `permute()` method (`#(4)`) to swap the first and the second axis before feeding it to the `linear_projection` layer (`#(5)`). Additionally, here I also print out the tensor dimension after each layer so that you can better understand the transformation made at each step.\nIn order to check if our `Patcher` class works properly, we can just pass a dummy tensor through the network. Look at the Codeblock 4 below to see how I do it.\npython\n\n```\n# Codeblock 4\npatcher  = Patcher()\n\nimages   = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nfeatures = patcher(images)\n```\n\npython\n\n```\n# Codeblock 4 Output\nimages         : torch.Size([1, 3, 384, 384])\nafter unfold   : torch.Size([1, 768, 576])  #(1)\nafter permute  : torch.Size([1, 576, 768])  #(2)\nafter lin proj : torch.Size([1, 576, 768])  #(3)\n```\n\nThe tensor I passed above represents an RGB image of size 384×384. Here we can see that after the unfold operation is performed, the tensor dimension changed to 1×768×576 (`#(1)`), denoting the flattened 3×16×16 patch for each of the 576 patches. Unfortunately, this output shape does not match what we need. Remember that in ViT, we perceive image patches as a sequence, so we need to swap the 1st and 2nd axes because typically, the 1st dimension of a tensor represents the temporal axis, while the 2nd one represents the feature vector of each timestep. As the `permute()` operation is performed, our tensor is now having the dimension of 1×576×768 (`#(2)`). Lastly, we pass this tensor through the linear projection layer, which the resulting tensor shape remains the same since we set the `EMBED_DIM` parameter to the same size (768) (`#(3)`). Despite having the same dimension, the information contained in the final tensor should be richer thanks to the transformation applied by the trainable weights of the linear projection layer.\n### Learnable positional embedding\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-6.png)\nFigure 6. Injecting the learnable positional embeddings into the embedded image patches [5].\nAfter the input image has successfully been converted into a sequence of patches, the next thing to do is to inject the so-called _positional embedding_ tensor. This is essentially done because a transformer without positional embedding is permutation-invariant, meaning that it treats the input sequence as if their order does not matter. Interestingly, since an image is not a literal sequence, we should set the positional embedding to be _learnable_ such that it will be able to somewhat reorder the patch sequence that it thinks works best in representing the spatial information. However, keep in mind that the term “reordering” here does not mean that we physically rearrange the sequence. Rather, it does so by adjusting the embedding weights.\nThe implementation is pretty simple. All we need to do is just to initialize a tensor using `nn.Parameter` which the dimension is set to match with the output from the `Patcher` model, i.e., 576×768. Also, don’t forget to write `requires_grad=True` just to ensure that the tensor is trainable. Look at the Codeblock 5 below for the details.\npython\n\n```\n# Codeblock 5\nclass LearnableEmbedding(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.learnable_embedding = nn.Parameter(torch.randn(size=(NUM_PATCHES, EMBED_DIM)),\n                                               requires_grad=True)\n      \n   def forward(self):\n       pos_embed = self.learnable_embedding\n       print(f'learnable embedding\\t: {pos_embed.size()}')\n      \n       return pos_embed\n```\n\nNow let’s run the following codeblock to see whether our `LearnableEmbedding` class works properly. You can see in the printed output that it successfully created the positional embedding tensor as expected.\npython\n\n```\n# Codeblock 6\nlearnable_embedding = LearnableEmbedding()\n\npos_embed = learnable_embedding()\n```\n\ntext\n\n```\n# Codeblock 6 Output\nlearnable embedding : torch.Size([576, 768])\n```\n\n### The main encoder block\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-7.png)\nFigure 7. The main encoder block [5].\nThe next thing we are going to do is to construct the main encoder block displayed in the Figure 7 above. Here you can see that this block consists of several sub-components, namely _self-attention_ , _layer norm_ , FFN (Feed-Forward Network), and another _layer norm_. The Codeblock 7a below shows how I initialize these layers inside the `__init__()` method of the `EncoderBlock` class.\npython\n\n```\n# Codeblock 7a\nclass EncoderBlock(nn.Module):\n   def __init__(self):\n       super().__init__()\n      \n       #(1)\n       self.self_attention = nn.MultiheadAttention(embed_dim=EMBED_DIM,\n                                                   num_heads=NUM_HEADS,\n                                                   batch_first=True,  #(2)\n                                                   dropout=DROP_PROB)\n      \n       self.layer_norm_0 = nn.LayerNorm(EMBED_DIM)  #(3)\n      \n       self.ffn = nn.Sequential(  #(4)\n           nn.Linear(in_features=EMBED_DIM, out_features=HIDDEN_DIM),\n           nn.GELU(),\n           nn.Dropout(p=DROP_PROB),\n           nn.Linear(in_features=HIDDEN_DIM, out_features=EMBED_DIM),\n       )\n      \n       self.layer_norm_1 = nn.LayerNorm(EMBED_DIM)  #(5)\n```\n\nI’ve previously mentioned that the idea of ViT is to capture the relationships between patches within an image. This process is done by the _multihead attention_ layer I initialize at line `#(1)` in the above codeblock. One thing to keep in mind here is that we need to set the batch_first parameter to `True` (`#(2)`). This is essentially done so that the attention layer will be compatible with our tensor shape, in which the batch dimension (`batch_size`) is at the 0th axis of the tensor. Next, the two layer normalization layers need to be initialized separately, as shown at line `#(3)` and `#(5)`. Lastly, we initialize the FFN block at line `#(4)`, which the layers stacked using `nn.Sequential` follows the structure defined in the following equation.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-8.png)\nFigure 8. The operations done inside the FFN block [1].\nAs the `__init__()` method is complete, we will now continue with the `forward()` method. Let’s take a look at the Codeblock 7b below.\npython\n\n```\n# Codeblock 7b\n   def forward(self, features):  #(1)\n      \n       residual = features  #(2)\n       print(f'features & residual\\t: {residual.size()}')\n      \n       #(3)\n       features, self_attn_weights = self.self_attention(query=features,\n                                                         key=features,\n                                                         value=features)\n       print(f'after self attention\\t: {features.size()}')\n       print(f\"self attn weights\\t: {self_attn_weights.shape}\")\n      \n       features = self.layer_norm_0(features + residual)  #(4)\n       print(f'after norm\\t\\t: {features.size()}')\n      \n\n       residual = features\n       print(f'\\nfeatures & residual\\t: {residual.size()}')\n      \n       features = self.ffn(features)  #(5)\n       print(f'after ffn\\t\\t: {features.size()}')\n      \n       features = self.layer_norm_1(features + residual)\n       print(f'after norm\\t\\t: {features.size()}')\n      \n       return features\n```\n\nHere you can see that the input tensor is named `features (#(1)`). I name it this way because the input of the `EncoderBlock` is the image that has already been processed with `Patcher` and `LearnableEmbedding`, instead of a raw image. Before doing anything, notice in the `encoder` block that there is a branch separated from the main flow which then returns back to the normalization layer. This branch is commonly known as a _residual connection_. To implement this, we need to store the original input tensor to the residual variable as I demonstrate at line `#(2)`. As the input tensor has been copied, now we are ready to process the original input with the multihead attention layer (`#(3)`). Since this is a _self_ -attention (not a _cross_ -attention), the `query`, `key`, and `value` inputs for this layer are all derived from the `features` tensor. Next, the layer normalization operation is then performed at line `#(4)`, which the input for this layer already contains information from the attention block as well as the residual connection. The remaining steps are basically the same as what I just explained, except that here we replace the self-attention block with FFN (`#(5)`).\nIn the following codeblock, I’ll test the `EncoderBlock` class by passing a dummy tensor of size 1×576×768, simulating an output tensor from the previous operations.\npython\n\n```\n# Codeblock 8\nencoder_block = EncoderBlock()\n\nfeatures = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM)\nfeatures = encoder_block(features)\n```\n\nBelow is what the tensor dimension looks like throughout the entire process inside the model.\npython\n\n```\n# Codeblock 8 Output\nfeatures & residual  : torch.Size([1, 576, 768])  #(1)\nafter self attention : torch.Size([1, 576, 768])\nself attn weights    : torch.Size([1, 576, 576])  #(2)\nafter norm           : torch.Size([1, 576, 768])\n\nfeatures & residual  : torch.Size([1, 576, 768])\nafter ffn            : torch.Size([1, 576, 768])  #(3)\nafter norm           : torch.Size([1, 576, 768])  #(4)\n```\n\nHere you can see that the final output tensor (`#(4)`) has the same size as the input (`#(1)`), allowing us to stack multiple encoder blocks without having to worry about messing up the tensor dimensions. Not only that, the size of the tensor also appears to be unchanged from the beginning all the way to the last layer. In fact, there are actually lots of transformations performed inside the attention block, but we just can’t see it since the entire process is done internally by the `nn.MultiheadAttention` layer. One of the tensors produced in the layer that we can observe is the attention weight (`#(2)`). This weight matrix, which has the size of 576×576, is responsible for storing information regarding the relationships between one patch and every other patch in the image. Furthermore, changes in tensor dimension actually also happened inside the FFN layer. The feature vector of each patch which has the initial length of 768 changed to 3072 and immediately shrunk back to 768 again (`#(3)`). However, this transformation is not printed since the process is wrapped with `nn.Sequential` back at line #(4) in Codeblock 7a.\n### ViT encoder\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-9.png)\nFigure 9. The entire ViT Encoder in the CPTR architecture [5].\nAs we have finished implementing all encoder components, now that we will assemble them to construct the actual ViT Encoder. We are going to do it in the `Encoder` class in Codeblock 9.\npython\n\n```\n# Codeblock 9\nclass Encoder(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.patcher = Patcher()  #(1)\n       self.learnable_embedding = LearnableEmbedding()  #(2)\n\n       #(3)\n       self.encoder_blocks = nn.ModuleList(EncoderBlock() for _ in range(NUM_ENCODER_BLOCKS))\n  \n   def forward(self, images):  #(4)\n       print(f'images\\t\\t\\t: {images.size()}')\n      \n       features = self.patcher(images)  #(5)\n       print(f'after patcher\\t\\t: {features.size()}')\n      \n       features = features + self.learnable_embedding()  #(6)\n       print(f'after learn embed\\t: {features.size()}')\n      \n       for i, encoder_block in enumerate(self.encoder_blocks):\n           features = encoder_block(features)  #(7)\n           print(f\"after encoder block #{i}\\t: {features.shape}\")\n\n       return features\n```\n\nInside the` __init__()` method, what we need to do is to initialize all components we created earlier, i.e., `Patcher` (`#(1)`), `LearnableEmbedding` (`#(2)`), and `EncoderBlock` (`#(3)`). In this case, the `EncoderBlock` is initialized inside `nn.ModuleList` since we want to repeat it `NUM_ENCODER_BLOCKS` (12) times. To the `forward()` method, it initially works by accepting raw image as the input (`#(4)`). We then process it with the `patcher` layer (`#(5)`) to divide the image into small patches and transform them with the linear projection operation. The learnable positional embedding tensor is then injected into the resulting output by element-wise addition (`#(6)`). Lastly, we pass it into the 12 encoder blocks sequentially with a simple for loop (`#(7)`).\nNow, in Codeblock 10, I am going to pass a dummy image through the entire encoder. Note that since I want to focus on the flow of this Encoder class, I re-run the previous classes we created earlier with the `print()` functions commented out so that the outputs will look neat.\npython\n\n```\n# Codeblock 10\nencoder = Encoder()\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nfeatures = encoder(images)\n```\n\nAnd below is what the flow of the tensor looks like. Here, we can see that our dummy input image successfully passed through all layers in the network, including the encoder blocks that we repeat 12 times. The resulting output tensor is now context-aware, meaning that it already contains information about the relationships between patches within the image. Therefore, this tensor is now ready to be processed further with the decoder, which will later be discussed in the subsequent section.\npython\n\n```\n# Codeblock 10 Output\nimages                  : torch.Size([1, 3, 384, 384])\nafter patcher           : torch.Size([1, 576, 768])\nafter learn embed       : torch.Size([1, 576, 768])\nafter encoder block #0  : torch.Size([1, 576, 768])\nafter encoder block #1  : torch.Size([1, 576, 768])\nafter encoder block #2  : torch.Size([1, 576, 768])\nafter encoder block #3  : torch.Size([1, 576, 768])\nafter encoder block #4  : torch.Size([1, 576, 768])\nafter encoder block #5  : torch.Size([1, 576, 768])\nafter encoder block #6  : torch.Size([1, 576, 768])\nafter encoder block #7  : torch.Size([1, 576, 768])\nafter encoder block #8  : torch.Size([1, 576, 768])\nafter encoder block #9  : torch.Size([1, 576, 768])\nafter encoder block #10 : torch.Size([1, 576, 768])\nafter encoder block #11 : torch.Size([1, 576, 768])\n```\n\n### ViT encoder (alternative)\nI want to show you something before we talk about the decoder. If you think that our approach above is too complicated, it is actually possible for you to use `nn.TransformerEncoderLayer` from PyTorch so that you don’t need to implement the `EncoderBlock` class from scratch. To do so, I am going to reimplement the `Encoder` class, but this time I’ll name it `EncoderTorch`.\npython\n\n```\n# Codeblock 11\nclass EncoderTorch(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.patcher = Patcher()\n       self.learnable_embedding = LearnableEmbedding()\n      \n       #(1)\n       encoder_block = nn.TransformerEncoderLayer(d_model=EMBED_DIM,\n                                                  nhead=NUM_HEADS,\n                                                  dim_feedforward=HIDDEN_DIM,\n                                                  dropout=DROP_PROB,\n                                                  batch_first=True)\n      \n       #(2)\n       self.encoder_blocks = nn.TransformerEncoder(encoder_layer=encoder_block,\n                                                   num_layers=NUM_ENCODER_BLOCKS)\n  \n   def forward(self, images):\n       print(f'images\\t\\t\\t: {images.size()}')\n      \n       features = self.patcher(images)\n       print(f'after patcher\\t\\t: {features.size()}')\n      \n       features = features + self.learnable_embedding()\n       print(f'after learn embed\\t: {features.size()}')\n      \n       features = self.encoder_blocks(features)  #(3)\n       print(f'after encoder blocks\\t: {features.size()}')\n\n       return features\n```\n\nWhat we basically do in the above codeblock is that instead of using the EncoderBlock class, here we use `nn.TransformerEncoderLayer` (`#(1)`), which will automatically create a single encoder block based on the parameters we pass to it. To repeat it multiple times, we can just use nn.`TransformerEncoder` and pass a number to the `num_layers` parameter (`#(2)`). With this approach, we don’t necessarily need to write the forward pass in a loop like what we did earlier (`#(3)`).\nThe testing code in the Codeblock 12 below is exactly the same as the one in Codeblock 10, except that here I use the `EncoderTorch` class. You can also see here that the output is basically the same as the previous one.\npython\n\n```\n# Codeblock 12\nencoder_torch = EncoderTorch()\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nfeatures = encoder_torch(images)\n```\n\npython\n\n```\n# Codeblock 12 Output\nimages               : torch.Size([1, 3, 384, 384])\nafter patcher        : torch.Size([1, 576, 768])\nafter learn embed    : torch.Size([1, 576, 768])\nafter encoder blocks : torch.Size([1, 576, 768])\n```\n\n## Decoder\nAs we have successfully created the encoder part of the CPTR architecture, now that we will talk about the decoder. In this section I am going to implement every single component inside the blue box in Figure 4. Based on the figure, we can see that the decoder accepts two inputs, i.e., the image caption ground truth (the lower part of the blue box) and the sequence of embedded patches produced by the encoder (the arrow coming from the green box). It is important to know that the architecture drawn in Figure 4 is intended to illustrate the training phase, where the entire caption ground truth is fed into the decoder. Later in the inference phase, we only provide a <BOS> (_Beginning of Sentence_) token for the caption input. The decoder will then predict each word sequentially based on the given image and the previously generated words. This process is commonly known as an _autoregressive_ mechanism.\n### Sinusoidal positional embedding\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-10.png)\nFigure 10. Where the sinusoidal positional embedding component is located in the decoder [5].\nIf you take a look at the CPTR model, you’ll see that the first step in the decoder is to convert each word into the corresponding feature vector representation using the _word embedding_ block. However, since this step is very easy, we are going to implement it later. Now let’s assume that this word vectorization process is already done, so we can move to the positional embedding part.\nAs I’ve mentioned earlier, since transformer is permutation-invariant by nature, we need to apply positional embedding to the input sequence. Different from the previous one, here we use the so-called _sinusoidal positional embedding_. We can think of it like a method to label each word vector by assigning numbers obtained from a sinusoidal wave. By doing so, we can expect our model to understand word orders thanks to the information given by the wave patterns.\nIf you go back to Codeblock 6 Output, you’ll see that the positional embedding tensor in the encoder has the size of `NUM_PATCHES` × `EMBED_DIM` (576×768). What we basically want to do in the decoder is to create a tensor having the size of `SEQ_LENGTH` × `EMBED_DIM` (30×768), which the values are computed based on the equation shown in Figure 11. This tensor is then set to be non-trainable because a sequence of words must maintain a fixed order to preserve its meaning.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-11.png)\nFigure 11. The equation for creating sinusoidal positional encoding proposed in the Transformer paper [6].\nHere I want to explain the following code quickly because I actually have discussed this more thoroughly in my previous article about Transformer. Generally speaking, what we basically do here is to create the sine and cosine wave using `torch.sin()` (`#(1)`) and `torch.cos()` (`#(2)`). The resulting two tensors are then merged using the code at line `#(3)` and `#(4)`.\npython\n\n```\n# Codeblock 13\nclass SinusoidalEmbedding(nn.Module):\n   def forward(self):\n       pos = torch.arange(SEQ_LENGTH).reshape(SEQ_LENGTH, 1)\n       print(f\"pos\\t\\t: {pos.shape}\")\n      \n       i = torch.arange(0, EMBED_DIM, 2)\n       denominator = torch.pow(10000, i/EMBED_DIM)\n       print(f\"denominator\\t: {denominator.shape}\")\n      \n       even_pos_embed = torch.sin(pos/denominator)  #(1)\n       odd_pos_embed  = torch.cos(pos/denominator)  #(2)\n       print(f\"even_pos_embed\\t: {even_pos_embed.shape}\")\n      \n       stacked = torch.stack([even_pos_embed, odd_pos_embed], dim=2)  #(3)\n       print(f\"stacked\\t\\t: {stacked.shape}\")\n\n       pos_embed = torch.flatten(stacked, start_dim=1, end_dim=2)  #(4)\n       print(f\"pos_embed\\t: {pos_embed.shape}\")\n      \n       return pos_embed\n```\n\nNow we can check if the `SinusoidalEmbedding` class above works properly by running the Codeblock 14 below. As expected earlier, here you can see that the resulting tensor has the size of 30×768. This dimension matches with the tensor obtained by the process done in the _word embedding_ block, allowing them to be summed in an element-wise manner.\npython\n\n```\n# Codeblock 14\nsinusoidal_embedding = SinusoidalEmbedding()\npos_embed = sinusoidal_embedding()\n```\n\npython\n\n```\n# Codeblock 14 Output\npos            : torch.Size([30, 1])\ndenominator    : torch.Size([384])\neven_pos_embed : torch.Size([30, 384])\nstacked        : torch.Size([30, 384, 2])\npos_embed      : torch.Size([30, 768])\n```\n\n### Look-ahead mask\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-12.png)\nFigure 12. A look-ahead mask needs to be applied to the masked-self attention layer [5].\nThe next thing I am going to talk about in the decoder is the _masked self-attention_ layer highlighted in the above figure. I am not going to code the attention mechanism from scratch. Rather, I’ll only implement the so-called _look-ahead mask_ , which will be useful for the self-attention layer so that it doesn’t attend to the subsequent words in the caption during the training phase.\nThe way to do it is pretty easy, what we need to do is just to create a triangular matrix which the size is set to match with the attention weight matrix, i.e., `SEQ_LENGTH` × `SEQ_LENGTH` (30×30). Look at the `create_mask()`function below for the details.\npython\n\n```\n# Codeblock 15\ndef create_mask(seq_length):\n   mask = torch.tril(torch.ones((seq_length, seq_length)))  #(1)\n   mask[mask == 0] = -float('inf')  #(2)\n   mask[mask == 1] = 0  #(3)\n   return mask\n```\n\nEven though creating a triangular matrix can simply be done with `torch.tril()` and `torch.ones()` (`#(1)`), but here we need to make a little modification by changing the 0 values to _-inf_ (`#(2)`) and the 1s to 0 (`#(3)`). This is essentially done because the `nn.MultiheadAttention` layer applies the mask by element-wise addition. By assigning _-inf_ to the subsequent words, the attention mechanism will completely ignore them. Again, the internal process inside an attention layer has also been discussed in detail in[ my previous article about transformer](https://contributor.insightmediagroup.io/paper-walkthrough-attention-is-all-you-need-80399cdc59e1).\nNow I am going to run the function with `seq_length=7` so that you can see what the mask actually looks like. Later in the complete flow, we need to set the `seq_length` parameter to `SEQ_LENGTH` (30) so that it matches with the actual caption length.\npython\n\n```\n# Codeblock 16\nmask_example = create_mask(seq_length=7)\nmask_example\n```\n\npython\n\n```\n# Codeblock 16 Output\ntensor([[0., -inf, -inf, -inf, -inf, -inf, -inf],\n       [0., 0., -inf, -inf, -inf, -inf, -inf],\n       [0., 0., 0., -inf, -inf, -inf, -inf],\n       [0., 0., 0., 0., -inf, -inf, -inf],\n       [0., 0., 0., 0., 0., -inf, -inf],\n       [0., 0., 0., 0., 0., 0., -inf],\n       [0., 0., 0., 0., 0., 0., 0.]])\n```\n\n### The main decoder block\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-13.png)\nFigure 13. The main decoder block [5].\nWe can see in the above figure that the structure of the decoder block is a bit longer than that of the encoder block. It seems like everything is nearly the same, except that the decoder part has a _cross-attention_ mechanism and an additional layer normalization step placed after it. This cross-attention layer can actually be perceived as the bridge between the encoder and the decoder, as it is employed to capture the relationships between each word in the caption and every single patch in the input image. The two arrows coming from the encoder are the _key_ and _value_ inputs for the attention layer, whereas the _query_ is derived from the previous layer in the decoder itself. Look at the Codeblock 17a and 17b below to see the implementation of the entire decoder block.\npython\n\n```\n# Codeblock 17a\nclass DecoderBlock(nn.Module):\n   def __init__(self):\n       super().__init__()\n      \n       #(1)\n       self.self_attention = nn.MultiheadAttention(embed_dim=EMBED_DIM,\n                                                   num_heads=NUM_HEADS,\n                                                   batch_first=True,\n                                                   dropout=DROP_PROB)\n       #(2)\n       self.layer_norm_0 = nn.LayerNorm(EMBED_DIM)\n       #(3)\n       self.cross_attention = nn.MultiheadAttention(embed_dim=EMBED_DIM,\n                                                    num_heads=NUM_HEADS,\n                                                    batch_first=True,\n                                                    dropout=DROP_PROB)\n\n       #(4)\n       self.layer_norm_1 = nn.LayerNorm(EMBED_DIM)\n      \n       #(5)      \n       self.ffn = nn.Sequential(\n           nn.Linear(in_features=EMBED_DIM, out_features=HIDDEN_DIM),\n           nn.GELU(),\n           nn.Dropout(p=DROP_PROB),\n           nn.Linear(in_features=HIDDEN_DIM, out_features=EMBED_DIM),\n       )\n      \n       #(6)\n       self.layer_norm_2 = nn.LayerNorm(EMBED_DIM)\n```\n\nIn the `__init__()` method, we first initialize both self-attention (`#(1)`) and cross-attention (`#(3)`) layers with `nn.MultiheadAttention`. These two layers appear to be exactly the same now, but later you’ll see the difference in the `forward()` method. The three layer normalization operations are initialized separately as shown at line `#(2)`, `#(4)` and `#(6)`, since each of them will contain different normalization parameters. Lastly, the `ffn` layer (`#(5)`) is exactly the same as the one in the encoder, which basically follows the equation back in Figure 8.\nTalking about the `forward()` method below, it initially works by accepting three inputs: `features`, `captions`, and `attn_mask`, which each of them denotes the tensor coming from the encoder, the tensor from the decoder itself, and a look-ahead mask, respectively (`#(1)`). The remaining steps are somewhat similar to that of the `EncoderBlock`, except that here we repeat the multihead attention block twice. The first attention mechanism takes captions as the `query`, `key`, and `value` parameters (`#(2)`). This is essentially done because we want the layer to capture the context within the captions tensor itself — hence the name _self-attention_. Here we also need to pass the attn_mask parameter to this layer so that it cannot see the subsequent words during the training phase. The second attention mechanism is different (`#(3)`). Since we want to combine the information from the encoder and the decoder, we need to pass the `captions` tensor as the `query`, whereas the `features` tensor will be passed as the `key` and `value` — hence the name _cross-attention_. A look-ahead mask is not necessary in the cross-attention layer since later in the inference phase the model will be able to see the entire input image at once rather than looking at the patches one by one. As the tensor has been processed by the two attention layers, we will then pass it through the feed forward network (`#(4)`). Lastly, don’t forget to create the residual connections and apply the layer normalization steps after each sub-component.\npython\n\n```\n# Codeblock 17b\n   def forward(self, features, captions, attn_mask):  #(1)\n       print(f\"attn_mask\\t\\t: {attn_mask.shape}\")\n       residual = captions\n       print(f\"captions & residual\\t: {captions.shape}\")\n      \n       #(2)\n       captions, self_attn_weights = self.self_attention(query=captions,\n                                                         key=captions,\n                                                         value=captions,\n                                                         attn_mask=attn_mask)\n       print(f\"after self attention\\t: {captions.shape}\")\n       print(f\"self attn weights\\t: {self_attn_weights.shape}\")\n      \n       captions = self.layer_norm_0(captions + residual)\n       print(f\"after norm\\t\\t: {captions.shape}\")\n      \n      \n       print(f\"\\nfeatures\\t\\t: {features.shape}\")\n       residual = captions\n       print(f\"captions & residual\\t: {captions.shape}\")\n      \n       #(3)\n       captions, cross_attn_weights = self.cross_attention(query=captions,\n                                                           key=features,\n                                                           value=features)\n       print(f\"after cross attention\\t: {captions.shape}\")\n       print(f\"cross attn weights\\t: {cross_attn_weights.shape}\")\n      \n       captions = self.layer_norm_1(captions + residual)\n       print(f\"after norm\\t\\t: {captions.shape}\")\n      \n       residual = captions\n       print(f\"\\ncaptions & residual\\t: {captions.shape}\")\n      \n       captions = self.ffn(captions)  #(4)\n       print(f\"after ffn\\t\\t: {captions.shape}\")\n      \n       captions = self.layer_norm_2(captions + residual)\n       print(f\"after norm\\t\\t: {captions.shape}\")\n      \n       return captions\n\n\n```\n\nAs the `DecoderBlock` class is completed, we can now test it with the following code.\npython\n\n```\n# Codeblock 18\ndecoder_block = DecoderBlock()\n\nfeatures = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM)  #(1)\ncaptions = torch.randn(BATCH_SIZE, SEQ_LENGTH, EMBED_DIM)   #(2)\nlook_ahead_mask = create_mask(seq_length=SEQ_LENGTH)  #(3)\n\ncaptions = decoder_block(features, captions, look_ahead_mask)\n```\n\nHere we assume that features is a tensor containing a sequence of patch embeddings produced by the `encoder` (`#(1)`), while captions is a sequence of embedded words (`#(2)`). The `seq_length` parameter of the look-ahead mask is set to `SEQ_LENGTH` (30) to match it to the number of words in the caption (`#(3)`). The tensor dimensions after each step are displayed in the following output.\npython\n\n```\n# Codeblock 18 Output\nattn_mask             : torch.Size([30, 30])\ncaptions & residual   : torch.Size([1, 30, 768])\nafter self attention  : torch.Size([1, 30, 768])\nself attn weights     : torch.Size([1, 30, 30])    #(1)\nafter norm            : torch.Size([1, 30, 768])\n\nfeatures              : torch.Size([1, 576, 768])\ncaptions & residual   : torch.Size([1, 30, 768])\nafter cross attention : torch.Size([1, 30, 768])\ncross attn weights    : torch.Size([1, 30, 576])   #(2)\nafter norm            : torch.Size([1, 30, 768])\n\ncaptions & residual   : torch.Size([1, 30, 768])\nafter ffn             : torch.Size([1, 30, 768])\nafter norm            : torch.Size([1, 30, 768])\n```\n\nHere we can see that our `DecoderBlock` class works properly as it successfully processed the input tensors all the way to the last layer in the network. Here I want you to take a closer look at the attention weights at lines `#(1)` and `#(2)`. Based on these two lines, we can confirm that our decoder implementation is correct since the attention weight produced by the self-attention layer has the size of 30×30 (`#(1)`), which basically means that this layer really captured the context within the input caption. Meanwhile, the attention weight matrix generated by the cross-attention layer has the size of 30×576 (`#(2)`), indicating that it successfully captured the relationships between the words and the patches. This essentially implies that after cross-attention operation is performed, the resulting captions tensor has been enriched with the information from the image.\n### Transformer decoder\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/03/CPTR-14.png)\nFigure 14. The entire Transformer Decoder in the CPTR architecture [5].\nNow that we have successfully created all components for the entire decoder, what I am going to do next is to put them together into a single class. Look at the Codeblock 19a and 19b below to see how I do that.\npython\n\n```\n# Codeblock 19a\nclass Decoder(nn.Module):\n   def __init__(self):\n       super().__init__()\n\n       #(1)\n       self.embedding = nn.Embedding(num_embeddings=VOCAB_SIZE,\n                                     embedding_dim=EMBED_DIM)\n\n       #(2)\n       self.sinusoidal_embedding = SinusoidalEmbedding()\n\n       #(3)\n       self.decoder_blocks = nn.ModuleList(DecoderBlock() for _ in range(NUM_DECODER_BLOCKS))\n\n       #(4)\n       self.linear = nn.Linear(in_features=EMBED_DIM,\n                               out_features=VOCAB_SIZE)\n```\n\nIf you compare this `Decoder` class with the `Encoder` class from codeblock 9, you’ll notice that they are somewhat similar in terms of the structure. In the encoder, we convert image patches into vectors using `Patcher`, while in the decoder we convert every single word in the caption into a vector using the `nn.Embedding layer` (`#(1)`), which I haven’t explained earlier. Afterward, we initialize the positional embedding layer, where for the decoder we use the _sinusoidal_ rather than the _trainable_ one (`#(2)`). Next, we stack multiple decoder blocks using `nn.ModuleList` (`#(3)`). The linear layer written at line #(4), which doesn’t exist in the encoder, is necessary to be implemented here since it will be responsible to map each of the embedded words into a vector of length `VOCAB_SIZE` (10000). Later on, this vector will contain the logit of every word in the dictionary, and what we need to do afterward is just to take the index containing the highest value, i.e., the most likely word to be predicted.\nThe flow of the tensors within the `forward()` method itself is also pretty similar to the one in the `Encoder` class. In the Codeblock 19b below we pass features, captions, and `attn_mask` as the input (`#(1)`). Keep in mind that in this case the captions tensor contains the raw word sequence, so we need to vectorize these words with the embedding layer beforehand (`#(2)`). Next, we inject the sinusoidal positional embedding tensor using the code at line `#(3)` before eventually passing it through the four decoder blocks sequentially (`#(4)`). Finally, we pass the resulting tensor through the last linear layer to obtain the `prediction` logits (`#(5)`).\npython\n\n```\n# Codeblock 19b\n   def forward(self, features, captions, attn_mask):  #(1)\n       print(f\"features\\t\\t: {features.shape}\")\n       print(f\"captions\\t\\t: {captions.shape}\")\n      \n       captions = self.embedding(captions)  #(2)\n       print(f\"after embedding\\t\\t: {captions.shape}\")\n      \n       captions = captions + self.sinusoidal_embedding()  #(3)\n       print(f\"after sin embed\\t\\t: {captions.shape}\")\n      \n       for i, decoder_block in enumerate(self.decoder_blocks):\n           captions = decoder_block(features, captions, attn_mask)  #(4)\n           print(f\"after decoder block #{i}\\t: {captions.shape}\")\n      \n       captions = self.linear(captions)  #(5)\n       print(f\"after linear\\t\\t: {captions.shape}\")\n      \n       return captions\n```\n\nAt this point you might be wondering why we don’t implement the softmax activation function as drawn in the illustration. This is essentially because during the training phase, softmax is typically included within the loss function, whereas in the inference phase, the index of the largest value will remain the same regardless of whether softmax is applied.\nNow let’s run the following testing code to check whether there are errors in our implementation. Previously I mentioned that the captions input of the `Decoder` class is a raw word sequence. To simulate this, we can simply create a sequence of random integers ranging between 0 and `VOCAB_SIZE` (10000) with the length of `SEQ_LENGTH` (30) words (`#(1)`).\npython\n\n```\n# Codeblock 20\ndecoder = Decoder()\n\nfeatures = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH))  #(1)\n\ncaptions = decoder(features, captions, look_ahead_mask)\n```\n\nAnd below is what the resulting output looks like. Here you can see in the last line that the linear layer produced a tensor of size 30×10000, indicating that our decoder model is now capable of predicting the logit scores for each word in the vocabulary across all 30 sequence positions.\npython\n\n```\n# Codeblock 20 Output\nfeatures               : torch.Size([1, 576, 768])\ncaptions               : torch.Size([1, 30])\nafter embedding        : torch.Size([1, 30, 768])\nafter sin embed        : torch.Size([1, 30, 768])\nafter decoder block #0 : torch.Size([1, 30, 768])\nafter decoder block #1 : torch.Size([1, 30, 768])\nafter decoder block #2 : torch.Size([1, 30, 768])\nafter decoder block #3 : torch.Size([1, 30, 768])\nafter linear           : torch.Size([1, 30, 10000])\n```\n\n### Transformer decoder (alternative)\nIt is actually also possible to make the code simpler by replacing the `DecoderBlock` class with the `nn.TransformerDecoderLayer`, just like what we did in the ViT Encoder. Below is what the code looks like if we use this approach instead.\npython\n\n```\n# Codeblock 21\nclass DecoderTorch(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.embedding = nn.Embedding(num_embeddings=VOCAB_SIZE,\n                                     embedding_dim=EMBED_DIM)\n      \n       self.sinusoidal_embedding = SinusoidalEmbedding()\n      \n       #(1)\n       decoder_block = nn.TransformerDecoderLayer(d_model=EMBED_DIM,\n                                                  nhead=NUM_HEADS,\n                                                  dim_feedforward=HIDDEN_DIM,\n                                                  dropout=DROP_PROB,\n                                                  batch_first=True)\n      \n       #(2)\n       self.decoder_blocks = nn.TransformerDecoder(decoder_layer=decoder_block,\n                                                   num_layers=NUM_DECODER_BLOCKS)\n      \n       self.linear = nn.Linear(in_features=EMBED_DIM,\n                               out_features=VOCAB_SIZE)\n      \n   def forward(self, features, captions, tgt_mask):\n       print(f\"features\\t\\t: {features.shape}\")\n       print(f\"captions\\t\\t: {captions.shape}\")\n      \n       captions = self.embedding(captions)\n       print(f\"after embedding\\t\\t: {captions.shape}\")\n      \n       captions = captions + self.sinusoidal_embedding()\n       print(f\"after sin embed\\t\\t: {captions.shape}\")\n      \n       #(3)\n       captions = self.decoder_blocks(tgt=captions,\n                                      memory=features,\n                                      tgt_mask=tgt_mask)\n       print(f\"after decoder blocks\\t: {captions.shape}\")\n      \n       captions = self.linear(captions)\n       print(f\"after linear\\t\\t: {captions.shape}\")\n      \n       return captions\n```\n\nThe main difference you will see in the `__init__()` method is the use of `nn.TransformerDecoderLayer` and `nn.TransformerDecoder` at line `#(1)` and `#(2)`, where the former is used to initialize a single decoder block, and the latter is for repeating the block multiple times. Next, the `forward()` method is mostly similar to the one in the `Decoder` class, except that the forward propagation on the decoder blocks is automatically repeated four times without needing to be put inside a loop (`#(3)`). One thing that you need to pay attention to in the `decoder_blocks` layer is that the tensor coming from the encoder (features) must be passed as the argument for the `memory` parameter. Meanwhile, the tensor from the decoder itself (captions) has to be passed as the input to the `tgt` parameter.\nThe testing code for the `DecoderTorch` model below is basically the same as the one written in Codeblock 20. Here you can see that this model also generates the final output tensor of size 30×10000.\npython\n\n```\n# Codeblock 22\ndecoder_torch = DecoderTorch()\n\nfeatures = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH))\n\ncaptions = decoder_torch(features, captions, look_ahead_mask)\n```\n\npython\n\n```\n# Codeblock 22 Output\nfeatures             : torch.Size([1, 576, 768])\ncaptions             : torch.Size([1, 30])\nafter embedding      : torch.Size([1, 30, 768])\nafter sin embed      : torch.Size([1, 30, 768])\nafter decoder blocks : torch.Size([1, 30, 768])\nafter linear         : torch.Size([1, 30, 10000])\n```\n\n## The entire CPTR model\nFinally, it’s time to put the encoder and the decoder part we just created into a single class to actually construct the CPTR architecture. You can see in Codeblock 23 below that the implementation is very simple. All we need to do here is just to initialize the encoder (`#(1)`) and the decoder (`#(2)`) components, then pass the raw images and the corresponding caption ground truths as well as the look-ahead mask to the `forward()` method (#(3)). Additionally, it is also possible for you to replace the `Encoder` and the `Decoder` with `EncoderTorch` and `DecoderTorch`, respectively.\npython\n\n```\n# Codeblock 23\nclass EncoderDecoder(nn.Module):\n   def __init__(self):\n       super().__init__()\n       self.encoder = Encoder()  #EncoderTorch()  #(1)\n       self.decoder = Decoder()  #DecoderTorch()  #(2)\n      \n   def forward(self, images, captions, look_ahead_mask):  #(3)\n       print(f\"images\\t\\t\\t: {images.shape}\")\n       print(f\"captions\\t\\t: {captions.shape}\")\n      \n       features = self.encoder(images)\n       print(f\"after encoder\\t\\t: {features.shape}\")\n      \n       captions = self.decoder(features, captions, look_ahead_mask)\n       print(f\"after decoder\\t\\t: {captions.shape}\")\n      \n       return captions\n```\n\nWe can do the testing by passing dummy tensors through it. See the Codeblock 24 below for the details. In this case, images is basically just a tensor of random numbers having the dimension of 1×3×384×384 (`#(1)`), while captions is a tensor of size 1×30 containing random integers (`#(2)`).\npython\n\n```\n# Codeblock 24\nencoder_decoder = EncoderDecoder()\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)  #(1)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH))  #(2)\n\ncaptions = encoder_decoder(images, captions, look_ahead_mask)\n```\n\nBelow is what the output looks like. We can see here that our input images and captions successfully went through all layers in the network, which basically means that the CPTR model we created is now ready to actually be trained on image captioning datasets.\npython\n\n```\n# Codeblock 24 Output\nimages         : torch.Size([1, 3, 384, 384])\ncaptions       : torch.Size([1, 30])\nafter encoder  : torch.Size([1, 576, 768])\nafter decoder  : torch.Size([1, 30, 10000])\n```\n\n## Ending\nThat was pretty much everything about the theory and implementation of the CaPtion TransformeR architecture. Let me know what deep learning architecture I should implement next. Feel free to leave a comment if you spot any mistakes in this article!\n_The code used in this article is available in_[ _my GitHub repo_](https://github.com/MuhammadArdiPutra/medium_articles/blob/main/Image%20Captioning%2C%20Transformer%20Mode%20On.ipynb) _. Here’s the link to my previous article about_[ _image captioning_](https://contributor.insightmediagroup.io/show-and-tell-e1a1142456e2/) _,_[_Vision Transformer (ViT)_](https://contributor.insightmediagroup.io/paper-walkthrough-vision-transformer-vit-c5dcf76f1a7a)_, and the original_[ _Transformer_](https://contributor.insightmediagroup.io/paper-walkthrough-attention-is-all-you-need-80399cdc59e1) _._\n## References\n[1] Wei Liu _et al._ CPTR: Full Transformer Network for Image Captioning. Arxiv.[ https://arxiv.org/pdf/2101.10804](https://arxiv.org/pdf/2101.10804) [Accessed November 16, 2024].\n[2] Oriol Vinyals _et al._ Show and Tell: A Neural Image Caption Generator. Arxiv.[ https://arxiv.org/pdf/1411.4555](https://arxiv.org/pdf/1411.4555) [Accessed December 3, 2024].\n[3] Image originally created by author based on: Alexey Dosovitskiy _et al._ An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. Arxiv.[ https://arxiv.org/pdf/2010.11929](https://arxiv.org/pdf/2010.11929) [Accessed December 3, 2024].\n[4] Image originally created by author based on [6].\n[5] Image originally created by author based on [1].\n[6] Ashish Vaswani _et al._ Attention Is All You Need. Arxiv.[ https://arxiv.org/pdf/1706.03762](https://arxiv.org/pdf/1706.03762) [Accessed December 3, 2024].\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657598287_taxhya.jpeg)\nWritten By\n## [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nDeep learning storyteller\n[See all from Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Image Captioning](https://towardsdatascience.com/tag/image-captioning/)[Transformer](https://towardsdatascience.com/tag/transformer/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F&title=CPTR%20Paper%20Walkthrough%3A%20Image%20Captioning%2C%20Transformer%20Mode%20On)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F&title=CPTR%20Paper%20Walkthrough%3A%20Image%20Captioning%2C%20Transformer%20Mode%20On)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fimage-captioning-transformer-mode-on%2F&text=CPTR%20Paper%20Walkthrough%3A%20Image%20Captioning%2C%20Transformer%20Mode%20On)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/0RkS_9a39Nah1bXso-scaled.jpg)](https://towardsdatascience.com/you-dont-need-a-job-to-get-data-science-experience-26af4fd4c200/)\n## [You Don't Need A Job To Get Data Science Experience](https://towardsdatascience.com/you-dont-need-a-job-to-get-data-science-experience-26af4fd4c200/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nFlipping the coin on its head\n[Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/)\nOctober 4, 20205 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/04/0ujxW0qAvKh1FzgJE.png)](https://towardsdatascience.com/image-data-generators-in-keras-7c5fc6928400/)\n## [Image Data Generators in Keras](https://towardsdatascience.com/image-data-generators-in-keras-7c5fc6928400/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow to effectively and efficiently use data generators in Keras for Computer Vision applications of Deep Learning\n[Manpreet Singh Minhas](https://towardsdatascience.com/author/manpreetsinghminhas/)\nApril 24, 20194 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/07/0jxBva0jm4TGpWN__-scaled.jpg)](https://towardsdatascience.com/can-quantum-machine-learning-help-us-solve-climate-change-3500741d583b/)\n## [Can Quantum Machine Learning Help Us Solve Climate Change?](https://towardsdatascience.com/can-quantum-machine-learning-help-us-solve-climate-change-3500741d583b/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nThe history and potential use cases of quantum-derived technology\n[Daniel García Solla](https://towardsdatascience.com/author/cardstdani/)\nJuly 13, 202219 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/11/1RScKhJWtCNHxJiEFcFrH_A.jpeg)](https://towardsdatascience.com/multi-label-land-cover-classification-with-deep-learning-d39ce2944a3d/)\n## [Multi-label Land Cover Classification with Deep Learning](https://towardsdatascience.com/multi-label-land-cover-classification-with-deep-learning-d39ce2944a3d/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nA step by step guide on Classifying Multi-label Land cover classification using Deep Neural Networks\n[Abdishakur](https://towardsdatascience.com/author/shakasom/)\nNovember 26, 20195 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/04/image-24.jpg)](https://towardsdatascience.com/how-to-use-claude-code-to-build-a-minimum-viable-product/)\n## [How to Use Claude Code to Build a Minimum Viable Product](https://towardsdatascience.com/how-to-use-claude-code-to-build-a-minimum-viable-product/)\n[Agentic AI](https://towardsdatascience.com/category/artificial-intelligence/agentic-ai/)\nLearn how to effectively present product ideas by building MVPs with coding agents\n[Eivind Kjosbakken](https://towardsdatascience.com/author/oieivind/)\nApril 8, 20267 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/07/1TXjs1ZXXN7TEOeUBTn1dkw-scaled.jpeg)](https://towardsdatascience.com/detecting-scams-using-ai-for-real-6c96e712cf09/)\n## [Detecting scams using AI, for real](https://towardsdatascience.com/detecting-scams-using-ai-for-real-6c96e712cf09/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA year-long journey to develop one of the best fraud prevention systems on the market\n[Massimo Belloni](https://towardsdatascience.com/author/massibelloni/)\nJuly 21, 20197 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/06/1hRXZK7oEAf2FA7cwpcqgfw.png)](https://towardsdatascience.com/understanding-cuped-a822523641af/)\n## [Understanding CUPED](https://towardsdatascience.com/understanding-cuped-a822523641af/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nAn in-depth guide to the state-of-the-art variance reduction technique for A/B tests\n[Matteo Courthoud](https://towardsdatascience.com/author/matteo.courthoud/)\nJune 29, 20224 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/openvla-colab-featured-image-clean.png)](https://towardsdatascience.com/i-tried-fine-tuning-a-robot-ai-model-on-colab-here-is-what-worked/)\n## [I Fine-Tuned OpenVLA-7B with LoRA on Google Colab: What Worked and What 100 Steps Proved](https://towardsdatascience.com/i-tried-fine-tuning-a-robot-ai-model-on-colab-here-is-what-worked/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA reproducible 100-step LoRA fine-tuning run for OpenVLA, with dataset checks, Colab setup, training metrics, and W&B evidence.\n[Abdullahi Dattijo](https://towardsdatascience.com/author/abdullahi.dattijo/)\nJuly 21, 202612 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/10271c9WJID5z7I_Gj0KuLg.png)](https://towardsdatascience.com/multi-headed-self-attention-by-hand-d2ce1ae031db/)\n## [Multi-Headed Self Attention - By Hand](https://towardsdatascience.com/multi-headed-self-attention-by-hand-d2ce1ae031db/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHand computing the cornerstone of modern AI.\n[Daniel Warfield](https://towardsdatascience.com/author/danielwarfield1/)\nJuly 12, 20245 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "body_base_path": "attachments/3d5614bc7b74/pages/",
  "body_markdown_path": "attachments/3d5614bc7b74/pages/archived-body.md",
  "local_html": "attachments/3d5614bc7b74/pages/archived-body.html",
  "body_source": "prior_capture"
}