{ "source_file": "0930", "source_platform": "Medium", "author": "Muhammad Ardi Putra", "title_listed": "Image Captioning Paper Walkthrough: Show and Tell", "keywords": "", "domain": "图像处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/show-and-tell-e1a1142456e2/", "publish_date": "2025-02-03", "corpus_stage": "", "notes": "", "id": "642d3128a2ad", "collected_at": "2026-10-02T03:01:11", "title": "Image Captioning Paper Walkthrough: Show and Tell | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Image Captioning Paper Walkthrough: Show and Tell\nImplementing one of the earliest neural image caption generator models with PyTorch.\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nFebruary 3, 202517 min read\n![Photo by Ståle Grut on Unsplash](../images/000.jpg)\nPhoto by [Ståle Grut](https://unsplash.com/@stalebg?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\n## Introduction\nNatural Language Processing and Computer Vision used to be two completely different fields. Well, at least back when I started to learn machine learning and deep learning, I feel like there are multiple paths to follow, and each of them, including NLP and Computer Vision, directs me to a completely different world. Over time, we can now observe that AI becomes more and more advanced, with the intersection between multiple fields of study getting more common, including the two I just mentioned.\nToday, many language models have capability to generate images based on the given prompt. That's one example of the bridge between NLP and Computer Vision. But I guess I'll save it for my upcoming article as it is a bit more complex. Instead, in this article I am going to discuss the simpler one: image captioning. As the name suggests, this is essentially a technique where a specific model accepts an image and returns a text that describes the input image.\nOne of the earliest papers in this topic is the one titled \"_Show and Tell: A Neural Image Caption Generator_ \" written by Vinyals et al. back in 2015 [1]. In this article, I will focus on implementing the deep learning model proposed in the paper using PyTorch. Note that I won't actually demonstrate the training process here as that's a topic on its own. Let me know in the comments if you want a separate tutorial on that.\n* * *\n## Image Captioning Framework\nGenerally speaking, image captioning can be done by combining two types of models: the one specialized to process images and another one capable of processing sequences. I believe you already know what kind of models work best for the two tasks - yes, you're right, those are CNN and RNN, respectively. The idea here is that the CNN is utilized to encode the input image (hence this part is called _encoder_), whereas the RNN is used for generating a sequence of words based on the features encoded by the CNN (hence the RNN part is called _decoder_).\nIt is discussed in the paper that the authors attempted to do so using GoogLeNet (a.k.a., Inception V1) for the encoder and LSTM for the decoder. In fact, the use of GoogLeNet is not explicitly mentioned, yet based on the illustration provided in the paper it seems like the architecture used in the encoder is adopted from the original GoogLeNet paper [2]. The figure below shows what the proposed architecture looks like.\n![Figure 1. The image captioning model proposed in \\[1\\], where the encoder part \\(the leftmost block\\) implements the GoogLeNet model \\[2\\].](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/02/1kKTqOvW7PgvE7vVZKDjAJg.png)\nFigure 1. The image captioning model proposed in [1], where the encoder part (the leftmost block) implements the GoogLeNet model [2].\nTalking more specifically about the connection between the encoder and the decoder, there are several methods available for connecting the two, namely _init-inject_ , _pre-inject_ , _par-inject_ and _merge_ , as mentioned in [3]. In the case of the _Show and Tell_ paper, authors used _pre-inject_ , a method where the features extracted by the encoder are perceived as the 0th word in the caption. Later in the inference phase, we expect the decoder to generate a caption based solely on these image features.\n![Figure 2. The four methods possible to be used to connect the encoder and the decoder part of an image captioning model \\[3\\]. In our case we are going to use the pre-inject method \\(b\\).](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/02/1lIqALUziG9p9abVaosyyVA.png)\nFigure 2. The four methods possible to be used to connect the encoder and the decoder part of an image captioning model [3]. In our case we are going to use the _pre-inject_ method (b).\nAs we already understood the theory behind the image captioning model, we can now jump into the code!\n* * *\n# Implementation\nI'll break the implementation part into three sections: the Encoder, the Decoder, and the combination of the two. Before we actually get into them, we need to import the modules and initialize the required parameters in advance. Look at the Codeblock 1 below to see the modules I use.\npython\n\n```\n# Codeblock 1\nimport torch #(1)\nimport torch.nn as nn #(2)\nimport torchvision.models as models #(3)\nfrom torchvision.models import GoogLeNet_Weights #(4)\n```\n\nLet's break down these imports quickly: the line marked with `#(1)` is used for basic operations, line `#(2)` is for initializing neural network layers, line `#(3)` is for loading various deep learning models, and `#(4)` is the pretrained weights for the GoogLeNet model.\nTalking about the parameter configuration, `EMBED_DIM` and `LSTM_HIDDEN_DIM` are the only two parameters mentioned in the paper, which are both set to 512 as shown at line `#(1)` and `#(2)` in the Codeblock 2 below. The `EMBED_DIM` variable essentially indicates the feature vector size representing a single token in the caption. In this case, we can simply think of a single token as an individual word. Meanwhile, `LSTM_HIDDEN_DIM` is a variable representing the hidden state size inside the LSTM cell. This paper does not mention how many times this RNN-based layer is repeated, but based on the diagram in Figure 1, it seems like it only implements a single LSTM cell. Thus, at line `#(3)` I set the `NUM_LSTM_LAYERS` variable to 1.\npython\n\n```\n# Codeblock 2\nEMBED_DIM = 512 #(1)\nLSTM_HIDDEN_DIM = 512 #(2)\nNUM_LSTM_LAYERS = 1 #(3)\n\nIMAGE_SIZE = 224 #(4)\nIN_CHANNELS = 3 #(5)\n\nSEQ_LENGTH = 30 #(6)\nVOCAB_SIZE = 10000 #(7)\n\nBATCH_SIZE = 1\n```\n\nThe next two parameters are related to the input image, namely `IMAGE_SIZE` (`#(4)`) and `IN_CHANNELS` (`#(5)`). Since we are about to use GoogLeNet for the encoder, we need to match it with its original input shape (3×224×224). Not only for the image, but we also need to configure the parameters for the caption. Here we assume that the caption length is no more than 30 words (`#(6)`) and the number of unique words in the dictionary is 10000 (`#(7)`). Lastly, the `BATCH_SIZE` parameter is used because by default PyTorch processes tensors in a batch. Just to make things simple, the number of image-caption pair within a single batch is set to 1.\n### GoogLeNet Encoder\nIt is actually possible to use any kind of CNN-based model for the encoder. I found on the internet that [4] uses DenseNet, [5] uses Inception V3, and [6] utilizes ResNet for the similar tasks. However, since my goal is to reproduce the model proposed in the paper as closely as possible, I am using the pretrained GoogLeNet model instead. Before we get into the encoder implementation, let's see what the GoogLeNet architecture looks like using the following code.\npython\n\n```\n# Codeblock 3\nmodels.googlenet()\n```\n\nThe resulting output is very long as it lists literally all layers inside the architecture. Here I truncate the output since I only want you to focus on the last layer (the `fc` layer marked with `#(1)` in the Codeblock 3 Output below). You can see that this linear layer maps a feature vector of size 1024 into 1000. Normally, in a standard image classification task, each of these 1000 neurons corresponds to a specific class. So, for example, if you want to perform a 5-class classification task, you would need to modify this layer such that it projects the outputs to 5 neurons only. In our case, we need to make this layer produce a feature vector of length 512 (`EMBED_DIM`). With this, the input image will later be represented as a 512-dimensional vector after being processed by the GoogLeNet model. This feature vector size will exactly match with the token embedding dimension, allowing it to be treated as a part of our word sequence.\ncsharp\n\n```\n# Codeblock 3 Output\nGoogLeNet(\n (conv1): BasicConv2d(\n (conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n )\n (maxpool1): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=True)\n (conv2): BasicConv2d(\n (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n )\n\n .\n .\n .\n .\n\n (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n (dropout): Dropout(p=0.2, inplace=False)\n (fc): Linear(in_features=1024, out_features=1000, bias=True) #(1)\n)\n```\n\nNow let's actually load and modify the GoogLeNet model, which I do in the `InceptionEncoder` class below.\npython\n\n```\n# Codeblock 4a\nclass InceptionEncoder(nn.Module):\n def __init__(self, fine_tune): #(1)\n super().__init__()\n self.googlenet = models.googlenet(weights=GoogLeNet_Weights.IMAGENET1K_V1) #(2)\n self.googlenet.fc = nn.Linear(in_features=self.googlenet.fc.in_features, #(3)\n out_features=EMBED_DIM) #(4)\n\n if fine_tune == True: #(5)\n for param in self.googlenet.parameters():\n param.requires_grad = True\n else:\n for param in self.googlenet.parameters():\n param.requires_grad = False\n\n for param in self.googlenet.fc.parameters():\n param.requires_grad = True\n```\n\nThe first thing we do in the above code is to load the model using `models.googlenet()`. It is mentioned in the paper that the model is already pretrained on the ImageNet dataset. Thus, we need to pass `GoogLeNet_Weights.IMAGENET1K_V1` into the `weights` parameter, as shown at line `#(2)` in Codeblock 4a. Next, at line `#(3)` we access the classification head through the `fc` attribute, where we replace the existing linear layer with a new one having the output dimension of 512 (`EMBED_DIM`) (`#(4)`). Since this GoogLeNet model is already trained, we don't need to train it from scratch. Instead, we can either perform _fine-tuning_ or _transfer learning_ in order to adapt it to the image captioning task.\nIn case you're not yet familiar with the two terms, _fine-tuning_ is a method where we update the weights of the entire model. On the other hand, _transfer learning_ is a technique where we only update the weights of the layers we replaced (in this case it's the last fully-connected layer), while setting the weights of the existing layers non-trainable. To do so, I implement a flag named `fine_tune` at line `#(1)` which will let the model to perform fine-tuning whenever it is set to `True` (`#(5)`).\nThe `forward()` method is pretty straightforward since what we do here is simply passing the input image through the modified GoogLeNet model. See the Codeblock 4b below for the details. Additionally, here I also print out the tensor dimension before and after processing so that you can better understand how the `InceptionEncoder` model works.\npython\n\n```\n# Codeblock 4b\n def forward(self, images):\n print(f'originalt: {images.size()}')\n features = self.googlenet(images)\n print(f'after googlenett: {features.size()}')\n\n return features\n```\n\nTo test whether our decoder works properly, we can pass a dummy tensor of size 1×3×224×224 through the network as demonstrated in Codeblock 5. This tensor dimension simulates a single RGB image of size 224×224. You can see in the resulting output that our image now becomes a single-dimensional feature vector with the length of 512.\npython\n\n```\n# Codeblock 5\ninception_encoder = InceptionEncoder(fine_tune=True)\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nfeatures = inception_encoder(images)\n```\n\ncsharp\n\n```\n# Codeblock 5 Output\noriginal : torch.Size([1, 3, 224, 224])\nafter googlenet : torch.Size([1, 512])\n```\n\n### LSTM Decoder\nAs we have successfully implemented the encoder, now that we are going to create the LSTM decoder, which I demonstrate in Codeblock 6a and 6b. What we need to do first is to initialize the required layers, namely an embedding layer (`#(1)`), the LSTM layer itself (`#(2)`), and a standard linear layer (`#(3)`). The first one (`nn.Embedding`) is responsible for mapping every single token into a 512 (`EMBED_DIM`)-dimensional vector. Meanwhile, the LSTM layer is going to generate a sequence of embedded tokens, where each of these tokens will be mapped into a 10000 (`VOCAB_SIZE`)-dimensional vector by the linear layer. Later on, the values contained in this vector will represent the likelihood of each word in the dictionary being chosen.\npython\n\n```\n# Codeblock 6a\nclass LSTMDecoder(nn.Module):\n def __init__(self):\n super().__init__()\n\n #(1)\n self.embedding = nn.Embedding(num_embeddings=VOCAB_SIZE,\n embedding_dim=EMBED_DIM)\n #(2)\n self.lstm = nn.LSTM(input_size=EMBED_DIM, \n hidden_size=LSTM_HIDDEN_DIM, \n num_layers=NUM_LSTM_LAYERS, \n batch_first=True)\n #(3) \n self.linear = nn.Linear(in_features=LSTM_HIDDEN_DIM, \n out_features=VOCAB_SIZE)\n```\n\nNext, let's define the flow of the network using the following code.\npython\n\n```\n# Codeblock 6b\n def forward(self, features, captions): #(1)\n print(f'features originalt: {features.size()}')\n features = features.unsqueeze(1) #(2)\n print(f\"after unsqueezett: {features.shape}\")\n\n print(f'captions originalt: {captions.size()}')\n captions = self.embedding(captions) #(3)\n print(f\"after embeddingtt: {captions.shape}\")\n\n captions = torch.cat([features, captions], dim=1) #(4)\n print(f\"after concattt: {captions.shape}\")\n\n captions, _ = self.lstm(captions) #(5)\n print(f\"after lstmtt: {captions.shape}\")\n\n captions = self.linear(captions) #(6)\n print(f\"after lineartt: {captions.shape}\")\n\n return captions\n```\n\nYou can see in the above code that the `forward()` method of the `LSTMDecoder` class accepts two inputs: `features` and `captions`, where the former is the image that has been processed by the `InceptionEncoder`, while the latter is the caption of the corresponding image serving as the ground truth (`#(1)`). The idea here is that we are going to perform _pre-inject_ operation by prepending the `features` tensor into `captions` using the code at line `#(4)`. However, keep in mind that we need to adjust the shape of both tensors beforehand. To do so, we have to insert a single dimension at the 1st axis of the image features (`#(2)`). Meanwhile, the shape of the `captions` tensor will align with our requirement right after being processed by the embedding layer (`#(3)`). As the `features` and `captions` have been concatenated, we then pass this tensor through the LSTM layer (`#(5)`) before it is eventually processed by the linear layer (`#(6)`). Look at the testing code below to better understand the flow of the two tensors.\npython\n\n```\n# Codeblock 7\nlstm_decoder = LSTMDecoder()\n\nfeatures = torch.randn(BATCH_SIZE, EMBED_DIM) #(1)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH)) #(2)\n\ncaptions = lstm_decoder(features, captions)\n```\n\nIn Codeblock 7, I assume that `features` is a dummy tensor that represents the output of the `InceptionEncoder` model (`#(1)`). Meanwhile, `captions` is the tensor representing a sequence of tokenized words, where in this case I initialize it as random numbers ranging between 0 to 10000 (`VOCAB_SIZE`) with the length of 30 (`SEQ_LENGTH`) (`#(2)`).\nWe can see in the output below that the features tensor initially has the dimension of 1×512 (`#(1)`). This tensor shape changed to 1×1×512 after being processed with the `unsqueeze()` operation (`#(2)`). The additional dimension in the middle (1) allows the tensor to be treated as a feature vector corresponding to a single timestep, which is necessary for compatibility with the LSTM layer. To the `captions` tensor, its shape changed from 1×30 (`#(3)`) to 1×30×512 (`#(4)`), indicating that every single word is now represented as a 512-dimensional vector.\ncsharp\n\n```\n# Codeblock 7 Output\nfeatures original : torch.Size([1, 512]) #(1)\nafter unsqueeze : torch.Size([1, 1, 512]) #(2)\ncaptions original : torch.Size([1, 30]) #(3)\nafter embedding : torch.Size([1, 30, 512]) #(4)\nafter concat : torch.Size([1, 31, 512]) #(5)\nafter lstm : torch.Size([1, 31, 512]) #(6)\nafter linear : torch.Size([1, 31, 10000]) #(7)\n```\n\nAfter _pre-inject_ operation is performed, our tensor is now having the dimension of 1×31×512, where the `features` tensor becomes the token at the 0th timestep in the sequence (`#(5)`). See the following figure to better illustrate this idea.\n![Figure 3. What the resulting tensor looks like after the pre-injection operation. \\[3\\].](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/02/1RhGAyYwE16KBpEtLGCz5_A.png)\nFigure 3. What the resulting tensor looks like after the pre-injection operation. [3].\nNext, we pass the tensor through the LSTM layer, which in this particular case the output tensor dimension remains the same. However, it is important to note that the tensor shapes at line `#(5)` and `#(6)` in the above output are actually specified by different parameters. The dimensions appear to match here because `EMBED_DIM` and `LSTM_HIDDEN_DIM` were both set to 512. Normally, if we use a different value for `LSTM_HIDDEN_DIM`, then the output dimension is going to be different as well. Finally, we projected each of the 31 token embeddings to a vector of size 10000, which will later contain the probability of every possible token being predicted (`#(7)`).\n### GoogLeNet Encoder + LSTM Decoder\nAt this point, we have successfully created both the encoder and the decoder parts of the image captioning model. What I am going to do next is to combine them together in the `ShowAndTell` class below.\npython\n\n```\n# Codeblock 8a\nclass ShowAndTell(nn.Module):\n def __init__(self):\n super().__init__()\n self.encoder = InceptionEncoder(fine_tune=True) #(1)\n self.decoder = LSTMDecoder() #(2)\n\n def forward(self, images, captions):\n features = self.encoder(images) #(3)\n print(f\"after encodert: {features.shape}\")\n\n captions = self.decoder(features, captions) #(4)\n print(f\"after decodert: {captions.shape}\")\n\n return captions\n```\n\nI think the above code is pretty straightforward. In the `__init__()` method, we only need to initialize the `InceptionEncoder` as well as the `LSTMDecoder` models (`#(1)` and `#(2)`). Here I assume that we are about to perform fine-tuning rather than transfer learning, so I set the `fine_tune` parameter to `True`. Theoretically speaking, fine-tuning is better than transfer learning if you have a relatively large dataset since it works by re-adjusting the weights of the entire model. However, if your dataset is rather small, you should go with transfer learning instead - but that's just the theory. It's definitely a good idea to experiment with both options to see which works best in your case.\nStill with the above codeblock, we configure the `forward()` method to accept image-caption pairs as input. With this configuration, we basically design this method such that it can only be used for training purpose. Here we initially process the raw image with the GoogLeNet inside the encoder block (`#(3)`). Afterwards, we pass the extracted features as well as the tokenized captions into the decoder block and let it produce another token sequence (`#(4)`). In the actual training, this caption output will then be compared with the ground truth to compute the error. This error value is going to be used to compute gradients through backpropagation, which determines how the weights in the network are updated.\nIt is important to know that we cannot use the `forward()` method to perform inference, so we need a separate one for that. In this case, I am going to implement the code specifically to perform inference in the `generate()` method below.\npython\n\n```\n# Codeblock 8b\n def generate(self, images): #(1)\n features = self.encoder(images) #(2)\n print(f\"after encodertt: {features.shape}n\")\n\n words = [] #(3)\n for i in range(SEQ_LENGTH): #(4)\n print(f\"iteration #{i}\")\n features = features.unsqueeze(1)\n print(f\"after unsqueezett: {features.shape}\")\n\n features, _ = self.decoder.lstm(features)\n print(f\"after lstmtt: {features.shape}\")\n\n features = features.squeeze(1) #(5)\n print(f\"after squeezett: {features.shape}\")\n\n probs = self.decoder.linear(features) #(6)\n print(f\"after lineartt: {probs.shape}\")\n\n _, word = probs.max(dim=1) #(7)\n print(f\"after maxtt: {word.shape}\")\n\n words.append(word.item()) #(8)\n\n if word == 1: #(9)\n break\n\n features = self.decoder.embedding(word) #(10)\n print(f\"after embeddingtt: {features.shape}n\")\n\n return words #(11)\n```\n\nInstead of taking two inputs like the previous one, the `generate()` method takes raw image as the only input (`#(1)`). Since we want the features extracted from the image to be the initial input token, we first need to process the raw input image with the encoder block prior to actually generating the subsequent tokens (`#(2)`). Next, we allocate an empty list for storing the token sequence to be produced later (`#(3)`). The tokens themselves are generated one by one, so we wrap the entire process inside a `for` loop, which is going to stop iterating once it reaches at most 30 (`SEQ_LENGTH`) words (`#(4)`).\nThe steps done inside the loop is algorithmically similar to the ones we discussed earlier. However, since the LSTM cell here generates a single token at a time, the process requires the tensor to be treated a bit differently from the one passed through the `forward()` method of the `LSTMDecoder` class back in Codeblock 6b. The first difference you might notice is the `squeeze()` operation (`#(5)`), which is basically just a technical step to be done such that the subsequent layer does the linear projection correctly (`#(6)`). Then, we take the index of the feature vector having the highest value, which corresponds to the token most likely to come next (`#(7)`), and append it to the list we allocated earlier (`#(8)`). The loop is going to break whenever the predicted index is a _stop token_ , which in this case I assume that this token is at the 1st index of the `probs` vector. Otherwise, if the model does not find the _stop token_ , then it is going to convert the last predicted word into its 512 (`EMBED_DIM`)-dimensional vector (`#(10)`), allowing it to be used as the input features for the next iteration. Lastly, the generated word sequence will be returned once the loop is completed (`#(11)`).\nWe are going to simulate the forward pass for the training phase using the Codeblock 9 below. Here I pass two tensors through the `show_and_tell` model (`#(1)`), each representing a raw image of size 3×224×224 (`#(2)`) and a sequence of tokenized words (`#(3)`). Based on the resulting output, we found that our model works properly as the two input tensors successfully passed through the `InceptionEncoder` and the `LSTMDecoder` part of the network.\npython\n\n```\n# Codeblock 9\nshow_and_tell = ShowAndTell() #(1)\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE) #(2)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH)) #(3)\n\ncaptions = show_and_tell(images, captions)\n```\n\ncsharp\n\n```\n# Codeblock 9 Output\nafter encoder : torch.Size([1, 512])\nafter decoder : torch.Size([1, 31, 10000])\n```\n\nNow, let's assume that our `show_and_tell` model is already trained on an image captioning dataset, and thus ready to be used for inference. Look at the Codeblock 10 below to see how I do it. Here we set the model to `eval()` mode (`#(1)`), initialize the input image (`#(2)`), and pass it through the model using the `generate()` method (`#(3)`).\npython\n\n```\n# Codeblock 10\nshow_and_tell.eval() #(1)\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE) #(2)\n\nwith torch.no_grad():\n generated_tokens = show_and_tell.generate(images) #(3)\n```\n\nThe flow of the tensor can be seen in the output below. Here I truncate the resulting outputs because it only shows the same token generation process 30 times.\ncsharp\n\n```\n# Codeblock 10 Output\nafter encoder : torch.Size([1, 512])\n\niteration #0\nafter unsqueeze : torch.Size([1, 1, 512])\nafter lstm : torch.Size([1, 1, 512])\nafter squeeze : torch.Size([1, 512])\nafter linear : torch.Size([1, 10000])\nafter max : torch.Size([1])\nafter embedding : torch.Size([1, 512])\n\niteration #1\nafter unsqueeze : torch.Size([1, 1, 512])\nafter lstm : torch.Size([1, 1, 512])\nafter squeeze : torch.Size([1, 512])\nafter linear : torch.Size([1, 10000])\nafter max : torch.Size([1])\nafter embedding : torch.Size([1, 512])\n\n.\n.\n.\n.\n```\n\nTo see what the resulting caption looks like, we can just print out the `generated_tokens` list as shown below. Keep in mind that this sequence is still in the form of tokenized words. Later, in the post-processing stage, we will need to convert them back to the words corresponding to these numbers.\npython\n\n```\n# Codeblock 11\ngenerated_tokens\n```\n\ncsharp\n\n```\n# Codeblock 11 Output\n[5627,\n 3906,\n 2370,\n 2299,\n 4952,\n 9933,\n 402,\n 7775,\n 602,\n 4414,\n 8667,\n 6774,\n 9345,\n 8750,\n 3680,\n 4458,\n 1677,\n 5998,\n 8572,\n 9556,\n 7347,\n 6780,\n 9672,\n 2596,\n 9218,\n 1880,\n 4396,\n 6168,\n 7999,\n 454]\n```\n\n* * *\n## Ending\nWith the above output, we've reached the end of our discussion on image captioning. Over time, many other researchers attempted to make improvements to accomplish this task. So, I think in the upcoming article I will discuss the state-of-the-art method on this topic.\nThanks for reading, I hope you learn something new today!\n_By the way you can also find the code used in this article [here](https://github.com/MuhammadArdiPutra/medium_articles/blob/main/Show%20and%20Tell.ipynb)._\n* * *\n## References\n[1] Oriol Vinyals et al. Show and Tell: A Neural Image Caption Generator. Arxiv. [Accessed November 13, 2024].\n[2] Christian Szegedy et al. Going Deeper with Convolutions. Arxiv. [Accessed November 13, 2024].\n[3] Marc Tanti et al. Where to put the Image in an Image Caption Generator. Arxiv. [Accessed November 13, 2024].\n[4] [Stepan Ulyanin](https://towardsdatascience.com/show-and-tell-e1a1142456e2/None). Captioning Images with CNN and RNN, using PyTorch. Medium. [Accessed November 16, 2024].\n[5] [Saketh Kotamraju](https://towardsdatascience.com/show-and-tell-e1a1142456e2/None). How to Build an Image-Captioning Model in Pytorch. Towards Data Science. [Accessed November 16, 2024].\n[6] Code with Aarohi. Image Captioning using CNN and RNN | Image Captioning using Deep Learning. YouTube. [Accessed November 16, 2024].\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-59e1c17daab1fcb582c5.jpeg)\nWritten By\n## [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nDeep learning storyteller\n[See all from Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\n[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Computer Vision](https://towardsdatascience.com/tag/computer-vision/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[NLP](https://towardsdatascience.com/tag/nlp/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F&title=Image%20Captioning%20Paper%20Walkthrough%3A%20Show%20and%20Tell)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F&title=Image%20Captioning%20Paper%20Walkthrough%3A%20Show%20and%20Tell)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F&text=Image%20Captioning%20Paper%20Walkthrough%3A%20Show%20and%20Tell)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/004.jpeg)](https://towardsdatascience.com/image-processing-using-streamlit-d650fb0ccf8/)\n## [Image Processing using Streamlit](https://towardsdatascience.com/image-processing-using-streamlit-d650fb0ccf8/)\n[](https://towardsdatascience.com/show-and-tell-e1a1142456e2/)\nIn this article, we will see how we can use Streamlit with Image Processing Techniques and Object Detection Algorithms. I am assuming that...\n[Aniket Wattamwar](https://towardsdatascience.com/author/aniket-wattamwar/)\nNovember 1, 20205 min read\n * [![](../images/005.jpeg)](https://towardsdatascience.com/nlp-video-summarization-with-watson-and-gpt-4d4db219b599/)\n## [NLP-Video Summarization with Watson and GPT](https://towardsdatascience.com/nlp-video-summarization-with-watson-and-gpt-4d4db219b599/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nSiri, Please watch my 1-hour long University lecture video and make a summary for me\n[Alex Aman](https://towardsdatascience.com/author/alexamannnn/)\nSeptember 11, 20203 min read\n * [![](../images/006.png)](https://towardsdatascience.com/a-loss-function-suitable-for-class-imbalanced-data-focal-loss-af1702d75d75/)\n## [A Loss Function Suitable for Class Imbalanced Data: \"Focal Loss\"](https://towardsdatascience.com/a-loss-function-suitable-for-class-imbalanced-data-focal-loss-af1702d75d75/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nDeep Learning with Class Imbalanced Data\n[Saptashwa Bhattacharyya](https://towardsdatascience.com/author/saptashwa/)\nFebruary 15, 20219 min read\n * [![](../images/007.jpeg)](https://towardsdatascience.com/back-propagation-the-easy-way-part-2-bea37046c897/)\n## [Back Propagation, the Easy Way (Part 2)](https://towardsdatascience.com/back-propagation-the-easy-way-part-2-bea37046c897/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nPractical implementation\n[Ziad SALLOUM](https://towardsdatascience.com/author/zsalloum/)\nJanuary 19, 20196 min read\n * [![](../images/008.png)](https://towardsdatascience.com/knowledge-distillation-simplified-dd4973dbc764/)\n## [Knowledge Distillation : Simplified](https://towardsdatascience.com/knowledge-distillation-simplified-dd4973dbc764/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nTake a peek into the world of Teacher Student networks\n[Prakhar Ganesh](https://towardsdatascience.com/author/prakhargannu/)\nAugust 12, 20194 min read\n * [![](../images/009.jpeg)](https://towardsdatascience.com/medical-imaging-with-azure-machine-learning-b5acfd772dd5/)\n## [Medical Imaging with Azure Machine Learning](https://towardsdatascience.com/medical-imaging-with-azure-machine-learning-b5acfd772dd5/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow Azure ML can be leveraged to support medical imaging use cases in areas like data, model management, deployment and MLOps\n[Harmke Alkemade](https://towardsdatascience.com/author/harmkealkemade/)\nDecember 20, 202113 min read\n * [![](../images/010.jpeg)](https://towardsdatascience.com/3-lessons-i-have-learned-after-i-started-working-as-a-data-scientist-893207d954b2/)\n## [3 Lessons I Have Learned After I Started Working as a Data Scientist](https://towardsdatascience.com/3-lessons-i-have-learned-after-i-started-working-as-a-data-scientist-893207d954b2/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat I would have done differently\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nMarch 15, 20214 min read\n * [![](../images/011.jpg)](https://towardsdatascience.com/graph-embeddings-explained-f0d8d1c49ec/)\n## [Graph Embeddings Explained](https://towardsdatascience.com/graph-embeddings-explained-f0d8d1c49ec/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nOverview and Python Implementation of Node, Edge and Graph Embedding Methods\n[Vatsal](https://towardsdatascience.com/author/vatsal12-p/)\nNovember 7, 20227 min read\n * [![](../images/012.png)](https://towardsdatascience.com/calculating-audio-song-similarity-using-siamese-neural-networks-62730e8f3e3d/)\n## [Calculating Audio Song Similarity Using Siamese Neural Networks](https://towardsdatascience.com/calculating-audio-song-similarity-using-siamese-neural-networks-62730e8f3e3d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAt AI Music, where our back catalogue of content grows every day, it is becoming increasingly necessary for us to create more intelligent...\n[Thomas Vassallo](https://towardsdatascience.com/author/tom_26639/)\nAugust 28, 20206 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "metadata": { "title": "Image Captioning Paper Walkthrough: Show and Tell | Towards Data Science", "description": "Implementing one of the earliest neural image caption generator models with PyTorch.", "keywords": null, "author": "Muhammad Ardi Putra", "og:locale": "en_US", "og:type": "article", "og:title": "Image Captioning Paper Walkthrough: Show and Tell", "og:description": "Implementing one of the earliest neural image caption generator models with PyTorch.", "og:url": "https://towardsdatascience.com/show-and-tell-e1a1142456e2/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/02/0PwKPzH0bOaxiLRQH-scaled.jpg", "og:image:width": "2560", "og:image:height": "1707", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Muhammad Ardi Putra", "twitter:label2": "Est. reading time", "twitter:data2": "17 min read", "article:published_time": "2025-02-03T16:30:24.000Z", "article:modified_time": "2025-02-03T16:30:24.000Z", "article:author": "https://towardsdatascience.com/author/muhammad_ardi/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 202, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/642d3128a2ad/code/block_000.txt", "chars": 158 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_001.txt", "chars": 247 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_002.txt", "chars": 33 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_003.txt", "chars": 702 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_004.txt", "chars": 706 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_005.txt", "chars": 212 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_006.txt", "chars": 175 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_007.txt", "chars": 109 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_008.txt", "chars": 603 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_009.txt", "chars": 799 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_010.txt", "chars": 213 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_011.txt", "chars": 385 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_012.txt", "chars": 482 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_013.txt", "chars": 1069 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_014.txt", "chars": 247 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_015.txt", "chars": 101 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_016.txt", "chars": 202 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_017.txt", "chars": 585 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_018.txt", "chars": 32 }, { "language": "txt", "file": "attachments/642d3128a2ad/code/block_019.txt", "chars": 229 } ], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:56:53Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:56:53Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:56:53Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:56:53Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=htNmFL2BG34>", "kind": "invalid_video_url", "status": "download_failed", "title": "", "error": "[youtube] Extracting URL: https://www.youtube.com/watch?v=htNmFL2BG34>\n[youtube] htNmFL2BG34: Downloading webpage\n[youtube] htNmFL2BG34: Downloading web embedded client config\n[youtube] htNmFL2BG34: Downloading player 67288f35-main\n[youtube] htNmFL2BG34: Downloading web embedded player API JSON\n[youtube] htNmFL2BG34: Downloading tv downgraded player API JSON\n[youtube] [jsc:deno] Solving JS challenges using deno\n[info] htNmFL2BG34: Downloading 1 format(s): 137+251\n[info] Video metadata is already present\n[download] /www/wwwroot/dataset/xinxing-zimeiti-corpus/attachments/642d3128a2ad/videos/repair-htNmFL2BG34.mp4 has already been downloaded\n", "retried_at": "2026-10-20T22:24:26Z" }, { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" }, { "url": "https://www.youtube.com/watch?v=htNmFL2BG34", "status": "ok", "local_path": "attachments/642d3128a2ad/videos/htNmFL2BG34.mp4", "bytes": 392991539, "valid": true, "height": 1080, "width": 1920, "duration": "2744.841000" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/muhammad_ardi/" }, { "type": "internal", "href": "https://towardsdatascience.com/show-and-tell-e1a1142456e2/None" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/computer-vision/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/deep-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/hands-on-tutorials/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/nlp/" }, { "type": "internal", "href": "https://towardsdatascience.com/image-processing-using-streamlit-d650fb0ccf8/" }, { "type": "internal", "href": "https://towardsdatascience.com/show-and-tell-e1a1142456e2/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/aniket-wattamwar/" }, { "type": "internal", "href": "https://towardsdatascience.com/nlp-video-summarization-with-watson-and-gpt-4d4db219b599/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/alexamannnn/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://unsplash.com/@stalebg" }, { "type": "external", "href": "https://unsplash.com" }, { "type": "external", "href": "https://github.com/MuhammadArdiPutra/medium_articles/blob/main/Show%20and%20Tell.ipynb" }, { "type": "external", "href": "https://arxiv.org/pdf/1411.4555" }, { "type": "external", "href": "https://arxiv.org/pdf/1409.4842" }, { "type": "external", "href": "https://arxiv.org/pdf/1703.09137" }, { "type": "external", "href": "https://medium.com/@stepanulyanin/captioning-images-with-pytorch-bc592e5fd1a3" }, { "type": "external", "href": "https://contributor.insightmediagroup.io/how-to-build-an-image-captioning-model-in-pytorch-29b9d8fe2f8c" }, { "type": "external", "href": "https://www.youtube.com/watch?v=htNmFL2BG34" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=Image+Captioning+Paper+Walkthrough%3A+Show+and+Tell&u=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=Image+Captioning+Paper+Walkthrough%3A+Show+and+Tell&url=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F" }, { "type": "external", "href": "https://x.com/share?text=Image+Captioning+Paper+Walkthrough%3A+Show+and+Tell&url=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F" }, { "type": "external", "href": "https://www.youtube.com/c/TowardsDataScience" }, { "type": "external", "href": "https://www.threads.net/@towardsdatascience" }, { "type": "external", "href": "https://bsky.app/profile/towardsdatascience.com" }, { "type": "external", "href": "https://thenewstack.io" }, { "type": "external", "href": "https://roadmap.sh" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:29:26Z", "source_url": "https://towardsdatascience.com/show-and-tell-e1a1142456e2/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [ { "local_path": "attachments/642d3128a2ad/videos/htNmFL2BG34.mp4", "bytes": 392991539, "valid": true, "height": 1080, "width": 1920, "duration": "2744.841000", "status": "ok" }, { "local_path": "attachments/642d3128a2ad/videos/repair-htNmFL2BG34.mp4", "bytes": 392991539, "valid": true, "height": 1080, "width": 1920, "duration": "2744.841000", "status": "ok" } ], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Image Captioning Paper Walkthrough: Show and Tell\nImplementing one of the earliest neural image caption generator models with PyTorch.\n[Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nFebruary 3, 202517 min read\n![Photo by Ståle Grut on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/02/0PwKPzH0bOaxiLRQH-scaled.jpg)\nPhoto by [Ståle Grut](https://unsplash.com/@stalebg?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\n## Introduction\nNatural Language Processing and Computer Vision used to be two completely different fields. Well, at least back when I started to learn machine learning and deep learning, I feel like there are multiple paths to follow, and each of them, including NLP and Computer Vision, directs me to a completely different world. Over time, we can now observe that AI becomes more and more advanced, with the intersection between multiple fields of study getting more common, including the two I just mentioned.\nToday, many language models have capability to generate images based on the given prompt. That's one example of the bridge between NLP and Computer Vision. But I guess I'll save it for my upcoming article as it is a bit more complex. Instead, in this article I am going to discuss the simpler one: image captioning. As the name suggests, this is essentially a technique where a specific model accepts an image and returns a text that describes the input image.\nOne of the earliest papers in this topic is the one titled \"_Show and Tell: A Neural Image Caption Generator_ \" written by Vinyals et al. back in 2015 [1]. In this article, I will focus on implementing the deep learning model proposed in the paper using PyTorch. Note that I won't actually demonstrate the training process here as that's a topic on its own. Let me know in the comments if you want a separate tutorial on that.\n* * *\n## Image Captioning Framework\nGenerally speaking, image captioning can be done by combining two types of models: the one specialized to process images and another one capable of processing sequences. I believe you already know what kind of models work best for the two tasks - yes, you're right, those are CNN and RNN, respectively. The idea here is that the CNN is utilized to encode the input image (hence this part is called _encoder_), whereas the RNN is used for generating a sequence of words based on the features encoded by the CNN (hence the RNN part is called _decoder_).\nIt is discussed in the paper that the authors attempted to do so using GoogLeNet (a.k.a., Inception V1) for the encoder and LSTM for the decoder. In fact, the use of GoogLeNet is not explicitly mentioned, yet based on the illustration provided in the paper it seems like the architecture used in the encoder is adopted from the original GoogLeNet paper [2]. The figure below shows what the proposed architecture looks like.\n![Figure 1. The image captioning model proposed in \\[1\\], where the encoder part \\(the leftmost block\\) implements the GoogLeNet model \\[2\\].](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/02/1kKTqOvW7PgvE7vVZKDjAJg.png)\nFigure 1. The image captioning model proposed in [1], where the encoder part (the leftmost block) implements the GoogLeNet model [2].\nTalking more specifically about the connection between the encoder and the decoder, there are several methods available for connecting the two, namely _init-inject_ , _pre-inject_ , _par-inject_ and _merge_ , as mentioned in [3]. In the case of the _Show and Tell_ paper, authors used _pre-inject_ , a method where the features extracted by the encoder are perceived as the 0th word in the caption. Later in the inference phase, we expect the decoder to generate a caption based solely on these image features.\n![Figure 2. The four methods possible to be used to connect the encoder and the decoder part of an image captioning model \\[3\\]. In our case we are going to use the pre-inject method \\(b\\).](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/02/1lIqALUziG9p9abVaosyyVA.png)\nFigure 2. The four methods possible to be used to connect the encoder and the decoder part of an image captioning model [3]. In our case we are going to use the _pre-inject_ method (b).\nAs we already understood the theory behind the image captioning model, we can now jump into the code!\n* * *\n# Implementation\nI'll break the implementation part into three sections: the Encoder, the Decoder, and the combination of the two. Before we actually get into them, we need to import the modules and initialize the required parameters in advance. Look at the Codeblock 1 below to see the modules I use.\npython\n\n```\n# Codeblock 1\nimport torch #(1)\nimport torch.nn as nn #(2)\nimport torchvision.models as models #(3)\nfrom torchvision.models import GoogLeNet_Weights #(4)\n```\n\nLet's break down these imports quickly: the line marked with `#(1)` is used for basic operations, line `#(2)` is for initializing neural network layers, line `#(3)` is for loading various deep learning models, and `#(4)` is the pretrained weights for the GoogLeNet model.\nTalking about the parameter configuration, `EMBED_DIM` and `LSTM_HIDDEN_DIM` are the only two parameters mentioned in the paper, which are both set to 512 as shown at line `#(1)` and `#(2)` in the Codeblock 2 below. The `EMBED_DIM` variable essentially indicates the feature vector size representing a single token in the caption. In this case, we can simply think of a single token as an individual word. Meanwhile, `LSTM_HIDDEN_DIM` is a variable representing the hidden state size inside the LSTM cell. This paper does not mention how many times this RNN-based layer is repeated, but based on the diagram in Figure 1, it seems like it only implements a single LSTM cell. Thus, at line `#(3)` I set the `NUM_LSTM_LAYERS` variable to 1.\npython\n\n```\n# Codeblock 2\nEMBED_DIM = 512 #(1)\nLSTM_HIDDEN_DIM = 512 #(2)\nNUM_LSTM_LAYERS = 1 #(3)\n\nIMAGE_SIZE = 224 #(4)\nIN_CHANNELS = 3 #(5)\n\nSEQ_LENGTH = 30 #(6)\nVOCAB_SIZE = 10000 #(7)\n\nBATCH_SIZE = 1\n```\n\nThe next two parameters are related to the input image, namely `IMAGE_SIZE` (`#(4)`) and `IN_CHANNELS` (`#(5)`). Since we are about to use GoogLeNet for the encoder, we need to match it with its original input shape (3×224×224). Not only for the image, but we also need to configure the parameters for the caption. Here we assume that the caption length is no more than 30 words (`#(6)`) and the number of unique words in the dictionary is 10000 (`#(7)`). Lastly, the `BATCH_SIZE` parameter is used because by default PyTorch processes tensors in a batch. Just to make things simple, the number of image-caption pair within a single batch is set to 1.\n### GoogLeNet Encoder\nIt is actually possible to use any kind of CNN-based model for the encoder. I found on the internet that [4] uses DenseNet, [5] uses Inception V3, and [6] utilizes ResNet for the similar tasks. However, since my goal is to reproduce the model proposed in the paper as closely as possible, I am using the pretrained GoogLeNet model instead. Before we get into the encoder implementation, let's see what the GoogLeNet architecture looks like using the following code.\npython\n\n```\n# Codeblock 3\nmodels.googlenet()\n```\n\nThe resulting output is very long as it lists literally all layers inside the architecture. Here I truncate the output since I only want you to focus on the last layer (the `fc` layer marked with `#(1)` in the Codeblock 3 Output below). You can see that this linear layer maps a feature vector of size 1024 into 1000. Normally, in a standard image classification task, each of these 1000 neurons corresponds to a specific class. So, for example, if you want to perform a 5-class classification task, you would need to modify this layer such that it projects the outputs to 5 neurons only. In our case, we need to make this layer produce a feature vector of length 512 (`EMBED_DIM`). With this, the input image will later be represented as a 512-dimensional vector after being processed by the GoogLeNet model. This feature vector size will exactly match with the token embedding dimension, allowing it to be treated as a part of our word sequence.\ncsharp\n\n```\n# Codeblock 3 Output\nGoogLeNet(\n (conv1): BasicConv2d(\n (conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n )\n (maxpool1): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=True)\n (conv2): BasicConv2d(\n (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n )\n\n .\n .\n .\n .\n\n (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n (dropout): Dropout(p=0.2, inplace=False)\n (fc): Linear(in_features=1024, out_features=1000, bias=True) #(1)\n)\n```\n\nNow let's actually load and modify the GoogLeNet model, which I do in the `InceptionEncoder` class below.\npython\n\n```\n# Codeblock 4a\nclass InceptionEncoder(nn.Module):\n def __init__(self, fine_tune): #(1)\n super().__init__()\n self.googlenet = models.googlenet(weights=GoogLeNet_Weights.IMAGENET1K_V1) #(2)\n self.googlenet.fc = nn.Linear(in_features=self.googlenet.fc.in_features, #(3)\n out_features=EMBED_DIM) #(4)\n\n if fine_tune == True: #(5)\n for param in self.googlenet.parameters():\n param.requires_grad = True\n else:\n for param in self.googlenet.parameters():\n param.requires_grad = False\n\n for param in self.googlenet.fc.parameters():\n param.requires_grad = True\n```\n\nThe first thing we do in the above code is to load the model using `models.googlenet()`. It is mentioned in the paper that the model is already pretrained on the ImageNet dataset. Thus, we need to pass `GoogLeNet_Weights.IMAGENET1K_V1` into the `weights` parameter, as shown at line `#(2)` in Codeblock 4a. Next, at line `#(3)` we access the classification head through the `fc` attribute, where we replace the existing linear layer with a new one having the output dimension of 512 (`EMBED_DIM`) (`#(4)`). Since this GoogLeNet model is already trained, we don't need to train it from scratch. Instead, we can either perform _fine-tuning_ or _transfer learning_ in order to adapt it to the image captioning task.\nIn case you're not yet familiar with the two terms, _fine-tuning_ is a method where we update the weights of the entire model. On the other hand, _transfer learning_ is a technique where we only update the weights of the layers we replaced (in this case it's the last fully-connected layer), while setting the weights of the existing layers non-trainable. To do so, I implement a flag named `fine_tune` at line `#(1)` which will let the model to perform fine-tuning whenever it is set to `True` (`#(5)`).\nThe `forward()` method is pretty straightforward since what we do here is simply passing the input image through the modified GoogLeNet model. See the Codeblock 4b below for the details. Additionally, here I also print out the tensor dimension before and after processing so that you can better understand how the `InceptionEncoder` model works.\npython\n\n```\n# Codeblock 4b\n def forward(self, images):\n print(f'originalt: {images.size()}')\n features = self.googlenet(images)\n print(f'after googlenett: {features.size()}')\n\n return features\n```\n\nTo test whether our decoder works properly, we can pass a dummy tensor of size 1×3×224×224 through the network as demonstrated in Codeblock 5. This tensor dimension simulates a single RGB image of size 224×224. You can see in the resulting output that our image now becomes a single-dimensional feature vector with the length of 512.\npython\n\n```\n# Codeblock 5\ninception_encoder = InceptionEncoder(fine_tune=True)\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)\nfeatures = inception_encoder(images)\n```\n\ncsharp\n\n```\n# Codeblock 5 Output\noriginal : torch.Size([1, 3, 224, 224])\nafter googlenet : torch.Size([1, 512])\n```\n\n### LSTM Decoder\nAs we have successfully implemented the encoder, now that we are going to create the LSTM decoder, which I demonstrate in Codeblock 6a and 6b. What we need to do first is to initialize the required layers, namely an embedding layer (`#(1)`), the LSTM layer itself (`#(2)`), and a standard linear layer (`#(3)`). The first one (`nn.Embedding`) is responsible for mapping every single token into a 512 (`EMBED_DIM`)-dimensional vector. Meanwhile, the LSTM layer is going to generate a sequence of embedded tokens, where each of these tokens will be mapped into a 10000 (`VOCAB_SIZE`)-dimensional vector by the linear layer. Later on, the values contained in this vector will represent the likelihood of each word in the dictionary being chosen.\npython\n\n```\n# Codeblock 6a\nclass LSTMDecoder(nn.Module):\n def __init__(self):\n super().__init__()\n\n #(1)\n self.embedding = nn.Embedding(num_embeddings=VOCAB_SIZE,\n embedding_dim=EMBED_DIM)\n #(2)\n self.lstm = nn.LSTM(input_size=EMBED_DIM, \n hidden_size=LSTM_HIDDEN_DIM, \n num_layers=NUM_LSTM_LAYERS, \n batch_first=True)\n #(3) \n self.linear = nn.Linear(in_features=LSTM_HIDDEN_DIM, \n out_features=VOCAB_SIZE)\n```\n\nNext, let's define the flow of the network using the following code.\npython\n\n```\n# Codeblock 6b\n def forward(self, features, captions): #(1)\n print(f'features originalt: {features.size()}')\n features = features.unsqueeze(1) #(2)\n print(f\"after unsqueezett: {features.shape}\")\n\n print(f'captions originalt: {captions.size()}')\n captions = self.embedding(captions) #(3)\n print(f\"after embeddingtt: {captions.shape}\")\n\n captions = torch.cat([features, captions], dim=1) #(4)\n print(f\"after concattt: {captions.shape}\")\n\n captions, _ = self.lstm(captions) #(5)\n print(f\"after lstmtt: {captions.shape}\")\n\n captions = self.linear(captions) #(6)\n print(f\"after lineartt: {captions.shape}\")\n\n return captions\n```\n\nYou can see in the above code that the `forward()` method of the `LSTMDecoder` class accepts two inputs: `features` and `captions`, where the former is the image that has been processed by the `InceptionEncoder`, while the latter is the caption of the corresponding image serving as the ground truth (`#(1)`). The idea here is that we are going to perform _pre-inject_ operation by prepending the `features` tensor into `captions` using the code at line `#(4)`. However, keep in mind that we need to adjust the shape of both tensors beforehand. To do so, we have to insert a single dimension at the 1st axis of the image features (`#(2)`). Meanwhile, the shape of the `captions` tensor will align with our requirement right after being processed by the embedding layer (`#(3)`). As the `features` and `captions` have been concatenated, we then pass this tensor through the LSTM layer (`#(5)`) before it is eventually processed by the linear layer (`#(6)`). Look at the testing code below to better understand the flow of the two tensors.\npython\n\n```\n# Codeblock 7\nlstm_decoder = LSTMDecoder()\n\nfeatures = torch.randn(BATCH_SIZE, EMBED_DIM) #(1)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH)) #(2)\n\ncaptions = lstm_decoder(features, captions)\n```\n\nIn Codeblock 7, I assume that `features` is a dummy tensor that represents the output of the `InceptionEncoder` model (`#(1)`). Meanwhile, `captions` is the tensor representing a sequence of tokenized words, where in this case I initialize it as random numbers ranging between 0 to 10000 (`VOCAB_SIZE`) with the length of 30 (`SEQ_LENGTH`) (`#(2)`).\nWe can see in the output below that the features tensor initially has the dimension of 1×512 (`#(1)`). This tensor shape changed to 1×1×512 after being processed with the `unsqueeze()` operation (`#(2)`). The additional dimension in the middle (1) allows the tensor to be treated as a feature vector corresponding to a single timestep, which is necessary for compatibility with the LSTM layer. To the `captions` tensor, its shape changed from 1×30 (`#(3)`) to 1×30×512 (`#(4)`), indicating that every single word is now represented as a 512-dimensional vector.\ncsharp\n\n```\n# Codeblock 7 Output\nfeatures original : torch.Size([1, 512]) #(1)\nafter unsqueeze : torch.Size([1, 1, 512]) #(2)\ncaptions original : torch.Size([1, 30]) #(3)\nafter embedding : torch.Size([1, 30, 512]) #(4)\nafter concat : torch.Size([1, 31, 512]) #(5)\nafter lstm : torch.Size([1, 31, 512]) #(6)\nafter linear : torch.Size([1, 31, 10000]) #(7)\n```\n\nAfter _pre-inject_ operation is performed, our tensor is now having the dimension of 1×31×512, where the `features` tensor becomes the token at the 0th timestep in the sequence (`#(5)`). See the following figure to better illustrate this idea.\n![Figure 3. What the resulting tensor looks like after the pre-injection operation. \\[3\\].](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/02/1RhGAyYwE16KBpEtLGCz5_A.png)\nFigure 3. What the resulting tensor looks like after the pre-injection operation. [3].\nNext, we pass the tensor through the LSTM layer, which in this particular case the output tensor dimension remains the same. However, it is important to note that the tensor shapes at line `#(5)` and `#(6)` in the above output are actually specified by different parameters. The dimensions appear to match here because `EMBED_DIM` and `LSTM_HIDDEN_DIM` were both set to 512. Normally, if we use a different value for `LSTM_HIDDEN_DIM`, then the output dimension is going to be different as well. Finally, we projected each of the 31 token embeddings to a vector of size 10000, which will later contain the probability of every possible token being predicted (`#(7)`).\n### GoogLeNet Encoder + LSTM Decoder\nAt this point, we have successfully created both the encoder and the decoder parts of the image captioning model. What I am going to do next is to combine them together in the `ShowAndTell` class below.\npython\n\n```\n# Codeblock 8a\nclass ShowAndTell(nn.Module):\n def __init__(self):\n super().__init__()\n self.encoder = InceptionEncoder(fine_tune=True) #(1)\n self.decoder = LSTMDecoder() #(2)\n\n def forward(self, images, captions):\n features = self.encoder(images) #(3)\n print(f\"after encodert: {features.shape}\")\n\n captions = self.decoder(features, captions) #(4)\n print(f\"after decodert: {captions.shape}\")\n\n return captions\n```\n\nI think the above code is pretty straightforward. In the `__init__()` method, we only need to initialize the `InceptionEncoder` as well as the `LSTMDecoder` models (`#(1)` and `#(2)`). Here I assume that we are about to perform fine-tuning rather than transfer learning, so I set the `fine_tune` parameter to `True`. Theoretically speaking, fine-tuning is better than transfer learning if you have a relatively large dataset since it works by re-adjusting the weights of the entire model. However, if your dataset is rather small, you should go with transfer learning instead - but that's just the theory. It's definitely a good idea to experiment with both options to see which works best in your case.\nStill with the above codeblock, we configure the `forward()` method to accept image-caption pairs as input. With this configuration, we basically design this method such that it can only be used for training purpose. Here we initially process the raw image with the GoogLeNet inside the encoder block (`#(3)`). Afterwards, we pass the extracted features as well as the tokenized captions into the decoder block and let it produce another token sequence (`#(4)`). In the actual training, this caption output will then be compared with the ground truth to compute the error. This error value is going to be used to compute gradients through backpropagation, which determines how the weights in the network are updated.\nIt is important to know that we cannot use the `forward()` method to perform inference, so we need a separate one for that. In this case, I am going to implement the code specifically to perform inference in the `generate()` method below.\npython\n\n```\n# Codeblock 8b\n def generate(self, images): #(1)\n features = self.encoder(images) #(2)\n print(f\"after encodertt: {features.shape}n\")\n\n words = [] #(3)\n for i in range(SEQ_LENGTH): #(4)\n print(f\"iteration #{i}\")\n features = features.unsqueeze(1)\n print(f\"after unsqueezett: {features.shape}\")\n\n features, _ = self.decoder.lstm(features)\n print(f\"after lstmtt: {features.shape}\")\n\n features = features.squeeze(1) #(5)\n print(f\"after squeezett: {features.shape}\")\n\n probs = self.decoder.linear(features) #(6)\n print(f\"after lineartt: {probs.shape}\")\n\n _, word = probs.max(dim=1) #(7)\n print(f\"after maxtt: {word.shape}\")\n\n words.append(word.item()) #(8)\n\n if word == 1: #(9)\n break\n\n features = self.decoder.embedding(word) #(10)\n print(f\"after embeddingtt: {features.shape}n\")\n\n return words #(11)\n```\n\nInstead of taking two inputs like the previous one, the `generate()` method takes raw image as the only input (`#(1)`). Since we want the features extracted from the image to be the initial input token, we first need to process the raw input image with the encoder block prior to actually generating the subsequent tokens (`#(2)`). Next, we allocate an empty list for storing the token sequence to be produced later (`#(3)`). The tokens themselves are generated one by one, so we wrap the entire process inside a `for` loop, which is going to stop iterating once it reaches at most 30 (`SEQ_LENGTH`) words (`#(4)`).\nThe steps done inside the loop is algorithmically similar to the ones we discussed earlier. However, since the LSTM cell here generates a single token at a time, the process requires the tensor to be treated a bit differently from the one passed through the `forward()` method of the `LSTMDecoder` class back in Codeblock 6b. The first difference you might notice is the `squeeze()` operation (`#(5)`), which is basically just a technical step to be done such that the subsequent layer does the linear projection correctly (`#(6)`). Then, we take the index of the feature vector having the highest value, which corresponds to the token most likely to come next (`#(7)`), and append it to the list we allocated earlier (`#(8)`). The loop is going to break whenever the predicted index is a _stop token_ , which in this case I assume that this token is at the 1st index of the `probs` vector. Otherwise, if the model does not find the _stop token_ , then it is going to convert the last predicted word into its 512 (`EMBED_DIM`)-dimensional vector (`#(10)`), allowing it to be used as the input features for the next iteration. Lastly, the generated word sequence will be returned once the loop is completed (`#(11)`).\nWe are going to simulate the forward pass for the training phase using the Codeblock 9 below. Here I pass two tensors through the `show_and_tell` model (`#(1)`), each representing a raw image of size 3×224×224 (`#(2)`) and a sequence of tokenized words (`#(3)`). Based on the resulting output, we found that our model works properly as the two input tensors successfully passed through the `InceptionEncoder` and the `LSTMDecoder` part of the network.\npython\n\n```\n# Codeblock 9\nshow_and_tell = ShowAndTell() #(1)\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE) #(2)\ncaptions = torch.randint(0, VOCAB_SIZE, (BATCH_SIZE, SEQ_LENGTH)) #(3)\n\ncaptions = show_and_tell(images, captions)\n```\n\ncsharp\n\n```\n# Codeblock 9 Output\nafter encoder : torch.Size([1, 512])\nafter decoder : torch.Size([1, 31, 10000])\n```\n\nNow, let's assume that our `show_and_tell` model is already trained on an image captioning dataset, and thus ready to be used for inference. Look at the Codeblock 10 below to see how I do it. Here we set the model to `eval()` mode (`#(1)`), initialize the input image (`#(2)`), and pass it through the model using the `generate()` method (`#(3)`).\npython\n\n```\n# Codeblock 10\nshow_and_tell.eval() #(1)\n\nimages = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE) #(2)\n\nwith torch.no_grad():\n generated_tokens = show_and_tell.generate(images) #(3)\n```\n\nThe flow of the tensor can be seen in the output below. Here I truncate the resulting outputs because it only shows the same token generation process 30 times.\ncsharp\n\n```\n# Codeblock 10 Output\nafter encoder : torch.Size([1, 512])\n\niteration #0\nafter unsqueeze : torch.Size([1, 1, 512])\nafter lstm : torch.Size([1, 1, 512])\nafter squeeze : torch.Size([1, 512])\nafter linear : torch.Size([1, 10000])\nafter max : torch.Size([1])\nafter embedding : torch.Size([1, 512])\n\niteration #1\nafter unsqueeze : torch.Size([1, 1, 512])\nafter lstm : torch.Size([1, 1, 512])\nafter squeeze : torch.Size([1, 512])\nafter linear : torch.Size([1, 10000])\nafter max : torch.Size([1])\nafter embedding : torch.Size([1, 512])\n\n.\n.\n.\n.\n```\n\nTo see what the resulting caption looks like, we can just print out the `generated_tokens` list as shown below. Keep in mind that this sequence is still in the form of tokenized words. Later, in the post-processing stage, we will need to convert them back to the words corresponding to these numbers.\npython\n\n```\n# Codeblock 11\ngenerated_tokens\n```\n\ncsharp\n\n```\n# Codeblock 11 Output\n[5627,\n 3906,\n 2370,\n 2299,\n 4952,\n 9933,\n 402,\n 7775,\n 602,\n 4414,\n 8667,\n 6774,\n 9345,\n 8750,\n 3680,\n 4458,\n 1677,\n 5998,\n 8572,\n 9556,\n 7347,\n 6780,\n 9672,\n 2596,\n 9218,\n 1880,\n 4396,\n 6168,\n 7999,\n 454]\n```\n\n* * *\n## Ending\nWith the above output, we've reached the end of our discussion on image captioning. Over time, many other researchers attempted to make improvements to accomplish this task. So, I think in the upcoming article I will discuss the state-of-the-art method on this topic.\nThanks for reading, I hope you learn something new today!\n_By the way you can also find the code used in this article [here](https://github.com/MuhammadArdiPutra/medium_articles/blob/main/Show%20and%20Tell.ipynb)._\n* * *\n## References\n[1] Oriol Vinyals et al. Show and Tell: A Neural Image Caption Generator. Arxiv. [Accessed November 13, 2024].\n[2] Christian Szegedy et al. Going Deeper with Convolutions. Arxiv. [Accessed November 13, 2024].\n[3] Marc Tanti et al. Where to put the Image in an Image Caption Generator. Arxiv. [Accessed November 13, 2024].\n[4] [Stepan Ulyanin](https://towardsdatascience.com/show-and-tell-e1a1142456e2/None). Captioning Images with CNN and RNN, using PyTorch. Medium. [Accessed November 16, 2024].\n[5] [Saketh Kotamraju](https://towardsdatascience.com/show-and-tell-e1a1142456e2/None). How to Build an Image-Captioning Model in Pytorch. Towards Data Science. [Accessed November 16, 2024].\n[6] Code with Aarohi. Image Captioning using CNN and RNN | Image Captioning using Deep Learning. YouTube. [Accessed November 16, 2024].\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657598287_taxhya.jpeg)\nWritten By\n## [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\nDeep learning storyteller\n[See all from Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/)\n[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Computer Vision](https://towardsdatascience.com/tag/computer-vision/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[NLP](https://towardsdatascience.com/tag/nlp/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F&title=Image%20Captioning%20Paper%20Walkthrough%3A%20Show%20and%20Tell)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F&title=Image%20Captioning%20Paper%20Walkthrough%3A%20Show%20and%20Tell)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fshow-and-tell-e1a1142456e2%2F&text=Image%20Captioning%20Paper%20Walkthrough%3A%20Show%20and%20Tell)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/1kXigF2ZIW5UR3t1s7cn6IA.jpeg)](https://towardsdatascience.com/image-processing-using-streamlit-d650fb0ccf8/)\n## [Image Processing using Streamlit](https://towardsdatascience.com/image-processing-using-streamlit-d650fb0ccf8/)\n[](https://towardsdatascience.com/show-and-tell-e1a1142456e2/)\nIn this article, we will see how we can use Streamlit with Image Processing Techniques and Object Detection Algorithms. I am assuming that...\n[Aniket Wattamwar](https://towardsdatascience.com/author/aniket-wattamwar/)\nNovember 1, 20205 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/1U0hBA8NnTD6l7L8Z2j-eiQ-scaled.jpeg)](https://towardsdatascience.com/nlp-video-summarization-with-watson-and-gpt-4d4db219b599/)\n## [NLP-Video Summarization with Watson and GPT](https://towardsdatascience.com/nlp-video-summarization-with-watson-and-gpt-4d4db219b599/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nSiri, Please watch my 1-hour long University lecture video and make a summary for me\n[Alex Aman](https://towardsdatascience.com/author/alexamannnn/)\nSeptember 11, 20203 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/02/1CbQ2VWJ1HV8ypZDiTvY-A.png)](https://towardsdatascience.com/a-loss-function-suitable-for-class-imbalanced-data-focal-loss-af1702d75d75/)\n## [A Loss Function Suitable for Class Imbalanced Data: \"Focal Loss\"](https://towardsdatascience.com/a-loss-function-suitable-for-class-imbalanced-data-focal-loss-af1702d75d75/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nDeep Learning with Class Imbalanced Data\n[Saptashwa Bhattacharyya](https://towardsdatascience.com/author/saptashwa/)\nFebruary 15, 20219 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/01/1aQ8yYq4PiC49TTbMS_wD7Q-scaled.jpeg)](https://towardsdatascience.com/back-propagation-the-easy-way-part-2-bea37046c897/)\n## [Back Propagation, the Easy Way (Part 2)](https://towardsdatascience.com/back-propagation-the-easy-way-part-2-bea37046c897/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nPractical implementation\n[Ziad SALLOUM](https://towardsdatascience.com/author/zsalloum/)\nJanuary 19, 20196 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/08/1DdClMPqhErordaun8Dw14Q.png)](https://towardsdatascience.com/knowledge-distillation-simplified-dd4973dbc764/)\n## [Knowledge Distillation : Simplified](https://towardsdatascience.com/knowledge-distillation-simplified-dd4973dbc764/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nTake a peek into the world of Teacher Student networks\n[Prakhar Ganesh](https://towardsdatascience.com/author/prakhargannu/)\nAugust 12, 20194 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/12/1hCiQAk9K2KL0w6MfjHxnnA.jpeg)](https://towardsdatascience.com/medical-imaging-with-azure-machine-learning-b5acfd772dd5/)\n## [Medical Imaging with Azure Machine Learning](https://towardsdatascience.com/medical-imaging-with-azure-machine-learning-b5acfd772dd5/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow Azure ML can be leveraged to support medical imaging use cases in areas like data, model management, deployment and MLOps\n[Harmke Alkemade](https://towardsdatascience.com/author/harmkealkemade/)\nDecember 20, 202113 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/03/17dBp1vkwZA58FF23OhAEoA-scaled.jpeg)](https://towardsdatascience.com/3-lessons-i-have-learned-after-i-started-working-as-a-data-scientist-893207d954b2/)\n## [3 Lessons I Have Learned After I Started Working as a Data Scientist](https://towardsdatascience.com/3-lessons-i-have-learned-after-i-started-working-as-a-data-scientist-893207d954b2/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat I would have done differently\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nMarch 15, 20214 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/11/0oK6SRDQwou4BWXkW.jpg)](https://towardsdatascience.com/graph-embeddings-explained-f0d8d1c49ec/)\n## [Graph Embeddings Explained](https://towardsdatascience.com/graph-embeddings-explained-f0d8d1c49ec/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nOverview and Python Implementation of Node, Edge and Graph Embedding Methods\n[Vatsal](https://towardsdatascience.com/author/vatsal12-p/)\nNovember 7, 20227 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/08/1YlFqBQxeG4Ml4XRvm_X3hQ.png)](https://towardsdatascience.com/calculating-audio-song-similarity-using-siamese-neural-networks-62730e8f3e3d/)\n## [Calculating Audio Song Similarity Using Siamese Neural Networks](https://towardsdatascience.com/calculating-audio-song-similarity-using-siamese-neural-networks-62730e8f3e3d/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAt AI Music, where our back catalogue of content grows every day, it is becoming increasingly necessary for us to create more intelligent...\n[Thomas Vassallo](https://towardsdatascience.com/author/tom_26639/)\nAugust 28, 20206 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "body_base_path": "attachments/642d3128a2ad/pages/", "body_markdown_path": "attachments/642d3128a2ad/pages/archived-body.md", "local_html": "attachments/642d3128a2ad/pages/archived-body.html", "body_source": "prior_capture" }