![Revisit consent button](../images/body-df29beacd7496220cc94.svg) We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg) We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data. * Cookie BCTempID * Duration 10 minutes * Description No description available. * Cookie __cf_bm * Duration 1 hour * Description This cookie, set by Cloudflare, is used to support Cloudflare Bot Management. * Cookie AWSALBCORS * Duration 7 days * Description Amazon Web Services set this cookie for load balancing. * Cookie _cfuvid * Duration session * Description Cloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services * Cookie li_gc * Duration 6 months * Description Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes. * Cookie __hssrc * Duration session * Description This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session. * Cookie __hssc * Duration 1 hour * Description HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie. * Cookie wpEmojiSettingsSupports * Duration session * Description WordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly. * Cookie BCSessionID * Duration 1 year 1 month 4 days * Description Blueconic sets this cookie as a unique identifier for the BlueConic profile. * Cookie _octo * Duration 1 year * Description No description available. * Cookie logged_in * Duration 1 year * Description No description available. * Cookie __Secure-YEC * Duration past * Description YouTube sets this cookie to stores the user's video player preferences using embedded YouTube video * Cookie __eoi * Duration 6 months * Description Description is currently not available. * Cookie AWSALBTGCORS * Duration 7 days * Description No description available. * Cookie login-status-p * Duration past * Description Description is currently not available. * Cookie AWSALBTG * Duration 7 days * Description No description available. * Cookie csrf_token * Duration session * Description No description available. * Cookie token_v2 * Duration 1 day * Description Description is currently not available. * Cookie D * Duration 1 year * Description Description is currently not available. * Cookie PHPSESSID * Duration session * Description This cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed. * Cookie VISITOR_PRIVACY_METADATA * Duration 6 months * Description YouTube sets this cookie to store the user's cookie consent state for the current domain. * Cookie cookietest * Duration session * Description The cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience. * Cookie __Host-airtable-session * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie __Host-airtable-session.sig * Duration 1 year * Description This cookie is used to enable us to integrate the services of Airtable. * Cookie m * Duration 1 year 1 month 4 days * Description Stripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly. * Cookie BIGipServer* * Duration session * Description Marketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. * Cookie __cfruid * Duration session * Description Cloudflare sets this cookie to identify trusted web traffic. * Cookie _GRECAPTCHA * Duration 6 months * Description Google Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks. * Cookie __Secure-YNID * Duration 6 months * Description Google cookie used to protect user security and prevent fraud, especially during the login process. * Cookie cookieyes-consent * Duration 1 year * Description CookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors. Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features. * Cookie lidc * Duration 1 day * Description LinkedIn sets the lidc cookie to facilitate data center selection. * Cookie brw * Duration 1 year * Description No description available. * Cookie brwConsent * Duration 5 minutes * Description Description is currently not available. * Cookie WMF-Uniq * Duration 1 year * Description Description is currently not available. * Cookie loom_anon_comment * Duration 1 year * Description No description available. * Cookie loom_referral_video * Duration session * Description Description is currently not available. * Cookie VISITOR_INFO1_LIVE * Duration 6 months * Description A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface. * Cookie yt-remote-connected-devices * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie ytidb::LAST_RESULT_ENTRY_KEY * Duration Never Expires * Description The cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future. * Cookie yt-remote-device-id * Duration Never Expires * Description YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. * Cookie yt-remote-session-name * Duration session * Description The yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video. * Cookie yt-remote-fast-check-period * Duration session * Description The yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos. * Cookie yt-remote-session-app * Duration session * Description The yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player. * Cookie yt-remote-cast-available * Duration session * Description The yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player. * Cookie yt-remote-cast-installed * Duration session * Description The yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video. * Cookie cp_session * Duration 3 months * Description Codepen sets this cookie for Help systems found in the website. * Cookie loid * Duration 1 year 1 month 4 days * Description This cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform. Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc. * Cookie __hstc * Duration 6 months * Description Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session). * Cookie hubspotutk * Duration 6 months * Description HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts. * Cookie _ga * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. * Cookie _ga_* * Duration 1 year 1 month 4 days * Description Google Analytics sets this cookie to store and count page views. * Cookie __Host-psifi.analyticsTrace * Duration 6 hours * Description Description is currently not available. * Cookie __Host-psifi.analyticsTraceV2 * Duration 6 hours * Description Description is currently not available. * Cookie _gh_sess * Duration session * Description GitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form. * Cookie YSC * Duration session * Description YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages. * Cookie ajs_anonymous_id * Duration 1 year * Description This cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before. * Cookie vuid * Duration 1 year 1 month 4 days * Description Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors. * Cookie AWSALB * Duration 7 days * Description AWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target. * Cookie acq * Duration past * Description Description is currently not available. * Cookie acq.sig * Duration past * Description Description is currently not available. * Cookie ptc * Duration 2 years * Description No description available. Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns. * Cookie muc_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to collect user behaviour and interaction data to optimize the website. * Cookie guest_id_marketing * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie guest_id_ads * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. * Cookie personalization_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting. * Cookie guest_id * Duration 1 year 1 month 4 days * Description Twitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences. * Cookie bcookie * Duration 1 year * Description LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs. * Cookie __Secure-ROLLOUT_TOKEN * Duration 6 months * Description YouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment. * Cookie yt.innertube::nextId * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie yt.innertube::requests * Duration Never Expires * Description YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. * Cookie session_tracker * Duration session * Description This cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website. * Cookie edgebucket * Duration session * Description Reddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content. * Cookie did * Duration 1 year * Description Arbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year. Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation [](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. Sign in[Submit an Article](https://write.insightmediagroup.io) [Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/) * * * [Write For TDS](https://towardsdatascience.com/submissions/) [LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) # DeiT Paper Walkthrough: Vision Transformer on a Budget Understanding how DeiT works via knowledge distillation —and how to implement it with PyTorch [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/) June 2, 202520 min read ![](../images/000.jpg) [Photo](https://unsplash.com/photos/person-holding-camera-lens-7KLa-xLbSXA) by Paul Skorupskas on Unsplash ## Introduction The vanilla ViT is problematic. If you take a look at the original ViT paper [1], you’ll notice that although this deep learning model proved to work extremely well, it requires hundreds of millions of labeled training images to achieve this. Well, that’s a lot. This requirement of an enormous amount of data is definitely a problem, and thus, we need a solution for that. Touvron _et al._ back in December 2020 brought an idea in their research paper titled “ _Training data-efficient image transformers & distillation through attention_” [2] to make training a ViT model to be computationally much cheaper. The authors came up with an idea where instead of training the transformer-based model from scratch, they exploited the knowledge of the existing model through distillation. With this approach, they managed to solve the ViT’s data-hungry problem while still maintaining high accuracy. What’s even more interesting is that this paper came out only two months after the original ViT! In this article I am going to discuss the model which the authors referred to as DeiT (Data-efficient image Transformer) as well as how to implement the architecture from scratch. Since DeiT is directly derived from ViT, it is highly recommended to have prior knowledge about ViT before reading this article. You can find my previous article about it in reference [3] at the end of this post. ··· Learn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap. ## The Idea of DeiT DeiT leverages the idea of _knowledge distillation_. In case you’re not yet familiar with the term, it is essentially a method to transfer the knowledge of a model (_teacher_) to another one (_student_) during the training phase. In this case, DeiT acts as the student whereas the teacher is RegNet, a CNN-based model. Later in the inference phase, we will completely omit the RegNet teacher and let the DeiT student make predictions on its own. The knowledge distillation technique allows the student model to learn more efficiently, which makes sense since it not only learns the patterns in the dataset from scratch but also benefits from the knowledge of the teacher during training. Think of it like someone learning a new subject. They could study purely from books, but it will be much more efficient if they also had a mentor to provide guidance. In this analogy, the learner acts as the student, the books are the dataset, while the mentor is the teacher. So, with this mechanism, the student essentially derives knowledge from both the dataset and the teacher simultaneously. As a result, training a student model requires much less amount of data. To better illustrate this, the original ViT needed 300 million images for training (JFT-300M dataset), while DeiT relies only on 1 million images (ImageNet-1K dataset). That’s 300x smaller! Technically speaking, knowledge distillation can be done without making any modifications to the student or teacher models. Rather, the changes are only made to the loss function and the training procedure. However, authors found that they can achieve more by slightly modifying the network structure, which at the same time also changing the distillation mechanism. Specifically, instead of sticking with the original ViT and apply a standard distillation process on it, they modify the architecture which they finally refer to as DeiT. It is important to know that this modification also causes the knowledge distillation mechanism to be different from the conventional one. To be exact, in ViT we only have the so-called _class token_ , but in DeiT, we will utilize the _class token_ itself and an additional one called _distillation token_. Look at the Figure 1 below to see where these two tokens are placed in the network. ![](../images/001.png) Figure 1. The DeiT architecture [2]. ### DeiT and ViT Variants There are three DeiT variants proposed in the paper, namely DeiT-Ti (Tiny), DeiT-S (Small) and DeiT-B (Base). Notice in Figure 2 that the largest DeiT variant (DeiT-B) is equivalent to the smallest ViT variant (ViT-B) in terms of the model size. So, this implicitly means that DeiT was indeed designed to challenge ViT by prioritizing efficiency. ![](../images/body-689645bbbace9508444b.png) Figure 2. DeiT and ViT variants [1, 2, 3]. Later in the coding part, I am going to implement the DeiT-B architecture. I will make the code as versatile as possible so that you can easily adjust the parameters if you want to implement the other variants instead. Taking a closer look at the DeiT-B row in the above table, we are going to configure the model such that it maps each image patch to a single-dimensional tensor of size 768. The elements in this tensor will then be grouped into 12 heads inside the attention layer. By doing so, every single of these attention heads will be responsible to process 64 features. Remember that the attention layer we are talking about is essentially a component of a Transformer encoder layer. In the case of DeiT-B, this layer is repeated 12 times before the tensor is eventually forwarded to the output layer. If we implement it correctly according to these configurations, the model should contain 86 million trainable parameters. ### Experimental Results There are plenty of experiments reported in the DeiT paper. Below is one of them that grabbed my attention the most. ![](../images/body-490c34c1215fdf730521.png) Figure 3. The performance of different models on ImageNet-1K dataset without additional training data [2]. The above figure was obtained by training multiple models on ImageNet-1K dataset, including EfficientNet, ViT, and the DeiT itself. In fact, there are two DeiT versions displayed in the figure: DeiT and DeiT⚗ — yes with that strange symbol for the latter (called “alembic”), which basically refers to the DeiT model trained using their proposed distillation mechanism. It is seen in the figure that the accuracy of ViT is already far behind DeiT with conventional distillation while still having the similar processing speed. The accuracy improved even further when the novel distillation mechanism was applied and the model was fine-tuned using the same images upscaled to 384×384 — hence the name DeiT-B⚗↑384. In theory, ViT should have performed better than its current result, yet in this experiment it could not unleash its full potential since it wasn’t allowed to be trained on the enormous JFT-300M dataset. And that’s just one result that proves the superiority of DeiT over ViT in a data-limited situation. I think that was probably all the things you need to understand to implement the DeiT architecture from scratch. Don’t worry if you haven’t fully grasped the entire idea of this model yet since we will get into the details in a minute. ··· ## DeiT Implementation As I mentioned earlier, the model we are about to implement is the DeiT-B variant. But since I also want to show you the novel knowledge distillation mechanism, I’ll specifically focus on the one referred to as DeiT-B⚗↑384. Now let’s start by importing the required modules. python ``` # Codeblock 1 import torch import torch.nn as nn from timm.models.layers import trunc_normal_ from torchinfo import summary ``` As the modules have been imported, what we need to do next is to initialize some configurable parameters in the Codeblock 2 below, which are all adjusted according to the DeiT-B specifications. At the line `#(1)`, the `IMAGE_SIZE` variable is set to 384 since we are about to simulate the DeiT version that accepts the upscaled images. Despite this higher resolution input, we still keep the patch size the same as when working with 224×224 images, i.e., 16×16, as written at line `#(2)`. Next, we set `EMBED_DIM` to 768 (`#(3)`), while the `NUM_HEADS` and `NUM_LAYERS` variables are both set to 12 (`#(4–5)`). Authors decided to use the same FFN structure as the one used in ViT, in which the size of its hidden layer is four times larger than the embedding dimension (`#(6)`). The number of patches itself can be calculated using a simple formula shown at line `#(7)`. In this case, since our image size is 384 and the patch size is 16, the value of `NUM_PATCHES` is going to be 576. Lastly, here I set `NUM_CLASSES` to 1000, simulating a classification task on ImageNet-1K dataset (`#(8)`). python ``` # Codeblock 2 BATCH_SIZE = 1 IMAGE_SIZE = 384 #(1) IN_CHANNELS = 3 PATCH_SIZE = 16 #(2) EMBED_DIM = 768 #(3) NUM_HEADS = 12 #(4) NUM_LAYERS = 12 #(5) FFN_SIZE = EMBED_DIM * 4 #(6) NUM_PATCHES = (IMAGE_SIZE//PATCH_SIZE) ** 2 #(7) NUM_CLASSES = 1000 #(8) ``` ### Treating an Image as a Sequence of Patches When it comes to processing images using transformers, what we need to do is to treat them as a sequence of patches. Such a patching mechanism is implemented in the `Patcher` class below. python ``` # Codeblock 3 class Patcher(nn.Module): def __init__(self): super().__init__() self.conv = nn.Conv2d(in_channels=IN_CHANNELS, #(1) out_channels=EMBED_DIM, kernel_size=PATCH_SIZE, #(2) stride=PATCH_SIZE) #(3) self.flatten = nn.Flatten(start_dim=2) #(4) def forward(self, x): print(f'original\t: {x.size()}') x = self.conv(x) #(5) print(f'after conv\t: {x.size()}') x = self.flatten(x) #(6) print(f'after flatten\t: {x.size()}') x = x.permute(0, 2, 1) #(7) print(f'after permute\t: {x.size()}') return x ``` You can see in Codeblock 3 that we use an `nn.Conv2d` layer to do so (`#(1)`). Keep in mind that the operation done by this layer is not intended to actually perform convolution like in CNN-based models. Instead, we use it as a trick to extract the information of each patch in a non-overlapping manner, which is the reason that we set both `kernel_size` (`#(2)`) and `stride` (`#(3)`) to `PATCH_SIZE` (16). The operation done by this convolution layer involves the patching mechanism only — we haven’t actually put these patches into sequence just yet. In order to do so, we can simply utilize an `nn.Flatten` layer which I initialize at line `#(4)` in the above codeblock. What we need to do inside the `forward()` method is to pass the input tensor through the `conv` (`#(5)`) and `flatten` (`#(6)`) layers. It is also necessary to perform the permute operation afterwards because we want the patch sequence to be placed along axis 1 and the embedding dimension along axis 2 (`#(7)`). Now let’s test the `Patcher()` class above using the following codeblock. Here I test it with a dummy tensor which the dimension is set to 1×3×384×384, simulating a single RGB image of size 384×384. python ``` # Codeblock 4 patcher = Patcher() x = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE) x = patcher(x) ``` And below is what the output looks like. Here I print out the tensor dimension after each step so that you can clearly see the flow inside the network. markup ``` # Codeblock 4 Outputoriginal : torch.Size([1, 3, 384, 384])after conv : torch.Size([1, 768, 24, 24]) #(1)after flatten : torch.Size([1, 768, 576]) #(2)after permute : torch.Size([1, 576, 768]) #(3) ``` Notice at line `#(1)` that the spatial dimension of the tensor changed from 384×384 to 24×24. This indicates that our convolution layer successfully done the patching process. By doing so, every single pixel in the 24×24 image now represents each 16×16 patch of the input image. Furthermore, notice in the same line that the number of channels increased from 3 to `EMBED_DIM` (768). Later on, we will perceive this as the number of features that stores the information of a single patch. Next, we can see at line `#(2)` that our `flatten` layer successfully flattened the 24×24 tensor into a single-dimensional tensor of length 576, which means that we already got our image represented as a sequence of patch tokens. The permute operation I mentioned earlier was essentially done because in the case of time-series data PyTorch treats the axis 1 of a tensor as a sequence (`#(3)`). ### Transformer Encoder Now let’s put our `Patcher` class aside for a while since in this section we are going to implement the transformer encoder layer. This layer is directly derived from the original ViT paper which the architecture can be seen in the Figure 4 below. Take a look at Codeblock 5 to see how I implement it. ![](../images/body-72c81860865afe3af641.png) Figure 4. The Transformer encoder layer used in ViT [1]. python ``` # Codeblock 5 class Encoder(nn.Module): def __init__(self): super().__init__() self.norm_0 = nn.LayerNorm(EMBED_DIM) #(1) self.multihead_attention = nn.MultiheadAttention(EMBED_DIM, #(2) num_heads=NUM_HEADS, batch_first=True) self.norm_1 = nn.LayerNorm(EMBED_DIM) #(3) self.ffn = nn.Sequential( #(4) nn.Linear(in_features=EMBED_DIM, out_features=FFN_SIZE), nn.GELU(), nn.Linear(in_features=FFN_SIZE, out_features=EMBED_DIM), ) def forward(self, x): residual = x print(f'residual dim\t: {residual.size()}') x = self.norm_0(x) print(f'after norm\t: {x.size()}') x = self.multihead_attention(x, x, x)[0] print(f'after attention\t: {x.size()}') x = x + residual print(f'after addition\t: {x.size()}') residual = x print(f'residual dim\t: {residual.size()}') x = self.norm_1(x) print(f'after norm\t: {x.size()}') x = self.ffn(x) print(f'after ffn\t: {x.size()}') x = x + residual print(f'after addition\t: {x.size()}') return x ``` According to the above figure, there are four layers need to be initialized in the `__init__()` method, namely a multihead attention layer (`#(2)`), an MLP layer — which is equivalent to FFN in Figure 1 (`#(4)`), and two layer normalization layers (`#(1,3)`). I am not going to get deeper into the above code since it is exactly the same as what I explained in my previous article about ViT [4]. So, I do recommend you check that article to better understand how the `Encoder` class works. And additionally, if you need an in-depth explanation specifically about the attention mechanism, you can also read my previous transformer article [5] where I implemented the entire transformer architecture from scratch. We can now just go ahead to the testing code to see how the tensor flows through the network. In the following codeblock, I assume that the input tensor `x` is an image that has already been processed by the `Patcher` block we created earlier, which is the reason why I set it to have the size of 1×576×768. python ``` # Codeblock 6 encoder = Encoder() x = torch.randn(BATCH_SIZE, NUM_PATCHES, EMBED_DIM) x = encoder(x) ``` markup ``` # Codeblock 6 Outputresidual dim : torch.Size([1, 576, 768])after norm : torch.Size([1, 576, 768])after attention : torch.Size([1, 576, 768])after addition : torch.Size([1, 576, 768])residual dim : torch.Size([1, 576, 768])after norm : torch.Size([1, 576, 768])after ffn : torch.Size([1, 576, 768])after addition : torch.Size([1, 576, 768]) ``` According to the above result, we can see that the final output tensor dimension is exactly the same as that of the input. This property allows us to stack multiple encoder blocks without disrupting the entire network structure. Furthermore, although the shape of the tensor appears to be constant along its way to the last layer, there are actually lots of dimensionality changes happening especially inside the attention and the FFN layers. However, these changes are not printed since the processes are done internally by `nn.MultiheadAttention` and `nn.Sequential`, respectively. ### The Entire DeiT Architecture All the codes I explained in the previous sections are actually identical to those used for constructing the ViT architecture. In this section, you will finally find the ones that clearly differentiate DeiT from ViT. Let’s now focus on the layers we need to initialize in the `__init__()` method of the `DeiT` class below. python ``` # Codeblock 7a class DeiT(nn.Module): def __init__(self): super().__init__() self.patcher = Patcher() #(1) self.class_token = nn.Parameter(torch.zeros(BATCH_SIZE, 1, EMBED_DIM)) #(2) self.dist_token = nn.Parameter(torch.zeros(BATCH_SIZE, 1, EMBED_DIM)) #(3) trunc_normal_(self.class_token, std=.02) #(4) trunc_normal_(self.dist_token, std=.02) #(5) self.pos_embedding = nn.Parameter(torch.zeros(BATCH_SIZE, NUM_PATCHES+2, EMBED_DIM)) #(6) trunc_normal_(self.pos_embedding, std=.02) #(7) self.encoders = nn.ModuleList([Encoder() for _ in range(NUM_LAYERS)]) #(8) self.norm_out = nn.LayerNorm(EMBED_DIM) #(9) self.class_head = nn.Linear(in_features=EMBED_DIM, out_features=NUM_CLASSES) #(10) self.dist_head = nn.Linear(in_features=EMBED_DIM, out_features=NUM_CLASSES) #(11) ``` The first component I initialized here is `Patcher` we created earlier (`#(1)`). Next, instead of only using class token, DeiT utilizes another one named distillation token. These two tokens, which in the above code are referred to as `class_token` (`#(2)`) and `dist_token` (`#(3)`), will later be appended to the patch token sequence. We set these two additional tokens to be trainable, allowing them to interact with and learn from the patch tokens later during the processing in the attention layer. Notice that I initialized these two trainable tensors using `trunc_normal_()` with a standard deviation of 0.02 (`#(4–5)`). In case you’re not yet familiar with the function, it essentially generates a truncated normal distribution, which ensures that no value lies beyond two standard deviations from the mean, avoiding the presence of extreme values for weight initialization. This approach is actually better than directly using `torch.randn()` since this function does not have such a value truncation mechanism. Afterwards, we create a learnable positional embedding tensor using the same technique which I do at lines `#(6)` and `#(7)`. It is important to keep in mind that this tensor will then be element-wise summed with the sequence of patch tokens that has been appended with the class and distillation tokens. Due to this reason, we need to set the length of axis 1 of this embedding tensor to `NUM_PATCHES+2`. Meanwhile, the transformer encoder layer is initialized inside `nn.ModuleList` which allows us to repeat the layer `NUM_LAYERS` (12) times (`#(8)`). The output produced by the last encoder layer in the stack will be processed with a layer norm (`#(9)`) before eventually being forwarded to the classification (`#(10)`) and distillation heads (`#(11)`). Now let’s move on to the `forward()` method which you can see in the Codeblock 7b below. python ``` # Codeblock 7b def forward(self, x): print(f'original\t\t: {x.size()}') x = self.patcher(x) #(1) print(f'after patcher\t\t: {x.size()}') x = torch.cat([self.class_token, self.dist_token, x], dim=1) #(2) print(f'after concat\t\t: {x.size()}') x = x + self.pos_embedding #(3) print(f'after pos embed\t\t: {x.size()}') for i, encoder in enumerate(self.encoders): x = encoder(x) #(4) print(f"after encoder #{i}\t: {x.size()}") x = self.norm_out(x) #(5) print(f'after norm\t\t: {x.size()}') class_out = x[:, 0] #(6) print(f'class_out\t\t: {class_out.size()}') dist_out = x[:, 1] #(7) print(f'dist_out\t\t: {dist_out.size()}') class_out = self.class_head(class_out) #(8) print(f'after class_head\t: {class_out.size()}') dist_out = self.dist_head(dist_out) #(9) print(f'after dist_head\t\t: {class_out.size()}') return class_out, dist_out ``` After taking raw image as the input, this `forward()` method will process the image using the `patcher` layer (`#(1)`). As we have previously discussed, this layer is responsible to convert the image into a sequence of patches. Subsequently, we will concatenate the class and distillation tokens to it using `torch.cat()` (`#(2)`). It might be worth noting that even though the illustration in Figure 1 places the class token in the beginning of the sequence and the distillation token at the end, but the code in the official GitHub repository [6] says that the distillation token is placed right after the class token. Thus, I decided to follow this approach in our implementation. Figure 5 below illustrates what the resulting tensor looks like. ![](../images/body-8a200baea80a4f471bd6.png) Figure 5. How class and distillation tokens are concatenated to the patch token sequence in our implementation [3]. Still with Codeblock 7b, what we need to do next is to inject the positional embedding tensor to the token sequence which the process is done at line (`#(3)`). We then pass the tensor through the stack of encoders using a simple loop (`#(4)`) and normalize the output produced by the last encoder layer (`#(5)`). At lines `#(6)` and `#(7)` we extract the information from the class and distillation tokens we appended earlier using a standard array slicing method. These two tokens should now contain meaningful information for classification task since they already learned the context of the image through the self-attention layers. The resulting `class_out` and `dist_out` tensors are then forwarded to two identical output layers and will undergo processing independently (`#(8–9)`). Since this model is intended for classification, these two output layers will produce tensors containing logits, in which every single element represents the raw prediction score of a class. We can see the flow of the DeiT model with the following testing code, where we initially start with the raw input image (`#(1)`), turning it into sequence of patches (`#(2)`), concatenating class and distillation tokens (`#(3)`), and so on until eventually getting the output from both classification and distillation heads (`#(4–5)`). python ``` # Codeblock 8 deit = DeiT() x = torch.randn(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE) class_out, dist_out = deit(x) ``` markup ``` # Codeblock 8 Outputoriginal : torch.Size([1, 3, 384, 384]) #(1)after patcher : torch.Size([1, 576, 768]) #(2)after concat : torch.Size([1, 578, 768]) #(3)after pos embed : torch.Size([1, 578, 768])after encoder #0 : torch.Size([1, 578, 768])after encoder #1 : torch.Size([1, 578, 768])after encoder #2 : torch.Size([1, 578, 768])after encoder #3 : torch.Size([1, 578, 768])after encoder #4 : torch.Size([1, 578, 768])after encoder #5 : torch.Size([1, 578, 768])after encoder #6 : torch.Size([1, 578, 768])after encoder #7 : torch.Size([1, 578, 768])after encoder #8 : torch.Size([1, 578, 768])after encoder #9 : torch.Size([1, 578, 768])after encoder #10 : torch.Size([1, 578, 768])after encoder #11 : torch.Size([1, 578, 768])after norm : torch.Size([1, 578, 768])class_out : torch.Size([1, 768])dist_out : torch.Size([1, 768])after class_head : torch.Size([1, 1000]) #(4)after dist_head : torch.Size([1, 1000]) #(5) ``` You can also run the following code if you want to see even more details of the architecture. It is seen in the resulting output that this network contains 87 million number of parameters, which is slightly higher than reported in the paper (86 million). I do acknowledge that the code I wrote above is indeed much simpler than the one in the documentation, so I might probably miss something that leads to such a difference in the number of params — please let me know if you spot any mistakes in my code! python ``` # Codeblock 9 summary(deit, input_size=(BATCH_SIZE, IN_CHANNELS, IMAGE_SIZE, IMAGE_SIZE)) ``` markup ``` # Codeblock 9 Output==========================================================================================Layer (type:depth-idx) Output Shape Param #==========================================================================================DeiT [1, 1000] 445,440├─Patcher: 1-1 [1, 576, 768] --│ └─Conv2d: 2-1 [1, 768, 24, 24] 590,592│ └─Flatten: 2-2 [1, 768, 576] --├─ModuleList: 1-2 -- --│ └─Encoder: 2-3 [1, 578, 768] --│ │ └─LayerNorm: 3-1 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-2 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-3 [1, 578, 768] 1,536│ │ └─Sequential: 3-4 [1, 578, 768] 4,722,432│ └─Encoder: 2-4 [1, 578, 768] --│ │ └─LayerNorm: 3-5 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-6 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-7 [1, 578, 768] 1,536│ │ └─Sequential: 3-8 [1, 578, 768] 4,722,432│ └─Encoder: 2-5 [1, 578, 768] --│ │ └─LayerNorm: 3-9 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-10 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-11 [1, 578, 768] 1,536│ │ └─Sequential: 3-12 [1, 578, 768] 4,722,432│ └─Encoder: 2-6 [1, 578, 768] --│ │ └─LayerNorm: 3-13 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-14 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-15 [1, 578, 768] 1,536│ │ └─Sequential: 3-16 [1, 578, 768] 4,722,432│ └─Encoder: 2-7 [1, 578, 768] --│ │ └─LayerNorm: 3-17 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-18 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-19 [1, 578, 768] 1,536│ │ └─Sequential: 3-20 [1, 578, 768] 4,722,432│ └─Encoder: 2-8 [1, 578, 768] --│ │ └─LayerNorm: 3-21 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-22 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-23 [1, 578, 768] 1,536│ │ └─Sequential: 3-24 [1, 578, 768] 4,722,432│ └─Encoder: 2-9 [1, 578, 768] --│ │ └─LayerNorm: 3-25 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-26 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-27 [1, 578, 768] 1,536│ │ └─Sequential: 3-28 [1, 578, 768] 4,722,432│ └─Encoder: 2-10 [1, 578, 768] --│ │ └─LayerNorm: 3-29 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-30 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-31 [1, 578, 768] 1,536│ │ └─Sequential: 3-32 [1, 578, 768] 4,722,432│ └─Encoder: 2-11 [1, 578, 768] --│ │ └─LayerNorm: 3-33 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-34 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-35 [1, 578, 768] 1,536│ │ └─Sequential: 3-36 [1, 578, 768] 4,722,432│ └─Encoder: 2-12 [1, 578, 768] --│ │ └─LayerNorm: 3-37 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-38 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-39 [1, 578, 768] 1,536│ │ └─Sequential: 3-40 [1, 578, 768] 4,722,432│ └─Encoder: 2-13 [1, 578, 768] --│ │ └─LayerNorm: 3-41 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-42 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-43 [1, 578, 768] 1,536│ │ └─Sequential: 3-44 [1, 578, 768] 4,722,432│ └─Encoder: 2-14 [1, 578, 768] --│ │ └─LayerNorm: 3-45 [1, 578, 768] 1,536│ │ └─MultiheadAttention: 3-46 [1, 578, 768] 2,362,368│ │ └─LayerNorm: 3-47 [1, 578, 768] 1,536│ │ └─Sequential: 3-48 [1, 578, 768] 4,722,432├─LayerNorm: 1-3 [1, 578, 768] 1,536├─Linear: 1-4 [1, 1000] 769,000├─Linear: 1-5 [1, 1000] 769,000==========================================================================================Total params: 87,630,032Trainable params: 87,630,032Non-trainable params: 0Total mult-adds (Units.MEGABYTES): 398.43==========================================================================================Input size (MB): 1.77Forward/backward pass size (MB): 305.41Params size (MB): 235.34Estimated Total Size (MB): 542.52========================================================================================== ``` ### How Classification and Distillation Heads Work I would like to talk a little bit about the tensors produced by the two output heads. During the training phase, the output from the classification head is compared with the original ground truth (one-hot label) which the classification performance is evaluated using cross entropy loss. Meanwhile, the output from the distillation head is compared with the output produced by the teacher model, i.e., RegNet. We always perceive the output of the teacher as a truth regardless of whether its prediction is correct. And that’s essentially how knowledge is distilled from RegNet to DeiT. There are actually two methods possible to be used to perform knowledge distillation: _soft distillation_ and _hard distillation_. The former is a technique where we use the logits produced by the teacher model as is (rather than the argmaxed logits) for the label. This kind of additional ground truth is referred to as _soft label_. If we decided to use this technique, we should use the so-called Kullback-Leibler (KL) loss, which is suitable for comparing two logits: one from the distillation head and another one from the teacher output. On the other hand, _hard distillation_ is a technique where the prediction made by the teacher is argmaxed prior to being compared with the output from the distillation head. In this case, the teacher output is referred to as _hard label_ , which is similar to a typical one-hot-encoded label. Thanks to this reason, if we were to use hard label instead, we can simply use the standard cross-entropy loss for this head. Although the authors found that hard distillation performed better than soft distillation, I still think that it is worth experimenting with the two approaches if you plan to use DeiT for your upcoming project to see if this notion also applies to your case. During the inference phase, we will no longer use the teacher model. Think of it like the student has graduated and is ready to work on its own. Despite the absence of the teacher, the output from the distillation head is still utilized. According to their GitHub documentation [6], the logits produced by both the classification and distillation heads are combined using a standard averaging mechanism before being argmaxed to obtain the final prediction. ··· ## Ending I think that’s everything about the main idea and implementation of DeiT. It is important to note that there are still lots of things I haven’t covered in this article. So, I do recommend you read the paper [2] if you want to get even deeper into the details of this deep learning model. Thanks for reading, I hope you learn something new today! _By the way you can access the code used in this article in the link at reference number [7]._ ··· ### References [1] Alexey Dosovitskiy _et al_. An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. Arxiv. [Accessed February 17, 2025]. [2] Hugo Touvron _et al_. Training Data-Efficient Image Transformers & Distillation Through Attention. Arxiv. [Accessed February 17, 2025]. [3] Image originally created by author. [4] Muhammad Ardi. Paper Walkthrough: Vision Transformer (ViT). Towards Data Science. [Accessed February 17, 2025]. [5] Muhammad Ardi. Paper Walkthrough: Attention Is All You Need. Towards Data Science. [Accessed February 17, 2025]. [6] facebookresearch. GitHub. [Accessed February 17, 2025]. [7] MuhammadArdiPutra. Vision Transformer on a Budget. GitHub. [Accessed February 17, 2025]. ## Register Now to Continue Reading ### Unlock Towards Data Science with a Free Account All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account * * * ![](../images/body-59e1c17daab1fcb582c5.jpeg) Written By ## [Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/) Deep learning storyteller [See all from Muhammad Ardi Putra](https://towardsdatascience.com/author/muhammad_ardi/) [Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Computer Vision](https://towardsdatascience.com/tag/computer-vision/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Neural Network](https://towardsdatascience.com/tag/neural-network/)[Vision Transformer](https://towardsdatascience.com/tag/vision-transformer/) Share This Article * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fvision-transformer-on-a-budget%2F&title=DeiT%20Paper%20Walkthrough%3A%20Vision%20Transformer%20on%20a%20Budget) * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fvision-transformer-on-a-budget%2F&title=DeiT%20Paper%20Walkthrough%3A%20Vision%20Transformer%20on%20a%20Budget) * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fvision-transformer-on-a-budget%2F&text=DeiT%20Paper%20Walkthrough%3A%20Vision%20Transformer%20on%20a%20Budget) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/submissions/) ## Related Articles * [![](../images/002.jpeg)](https://towardsdatascience.com/hybrid-rule-based-machine-learning-with-scikit-learn-9cb9841bebf2/) ## [Hybrid Rule-Based Machine Learning With scikit-learn](https://towardsdatascience.com/hybrid-rule-based-machine-learning-with-scikit-learn-9cb9841bebf2/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Use Domain Knowledge to Augment Your scikit-learn Models With Hard-Coded Rules [Lukas Haas](https://towardsdatascience.com/author/lukhaas/) July 29, 202011 min read * [![](../images/003.jpeg)](https://towardsdatascience.com/concept-learning-making-your-network-interpretable-735a0698fc03/) ## [Concept Learning: Making your network interpretable](https://towardsdatascience.com/concept-learning-making-your-network-interpretable-735a0698fc03/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) Over the last decade, neural networks have been showing superb performance across a large variety of datasets and problems. While metrics... [Lukas Huber](https://towardsdatascience.com/author/lukas.huber1/) August 2, 20225 min read * [![](../images/004.jpg)](https://towardsdatascience.com/land-use-land-cover-classification-with-deep-learning-9a5041095ddb/) ## [Land use/Land cover classification with Deep Learning](https://towardsdatascience.com/land-use-land-cover-classification-with-deep-learning-9a5041095ddb/) [Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/) Identifying the physical aspect of the earth's surface (Land cover) as well as how we exploit the land (Land use) is a challenging problem... [Abdishakur](https://towardsdatascience.com/author/shakasom/) August 24, 20184 min read * [![](../images/005.jpg)](https://towardsdatascience.com/quantization-linear-regression-and-hardware-for-ai-our-best-recent-deep-dives-a6baf108db0c/) ## [Quantization, Linear Regression, and Hardware for AI: Our Best Recent Deep Dives](https://towardsdatascience.com/quantization-linear-regression-and-hardware-for-ai-our-best-recent-deep-dives-a6baf108db0c/) [Data Science](https://towardsdatascience.com/category/data-science/) Our weekly selection of must-read Editors' Picks and original features [TDS Editors](https://towardsdatascience.com/author/towardsdatascience/) April 18, 20243 min read * [![](../images/006.jpeg)](https://towardsdatascience.com/engineers-should-learn-data-science-differently-99c7bf5caa6b/) ## [Engineers should learn data science differently.](https://towardsdatascience.com/engineers-should-learn-data-science-differently-99c7bf5caa6b/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) An HOV lane for engineers and engineering students to learn data science. [Naser Tamimi](https://towardsdatascience.com/author/tamimi-naser/) December 26, 20205 min read * [![](../images/007.jpg)](https://towardsdatascience.com/using-machine-learning-to-detect-fraud-f204910389cf/) ## [Using Machine Learning To Detect Fraud](https://towardsdatascience.com/using-machine-learning-to-detect-fraud-f204910389cf/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) The beginning of an End-to-End Machine Learning Project [Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/) September 21, 20207 min read * [![](../images/008.png)](https://towardsdatascience.com/water-cooler-small-talk-ep-10-so-what-about-the-ai-bubble/) ## [Water Cooler Small Talk, Ep. 10: So, What About the AI Bubble?](https://towardsdatascience.com/water-cooler-small-talk-ep-10-so-what-about-the-ai-bubble/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Have we all been tricked into believing in an impossible, extremely expensive future? [Maria Mouschoutzi](https://towardsdatascience.com/author/m.mouschoutzi/) November 27, 20258 min read * [![](../images/009.jpeg)](https://towardsdatascience.com/dask-dataframes-how-to-run-pandas-in-parallel-with-ease-b8b1f6b2646b/) ## [Dask DataFrames - How to Run Pandas in Parallel With Ease](https://towardsdatascience.com/dask-dataframes-how-to-run-pandas-in-parallel-with-ease-b8b1f6b2646b/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) Learn how to fit 20GB of CSV files into 16GB of RAM. So you have... [Dario Radečić](https://towardsdatascience.com/author/radecicdario/) May 28, 20216 min read * [![](../images/010.png)](https://towardsdatascience.com/self-attention-and-transformers-882e9de5edda/) ## [Self Attention and Transformers](https://towardsdatascience.com/self-attention-and-transformers-882e9de5edda/) [Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/) From Attention to Self Attention to Transformers [Mahendran Venkatachalam](https://towardsdatascience.com/author/mahendran.venkatachalam/) July 11, 20197 min read * [YouTube](https://www.youtube.com/c/TowardsDataScience) * [X](https://x.com/TDataScience) * [LinkedIn](https://www.linkedin.com/company/towards-data-science/) * [Threads](https://www.threads.net/@towardsdatascience) * [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/) © Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network) #### Recommended Articles Close * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ "How Relevance Models Foreshadowed Transformers for NLP") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ "Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ "The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs") * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ "Does Your Company Have a Data Strategy?")