{ "source_file": "0930", "source_platform": "Medium", "author": "Cristian Leo", "title_listed": "The Math Behind Multi-Head Attention in Transformers", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/the-math-behind-multi-head-attention-in-transformers-c26cba15f625/", "publish_date": "2024-07-16", "corpus_stage": "", "notes": "", "id": "767188f87239", "collected_at": "2026-10-02T06:05:18", "title": "The Math Behind Multi-Head Attention in Transformers | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# The Math Behind Multi-Head Attention in Transformers\nDeep Dive into Multi-Head Attention, the secret element in Transformers and LLMs. Let's explore its math, and build it from scratch.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nJuly 16, 202416 min read\n### Deep Dive into Multi-Head Attention, the secret element in Transformers and LLMs. Let's explore its math, and build it from scratch in Python\n![Image generated by DALL-E](../images/000.jpg)\nImage generated by DALL-E\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer), [Computer Science](https://roadmap.sh/computer-science) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n## 1: Introduction\n### 1.1: Transformers Overview\nThe Transformer architecture, introduced by Vaswani et al. in their paper \"[Attention is All You Need](https://arxiv.org/abs/1706.03762)\" has transformed deep learning, particularly in natural language processing (NLP). Transformers use a self-attention mechanism, enabling them to handle input sequences all at once. This parallel processing allows for faster computation and better management of long-range dependencies within the data. This doesn't sound familiar? Don't worry as it will be at the end of this article. Let's first take a brief look at what a Transformer looks like.\n![Transformer Architecture \\(Architecture in "Attention is all you need"\\) - Image by Author](../images/001.jpeg)\nTransformer Architecture (Architecture in \"[Attention is all you need](https://arxiv.org/abs/1706.03762)\") - Image by Author\nA Transformer consists of two main parts: an **encoder** and a **decoder**. The encoder processes the input sequence to create a continuous representation, while the decoder generates the output sequence from this representation. Both the encoder and the decoder have multiple layers, each containing two essential components: a multi-head self-attention mechanism and a position-wise feed-forward network. In this article, we'll focus on the multi-head attention mechanism, but we'll explore the entire Transformer architecture in future articles.\n### 1.2: Multi-Head Attention Overview\nMulti-head attention enables the model to focus on different parts of the input sequence at the same time, capturing various aspects of the data. Think of it like having multiple spotlights shining on different parts of a stage. Each spotlight (or \"head\") can illuminate a different performer (or data feature), allowing the audience (or model) to see the whole scene more clearly. By splitting the input into multiple subspaces, each with its attention mechanism, multi-head attention provides the model with several views of the input data. This setup helps the model understand complex relationships within the data more effectively.\nThis mechanism allows the _Transformer_ to capture different relationships in the data by attending to various parts of the sequence. This improves the learning process by offering multiple perspectives of the input, enhancing the model's ability to generalize. It also increases the model's expressiveness by enabling it to learn different aspects of the input data simultaneously.\nThese capabilities make multi-head attention a crucial component in the success of Transformer models across a range of applications, from language translation to image processing.\n## 2: The Mathematical Foundations\n![Multi-Head Attention Architecture - Image by Author](../images/002.jpeg)\nMulti-Head Attention Architecture - Image by Author\n### **2.1: Attention Mechanism**\nThe attention mechanism in neural networks is designed to mimic the human ability to focus on specific parts of information while processing data. Imagine reading a book: your eyes don't pay equal attention to every word on the page. Instead, they focus more on the important words that help you understand the story. Similarly, in neural networks, attention allows the model to dynamically weigh the importance of different input elements. This means the model can prioritize parts of the input sequence that are more relevant for generating the output, improving its performance in tasks like language translation, text summarization, and more.\nMathematically, the attention mechanism can be described using a set of queries, keys, and values. Let's denote the input as a set of queries **Q** , keys **K** , and values **V**. These are typically linear transformations of the input data.\nThe attention scores are calculated by taking the dot product of the query with all keys, which gives a measure of similarity. For a query **q** and a set of keys **k1, k2, ..., kn** ​, the attention scores are given by:\n![Attention Score - Image by Author](../images/003.png)\nAttention Score - Image by Author\nThink of this as comparing how similar each word (key) in a sentence is to the word (query) you're focusing on. Higher scores indicate greater similarity.\nTo prevent the dot products from becoming too large, especially when dealing with high-dimensional vectors, we scale the scores by the square root of the dimension of the keys, **d_k** :\n![Scaled Score Formula - Image by Author](../images/004.png)\nScaled Score Formula - Image by Author\nThis is like adjusting the intensity of the spotlight based on how large the stage is. It ensures that the scores remain manageable and helps maintain stable gradients during training. This scaling ensures that the values passed to the softmax function have a standard deviation close to 1, which helps maintain stable gradients during training.\nTo see why this is necessary, consider the properties of dot products and high-dimensional vectors. When we compute the dot product of two vectors **q** and **k_i** ​ of dimension **d_k** ​, the expected value of their dot product is proportional to **d_k​**. Without scaling, as **d_k** ​ increases, the variance of the dot product grows, leading to very large values which can cause the softmax function to produce near-binary outputs (i.e., probabilities close to 0 or 1). This sharpness reduces the model's ability to learn effectively because it makes the gradients very small.\nBy dividing the dot product by **d_k** ​​, we normalize the input to the softmax function, ensuring that the scores remain within a reasonable range. This normalization helps the model to maintain a balance, enabling it to learn more effectively and stably.\nThese scaled scores are then passed through a softmax function to obtain the attention weights. The softmax function converts the scores into probabilities, which represent the importance of each key relative to the query:\n![Attention Weight using Softmax Function - Image by Author](../images/005.png)\nAttention Weight using Softmax Function - Image by Author\nThis step is like converting the adjusted spotlight intensities into a clear ranking, highlighting the most relevant parts of the scene more brightly.\nThe final attention output is obtained by taking a weighted sum of the values, using the attention weights:\n![Attention Output Formula - Image by Author](../images/006.png)\nAttention Output Formula - Image by Author\nHere, **v_i** ​ represents the value corresponding to key **k_i**. This weighted sum combines the most relevant information from the values, much like focusing your attention on the most important parts of the book to understand the story better.\n### **2.2: Multi-Head Attention**\nMulti-head attention is an advanced form of the attention mechanism that allows a model to focus on different parts of the input sequence simultaneously, capturing various relationships within the data. Instead of having a single attention mechanism, multi-head attention processes the input into multiple \"heads,\" each with its own set of queries, keys, and values. Each head performs the attention operation independently, and their outputs are then combined. This enhances the model's ability to understand complex patterns and dependencies in the data.\nImagine you're trying to understand a complex scene with many elements. If you had multiple pairs of eyes, each looking at different parts of the scene, you'd get a more comprehensive understanding. Similarly, multi-head attention allows the model to focus on different parts of the input data at once, providing a richer and more detailed representation.\nGiven an input sequence **X** , we project it into queries **Q** , keys **K** , and values **V** using learned linear transformations. For each head **i** , we have separate weight matrices **W_Q** ​, **W_K** ​, and **W_V** ​:\n![Queries, Keys, and Values Linear Transformations - Image by Author](../images/007.png)\nQueries, Keys, and Values Linear Transformations - Image by Author\nThese projections allow each head to focus on different aspects of the input data. For each head **i** , we compute the attention scores using the scaled dot-product attention mechanism. The attention output for head **i** is:\n![Attention Formula - Image by Author](../images/008.png)\nAttention Formula - Image by Author\nHere, **d_k** ​ is the dimension of the key vectors, ensuring the scores are appropriately scaled.\nAfter computing the attention outputs for all heads, we concatenate them along the feature dimension. If we have **h** heads, each producing an output of dimension **d_v** ​, the concatenated output will have a dimension of **h** ×**d_v** ​:\n![Multi-Head Concatenation - Image by Author](../images/009.png)\nMulti-Head Concatenation - Image by Author\nThe concatenated output is then projected back to the original input dimension **d** ​ using a learned weight matrix **W_O** :\n![Final Layer of Multi-Head Attention - Image by Author](../images/010.png)\nFinal Layer of Multi-Head Attention - Image by Author\nThis final linear transformation combines the outputs from all heads into a single representation.\nThe core idea behind combining multiple attention heads is to allow the model to capture different types of information from the input sequence simultaneously. By having multiple heads, each head can learn to attend to different parts of the input or different features. This diversity in attention leads to a richer and more nuanced representation of the data.\n### **2.3: Position-wise Feed-Forward Networks**\nIn the Transformer architecture, each layer consists of a multi-head attention mechanism followed by a position-wise feed-forward network. These feed-forward layers are applied independently to each position in the sequence, hence the term \"position-wise.\" Essentially, they are simple fully connected neural networks applied separately and identically to each position of the input sequence.\nImagine a factory where every product on the conveyor belt goes through the same set of machines. Each machine processes the product in a specific way, adding something new or refining it. Similarly, each position in the sequence is processed independently by the feed-forward layers, transforming and enhancing the representation.\nThe purpose of these feed-forward layers is to introduce non-linearity and additional learning capacity to the model. After the attention mechanism has aggregated information from different parts of the sequence, the feed-forward network processes this information to further transform and refine the representation.\nMathematically, a position-wise feed-forward network consists of two linear transformations with a ReLU activation function in between. Given an input **x** at a particular position, the feed-forward network can be represented as:\n![Feed-Forward Network Formula - Image by Author](../images/011.png)\nFeed-Forward Network Formula - Image by Author\nHere:\n * **W1** ​ and **W2** ​ are learned weight matrices.\n * **b1** ​ and **b2** ​ are learned bias vectors.\n * **max(0, xW1​+b1​)** represents the ReLU activation function applied element-wise.\n\n\nThe input **x** is first linearly transformed using the weight matrix **W1** ​ and bias **b1** ​:\n![Linear Transformation of the input - Image by Author](../images/012.png)\nLinear Transformation of the input - Image by Author\nThink of this step as passing the input through the first machine in the factory, which adds initial modifications based on learned weights and biases.\nThe linear transformation is followed by a ReLU activation function, which introduces non-linearity:\n![ReLU Formula - Image by Author](../images/013.png)\nReLU Formula - Image by Author\nReLU (Rectified Linear Unit) sets all negative values to zero, allowing the model to capture non-linear relationships in the data. This step is like ensuring only positive contributions from the first machine are passed on.\nThe activated output is then passed through a second linear transformation using weight matrix **W2** ​ and bias **b2** ​:\n![Second FFN - Image by Author](../images/body-52896ff70587b76501e3.png)\nSecond FFN - Image by Author\nThis final step further refines the output, much like the second machine in the factory making additional modifications to produce a finished product.\nThe position-wise feed-forward network in the Transformer architecture further processes the information captured by the multi-head attention mechanism. While the attention mechanism allows the model to focus on different parts of the sequence and aggregate context-specific information, the feed-forward network refines and transforms this information at each position. This enhances the model's ability to capture complex patterns and dependencies.\n## 3: Building Multi-Head Attention from Scratch\nIn this section, we will break down and explain the implementation of a multi-head attention mechanism from scratch using Python and `numpy`. The goal is to understand how the input is modified during the process. Before proceed with your reading, take a look at the code we will cover in this section. You should be able to get a general understanding, but don't worry if now, as we will go over each line in this section.\n> [**models-from-scratch-python/Multi-Head Attention/demo.py at main ·...**](https://github.com/cristianleoo/models-from-scratch-python/blob/main/Multi-Head%20Attention/demo.py)\nTo begin, we define the `MultiHeadAttention` class, which is responsible for managing the parameters needed for the multi-head attention mechanism. Let's go through this step-by-step to understand how we set it up.\npython\n\n```\nimport numpy\n\nclass MultiHeadAttention:\n def __init__(self, num_hiddens, num_heads, dropout=0.0, bias=False):\n self.num_heads = num_heads\n self.num_hiddens = num_hiddens\n self.d_k = self.d_v = num_hiddens // num_heads\n```\n\nIn the initialization method, we first set the number of attention heads and the total number of hidden units in the model. These values are provided as arguments when the class is instantiated.\n * `num_hiddens`: This represents the total number of hidden units in the model. It's a crucial parameter because it determines the size of the linear transformations applied to the input data.\n * `num_heads`: This indicates the number of attention heads. Each head will independently learn to focus on different parts of the input, enabling the model to capture various aspects of the data.\n * `dropout`: This is the dropout rate, which is not used in this particular implementation but is included for completeness.\n * `bias`: This is a boolean flag indicating whether to include biased terms in the linear transformations.\n\n\nWe then calculate the dimensions of the queries and values for each head. Since the total number of hidden units (`num_hiddens`) is split across all heads (`num_heads`), each head will have a query and value dimension of `num_hiddens // num_heads`.\npython\n\n```\n self.W_q = np.random.rand(num_hiddens, num_hiddens)\n self.W_k = np.random.rand(num_hiddens, num_hiddens)\n self.W_v = np.random.rand(num_hiddens, num_hiddens)\n self.W_o = np.random.rand(num_hiddens, num_hiddens)\n```\n\nNext, we initialize the weight matrices for the queries, keys, values, and output transformations. These weight matrices are randomly initialized:\n * `W_q` is used to transform the input data into queries. It has dimensions `num_hiddens x num_hiddens`, meaning it maps the input features to the query space.\n * `W_k` is used to transform the input data into keys. It also has dimensions `num_hiddens x num_hiddens`, mapping the input features to the key space.\n * `W_v` is used to transform the input data into values, with the same dimensions as the previous matrices.\n * `W_o` is used to transform the concatenated output of all heads back to the original input dimensions.\n\n\npython\n\n```\n if bias:\n self.b_q = np.random.rand(num_hiddens)\n self.b_k = np.random.rand(num_hiddens)\n self.b_v = np.random.rand(num_hiddens)\n self.b_o = np.random.rand(num_hiddens)\n else:\n self.b_q = self.b_k = self.b_v = self.b_o = np.zeros(num_hiddens)\n```\n\nFinally, we initialize the bias vectors for the queries, keys, values, and output transformations. If the `bias` parameter is set to `True`, these biases are randomly initialized. Otherwise, they are set to zero:\n * `b_q`: Bias for the query transformation.\n * `b_k`: Bias for the key transformation.\n * `b_v`: Bias for the value transformation.\n * `b_o`: Bias for the output transformation.\n\n\nThe biases have dimensions equal to the number of hidden units, `num_hiddens`.\nBy setting up these weights and biases, we ensure that each attention head can independently learn to focus on different parts of the input data.\nNext, we define methods to prepare and transform the data for multi-head attention. First, let's look at the `transpose_qkv` method:\npython\n\n```\ndef transpose_qkv(self, X):\n X = X.reshape(X.shape[0], X.shape[1], self.num_heads, -1)\n X = X.transpose(0, 2, 1, 3)\n return X.reshape(-1, X.shape[2], X.shape[3])\n```\n\nThis method is responsible for reshaping and transposing the input data to prepare it for multi-head attention. In particular:\npython\n\n```\nX = X.reshape(X.shape[0], X.shape[1], self.num_heads, -1)\n```\n\nThis line reshapes the input tensor `X` to have four dimensions: `(batch_size, sequence_length, num_heads, depth_per_head)`.\n * `X.shape[0]` is the batch size.\n * `X.shape[1]` is the sequence length (number of positions in the input sequence).\n * `self.num_heads` is the number of attention heads.\n * `-1` automatically infers the size of the last dimension (depth per head) so that the total number of elements remains the same.\n\n\npython\n\n```\nX = X.transpose(0, 2, 1, 3)\n```\n\nThis line transposes the tensor so that the dimensions are reordered to `(batch_size, num_heads, sequence_length, depth_per_head)`.\nThis rearrangement ensures that each attention head processes its part of the input sequence independently.\npython\n\n```\nreturn X.reshape(-1, X.shape[2], X.shape[3])\n```\n\nThis final reshape flattens the batch and head dimensions into a single dimension, resulting in a tensor of shape `(batch_size * num_heads, sequence_length, depth_per_head)`.\nBy doing this, `transpose_qkv` ensures that the input data is split correctly among the multiple heads, with each head having the appropriate dimensions to process its segment of the data.\nNext, we have the `transpose_output` method:\npython\n\n```\ndef transpose_output(self, X):\n X = X.reshape(-1, self.num_heads, X.shape[1], X.shape[2])\n X = X.transpose(0, 2, 1, 3)\n return X.reshape(X.shape[0], X.shape[1], -1)\n```\n\nThis method reverses the transformation done by `transpose_qkv` to combine the outputs from all heads back into the original shape.\nAfter transposing our matrices, we can process with the scaled dot-product attention mechanism, which allows the model to focus on different parts of the input sequence with varying degrees of importance.\npython\n\n```\ndef scaled_dot_product_attention(self, Q, K, V, valid_lens):\n d_k = Q.shape[-1]\n scores = np.matmul(Q, K.transpose(0, 2, 1)) / np.sqrt(d_k)\n if valid_lens is not None:\n mask = np.arange(scores.shape[-1]) < valid_lens[:, None]\n scores = np.where(mask[:, None, :], scores, -np.inf)\n attention_weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))\n attention_weights /= attention_weights.sum(axis=-1, keepdims=True)\n return np.matmul(attention_weights, V)\n```\n\nThe inputs to this method are the query (**Q**), key (**K**), and value (**V**) matrices. These matrices are derived from the input data through linear transformations.\npython\n\n```\nd_k = Q.shape[-1]\n```\n\nHere, we extract the dimension of the key vectors, **d_k** ​, from the last dimension of the query matrix **Q**. This value is used for scaling the attention scores.\npython\n\n```\nscores = np.matmul(Q, K.transpose(0, 2, 1)) / np.sqrt(d_k)\n```\n\nWe calculate the attention scores by performing a matrix multiplication of **Q** and the transpose of **K**. The scores are then scaled by the square root of **d_k** ​. This scaling helps prevent the scores from growing too large, which can lead to issues during the softmax calculation.\nNext, we define the forward pass method to process the input data through the multi-head attention mechanism. This method is crucial as it orchestrates the entire multi-head attention process, from transforming the input data to combining the outputs from multiple heads.\npython\n\n```\ndef forward(self, queries, keys, values, valid_lens):\n queries = self.transpose_qkv(np.dot(queries, self.W_q) + self.b_q)\n keys = self.transpose_qkv(np.dot(keys, self.W_k) + self.b_k)\n values = self.transpose_qkv(np.dot(values, self.W_v) + self.b_v)\n\n if valid_lens is not None:\n valid_lens = np.repeat(valid_lens, self.num_heads, axis=0)\n\n output = self.scaled_dot_product_attention(queries, keys, values, valid_lens)\n output_concat = self.transpose_output(output)\n return np.dot(output_concat, self.W_o) + self.b_o\n```\n\nLet's break down the forward method:\npython\n\n```\nqueries = self.transpose_qkv(np.dot(queries, self.W_q) + self.b_q)\nkeys = self.transpose_qkv(np.dot(keys, self.W_k) + self.b_k)\nvalues = self.transpose_qkv(np.dot(values, self.W_v) + self.b_v)\n```\n\nFirst, the input queries, keys, and values are projected into their respective subspaces using the learned weight matrices (`W_q`, `W_k`, `W_v`) and biases (`b_q`, `b_k`, `b_v`). This is done by performing matrix multiplication with the weight matrices and adding the biases. The results are then transformed for multi-head attention using the `transpose_qkv` method, which reshapes and transposes the data to ensure each head processes the input independently.\n**Queries, keys, and values** are the transformed inputs, now prepared for multi-head attention.\npython\n\n```\nif valid_lens is not None:\n valid_lens = np.repeat(valid_lens, self.num_heads, axis=0)\n```\n\nIf `valid_lens` (valid lengths) are provided, they are repeated for each head. This ensures that the appropriate mask is created for each attention head, allowing the model to focus only on valid positions within the sequences.\npython\n\n```\noutput = self.scaled_dot_product_attention(queries, keys, values, valid_lens)\n```\n\nThe method then calls `scaled_dot_product_attention` with the transformed queries, keys, values, and repeated valid lengths. This function calculates the attention scores, applies the softmax function to obtain attention weights, and computes the weighted sum of the values to produce the attention output for each head.\npython\n\n```\noutput_concat = self.transpose_output(output)\nreturn np.dot(output_concat, self.W_o) + self.b_o\n```\n\nAfter obtaining the attention outputs from all heads, the method concatenates these outputs along the feature dimension using `transpose_output`. This method reverses the initial transformation, combining the outputs from all heads back into a single representation. The concatenated output is then transformed back to the original input dimension using a final linear transformation with weight matrix `W_o` and bias `b_o`.\nLastly, we test the class with some sample data. Here's how we do it:\ngraphql\n\n```\n# Define dimensions and initialize multi-head attention\nnum_hiddens, num_heads = 100, 5\nattention = MultiHeadAttention(num_hiddens, num_heads, dropout=0.5, bias=False)\n```\n\nWe initialize the `MultiHeadAttention` class with 100 hidden units and 5 attention heads. This sets up the necessary parameters and weight matrices for the multi-head attention mechanism.\npython\n\n```\n# Define sample data\nbatch_size, num_queries, num_kvpairs = 2, 4, 6\nvalid_lens = np.array([3, 2])\nX = np.random.rand(batch_size, num_queries, num_hiddens) # Use random data to simulate input queries\nY = np.random.rand(batch_size, num_kvpairs, num_hiddens) # Use random data to simulate key-value pairs\n```\n\nWe create random data to simulate input queries (`X`) and key-value pairs (`Y`). The batch size is 2, the number of queries is 4, and the number of key-value pairs is 6. We also define valid lengths (`valid_lens`) to indicate the valid positions within the sequences.\npython\n\n```\n# Apply multi-head attention\noutput = attention.forward(X, Y, Y, valid_lens)\n```\n\nWe pass the sample data through the multi-head attention mechanism using the `forward` method. This processes the input queries, keys, and values, applying the multi-head attention calculations.\npython\n\n```\nprint(\"Output shape:\", output.shape)\n# Output should be: (2, 4, 100)\n\nprint(\"Output data:\", output)\n```\n\nWe print the shape and content of the output. The expected output shape ensures that the output dimensions match the original input dimensions. Then, we print the output data after computing Multi-Head Attention. Now that you have an understanding of how the Multi-Head Attention mechanism works, try tweaking it. For instance, change the number of heads, try adding multiple FFN before and after Multi-Head Attention. Also, you could try to implement it in a machine translation task, and see it in action. Let me know if you would like me to do that in a next article.\n## Conclusion\nTransformers have transformed deep learning, especially in NLP, by using self-attention mechanisms that allow for parallel processing of input sequences. This approach not only speeds up computation but also handles long-range dependencies more effectively than traditional recurrent neural networks).\nIn this article, we've gained a comprehensive understanding of multi-head attention in Transformers, from it's math theory to a practical code implementation. Maybe for now the concepts we will still be abstracts, as you can't really do anything with the outputs from Multi-Head Attention, but soon we will see how they play a key role in the transformer architecture, which is the base of the well-known LLMs around (Claude, ChatGPT, ...). Stay tuned for future articles, where we'll explore the remaining components of the Transformer architecture, offering deeper insights into this powerful model.\n## References\n * Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). [Attention is All You Need](https://arxiv.org/abs/1706.03762). In Advances in Neural Information Processing Systems (NeurIPS).\n * [Alammar, J. (2018). The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/). jalammar.github.io.\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-8f73ac5a40c91a7b503f.jpeg)\nWritten By\n## [Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[See all from Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Math](https://towardsdatascience.com/tag/math/)[Programming](https://towardsdatascience.com/tag/programming/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F&title=The%20Math%20Behind%20Multi-Head%20Attention%20in%20Transformers)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F&title=The%20Math%20Behind%20Multi-Head%20Attention%20in%20Transformers)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F&text=The%20Math%20Behind%20Multi-Head%20Attention%20in%20Transformers)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/014.gif)](https://towardsdatascience.com/demystifying-artificial-intelligence-d2887879aaf1/)\n## [Demystifying Artificial Intelligence!](https://towardsdatascience.com/demystifying-artificial-intelligence-d2887879aaf1/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nBreaking down the field of Artificial Intelligence and what the future holds for it.\n[Bharath K](https://towardsdatascience.com/author/bharath-k1297/)\nNovember 2, 20208 min read\n * [![](../images/015.jpg)](https://towardsdatascience.com/demystifying-the-lottery-ticket-hypothesis-in-deep-learning-158570b62674/)\n## [Demystifying the Lottery Ticket Hypothesis in Deep Learning](https://towardsdatascience.com/demystifying-the-lottery-ticket-hypothesis-in-deep-learning-158570b62674/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nWhy lottery tickets are the next big thing in training neural networks\n[David Yastremsky](https://towardsdatascience.com/author/davidyastremsky/)\nMarch 3, 20224 min read\n * [![](../images/016.jpeg)](https://towardsdatascience.com/fitbit-google-ai-5g-e6764d39e44e/)\n## [Fitbit + Google AI + 5G = ?](https://towardsdatascience.com/fitbit-google-ai-5g-e6764d39e44e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nGoogle Buys Fitbit for $2.1 billion\n[Alex Moltzau](https://towardsdatascience.com/author/alexmoltzau/)\nNovember 2, 20195 min read\n * [![](../images/017.jpg)](https://towardsdatascience.com/transformers-for-tabular-data-tabtransformer-deep-dive-5fb2438da820/)\n## [Transformers for Tabular Data: TabTransformer Deep Dive](https://towardsdatascience.com/transformers-for-tabular-data-tabtransformer-deep-dive-5fb2438da820/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nMaking sense out of TabTransformer and learning to apply it in TensorFlow\n[Antons Tocilins-Ruberts](https://towardsdatascience.com/author/antonsruberts/)\nSeptember 13, 20227 min read\n * [![](../images/018.jpeg)](https://towardsdatascience.com/openai-api-intro-11-practical-implementation-examples-of-the-models-behind-chatgpt-18601f68b51b/)\n## [OpenAI API - Intro & 11 Practical Implementation Examples of the Models Behind ChatGPT](https://towardsdatascience.com/openai-api-intro-11-practical-implementation-examples-of-the-models-behind-chatgpt-18601f68b51b/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA programmatic approach to use models behind ChatGPT.\n[Farzad Nobar](https://towardsdatascience.com/author/fmnobar/)\nNovember 7, 202319 min read\n * [![](../images/019.jpg)](https://towardsdatascience.com/these-are-the-top-6-things-people-get-wrong-when-teaching-themselves-data-science-ae8a0bd72fe7/)\n## [These Are the Top 6 Things People Get Wrong When Teaching Themselves Data Science](https://towardsdatascience.com/these-are-the-top-6-things-people-get-wrong-when-teaching-themselves-data-science-ae8a0bd72fe7/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAnd how to avoid making the same mistakes yourself.\n[Madison Hunter](https://towardsdatascience.com/author/madison13/)\nSeptember 6, 202110 min read\n * [![](../images/020.jpg)](https://towardsdatascience.com/7-handy-built-in-python-functions-3a32d2aa0301/)\n## [7 Handy Built-In Python Functions](https://towardsdatascience.com/7-handy-built-in-python-functions-3a32d2aa0301/)\n[Programming](https://towardsdatascience.com/category/programming/)\nDo you know how to properly use these functions already built into Python? Awareness of...\n[Vivienne DiFrancesco](https://towardsdatascience.com/author/heythatsviv/)\nMay 2, 20215 min read\n * [![](../images/body-63beb50f664a055ed739.jpg)](https://towardsdatascience.com/deploying-a-serverless-inference-service-with-amazon-sagemaker-pipelines-2d2f3cc96c39/)\n## [Deploying a Serverless Inference Service with Amazon SageMaker Pipelines](https://towardsdatascience.com/deploying-a-serverless-inference-service-with-amazon-sagemaker-pipelines-2d2f3cc96c39/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nDeploying a serverless inference service with Amazon SageMaker Pipelines, AWS Lambda, Amazon API Gateway, and CDK\n[Sofian Hamiti](https://towardsdatascience.com/author/sofian-hamiti/)\nMarch 15, 20216 min read\n * [![](../images/022.jpg)](https://towardsdatascience.com/how-to-effectively-work-with-frontend-and-backend-code/)\n## [How to Work Effectively with Frontend and Backend Code](https://towardsdatascience.com/how-to-effectively-work-with-frontend-and-backend-code/)\n[LLM Applications](https://towardsdatascience.com/category/artificial-intelligence/llm-applications/)\nLearn how to be an effective full-stack engineer with Claude Code\n[Eivind Kjosbakken](https://towardsdatascience.com/author/oieivind/)\nFebruary 4, 20266 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "metadata": { "title": "The Math Behind Multi-Head Attention in Transformers | Towards Data Science", "description": "Deep Dive into Multi-Head Attention, the secret element in Transformers and LLMs. Let's explore its math, and build it from scratch.", "keywords": null, "author": "Cristian Leo", "og:locale": "en_US", "og:type": "article", "og:title": "The Math Behind Multi-Head Attention in Transformers", "og:description": "Deep Dive into Multi-Head Attention, the secret element in Transformers and LLMs. Let's explore its math, and build it from scratch.", "og:url": "https://towardsdatascience.com/the-math-behind-multi-head-attention-in-transformers-c26cba15f625/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/0tcKqpHLdTSoUYJfU.jpg", "og:image:width": "1792", "og:image:height": "1024", "og:image:type": "image/jpeg", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Cristian Leo", "twitter:label2": "Est. reading time", "twitter:data2": "16 min read", "article:published_time": "2024-07-16T05:32:42.000Z", "article:modified_time": "2024-07-16T05:32:42.000Z", "article:author": "https://towardsdatascience.com/author/cristianleo120/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 202, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/767188f87239/code/block_000.txt", "chars": 242 }, { "language": "txt", "file": "attachments/767188f87239/code/block_001.txt", "chars": 240 }, { "language": "txt", "file": "attachments/767188f87239/code/block_002.txt", "chars": 313 }, { "language": "txt", "file": "attachments/767188f87239/code/block_003.txt", "chars": 171 }, { "language": "txt", "file": "attachments/767188f87239/code/block_004.txt", "chars": 58 }, { "language": "txt", "file": "attachments/767188f87239/code/block_005.txt", "chars": 28 }, { "language": "txt", "file": "attachments/767188f87239/code/block_006.txt", "chars": 45 }, { "language": "txt", "file": "attachments/767188f87239/code/block_007.txt", "chars": 174 }, { "language": "txt", "file": "attachments/767188f87239/code/block_008.txt", "chars": 497 }, { "language": "txt", "file": "attachments/767188f87239/code/block_009.txt", "chars": 18 }, { "language": "txt", "file": "attachments/767188f87239/code/block_010.txt", "chars": 59 }, { "language": "txt", "file": "attachments/767188f87239/code/block_011.txt", "chars": 545 }, { "language": "txt", "file": "attachments/767188f87239/code/block_012.txt", "chars": 193 }, { "language": "txt", "file": "attachments/767188f87239/code/block_013.txt", "chars": 90 }, { "language": "txt", "file": "attachments/767188f87239/code/block_014.txt", "chars": 78 }, { "language": "txt", "file": "attachments/767188f87239/code/block_015.txt", "chars": 96 }, { "language": "txt", "file": "attachments/767188f87239/code/block_016.txt", "chars": 168 }, { "language": "txt", "file": "attachments/767188f87239/code/block_017.txt", "chars": 304 }, { "language": "txt", "file": "attachments/767188f87239/code/block_018.txt", "chars": 77 }, { "language": "txt", "file": "attachments/767188f87239/code/block_019.txt", "chars": 100 } ], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:37:18Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:37:18Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:37:18Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:37:18Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/cristianleo120/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/deep-dives/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/math/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/programming/" }, { "type": "internal", "href": "https://towardsdatascience.com/demystifying-artificial-intelligence-d2887879aaf1/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/bharath-k1297/" }, { "type": "internal", "href": "https://towardsdatascience.com/demystifying-the-lottery-ticket-hypothesis-in-deep-learning-158570b62674/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/davidyastremsky/" }, { "type": "internal", "href": "https://towardsdatascience.com/fitbit-google-ai-5g-e6764d39e44e/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/alexmoltzau/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://roadmap.sh/ai-engineer" }, { "type": "external", "href": "https://roadmap.sh/computer-science" }, { "type": "external", "href": "https://roadmap.sh/machine-learning" }, { "type": "external", "href": "https://arxiv.org/abs/1706.03762" }, { "type": "external", "href": "https://github.com/cristianleoo/models-from-scratch-python/blob/main/Multi-Head%20Attention/demo.py" }, { "type": "external", "href": "https://jalammar.github.io/illustrated-transformer/" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=The+Math+Behind+Multi-Head+Attention+in+Transformers&u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=The+Math+Behind+Multi-Head+Attention+in+Transformers&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F" }, { "type": "external", "href": "https://x.com/share?text=The+Math+Behind+Multi-Head+Attention+in+Transformers&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F" }, { "type": "external", "href": "https://www.youtube.com/c/TowardsDataScience" }, { "type": "external", "href": "https://www.threads.net/@towardsdatascience" }, { "type": "external", "href": "https://bsky.app/profile/towardsdatascience.com" }, { "type": "external", "href": "https://thenewstack.io" }, { "type": "external", "href": "https://roadmap.sh" }, { "type": "external", "href": "https://insightmediagroup.io/creator-network" } ], "repair_version": 2, "repaired_at": "2026-10-20T17:23:11Z", "source_url": "https://towardsdatascience.com/the-math-behind-multi-head-attention-in-transformers-c26cba15f625/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# The Math Behind Multi-Head Attention in Transformers\nDeep Dive into Multi-Head Attention, the secret element in Transformers and LLMs. Let's explore its math, and build it from scratch.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nJuly 16, 202416 min read\n### Deep Dive into Multi-Head Attention, the secret element in Transformers and LLMs. Let's explore its math, and build it from scratch in Python\n![Image generated by DALL-E](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/0tcKqpHLdTSoUYJfU.jpg)\nImage generated by DALL-E\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer), [Computer Science](https://roadmap.sh/computer-science) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n## 1: Introduction\n### 1.1: Transformers Overview\nThe Transformer architecture, introduced by Vaswani et al. in their paper \"[Attention is All You Need](https://arxiv.org/abs/1706.03762)\" has transformed deep learning, particularly in natural language processing (NLP). Transformers use a self-attention mechanism, enabling them to handle input sequences all at once. This parallel processing allows for faster computation and better management of long-range dependencies within the data. This doesn't sound familiar? Don't worry as it will be at the end of this article. Let's first take a brief look at what a Transformer looks like.\n![Transformer Architecture \\(Architecture in "Attention is all you need"\\) - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1gcISnADMZ9vvgHn-cYIC0Q.jpeg)\nTransformer Architecture (Architecture in \"[Attention is all you need](https://arxiv.org/abs/1706.03762)\") - Image by Author\nA Transformer consists of two main parts: an **encoder** and a **decoder**. The encoder processes the input sequence to create a continuous representation, while the decoder generates the output sequence from this representation. Both the encoder and the decoder have multiple layers, each containing two essential components: a multi-head self-attention mechanism and a position-wise feed-forward network. In this article, we'll focus on the multi-head attention mechanism, but we'll explore the entire Transformer architecture in future articles.\n### 1.2: Multi-Head Attention Overview\nMulti-head attention enables the model to focus on different parts of the input sequence at the same time, capturing various aspects of the data. Think of it like having multiple spotlights shining on different parts of a stage. Each spotlight (or \"head\") can illuminate a different performer (or data feature), allowing the audience (or model) to see the whole scene more clearly. By splitting the input into multiple subspaces, each with its attention mechanism, multi-head attention provides the model with several views of the input data. This setup helps the model understand complex relationships within the data more effectively.\nThis mechanism allows the _Transformer_ to capture different relationships in the data by attending to various parts of the sequence. This improves the learning process by offering multiple perspectives of the input, enhancing the model's ability to generalize. It also increases the model's expressiveness by enabling it to learn different aspects of the input data simultaneously.\nThese capabilities make multi-head attention a crucial component in the success of Transformer models across a range of applications, from language translation to image processing.\n## 2: The Mathematical Foundations\n![Multi-Head Attention Architecture - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1_CY-5FTUDoJrWjPRRlqnkQ.jpeg)\nMulti-Head Attention Architecture - Image by Author\n### **2.1: Attention Mechanism**\nThe attention mechanism in neural networks is designed to mimic the human ability to focus on specific parts of information while processing data. Imagine reading a book: your eyes don't pay equal attention to every word on the page. Instead, they focus more on the important words that help you understand the story. Similarly, in neural networks, attention allows the model to dynamically weigh the importance of different input elements. This means the model can prioritize parts of the input sequence that are more relevant for generating the output, improving its performance in tasks like language translation, text summarization, and more.\nMathematically, the attention mechanism can be described using a set of queries, keys, and values. Let's denote the input as a set of queries **Q** , keys **K** , and values **V**. These are typically linear transformations of the input data.\nThe attention scores are calculated by taking the dot product of the query with all keys, which gives a measure of similarity. For a query **q** and a set of keys **k1, k2, ..., kn** ​, the attention scores are given by:\n![Attention Score - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1A5K2jQyuRGyL0ScnuBMGqw.png)\nAttention Score - Image by Author\nThink of this as comparing how similar each word (key) in a sentence is to the word (query) you're focusing on. Higher scores indicate greater similarity.\nTo prevent the dot products from becoming too large, especially when dealing with high-dimensional vectors, we scale the scores by the square root of the dimension of the keys, **d_k** :\n![Scaled Score Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1pE9Wyffo2Em9vYhAYMlKtw.png)\nScaled Score Formula - Image by Author\nThis is like adjusting the intensity of the spotlight based on how large the stage is. It ensures that the scores remain manageable and helps maintain stable gradients during training. This scaling ensures that the values passed to the softmax function have a standard deviation close to 1, which helps maintain stable gradients during training.\nTo see why this is necessary, consider the properties of dot products and high-dimensional vectors. When we compute the dot product of two vectors **q** and **k_i** ​ of dimension **d_k** ​, the expected value of their dot product is proportional to **d_k​**. Without scaling, as **d_k** ​ increases, the variance of the dot product grows, leading to very large values which can cause the softmax function to produce near-binary outputs (i.e., probabilities close to 0 or 1). This sharpness reduces the model's ability to learn effectively because it makes the gradients very small.\nBy dividing the dot product by **d_k** ​​, we normalize the input to the softmax function, ensuring that the scores remain within a reasonable range. This normalization helps the model to maintain a balance, enabling it to learn more effectively and stably.\nThese scaled scores are then passed through a softmax function to obtain the attention weights. The softmax function converts the scores into probabilities, which represent the importance of each key relative to the query:\n![Attention Weight using Softmax Function - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/13aTxqvE3_-vi1-xdhwbLQw.png)\nAttention Weight using Softmax Function - Image by Author\nThis step is like converting the adjusted spotlight intensities into a clear ranking, highlighting the most relevant parts of the scene more brightly.\nThe final attention output is obtained by taking a weighted sum of the values, using the attention weights:\n![Attention Output Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/19G1SFMBl5uZtIvhvgEI6Zw.png)\nAttention Output Formula - Image by Author\nHere, **v_i** ​ represents the value corresponding to key **k_i**. This weighted sum combines the most relevant information from the values, much like focusing your attention on the most important parts of the book to understand the story better.\n### **2.2: Multi-Head Attention**\nMulti-head attention is an advanced form of the attention mechanism that allows a model to focus on different parts of the input sequence simultaneously, capturing various relationships within the data. Instead of having a single attention mechanism, multi-head attention processes the input into multiple \"heads,\" each with its own set of queries, keys, and values. Each head performs the attention operation independently, and their outputs are then combined. This enhances the model's ability to understand complex patterns and dependencies in the data.\nImagine you're trying to understand a complex scene with many elements. If you had multiple pairs of eyes, each looking at different parts of the scene, you'd get a more comprehensive understanding. Similarly, multi-head attention allows the model to focus on different parts of the input data at once, providing a richer and more detailed representation.\nGiven an input sequence **X** , we project it into queries **Q** , keys **K** , and values **V** using learned linear transformations. For each head **i** , we have separate weight matrices **W_Q** ​, **W_K** ​, and **W_V** ​:\n![Queries, Keys, and Values Linear Transformations - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1MtmF1ce0bI_A8z6LmH567w.png)\nQueries, Keys, and Values Linear Transformations - Image by Author\nThese projections allow each head to focus on different aspects of the input data. For each head **i** , we compute the attention scores using the scaled dot-product attention mechanism. The attention output for head **i** is:\n![Attention Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1Nm2P0YbF6NzwsHQVkpQmEQ.png)\nAttention Formula - Image by Author\nHere, **d_k** ​ is the dimension of the key vectors, ensuring the scores are appropriately scaled.\nAfter computing the attention outputs for all heads, we concatenate them along the feature dimension. If we have **h** heads, each producing an output of dimension **d_v** ​, the concatenated output will have a dimension of **h** ×**d_v** ​:\n![Multi-Head Concatenation - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/12njRpNxfc4_MlLGnT-giOw.png)\nMulti-Head Concatenation - Image by Author\nThe concatenated output is then projected back to the original input dimension **d** ​ using a learned weight matrix **W_O** :\n![Final Layer of Multi-Head Attention - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1dBgIc27GDIA-7ebnmOMJlQ.png)\nFinal Layer of Multi-Head Attention - Image by Author\nThis final linear transformation combines the outputs from all heads into a single representation.\nThe core idea behind combining multiple attention heads is to allow the model to capture different types of information from the input sequence simultaneously. By having multiple heads, each head can learn to attend to different parts of the input or different features. This diversity in attention leads to a richer and more nuanced representation of the data.\n### **2.3: Position-wise Feed-Forward Networks**\nIn the Transformer architecture, each layer consists of a multi-head attention mechanism followed by a position-wise feed-forward network. These feed-forward layers are applied independently to each position in the sequence, hence the term \"position-wise.\" Essentially, they are simple fully connected neural networks applied separately and identically to each position of the input sequence.\nImagine a factory where every product on the conveyor belt goes through the same set of machines. Each machine processes the product in a specific way, adding something new or refining it. Similarly, each position in the sequence is processed independently by the feed-forward layers, transforming and enhancing the representation.\nThe purpose of these feed-forward layers is to introduce non-linearity and additional learning capacity to the model. After the attention mechanism has aggregated information from different parts of the sequence, the feed-forward network processes this information to further transform and refine the representation.\nMathematically, a position-wise feed-forward network consists of two linear transformations with a ReLU activation function in between. Given an input **x** at a particular position, the feed-forward network can be represented as:\n![Feed-Forward Network Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1l0krgngwShFmWicrDByuBw.png)\nFeed-Forward Network Formula - Image by Author\nHere:\n * **W1** ​ and **W2** ​ are learned weight matrices.\n * **b1** ​ and **b2** ​ are learned bias vectors.\n * **max(0, xW1​+b1​)** represents the ReLU activation function applied element-wise.\n\n\nThe input **x** is first linearly transformed using the weight matrix **W1** ​ and bias **b1** ​:\n![Linear Transformation of the input - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1XV0VGR5suzDoVSa9j3OLiQ.png)\nLinear Transformation of the input - Image by Author\nThink of this step as passing the input through the first machine in the factory, which adds initial modifications based on learned weights and biases.\nThe linear transformation is followed by a ReLU activation function, which introduces non-linearity:\n![ReLU Formula - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/10bz2liFMMTCyd_nBCZuSLw.png)\nReLU Formula - Image by Author\nReLU (Rectified Linear Unit) sets all negative values to zero, allowing the model to capture non-linear relationships in the data. This step is like ensuring only positive contributions from the first machine are passed on.\nThe activated output is then passed through a second linear transformation using weight matrix **W2** ​ and bias **b2** ​:\n![Second FFN - Image by Author](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/07/1nbilhl7VwK3i01LflzowlA.png)\nSecond FFN - Image by Author\nThis final step further refines the output, much like the second machine in the factory making additional modifications to produce a finished product.\nThe position-wise feed-forward network in the Transformer architecture further processes the information captured by the multi-head attention mechanism. While the attention mechanism allows the model to focus on different parts of the sequence and aggregate context-specific information, the feed-forward network refines and transforms this information at each position. This enhances the model's ability to capture complex patterns and dependencies.\n## 3: Building Multi-Head Attention from Scratch\nIn this section, we will break down and explain the implementation of a multi-head attention mechanism from scratch using Python and `numpy`. The goal is to understand how the input is modified during the process. Before proceed with your reading, take a look at the code we will cover in this section. You should be able to get a general understanding, but don't worry if now, as we will go over each line in this section.\n> [**models-from-scratch-python/Multi-Head Attention/demo.py at main ·...**](https://github.com/cristianleoo/models-from-scratch-python/blob/main/Multi-Head%20Attention/demo.py)\nTo begin, we define the `MultiHeadAttention` class, which is responsible for managing the parameters needed for the multi-head attention mechanism. Let's go through this step-by-step to understand how we set it up.\npython\n\n```\nimport numpy\n\nclass MultiHeadAttention:\n def __init__(self, num_hiddens, num_heads, dropout=0.0, bias=False):\n self.num_heads = num_heads\n self.num_hiddens = num_hiddens\n self.d_k = self.d_v = num_hiddens // num_heads\n```\n\nIn the initialization method, we first set the number of attention heads and the total number of hidden units in the model. These values are provided as arguments when the class is instantiated.\n * `num_hiddens`: This represents the total number of hidden units in the model. It's a crucial parameter because it determines the size of the linear transformations applied to the input data.\n * `num_heads`: This indicates the number of attention heads. Each head will independently learn to focus on different parts of the input, enabling the model to capture various aspects of the data.\n * `dropout`: This is the dropout rate, which is not used in this particular implementation but is included for completeness.\n * `bias`: This is a boolean flag indicating whether to include biased terms in the linear transformations.\n\n\nWe then calculate the dimensions of the queries and values for each head. Since the total number of hidden units (`num_hiddens`) is split across all heads (`num_heads`), each head will have a query and value dimension of `num_hiddens // num_heads`.\npython\n\n```\n self.W_q = np.random.rand(num_hiddens, num_hiddens)\n self.W_k = np.random.rand(num_hiddens, num_hiddens)\n self.W_v = np.random.rand(num_hiddens, num_hiddens)\n self.W_o = np.random.rand(num_hiddens, num_hiddens)\n```\n\nNext, we initialize the weight matrices for the queries, keys, values, and output transformations. These weight matrices are randomly initialized:\n * `W_q` is used to transform the input data into queries. It has dimensions `num_hiddens x num_hiddens`, meaning it maps the input features to the query space.\n * `W_k` is used to transform the input data into keys. It also has dimensions `num_hiddens x num_hiddens`, mapping the input features to the key space.\n * `W_v` is used to transform the input data into values, with the same dimensions as the previous matrices.\n * `W_o` is used to transform the concatenated output of all heads back to the original input dimensions.\n\n\npython\n\n```\n if bias:\n self.b_q = np.random.rand(num_hiddens)\n self.b_k = np.random.rand(num_hiddens)\n self.b_v = np.random.rand(num_hiddens)\n self.b_o = np.random.rand(num_hiddens)\n else:\n self.b_q = self.b_k = self.b_v = self.b_o = np.zeros(num_hiddens)\n```\n\nFinally, we initialize the bias vectors for the queries, keys, values, and output transformations. If the `bias` parameter is set to `True`, these biases are randomly initialized. Otherwise, they are set to zero:\n * `b_q`: Bias for the query transformation.\n * `b_k`: Bias for the key transformation.\n * `b_v`: Bias for the value transformation.\n * `b_o`: Bias for the output transformation.\n\n\nThe biases have dimensions equal to the number of hidden units, `num_hiddens`.\nBy setting up these weights and biases, we ensure that each attention head can independently learn to focus on different parts of the input data.\nNext, we define methods to prepare and transform the data for multi-head attention. First, let's look at the `transpose_qkv` method:\npython\n\n```\ndef transpose_qkv(self, X):\n X = X.reshape(X.shape[0], X.shape[1], self.num_heads, -1)\n X = X.transpose(0, 2, 1, 3)\n return X.reshape(-1, X.shape[2], X.shape[3])\n```\n\nThis method is responsible for reshaping and transposing the input data to prepare it for multi-head attention. In particular:\npython\n\n```\nX = X.reshape(X.shape[0], X.shape[1], self.num_heads, -1)\n```\n\nThis line reshapes the input tensor `X` to have four dimensions: `(batch_size, sequence_length, num_heads, depth_per_head)`.\n * `X.shape[0]` is the batch size.\n * `X.shape[1]` is the sequence length (number of positions in the input sequence).\n * `self.num_heads` is the number of attention heads.\n * `-1` automatically infers the size of the last dimension (depth per head) so that the total number of elements remains the same.\n\n\npython\n\n```\nX = X.transpose(0, 2, 1, 3)\n```\n\nThis line transposes the tensor so that the dimensions are reordered to `(batch_size, num_heads, sequence_length, depth_per_head)`.\nThis rearrangement ensures that each attention head processes its part of the input sequence independently.\npython\n\n```\nreturn X.reshape(-1, X.shape[2], X.shape[3])\n```\n\nThis final reshape flattens the batch and head dimensions into a single dimension, resulting in a tensor of shape `(batch_size * num_heads, sequence_length, depth_per_head)`.\nBy doing this, `transpose_qkv` ensures that the input data is split correctly among the multiple heads, with each head having the appropriate dimensions to process its segment of the data.\nNext, we have the `transpose_output` method:\npython\n\n```\ndef transpose_output(self, X):\n X = X.reshape(-1, self.num_heads, X.shape[1], X.shape[2])\n X = X.transpose(0, 2, 1, 3)\n return X.reshape(X.shape[0], X.shape[1], -1)\n```\n\nThis method reverses the transformation done by `transpose_qkv` to combine the outputs from all heads back into the original shape.\nAfter transposing our matrices, we can process with the scaled dot-product attention mechanism, which allows the model to focus on different parts of the input sequence with varying degrees of importance.\npython\n\n```\ndef scaled_dot_product_attention(self, Q, K, V, valid_lens):\n d_k = Q.shape[-1]\n scores = np.matmul(Q, K.transpose(0, 2, 1)) / np.sqrt(d_k)\n if valid_lens is not None:\n mask = np.arange(scores.shape[-1]) < valid_lens[:, None]\n scores = np.where(mask[:, None, :], scores, -np.inf)\n attention_weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))\n attention_weights /= attention_weights.sum(axis=-1, keepdims=True)\n return np.matmul(attention_weights, V)\n```\n\nThe inputs to this method are the query (**Q**), key (**K**), and value (**V**) matrices. These matrices are derived from the input data through linear transformations.\npython\n\n```\nd_k = Q.shape[-1]\n```\n\nHere, we extract the dimension of the key vectors, **d_k** ​, from the last dimension of the query matrix **Q**. This value is used for scaling the attention scores.\npython\n\n```\nscores = np.matmul(Q, K.transpose(0, 2, 1)) / np.sqrt(d_k)\n```\n\nWe calculate the attention scores by performing a matrix multiplication of **Q** and the transpose of **K**. The scores are then scaled by the square root of **d_k** ​. This scaling helps prevent the scores from growing too large, which can lead to issues during the softmax calculation.\nNext, we define the forward pass method to process the input data through the multi-head attention mechanism. This method is crucial as it orchestrates the entire multi-head attention process, from transforming the input data to combining the outputs from multiple heads.\npython\n\n```\ndef forward(self, queries, keys, values, valid_lens):\n queries = self.transpose_qkv(np.dot(queries, self.W_q) + self.b_q)\n keys = self.transpose_qkv(np.dot(keys, self.W_k) + self.b_k)\n values = self.transpose_qkv(np.dot(values, self.W_v) + self.b_v)\n\n if valid_lens is not None:\n valid_lens = np.repeat(valid_lens, self.num_heads, axis=0)\n\n output = self.scaled_dot_product_attention(queries, keys, values, valid_lens)\n output_concat = self.transpose_output(output)\n return np.dot(output_concat, self.W_o) + self.b_o\n```\n\nLet's break down the forward method:\npython\n\n```\nqueries = self.transpose_qkv(np.dot(queries, self.W_q) + self.b_q)\nkeys = self.transpose_qkv(np.dot(keys, self.W_k) + self.b_k)\nvalues = self.transpose_qkv(np.dot(values, self.W_v) + self.b_v)\n```\n\nFirst, the input queries, keys, and values are projected into their respective subspaces using the learned weight matrices (`W_q`, `W_k`, `W_v`) and biases (`b_q`, `b_k`, `b_v`). This is done by performing matrix multiplication with the weight matrices and adding the biases. The results are then transformed for multi-head attention using the `transpose_qkv` method, which reshapes and transposes the data to ensure each head processes the input independently.\n**Queries, keys, and values** are the transformed inputs, now prepared for multi-head attention.\npython\n\n```\nif valid_lens is not None:\n valid_lens = np.repeat(valid_lens, self.num_heads, axis=0)\n```\n\nIf `valid_lens` (valid lengths) are provided, they are repeated for each head. This ensures that the appropriate mask is created for each attention head, allowing the model to focus only on valid positions within the sequences.\npython\n\n```\noutput = self.scaled_dot_product_attention(queries, keys, values, valid_lens)\n```\n\nThe method then calls `scaled_dot_product_attention` with the transformed queries, keys, values, and repeated valid lengths. This function calculates the attention scores, applies the softmax function to obtain attention weights, and computes the weighted sum of the values to produce the attention output for each head.\npython\n\n```\noutput_concat = self.transpose_output(output)\nreturn np.dot(output_concat, self.W_o) + self.b_o\n```\n\nAfter obtaining the attention outputs from all heads, the method concatenates these outputs along the feature dimension using `transpose_output`. This method reverses the initial transformation, combining the outputs from all heads back into a single representation. The concatenated output is then transformed back to the original input dimension using a final linear transformation with weight matrix `W_o` and bias `b_o`.\nLastly, we test the class with some sample data. Here's how we do it:\ngraphql\n\n```\n# Define dimensions and initialize multi-head attention\nnum_hiddens, num_heads = 100, 5\nattention = MultiHeadAttention(num_hiddens, num_heads, dropout=0.5, bias=False)\n```\n\nWe initialize the `MultiHeadAttention` class with 100 hidden units and 5 attention heads. This sets up the necessary parameters and weight matrices for the multi-head attention mechanism.\npython\n\n```\n# Define sample data\nbatch_size, num_queries, num_kvpairs = 2, 4, 6\nvalid_lens = np.array([3, 2])\nX = np.random.rand(batch_size, num_queries, num_hiddens) # Use random data to simulate input queries\nY = np.random.rand(batch_size, num_kvpairs, num_hiddens) # Use random data to simulate key-value pairs\n```\n\nWe create random data to simulate input queries (`X`) and key-value pairs (`Y`). The batch size is 2, the number of queries is 4, and the number of key-value pairs is 6. We also define valid lengths (`valid_lens`) to indicate the valid positions within the sequences.\npython\n\n```\n# Apply multi-head attention\noutput = attention.forward(X, Y, Y, valid_lens)\n```\n\nWe pass the sample data through the multi-head attention mechanism using the `forward` method. This processes the input queries, keys, and values, applying the multi-head attention calculations.\npython\n\n```\nprint(\"Output shape:\", output.shape)\n# Output should be: (2, 4, 100)\n\nprint(\"Output data:\", output)\n```\n\nWe print the shape and content of the output. The expected output shape ensures that the output dimensions match the original input dimensions. Then, we print the output data after computing Multi-Head Attention. Now that you have an understanding of how the Multi-Head Attention mechanism works, try tweaking it. For instance, change the number of heads, try adding multiple FFN before and after Multi-Head Attention. Also, you could try to implement it in a machine translation task, and see it in action. Let me know if you would like me to do that in a next article.\n## Conclusion\nTransformers have transformed deep learning, especially in NLP, by using self-attention mechanisms that allow for parallel processing of input sequences. This approach not only speeds up computation but also handles long-range dependencies more effectively than traditional recurrent neural networks).\nIn this article, we've gained a comprehensive understanding of multi-head attention in Transformers, from it's math theory to a practical code implementation. Maybe for now the concepts we will still be abstracts, as you can't really do anything with the outputs from Multi-Head Attention, but soon we will see how they play a key role in the transformer architecture, which is the base of the well-known LLMs around (Claude, ChatGPT, ...). Stay tuned for future articles, where we'll explore the remaining components of the Transformer architecture, offering deeper insights into this powerful model.\n## References\n * Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). [Attention is All You Need](https://arxiv.org/abs/1706.03762). In Advances in Neural Information Processing Systems (NeurIPS).\n * [Alammar, J. (2018). The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/). jalammar.github.io.\n\n\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657505387_wlyc0o.jpeg)\nWritten By\n## [Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[See all from Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\n[Deep Dives](https://towardsdatascience.com/tag/deep-dives/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Math](https://towardsdatascience.com/tag/math/)[Programming](https://towardsdatascience.com/tag/programming/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F&title=The%20Math%20Behind%20Multi-Head%20Attention%20in%20Transformers)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F&title=The%20Math%20Behind%20Multi-Head%20Attention%20in%20Transformers)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fthe-math-behind-multi-head-attention-in-transformers-c26cba15f625%2F&text=The%20Math%20Behind%20Multi-Head%20Attention%20in%20Transformers)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/1s3hc0Foz31LJVq3U0xWPkQ.gif)](https://towardsdatascience.com/demystifying-artificial-intelligence-d2887879aaf1/)\n## [Demystifying Artificial Intelligence!](https://towardsdatascience.com/demystifying-artificial-intelligence-d2887879aaf1/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nBreaking down the field of Artificial Intelligence and what the future holds for it.\n[Bharath K](https://towardsdatascience.com/author/bharath-k1297/)\nNovember 2, 20208 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/03/09_NWVjR3bcaefsNq-scaled.jpg)](https://towardsdatascience.com/demystifying-the-lottery-ticket-hypothesis-in-deep-learning-158570b62674/)\n## [Demystifying the Lottery Ticket Hypothesis in Deep Learning](https://towardsdatascience.com/demystifying-the-lottery-ticket-hypothesis-in-deep-learning-158570b62674/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nWhy lottery tickets are the next big thing in training neural networks\n[David Yastremsky](https://towardsdatascience.com/author/davidyastremsky/)\nMarch 3, 20224 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/11/1dNHl6L7ii0zgDL4yziIrnA-scaled.jpeg)](https://towardsdatascience.com/fitbit-google-ai-5g-e6764d39e44e/)\n## [Fitbit + Google AI + 5G = ?](https://towardsdatascience.com/fitbit-google-ai-5g-e6764d39e44e/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nGoogle Buys Fitbit for $2.1 billion\n[Alex Moltzau](https://towardsdatascience.com/author/alexmoltzau/)\nNovember 2, 20195 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/09/0iPDX0cNVz95Ww3cO-scaled.jpg)](https://towardsdatascience.com/transformers-for-tabular-data-tabtransformer-deep-dive-5fb2438da820/)\n## [Transformers for Tabular Data: TabTransformer Deep Dive](https://towardsdatascience.com/transformers-for-tabular-data-tabtransformer-deep-dive-5fb2438da820/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nMaking sense out of TabTransformer and learning to apply it in TensorFlow\n[Antons Tocilins-Ruberts](https://towardsdatascience.com/author/antonsruberts/)\nSeptember 13, 20227 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/11/1nZCF9XBsCMn-eFcEB-lLDw-scaled.jpeg)](https://towardsdatascience.com/openai-api-intro-11-practical-implementation-examples-of-the-models-behind-chatgpt-18601f68b51b/)\n## [OpenAI API - Intro & 11 Practical Implementation Examples of the Models Behind ChatGPT](https://towardsdatascience.com/openai-api-intro-11-practical-implementation-examples-of-the-models-behind-chatgpt-18601f68b51b/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA programmatic approach to use models behind ChatGPT.\n[Farzad Nobar](https://towardsdatascience.com/author/fmnobar/)\nNovember 7, 202319 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/00qLDudVeAlJhiV8r-scaled.jpg)](https://towardsdatascience.com/these-are-the-top-6-things-people-get-wrong-when-teaching-themselves-data-science-ae8a0bd72fe7/)\n## [These Are the Top 6 Things People Get Wrong When Teaching Themselves Data Science](https://towardsdatascience.com/these-are-the-top-6-things-people-get-wrong-when-teaching-themselves-data-science-ae8a0bd72fe7/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAnd how to avoid making the same mistakes yourself.\n[Madison Hunter](https://towardsdatascience.com/author/madison13/)\nSeptember 6, 202110 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/05/01SmrWP_D99v2k7zL-scaled.jpg)](https://towardsdatascience.com/7-handy-built-in-python-functions-3a32d2aa0301/)\n## [7 Handy Built-In Python Functions](https://towardsdatascience.com/7-handy-built-in-python-functions-3a32d2aa0301/)\n[Programming](https://towardsdatascience.com/category/programming/)\nDo you know how to properly use these functions already built into Python? Awareness of...\n[Vivienne DiFrancesco](https://towardsdatascience.com/author/heythatsviv/)\nMay 2, 20215 min read\n * [![](https://towardsdatascience.com/images/fallback-image.jpg)](https://towardsdatascience.com/deploying-a-serverless-inference-service-with-amazon-sagemaker-pipelines-2d2f3cc96c39/)\n## [Deploying a Serverless Inference Service with Amazon SageMaker Pipelines](https://towardsdatascience.com/deploying-a-serverless-inference-service-with-amazon-sagemaker-pipelines-2d2f3cc96c39/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nDeploying a serverless inference service with Amazon SageMaker Pipelines, AWS Lambda, Amazon API Gateway, and CDK\n[Sofian Hamiti](https://towardsdatascience.com/author/sofian-hamiti/)\nMarch 15, 20216 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/02/image-216.jpg)](https://towardsdatascience.com/how-to-effectively-work-with-frontend-and-backend-code/)\n## [How to Work Effectively with Frontend and Backend Code](https://towardsdatascience.com/how-to-effectively-work-with-frontend-and-backend-code/)\n[LLM Applications](https://towardsdatascience.com/category/artificial-intelligence/llm-applications/)\nLearn how to be an effective full-stack engineer with Claude Code\n[Eivind Kjosbakken](https://towardsdatascience.com/author/oieivind/)\nFebruary 4, 20266 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n\n\n", "body_base_path": "attachments/767188f87239/pages/", "body_markdown_path": "attachments/767188f87239/pages/archived-body.md", "local_html": "attachments/767188f87239/pages/archived-body.html", "body_source": "prior_capture" }