{ "source_file": "0930", "source_platform": "Medium", "author": "Bradney Smith", "title_listed": "Self-Attention Explained with Code", "keywords": "", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/", "publish_date": "2024-02-10", "corpus_stage": "", "notes": "", "id": "34afc94764d0", "collected_at": "2026-10-02T01:02:29", "title": "Self-Attention Explained with Code | Towards Data Science", "body_markdown": "![Revisit consent button](../images/body-df29beacd7496220cc94.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](../images/body-2159bb67f3de8ad870eb.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Self-Attention Explained with Code\nHow large language models create rich, contextual embeddings\n[Bradney Smith](https://towardsdatascience.com/author/bradneysmith/)\nFebruary 9, 202435 min read\n### **Part 3 in the \"LLMs from Scratch\" series - a complete guide to understanding and building Large Language Models. If you are interested in learning more about how these models work I encourage you to read:**\n * [Prelude: A Brief History of Large Language Models](https://medium.com/p/98a1320e7650/)\n * [Part 1: Tokenization - A Complete Guide](https://medium.com/p/cedc9f72de4e)\n * [Part 2: Word Embeddings with word2vec from Scratch in Python](https://medium.com/p/eb9326c6ab7c)\n * **Part 3: Self-Attention Explained with Code**\n * [Part 4: A Complete Guide to BERT with Code](https://medium.com/p/9f87602e4a11/)\n\n\n## Introduction\nThe paper \"Attention is All You Need\" debuted perhaps the single largest advancement in Natural Language Processing (NLP) in the last 10 years: the Transformer [1]. This architecture massively simplified the complex designs of language models at the time while achieving unparalleled results. State-of-the-art (SOTA) models, such as those in the GPT, Claude, and Llama families, owe their success to this design, at the heart of which is self-attention. In this deep dive, we will explore how this mechanism works and how it is used by transformers to create contextually rich embeddings that enable these models to perform so well.\n## Contents\n**1** - [Overview of the Transformer Embedding Process](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#cc4f)\n**2** - [Positional Encoding](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#365f)\n**3** -[ The Self-Attention Mechanism](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#4489)\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n**4** - [Transformer Embeddings in Python](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#cb34)\n**5** -[ Conclusion](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#126e)\n**6** - [Further Reading](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#776b)\n## 1 - Overview of the Transformer Embedding Process\n### 1.1 - Recap on Transformers\nIn the prelude article of this series, we briefly explored the history of the Transformer and its impact on NLP. To recap: the Transformer is a deep neural network architecture that is the foundation for almost all LLMs today. Derivative models are often called Transformer-based models or **transformers** for short, and so these terms will be used interchangeably here. Like all machine learning models, transformers work with numbers and linear algebra rather than processing human language directly. Because of this, they must convert textual inputs from users into numerical representations through several steps. Perhaps the most important of these steps is applying the self-attention mechanism, which is the focus of this article. The process of representing text with vectors is called **embedding** (or **encoding**), hence the numerical representations of the input text are known as **transformer embeddings**.\n### 1.2 - The Issue with Static Embeddings\nIn [Part 2 of this series](https://medium.com/p/eb9326c6ab7c), we explored static embeddings for language models using word2vec as an example. This embedding method predates transformers and suffers from one major drawback: the lack of contextual information. Words with multiple meanings (called **polysemous** words) are encoded with somewhat ambiguous representations since they lack the context needed for precise meaning. A classic example of a polysemous word is `bank`. Using a static embedding model, the word `bank` would be represented in vector space with some degree of similarity to words such as `money` and `deposit` and some degree of similarity to words such as `river` and `nature`. This is because the word will occur in many different contexts within the training data. This is the core problem with static embeddings: they do not change based on context - hence the term \"static\".\n### 1.3 - Fixing Static Embeddings\nTransformers overcome the limitations of static embeddings by producing their own context-aware transformer embeddings. In this approach, fixed word embeddings are augmented with positional information (where the words occur in the input text) and contextual information (how the words are used). These two steps take place in distinct components in transformers, namely the positional encoder and the self-attention blocks, respectively. We will look at each of these in detail in the following sections. By incorporating this additional information, transformers can produce much more powerful vector representations of words based on their usage in the input sequence. Extending the vector representations beyond static embeddings is what enables Transformer-based models to handle polysemous words and gain a deeper understanding of language compared to previous models.\n### 1.4 - Introducing Learned Embeddings\nMuch like the word2vec approach released four years prior, transformers store the initial vector representation for each token in the weights of a linear layer (a small neural network). In the word2vec model, these representations form the static embeddings, but in the Transformer context these are known as **learned embeddings**. In practice they are very similar, but using a different name emphasises that these representations are only a starting point for the transformer embeddings and not the final form.\nThe linear layer sits at the beginning of the Transformer architecture and contains only weights and no bias terms (bias = 0 for every neuron). The layer weights can be represented as a matrix of size _V_ × _d _model_ , where _V_ is the vocabulary size (the number of unique words in the training data) and _d _model_ is the number of embedding dimensions. In the previous article, we denoted _d _model_ as _N_ , in line with word2vec notation, but here we will use _d _model_ which is more common in the Transformer context. The original Transformer was proposed with a _d _model_ size of 512 dimensions, but in practice any reasonable value can be used.\n![A diagram showing the location of the linear layer in the Transformer architecture, which stores the learned embeddings. Image by author, adapted from the Transformer architecture diagram in the "Attention is All You Need" paper \\[1\\].](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1ZPePnPodMZeehez9YFmr9A.png)\nA diagram showing the location of the linear layer in the Transformer architecture, which stores the learned embeddings. Image by author, adapted from the Transformer architecture diagram in the \"Attention is All You Need\" paper [1].\n### **1.5 - Creating Learned Embeddings**\nA key difference between static and learned embeddings is the way in which they are trained. Static embeddings are trained in a separate neural network (using the Skip-Gram or Continuous Bag of Words architectures) using a word prediction task within a given window size. Once trained, the embeddings are then extracted and used with a range of different language models. Learned embeddings, however, are integral to the transformer you are using and are stored as weights in the first linear layer of the model. These weights, and consequently the learned embedding for each token in the vocabulary, are trained in the same backpropagation steps as the rest of the model parameters. Below is a summary of the training process for learned embeddings.\n**Step 1: Initialisation**\nRandomly initialise the weights for each neuron in the linear layer at the beginning of the model, and set the bias terms to 0. This layer is also called the **embedding layer,** since it is the linear layer that will store the learned embeddings. The weights can be represented as a matrix of size _V_ × _d _model_ , where the word embedding for each word in the vocabulary is stored along the rows. For example, the embedding for the first word in the vocabulary is stored in the first row, the second word is stored in the second row, and so on.\n**Step 2: Training**\nAt each training step, the Transformer receives an input word and the aim is to predict the next word in the sequence - a task known as Next Token Prediction (NTP). Initially, these predictions will be very poor, and so every weight and bias term in the network will be updated to improve performance against the loss function, including the embeddings. After many training iterations, the learned embeddings should provide a strong vector representation for each word in the vocabulary.\n**Step 3: Extract the Learned Embeddings**\nWhen new input sequences are given to the model, the words are converted into tokens with an associated token ID, which corresponds to the position of the token in the tokenizer's vocabulary. For example, the word `cat` may lie at position `349` in the tokenizer's vocabulary and so will take the ID `349`. Token IDs are used to create one-hot encoded vectors that extract the correct learned embeddings from the weights matrix (that is, _V_ -dimensional vectors where every element is 0 except for the element at the token ID position, which is 1).\n> _**Note:**__PyTorch is a very popular deep learning library in Python that powers some of the most well-known machine learning packages, such as the HuggingFace_` _Transformers_`_library [2]. If you are familiar with PyTorch, you may have encountered the_` _nn.Embedding_`_class, which is often used to form the first layer of transformer networks (the_` _nn_`_denotes that the class belongs to the neural network package). This class returns a regular linear layer that is initialised with the identity function as the activation function and with no bias term. The weights are randomly initialised since they are parameters to be learned by the model during training. This essentially carries out the steps described above in one simple line of code. Remember, the_` _nn.Embeddings_`_layer does not provide pre-trained word embeddings out-of-the-box, but rather initialises a blank canvas of embeddings before training. This is to allow the transformer to learn its own embeddings during the training phase._\n### 1.6 - Transformer Embedding Process\nOnce the learned embeddings have been trained, the weights in the embedding layer never change. That is, the learned embedding for each word (or more specifically, token) always provides the same starting point for a word's vector representation. From here, the positional and contextual information will be added to produce a unique representation of the word that is reflective of its usage in the input sequence.\nTransformer embeddings are created in a four-step process, which is demonstrated below using the example prompt: `Write a poem about a man fishing on a river bank.`. Note that the first two steps are the same as the word2vec approach we saw before. Steps 3 and 4 are the further processing that add contextual information to the embeddings.\n**Step 1) Tokenization:**\nTokenization is the process of dividing a longer input sequence into individual words (and parts of words) called tokens. In this case, the sentence will be broken down into:\n`write`, `a`, `poem`, `about`, `a`, `man`, `fishing`, `on`, `a`, `river`, `bank`\nNext, the tokens are associated with their token IDs, which are integer values corresponding to the position of the token in the tokenizer's vocabulary (see [Part 1 of this series](https://medium.com/p/cedc9f72de4e) for an in-depth look at the tokenization process).\n**Step 2) Map the Tokens to Learned Embeddings:**\nOnce the input sequence has been converted into a set of token IDs, the tokens are then mapped to their learned embedding vector representations, which were acquired during the transformer's training. These learned embeddings have the \"lookup table\" behaviour as we saw in the word2vec example in [Part 2 of this series](https://medium.com/p/eb9326c6ab7c)). The mapping takes place by multiplying a one-hot encoded vector created from the token ID with the weights matrix, just as in the word2vec approach. The learned embeddings are denoted _V_ in the image below.\n**Step 3) Add Positional Information with Positional Encoding:**\n**Positional Encoding** is then used to add positional information to the word embeddings. Whereas Recurrent Neural Networks (RNNs) process text sequentially (one word at a time), transformers process all words in parallel. This removes any implicit information about the position of each word in the sentence. For example, the sentences `the cat ate the mouse` and `the mouse ate the cat` use the same words but have very different meanings. To preserve the word order, positional encoding vectors are generated and added to the learned embedding for each word. In the image below, the positional encoding vectors are denoted _P_ , and the sums of the learned embeddings and positional encodings are denoted _X_.\n**Step 4) Modify the Embeddings using Self-Attention:**\nThe final step is to add contextual information using the self-attention mechanism. This determines which words give context to other words in the input sequence. In the image below, the transformer embeddings are denoted _y_.\n![An overview of the transformer embedding process from input text through to transformer embeddings. Image by author.](../images/001.png)\nAn overview of the transformer embedding process from input text through to transformer embeddings. Image by author.\n## 2 - Positional Encoding\n### 2.1 - The Need for Positional Encoding\nBefore the self-attention mechanism is applied, positional encoding is used to add information about the order of tokens to the learned embeddings. This compensates for the loss of positional information caused by the parallel processing used by transformers described earlier. There are many feasible approaches for injecting this information, but all methods must adhere to a set of constraints. The functions used to generate positional information must produce values that are:\n * **Bounded** - values should not explode in the positive or negative direction but be constrained (e.g. between 0 and 1, -1 and 1, etc)\n * **Periodic** - the function should produce a repeating pattern that the model can learn to recognise and discern position from\n * **Predictable** - positional information should be generated in such a way that the model can understand the position of words in sequence lengths it was not trained on. For example, even if the model has not seen a sequence length of exactly 412 tokens in its training, the transformer should be able to understand the position of each of the embeddings in the sequence.\n\n\nThese constraints ensure that the positional encoder produces positional information that allows words to **attend** to (gain context from) any other important word, regardless of their relative positions in the sequence. In theory, with a sufficiently powerful computer, words should be able to gain context from every relevant word in an infinitely long input sequence. The length of a sequence from which a model can derive context is called the **context length**. In chatbots like ChatGPT, the context includes the current prompt as well as all previous prompts and responses in the conversation (within the context length limit). This limit is typically in the range of a few thousand tokens, with GPT-3 supporting up to 4096 tokens and GPT-4 enterprise edition capping at around 128,000 tokens [3].\n### 2.2 - Positional Encoding in \"Attention is All You Need\"\nThe original transformer model was proposed with the following positional encoding functions:\n![An image of the equations for positional encoding, as proposed in the paper "Attention is All You Need" \\[1\\]. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1xzeVzfdB7_MtQnZb7wv3ng.png)\nAn image of the equations for positional encoding, as proposed in the paper \"Attention is All You Need\" [1]. Image by author.\nwhere:\n * _pos_ is the position of the word in the input, where _pos = 0_ corresponds to the first word in the sequence\n * _i_ is the index of each embedding dimension, ranging from _i=0_ (for the first embedding dimension) up to _d _model_\n * _d _model_ is the number of embedding dimensions for each learned embedding vector (and therefore each positional encoding vector). This was previously denoted _N_ in the article on word2vec.\n\n\nThe two proposed functions take arguments of _2i_ and _2i+1_ , which in practice means that the sine function generates positional information for the even-numbered dimensions of each word vector (_i_ is even), and the cosine function does so for the odd-numbered dimensions (_i_ is odd). According to the authors of the transformer:\n> _\"The positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000·2π. We chose this function because we hypothesised it would allow the model to easily learn to attend by relative positions, since for any fixed offset_ k _,_ PE_pos+k _can be represented as a linear function of_ PE _pos_ \"._\nThe value of the constant in the denominator being `10_000` was found to be suitable after some experimentation, but is a somewhat arbitrary choice by the authors.\n### 2.3 - Other Positional Encoding Approaches\nThe positional encodings shown above are considered **fixed** because they are generated by a known function with deterministic (predictable) outputs. This represents the most simple form of positional encoding. It is also possible to use **learned positional encodings** by randomly initialising some positional encodings and training them with backpropagation. Derivatives of the BERT architecture are examples of models that take this learned encoding approach. More recently, the Rotary Positional Encoding (RoPE) method has gained popularity, finding use in models such as Llama 2 and PaLM, among other positional encoding methods.\n### 2.4 - Implementing a Positional Encoder in Python\nCreating a positional encoder class in Python is fairly straightforward. We can start by defining a function that accepts the number of embedding dimensions (`d_model`), the maximum length of the input sequence (`max_length`), and the number of decimal places to round each value in the vectors to (`rounding`). Note that transformers define a maximum input sequence length, and any sequence that has fewer tokens than this limit is appended with padding tokens until the limit is reached. To account for this behaviour in our positional encoder, we accept a `max_length` argument. In practice, this limit is typically thousands of characters long.\nWe can also exploit a mathematical trick to save computation. Instead of calculating the denominator for both _PE_{pos, 2i}_ and _PE_{pos, 2i}_, we can note that the denominator is identical for consecutive pairs of _i_. For example, the denominators for _i=0_ and _i=1_ are the same, as are the denominators for _i=2_ and _i=3_. Hence, we can perform the calculations to determine the denominators once for the even values of _i_ and reuse them for the odd values of _i_.\npython\n\n```\nimport numpy as np\n\nclass PositionalEncoder():\n \"\"\" An implementation of positional encoding.\n\n Attributes:\n d_model (int): The number of embedding dimensions in the learned\n embeddings. This is used to determine the length of the positional\n encoding vectors, which make up the rows of the positional encoding\n matrix.\n max_length (int): The maximum sequence length in the transformer. This\n is used to determine the size of the positional encoding matrix.\n rounding (int): The number of decimal places to round each of the\n values to in the output positional encoding matrix.\n \"\"\"\n\n def __init__(self, d_model, max_length, rounding):\n self.d_model = d_model\n self.max_length = max_length\n self.rounding = rounding\n\n def generate_positional_encoding(self):\n \"\"\" Generate positional information to add to inputs for encoding.\n\n The positional information is generated using the number of embedding\n dimensions (d_model), the maximum length of the sequence (max_length),\n and the number of decimal places to round to (rounding). The output\n matrix generated is of size (max_length X embedding_dim), where each\n row is the positional information to be added to the learned\n embeddings, and each column is an embedding dimension.\n \"\"\"\n\n position = np.arange(0, self.max_length).reshape(self.max_length, 1)\n even_i = np.arange(0, self.d_model, 2)\n denominator = 10_000**(even_i / self.d_model)\n\n even_encoded = np.round(np.sin(position / denominator), self.rounding)\n odd_encoded = np.round(np.cos(position / denominator), self.rounding)\n\n # Interleave the even and odd encodings\n positional_encoding = np.stack((even_encoded, odd_encoded),2)\n .reshape(even_encoded.shape[0],-1)\n\n # If self.d_model is odd remove the extra column generated\n if self.d_model % 2 == 1:\n positional_encoding = np.delete(positional_encoding, -1, axis=1)\n\n return positional_encoding\n\n def encode(self, input):\n \"\"\" Encode the input by adding positional information.\n\n Args:\n input (np.array): A two-dimensional array of embeddings. The array\n should be of size (self.max_length x self.d_model).\n\n Returns:\n output (np.array): A two-dimensional array of embeddings plus the\n positional information. The array has size (self.max_length x\n self.d_model).\n \"\"\"\n positional_encoding = self.generate_positional_encoding()\n output = input + positional_encoding\n\n return output\n\nMAX_LENGTH = 5\nEMBEDDING_DIM = 3\nROUNDING = 2\n\n# Instantiate the encoder\nPE = PositionalEncoder(d_model=EMBEDDING_DIM,\n max_length=MAX_LENGTH,\n rounding=ROUNDING)\n\n# Create an input matrix of word embeddings without positional encoding\ninput = np.round(np.random.rand(MAX_LENGTH, EMBEDDING_DIM), ROUNDING)\n\n# Create an output matrix of word embeddings by adding positional encoding\noutput = PE.encode(input)\n\n# Print the results\nprint(f'Embeddings without positional encoding:nn{input}n')\nprint(f'Positional encoding:nn{output-input}n')\nprint(f'Embeddings with positional encoding:nn{output}')\n```\n\nshell\n\n```\nEmbeddings without positional encoding:\n\n[[0.12 0.94 0.9 ]\n [0.14 0.65 0.22]\n [0.29 0.58 0.31]\n [0.69 0.37 0.62]\n [0.25 0.61 0.65]]\n\nPositional encoding:\n\n[[ 0. 1. 0. ]\n [ 0.84 0.54 0. ]\n [ 0.91 -0.42 0. ]\n [ 0.14 -0.99 0.01]\n [-0.76 -0.65 0.01]]\n\nEmbeddings with positional encoding:\n\n[[ 0.12 1.94 0.9 ]\n [ 0.98 1.19 0.22]\n [ 1.2 0.16 0.31]\n [ 0.83 -0.62 0.63]\n [-0.51 -0.04 0.66]]\n```\n\n### 2.5 - Visualising the Positional Encoding Matrix\nRecall that the positional information generated must be bounded, periodic, and predictable. The outputs of the sinusoidal functions presented earlier can be collected into a matrix, which can then be easily combined with the learned embeddings using element-wise addition. Plotting this matrix gives a nice visualisation of the desired properties. In the plot below, curving bands of negative values (blue) emanate from the left edge of the matrix. These bands form a pattern that the transformer can easily learn to predict.\npython\n\n```\nimport matplotlib.pyplot as plt\n\n# Instantiate a PositionalEncoder class\nd_model = 400\nmax_length = 100\nrounding = 4\n\nPE = PositionalEncoder(d_model=d_model,\n max_length=max_length,\n rounding=rounding)\n\n# Generate positional encodings\ninput = np.round(np.random.rand(max_length, d_model), 4)\npositional_encoding = PE.generate_positional_encoding()\n\n# Plot positional encodings\ncax = plt.matshow(positional_encoding, cmap='coolwarm')\nplt.title(f'Positional Encoding Matrix ({d_model=}, {max_length=})')\nplt.ylabel('Position of the Embeddingnin the Sequence, pos')\nplt.xlabel('Embedding Dimension, i')\nplt.gcf().colorbar(cax)\nplt.gca().xaxis.set_ticks_position('bottom')\n```\n\n![A visualisation of the positional encoding matrix for a model with 400 embedding dimensions \\(d_model = 400\\), and a maximum sequence length of 100 \\(max_length = 100\\). Image by author.](../images/003.png)\nA visualisation of the positional encoding matrix for a model with 400 embedding dimensions (d_model = 400), and a maximum sequence length of 100 (max_length = 100). Image by author.\n## 3 - The Self-Attention Mechanism\n### 3.1 - Overview of Attention Mechanisms\nNow that we have covered an overview of transformer embeddings and the positional encoding step, we can turn our focus to the self-attention mechanism itself. In short, self-attention modifies the vector representation of words to capture the context of their usage in an input sequence. The \"self\" in self-attention refers to the fact that the mechanism uses the surrounding words within a single sequence to provide context. As such, self-attention requires all words to be processed in parallel. This is actually one of the main benefits of transformers (especially compared to RNNs) since the models can leverage parallel processing for a significant performance boost. In recent times, there has been some rethinking around this approach, and in the future we may see this core mechanism being replaced [4].\nAnother form of attention used in transformers is cross-attention. Unlike self-attention, which operates within a single sequence, cross-attention compares each word in an output sequence to each word in an input sequence, crossing between the two embedding matrices. Note the difference here compared to self-attention, which focuses entirely within a single sequence.\n### 3.2 - Visualising How Self-Attention Contextualises Embeddings\nThe plots below show a simplified set of learned embedding vectors in two dimensions. Words associated with nature and rivers are concentrated in the top right quadrant of the graph, while words associated with money are concentrated in the bottom left. The vector representing the word `bank` is positioned between the two clusters due to its polysemic nature. The objective of self-attention is to move the learned embedding vectors to regions of vector space that more accurately capture their meaning within the context of the input sequence. In the example input `Write a poem about a man fishing on a river bank.`, the aim is to move the vector for `bank` in such a way that captures more of the meaning of nature and rivers, and less of the meaning of money and deposits.\n> _**Note:**__More accurately, the goal of self-attention here is to update the vector for every word in the input, so that all embeddings better represent the context in which they were used. There is nothing special about the word_` _bank_`_here that transformers have some special knowledge of - self-attention is applied across all the words. We will look more at this shortly, but for now, considering solely how_` _bank_`_is affected by self-attention gives a good intuition for what is happening in the attention block. For the purpose of this visualisation, the positional encoding information has not been explicitly shown. The effect of this will be minimal, but note that the self-attention mechanism will technically operate on the sum of the learned embedding plus the positional information and not solely the learned embedding itself._\npython\n\n```\nimport matplotlib.pyplot as plt\n\n# Create word embeddings\nxs = [0.5, 1.5, 2.5, 6.0, 7.5, 8.0]\nys = [3.0, 1.2, 0.5, 8.0, 7.5, 5.5]\nwords = ['money', 'deposit', 'withdraw', 'nature', 'river', 'water']\nbank = [[4.5, 4.5], [6.7, 6.5]]\n\n# Create figure\nfig, ax = plt.subplots(ncols=2, figsize=(8,4))\n\n# Add titles\nax[0].set_title('Learned Embedding for \"bank\"nwithout context')\nax[1].set_title('Contextual Embedding forn\"bank\" after self-attention')\n\n# Add trace on plot 2 to show the movement of \"bank\"\nax[1].scatter(bank[0][0], bank[0][1], c='blue', s=50, alpha=0.3)\nax[1].plot([bank[0][0]+0.1, bank[1][0]],\n [bank[0][1]+0.1, bank[1][1]],\n linestyle='dashed',\n zorder=-1)\n\nfor i in range(2):\n ax[i].set_xlim(0,10)\n ax[i].set_ylim(0,10)\n\n # Plot word embeddings\n for (x, y, word) in list(zip(xs, ys, words)):\n ax[i].scatter(x, y, c='red', s=50)\n ax[i].text(x+0.5, y, word)\n\n # Plot \"bank\" vector\n x = bank[i][0]\n y = bank[i][1]\n\n color = 'blue' if i == 0 else 'purple'\n\n ax[i].text(x+0.5, y, 'bank')\n ax[i].scatter(x, y, c=color, s=50)\n```\n\n![A visualisation of the vector representation for the word "bank" moving through the embedding space following the addition of contextual information. Image by author.](../images/004.png)\nA visualisation of the vector representation for the word \"bank\" moving through the embedding space following the addition of contextual information. Image by author.\n### 3.3 - The Self-Attention Algorithm\nIn the section above, we stated that the goal of self-attention is to move the embedding for each token to a region of vector space that better represents the context of its use in the input sequence. What we didn't discuss is how this is done. Here we will show a step-by-step example of how the self-attention mechanism modifies the embedding for `bank`, by adding context from the surrounding tokens.\n![A simplified overview of a self-attention block \\(with the key, query, and value matrices excluded\\). Image by author.](../images/005.png)\nA simplified overview of a self-attention block (with the key, query, and value matrices excluded). Image by author.\n**Step 1) Calculate the Similarity Between Words using the Dot Product:**\nThe context of a token is given by the surrounding tokens in the sentence. Therefore, we can use the embeddings of all the tokens in the input sequence to update the embedding for any word, such as `bank`. Ideally, words that provide significant context (such as `river`) will heavily influence the embedding, while words that provide less context (such as `a`) will have minimal effect.\nThe degree of context one word contributes to another is measured by a similarity score. Tokens with similar learned embeddings are likely to provide more context than those with dissimilar embeddings. The similarity scores are calculated by taking the dot product of the current embedding for one token (its learned embedding plus positional information) with the current embeddings of every other token in the sequence. For clarity, the current embeddings have been termed s**elf-attention inputs** in this article and are denoted _x_.\nThere are several options for measuring the similarity between two vectors, which can be broadly categorised into: distance-based and angle-based metrics. Distance-based metrics characterise the similarity of vectors using the straight-line distance between them. This calculation is relatively simple and can be thought of as applying Pythagoras's theorem in _d _model_ -dimensional space. While intuitive, this approach is computationally expensive.\nFor angle-based similarity metrics, the two main candidates are: **cosine similarity** and **dot-product similarity**. Both of these characterise similarity using the cosine of the angle between the two vectors, θ. For orthogonal vectors (vectors that are at right angles to each other) _cos(θ)_ _= 0_ , which represents no similarity. For parallel vectors, _cos(θ) = 1_ , which represents that the vectors are identical. Solely using the angle between vectors, as is the case with cosine similarity, is not ideal for two reasons. The first is that the magnitude of the vectors is not considered, so distant vectors that happen to be aligned will produce inflated similarity scores. The second is that cosine similarity requires first computing the dot product and then dividing by the product of the vectors' magnitudes - making cosine similarity a computationally expensive metric. Therefore, the dot product is used to determine similarity. The dot product formula is given below for two vectors _x _1_ and _x _2_.\n![The dot product formula for two vectors x_1 and x_2. Image by author.](../images/006.png)\nThe dot product formula for two vectors x_1 and x_2. Image by author.\nThe diagram below shows the dot product between the self-attention input vector for `bank`, _x _bank,_ and the matrix of vector representations for every token in the input sequence, _X^T_. We can also write _x _bank_ as _x _11_ to reflect its position in the input sequence. The matrix _X_ stores the self-attention inputs for every token in the input sequence as rows. The number of columns in this matrix is given by _L _max_ , the maximum sequence length of the model. In this example, we will assume that the maximum sequence length is equal to the number of words in the input prompt, removing the need for any padding tokens (see [Part 4 in this series](https://medium.com/p/9f87602e4a11) for more about padding). To compute the dot product directly, we can transpose _X_ and calculate the vector of similarity scores, _S _bank_ using _S_bank = x _bank ⋅ X^T_. The individual elements of _S _bank_ represent the similarity scores between `bank` and each token in the input sequence.\n![An example calculation of the similarity scores for X_11 with every self-attention input \\(the sum of the learned embedding and positional information for each token in the input sequence\\). Image by author.](../images/007.png)\nAn example calculation of the similarity scores for X_11 with every self-attention input (the sum of the learned embedding and positional information for each token in the input sequence). Image by author.\n**Step 2) Scale the Similarity Scores:**\nThe dot product approach lacks any form of normalisation (unlike cosine similarity), which can cause the similarity scores to become very large. This can pose computational challenges, so normalisation of some form becomes necessary. The most common method is to divide each score by √_d _model_ , resulting in **scaled dot-product attention**. Scaled dot-product attention is not restricted to self-attention and is also used for cross-attention in transformers.\n**Step 3) Calculate the Attention Weights using the Softmax Function:**\nThe output of the previous step is the vector _S _bank_ , which contains the similarity scores between `bank` and every token in the input sequence. These similarity scores are used as weights to construct a transformer embedding for `bank` from the weighted sum of embeddings for each surrounding token in the prompt. The weights, known as **attention weights** , are calculated by passing _S _bank_ into the softmax function. The outputs are stored in a vector denoted _W _bank_. To see more about the softmax function, refer to the previous article on word2vec.\n![An example calculation of the attention weights for "bank" based on the similarity with every self-attention input. Image by author.](../images/008.png)\nAn example calculation of the attention weights for \"bank\" based on the similarity with every self-attention input. Image by author.\n**Step 4) Calculate the Transformer Embedding**\nFinally, the transformer embedding for `bank` is obtained by taking the weighted sum of `write`, `a`, `prompt`, ..., `bank`. Of course, `bank` will have the highest similarity score with itself (and therefore the largest attention weight), so the output embedding after this process will remain similar to before. This behaviour is ideal since the initial embedding already occupies a region of vector space that encodes some meaning for bank. The goal is to nudge the embedding towards the words that provide more context. The weights for words that provide little context, such as `a` and `man`, are very small. Hence, their influence on the output embedding will be minimal. Words that provide significant context, such as `river` and `fishing`, will have higher weights, and therefore pull the output embedding closer to their regions of vector space. The end result is a new embedding, _y _bank_ , that reflects the context of the entire input sequence.\n![An example calculation for the new embedding of "bank" by taking a weighted sum of the other embeddings for each token in the sequence. Image by author.](../images/009.png)\nAn example calculation for the new embedding of \"bank\" by taking a weighted sum of the other embeddings for each token in the sequence. Image by author.\n### 3.4 - Expanding Self-Attention using Matrices\nAbove, we walked through the steps to calculate the transformer embedding for the singular word `bank`. The input consisted of the learned embedding vector for bank plus its positional information, which we denoted _x _11_ or _x _bank_. The key point here, is that we considered only one vector as the input. If we instead pass in the matrix _X_ (with dimensions _L _max_ × _d _model_) to the self-attention block, we can calculate the transformer embedding for every token in the input prompt simultaneously. The output matrix, _Y_ , contains the transformer embedding for every token along the rows of the matrix. This approach is what enables transformers to quickly process text.\n![A black box diagram of a self-attention block. The matrix of word vectors is represented by X for the input sequence, and Y for the output sequence. Image by author.](../images/010.png)\nA black box diagram of a self-attention block. The matrix of word vectors is represented by X for the input sequence, and Y for the output sequence. Image by author.\n### 3.5 - The Query, Key, and Value Matrices\nThe above description gives an overview of the core functionality of the self-attention block, but there is one more piece of the puzzle. The simple weighted sum above does not include any trainable parameters, but we can introduce some to the process. Without trainable parameters, the performance of the model may still be good, but by allowing the model to learn more intricate patterns and hidden features from the training data, we observe much stronger model performance.\nThe self-attention inputs are used three times to calculate the new embeddings, these include the _x _bank_ vector, the _X^T_ matrix in the dot product step, and the _X^T_ matrix in the weighted sum step. These three sites are the perfect candidates to introduce some weights, which are added in the form of matrices (shown in red). When pre-multiplied by their respective inputs (shown in blue), these form the **key** , **query** , and **value** matrices, _K_ , _Q_ , and _V_ (shown in purple). The number of columns in these weight matrices is an architectural choice by the user. Choosing a value for _d _q_ , _d _k_ , and _d _v_ that is less than _d _model_ will result in dimensionality reduction, which can improve model speed. Ultimately, these values are hyperparameters that can be changed based on the specific implementation of the model and the use-case, and are often all set equal to _d _model_ if unsure [5].\n![A diagram of a complete self-attention block including the key, query, and value matrices. Image by author.](../images/011.png)\nA diagram of a complete self-attention block including the key, query, and value matrices. Image by author.\n### 3.6 - The Database Analogy\nThe names for these matrices come from an analogy with databases, which is explained briefly below.\n**Query:**\n * A query in a database is what you are looking for when performing a search. For example, \"show me all the albums in the database that have sold more than 1,000,000 records\". In the self-attention block, we are essentially asking the same question, but phrased as \"show me the transformer embedding for this vector (e.g. _x _bank_)\". For the sake of this example, we have only considered a single vector, _x _bank_ , but recall that we can perform the self-attention step on as many vectors as we like by collecting them into a matrix. Therefore, we can just as easily pass in the matrix _X_ as the query, which changes the question to \"show me the transformer embedding for each vector in the input sequence\". This is what actually happens in Transformer-based models.\n\n\n**Key:**\n * The keys in the database are the attributes or columns that are being searched against. In the example given earlier, you can think of this as the \"Albums Sold\" column, which stores the information we are interested in. In self-attention, we are interested in the embeddings for every word in the input prompt, so we can compute a set of attention weights. Therefore, the key matrix is a collection of all the input embeddings.\n\n\n**Value:**\n * The values correspond to the actual data in the database, that is, the actual sale figures in our example (e.g. 2,300,000 copies). For self-attention, this is exactly the same as the input for the key matrix (and query matrix as we just discussed): a collection of all the input embeddings. Hence, the query, key, and value matrices all take in the matrix _X_ as the input.\n\n\n### 3.7 - A Note on Multi-Head Attention\n**Distributing Computation Across Multiple Heads:**\nThe \"Attention is All You Need\" paper extends standard self-attention into **Multi-Head Attention (MHA)** by dividing the attention mechanism into multiple **heads**. In standard self-attention, the model learns a single set of weight matrices (_W _Q_ , _W _K_ , and _W _V_) that transform the token embedding matrix _X_ into query, key, and value matrices (_Q_ , _K_ , and _V_). These matrices are then used to compute attention scores and update _X_ with contextual information as we have seen above.\nIn contrast, MHA splits the attention mechanism into _H_ independent heads, each learning its own smaller set of weight matrices. These weights are used to calculate a set of smaller, head-specific query, key, and value matrices (denoted _Q^h_ , _K^h_ , and _V^h_). Each head processes the input sequence independently, generating distinct attention outputs. These outputs are then concatenated (stacked on top of each other) and passed through a final linear layer to produce the updated _X_ matrix, shown as _Y_ in the diagram below, with rich contextual information.\nBy introducing multiple heads, MHA increases the number of learnable parameters in the attention process, enabling the model to capture more complex relationships within the data. Each head learns its own weight matrices, allowing them to focus on different aspects of the input such as long-range dependencies (relationships between distant words), short-range dependencies (relationships between nearby words), grammatical syntax, etc. The overall effect produces a model with a more nuanced understanding of the input sequence.\n![An overview of the Multi-Head Attention process. For an in-depth explanation of terms and each process step, see Section 2.8 in Part 5 of this series. Image by author.](../images/012.png)\nAn overview of the Multi-Head Attention process. For an in-depth explanation of terms and each process step, see Section 2.8 in Part 5 of this series. Image by author.\nThe paragraphs below focus on building a broad intuition for how this works and why this step is useful. However, if you are keen to dive into the implementation details for MHA, see Section 2.8 in the Part 5 - A Complete Guide to Mistral 7B with Code [Link coming soon!].\n**The Benefits of Using Multiple Heads:**\nThe core idea is to allow each head to learn different types of relationships between words in the input sequence, and to combine them to create deep text representations. For example, some heads might learn to capture long-term dependencies (relationships between words that are distant in the text), while others might focus on short-term dependencies (words that are close in text).\nIf the model is given the sentence `A man withdrew money from the bank then sat on the river bank`, the use of multi-head attention allows the model capture the short-term dependency between `money` and the first instance of `bank`, and a separate dependency between `river` and the second instance of `bank`. Thus, the two uses of the word `bank` would be correctly updated with different contextual information for their respective meanings.\n**Building Intuition for Multi-Head Attention:**\nTo deepen this intuition for the usefulness of multiple attention heads, consider words in a sentence that require a lot of context. For example, in the sentence `I ate some of Bob's chocolate cake`, the word `ate` should attend to `I`, `Bob's` and `cake` to gain full context. This is a rather simple example, but if you extend this concept to complex sequences spanning thousands of words, hopefully it seems reasonable that distributing the computational load across separate attention mechanisms will be beneficial.\n![An example of attention heads capturing different word dependencies in an input sequence. Image by author.](../images/body-4b78e1a5d8043124e0a9.png)\nAn example of attention heads capturing different word dependencies in an input sequence. Image by author.\n## 4 - Transformer Embeddings in Python\n### 4.1 - Extracting Learned Embeddings and Transformer Embeddings from Transformer Models\nPython has many options for working with transformer models, but none are perhaps as well-known as Hugging Face. Hugging Face provides a centralised resource hub for NLP researchers and developers alike, including tools such as:\n * `transformers`: The library at the core of Hugging Face, which provides an interface for using, training, and fine-tuning pre-trained transformer models.\n * `tokenizers`: A library for working with tokenizers for many kinds of transformers, either using pre-built tokenizer models or constructing brand new ones from scratch.\n * `datasets`: A collection of datasets to train models on a variety of tasks, not just restricted to NLP.\n * Model Hub: A large repository of cutting-edge models from published papers, community-developed models, and everything in between. These are made freely available and can be easily imported into Python via the `transformers` API.\n\n\nThe code cell below shows how the `transformers` library can be used to load a transformer-based model into Python, and how to extract both the learned embeddings for words (without context) and the transformer embeddings (with context). The remainder of this article will break down the steps shown in this cell and describe additional functionalities available when working with embeddings.\npython\n\n```\nimport torch\nfrom transformers import AutoModel, AutoTokenizer\n\ndef extract_le(sequence, tokenizer, model):\n \"\"\" Extract the learned embedding for each token in an input sequence.\n\n Tokenize an input sequence (string) to produce a tensor of token IDs.\n Return a tensor containing the learned embedding for each token in the\n input sequence.\n\n Args:\n sequence (str): The input sentence(s) to tokenize and extract\n embeddings from.\n tokenizer: The tokenizer used to produce tokens.\n model: The model to extract learned embeddings from.\n\n Returns:\n learned_embeddings (torch.tensor): A tensor containing tensors of\n learned embeddings for each token in the input sequence.\n \"\"\"\n token_dict = tokenizer(sequence, return_tensors='pt')\n token_ids = token_dict['input_ids']\n learned_embeddings = model.embeddings.word_embeddings(token_ids)[0]\n\n # Additional processing for display purposes\n learned_embeddings = learned_embeddings.tolist()\n learned_embeddings = [[round(i,2) for i in le] \n for le in learned_embeddings]\n\n return learned_embeddings\n\ndef extract_te(sequence, tokenizer, model):\n \"\"\" Extract the tranformer embedding for each token in an input sequence.\n\n Tokenize an input sequence (string) to produce a tensor of token IDs.\n Return a tensor containing the transformer embedding for each token in the\n input sequence.\n\n Args:\n sequence (str): The input sentence(s) to tokenize and extract\n embeddings from.\n tokenizer: The tokenizer used to produce tokens.\n model: The model to extract learned embeddings from.\n\n Returns:\n transformer_embeddings (torch.tensor): A tensor containing tensors of\n transformer embeddings for each token in the input sequence.\n \"\"\"\n token_dict = tokenizer(sequence, return_tensors='pt')\n\n with torch.no_grad():\n base_model_output = model(**token_dict)\n\n transformer_embeddings = base_model_output.last_hidden_state[0]\n\n # Additional processing for display purposes\n transformer_embeddings = transformer_embeddings.tolist()\n transformer_embeddings = [[round(i,2) for i in te] \n for te in transformer_embeddings]\n\n return transformer_embeddings\n\n# Instantiate DistilBERT tokenizer and model\ntokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')\nmodel = AutoModel.from_pretrained('distilbert-base-uncased')\n\n# Extract the learned embedding for bank from DistilBERT\nle_bank = extract_le('bank', tokenizer, model)[1]\n\n# Write sentences containing \"bank\" in two different contexts\ns1 = 'Write a poem about a man fishing on a river bank.'\ns2 = 'Write a poem about a man withdrawing money from a bank.'\n\n# Extract the transformer embedding for bank from DistilBERT in each sentence\ns1_te_bank = extract_te(s1, tokenizer, model)[11]\ns2_te_bank = extract_te(s2, tokenizer, model)[11]\n\n# Print the results\nprint('------------------- Embedding vectors for \"bank\" -------------------n')\nprint(f'Learned embedding: {le_bank[:5]}')\nprint(f'Transformer embedding (sentence 1): {s1_te_bank[:5]}')\nprint(f'Transformer embedding (sentence 2): {s2_te_bank[:5]}')\n```\n\nshell\n\n```\n------------------- Embedding vectors for \"bank\" -------------------\n\nLearned embedding: [-0.03, -0.06, -0.09, -0.07, -0.03]\nTransformer embedding (sentence 1): [0.15, -0.16, -0.17, -0.08, 0.44]\nTransformer embedding (sentence 2): [0.27, -0.23, -0.23, -0.21, 0.79]\n```\n\n### 4.2 - Import the `Transformers` Library\nThe first step to produce transformer embeddings is to choose a model from the Hugging Face `transformers` library. In this article, we will not use the model for inference but solely to examine the embeddings it produces. This is not a standard use-case, and so we will have to do some extra digging in order to access the embeddings. Since the `transformers` library is written in PyTorch (referred to as `torch` in the code), we can import `torch` to extract data from the inner workings of the models.\n### 4.3 - Choose a Model\nFor this example, we will use DistilBERT, a smaller version of Google's BERT model which was released by Hugging Face themselves in October 2019 [6]. According to the Hugging Face documentation [7]:\n> _DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than_` _bert-base-uncased_`_, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark_.\nWe can import DistilBERT and its corresponding tokenizer into Python either directly from the `transformers` library or using the `AutoModel` and `AutoTokenizer` classes. There is very little difference between the two, although `AutoModel` and `AutoTokenizer` are often preferred since the model name can be parameterised and stored in a string, which makes it simpler to change the model being used.\npython\n\n```\nimport torch\nfrom transformers import DistilBertTokenizerFast, DistilBertModel\n\n# Instantiate DistilBERT tokenizer and model\ntokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')\nmodel = DistilBertModel.from_pretrained('distilbert-base-uncased')\n```\n\npython\n\n```\nimport torch\nfrom transformers import AutoModel, AutoTokenizer\n\n# Instantiate DistilBERT tokenizer and model\ntokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')\nmodel = AutoModel.from_pretrained('distilbert-base-uncased')\n```\n\nAfter importing DistilBERT and its corresponding tokenizer, we can call the `from_pretrained` method for each to load in the specific version of the DistilBERT model and tokenizer we want to use. In this case, we have chosen `distilbert-base-uncased`, where `base` refers to the size of the model, and `uncased` indicates that the model was trained on uncased text (all text is converted to lowercase).\n### 4.4 - Create Some Example Sentences\nNext, we can create some sentences to give the model some words to embed. The two sentences, `s1` and `s2`, both contain the word `bank` but in different contexts. The goal here is to show that the word `bank` will begin with the same learned embedding in both sentences, then be modified by DistilBERT using self-attention to produce a unique, contextualised embedding for each input sequence.\npython\n\n```\n# Create example sentences to produce embeddings for\ns1 = 'Write a poem about a man fishing on a river bank.'\ns2 = 'Write a poem about a man withdrawing money from a bank.'\n```\n\n### 4.5 - Tokenize an Input Sequence\nThe tokenizer class can be used to tokenize an input sequence (as shown below) and convert a string into a list of token IDs. Optionally, we can also pass a `return_tensors` argument to format the token IDs as a PyTorch tensor (`return_tensors=pt`) or as TensorFlow constants (`return_tensors=tf`). Leaving this argument empty will return the token IDs in a Python list. The return value is a dictionary that contains `input_ids`: the list-like object containing token IDs, and `attention_mask` which we will ignore for now.\n> **Note:** BERT-based models include a `[CLS]` token at the beginning of each sequence, and a `[SEP]` token to distinguish between two bodies of text in the input. These are present due to the tasks that BERT was originally trained on and can largely be ignored here. For a discussion on BERT special tokens, model sizes, `cased` vs `uncased`, and the attention mask, see [Part 4 of this series](https://medium.com/p/9f87602e4a11).\npython\n\n```\ntoken_dict = tokenizer(s1, return_tensors='pt')\ntoken_ids = token_dict['input_ids'][0]\n```\n\n### 4.6 - Extract the Learned Embeddings from a Model\nEach transformer model provides access to its learned embeddings via the `embeddings.word_embeddings` method. This method accepts a token ID or collection of token IDs and returns the learned embedding(s) as a PyTorch tensor.\npython\n\n```\nlearned_embeddings = model.embeddings.word_embeddings(token_ids)\nlearned_embeddings\n```\n\nshell\n\n```\ntensor([[ 0.0390, -0.0123, -0.0208, ..., 0.0607, 0.0230, 0.0238],\n [-0.0300, -0.0070, -0.0247, ..., 0.0203, -0.0566, -0.0264],\n [ 0.0062, 0.0100, 0.0071, ..., -0.0043, -0.0132, 0.0166],\n ...,\n [-0.0261, -0.0571, -0.0934, ..., -0.0351, -0.0396, -0.0389],\n [-0.0244, -0.0138, -0.0078, ..., 0.0069, 0.0057, -0.0016],\n [-0.0199, -0.0095, -0.0099, ..., -0.0235, 0.0071, -0.0071]],\n grad_fn=)\n```\n\n### 4.7 - Extract the Transformer Embeddings from a Model\nConverting a context-lacking learned embedding into a context-aware transformer embedding requires a forward pass of the model. Since we are not updating the weights of the model here (i.e. training the model), we can use the `torch.no_grad()` context manager to save on memory. This allows us to pass the tokens directly into the model and compute the transformer embeddings without any unnecessary calculations. Once the tokens have been passed into the model, a `BaseModelOutput` is returned, which contains various information about the forward pass. The only data that is of interest here is the activations in the last hidden state, which form the transformer embeddings. These can be accessed using the `last_hidden_state` attribute, as shown below, which concludes the explanation for the code cell shown at the top of this section.\npython\n\n```\nwith torch.no_grad():\n base_model_output = model(**token_dict)\n\ntransformer_embeddings = base_model_output.last_hidden_state\ntransformer_embeddings\n```\n\nshell\n\n```\ntensor([[[-0.0957, -0.2030, -0.5024, ..., 0.0490, 0.3114, 0.1348],\n [ 0.4535, 0.5324, -0.2670, ..., 0.0583, 0.2880, -0.4577],\n [-0.1893, 0.1717, -0.4159, ..., -0.2230, -0.2225, 0.0207],\n ...,\n [ 0.1536, -0.1616, -0.1735, ..., -0.3608, -0.3879, -0.1812],\n [-0.0182, -0.4264, -0.6702, ..., 0.3213, 0.5881, -0.5163],\n [ 0.7911, 0.2633, -0.4892, ..., -0.2303, -0.6364, -0.3311]]])\n```\n\n### 4.8 - Convert Token IDs to Tokens\nIt is possible to convert token IDs back into textual tokens, which shows exactly how the tokenizer divided the input sequence. This is useful when longer or rarer words are divided into multiple subwords when using subword tokenizers such as WordPiece (e.g. in BERT-based models) or Byte-Pair Encoding (e.g. in the GPT family of models).\npython\n\n```\ntokens = tokenizer.convert_ids_to_tokens(token_ids)\ntokens\n```\n\nshell\n\n```\n['[CLS]', 'write', 'a', 'poem', 'about', 'a', 'man', 'fishing', 'on', 'a',\n 'river', 'bank', '.', '[SEP]']\n```\n\n## 5 - Conclusion\nThe self-attention mechanism generates rich, context-aware transformer embeddings for text by processing each token in an input sequence simultaneously. These embeddings build on the foundations of static word embeddings (such as word2vec) and enable more capable language models such as BERT and GPT. Further work in this field will continue to improve the capabilities of LLMs and NLP as a whole.\n## 6 - Further Reading\n[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, [Attention is All You Need](https://arxiv.org/pdf/1706.03762) (2017), Advances in Neural Information Processing Systems 30 (NIPS 2017)\n[2] Hugging Face, [Transformers](https://huggingface.co/docs/transformers/index) (2024), HuggingFace.co\n[3] OpenAI, [ChatGPT Pricing](https://openai.com/chatgpt/pricing/) (2024), OpenAI.com\n[4] A. Gu and T. Dao, [Mamba: Linear-Time Sequence Modelling with Selective State Spaces](https://arxiv.org/abs/2312.00752) (2023), ArXiv abs/2312.00752\n[5] J. Alammar, [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) (2018). GitHub\n[6] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/pdf/1910.01108) (2019), 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019\n[7] Hugging Face, [DistilBERT Documentation](https://huggingface.co/docs/transformers/en/model_doc/distilbert) (2024) HuggingFace.co\n[8] Hugging Face, [BERT Documentation](https://huggingface.co/docs/transformers/model_doc/bert) (2024) HuggingFace.co\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](../images/body-8a241db34a497a6bbd1d.jpeg)\nWritten By\n## [Bradney Smith](https://towardsdatascience.com/author/bradneysmith/)\n[See all from Bradney Smith](https://towardsdatascience.com/author/bradneysmith/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[NLP](https://towardsdatascience.com/tag/nlp/)[Self Attention](https://towardsdatascience.com/tag/self-attention/)[Transformers](https://towardsdatascience.com/tag/transformers/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fcontextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e%2F&title=Self-Attention%20Explained%20with%20Code)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fcontextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e%2F&title=Self-Attention%20Explained%20with%20Code)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fcontextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e%2F&text=Self-Attention%20Explained%20with%20Code)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](../images/013.jpg)](https://towardsdatascience.com/graphrag-in-practice-how-to-build-cost-efficient-high-recall-retrieval-systems/)\n## [GraphRAG in Practice: How to Build Cost-Efficient, High-Recall Retrieval Systems](https://towardsdatascience.com/graphrag-in-practice-how-to-build-cost-efficient-high-recall-retrieval-systems/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nSmarter retrieval strategies that outperform dense graphs — with hybrid pipelines and lower cost\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nDecember 9, 202511 min read\n * [![](../images/body-63beb50f664a055ed739.jpg)](https://towardsdatascience.com/latest-picks-improving-named-entity-disambiguation-using-entity-relatedness-within-wikipedia-a7970d4ce685/)\n## [Latest picks: Improving Named Entity Disambiguation using Entity Relatedness within Wikipedia](https://towardsdatascience.com/latest-picks-improving-named-entity-disambiguation-using-entity-relatedness-within-wikipedia-a7970d4ce685/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nYour daily dose of data science\n[TDS Editors](https://towardsdatascience.com/author/towardsdatascience/)\nJanuary 5, 20211 min read\n * [![](../images/015.jpeg)](https://towardsdatascience.com/how-to-perform-ordinal-regression-classification-in-pytorch-361a2a095a99/)\n## [How to Perform Ordinal Regression / Classification in PyTorch](https://towardsdatascience.com/how-to-perform-ordinal-regression-classification-in-pytorch-361a2a095a99/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA simple trick for improving model performance when labels are ordered.\n[Mathias Gruber](https://towardsdatascience.com/author/nanomathias/)\nJuly 10, 20217 min read\n * [![](../images/016.jpg)](https://towardsdatascience.com/debunking-data-science-myths-83d693fb5ba4/)\n## [Debunking Data Science Myths](https://towardsdatascience.com/debunking-data-science-myths-83d693fb5ba4/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nThe Truth They Don't Tell You About Data Science\n[Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/)\nDecember 8, 20204 min read\n * [![](../images/017.jpg)](https://towardsdatascience.com/how-to-extract-and-convert-tables-from-pdf-files-to-pandas-dataframe-cb2e4c596fa8/)\n## [How to Extract and Convert Tables From PDF Files to Pandas Dataframe?](https://towardsdatascience.com/how-to-extract-and-convert-tables-from-pdf-files-to-pandas-dataframe-cb2e4c596fa8/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nSo you have some pdf files with tables in them and want to read them into a pandas data frame. Let me show you how.\n[Saed Hussain](https://towardsdatascience.com/author/saedhussain/)\nApril 19, 20223 min read\n * [![](../images/018.jpg)](https://towardsdatascience.com/develop-your-own-calendar-to-track-important-dates-with-python-c1af9e98ffc3/)\n## [Develop Your Own Calendar To Track Important Dates with Python](https://towardsdatascience.com/develop-your-own-calendar-to-track-important-dates-with-python-c1af9e98ffc3/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDeveloping a calendar GUI interface to regulate your plans for 2022 and beyond\n[Bharath K](https://towardsdatascience.com/author/bharath-k1297/)\nJanuary 13, 20227 min read\n * [![](../images/019.jpg)](https://towardsdatascience.com/use-pytorch-to-easily-access-your-gpu/)\n## [Use PyTorch to Easily Access Your GPU](https://towardsdatascience.com/use-pytorch-to-easily-access-your-gpu/)\n[Programming](https://towardsdatascience.com/category/programming/)\nOr… how an ML library can accelerate non-ML computations\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nMay 21, 202511 min read\n * [![](../images/020.jpg)](https://towardsdatascience.com/an-overview-of-forecasting-performance-metrics-ef548dad0134/)\n## [An Overview Of Forecasting Performance Metrics](https://towardsdatascience.com/an-overview-of-forecasting-performance-metrics-ef548dad0134/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn overview of some of the fundamental performance metrics for time series forecasting\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nNovember 16, 20225 min read\n * [![](../images/021.png)](https://towardsdatascience.com/an-intuitive-guide-to-track-your-ml-experiments-with-mlflow-7ac50e63b09/)\n## [An Intuitive Guide to Track Your ML Experiments with MLflow](https://towardsdatascience.com/an-intuitive-guide-to-track-your-ml-experiments-with-mlflow-7ac50e63b09/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHelping you on improving the reproducibility, management and tracking of experiments for your Sklearn model\n[Eugenia Anello](https://towardsdatascience.com/author/eugenia-anello/)\nFebruary 8, 20220 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500) Check Your Biases ](https://towardsdatascience.com/check-your-biases-ba2685a7f799/ \"Check Your Biases\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500) Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads ](https://towardsdatascience.com/machine-learning-tricks-learning-with-chatgpt-and-other-recent-must-reads-f277648a1921/ \"Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads\")\n\n\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1\n", "metadata": { "title": "Self-Attention Explained with Code | Towards Data Science", "description": "How large language models create rich, contextual embeddings", "keywords": null, "author": "Bradney Smith", "og:locale": "en_US", "og:type": "article", "og:title": "Self-Attention Explained with Code", "og:description": "How large language models create rich, contextual embeddings", "og:url": "https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1ZPePnPodMZeehez9YFmr9A.png", "og:image:width": "1458", "og:image:height": "1106", "og:image:type": "image/png", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Bradney Smith", "twitter:label2": "Est. reading time", "twitter:data2": "35 min read", "article:published_time": "2024-02-09T19:10:08.000Z", "article:modified_time": "2024-02-09T19:10:08.000Z", "article:author": "https://towardsdatascience.com/author/bradneysmith/" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "", "retried": true }, "code_blocks": [ { "language": "txt", "file": "attachments/34afc94764d0/code/block_000.txt", "chars": 3359 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_001.txt", "chars": 406 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_002.txt", "chars": 713 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_003.txt", "chars": 1106 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_004.txt", "chars": 3256 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_005.txt", "chars": 282 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_006.txt", "chars": 271 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_007.txt", "chars": 239 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_008.txt", "chars": 173 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_009.txt", "chars": 87 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_010.txt", "chars": 84 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_011.txt", "chars": 471 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_012.txt", "chars": 151 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_013.txt", "chars": 442 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_014.txt", "chars": 59 }, { "language": "txt", "file": "attachments/34afc94764d0/code/block_015.txt", "chars": 107 } ], "images": [ { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:47:33Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:47:33Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:47:33Z" }, { "status": "http_404", "url": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500", "src": "https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500", "alt": "", "downloaded": false, "discovery": "saved_markdown", "retried_at": "2026-10-20T17:47:33Z" } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/c/TowardsDataScience", "kind": "channel_reference", "status": "reference_only" } ], "links": [], "retry_at": "2026-10-03T08:04:09", "repair_version": 2, "repaired_at": "2026-10-20T17:26:15Z", "source_url": "https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/", "delivery_status": "unverified", "discovery_status": "http_403", "missing_reason": "源页面未重新核验:http_403；已按保留正文补采图片", "repair_error": "", "actual_video_files": [], "expected_saved_body_images": 4, "image_discovery": "saved_markdown", "source_body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Self-Attention Explained with Code\nHow large language models create rich, contextual embeddings\n[Bradney Smith](https://towardsdatascience.com/author/bradneysmith/)\nFebruary 9, 202435 min read\n### **Part 3 in the \"LLMs from Scratch\" series - a complete guide to understanding and building Large Language Models. If you are interested in learning more about how these models work I encourage you to read:**\n * [Prelude: A Brief History of Large Language Models](https://medium.com/p/98a1320e7650/)\n * [Part 1: Tokenization - A Complete Guide](https://medium.com/p/cedc9f72de4e)\n * [Part 2: Word Embeddings with word2vec from Scratch in Python](https://medium.com/p/eb9326c6ab7c)\n * **Part 3: Self-Attention Explained with Code**\n * [Part 4: A Complete Guide to BERT with Code](https://medium.com/p/9f87602e4a11/)\n\n\n## Introduction\nThe paper \"Attention is All You Need\" debuted perhaps the single largest advancement in Natural Language Processing (NLP) in the last 10 years: the Transformer [1]. This architecture massively simplified the complex designs of language models at the time while achieving unparalleled results. State-of-the-art (SOTA) models, such as those in the GPT, Claude, and Llama families, owe their success to this design, at the heart of which is self-attention. In this deep dive, we will explore how this mechanism works and how it is used by transformers to create contextually rich embeddings that enable these models to perform so well.\n## Contents\n**1** - [Overview of the Transformer Embedding Process](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#cc4f)\n**2** - [Positional Encoding](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#365f)\n**3** -[ The Self-Attention Mechanism](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#4489)\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) and [Machine Learning](https://roadmap.sh/machine-learning) roadmaps.\n**4** - [Transformer Embeddings in Python](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#cb34)\n**5** -[ Conclusion](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#126e)\n**6** - [Further Reading](https://towardsdatascience.com/contextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e/#776b)\n## 1 - Overview of the Transformer Embedding Process\n### 1.1 - Recap on Transformers\nIn the prelude article of this series, we briefly explored the history of the Transformer and its impact on NLP. To recap: the Transformer is a deep neural network architecture that is the foundation for almost all LLMs today. Derivative models are often called Transformer-based models or **transformers** for short, and so these terms will be used interchangeably here. Like all machine learning models, transformers work with numbers and linear algebra rather than processing human language directly. Because of this, they must convert textual inputs from users into numerical representations through several steps. Perhaps the most important of these steps is applying the self-attention mechanism, which is the focus of this article. The process of representing text with vectors is called **embedding** (or **encoding**), hence the numerical representations of the input text are known as **transformer embeddings**.\n### 1.2 - The Issue with Static Embeddings\nIn [Part 2 of this series](https://medium.com/p/eb9326c6ab7c), we explored static embeddings for language models using word2vec as an example. This embedding method predates transformers and suffers from one major drawback: the lack of contextual information. Words with multiple meanings (called **polysemous** words) are encoded with somewhat ambiguous representations since they lack the context needed for precise meaning. A classic example of a polysemous word is `bank`. Using a static embedding model, the word `bank` would be represented in vector space with some degree of similarity to words such as `money` and `deposit` and some degree of similarity to words such as `river` and `nature`. This is because the word will occur in many different contexts within the training data. This is the core problem with static embeddings: they do not change based on context - hence the term \"static\".\n### 1.3 - Fixing Static Embeddings\nTransformers overcome the limitations of static embeddings by producing their own context-aware transformer embeddings. In this approach, fixed word embeddings are augmented with positional information (where the words occur in the input text) and contextual information (how the words are used). These two steps take place in distinct components in transformers, namely the positional encoder and the self-attention blocks, respectively. We will look at each of these in detail in the following sections. By incorporating this additional information, transformers can produce much more powerful vector representations of words based on their usage in the input sequence. Extending the vector representations beyond static embeddings is what enables Transformer-based models to handle polysemous words and gain a deeper understanding of language compared to previous models.\n### 1.4 - Introducing Learned Embeddings\nMuch like the word2vec approach released four years prior, transformers store the initial vector representation for each token in the weights of a linear layer (a small neural network). In the word2vec model, these representations form the static embeddings, but in the Transformer context these are known as **learned embeddings**. In practice they are very similar, but using a different name emphasises that these representations are only a starting point for the transformer embeddings and not the final form.\nThe linear layer sits at the beginning of the Transformer architecture and contains only weights and no bias terms (bias = 0 for every neuron). The layer weights can be represented as a matrix of size _V_ × _d _model_ , where _V_ is the vocabulary size (the number of unique words in the training data) and _d _model_ is the number of embedding dimensions. In the previous article, we denoted _d _model_ as _N_ , in line with word2vec notation, but here we will use _d _model_ which is more common in the Transformer context. The original Transformer was proposed with a _d _model_ size of 512 dimensions, but in practice any reasonable value can be used.\n![A diagram showing the location of the linear layer in the Transformer architecture, which stores the learned embeddings. Image by author, adapted from the Transformer architecture diagram in the "Attention is All You Need" paper \\[1\\].](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1ZPePnPodMZeehez9YFmr9A.png)\nA diagram showing the location of the linear layer in the Transformer architecture, which stores the learned embeddings. Image by author, adapted from the Transformer architecture diagram in the \"Attention is All You Need\" paper [1].\n### **1.5 - Creating Learned Embeddings**\nA key difference between static and learned embeddings is the way in which they are trained. Static embeddings are trained in a separate neural network (using the Skip-Gram or Continuous Bag of Words architectures) using a word prediction task within a given window size. Once trained, the embeddings are then extracted and used with a range of different language models. Learned embeddings, however, are integral to the transformer you are using and are stored as weights in the first linear layer of the model. These weights, and consequently the learned embedding for each token in the vocabulary, are trained in the same backpropagation steps as the rest of the model parameters. Below is a summary of the training process for learned embeddings.\n**Step 1: Initialisation**\nRandomly initialise the weights for each neuron in the linear layer at the beginning of the model, and set the bias terms to 0. This layer is also called the **embedding layer,** since it is the linear layer that will store the learned embeddings. The weights can be represented as a matrix of size _V_ × _d _model_ , where the word embedding for each word in the vocabulary is stored along the rows. For example, the embedding for the first word in the vocabulary is stored in the first row, the second word is stored in the second row, and so on.\n**Step 2: Training**\nAt each training step, the Transformer receives an input word and the aim is to predict the next word in the sequence - a task known as Next Token Prediction (NTP). Initially, these predictions will be very poor, and so every weight and bias term in the network will be updated to improve performance against the loss function, including the embeddings. After many training iterations, the learned embeddings should provide a strong vector representation for each word in the vocabulary.\n**Step 3: Extract the Learned Embeddings**\nWhen new input sequences are given to the model, the words are converted into tokens with an associated token ID, which corresponds to the position of the token in the tokenizer's vocabulary. For example, the word `cat` may lie at position `349` in the tokenizer's vocabulary and so will take the ID `349`. Token IDs are used to create one-hot encoded vectors that extract the correct learned embeddings from the weights matrix (that is, _V_ -dimensional vectors where every element is 0 except for the element at the token ID position, which is 1).\n> _**Note:**__PyTorch is a very popular deep learning library in Python that powers some of the most well-known machine learning packages, such as the HuggingFace_` _Transformers_`_library [2]. If you are familiar with PyTorch, you may have encountered the_` _nn.Embedding_`_class, which is often used to form the first layer of transformer networks (the_` _nn_`_denotes that the class belongs to the neural network package). This class returns a regular linear layer that is initialised with the identity function as the activation function and with no bias term. The weights are randomly initialised since they are parameters to be learned by the model during training. This essentially carries out the steps described above in one simple line of code. Remember, the_` _nn.Embeddings_`_layer does not provide pre-trained word embeddings out-of-the-box, but rather initialises a blank canvas of embeddings before training. This is to allow the transformer to learn its own embeddings during the training phase._\n### 1.6 - Transformer Embedding Process\nOnce the learned embeddings have been trained, the weights in the embedding layer never change. That is, the learned embedding for each word (or more specifically, token) always provides the same starting point for a word's vector representation. From here, the positional and contextual information will be added to produce a unique representation of the word that is reflective of its usage in the input sequence.\nTransformer embeddings are created in a four-step process, which is demonstrated below using the example prompt: `Write a poem about a man fishing on a river bank.`. Note that the first two steps are the same as the word2vec approach we saw before. Steps 3 and 4 are the further processing that add contextual information to the embeddings.\n**Step 1) Tokenization:**\nTokenization is the process of dividing a longer input sequence into individual words (and parts of words) called tokens. In this case, the sentence will be broken down into:\n`write`, `a`, `poem`, `about`, `a`, `man`, `fishing`, `on`, `a`, `river`, `bank`\nNext, the tokens are associated with their token IDs, which are integer values corresponding to the position of the token in the tokenizer's vocabulary (see [Part 1 of this series](https://medium.com/p/cedc9f72de4e) for an in-depth look at the tokenization process).\n**Step 2) Map the Tokens to Learned Embeddings:**\nOnce the input sequence has been converted into a set of token IDs, the tokens are then mapped to their learned embedding vector representations, which were acquired during the transformer's training. These learned embeddings have the \"lookup table\" behaviour as we saw in the word2vec example in [Part 2 of this series](https://medium.com/p/eb9326c6ab7c)). The mapping takes place by multiplying a one-hot encoded vector created from the token ID with the weights matrix, just as in the word2vec approach. The learned embeddings are denoted _V_ in the image below.\n**Step 3) Add Positional Information with Positional Encoding:**\n**Positional Encoding** is then used to add positional information to the word embeddings. Whereas Recurrent Neural Networks (RNNs) process text sequentially (one word at a time), transformers process all words in parallel. This removes any implicit information about the position of each word in the sentence. For example, the sentences `the cat ate the mouse` and `the mouse ate the cat` use the same words but have very different meanings. To preserve the word order, positional encoding vectors are generated and added to the learned embedding for each word. In the image below, the positional encoding vectors are denoted _P_ , and the sums of the learned embeddings and positional encodings are denoted _X_.\n**Step 4) Modify the Embeddings using Self-Attention:**\nThe final step is to add contextual information using the self-attention mechanism. This determines which words give context to other words in the input sequence. In the image below, the transformer embeddings are denoted _y_.\n![An overview of the transformer embedding process from input text through to transformer embeddings. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1_VVYcevZ4CtY3rIj5EQMzg.png)\nAn overview of the transformer embedding process from input text through to transformer embeddings. Image by author.\n## 2 - Positional Encoding\n### 2.1 - The Need for Positional Encoding\nBefore the self-attention mechanism is applied, positional encoding is used to add information about the order of tokens to the learned embeddings. This compensates for the loss of positional information caused by the parallel processing used by transformers described earlier. There are many feasible approaches for injecting this information, but all methods must adhere to a set of constraints. The functions used to generate positional information must produce values that are:\n * **Bounded** - values should not explode in the positive or negative direction but be constrained (e.g. between 0 and 1, -1 and 1, etc)\n * **Periodic** - the function should produce a repeating pattern that the model can learn to recognise and discern position from\n * **Predictable** - positional information should be generated in such a way that the model can understand the position of words in sequence lengths it was not trained on. For example, even if the model has not seen a sequence length of exactly 412 tokens in its training, the transformer should be able to understand the position of each of the embeddings in the sequence.\n\n\nThese constraints ensure that the positional encoder produces positional information that allows words to **attend** to (gain context from) any other important word, regardless of their relative positions in the sequence. In theory, with a sufficiently powerful computer, words should be able to gain context from every relevant word in an infinitely long input sequence. The length of a sequence from which a model can derive context is called the **context length**. In chatbots like ChatGPT, the context includes the current prompt as well as all previous prompts and responses in the conversation (within the context length limit). This limit is typically in the range of a few thousand tokens, with GPT-3 supporting up to 4096 tokens and GPT-4 enterprise edition capping at around 128,000 tokens [3].\n### 2.2 - Positional Encoding in \"Attention is All You Need\"\nThe original transformer model was proposed with the following positional encoding functions:\n![An image of the equations for positional encoding, as proposed in the paper "Attention is All You Need" \\[1\\]. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1xzeVzfdB7_MtQnZb7wv3ng.png)\nAn image of the equations for positional encoding, as proposed in the paper \"Attention is All You Need\" [1]. Image by author.\nwhere:\n * _pos_ is the position of the word in the input, where _pos = 0_ corresponds to the first word in the sequence\n * _i_ is the index of each embedding dimension, ranging from _i=0_ (for the first embedding dimension) up to _d _model_\n * _d _model_ is the number of embedding dimensions for each learned embedding vector (and therefore each positional encoding vector). This was previously denoted _N_ in the article on word2vec.\n\n\nThe two proposed functions take arguments of _2i_ and _2i+1_ , which in practice means that the sine function generates positional information for the even-numbered dimensions of each word vector (_i_ is even), and the cosine function does so for the odd-numbered dimensions (_i_ is odd). According to the authors of the transformer:\n> _\"The positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000·2π. We chose this function because we hypothesised it would allow the model to easily learn to attend by relative positions, since for any fixed offset_ k _,_ PE_pos+k _can be represented as a linear function of_ PE _pos_ \"._\nThe value of the constant in the denominator being `10_000` was found to be suitable after some experimentation, but is a somewhat arbitrary choice by the authors.\n### 2.3 - Other Positional Encoding Approaches\nThe positional encodings shown above are considered **fixed** because they are generated by a known function with deterministic (predictable) outputs. This represents the most simple form of positional encoding. It is also possible to use **learned positional encodings** by randomly initialising some positional encodings and training them with backpropagation. Derivatives of the BERT architecture are examples of models that take this learned encoding approach. More recently, the Rotary Positional Encoding (RoPE) method has gained popularity, finding use in models such as Llama 2 and PaLM, among other positional encoding methods.\n### 2.4 - Implementing a Positional Encoder in Python\nCreating a positional encoder class in Python is fairly straightforward. We can start by defining a function that accepts the number of embedding dimensions (`d_model`), the maximum length of the input sequence (`max_length`), and the number of decimal places to round each value in the vectors to (`rounding`). Note that transformers define a maximum input sequence length, and any sequence that has fewer tokens than this limit is appended with padding tokens until the limit is reached. To account for this behaviour in our positional encoder, we accept a `max_length` argument. In practice, this limit is typically thousands of characters long.\nWe can also exploit a mathematical trick to save computation. Instead of calculating the denominator for both _PE_{pos, 2i}_ and _PE_{pos, 2i}_, we can note that the denominator is identical for consecutive pairs of _i_. For example, the denominators for _i=0_ and _i=1_ are the same, as are the denominators for _i=2_ and _i=3_. Hence, we can perform the calculations to determine the denominators once for the even values of _i_ and reuse them for the odd values of _i_.\npython\n\n```\nimport numpy as np\n\nclass PositionalEncoder():\n \"\"\" An implementation of positional encoding.\n\n Attributes:\n d_model (int): The number of embedding dimensions in the learned\n embeddings. This is used to determine the length of the positional\n encoding vectors, which make up the rows of the positional encoding\n matrix.\n max_length (int): The maximum sequence length in the transformer. This\n is used to determine the size of the positional encoding matrix.\n rounding (int): The number of decimal places to round each of the\n values to in the output positional encoding matrix.\n \"\"\"\n\n def __init__(self, d_model, max_length, rounding):\n self.d_model = d_model\n self.max_length = max_length\n self.rounding = rounding\n\n def generate_positional_encoding(self):\n \"\"\" Generate positional information to add to inputs for encoding.\n\n The positional information is generated using the number of embedding\n dimensions (d_model), the maximum length of the sequence (max_length),\n and the number of decimal places to round to (rounding). The output\n matrix generated is of size (max_length X embedding_dim), where each\n row is the positional information to be added to the learned\n embeddings, and each column is an embedding dimension.\n \"\"\"\n\n position = np.arange(0, self.max_length).reshape(self.max_length, 1)\n even_i = np.arange(0, self.d_model, 2)\n denominator = 10_000**(even_i / self.d_model)\n\n even_encoded = np.round(np.sin(position / denominator), self.rounding)\n odd_encoded = np.round(np.cos(position / denominator), self.rounding)\n\n # Interleave the even and odd encodings\n positional_encoding = np.stack((even_encoded, odd_encoded),2)\n .reshape(even_encoded.shape[0],-1)\n\n # If self.d_model is odd remove the extra column generated\n if self.d_model % 2 == 1:\n positional_encoding = np.delete(positional_encoding, -1, axis=1)\n\n return positional_encoding\n\n def encode(self, input):\n \"\"\" Encode the input by adding positional information.\n\n Args:\n input (np.array): A two-dimensional array of embeddings. The array\n should be of size (self.max_length x self.d_model).\n\n Returns:\n output (np.array): A two-dimensional array of embeddings plus the\n positional information. The array has size (self.max_length x\n self.d_model).\n \"\"\"\n positional_encoding = self.generate_positional_encoding()\n output = input + positional_encoding\n\n return output\n\nMAX_LENGTH = 5\nEMBEDDING_DIM = 3\nROUNDING = 2\n\n# Instantiate the encoder\nPE = PositionalEncoder(d_model=EMBEDDING_DIM,\n max_length=MAX_LENGTH,\n rounding=ROUNDING)\n\n# Create an input matrix of word embeddings without positional encoding\ninput = np.round(np.random.rand(MAX_LENGTH, EMBEDDING_DIM), ROUNDING)\n\n# Create an output matrix of word embeddings by adding positional encoding\noutput = PE.encode(input)\n\n# Print the results\nprint(f'Embeddings without positional encoding:nn{input}n')\nprint(f'Positional encoding:nn{output-input}n')\nprint(f'Embeddings with positional encoding:nn{output}')\n```\n\nshell\n\n```\nEmbeddings without positional encoding:\n\n[[0.12 0.94 0.9 ]\n [0.14 0.65 0.22]\n [0.29 0.58 0.31]\n [0.69 0.37 0.62]\n [0.25 0.61 0.65]]\n\nPositional encoding:\n\n[[ 0. 1. 0. ]\n [ 0.84 0.54 0. ]\n [ 0.91 -0.42 0. ]\n [ 0.14 -0.99 0.01]\n [-0.76 -0.65 0.01]]\n\nEmbeddings with positional encoding:\n\n[[ 0.12 1.94 0.9 ]\n [ 0.98 1.19 0.22]\n [ 1.2 0.16 0.31]\n [ 0.83 -0.62 0.63]\n [-0.51 -0.04 0.66]]\n```\n\n### 2.5 - Visualising the Positional Encoding Matrix\nRecall that the positional information generated must be bounded, periodic, and predictable. The outputs of the sinusoidal functions presented earlier can be collected into a matrix, which can then be easily combined with the learned embeddings using element-wise addition. Plotting this matrix gives a nice visualisation of the desired properties. In the plot below, curving bands of negative values (blue) emanate from the left edge of the matrix. These bands form a pattern that the transformer can easily learn to predict.\npython\n\n```\nimport matplotlib.pyplot as plt\n\n# Instantiate a PositionalEncoder class\nd_model = 400\nmax_length = 100\nrounding = 4\n\nPE = PositionalEncoder(d_model=d_model,\n max_length=max_length,\n rounding=rounding)\n\n# Generate positional encodings\ninput = np.round(np.random.rand(max_length, d_model), 4)\npositional_encoding = PE.generate_positional_encoding()\n\n# Plot positional encodings\ncax = plt.matshow(positional_encoding, cmap='coolwarm')\nplt.title(f'Positional Encoding Matrix ({d_model=}, {max_length=})')\nplt.ylabel('Position of the Embeddingnin the Sequence, pos')\nplt.xlabel('Embedding Dimension, i')\nplt.gcf().colorbar(cax)\nplt.gca().xaxis.set_ticks_position('bottom')\n```\n\n![A visualisation of the positional encoding matrix for a model with 400 embedding dimensions \\(d_model = 400\\), and a maximum sequence length of 100 \\(max_length = 100\\). Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1-BW3AxwVXk3ZNV80TqTn3A.png)\nA visualisation of the positional encoding matrix for a model with 400 embedding dimensions (d_model = 400), and a maximum sequence length of 100 (max_length = 100). Image by author.\n## 3 - The Self-Attention Mechanism\n### 3.1 - Overview of Attention Mechanisms\nNow that we have covered an overview of transformer embeddings and the positional encoding step, we can turn our focus to the self-attention mechanism itself. In short, self-attention modifies the vector representation of words to capture the context of their usage in an input sequence. The \"self\" in self-attention refers to the fact that the mechanism uses the surrounding words within a single sequence to provide context. As such, self-attention requires all words to be processed in parallel. This is actually one of the main benefits of transformers (especially compared to RNNs) since the models can leverage parallel processing for a significant performance boost. In recent times, there has been some rethinking around this approach, and in the future we may see this core mechanism being replaced [4].\nAnother form of attention used in transformers is cross-attention. Unlike self-attention, which operates within a single sequence, cross-attention compares each word in an output sequence to each word in an input sequence, crossing between the two embedding matrices. Note the difference here compared to self-attention, which focuses entirely within a single sequence.\n### 3.2 - Visualising How Self-Attention Contextualises Embeddings\nThe plots below show a simplified set of learned embedding vectors in two dimensions. Words associated with nature and rivers are concentrated in the top right quadrant of the graph, while words associated with money are concentrated in the bottom left. The vector representing the word `bank` is positioned between the two clusters due to its polysemic nature. The objective of self-attention is to move the learned embedding vectors to regions of vector space that more accurately capture their meaning within the context of the input sequence. In the example input `Write a poem about a man fishing on a river bank.`, the aim is to move the vector for `bank` in such a way that captures more of the meaning of nature and rivers, and less of the meaning of money and deposits.\n> _**Note:**__More accurately, the goal of self-attention here is to update the vector for every word in the input, so that all embeddings better represent the context in which they were used. There is nothing special about the word_` _bank_`_here that transformers have some special knowledge of - self-attention is applied across all the words. We will look more at this shortly, but for now, considering solely how_` _bank_`_is affected by self-attention gives a good intuition for what is happening in the attention block. For the purpose of this visualisation, the positional encoding information has not been explicitly shown. The effect of this will be minimal, but note that the self-attention mechanism will technically operate on the sum of the learned embedding plus the positional information and not solely the learned embedding itself._\npython\n\n```\nimport matplotlib.pyplot as plt\n\n# Create word embeddings\nxs = [0.5, 1.5, 2.5, 6.0, 7.5, 8.0]\nys = [3.0, 1.2, 0.5, 8.0, 7.5, 5.5]\nwords = ['money', 'deposit', 'withdraw', 'nature', 'river', 'water']\nbank = [[4.5, 4.5], [6.7, 6.5]]\n\n# Create figure\nfig, ax = plt.subplots(ncols=2, figsize=(8,4))\n\n# Add titles\nax[0].set_title('Learned Embedding for \"bank\"nwithout context')\nax[1].set_title('Contextual Embedding forn\"bank\" after self-attention')\n\n# Add trace on plot 2 to show the movement of \"bank\"\nax[1].scatter(bank[0][0], bank[0][1], c='blue', s=50, alpha=0.3)\nax[1].plot([bank[0][0]+0.1, bank[1][0]],\n [bank[0][1]+0.1, bank[1][1]],\n linestyle='dashed',\n zorder=-1)\n\nfor i in range(2):\n ax[i].set_xlim(0,10)\n ax[i].set_ylim(0,10)\n\n # Plot word embeddings\n for (x, y, word) in list(zip(xs, ys, words)):\n ax[i].scatter(x, y, c='red', s=50)\n ax[i].text(x+0.5, y, word)\n\n # Plot \"bank\" vector\n x = bank[i][0]\n y = bank[i][1]\n\n color = 'blue' if i == 0 else 'purple'\n\n ax[i].text(x+0.5, y, 'bank')\n ax[i].scatter(x, y, c=color, s=50)\n```\n\n![A visualisation of the vector representation for the word "bank" moving through the embedding space following the addition of contextual information. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1n_t-kLuJpHy9WrnSA46kPw.png)\nA visualisation of the vector representation for the word \"bank\" moving through the embedding space following the addition of contextual information. Image by author.\n### 3.3 - The Self-Attention Algorithm\nIn the section above, we stated that the goal of self-attention is to move the embedding for each token to a region of vector space that better represents the context of its use in the input sequence. What we didn't discuss is how this is done. Here we will show a step-by-step example of how the self-attention mechanism modifies the embedding for `bank`, by adding context from the surrounding tokens.\n![A simplified overview of a self-attention block \\(with the key, query, and value matrices excluded\\). Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1Ghl4oQSB604tgS4R97mMLw.png)\nA simplified overview of a self-attention block (with the key, query, and value matrices excluded). Image by author.\n**Step 1) Calculate the Similarity Between Words using the Dot Product:**\nThe context of a token is given by the surrounding tokens in the sentence. Therefore, we can use the embeddings of all the tokens in the input sequence to update the embedding for any word, such as `bank`. Ideally, words that provide significant context (such as `river`) will heavily influence the embedding, while words that provide less context (such as `a`) will have minimal effect.\nThe degree of context one word contributes to another is measured by a similarity score. Tokens with similar learned embeddings are likely to provide more context than those with dissimilar embeddings. The similarity scores are calculated by taking the dot product of the current embedding for one token (its learned embedding plus positional information) with the current embeddings of every other token in the sequence. For clarity, the current embeddings have been termed s**elf-attention inputs** in this article and are denoted _x_.\nThere are several options for measuring the similarity between two vectors, which can be broadly categorised into: distance-based and angle-based metrics. Distance-based metrics characterise the similarity of vectors using the straight-line distance between them. This calculation is relatively simple and can be thought of as applying Pythagoras's theorem in _d _model_ -dimensional space. While intuitive, this approach is computationally expensive.\nFor angle-based similarity metrics, the two main candidates are: **cosine similarity** and **dot-product similarity**. Both of these characterise similarity using the cosine of the angle between the two vectors, θ. For orthogonal vectors (vectors that are at right angles to each other) _cos(θ)_ _= 0_ , which represents no similarity. For parallel vectors, _cos(θ) = 1_ , which represents that the vectors are identical. Solely using the angle between vectors, as is the case with cosine similarity, is not ideal for two reasons. The first is that the magnitude of the vectors is not considered, so distant vectors that happen to be aligned will produce inflated similarity scores. The second is that cosine similarity requires first computing the dot product and then dividing by the product of the vectors' magnitudes - making cosine similarity a computationally expensive metric. Therefore, the dot product is used to determine similarity. The dot product formula is given below for two vectors _x _1_ and _x _2_.\n![The dot product formula for two vectors x_1 and x_2. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1t_PaA8rj9RA3-TZ0cIpK9g.png)\nThe dot product formula for two vectors x_1 and x_2. Image by author.\nThe diagram below shows the dot product between the self-attention input vector for `bank`, _x _bank,_ and the matrix of vector representations for every token in the input sequence, _X^T_. We can also write _x _bank_ as _x _11_ to reflect its position in the input sequence. The matrix _X_ stores the self-attention inputs for every token in the input sequence as rows. The number of columns in this matrix is given by _L _max_ , the maximum sequence length of the model. In this example, we will assume that the maximum sequence length is equal to the number of words in the input prompt, removing the need for any padding tokens (see [Part 4 in this series](https://medium.com/p/9f87602e4a11) for more about padding). To compute the dot product directly, we can transpose _X_ and calculate the vector of similarity scores, _S _bank_ using _S_bank = x _bank ⋅ X^T_. The individual elements of _S _bank_ represent the similarity scores between `bank` and each token in the input sequence.\n![An example calculation of the similarity scores for X_11 with every self-attention input \\(the sum of the learned embedding and positional information for each token in the input sequence\\). Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/17edjXXmtsfFAWN-_OgCYaQ.png)\nAn example calculation of the similarity scores for X_11 with every self-attention input (the sum of the learned embedding and positional information for each token in the input sequence). Image by author.\n**Step 2) Scale the Similarity Scores:**\nThe dot product approach lacks any form of normalisation (unlike cosine similarity), which can cause the similarity scores to become very large. This can pose computational challenges, so normalisation of some form becomes necessary. The most common method is to divide each score by √_d _model_ , resulting in **scaled dot-product attention**. Scaled dot-product attention is not restricted to self-attention and is also used for cross-attention in transformers.\n**Step 3) Calculate the Attention Weights using the Softmax Function:**\nThe output of the previous step is the vector _S _bank_ , which contains the similarity scores between `bank` and every token in the input sequence. These similarity scores are used as weights to construct a transformer embedding for `bank` from the weighted sum of embeddings for each surrounding token in the prompt. The weights, known as **attention weights** , are calculated by passing _S _bank_ into the softmax function. The outputs are stored in a vector denoted _W _bank_. To see more about the softmax function, refer to the previous article on word2vec.\n![An example calculation of the attention weights for "bank" based on the similarity with every self-attention input. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/10xUGdaAZQvgUGMOvEWtGdQ.png)\nAn example calculation of the attention weights for \"bank\" based on the similarity with every self-attention input. Image by author.\n**Step 4) Calculate the Transformer Embedding**\nFinally, the transformer embedding for `bank` is obtained by taking the weighted sum of `write`, `a`, `prompt`, ..., `bank`. Of course, `bank` will have the highest similarity score with itself (and therefore the largest attention weight), so the output embedding after this process will remain similar to before. This behaviour is ideal since the initial embedding already occupies a region of vector space that encodes some meaning for bank. The goal is to nudge the embedding towards the words that provide more context. The weights for words that provide little context, such as `a` and `man`, are very small. Hence, their influence on the output embedding will be minimal. Words that provide significant context, such as `river` and `fishing`, will have higher weights, and therefore pull the output embedding closer to their regions of vector space. The end result is a new embedding, _y _bank_ , that reflects the context of the entire input sequence.\n![An example calculation for the new embedding of "bank" by taking a weighted sum of the other embeddings for each token in the sequence. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1GuL67Al9iUKDlhWhWmJViw.png)\nAn example calculation for the new embedding of \"bank\" by taking a weighted sum of the other embeddings for each token in the sequence. Image by author.\n### 3.4 - Expanding Self-Attention using Matrices\nAbove, we walked through the steps to calculate the transformer embedding for the singular word `bank`. The input consisted of the learned embedding vector for bank plus its positional information, which we denoted _x _11_ or _x _bank_. The key point here, is that we considered only one vector as the input. If we instead pass in the matrix _X_ (with dimensions _L _max_ × _d _model_) to the self-attention block, we can calculate the transformer embedding for every token in the input prompt simultaneously. The output matrix, _Y_ , contains the transformer embedding for every token along the rows of the matrix. This approach is what enables transformers to quickly process text.\n![A black box diagram of a self-attention block. The matrix of word vectors is represented by X for the input sequence, and Y for the output sequence. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1b5Gnqa6b1A-qa9aphv_JMQ.png)\nA black box diagram of a self-attention block. The matrix of word vectors is represented by X for the input sequence, and Y for the output sequence. Image by author.\n### 3.5 - The Query, Key, and Value Matrices\nThe above description gives an overview of the core functionality of the self-attention block, but there is one more piece of the puzzle. The simple weighted sum above does not include any trainable parameters, but we can introduce some to the process. Without trainable parameters, the performance of the model may still be good, but by allowing the model to learn more intricate patterns and hidden features from the training data, we observe much stronger model performance.\nThe self-attention inputs are used three times to calculate the new embeddings, these include the _x _bank_ vector, the _X^T_ matrix in the dot product step, and the _X^T_ matrix in the weighted sum step. These three sites are the perfect candidates to introduce some weights, which are added in the form of matrices (shown in red). When pre-multiplied by their respective inputs (shown in blue), these form the **key** , **query** , and **value** matrices, _K_ , _Q_ , and _V_ (shown in purple). The number of columns in these weight matrices is an architectural choice by the user. Choosing a value for _d _q_ , _d _k_ , and _d _v_ that is less than _d _model_ will result in dimensionality reduction, which can improve model speed. Ultimately, these values are hyperparameters that can be changed based on the specific implementation of the model and the use-case, and are often all set equal to _d _model_ if unsure [5].\n![A diagram of a complete self-attention block including the key, query, and value matrices. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/10vnPxdYDzzXqCQf2Ygr6Jg.png)\nA diagram of a complete self-attention block including the key, query, and value matrices. Image by author.\n### 3.6 - The Database Analogy\nThe names for these matrices come from an analogy with databases, which is explained briefly below.\n**Query:**\n * A query in a database is what you are looking for when performing a search. For example, \"show me all the albums in the database that have sold more than 1,000,000 records\". In the self-attention block, we are essentially asking the same question, but phrased as \"show me the transformer embedding for this vector (e.g. _x _bank_)\". For the sake of this example, we have only considered a single vector, _x _bank_ , but recall that we can perform the self-attention step on as many vectors as we like by collecting them into a matrix. Therefore, we can just as easily pass in the matrix _X_ as the query, which changes the question to \"show me the transformer embedding for each vector in the input sequence\". This is what actually happens in Transformer-based models.\n\n\n**Key:**\n * The keys in the database are the attributes or columns that are being searched against. In the example given earlier, you can think of this as the \"Albums Sold\" column, which stores the information we are interested in. In self-attention, we are interested in the embeddings for every word in the input prompt, so we can compute a set of attention weights. Therefore, the key matrix is a collection of all the input embeddings.\n\n\n**Value:**\n * The values correspond to the actual data in the database, that is, the actual sale figures in our example (e.g. 2,300,000 copies). For self-attention, this is exactly the same as the input for the key matrix (and query matrix as we just discussed): a collection of all the input embeddings. Hence, the query, key, and value matrices all take in the matrix _X_ as the input.\n\n\n### 3.7 - A Note on Multi-Head Attention\n**Distributing Computation Across Multiple Heads:**\nThe \"Attention is All You Need\" paper extends standard self-attention into **Multi-Head Attention (MHA)** by dividing the attention mechanism into multiple **heads**. In standard self-attention, the model learns a single set of weight matrices (_W _Q_ , _W _K_ , and _W _V_) that transform the token embedding matrix _X_ into query, key, and value matrices (_Q_ , _K_ , and _V_). These matrices are then used to compute attention scores and update _X_ with contextual information as we have seen above.\nIn contrast, MHA splits the attention mechanism into _H_ independent heads, each learning its own smaller set of weight matrices. These weights are used to calculate a set of smaller, head-specific query, key, and value matrices (denoted _Q^h_ , _K^h_ , and _V^h_). Each head processes the input sequence independently, generating distinct attention outputs. These outputs are then concatenated (stacked on top of each other) and passed through a final linear layer to produce the updated _X_ matrix, shown as _Y_ in the diagram below, with rich contextual information.\nBy introducing multiple heads, MHA increases the number of learnable parameters in the attention process, enabling the model to capture more complex relationships within the data. Each head learns its own weight matrices, allowing them to focus on different aspects of the input such as long-range dependencies (relationships between distant words), short-range dependencies (relationships between nearby words), grammatical syntax, etc. The overall effect produces a model with a more nuanced understanding of the input sequence.\n![An overview of the Multi-Head Attention process. For an in-depth explanation of terms and each process step, see Section 2.8 in Part 5 of this series. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/11/1R5rsOaTdYLHIJX_fJiqgPw.png)\nAn overview of the Multi-Head Attention process. For an in-depth explanation of terms and each process step, see Section 2.8 in Part 5 of this series. Image by author.\nThe paragraphs below focus on building a broad intuition for how this works and why this step is useful. However, if you are keen to dive into the implementation details for MHA, see Section 2.8 in the Part 5 - A Complete Guide to Mistral 7B with Code [Link coming soon!].\n**The Benefits of Using Multiple Heads:**\nThe core idea is to allow each head to learn different types of relationships between words in the input sequence, and to combine them to create deep text representations. For example, some heads might learn to capture long-term dependencies (relationships between words that are distant in the text), while others might focus on short-term dependencies (words that are close in text).\nIf the model is given the sentence `A man withdrew money from the bank then sat on the river bank`, the use of multi-head attention allows the model capture the short-term dependency between `money` and the first instance of `bank`, and a separate dependency between `river` and the second instance of `bank`. Thus, the two uses of the word `bank` would be correctly updated with different contextual information for their respective meanings.\n**Building Intuition for Multi-Head Attention:**\nTo deepen this intuition for the usefulness of multiple attention heads, consider words in a sentence that require a lot of context. For example, in the sentence `I ate some of Bob's chocolate cake`, the word `ate` should attend to `I`, `Bob's` and `cake` to gain full context. This is a rather simple example, but if you extend this concept to complex sequences spanning thousands of words, hopefully it seems reasonable that distributing the computational load across separate attention mechanisms will be beneficial.\n![An example of attention heads capturing different word dependencies in an input sequence. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/02/1fh_kaG4cMk4ep1fIajSwiA.png)\nAn example of attention heads capturing different word dependencies in an input sequence. Image by author.\n## 4 - Transformer Embeddings in Python\n### 4.1 - Extracting Learned Embeddings and Transformer Embeddings from Transformer Models\nPython has many options for working with transformer models, but none are perhaps as well-known as Hugging Face. Hugging Face provides a centralised resource hub for NLP researchers and developers alike, including tools such as:\n * `transformers`: The library at the core of Hugging Face, which provides an interface for using, training, and fine-tuning pre-trained transformer models.\n * `tokenizers`: A library for working with tokenizers for many kinds of transformers, either using pre-built tokenizer models or constructing brand new ones from scratch.\n * `datasets`: A collection of datasets to train models on a variety of tasks, not just restricted to NLP.\n * Model Hub: A large repository of cutting-edge models from published papers, community-developed models, and everything in between. These are made freely available and can be easily imported into Python via the `transformers` API.\n\n\nThe code cell below shows how the `transformers` library can be used to load a transformer-based model into Python, and how to extract both the learned embeddings for words (without context) and the transformer embeddings (with context). The remainder of this article will break down the steps shown in this cell and describe additional functionalities available when working with embeddings.\npython\n\n```\nimport torch\nfrom transformers import AutoModel, AutoTokenizer\n\ndef extract_le(sequence, tokenizer, model):\n \"\"\" Extract the learned embedding for each token in an input sequence.\n\n Tokenize an input sequence (string) to produce a tensor of token IDs.\n Return a tensor containing the learned embedding for each token in the\n input sequence.\n\n Args:\n sequence (str): The input sentence(s) to tokenize and extract\n embeddings from.\n tokenizer: The tokenizer used to produce tokens.\n model: The model to extract learned embeddings from.\n\n Returns:\n learned_embeddings (torch.tensor): A tensor containing tensors of\n learned embeddings for each token in the input sequence.\n \"\"\"\n token_dict = tokenizer(sequence, return_tensors='pt')\n token_ids = token_dict['input_ids']\n learned_embeddings = model.embeddings.word_embeddings(token_ids)[0]\n\n # Additional processing for display purposes\n learned_embeddings = learned_embeddings.tolist()\n learned_embeddings = [[round(i,2) for i in le] \n for le in learned_embeddings]\n\n return learned_embeddings\n\ndef extract_te(sequence, tokenizer, model):\n \"\"\" Extract the tranformer embedding for each token in an input sequence.\n\n Tokenize an input sequence (string) to produce a tensor of token IDs.\n Return a tensor containing the transformer embedding for each token in the\n input sequence.\n\n Args:\n sequence (str): The input sentence(s) to tokenize and extract\n embeddings from.\n tokenizer: The tokenizer used to produce tokens.\n model: The model to extract learned embeddings from.\n\n Returns:\n transformer_embeddings (torch.tensor): A tensor containing tensors of\n transformer embeddings for each token in the input sequence.\n \"\"\"\n token_dict = tokenizer(sequence, return_tensors='pt')\n\n with torch.no_grad():\n base_model_output = model(**token_dict)\n\n transformer_embeddings = base_model_output.last_hidden_state[0]\n\n # Additional processing for display purposes\n transformer_embeddings = transformer_embeddings.tolist()\n transformer_embeddings = [[round(i,2) for i in te] \n for te in transformer_embeddings]\n\n return transformer_embeddings\n\n# Instantiate DistilBERT tokenizer and model\ntokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')\nmodel = AutoModel.from_pretrained('distilbert-base-uncased')\n\n# Extract the learned embedding for bank from DistilBERT\nle_bank = extract_le('bank', tokenizer, model)[1]\n\n# Write sentences containing \"bank\" in two different contexts\ns1 = 'Write a poem about a man fishing on a river bank.'\ns2 = 'Write a poem about a man withdrawing money from a bank.'\n\n# Extract the transformer embedding for bank from DistilBERT in each sentence\ns1_te_bank = extract_te(s1, tokenizer, model)[11]\ns2_te_bank = extract_te(s2, tokenizer, model)[11]\n\n# Print the results\nprint('------------------- Embedding vectors for \"bank\" -------------------n')\nprint(f'Learned embedding: {le_bank[:5]}')\nprint(f'Transformer embedding (sentence 1): {s1_te_bank[:5]}')\nprint(f'Transformer embedding (sentence 2): {s2_te_bank[:5]}')\n```\n\nshell\n\n```\n------------------- Embedding vectors for \"bank\" -------------------\n\nLearned embedding: [-0.03, -0.06, -0.09, -0.07, -0.03]\nTransformer embedding (sentence 1): [0.15, -0.16, -0.17, -0.08, 0.44]\nTransformer embedding (sentence 2): [0.27, -0.23, -0.23, -0.21, 0.79]\n```\n\n### 4.2 - Import the `Transformers` Library\nThe first step to produce transformer embeddings is to choose a model from the Hugging Face `transformers` library. In this article, we will not use the model for inference but solely to examine the embeddings it produces. This is not a standard use-case, and so we will have to do some extra digging in order to access the embeddings. Since the `transformers` library is written in PyTorch (referred to as `torch` in the code), we can import `torch` to extract data from the inner workings of the models.\n### 4.3 - Choose a Model\nFor this example, we will use DistilBERT, a smaller version of Google's BERT model which was released by Hugging Face themselves in October 2019 [6]. According to the Hugging Face documentation [7]:\n> _DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than_` _bert-base-uncased_`_, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark_.\nWe can import DistilBERT and its corresponding tokenizer into Python either directly from the `transformers` library or using the `AutoModel` and `AutoTokenizer` classes. There is very little difference between the two, although `AutoModel` and `AutoTokenizer` are often preferred since the model name can be parameterised and stored in a string, which makes it simpler to change the model being used.\npython\n\n```\nimport torch\nfrom transformers import DistilBertTokenizerFast, DistilBertModel\n\n# Instantiate DistilBERT tokenizer and model\ntokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')\nmodel = DistilBertModel.from_pretrained('distilbert-base-uncased')\n```\n\npython\n\n```\nimport torch\nfrom transformers import AutoModel, AutoTokenizer\n\n# Instantiate DistilBERT tokenizer and model\ntokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased')\nmodel = AutoModel.from_pretrained('distilbert-base-uncased')\n```\n\nAfter importing DistilBERT and its corresponding tokenizer, we can call the `from_pretrained` method for each to load in the specific version of the DistilBERT model and tokenizer we want to use. In this case, we have chosen `distilbert-base-uncased`, where `base` refers to the size of the model, and `uncased` indicates that the model was trained on uncased text (all text is converted to lowercase).\n### 4.4 - Create Some Example Sentences\nNext, we can create some sentences to give the model some words to embed. The two sentences, `s1` and `s2`, both contain the word `bank` but in different contexts. The goal here is to show that the word `bank` will begin with the same learned embedding in both sentences, then be modified by DistilBERT using self-attention to produce a unique, contextualised embedding for each input sequence.\npython\n\n```\n# Create example sentences to produce embeddings for\ns1 = 'Write a poem about a man fishing on a river bank.'\ns2 = 'Write a poem about a man withdrawing money from a bank.'\n```\n\n### 4.5 - Tokenize an Input Sequence\nThe tokenizer class can be used to tokenize an input sequence (as shown below) and convert a string into a list of token IDs. Optionally, we can also pass a `return_tensors` argument to format the token IDs as a PyTorch tensor (`return_tensors=pt`) or as TensorFlow constants (`return_tensors=tf`). Leaving this argument empty will return the token IDs in a Python list. The return value is a dictionary that contains `input_ids`: the list-like object containing token IDs, and `attention_mask` which we will ignore for now.\n> **Note:** BERT-based models include a `[CLS]` token at the beginning of each sequence, and a `[SEP]` token to distinguish between two bodies of text in the input. These are present due to the tasks that BERT was originally trained on and can largely be ignored here. For a discussion on BERT special tokens, model sizes, `cased` vs `uncased`, and the attention mask, see [Part 4 of this series](https://medium.com/p/9f87602e4a11).\npython\n\n```\ntoken_dict = tokenizer(s1, return_tensors='pt')\ntoken_ids = token_dict['input_ids'][0]\n```\n\n### 4.6 - Extract the Learned Embeddings from a Model\nEach transformer model provides access to its learned embeddings via the `embeddings.word_embeddings` method. This method accepts a token ID or collection of token IDs and returns the learned embedding(s) as a PyTorch tensor.\npython\n\n```\nlearned_embeddings = model.embeddings.word_embeddings(token_ids)\nlearned_embeddings\n```\n\nshell\n\n```\ntensor([[ 0.0390, -0.0123, -0.0208, ..., 0.0607, 0.0230, 0.0238],\n [-0.0300, -0.0070, -0.0247, ..., 0.0203, -0.0566, -0.0264],\n [ 0.0062, 0.0100, 0.0071, ..., -0.0043, -0.0132, 0.0166],\n ...,\n [-0.0261, -0.0571, -0.0934, ..., -0.0351, -0.0396, -0.0389],\n [-0.0244, -0.0138, -0.0078, ..., 0.0069, 0.0057, -0.0016],\n [-0.0199, -0.0095, -0.0099, ..., -0.0235, 0.0071, -0.0071]],\n grad_fn=)\n```\n\n### 4.7 - Extract the Transformer Embeddings from a Model\nConverting a context-lacking learned embedding into a context-aware transformer embedding requires a forward pass of the model. Since we are not updating the weights of the model here (i.e. training the model), we can use the `torch.no_grad()` context manager to save on memory. This allows us to pass the tokens directly into the model and compute the transformer embeddings without any unnecessary calculations. Once the tokens have been passed into the model, a `BaseModelOutput` is returned, which contains various information about the forward pass. The only data that is of interest here is the activations in the last hidden state, which form the transformer embeddings. These can be accessed using the `last_hidden_state` attribute, as shown below, which concludes the explanation for the code cell shown at the top of this section.\npython\n\n```\nwith torch.no_grad():\n base_model_output = model(**token_dict)\n\ntransformer_embeddings = base_model_output.last_hidden_state\ntransformer_embeddings\n```\n\nshell\n\n```\ntensor([[[-0.0957, -0.2030, -0.5024, ..., 0.0490, 0.3114, 0.1348],\n [ 0.4535, 0.5324, -0.2670, ..., 0.0583, 0.2880, -0.4577],\n [-0.1893, 0.1717, -0.4159, ..., -0.2230, -0.2225, 0.0207],\n ...,\n [ 0.1536, -0.1616, -0.1735, ..., -0.3608, -0.3879, -0.1812],\n [-0.0182, -0.4264, -0.6702, ..., 0.3213, 0.5881, -0.5163],\n [ 0.7911, 0.2633, -0.4892, ..., -0.2303, -0.6364, -0.3311]]])\n```\n\n### 4.8 - Convert Token IDs to Tokens\nIt is possible to convert token IDs back into textual tokens, which shows exactly how the tokenizer divided the input sequence. This is useful when longer or rarer words are divided into multiple subwords when using subword tokenizers such as WordPiece (e.g. in BERT-based models) or Byte-Pair Encoding (e.g. in the GPT family of models).\npython\n\n```\ntokens = tokenizer.convert_ids_to_tokens(token_ids)\ntokens\n```\n\nshell\n\n```\n['[CLS]', 'write', 'a', 'poem', 'about', 'a', 'man', 'fishing', 'on', 'a',\n 'river', 'bank', '.', '[SEP]']\n```\n\n## 5 - Conclusion\nThe self-attention mechanism generates rich, context-aware transformer embeddings for text by processing each token in an input sequence simultaneously. These embeddings build on the foundations of static word embeddings (such as word2vec) and enable more capable language models such as BERT and GPT. Further work in this field will continue to improve the capabilities of LLMs and NLP as a whole.\n## 6 - Further Reading\n[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, [Attention is All You Need](https://arxiv.org/pdf/1706.03762) (2017), Advances in Neural Information Processing Systems 30 (NIPS 2017)\n[2] Hugging Face, [Transformers](https://huggingface.co/docs/transformers/index) (2024), HuggingFace.co\n[3] OpenAI, [ChatGPT Pricing](https://openai.com/chatgpt/pricing/) (2024), OpenAI.com\n[4] A. Gu and T. Dao, [Mamba: Linear-Time Sequence Modelling with Selective State Spaces](https://arxiv.org/abs/2312.00752) (2023), ArXiv abs/2312.00752\n[5] J. Alammar, [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) (2018). GitHub\n[6] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/pdf/1910.01108) (2019), 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019\n[7] Hugging Face, [DistilBERT Documentation](https://huggingface.co/docs/transformers/en/model_doc/distilbert) (2024) HuggingFace.co\n[8] Hugging Face, [BERT Documentation](https://huggingface.co/docs/transformers/model_doc/bert) (2024) HuggingFace.co\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657533258_n6gaio.jpeg)\nWritten By\n## [Bradney Smith](https://towardsdatascience.com/author/bradneysmith/)\n[See all from Bradney Smith](https://towardsdatascience.com/author/bradneysmith/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[NLP](https://towardsdatascience.com/tag/nlp/)[Self Attention](https://towardsdatascience.com/tag/self-attention/)[Transformers](https://towardsdatascience.com/tag/transformers/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fcontextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e%2F&title=Self-Attention%20Explained%20with%20Code)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fcontextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e%2F&title=Self-Attention%20Explained%20with%20Code)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fcontextual-transformer-embeddings-using-self-attention-explained-with-diagrams-and-python-code-d7a9f0f4d94e%2F&text=Self-Attention%20Explained%20with%20Code)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/12/Copilot_20251206_112042.jpg)](https://towardsdatascience.com/graphrag-in-practice-how-to-build-cost-efficient-high-recall-retrieval-systems/)\n## [GraphRAG in Practice: How to Build Cost-Efficient, High-Recall Retrieval Systems](https://towardsdatascience.com/graphrag-in-practice-how-to-build-cost-efficient-high-recall-retrieval-systems/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nSmarter retrieval strategies that outperform dense graphs — with hybrid pipelines and lower cost\n[Partha Sarkar](https://towardsdatascience.com/author/partha_sarkar/)\nDecember 9, 202511 min read\n * [![](https://towardsdatascience.com/images/fallback-image.jpg)](https://towardsdatascience.com/latest-picks-improving-named-entity-disambiguation-using-entity-relatedness-within-wikipedia-a7970d4ce685/)\n## [Latest picks: Improving Named Entity Disambiguation using Entity Relatedness within Wikipedia](https://towardsdatascience.com/latest-picks-improving-named-entity-disambiguation-using-entity-relatedness-within-wikipedia-a7970d4ce685/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nYour daily dose of data science\n[TDS Editors](https://towardsdatascience.com/author/towardsdatascience/)\nJanuary 5, 20211 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/1wZRDPXMCiMdiAAqXck6y6w-scaled.jpeg)](https://towardsdatascience.com/how-to-perform-ordinal-regression-classification-in-pytorch-361a2a095a99/)\n## [How to Perform Ordinal Regression / Classification in PyTorch](https://towardsdatascience.com/how-to-perform-ordinal-regression-classification-in-pytorch-361a2a095a99/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA simple trick for improving model performance when labels are ordered.\n[Mathias Gruber](https://towardsdatascience.com/author/nanomathias/)\nJuly 10, 20217 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/12/0M4gj9drSWPum4oBz-scaled.jpg)](https://towardsdatascience.com/debunking-data-science-myths-83d693fb5ba4/)\n## [Debunking Data Science Myths](https://towardsdatascience.com/debunking-data-science-myths-83d693fb5ba4/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nThe Truth They Don't Tell You About Data Science\n[Kurtis Pykes](https://towardsdatascience.com/author/kurtispykes/)\nDecember 8, 20204 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/0nz8GP5aEHNi_Zkw4-scaled.jpg)](https://towardsdatascience.com/how-to-extract-and-convert-tables-from-pdf-files-to-pandas-dataframe-cb2e4c596fa8/)\n## [How to Extract and Convert Tables From PDF Files to Pandas Dataframe?](https://towardsdatascience.com/how-to-extract-and-convert-tables-from-pdf-files-to-pandas-dataframe-cb2e4c596fa8/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nSo you have some pdf files with tables in them and want to read them into a pandas data frame. Let me show you how.\n[Saed Hussain](https://towardsdatascience.com/author/saedhussain/)\nApril 19, 20223 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/0QUZPrCzpAU1AsTcq-scaled.jpg)](https://towardsdatascience.com/develop-your-own-calendar-to-track-important-dates-with-python-c1af9e98ffc3/)\n## [Develop Your Own Calendar To Track Important Dates with Python](https://towardsdatascience.com/develop-your-own-calendar-to-track-important-dates-with-python-c1af9e98ffc3/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nDeveloping a calendar GUI interface to regulate your plans for 2022 and beyond\n[Bharath K](https://towardsdatascience.com/author/bharath-k1297/)\nJanuary 13, 20227 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/05/1_yNba9as-o0nTeh-4Aw89Qw.jpg)](https://towardsdatascience.com/use-pytorch-to-easily-access-your-gpu/)\n## [Use PyTorch to Easily Access Your GPU](https://towardsdatascience.com/use-pytorch-to-easily-access-your-gpu/)\n[Programming](https://towardsdatascience.com/category/programming/)\nOr… how an ML library can accelerate non-ML computations\n[Thomas Reid](https://towardsdatascience.com/author/thomas_reid/)\nMay 21, 202511 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/11/0ramo1Mj_UNrRQFdL-scaled.jpg)](https://towardsdatascience.com/an-overview-of-forecasting-performance-metrics-ef548dad0134/)\n## [An Overview Of Forecasting Performance Metrics](https://towardsdatascience.com/an-overview-of-forecasting-performance-metrics-ef548dad0134/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nAn overview of some of the fundamental performance metrics for time series forecasting\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nNovember 16, 20225 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/02/1zrGyk6kdaiGUXxY8MXOQ9A.png)](https://towardsdatascience.com/an-intuitive-guide-to-track-your-ml-experiments-with-mlflow-7ac50e63b09/)\n## [An Intuitive Guide to Track Your ML Experiments with MLflow](https://towardsdatascience.com/an-intuitive-guide-to-track-your-ml-experiments-with-mlflow-7ac50e63b09/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHelping you on improving the reproducibility, management and tracking of experiments for your Sklearn model\n[Eugenia Anello](https://towardsdatascience.com/author/eugenia-anello/)\nFebruary 8, 20220 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3642/image?etag=1780830904480&width=500&height=500) Check Your Biases ](https://towardsdatascience.com/check-your-biases-ba2685a7f799/ \"Check Your Biases\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/3660/image?etag=1780176246070&width=500&height=500) Does Your Company Have a Data Strategy? ](https://towardsdatascience.com/does-your-company-have-a-data-strategy-e8944dbec4ae/ \"Does Your Company Have a Data Strategy?\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/4973/image?etag=1782232567105&width=500&height=500) Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads ](https://towardsdatascience.com/machine-learning-tricks-learning-with-chatgpt-and-other-recent-must-reads-f277648a1921/ \"Machine Learning Tricks, Learning with ChatGPT, and Other Recent Must-Reads\")\n\n\nword word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word\n \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1 \nmmMwWLliI0fiflO&1\n", "body_base_path": "attachments/34afc94764d0/pages/", "body_markdown_path": "attachments/34afc94764d0/pages/archived-body.md", "local_html": "attachments/34afc94764d0/pages/archived-body.html", "body_source": "prior_capture" }