{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Maxime Labonne",
  "title_listed": "Graph Attention Networks: Self-Attention Explained",
  "keywords": "GNN,图注意力,GAT",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/graph-attention-networks-in-python-975736ac5c0c/",
  "publish_date": "2022-04",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "701ad9412fe8",
  "collected_at": "2026-10-01T22:10:38",
  "title": "Graph Attention Networks: Self-Attention Explained | Towards Data Science",
  "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Programming](https://towardsdatascience.com/category/programming/)\n# Graph Attention Networks: Self-Attention Explained\nA guide to GNNs with self-attention using PyTorch Geometric\n[Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\nApril 17, 202211 min read\n![Image by author, file icon by OpenMoji \\(CC BY-SA 4.0\\)](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1_p3JftpWqL3pJbRTxqflgg.gif)\nImage by author, file icon by [OpenMoji](https://openmoji.org/) ([CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/))\nGraph Attention Networks are **one of the most popular types** of Graph Neural Networks. For a good reason.\nWith Graph _Convolutional_ Networks (GCN), every neighbor has the **same importance**. Obviously, it should not be the case: some nodes are more essential than others.\n![Node 4 is more important than node 3, which is more important than node 2 \\(image by author\\)](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1h0KUV0wX5KckVNoiWy5P2A.png)\nNode 4 is more important than node 3, which is more important than node 2 (image by author)\nGraph _Attention_ Networks offer a solution to this problem. To consider the importance of each neighbor, an attention mechanism assigns a **weighting factor to every connection**.\nIn this article, we'll see how to **calculate** these attention scores and **implement** an efficient GAT in PyTorch Geometric (PyG). You can run the code of this tutorial with the following [Google Colab notebook](https://colab.research.google.com/drive/1B0vLpH_gSfrOLgsc2UZVyXrcofzA-t0L?usp=sharing).\n## 🌐 I. Graph data\n![CiteSeer dataset \\(image by author, made with yEd Live\\)](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/06DAVkX2UjMdH-ijK.png)\nCiteSeer dataset (image by author, made with [yEd Live](https://www.yworks.com/yed-live/))\nThere are three classic graph datasets we can use for this work (MIT license). They represent networks of research papers, where each connection is a citation.\n  * [**Cora**](http://www.kamalnigam.com/papers/cora-jnl.pdf): it consists of 2708 machine learning papers that belong to one of 7 categories. ➡️ Node features represent the presence (1) or absence (0) of 1433 words in a paper (binary [bag of words](https://en.wikipedia.org/wiki/Bag-of-words_model)).\n  * [**CiteSeer**](https://clgiles.ist.psu.edu/papers/DL-1998-citeseer.pdf): it is a bigger but similar dataset of 3312 scientific papers to classify into one of 6 categories. ➡️ Node features represent the presence (1) or absence (0) of 3703 words in a paper.\n  * [**PubMed**](http://eliassi.org/papers/ai-mag-tr08.pdf): it is an even bigger dataset with 19717 scientific publications about diabetes from PubMed's database, classified into 3 categories. ➡️ Node features are [TF-IDF weighted word vectors](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) from a dictionary of 500 unique words.\n\n\nThese datasets have been widely used by the scientific community. As a challenge, we can compare our accuracy scores to those obtained in the [literature](https://arxiv.org/pdf/1710.10903.pdf) using **Multilayer Perceptrons** (MLPs), **GCNs** , and **GATs** :\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1nnsT7G3fb_EfuNS9NVSK4w.png)\nPubMed is quite large so it would take longer to process it and train a GNN on it. Cora is the most studied one in the literature, so let's **focus on CiteSeer** as a middle ground.\nWe can directly import any of these datasets in PyTorch Geometric with the [Planetoid class](https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html#torch_geometric.datasets.Planetoid):\npython\n\n```\nfrom torch_geometric.datasets import Planetoid\n\n# Import dataset from PyTorch Geometric\ndataset = Planetoid(root=\".\", name=\"CiteSeer\")\ndata = dataset[0]\n\n# Print information about the dataset\nprint(f'Number of graphs: {len(dataset)}')\nprint(f'Number of nodes: {data.x.shape[0]}')\nprint(f'Number of features: {dataset.num_features}')\nprint(f'Number of classes: {dataset.num_classes}')\nprint(f'Has isolated nodes: {data.has_isolated_nodes()}')\n```\n\ntext\n\n```\nNumber of graphs: 1\nNumber of nodes: 3327\nNumber of features: 3703\nNumber of classes: 6\nHas isolated nodes: True\n```\n\nInterestingly enough, we have **3327 nodes instead of 3312.** I found that PyG actually uses [this paper's](https://arxiv.org/pdf/1603.08861.pdf) implementation of CiteSeer, which also displays 3327 nodes. Mystery solved for now.\nHowever, we observe that **some nodes are isolated** (48 to be precise)! Correctly classifying these isolated nodes will be a challenge since we cannot rely on any aggregation.\nLet's plot the number of connections of each node with `degree`:\npython\n\n```\nfrom torch_geometric.utils import degree\nfrom collections import Counter\n\n# Get list of degrees for each node\ndegrees = degree(data.edge_index[0]).numpy()\n\n# Count the number of nodes for each degree\nnumbers = Counter(degrees)\n\n# Bar plot\nfig, ax = plt.subplots(figsize=(18, 7))\nax.set_xlabel('Node degree')\nax.set_ylabel('Number of nodes')\nplt.bar(numbers.keys(),\n        numbers.values(),\n        color='#0A047A')\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1Md0aDjloaEGZCPssWvxkaQ.png)\nMost nodes only have **1 or 2 neighbors**. It could explain why CiteSeer **** obtains lower accuracy scores than the two other datasets...\n## ⚠️ II. Self-attention\nIntroduced by [Veličković et al.](https://arxiv.org/abs/1710.10903) in 2017, self-attention in GNNs relies on a simple idea: **nodes should not all have the same importance**.\nWe talk about _self_ -attention (and not just attention) because inputs are compared to each other.\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1AVQq-PJoU3GCffAuQJXXvA.png)\nImage by author\nThis mechanism assigns a **weighting factor** (attention score) **** to each connection. Let's call** __ α**ᵢⱼ the attention score between the node _s_ i an _d_ j.\nHere's how to calculate the embedding of node 1, where 𝐖 is a shared weight matrix:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1-7zYVLyhNdqj7yxlOJZa6A.png)\nBut how do we calculate the attention scores? We could write a static formula, but there's a smarter solution: we can **learn** **their values with a neural network**. There are three steps in this process:\n  1. **Linear transformation** ;\n  2. **Activation function** ;\n  3. **Softmax normalization.**\n\n\n### 1️⃣ Linear transformation\nWe want to calculate the **importance of each connection** , so we need pairs of hidden vectors. An easy way to create these pairs is to concatenate vectors from both nodes.\nOnly then can we apply a new **linear transformation** with a weight matrix 𝐖ₐ**ₜₜ:**\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/12zojhcpsK92_SneI0zmWaw.png)\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1chXBes9TjC6WMD637MYSIg.png)\nImage by author\n### 2️⃣ Activation function\nWe're building a **** neural network, so the second step is to add an activation function. In this case, the authors of the paper chose the _LeakyReLU_ function.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1onA_k8ocjpon4UovKwZSog.png)\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1-1RHa4aDJRhdbNLtVuNM2w.png)\nImage by author\n### 3️⃣ Softmax normalization\nThe output of our neural network is **not normalized** , which is a problem since we want to compare these scores. To be able to say if node 2 is more important to node 1 than node 3 (_α_₁₂ > _α_₁₃), we need to share the same scale.\nA common way to do it with neural networks is to use the _**softmax**_ function. Here, we apply it to every neighboring node:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1wdXTVPRQhs561BuWAoE8aw.png)\n![Image by author](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1ZY9HLOvwVXxLc-wMNH17fw.png)\nImage by author\nHere you have it: we can calculate every _α_ᵢⱼ. The only problem is... **self-attention is not very stable**. In order to improve performance, [Vaswani et al.](https://arxiv.org/abs/1706.03762) introduced multi-head attention in the transformer architecture.\n### 4️⃣ Bonus: multi-head attention\nThis is only slightly surprising since we've been talking about self-attention a lot but, in reality, **transformers are GNNs in disguise**. This is why we can reuse some ideas from Natural Language Processing here.\n![Multi-head attention \\(image by author\\)](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1A3C7OdAe1DrVEumyYvUdhw.png)\nMulti-head attention (image by author)\nIn GATs, multi-head attention consists of **replicating the same 3 steps several times** in order to average or concatenate the results. That's it. Instead of a single _h₁_ , we get one hidden vector _h₁ᵏ_ per attention head. One of the two following schemes can then be applied:\n  * **Average** : we sum the different _hᵢᵏ​_ and normalize the result by the number of attention heads _n_ ;\n\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1O77fo-OAXP5n5y6izG0lAw.png)\n  * **Concatenation** : we concatenate the different _hᵢᵏ_.​\n\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1RiEetyq7VOsTxHtOAPx3lQ.png)\nIn practice, we use the **concatenation scheme** when it's a hidden layer, and the **average scheme** when it's the last layer of the network.\n## 🧠 III. Graph Attention Networks\nLet's implement a GAT in PyTorch Geometric. This library has **two different graph attention layers** : `GATConv` and `GATv2Conv`.\nWhat we talked about so far is the `GatConv` layer, but in 2021 [Brody et al.](https://arxiv.org/abs/2105.14491) introduced an improvement by modifying the order of operations. The weight matrix 𝐖 is applied a**fter the concatenation,** and the attention weight matrix 𝐖ₐₜ**ₜ i** s used af**ter the Le_akyReLU f_unction.** In summary:\n  * `GatConv`:\n\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1zzxjH_OF43Hy8GZeo95aPA.png)\n  * `Gatv2Conv`:\n\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1XNA9GR4YDZdjPrqHBMcsJQ.png)\nWhich one should you use? According to Brody et al., `**Gatv2Conv**`**consistently outperforms**`**GatConv**`and thus should be preferred.\nNow let's classify the papers from CiteSeer! I tried to **roughly reproduce the experiments** of the original authors without adding too much complexity. You can find the official implementation of GAT [on GitHub](https://github.com/PetarV-/GAT).\nNote that we use graph attention layers in two configurations:\n  * The **first layer** concatenates 8 outputs (multi-head attention);\n  * The **second layer** only has 1 head, which produces our final embeddings.\n\n\nWe're also gonna train and test a GCN to compare the accuracy scores.\npython\n\n```\nimport torch.nn.functional as F\nfrom torch.nn import Linear, Dropout\nfrom torch_geometric.nn import GCNConv, GATv2Conv\n\nclass GCN(torch.nn.Module):\n  \"\"\"Graph Convolutional Network\"\"\"\n  def __init__(self, dim_in, dim_h, dim_out):\n    super().__init__()\n    self.gcn1 = GCNConv(dim_in, dim_h)\n    self.gcn2 = GCNConv(dim_h, dim_out)\n    self.optimizer = torch.optim.Adam(self.parameters(),\n                                      lr=0.01,\n                                      weight_decay=5e-4)\n\n  def forward(self, x, edge_index):\n    h = F.dropout(x, p=0.5, training=self.training)\n    h = self.gcn1(h, edge_index)\n    h = torch.relu(h)\n    h = F.dropout(h, p=0.5, training=self.training)\n    h = self.gcn2(h, edge_index)\n    return h, F.log_softmax(h, dim=1)\n\nclass GAT(torch.nn.Module):\n  \"\"\"Graph Attention Network\"\"\"\n  def __init__(self, dim_in, dim_h, dim_out, heads=8):\n    super().__init__()\n    self.gat1 = GATv2Conv(dim_in, dim_h, heads=heads)\n    self.gat2 = GATv2Conv(dim_h*heads, dim_out, heads=1)\n    self.optimizer = torch.optim.Adam(self.parameters(),\n                                      lr=0.005,\n                                      weight_decay=5e-4)\n\n  def forward(self, x, edge_index):\n    h = F.dropout(x, p=0.6, training=self.training)\n    h = self.gat1(x, edge_index)\n    h = F.elu(h)\n    h = F.dropout(h, p=0.6, training=self.training)\n    h = self.gat2(h, edge_index)\n    return h, F.log_softmax(h, dim=1)\n\ndef accuracy(pred_y, y):\n    \"\"\"Calculate accuracy.\"\"\"\n    return ((pred_y == y).sum() / len(y)).item()\n\ndef train(model, data):\n    \"\"\"Train a GNN model and return the trained model.\"\"\"\n    criterion = torch.nn.CrossEntropyLoss()\n    optimizer = model.optimizer\n    epochs = 200\n\n    model.train()\n    for epoch in range(epochs+1):\n        # Training\n        optimizer.zero_grad()\n        _, out = model(data.x, data.edge_index)\n        loss = criterion(out[data.train_mask], data.y[data.train_mask])\n        acc = accuracy(out[data.train_mask].argmax(dim=1), data.y[data.train_mask])\n        loss.backward()\n        optimizer.step()\n\n        # Validation\n        val_loss = criterion(out[data.val_mask], data.y[data.val_mask])\n        val_acc = accuracy(out[data.val_mask].argmax(dim=1), data.y[data.val_mask])\n\n        # Print metrics every 10 epochs\n        if(epoch % 10 == 0):\n            print(f'Epoch {epoch:>3} | Train Loss: {loss:.3f} | Train Acc: '\n                  f'{acc*100:>6.2f}% | Val Loss: {val_loss:.2f} | '\n                  f'Val Acc: {val_acc*100:.2f}%')\n\n    return model\n\ndef test(model, data):\n    \"\"\"Evaluate the model on test set and print the accuracy score.\"\"\"\n    model.eval()\n    _, out = model(data.x, data.edge_index)\n    acc = accuracy(out.argmax(dim=1)[data.test_mask], data.y[data.test_mask])\n    return acc\n```\n\npython\n\n```\n%%time\n\n# Create GCN\ngcn = GCN(dataset.num_features, 16, dataset.num_classes)\nprint(gcn)\n\n# Train\ntrain(gcn, data)\n\n# Test\nacc = test(gcn, data)\nprint(f'GCN test accuracy: {acc*100:.2f}%n')\n```\n\ntext\n\n```\nGCN(\n  (gcn1): GCNConv(3703, 16)\n  (gcn2): GCNConv(16, 6)\n)\n```\n\ntext\n\n```\nEpoch   0 | Train Loss: 1.782 | Train Acc:  20.83% | Val Loss: 1.79 \nEpoch  20 | Train Loss: 0.165 | Train Acc:  95.00% | Val Loss: 1.30 \nEpoch  40 | Train Loss: 0.069 | Train Acc:  99.17% | Val Loss: 1.66 \nEpoch  60 | Train Loss: 0.053 | Train Acc:  99.17% | Val Loss: 1.50 \nEpoch  80 | Train Loss: 0.054 | Train Acc: 100.00% | Val Loss: 1.67 \nEpoch 100 | Train Loss: 0.062 | Train Acc:  99.17% | Val Loss: 1.62 \nEpoch 120 | Train Loss: 0.043 | Train Acc: 100.00% | Val Loss: 1.66 \nEpoch 140 | Train Loss: 0.058 | Train Acc:  98.33% | Val Loss: 1.68 \nEpoch 160 | Train Loss: 0.037 | Train Acc: 100.00% | Val Loss: 1.44 \nEpoch 180 | Train Loss: 0.036 | Train Acc:  99.17% | Val Loss: 1.65 \nEpoch 200 | Train Loss: 0.093 | Train Acc:  95.83% | Val Loss: 1.73 \n\nGCN test accuracy: 67.70%\n\nCPU times: user 25.1 s, sys: 847 ms, total: 25.9 s\nWall time: 32.4 s\n```\n\npython\n\n```\n%%time\n\n# Create GAT\ngat = GAT(dataset.num_features, 8, dataset.num_classes)\nprint(gat)\n\n# Train\ntrain(gat, data)\n\n# Test\nacc = test(gat, data)\nprint(f'GAT test accuracy: {acc*100:.2f}%n')\n```\n\ntext\n\n```\nGAT(\n  (gat1): GATv2Conv(3703, 8, heads=8)\n  (gat2): GATv2Conv(64, 6, heads=1)\n)\n```\n\ntext\n\n```\nEpoch   0 | Train Loss: 1.790 | Val Loss: 1.81 | Val Acc: 12.80%\nEpoch  20 | Train Loss: 0.040 | Val Loss: 1.21 | Val Acc: 64.80%\nEpoch  40 | Train Loss: 0.027 | Val Loss: 1.20 | Val Acc: 67.20%\nEpoch  60 | Train Loss: 0.009 | Val Loss: 1.11 | Val Acc: 67.00%\nEpoch  80 | Train Loss: 0.013 | Val Loss: 1.16 | Val Acc: 66.80%\nEpoch 100 | Train Loss: 0.013 | Val Loss: 1.07 | Val Acc: 67.20%\nEpoch 120 | Train Loss: 0.014 | Val Loss: 1.12 | Val Acc: 66.40%\nEpoch 140 | Train Loss: 0.007 | Val Loss: 1.19 | Val Acc: 65.40%\nEpoch 160 | Train Loss: 0.007 | Val Loss: 1.16 | Val Acc: 68.40%\nEpoch 180 | Train Loss: 0.006 | Val Loss: 1.13 | Val Acc: 68.60%\nEpoch 200 | Train Loss: 0.007 | Val Loss: 1.13 | Val Acc: 68.40%\n\nGAT test accuracy: 70.00%\n\nCPU times: user 53.4 s, sys: 2.68 s, total: 56.1 s\nWall time: 55.9 s\n```\n\nThis experiment is not super rigorous: we'd need to **repeat it** _**n**_**times** and take the average accuracy with a standard deviation as the final result.\nWe can see in this example that the **GAT outperforms the GCN** in terms of accuracy (70.00% vs. 67.70%), but takes longer to train (55.9s vs. 32.4s). It's a tradeoff that can cause scalability issues when working with large graphs.\nThe authors obtained 72.5% for the GAT and 70.3% for the GCN, which is clearly better than what we did. The difference can be explained by **preprocessing** , some **tweaks in the models,** and a different **training setting** (_e.g.,_ a patience of 100 instead of a fixed number of epochs).\nLet's visualize what the GAT learned. We're gonna use [t-SNE](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding), a powerful method to plot high-dimensional data in 2D or 3D. First, let's see what the embeddings looked like before any training: it should be absolutely **random** since they're produced by randomly initialized weight matrices.\npython\n\n```\nuntrained_gat = GAT(dataset.num_features, 8, dataset.num_classes)\n\n# Get embeddings\nh, _ = untrained_gat(data.x, data.edge_index)\n\n# Train TSNE\ntsne = TSNE(n_components=2, learning_rate='auto',\n         init='pca').fit_transform(h.detach())\n\n# Plot TSNE\nplt.figure(figsize=(10, 10))\nplt.axis('off')\nplt.scatter(tsne[:, 0], tsne[:, 1], s=50, c=data.y)\nplt.show()\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/10qB0PpkvaER4bYwXTQIysQ.png)\nIndeed, there's **no apparent structure**. But do the embeddings produced by our trained model look better?\npython\n\n```\nh, _ = gat(data.x, data.edge_index)\n\n# Train TSNE\ntsne = TSNE(n_components=2, learning_rate='auto',\n         init='pca').fit_transform(h.detach())\n\n# Plot TSNE\nplt.figure(figsize=(10, 10))\nplt.axis('off')\nplt.scatter(tsne[:, 0], tsne[:, 1], s=50, c=data.y)\nplt.show()\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1RGEUniuYZllkCOgLlFDp1g.png)\nThe difference is noticeable: **nodes belonging to the same classes cluster together**. We can see 6 clusters, corresponding to the 6 classes of papers. There are outliers, but this was to be expected: our accuracy score is far from perfect.\nPreviously, I speculated that poorly connected nodes **** migh** t negatively impact** performance on CiteSeer. Let's calculate the model's accuracy for each degree.\npython\n\n```\nfrom torch_geometric.utils import degree\n\n# Get model's classifications\n_, out = gat(data.x, data.edge_index)\n\n# Calculate the degree of each node\ndegrees = degree(data.edge_index[0]).numpy()\n\n# Store accuracy scores and sample sizes\naccuracies = []\nsizes = []\n\n# Accuracy for degrees between 0 and 5\nfor i in range(0, 6):\n  mask = np.where(degrees == i)[0]\n  accuracies.append(accuracy(out.argmax(dim=1)[mask], data.y[mask]))\n  sizes.append(len(mask))\n\n# Accuracy for degrees > 5\nmask = np.where(degrees > 5)[0]\naccuracies.append(accuracy(out.argmax(dim=1)[mask], data.y[mask]))\nsizes.append(len(mask))\n\n# Bar plot\nfig, ax = plt.subplots(figsize=(18, 9))\nax.set_xlabel('Node degree')\nax.set_ylabel('Accuracy score')\nax.set_facecolor('#EFEEEA')\nplt.bar(['0','1','2','3','4','5','>5'],\n        accuracies,\n        color='#0A047A')\nfor i in range(0, 7):\n    plt.text(i, accuracies[i], f'{accuracies[i]*100:.2f}%',\n             ha='center', color='#0A047A')\nfor i in range(0, 7):\n    plt.text(i, accuracies[i]//2, sizes[i],\n             ha='center', color='white')\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1Q-CkQnJqXCPaohouE_OkcA.png)\nThese results confirm our intuition: nodes with few neighbors are indeed **harder to classify**. This is due to the nature of GNNs: the more relevant connections you have, the more information you can aggregate.\n## Conclusion\nWhile they take longer to train, GATs are a **substantial improvement** over GCNs in terms of accuracy. The self-attention mechanism automatically calculates weighting factors instead of static coefficients to produce better embeddings. In this article,\n  * We learned about the **self-attention** mechanism applied to GNNs;\n  * We implemented and **compared** two **** architectures (a GCN and a GAT) in PyTorch Geometric;\n  * We visualized how and what the GAT learns with a **t-SNE** plot and the accuracy score for each degree;\n\n\nGATs are the de facto standard in a lot of GNN applications. However, their **slow training time** can become a problem when applied to massive graph datasets. Scalability is an important factor in deep learning: most often, more data can lead to better performance.\nIn the next article, we'll see **how to improve scalability** with mini-batching and a new GNN architecture called GraphSAGE.\nIf you enjoyed this tutorial, feel free to [**follow me on Twitter**](https://twitter.com/maximelabonne) for more GNN content. Thank you and see you in the next article! 📣\n## Next article\n> [**Chapter 3: Introduction to GraphSAGE in Python**](https://towardsdatascience.com/introduction-to-graphsage-in-python-a9e7f9ecf9d7)\n* * *\n_Learn more about machine learning and support my work with one click - become a Medium member here:_\n> [**Join Medium with my referral link - Maxime Labonne**](https://medium.com/@mlabonne/membership)\n_If you're already a member, you can_[ _follow me on Medium_](https://medium.com/@mlabonne) _._\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657499317_sxy3ci.png)\nWritten By\n## [Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[See all from Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Hands On Tutorials](https://towardsdatascience.com/tag/hands-on-tutorials/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Programming](https://towardsdatascience.com/tag/programming/)[Python](https://towardsdatascience.com/tag/python/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fgraph-attention-networks-in-python-975736ac5c0c%2F&title=Graph%20Attention%20Networks%3A%20Self-Attention%20Explained)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fgraph-attention-networks-in-python-975736ac5c0c%2F&title=Graph%20Attention%20Networks%3A%20Self-Attention%20Explained)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fgraph-attention-networks-in-python-975736ac5c0c%2F&text=Graph%20Attention%20Networks%3A%20Self-Attention%20Explained)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/12/0OYfxFtMkZQbxoKf5-scaled.jpg)](https://towardsdatascience.com/why-does-a-graphics-card-help-in-machine-learning-8f365593b22/)\n## [The Power of GPUs: Revolutionizing Computing and Unlocking New Frontiers](https://towardsdatascience.com/why-does-a-graphics-card-help-in-machine-learning-8f365593b22/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nHow Graphics Processing Units are Changing the Game in Machine Learning\n[Niklas Lang](https://towardsdatascience.com/author/niklas_lang/)\nDecember 9, 20228 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/08/1yIzNrFlesrFD89nItgNn5g-scaled.jpeg)](https://towardsdatascience.com/machine-learning-for-store-demand-forecasting-and-inventory-optimization-part-2-replenishment-6ded544be81b/)\n## [Machine Learning for Store Delivery Scheduling](https://towardsdatascience.com/machine-learning-for-store-demand-forecasting-and-inventory-optimization-part-2-replenishment-6ded544be81b/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nDesign Replenishment Strategies to Optimize Inventory and Reduce Number of Deliveries from your Warehouse\n[Samir Saci](https://towardsdatascience.com/author/s-saci95/)\nAugust 24, 20204 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/0ZOHt8XCfa7lQ4JaJ.jpg)](https://towardsdatascience.com/keep-your-code-secure-by-using-environment-variables-and-env-files-4688a70ea286/)\n## [Keep your code secure by using environment variables and env files](https://towardsdatascience.com/keep-your-code-secure-by-using-environment-variables-and-env-files-4688a70ea286/)\n[Programming](https://towardsdatascience.com/category/programming/)\nSecurely load a file containing all of our app's required, confidential data like passwords, tokens, etc\n[Mike Huls](https://towardsdatascience.com/author/mikehuls/)\nNovember 1, 20216 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/06/1ije1eqpsiIPorinyyyisTw-scaled.jpeg)](https://towardsdatascience.com/how-to-convert-your-custom-model-into-tensorrt-5a2ea1dec2e4/)\n## [How to Convert Your Custom Model into TensorRT](https://towardsdatascience.com/how-to-convert-your-custom-model-into-tensorrt-5a2ea1dec2e4/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nEase your TensorRT pain with TPAT\n[Ivan Ralašić](https://towardsdatascience.com/author/ralasic1/)\nJune 2, 20226 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2019/12/0FbLSIR1RZfTKvT0y-scaled.jpg)](https://towardsdatascience.com/top-4-numpy-functions-you-dont-know-about-probably-28fcd5d7174f/)\n## [Top 4 Numpy Functions You Don't Know About (Probably)](https://towardsdatascience.com/top-4-numpy-functions-you-dont-know-about-probably-28fcd5d7174f/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nFunctions that saved me a lot of time and frustration\n[Dario Radečić](https://towardsdatascience.com/author/radecicdario/)\nDecember 7, 20193 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/10/1pfsS4MwFtPbvDncP0VwlCA-scaled.jpeg)](https://towardsdatascience.com/clustering-a-daunting-task-for-a-data-scientist-5c7f362ccfb8/)\n## [Clustering - A Daunting Task for a Data Scientist](https://towardsdatascience.com/clustering-a-daunting-task-for-a-data-scientist-5c7f362ccfb8/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nGuidelines for designing a clustering model\n[Poornachandra Sarang](https://towardsdatascience.com/author/profsarang/)\nOctober 4, 20227 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/1NGcaaVHPBOkje_M6-Seu3w.png)](https://towardsdatascience.com/an-interactive-visualisation-for-your-graph-neural-network-explanations-1ac79d8ddd0a/)\n## [An Interactive Visualisation for your Graph Neural Network Explanations](https://towardsdatascience.com/an-interactive-visualisation-for-your-graph-neural-network-explanations-1ac79d8ddd0a/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA step-by-step guide on how to build one in five easy steps, with code already written for you.\n[Benjamin Lee](https://towardsdatascience.com/author/bl3e967/)\nJanuary 29, 202417 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1bMl96mIuj5ZN3m9HEqJpLQ.png)](https://towardsdatascience.com/how-to-embed-interactive-plotly-visualizations-in-folium-map-pop-ups-c69c818a8cd9/)\n## [How to Embed Interactive Plotly Visualizations in Folium Map Pop-ups](https://towardsdatascience.com/how-to-embed-interactive-plotly-visualizations-in-folium-map-pop-ups-c69c818a8cd9/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nA Detailed Hands-on Tutorial to Level Up your Python, Plotly, and Folium Skills\n[My Data Talk](https://towardsdatascience.com/author/insightsbees/)\nJanuary 17, 20228 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/04sQLj8DkSD-dbprY-scaled.jpg)](https://towardsdatascience.com/importing-python-functions-from-other-scripts-8769ca65a3da/)\n## [Importing Python Functions from Other Scripts](https://towardsdatascience.com/importing-python-functions-from-other-scripts-8769ca65a3da/)\n[Programming](https://towardsdatascience.com/category/programming/)\nStop copying and pasting functions into new scripts\n[Katy Hagerty](https://towardsdatascience.com/author/katyhagerty19/)\nApril 12, 20224 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n####  Recommended Articles \nClose\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n  * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n",
  "metadata": {
    "title": "Graph Attention Networks: Self-Attention Explained | Towards Data Science",
    "description": "A guide to GNNs with self-attention using PyTorch Geometric",
    "keywords": null,
    "author": "Maxime Labonne",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Graph Attention Networks: Self-Attention Explained",
    "og:description": "A guide to GNNs with self-attention using PyTorch Geometric",
    "og:url": "https://towardsdatascience.com/graph-attention-networks-in-python-975736ac5c0c/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1_p3JftpWqL3pJbRTxqflgg.gif",
    "og:image:width": "1280",
    "og:image:height": "720",
    "og:image:type": "image/gif",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Maxime Labonne",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "11 min read",
    "article:published_time": "2022-04-17T13:18:34.000Z",
    "article:modified_time": "2022-04-17T13:18:34.000Z",
    "article:author": "https://towardsdatascience.com/author/mlabonne/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_000.txt",
      "chars": 442
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_001.txt",
      "chars": 113
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_002.txt",
      "chars": 416
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_003.txt",
      "chars": 2780
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_004.txt",
      "chars": 190
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_005.txt",
      "chars": 60
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_006.txt",
      "chars": 856
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_007.txt",
      "chars": 189
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_008.txt",
      "chars": 81
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_009.txt",
      "chars": 812
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_010.txt",
      "chars": 362
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_011.txt",
      "chars": 268
    },
    {
      "language": "txt",
      "file": "attachments/701ad9412fe8/code/block_012.txt",
      "chars": 1062
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1h0KUV0wX5KckVNoiWy5P2A.png",
      "alt": "Node 4 is more important than node 3, which is more important than node 2 (image by author)",
      "local_path": "attachments/701ad9412fe8/images/000.png",
      "bytes": 68051,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/06DAVkX2UjMdH-ijK.png",
      "alt": "CiteSeer dataset (image by author, made with yEd Live)",
      "local_path": "attachments/701ad9412fe8/images/001.png",
      "bytes": 878299,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1nnsT7G3fb_EfuNS9NVSK4w.png",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/002.png",
      "bytes": 17825,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1AVQq-PJoU3GCffAuQJXXvA.png",
      "alt": "Image by author",
      "local_path": "attachments/701ad9412fe8/images/003.png",
      "bytes": 75217,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1chXBes9TjC6WMD637MYSIg.png",
      "alt": "Image by author",
      "local_path": "attachments/701ad9412fe8/images/004.png",
      "bytes": 63006,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1-1RHa4aDJRhdbNLtVuNM2w.png",
      "alt": "Image by author",
      "local_path": "attachments/701ad9412fe8/images/005.png",
      "bytes": 70114,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1ZY9HLOvwVXxLc-wMNH17fw.png",
      "alt": "Image by author",
      "local_path": "attachments/701ad9412fe8/images/006.png",
      "bytes": 73932,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/1A3C7OdAe1DrVEumyYvUdhw.png",
      "alt": "Multi-head attention (image by author)",
      "local_path": "attachments/701ad9412fe8/images/007.png",
      "bytes": 110712,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/12/0OYfxFtMkZQbxoKf5-scaled.jpg",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/008.jpg",
      "bytes": 357404,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/08/1yIzNrFlesrFD89nItgNn5g-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/009.jpeg",
      "bytes": 743869,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/0ZOHt8XCfa7lQ4JaJ.jpg",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/010.jpg",
      "bytes": 148513,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/06/1ije1eqpsiIPorinyyyisTw-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/011.jpeg",
      "bytes": 556638,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2019/12/0FbLSIR1RZfTKvT0y-scaled.jpg",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/012.jpg",
      "bytes": 60006,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/10/1pfsS4MwFtPbvDncP0VwlCA-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/013.jpeg",
      "bytes": 250151,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/01/1NGcaaVHPBOkje_M6-Seu3w.png",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/014.png",
      "bytes": 1642415,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/01/1bMl96mIuj5ZN3m9HEqJpLQ.png",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/015.png",
      "bytes": 249115,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2022/04/04sQLj8DkSD-dbprY-scaled.jpg",
      "alt": "",
      "local_path": "attachments/701ad9412fe8/images/016.jpg",
      "bytes": 594724,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://www.youtube.com/c/TowardsDataScience",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/programming/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/mlabonne/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/introduction-to-graphsage-in-python-a9e7f9ecf9d7"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/hands-on-tutorials/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/programming/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/python/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/why-does-a-graphics-card-help-in-machine-learning-8f365593b22/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/niklas_lang/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/machine-learning-for-store-demand-forecasting-and-inventory-optimization-part-2-replenishment-6ded544be81b/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/s-saci95/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/keep-your-code-secure-by-using-environment-variables-and-env-files-4688a70ea286/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/mikehuls/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://openmoji.org/"
    },
    {
      "type": "external",
      "href": "https://creativecommons.org/licenses/by-sa/4.0/"
    },
    {
      "type": "external",
      "href": "https://colab.research.google.com/drive/1B0vLpH_gSfrOLgsc2UZVyXrcofzA-t0L?usp=sharing"
    },
    {
      "type": "external",
      "href": "https://www.yworks.com/yed-live/"
    },
    {
      "type": "external",
      "href": "http://www.kamalnigam.com/papers/cora-jnl.pdf"
    },
    {
      "type": "external",
      "href": "https://en.wikipedia.org/wiki/Bag-of-words_model"
    },
    {
      "type": "external",
      "href": "https://clgiles.ist.psu.edu/papers/DL-1998-citeseer.pdf"
    },
    {
      "type": "external",
      "href": "http://eliassi.org/papers/ai-mag-tr08.pdf"
    },
    {
      "type": "external",
      "href": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/1710.10903.pdf"
    },
    {
      "type": "external",
      "href": "https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/pdf/1603.08861.pdf"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/1710.10903"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/1706.03762"
    },
    {
      "type": "external",
      "href": "https://arxiv.org/abs/2105.14491"
    },
    {
      "type": "external",
      "href": "https://github.com/PetarV-/GAT"
    },
    {
      "type": "external",
      "href": "https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:48Z",
  "source_url": "https://towardsdatascience.com/graph-attention-networks-in-python-975736ac5c0c/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}