{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "How Much Do Attention Heads and Activation Functions Really Matter?",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/how-much-do-attention-heads-and-activation-functions-really-matter",
  "publish_date": "2026-07-21",
  "corpus_stage": "",
  "notes": "",
  "id": "29a394d6dafe",
  "collected_at": "2026-10-02T14:11:44",
  "title": "Do Attention Heads and Activation Functions Matter in GPT?",
  "body_markdown": "How Much Do Attention Heads and Activation Functions Really Matter?\nControlled experiments with attention-head counts and activation functions in a small GPT model.\nUpdated\nJuly 21, 2026\n•\n7\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nInside GPT: Experiments and Trade-offs\nTL;DR\nI tested two architectural choices in a small GPT model while keeping the rest of the setup stable:\nAttention heads:\n1, 4, and 8 heads under fixed-token and fixed-time budgets.\nActivation functions:\nGELU, ReLU, ReLU², and Tanh under a fixed-token budget.\nAt equal token exposure, increasing the number of attention heads produced only a small improvement in validation loss. Under the fixed-time budget, eight heads performed noticeably better—but primarily because that configuration processed more tokens.\nFor activation functions, ReLU² achieved the lowest validation loss, while Tanh performed the worst. Their internal behavior differed dramatically: ReLU and ReLU² were highly sparse, ReLU² produced extremely large positive activations, and Tanh bounded everything between −1 and 1.\nThe broader takeaway was simple:\nArchitectural choices can significantly change how a model computes without producing equally large differences in model quality.\nThe complete code for these experiments is available in my\nGitHub repository\n.\nWhy Experiment With Architecture?\nSo far in this series, I have experimented with learning rates, context length, model size, and fine-tuning strategies. Most of those experiments changed how the model was trained.\nThis time, I wanted to look inside the model itself.\nA GPT model contains several architectural decisions: how attention is divided, which activation function is used, where normalization happens, and how position is represented.\nTesting all of them together would make it difficult to identify what caused each result. I therefore focused on two controlled experiments:\nChanging the number of attention heads\nChanging the activation function\nThe baseline model contained approximately\n13.67 million parameters\n, used a context length of 128, and was trained on WikiText data.\nExperiment 1: How Many Attention Heads Does a Small GPT Need?\nMulti-head attention divides the model’s embedding space into smaller attention subspaces.\nThe embedding dimension remained fixed at 128:\nHeads\nDimension per head\n1\n128\n4\n32\n8\n16\nThe parameter count remained identical across all three models. The only difference was how the attention representation was divided.\nI evaluated the configurations under two budgets.\nHypothesis\nUnder a\nfixed-token budget\n, I expected the differences to be small. Every model would see the same amount of training data and complete the same number of optimizer updates.\nUnder a\nfixed-time budget\n, I expected multiple heads to potentially benefit from better hardware parallelism. However, I did not expect head count alone to guarantee higher computational efficiency.\nAt first glance, multi-head attention can look like a computational shortcut: instead of performing one large attention operation, we split the embedding space into smaller subspaces and process them separately. Conceptually, that is true—but it does not automatically reduce the total amount of work.\nPractical throughput can still change because of:\nHardware parallelism\nMemory-access patterns\nTensor shapes\nFramework and kernel overhead\nIn my fixed-time run, additional heads appeared to benefit from the execution environment. However, the timing results were not consistent across both budgets, so I would not generalize that more heads are inherently faster.\nFixed-Token Results\nEach model processed exactly\n200,192 tokens\nacross\n782 optimizer steps\n.\nEight heads achieved the lowest validation loss, followed by four heads and one head.\nHowever, the difference between the best and worst result was only\n0.027 loss points\n. Accuracy also improved only slightly, from approximately 12.07% with one head to 12.29% with eight heads.\nThis broadly supported the hypothesis. Multiple heads may have provided a modest representational advantage, but head count was not a major performance lever when training exposure was held constant.\nFixed-Time Results\nEach model received approximately\n300 seconds\nof measured training time.\nThe eight-head model processed the most tokens and achieved the lowest validation loss. It processed approximately:\n21% more tokens than the one-head model\n12% more tokens than the four-head model\nThe fixed-token experiment had shown only a small architectural advantage for eight heads. This suggests that much of the larger fixed-time improvement came from processing more training data—not simply from having more heads.\nFor this small GPT model, attention-head count appeared to be a\nsecondary architectural choice\n. Performance under equal token exposure changed only modestly, while the amount of training completed within the available time had a larger effect.\nExperiment 2: Do Activation Functions Change How GPT Learns?\nI chose four functions that represent meaningfully different behaviors.\nGELU\nGELU is a strong modern baseline for Transformer models. It is smooth, allows small negative outputs, and does not abruptly shut values off. I would use it when I want a balanced and reliable default.\nReLU\nReLU sets negative inputs to zero while allowing positive inputs to grow linearly. This creates sparse representations, but it can also discard useful information on the negative side.\nReLU²\nReLU² retains ReLU’s hard zeroing behavior but squares positive values. Small positive values stay modest, while larger values are amplified aggressively. I would use it to test whether emphasizing a small number of strong signals improves learning—while monitoring activation and gradient stability.\nTanh\nTanh is smooth and dense but bounded between −1 and 1. This controls activation magnitude, but it can also saturate. Once inputs become large, further input changes produce increasingly small output changes. I would primarily use it when bounded outputs are important or as an extreme comparison against unbounded functions.\nAll four models had identical parameter counts and processed exactly\n200,192 tokens\n.\nHypothesis\nI expected GELU to remain a stable baseline.\nReLU might create greater sparsity but lose information by discarding negative values.\nReLU² might learn efficiently by amplifying strong positive signals, but it could also create unstable activation magnitudes.\nTanh represented the opposite extreme. Its bounded outputs could control activation growth, but saturation might weaken learning.\nResults\nReLU² achieved the lowest validation loss, slightly ahead of GELU. The difference was small, so I would describe it as a promising result rather than a decisive win.\nIts internal behavior was much more dramatic.\nApproximately\n76% of ReLU² outputs were zero\n, but its maximum activation exceeded 203, compared with around 12 for GELU and ReLU. It suppressed most activations while strongly amplifying a small number of positive signals.\nTanh behaved exactly like the extreme comparison I wanted. Its activations remained bounded between −1 and 1, but it produced the highest validation loss, lowest accuracy, and slowest throughput.\nBounding activations prevented large values, but it may also have compressed useful distinctions between strongly activated features. Tanh showed that controlling activation magnitude too aggressively can restrict learning.\nThe activation functions changed the model’s internal behavior far more dramatically than they changed its final validation loss.\nFinal Takeaway\nFor this setup, eight attention heads and ReLU² achieved the lowest validation losses.\nHowever, neither result was decisive.\nChanging the number of heads produced only a modest improvement when token exposure was controlled. Changing the activation function created radically different sparsity and activation magnitudes, but the final loss differences remained relatively small.\nThese results describe early learning for one small model, dataset, training budget, and random seed. Repeating the closest configurations across multiple seeds would be necessary before treating the rankings as general conclusions.\nAppendix: Related Papers and Reading\n1.\nMoH: Multi-Head Attention as Mixture-of-Head Attention\n— 2024\n2.\nPeri-LN: Revisiting Layer Normalization in the Transformer Architecture\n— 2025\n3.\n2 OLMo 2 Furious\n— 2024/2025\n4.\nRound and Round We Go! What Makes Rotary Positional Encodings Useful?\n— 2024\n5.\nGLU Variants Improve Transformer\n— 2020\n#\nactivation-function\n#\nlarge-language-models\n#\nattention-mechanism\n#\ndeep-learning\n9\nviews",
  "metadata": {
    "title": "Do Attention Heads and Activation Functions Matter in GPT?",
    "description": "I compare attention-head counts and four activation functions in a small GPT model to understand effects on loss, throughput, sparsity, activation.",
    "keywords": null,
    "author": null,
    "og:title": "Do Attention Heads and Activation Functions Matter in GPT?",
    "og:description": "I compare attention-head counts and four activation functions in a small GPT model to understand effects on loss, throughput, sparsity, activation.",
    "og:url": "https://curious-pm.hashnode.dev/how-much-do-attention-heads-and-activation-functions-really-matter",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d0366832-7857-4788-a1a4-1ce757b9075c.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Do Attention Heads and Activation Functions Matter in GPT?",
    "twitter:description": "I compare attention-head counts and four activation functions in a small GPT model to understand effects on loss, throughput, sparsity, activation.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d0366832-7857-4788-a1a4-1ce757b9075c.png",
    "article:published_time": "2026-07-21T03:29:34.419Z",
    "article:modified_time": "2026-07-21T03:29:34.419Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d0366832-7857-4788-a1a4-1ce757b9075c.png",
      "bytes": 1173586,
      "reused": true,
      "local_path": "attachments/29a394d6dafe/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d0366832-7857-4788-a1a4-1ce757b9075c.png",
      "alt": "How Much Do Attention Heads and Activation Functions Really Matter?",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/25a85345-84ed-4ba8-bd9f-13e5f354d825.png",
      "bytes": 18315,
      "reused": true,
      "local_path": "attachments/29a394d6dafe/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/25a85345-84ed-4ba8-bd9f-13e5f354d825.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d6cec14c-fd4c-4b61-b08d-734c2fdfc7e6.png",
      "bytes": 32478,
      "reused": true,
      "local_path": "attachments/29a394d6dafe/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d6cec14c-fd4c-4b61-b08d-734c2fdfc7e6.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/f4dc6566-275b-4fd1-a892-3507f1009f22.png",
      "bytes": 30879,
      "reused": true,
      "local_path": "attachments/29a394d6dafe/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/f4dc6566-275b-4fd1-a892-3507f1009f22.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/6b5b2d4e-6c1d-4d10-a584-8f0b5da1506f.png",
      "bytes": 47997,
      "reused": true,
      "local_path": "attachments/29a394d6dafe/images/004.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/6b5b2d4e-6c1d-4d10-a584-8f0b5da1506f.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/ef365c44-f41f-4f82-999b-2539fa00ffbe.png",
      "bytes": 38421,
      "reused": true,
      "local_path": "attachments/29a394d6dafe/images/005.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/ef365c44-f41f-4f82-999b-2539fa00ffbe.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/c06cc713-6f42-4c88-bf30-eff05da1af96.png",
      "bytes": 13838,
      "reused": true,
      "local_path": "attachments/29a394d6dafe/images/006.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/c06cc713-6f42-4c88-bf30-eff05da1af96.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/51f1f298-b7dc-4a1d-9447-80132bd93513.png",
      "bytes": 13576,
      "reused": true,
      "local_path": "attachments/29a394d6dafe/images/007.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/51f1f298-b7dc-4a1d-9447-80132bd93513.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/how-much-do-attention-heads-and-activation-functions-really-matter.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/inside-gpt-experiments-and-trade-offs",
      "label": "Part of series Inside GPT: Experiments and Trade-offs",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work",
      "label": "GitHub repository",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2410.11842",
      "label": "MoH: Multi-Head Attention as Mixture-of-Head Attention",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2502.02732",
      "label": "Peri-LN: Revisiting Layer Normalization in the Transformer Architecture",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2501.00656",
      "label": "2 OLMo 2 Furious",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2410.06205",
      "label": "Round and Round We Go! What Makes Rotary Positional Encodings Useful?",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2002.05202",
      "label": "GLU Variants Improve Transformer",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/activation-function",
      "label": "# activation-function",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/large-language-models",
      "label": "# large-language-models",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/attention-mechanism",
      "label": "# attention-mechanism",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/deep-learning",
      "label": "# deep-learning",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:08:58",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:54Z",
  "source_url": "https://curious-pm.hashnode.dev/how-much-do-attention-heads-and-activation-functions-really-matter",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2410.11842",
      "bytes": 2631469,
      "reused": false,
      "local_path": "papers/repair/faddba30f95b62abd8e9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2502.02732",
      "bytes": 14105021,
      "reused": false,
      "local_path": "papers/repair/737aaf848ed66daa4558.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2501.00656",
      "bytes": 4213966,
      "reused": true,
      "local_path": "papers/repair/c3650555ef74a19c70cd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2410.06205",
      "bytes": 3280901,
      "reused": false,
      "local_path": "papers/repair/21fb9dfaaf36e3774b20.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2002.05202",
      "bytes": 109135,
      "reused": false,
      "local_path": "papers/repair/616318f661e747a41449.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 8733,
  "expected_body_images": 8,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/29a394d6dafe/pages/source.html",
  "local_html": "attachments/29a394d6dafe/pages/article.html",
  "body_base_path": "attachments/29a394d6dafe/pages/",
  "body_markdown_path": "attachments/29a394d6dafe/pages/article.md"
}