{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "LLM architecture (part 2): Inside a Transformer Block",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/llm-architecture-part-2-inside-a-transformer-block",
  "publish_date": "2026-07-14",
  "corpus_stage": "",
  "notes": "",
  "id": "cce07b5e3cda",
  "collected_at": "2026-10-02T14:11:45",
  "title": "GPT Transformer Block: Normalization, GELU and Activation",
  "body_markdown": "LLM architecture (part 2): Inside a Transformer Block\nUnderstanding normalization, linear layers, GELU activation, and how they work together inside GPT’s main processing unit\nUpdated\nJuly 14, 2026\n•\n8\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nBuilding a GPT Model From Scratch\nIn the previous article, we looked at tokenization and embeddings, which convert a finite vocabulary into continuous vectors that we can work with mathematically. We also explored attention, which helps the model understand how different tokens relate to and influence one another in context.\nIn this article, we will look at a few more building blocks of a GPT-style model: normalization, linear layers, and activation functions. We will then bring them together inside the transformer block—the main processing unit that GPT repeats throughout the model.\nNormalization\nAt its simplest, normalization is about putting different numerical features onto a more consistent scale so they can be compared or combined more meaningfully.\nImagine ranking football players using three features:\nGoals scored\nAssists\nPass-completion rate\nThese values naturally live on different scales, so simply adding the raw numbers together would not tell us much.\nOne way to solve this is to normalize each feature. For goals, we could look at the minimum and maximum scored during a tournament and convert each player’s performance into a score between 0 and 1. We could then do the same for assists and pass-completion rate.\nInstead of directly comparing 20 goals, 8 assists, and 90% pass completion, we would now have three scores on a common scale.\nThis football example uses min-max normalization. Layer normalization uses a different mathematical method, but the underlying motivation is similar: keeping values on a well-behaved scale.\nIn a GPT-style model, each token is represented by hundreds of numbers across its embedding dimensions. You can loosely think of these dimensions as the model’s internal features for representing that token.\nLayer normalization works on each token independently and normalizes across these embedding dimensions.\nclass LayerNorm(nn.Module):\n    def __init__(self, emb_dim):\n        super().__init__()\n        self.eps = 1e-5\n        self.scale = nn.Parameter(torch.ones(emb_dim))\n        self.shift = nn.Parameter(torch.zeros(emb_dim))\n\n    def forward(self, x):\n        mean = x.mean(dim=-1, keepdim=True)\n        var = x.var(dim=-1, keepdim=True, unbiased=False)\n\n        norm_x = (x - mean) / torch.sqrt(var + self.eps)\n\n        return self.scale * norm_x + self.shift\nHere,\ndim=-1\ntells PyTorch to calculate the mean and variance across the final dimension—the embedding dimension. It does not normalize across different tokens or across the batch.\nThe small\neps\nvalue prevents division by zero, while\nscale\nand\nshift\nare trainable parameters. They allow the model to adjust the normalized values rather than forcing every representation to remain at exactly the same scale.\nWhy does this matter?\nAs token representations move through different model layers, their values can drift and end up on very different scales. Layer normalization acts as a kind of numerical housekeeping, bringing each token’s representation back into a more predictable range before the next major transformation.\nWe use normalization frequently inside a GPT model. Within every transformer block, token representations are normalized before the attention mechanism and again before the feed-forward network. After the full stack of transformer blocks, GPT-2 also applies one final normalization layer.\nExactly where normalization appears is an architectural design choice. Some transformer architectures place it before the major sub-layers, while others place it afterward.\nThe GPT-style architecture we are building uses\npre-layer normalization\n, meaning normalization happens before attention and before the feed-forward network.\nLinear Layers\nAt a high level, a linear layer takes a set of input values and transforms them into a new set of values by combining the inputs with different weights.\nContinuing our sports analogy, suppose we start with three features for scouting football players:\nGoals\nAssists\nPass-completion rate\nWe may want to transform them into three different scores:\nAttacking ability\nPlaymaking ability\nOverall contribution\nEach score uses the same inputs, but with different weights. For example:\nAttacking ability = 0.8 × goals score + 0.4 × assists score\n+ 0.1 × passing score\n\nPlaymaking ability = 0.2 × goals score + 0.6 × assists score\n+ 0.8 × passing score\n\nOverall contribution = 0.4 × goals score + 0.4 × assists score + 0.6 × passing score\nA GPT model does something similar.\nThe same token representation needs to be interpreted differently at different stages of the network. Linear layers create these new views by learning different weighted combinations of the existing representation.\nImportantly, the weights inside a linear layer are\ntrainable parameters\n. During training, the model continually adjusts them to learn which combinations of features are useful.\nWe already saw linear layers inside the attention mechanism, where separate projections transform the same token representation into queries, keys, and values.\nThey appear again inside the feed-forward network, where the representation is expanded into a larger intermediate space and then brought back to its original embedding size.\nActivation Functions\nThere is an important limitation to linear layers: stacking them alone does not make the network much more expressive.\nWithout an activation function between them, multiple linear transformations can effectively collapse into a single linear transformation. Activation functions introduce\nnonlinearity\n, allowing the model to learn much richer patterns.\nIntuitively, a useful activation function should:\nIntroduce nonlinearity\nReduce weaker or less useful signals\nPreserve sufficiently strong signals\nSupport stable learning through useful gradients\nOur GPT model uses\nGELU\n, or the Gaussian Error Linear Unit:\nclass GELU(nn.Module):\n    def __init__(self):\n        super().__init__()\n\n    def forward(self, x):\n        return 0.5 * x * (\n            1 + torch.tanh(\n                torch.sqrt(torch.tensor(2.0 / torch.pi))\n                * (x + 0.044715 * torch.pow(x, 3))\n            )\n        )\nWhat GELU looks like\nUnlike ReLU, which completely removes negative values, GELU applies a smoother filter.\nStrong positive activations mostly pass through, strongly negative ones are largely suppressed, and values close to zero are only partially retained. This gives the model a more gradual way to decide which signals are useful.\nInside the GPT-2 feed-forward network, GELU sits between two linear layers:\nself.layers = nn.Sequential(\n    # Expand and then reduce the embedding dimensions\n    nn.Linear(cfg[\"emb_dim\"], 4 * cfg[\"emb_dim\"]),\n    GELU(),\n    nn.Linear(4 * cfg[\"emb_dim\"], cfg[\"emb_dim\"]),\n)\nThe first linear layer expands each token representation into a higher-dimensional space. For GPT-2 124M, this means expanding from 768 dimensions to 3,072.\nGELU then filters the resulting activations before the second linear layer projects them back to the original 768-dimensional embedding space.\nI think of this as temporarily giving the model a larger workspace. It creates a wider set of candidate features, filters them through GELU, and then compresses the useful information back into the standard representation required by the rest of the model.\nThe size of this intermediate space is another architectural design choice. A wider layer gives the model more capacity, but it also increases the number of trainable parameters and the computational cost.\nGPT-2 uses an intermediate dimension four times larger than the embedding dimension as a practical balance.\nOne important distinction is:\nAttention allows information to move between tokens, while the activation processes each token independently.\nThe Transformer Block\nWe now have all the components required for a transformer block.\nA transformer block is the main processing unit inside a GPT model. It helps each token gather relevant information from the sequence and then transforms the resulting representation into something richer.\nIts overall flow looks like this:\nReturning one final time to our sports analogy, we can think of each transformer block as another round of player evaluation.\nThe first normalization layer puts the player’s different features onto a stable numerical scale. Attention then evaluates the player in the context of teammates, opponents, and the wider game—not just through individual statistics.\nThe original player profile is added back through a shortcut connection, helping preserve the information we already had. The updated profile is then normalized again before entering the feed-forward network.\nThe feed-forward network reinterprets this updated profile. Its linear layers create new combinations of the available features, while GELU filters which signals should meaningfully influence the final assessment.\nAnother shortcut connection preserves the earlier representation instead of forcing every transformation to rebuild it from scratch.\nIn more technical terms, the sequence is:\nNormalize the token representation.\nExchange information between tokens through attention.\nAdd back the original representation.\nNormalize the updated representation.\nProcess each token independently through the feed-forward network.\nAdd back the representation from before the feed-forward network.\nNote that we also have a Dropout layer after step 2 and step 4 - that drops randomly some information to prevent overfitting while training. I have removed this detail to keep it simple.\nGPT-2 repeats this process across multiple transformer blocks. With every block, the representation of each token becomes more informed by both its own features and the context provided by the surrounding sequence.\nAt this point, we seem to have all the raw material needed for our GPT-2 model.\nIn the next article, we will assemble these components, walk through the end-to-end flow, and calculate how many trainable parameters we end up with.\nReferences\nSebastian Raschka —\nBuild a Large Language Model (From Scratch)\nVaswani et al. —\nAttention Is All You Need\nRadford et al. —\nLanguage Models are Unsupervised Multitask Learners\nBa, Kiros, and Hinton —\nLayer Normalization\nXiong et al. —\nOn Layer Normalization in the Transformer Architecture\nHendrycks and Gimpel —\nGaussian Error Linear Units (GELUs)\nAman Jain — Project repository.\nThe code and notebooks accompanying this learning journey are available on my\nGitHub profile\n.\n#\ngpt\n#\nlarge-language-models\n#\ntransformer\n#\ndeep-learning\n68\nviews",
  "metadata": {
    "title": "GPT Transformer Block: Normalization, GELU and Activation",
    "description": "Learn how layer normalization, linear layers, GELU activation, and feed-forward networks work together inside a GPT-style transformer block.",
    "keywords": null,
    "author": null,
    "og:title": "GPT Transformer Block: Normalization, GELU and Activation",
    "og:description": "Learn how layer normalization, linear layers, GELU activation, and feed-forward networks work together inside a GPT-style transformer block.",
    "og:url": "https://curious-pm.hashnode.dev/llm-architecture-part-2-inside-a-transformer-block",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5cf78424-15fa-4fb2-8ada-cc9dd86f1ab6.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "GPT Transformer Block: Normalization, GELU and Activation",
    "twitter:description": "Learn how layer normalization, linear layers, GELU activation, and feed-forward networks work together inside a GPT-style transformer block.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5cf78424-15fa-4fb2-8ada-cc9dd86f1ab6.png",
    "article:published_time": "2026-07-14T03:38:14.247Z",
    "article:modified_time": "2026-07-14T04:23:41.973Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/cce07b5e3cda/code/body-000.txt",
      "chars": 459,
      "language": "text"
    },
    {
      "file": "attachments/cce07b5e3cda/code/body-001.txt",
      "chars": 252,
      "language": "text"
    },
    {
      "file": "attachments/cce07b5e3cda/code/body-002.txt",
      "chars": 288,
      "language": "text"
    },
    {
      "file": "attachments/cce07b5e3cda/code/body-003.txt",
      "chars": 199,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5cf78424-15fa-4fb2-8ada-cc9dd86f1ab6.png",
      "bytes": 1371053,
      "reused": true,
      "local_path": "attachments/cce07b5e3cda/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5cf78424-15fa-4fb2-8ada-cc9dd86f1ab6.png",
      "alt": "LLM architecture (part 2): Inside a Transformer Block",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/6227054b-b107-4d2e-bf6b-de932179aa1a.png",
      "bytes": 38428,
      "reused": true,
      "local_path": "attachments/cce07b5e3cda/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/6227054b-b107-4d2e-bf6b-de932179aa1a.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/219be7e4-20ce-4f4d-8631-c0c3de7b4037.png",
      "bytes": 51660,
      "reused": true,
      "local_path": "attachments/cce07b5e3cda/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/219be7e4-20ce-4f4d-8631-c0c3de7b4037.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/1b1246be-cbb5-4fab-9770-a4dbc0657d5c.png",
      "bytes": 26527,
      "reused": true,
      "local_path": "attachments/cce07b5e3cda/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/1b1246be-cbb5-4fab-9770-a4dbc0657d5c.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/llm-architecture-part-2-inside-a-transformer-block.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/building-gpt-from-scratch",
      "label": "Part of series Building a GPT Model From Scratch",
      "type": "a"
    },
    {
      "href": "https://www.manning.com/books/build-a-large-language-model-from-scratch",
      "label": "Build a Large Language Model (From Scratch)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03762",
      "label": "Attention Is All You Need",
      "type": "a"
    },
    {
      "href": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "label": "Language Models are Unsupervised Multitask Learners",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1607.06450",
      "label": "Layer Normalization",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2002.04745",
      "label": "On Layer Normalization in the Transformer Architecture",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1606.08415",
      "label": "Gaussian Error Linear Units (GELUs)",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454",
      "label": "GitHub profile",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/gpt",
      "label": "# gpt",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/large-language-models",
      "label": "# large-language-models",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/transformer",
      "label": "# transformer",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/deep-learning",
      "label": "# deep-learning",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:25",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:56Z",
  "source_url": "https://curious-pm.hashnode.dev/llm-architecture-part-2-inside-a-transformer-block",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    },
    {
      "status": "ok",
      "url": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "bytes": 582775,
      "reused": true,
      "local_path": "papers/repair/3c62359064baa28a7743.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1607.06450",
      "bytes": 612446,
      "reused": true,
      "local_path": "papers/repair/52a2360d9b658baa64ff.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2002.04745",
      "bytes": 779713,
      "reused": true,
      "local_path": "papers/repair/eeee73d530f66e641d46.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1606.08415",
      "bytes": 2767669,
      "reused": true,
      "local_path": "papers/repair/74f293f3c925776e7545.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 10853,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/cce07b5e3cda/pages/source.html",
  "local_html": "attachments/cce07b5e3cda/pages/article.html",
  "body_base_path": "attachments/cce07b5e3cda/pages/",
  "body_markdown_path": "attachments/cce07b5e3cda/pages/article.md"
}