{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "LLM architecture (part 3): Assembling model",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-3-assembling-the-model",
  "publish_date": "2026-07-14",
  "corpus_stage": "",
  "notes": "",
  "id": "83a44523f22c",
  "collected_at": "2026-10-02T14:11:45",
  "title": "Assembling a GPT Model from Scratch | Part 3",
  "body_markdown": "LLM architecture (part 3): Assembling model\nBringing embeddings, transformer blocks and the output layer together—and tracing tensor dimensions and the model’s parameter count.\nUpdated\nJuly 16, 2026\n•\n9\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nBuilding a GPT Model From Scratch\nAt this point, we have almost everything we need to assemble a baseline GPT model. In this article, I will:\nBring the different components together and show how they form the complete model.\nDo a quick walkthrough of the tensor dimensions at each stage to understand how the data changes as it moves through the network.\nCalculate the total number of trainable parameters in the model. This parameter count is typically what people refer to when they describe the size of a model.\nAssembling the components\nLet us begin with a quick review of the components introduced in the previous articles:\nTokenization and embeddings\nconvert natural-language text into tokens and then represent those tokens as numerical vectors.\nAttention\nallows each token to incorporate context from the other relevant tokens in the sequence.\nActivation layers\nhelp the model identify and strengthen useful patterns within the data.\nNormalization layers\nkeep the values flowing through the model within a stable range.\nDropout layers\nrandomly remove some information during training to reduce overfitting. I have left dropout out of the diagram below to keep it simple.\nThe following diagram represents how these components fit together in the baseline model:\nThe following piece of code does the heavy lifting:\nclass GPTModel(nn.Module):\n    def __init__(self, cfg):\n        super().__init__()\n\n        self.tok_emb = nn.Embedding(\n            cfg[\"vocab_size\"],\n            cfg[\"emb_dim\"]\n        )\n        self.pos_emb = nn.Embedding(\n            cfg[\"context_length\"],\n            cfg[\"emb_dim\"]\n        )\n        self.drop = nn.Dropout(cfg[\"drop_rate\"])\n\n        # Transformer block repeated n_layers times\n        self.trf_blocks = nn.Sequential(\n            *[\n                TransformerBlock(cfg)\n                for _ in range(cfg[\"n_layers\"])\n            ]\n        )\n\n        # Final normalization layer\n        self.final_norm = LayerNorm(cfg[\"emb_dim\"])\n        self.out_head = nn.Linear(\n            cfg[\"emb_dim\"],\n            cfg[\"vocab_size\"],\n            bias=False\n        )\n\n    def forward(self, in_idx):\n        batch_size, seq_len = in_idx.shape\n\n        tok_embeds = self.tok_emb(in_idx)\n        pos_embeds = self.pos_emb(\n            torch.arange(seq_len, device=in_idx.device)\n        )\n\n        x = self.drop(tok_embeds + pos_embeds)\n        x = self.trf_blocks(x)\n        x = self.final_norm(x)\n        logits = self.out_head(x)\n\n        return logits\nTracking tensor dimensions\nRecall the model configuration introduced in Part 1:\nGPT_CONFIG_124M = {\n    \"vocab_size\": 50_257,\n    \"context_length\": 1_024,\n    \"emb_dim\": 768,\n    \"n_heads\": 12,\n    \"n_layers\": 12,\n    \"drop_rate\": 0.1,\n    \"qkv_bias\": False,\n}\nIn addition, let us assume that every input has been padded or truncated to\ncontext_length\nand that each input batch contains\nbatch_size\ninputs. Intuitively, the final output should give us\nbatch_size\npredicted tokens—one for each input.\nTo keep things simple, we do not need to follow every tensor transformation inside the model. At a high level, the input begins as a sequence of token IDs. Each token is then converted into an\nemb_dim\n-dimensional vector, and this overall shape is preserved as the data moves through the transformer blocks.\nInside each block, the attention and activation components temporarily reshape or expand the data, but the output returns to the same dimensions.\nFinally, the output layer converts each token representation into a score for every token in the vocabulary.\nStage\nWhat happens\nOutput dimensions\nTokenization\nEach input is converted into a sequence of token IDs.\nbatch_size * context_length\nToken embeddings\nEach token ID is converted into a vector in the embedding space.\nbatch_size * context_length * emb_dim\nPositional embeddings\nPosition information is added to each token. The overall dimensions remain unchanged.\nbatch_size * context_length * emb_dim\nInput to transformer blocks\nToken and positional embeddings are combined and passed into the transformer blocks.\nbatch_size * context_length * emb_dim\nAttention\nThe model creates query, key and value representations to understand relationships between tokens. The attention output returns to the original dimensions.\nbatch_size * context_length * emb_dim\nActivation expansion\nThe representation is temporarily expanded to four times the embedding dimension.\nbatch_size * context_length * (emb_dim * 4)\nActivation projection\nThe expanded representation is reduced back to the original embedding dimension.\nbatch_size * context_length * emb_dim\nOutput of each transformer block\nThe original tensor dimensions are preserved before the data enters the next block.\nbatch_size * context_length * emb_dim\nOutput layer\nEach token representation is converted into a score for every token in the vocabulary.\nbatch_size * context_length * vocab_size\nFinal-position logits\nThe vocabulary scores from the final token position are selected for next-token prediction.\nbatch_size * vocab_size\nToken prediction\nOne next token is selected for each input in the batch.\nbatch_size * 1\nCount of trainable parameters\nThere is one final piece to examine before we complete the model: the total number of trainable parameters.\nThese are the values the model adjusts during training as it learns patterns from the data. The overall parameter count is also commonly used to describe the size of a model. For example, GPT-2 Small is referred to as a 124-million-parameter model.\nLet us create an instance of our GPT model and count its parameters:\ntorch.manual_seed(123)\nmodel = GPTModel(GPT_CONFIG_124M)\n\n#setting output layer to use same weights as embedding layer. \n# as it is same dictionary. \nmodel.out_head.weight = model.tok_emb.weight\n\n\ntied_param_count = sum(p.numel() for p in model.parameters())\nprint(f\"Total number of unique parameters: {tied_param_count}\")\nThe model produces the following output:\nTotal number of parameters: 124402944\nLet us examine this in more detail:\nModel component\nTrainable parameters\nHow it is calculated\nToken embeddings\n38,597,376\n50,257 * 768\n: one 768-dimensional embedding vector for each of the 50,257 tokens in the vocabulary\nPositional embeddings\n786,432\n1,024 * 768\n: one 768-dimensional embedding vector for each of the 1,024 positions in the context window\n12 transformer blocks\n85,017,600\n7,084,800 * 12\n: 12 transformer blocks, each containing attention, activation and normalization layers\nFinal normalization layer\n1,536\n768 + 768\n: one trainable scale value and one trainable shift value for each of the 768 embedding dimensions\nOutput head\n0\nre-using token embedding weights.\nTotal in the current implementation\n124402944\nApproximately\n124 million parameters\nEach transformer block contains approximately 7.1 million parameters:\nComponent within one transformer block\nParameters\nHow it is calculated\nAttention\n2,359,296\nFour projection matrices—query, key, value and output—each containing\n768 * 768\nweights\nFirst activation layer\n2,362,368\n768 * 3,072\nweights +\n3,072\nbias values; expands each token representation to four times the embedding dimension\nSecond activation layer\n2,360,064\n3,072 * 768\nweights +\n768\nbias values; projects the expanded representation back to 768 dimensions\nTwo normalization layers\n3,072\nEach layer contains\n768\nscale values +\n768\nshift values, giving\n1,536 * 2\nparameters\nTotal per block\n7,084,800\nAttention + two activation layers + two normalization layers\nThis gives us approximately\n124 million unique trainable parameters\n, which is the convention used when describing GPT-2 Small as a 124M model.\nBefore we close, let us look at the structure of the model we have built. Most of these components should now look familiar: token and positional embeddings, multi-head attention, activation layers, normalization, and the final output layer.\nmodel = GPTModel(GPT_CONFIG_124M)\n\nmodel.load_state_dict(\n    torch.load(\"model.pth\", map_location=device)\n)\n\nmodel.eval()\nprint(model)\nThe output below has been shortened for readability. Each of the 12 transformer blocks follows the same structure.\nGPTModel(\n  (tok_emb): Embedding(50257, 768)\n  (pos_emb): Embedding(256, 768)\n  (drop): Dropout(p=0.1)\n\n  (trf_blocks): Sequential(\n    (0): TransformerBlock(\n      (attn): MultiHeadAttention(\n        (W_query): Linear(768 → 768, bias=False)\n        (W_key): Linear(768 → 768, bias=False)\n        (W_value): Linear(768 → 768, bias=False)\n        (out_proj): Linear(768 → 768, bias=False)\n        (dropout): Dropout(p=0.1)\n      )\n\n      (ff): FeedForward(\n        (layers): Sequential(\n          (0): Linear(768 → 3072)\n          (1): GELU()\n          (2): Linear(3072 → 768)\n        )\n      )\n\n      (norm1): LayerNorm()\n      (norm2): LayerNorm()\n      (drop_shortcut): Dropout(p=0.1)\n    )\n\n    ...\n\n    TransformerBlock structure repeated 12 times\n\n    ...\n  )\n\n  (final_norm): LayerNorm()\n  (out_head): Linear(768 → 50257, bias=False)\n)\nThe model starts with a vocabulary of\n50,257\ntokens and represents each token using\n768\ndimensions. Every transformer block preserves this 768-dimensional representation, while the activation layers temporarily expand it to\n3,072\ndimensions. Finally, the output layer maps the representation back to\n50,257\nlogits—one score for every token in the vocabulary.\nGenerating text from the untrained model\nWith that, we now have a complete GPT-style model assembled from the individual components covered in the previous articles.\nWe have also traced how the tensor dimensions change as data moves through the model and calculated the number of trainable parameters that define its size.\nHowever, the model currently begins with randomly initialized parameters. It has the right architecture, but it has not yet learned anything meaningful about language.\nLet us try generating some text:\nstart_text = \"Polish my unrevealed\"\nencoded = tokenizer.encode(start_text)\n\n# Add the batch dimension\nencoded_tensor = torch.tensor(encoded).unsqueeze(0)\n\n# Put the model into evaluation mode\nmodel.eval()\n\nout = generate_text_simple(\n    model=model,\n    idx=encoded_tensor,\n    max_new_tokens=6,\n    context_size=GPT_CONFIG_124M[\"context_length\"]\n)\n\ndecoded_text = tokenizer.decode(out.squeeze(0).tolist())\nprint(decoded_text)\nThe model generates:\nPolish my unrevealeditters Aminreally pocket cumbers reprodu\nAs expected, a model with random parameters produces random and largely meaningless output.\nWhat comes next?\nIn the next article, we will look at how a GPT model is trained, how its performance is evaluated during training and how text is generated from its predictions.\nWe will then download pretrained GPT-2 weights and load them into our custom-built architecture. This will allow us to compare a randomly initialized model with one that has already learned language patterns from a large body of text.\nReferences\nSebastian Raschka — *Build a Large Language Model (From Scratch).\n*\nhttps://www.manning.com/books/build-a-large-language-model-from-scratch\nGithub repo -\nhttps://github.com/amanjain454/llms-from-scratch-work\n#\ngpt\n#\ndeeplearning\n33\nviews",
  "metadata": {
    "title": "Assembling a GPT Model from Scratch | Part 3",
    "description": "See how a baseline GPT model comes together, how tensor dimensions flow through it and why GPT-2 Small is described as a 124M-parameter model.",
    "keywords": null,
    "author": null,
    "og:title": "Assembling a GPT Model from Scratch | Part 3",
    "og:description": "See how a baseline GPT model comes together, how tensor dimensions flow through it and why GPT-2 Small is described as a 124M-parameter model.",
    "og:url": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-3-assembling-the-model",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/54c0951f-d60a-4870-a2d5-9e2fd378cdd6.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Assembling a GPT Model from Scratch | Part 3",
    "twitter:description": "See how a baseline GPT model comes together, how tensor dimensions flow through it and why GPT-2 Small is described as a 124M-parameter model.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/54c0951f-d60a-4870-a2d5-9e2fd378cdd6.png",
    "article:published_time": "2026-07-14T05:56:24.977Z",
    "article:modified_time": "2026-07-16T03:04:43.751Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/83a44523f22c/code/body-000.txt",
      "chars": 1168,
      "language": "text"
    },
    {
      "file": "attachments/83a44523f22c/code/body-001.txt",
      "chars": 181,
      "language": "text"
    },
    {
      "file": "attachments/83a44523f22c/code/body-002.txt",
      "chars": 323,
      "language": "text"
    },
    {
      "file": "attachments/83a44523f22c/code/body-003.txt",
      "chars": 39,
      "language": "text"
    },
    {
      "file": "attachments/83a44523f22c/code/body-004.txt",
      "chars": 136,
      "language": "text"
    },
    {
      "file": "attachments/83a44523f22c/code/body-005.txt",
      "chars": 842,
      "language": "text"
    },
    {
      "file": "attachments/83a44523f22c/code/body-006.txt",
      "chars": 428,
      "language": "text"
    },
    {
      "file": "attachments/83a44523f22c/code/body-007.txt",
      "chars": 61,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/54c0951f-d60a-4870-a2d5-9e2fd378cdd6.png",
      "bytes": 1265787,
      "reused": true,
      "local_path": "attachments/83a44523f22c/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/54c0951f-d60a-4870-a2d5-9e2fd378cdd6.png",
      "alt": "LLM architecture (part 3): Assembling model",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/147d55ae-4a70-4fb2-b647-104ed5fab3d5.png",
      "bytes": 22854,
      "reused": true,
      "local_path": "attachments/83a44523f22c/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/147d55ae-4a70-4fb2-b647-104ed5fab3d5.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/ad5d3c50-da4a-4e57-b387-dc9c7a28fb81.png",
      "bytes": 28616,
      "reused": true,
      "local_path": "attachments/83a44523f22c/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/ad5d3c50-da4a-4e57-b387-dc9c7a28fb81.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-3-assembling-the-model.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/building-gpt-from-scratch",
      "label": "Part of series Building a GPT Model From Scratch",
      "type": "a"
    },
    {
      "href": "https://www.manning.com/books/build-a-large-language-model-from-scratch",
      "label": "https://www.manning.com/books/build-a-large-language-model-from-scratch",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work",
      "label": "https://github.com/amanjain454/llms-from-scratch-work",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/gpt",
      "label": "# gpt",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/deeplearning",
      "label": "# deeplearning",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:22",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:56Z",
  "source_url": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-3-assembling-the-model",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11458,
  "expected_body_images": 3,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/83a44523f22c/pages/source.html",
  "local_html": "attachments/83a44523f22c/pages/article.html",
  "body_base_path": "attachments/83a44523f22c/pages/",
  "body_markdown_path": "attachments/83a44523f22c/pages/article.md"
}