{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "LLM architecture (part 1): Tokenization, Embeddings, and Attention",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/building-gpt-from-scratch-tokenization-embeddings-attention",
  "publish_date": "2026-07-13",
  "corpus_stage": "",
  "notes": "",
  "id": "c715da42bb8e",
  "collected_at": "2026-10-02T14:11:46",
  "title": "GPT From Scratch: Tokenization, Embeddings & Attention",
  "body_markdown": "LLM architecture (part 1): Tokenization, Embeddings, and Attention\nHow raw text becomes token IDs, embeddings, and context-aware representations inside a GPT-2-style model.\nUpdated\nJuly 14, 2026\n•\n15\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nBuilding a GPT Model From Scratch\nIn the first article, I shared why I decided to build a GPT-style model from scratch and what I ultimately built. In this article, I want to open that box and look at the first set of building blocks that make the model work:\nTokenization\nPreparing the training data\nEmbeddings\nThe attention mechanism\nThis is not intended to be an exhaustive technical walkthrough. There is far more happening in the code than I can reasonably cover in one article. Instead, I will focus on the key ideas and configurations that appear in the accompanying GitHub repository and explain how they shape the model.\nTo help us keep track, I will use the following GPT-2 Small-style configuration throughout the article:\nGPT_CONFIG_124M = {\n    \"vocab_size\": 50_257,\n    \"context_length\": 1_024,\n    \"emb_dim\": 768,\n    \"n_heads\": 12,\n    \"n_layers\": 12,\n    \"drop_rate\": 0.1,\n    \"qkv_bias\": False,\n}\nI will also keep calling out the dimensions of the tensors as they move through the model. Personally, I found dimensions to be one of the easiest ways to understand what the code was actually doing.\n1. Tokenization: Converting Language into Model-Readable Units\nBefore an LLM can train on text, the text must be converted into tokens and then into token IDs.\nA tokenizer has a finite vocabulary: the complete set of tokens it knows. In our configuration,\nvocab_size = 50,257\n, which means every token produced by the tokenizer must map to one of 50,257 token IDs.\nThis does not mean that the vocabulary contains every possible English word. GPT-2 uses Byte Pair Encoding, or BPE, which can split unfamiliar text into smaller units that already exist in the vocabulary.\nFor example, some random text such as:\npdt magr\nmay be split into:\nTokens:    ['p', 'dt', ' mag', 'r']\nToken IDs: [79, 28664, 2153, 81]\nThe model never directly receives the original string. It receives the sequence of token IDs.\nTokenization is also sensitive to the exact text. Capitalization, pluralization, and even a preceding space can change the result:\n'cat'  → ['cat']     → [9246]\n'Cat'  → ['Cat']     → [21979]\n'CAT'  → ['C', 'AT'] → [34, 1404]\n'cats' → ['cats']    → [24619]\nThis was one of my first useful realizations: a word is not necessarily a token, and the same-looking word can be represented differently depending on how it appears in the text.\n2. Preparing the Input and Target Data\nOnce the text has been converted into token IDs, the next step is to create examples the model can learn from.\nThe basic training objective is simple: given the tokens so far, predict the next token.\nSuppose our tokenized text is conceptually:\n[I, had, always, thought, Jack, Gisburn, is, my, friend]\nLet us define a basic strategy with:\ncontext_length = 5\n: Each training example contains five input tokens.\nThe target sequence, or output, is created by shifting the input tokens one position to the right while keeping both sequences the same length. During training, the model predicts the next token at every position in the sequence. During generation, however, we usually use only the prediction from the final position to select the next token.\nstride = 2\n: The next example begins two positions after the previous one. The stride determines how much consecutive training samples overlap. A smaller stride creates more examples and uses the text more thoroughly, but it also creates more redundancy. A larger stride reduces overlap but may skip some possible training windows.\nThe first input-target pair would be:\nInput 1:  [I, had, always, thought, Jack]\nTarget 1: [had, always, thought, Jack, Gisburn]\nThe target is simply the input sequence shifted one position to the right.\nWith a stride of two, the next pair becomes:\nInput 2:  [always, thought, Jack, Gisburn, is]\nTarget 2: [thought, Jack, Gisburn, is, my]\nDuring pretraining, the model makes a next-token prediction at every position. For the first example, it learns to predict:\nI       → had\nhad     → always\nalways  → thought\nthought → Jack\nJack    → Gisburn\nThis data-preparation strategy also changes with the use case. For language-model pretraining, the target is the next-token sequence. For classification fine-tuning, the target may instead be a class such as\nspam\nor\nnot spam\n. The model architecture may remain largely familiar, but the way we construct the input and output data changes substantially.\nIn the GitHub repository, the dataset and data-loader classes handle this preparation and convert the raw text into batches that the model can process.\n3. Embeddings: Turning IDs into Learnable Vectors\nAt this point, our input consists of token IDs. These numbers are only identifiers.\nFor example, token ID\n9246\ndoes not carry any mathematical meaning by itself. It does not tell the model that\ncat\nis an animal, that it may be a pet, or that it is related to\ndog\n. It simply identifies one row in the vocabulary.\nTo build intuition, imagine a toy embedding space with only three dimensions:\nDimension 1: association with pets\nDimension 2: association with wildlife\nDimension 3: association with sports\nTokens might appear at hypothetical locations such as:\ncat     → [2.0, 0.1, 0.0]\nhorse   → [1.2, 0.8, 2.0]\ncricket → [0.2, 4.0, 8.0]\nIn this simplified example,\nhorse\nhas associations with pets, wildlife, and sports, while\ncricket\nmay be associated with both an animal and a sport.\nReal embeddings do not come with dimensions labelled\npet\n,\nwildlife\n, or\nsport\n. The individual dimensions have no predefined meanings. The model learns a distributed representation across hundreds of dimensions, and understanding what these internal representations capture is itself an active field of research.\nThe embedding values also do not start with meaningful coordinates. They begin as random numbers and are adjusted during training. When tokens repeatedly appear in similar contexts, the model may learn representations that place them closer together or make them useful in similar computations.\nThe embedding layer converts each token ID into a dense, trainable vector. In our model, the token-embedding table has the following shape:\nvocab_size × emb_dim\n50,257 × 768\nEvery vocabulary token therefore has one 768-dimensional vector.\nAdding Position\nToken embeddings tell the model what token it is looking at, but they do not tell it where that token appears.\nThat matters because these two sentences contain the same words but not the same meaning:\nThe dog chased the cat.\nThe cat chased the dog.\nGPT-2 therefore learns an absolute positional embedding for each location up to the configured context length. With\ncontext_length = 1,024\nand\nemb_dim = 768\n, the positional-embedding table has the shape:\n1,024 × 768\nFor every token, the model adds its token embedding and positional embedding:\ninput_embeddings = token_embeddings + positional_embeddings\nThe resulting vector contains information about both the token and its position before entering the transformer blocks.\nSome newer architectures use relative positional methods, which focus on how far tokens are from one another rather than only on their fixed positions. GPT-2, however, uses learned absolute positional embeddings.\nReturning to a Finite Vocabulary\nEmbeddings place tokens in a continuous vector space, but the model eventually has to choose one token from a finite vocabulary.\nNear the end of the model, the final hidden vector is projected into one score, or logit, for every possible vocabulary token:\n768-dimensional hidden vector\n              ↓\n50,257 logits\nSoftmax can convert these logits into probabilities, after which the next token is selected using a strategy such as greedy selection or sampling.\n4. Attention: Allowing Tokens to Exchange Information\nThe attention mechanism is one of the key ideas behind the Transformer architecture. The intuition is straightforward:\nEach token asks: Which of the available tokens are relevant to me?\nBefore self-attention, a token starts with its token and positional information. After self-attention, it has gathered information from other relevant tokens in the sequence. The resulting updated representation is commonly called a context vector.\nConsider our toy embedding for\nhorse\n:\nhorse → [1.2, 0.8, 2.0]\nNow suppose it appears in:\nticket for horse race\nThe words\nticket\nand\nrace\nprovide evidence that\nhorse\nis being used in a sporting context. Attention allows the representation for\nhorse\nto incorporate that surrounding information. Its context vector may therefore lean more strongly toward the hypothetical\nsports\ndimension.\nThe exact dimensions in this example are illustrative, but the idea is important: the initial token embedding is context-independent, while the representation created by the transformer becomes context-sensitive.\nAttention works only with the tokens inside the current context window. This is one reason\ncontext_length\nmatters: it limits how many tokens the model can consider in one sequence.\nStarting with Simple Self-Attention\nThe mathematical core of attention is a weighted combination of vectors.\nSuppose the input contains six tokens, each represented by a three-dimensional vector:\nimport torch\n\ninputs = torch.tensor([\n    [0.43, 0.15, 0.89],\n    [0.55, 0.87, 0.66],\n    [0.57, 0.85, 0.64],\n    [0.22, 0.58, 0.33],\n    [0.77, 0.25, 0.10],\n    [0.05, 0.80, 0.55],\n])\nThe input matrix has the shape:\nnumber of tokens × embedding dimensions\n6 × 3\nFor one token, we can take its dot product with every token in the sequence. A larger dot product generally indicates greater alignment in that vector space.\nquery = inputs[1]\nattention_scores = inputs @ query\nattention_weights = torch.softmax(attention_scores, dim=0)\ncontext_vector = attention_weights @ inputs\nSoftmax converts the raw scores into positive weights that sum to one. The context vector is then a weighted combination of all input vectors.\nWe can process every token at once using matrix multiplication:\nattention_scores = inputs @ inputs.T\nattention_weights = torch.softmax(attention_scores, dim=-1)\ncontext_vectors = attention_weights @ inputs\nThe dimensions are:\ninputs:             6 × 3\nattention_scores:   6 × 6\nattention_weights:  6 × 6\ncontext_vectors:    6 × 3\nThe\n6 × 6\nattention matrix tells us how strongly each of the six tokens attends to each of the six available tokens. After multiplying by the original input matrix, we return to one three-dimensional context vector per token.\nI could not resist experimenting with this. In this simplified version, if the exact same vector appears twice, both occurrences produce the same context vector. That is a useful sanity check for the basic calculation.\ninputs = torch.tensor(\n    [[0.43, 0.15, 0.89],\n     [0.43, 0.15, 0.89],    # parallel to the first\n     [0.57, 0.85, 0.64],\n     [0.22, 0.58, 0.33],\n     [0.77, 0.25, 0.10],\n     [0.05, 0.80, 0.55]])\nAll context vectors:\ntensor([[0.4181, 0.4459, 0.6276],\n        [0.4181, 0.4459, 0.6276],\n        [0.4130, 0.5147, 0.5924],\n        [0.4038, 0.5031, 0.5789],\n        [0.4442, 0.4635, 0.5573],\n        [0.3882, 0.5229, 0.5909]])\nIt is not generally true in the complete GPT model, though. Once positional embeddings and causal masking are included, two occurrences of the same token can receive different representations because they appear at different locations and have access to different preceding context.\nLayer 1: Learning What to Compare Through Query, Key, and Value\nThe simple version compares the input vectors directly. A real attention layer gives the model trainable weights that teach it how to compare tokens and what information to exchange. These trainable attention weights help the model learn relationships between tokens, including patterns that reflect grammar, syntax, reference, and meaning. These relationships are learned statistically from text rather than encoded as explicit linguistic rules.\nThe same input is projected into three different representations:\nQuery: What information am I looking for?\nKey:   What information do I contain?\nValue: What information should I contribute?\nMathematically:\nqueries = inputs @ W_query\nkeys = inputs @ W_key\nvalues = inputs @ W_value\nThe attention scores are calculated by comparing queries with keys, and the resulting weights determine how the value vectors are combined:\nattention_scores = queries @ keys.T\nattention_weights = torch.softmax(\n    attention_scores / keys.shape[-1] ** 0.5,\n    dim=-1,\n)\ncontext_vectors = attention_weights @ values\nThe division by the square root of the key dimension keeps the scores from becoming excessively large as the vector dimension increases.\nThe intuition behind trainable Query, Key, and Value matrices is that attention should not be limited to raw embedding similarity. Two tokens may need to connect because of grammar, pronoun reference, sentence structure, or meaning, even when their original embeddings are not especially similar.\nConsider:\nThe animal did not cross the road because it was tired.\nWhen processing\nit\n, the query projection can learn to search for a likely reference. The key representation for\nanimal\ncan become a strong match, while its value representation carries useful information into the updated representation for\nit\n.\nThese are not manually programmed grammar rules. They are statistical patterns learned through repeated next-token prediction. As with the embedding table, the attention weights begin randomly and are gradually adjusted whenever the model makes prediction errors.\nLayer 2: Masking Future Tokens\nGPT generates text from left to right. When predicting the next token, it should not be able to look ahead and see the answer.\nFor a six-token sequence, the context vector at position three may use tokens one, two, and three, but not tokens four, five, or six.\nA causal attention mask enforces this by hiding all future positions in the attention matrix:\nself.register_buffer(\n    \"mask\",\n    torch.triu(\n        torch.ones(context_length, context_length),\n        diagonal=1,\n    ),\n)\nBefore softmax, the masked scores are replaced with negative infinity. Their softmax probabilities therefore become zero.\nThis ensures that training matches generation: every prediction can use only the information that would have been available at that point in the sequence.\nLayer 3: Adding Dropout\nThe intuition behind dropout is to prevent the model from depending too heavily on a small set of activations or connections.\nWith\ndrop_rate = 0.1\n, roughly 10% of the selected activations are randomly set to zero during each training step. Because a different subset is removed each time, the model is encouraged to distribute useful information across multiple pathways.\nI think of it like training a team while randomly making a few members unavailable each day. The team becomes less dependent on any one person and learns to function through different combinations of people.\nIn a GPT-style model, dropout may be applied to the combined token and positional embeddings, the attention weights, and the outputs of attention or feed-forward layers. It is active during training and disabled during evaluation or generation.\nLayer 4: Using Multiple Attention Heads\nA single attention head gives the model one way to relate tokens. But language can contain several relevant relationships at the same time.\nOne token may need to track a nearby modifier, connect a pronoun to a distant noun, maintain subject-verb agreement, and preserve the broader meaning of the sentence. Multi-head attention gives the model several learned relationship spaces in parallel.\nI find the analogy of a group of reviewers useful. Everyone reads the same sentence, but one reviewer may focus on grammar, another on meaning, another on nearby words, and another on long-range relationships. Their observations are then combined.\nThe heads are not explicitly assigned these responsibilities, and their learned behavior is not always cleanly interpretable. Training simply allows them to discover patterns that improve next-token prediction.\nIn GPT-2 Small, the 768-dimensional representation is divided across 12 heads:\n768 ÷ 12 = 64 dimensions per head\nEach head performs attention in its own 64-dimensional subspace. The context vectors from all 12 heads are then concatenated:\n12 heads × 64 dimensions = 768 dimensions\nA final output projection combines the information from the heads and returns one 768-dimensional representation for each token, ready for the next part of the transformer block.\nWhere We Are Now\nWe started with raw text and converted it into token IDs. We turned those IDs into token and positional embeddings. Attention then allowed every token to gather relevant information from the context available to it, while trainable Query, Key, and Value matrices taught the model how to compare tokens and what information to exchange.\nBy this stage, each token is no longer represented only by what it is. Its vector also reflects where it appears and what the preceding text suggests it means in that specific context.\nThat still leaves several important components of the baseline GPT architecture. In the next article, I will cover the feed-forward network, GELU activation, layer normalization, shortcut connections, and how these pieces come together inside a transformer block.\nReferences\n3Blue1Brown — Transformer and LLM visualizations.\nTransformers, the Tech Behind LLMs\nand\nAttention in Transformers, Step by Step\nprovide intuitive visual explanations of embeddings, attention, and the broader Transformer architecture. The accompanying lesson,\nHow Might LLMs Store Facts?\n, explores the role of feed-forward networks.\n**Sebastian Raschka —\nBuild a Large Language Model (From Scratch).\n** The\nbook and companion materials\ncover tokenization, embeddings, attention, GPT-style model construction, pretraining, and fine-tuning. The\nofficial GitHub repository\ncontains the accompanying implementation.\nOpenAI — GPT-2 publications and release materials.\nBetter Language Models and Their Implications\nintroduces GPT-2, while the paper\nLanguage Models Are Unsupervised Multitask Learners\ndescribes its architecture, training approach, and evaluation.\nAman Jain — Project repository.\nThe code and notebooks accompanying this learning journey are available on my\nGitHub profile\n.\nVaswani et al. — “Attention Is All You Need.”\nThe original\nGoogle Research paper\nintroduced the Transformer architecture and its use of scaled dot-product attention, multi-head attention, and positional encoding.\n#\nlarge-language-models\n#\ngpt-2\n#\ntransformers\n#\ndeep-learning\n57\nviews",
  "metadata": {
    "title": "GPT From Scratch: Tokenization, Embeddings & Attention",
    "description": "A practical walkthrough of how a GPT-2-style model turns text into tokens, embeddings, and context-aware representations through self-attention.",
    "keywords": null,
    "author": null,
    "og:title": "GPT From Scratch: Tokenization, Embeddings & Attention",
    "og:description": "A practical walkthrough of how a GPT-2-style model turns text into tokens, embeddings, and context-aware representations through self-attention.",
    "og:url": "https://curious-pm.hashnode.dev/building-gpt-from-scratch-tokenization-embeddings-attention",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/2bd7735c-ea15-4b4b-a9a0-821496e2c43d.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "GPT From Scratch: Tokenization, Embeddings & Attention",
    "twitter:description": "A practical walkthrough of how a GPT-2-style model turns text into tokens, embeddings, and context-aware representations through self-attention.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/2bd7735c-ea15-4b4b-a9a0-821496e2c43d.png",
    "article:published_time": "2026-07-13T05:07:40.842Z",
    "article:modified_time": "2026-07-14T04:33:23.746Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/c715da42bb8e/code/body-000.txt",
      "chars": 181,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-001.txt",
      "chars": 9,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-002.txt",
      "chars": 69,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-003.txt",
      "chars": 126,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-004.txt",
      "chars": 57,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-005.txt",
      "chars": 90,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-006.txt",
      "chars": 90,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-007.txt",
      "chars": 82,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-008.txt",
      "chars": 111,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-009.txt",
      "chars": 78,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-010.txt",
      "chars": 34,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-011.txt",
      "chars": 48,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-012.txt",
      "chars": 12,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-013.txt",
      "chars": 60,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-014.txt",
      "chars": 60,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-015.txt",
      "chars": 24,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-016.txt",
      "chars": 22,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-017.txt",
      "chars": 185,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-018.txt",
      "chars": 46,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-019.txt",
      "chars": 155,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-020.txt",
      "chars": 142,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-021.txt",
      "chars": 104,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-022.txt",
      "chars": 201,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-023.txt",
      "chars": 226,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-024.txt",
      "chars": 125,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-025.txt",
      "chars": 75,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-026.txt",
      "chars": 176,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-027.txt",
      "chars": 56,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-028.txt",
      "chars": 131,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-029.txt",
      "chars": 34,
      "language": "text"
    },
    {
      "file": "attachments/c715da42bb8e/code/body-030.txt",
      "chars": 42,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/2bd7735c-ea15-4b4b-a9a0-821496e2c43d.png",
      "bytes": 1758653,
      "reused": true,
      "local_path": "attachments/c715da42bb8e/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/2bd7735c-ea15-4b4b-a9a0-821496e2c43d.png",
      "alt": "LLM architecture (part 1): Tokenization, Embeddings, and Attention",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/dc9494d0-a10e-4b43-b206-88c5f412a849.png",
      "bytes": 56307,
      "reused": true,
      "local_path": "attachments/c715da42bb8e/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/dc9494d0-a10e-4b43-b206-88c5f412a849.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/building-gpt-from-scratch-tokenization-embeddings-attention.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/building-gpt-from-scratch",
      "label": "Part of series Building a GPT Model From Scratch",
      "type": "a"
    },
    {
      "href": "https://www.3blue1brown.com/lessons/gpt/",
      "label": "Transformers, the Tech Behind LLMs",
      "type": "a"
    },
    {
      "href": "https://www.3blue1brown.com/lessons/attention/",
      "label": "Attention in Transformers, Step by Step",
      "type": "a"
    },
    {
      "href": "https://www.3blue1brown.com/lessons/mlp/",
      "label": "How Might LLMs Store Facts?",
      "type": "a"
    },
    {
      "href": "https://sebastianraschka.com/llms-from-scratch/",
      "label": "book and companion materials",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch",
      "label": "official GitHub repository",
      "type": "a"
    },
    {
      "href": "https://openai.com/index/better-language-models/",
      "label": "Better Language Models and Their Implications",
      "type": "a"
    },
    {
      "href": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "label": "Language Models Are Unsupervised Multitask Learners",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454",
      "label": "GitHub profile",
      "type": "a"
    },
    {
      "href": "https://research.google/pubs/attention-is-all-you-need/",
      "label": "Google Research paper",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/large-language-models",
      "label": "# large-language-models",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/gpt-2",
      "label": "# gpt-2",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/transformers",
      "label": "# transformers",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/deep-learning",
      "label": "# deep-learning",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:27",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:57Z",
  "source_url": "https://curious-pm.hashnode.dev/building-gpt-from-scratch-tokenization-embeddings-attention",
  "papers": [
    {
      "status": "ok",
      "url": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "bytes": 582775,
      "reused": true,
      "local_path": "papers/repair/3c62359064baa28a7743.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 18884,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c715da42bb8e/pages/source.html",
  "local_html": "attachments/c715da42bb8e/pages/article.html",
  "body_base_path": "attachments/c715da42bb8e/pages/",
  "body_markdown_path": "attachments/c715da42bb8e/pages/article.md"
}