{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "Building a GPT Model from Scratch (Part 4): Training and Loading Pretrained Weights",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-4-training-and-pretrained-weights",
  "publish_date": "2026-07-16",
  "corpus_stage": "",
  "notes": "",
  "id": "7ec878e7a673",
  "collected_at": "2026-10-02T14:11:45",
  "title": "Train a GPT Model and Load Pretrained GPT-2 Weights",
  "body_markdown": "Building a GPT Model from Scratch (Part 4): Training and Loading Pretrained Weights\nFrom cross-entropy loss and backpropagation to decoding strategies and loading Open AI’s pretrained GPT-2 weights.\nUpdated\nJuly 16, 2026\n•\n13\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nBuilding a GPT Model From Scratch\nIn the previous article, we completed the base architecture of our GPT model. The token and positional embeddings, transformer blocks, normalization layers, and output layer were all in place. The model worked, but its weights were randomly initialized, so its output was random. We had built the machinery, but it had not learned anything yet.\nIn this article, we will look at how to turn that architecture into a model that can generate more meaningful text. We will cover this in three parts.\nFirst, we will look at training: preparing the data pipeline, dividing the data into training and validation sets, and defining a loss function that measures how far the model’s predictions are from the expected output.\nSecond, we will explore decoding strategies used during inference. The model produces a score for every token in the vocabulary, but we still need a method for selecting the next token and converting those scores into generated text.\nFinally, we will download the publicly available pretrained GPT-2 weights released by OpenAI and load them into our custom architecture. Training a model on internet-scale text would require significant data, computing power, and time, so we will reuse OpenAI’s trained weights and map them to the corresponding layers in our model.\nModel Training and Validation\nFor our initial training run, we will use a short text file,\nThe Verdict\n:\nfile_path = \"the-verdict.txt\"\n\nwith open(file_path, \"r\", encoding=\"utf-8\") as file:\n    text_data = file.read()\n\ntotal_characters = len(text_data)\ntotal_tokens = len(tokenizer.encode(text_data))\n\nprint(f\"Total characters: {total_characters}\")\nprint(f\"Total tokens: {total_tokens}\")\nTotal characters: 20,479\nTotal tokens: 5,145\nWe divide this text into smaller batches using the sliding-window approach discussed in the earlier articles. Each input sequence is paired with the same sequence shifted one position forward, allowing the model to learn next-token prediction.\nOnce the input and target sequences are ready, the next step is to define the loss function that training will minimize. Before doing that, it helps to understand exactly what the model produces.\nWhat Does the Model Output?\nWhen a sequence enters the model, each token is first represented by an embedding vector. In our GPT-2 configuration, each vector has\n768\ndimensions.\nAs these vectors pass through the transformer blocks, they are updated using information from the preceding tokens. The final transformer block therefore produces a contextualized 768-dimensional vector for every position in the sequence.\nThe output layer then maps each vector from the embedding dimension to the vocabulary dimension. Since our vocabulary contains\n50,257\ntokens, the model produces\n50,257\nnumbers at every sequence position.\nThese numbers are called\nlogits\n. Each logit represents how strongly the model favors one vocabulary token as the next token. Logits are scores rather than probabilities, so they do not need to add up to one. We can apply softmax to convert them into probabilities.\nDuring training, the model makes one prediction at every position in the sequence. Consider the following input and target pair:\nInput:  [15432, 4981, 2198, 684, 5246, 11987]\nTarget: [4981, 2198, 684, 5246, 11987, 268]\nThis creates six next-token prediction tasks:\nPosition\nContext available to the model\nTarget token\n1\n15432\n4981\n2\n15432, 4981\n2198\n3\n15432, 4981, 2198\n684\n4\n15432, 4981, 2198, 684\n5246\n5\n15432, 4981, 2198, 684, 5246\n11987\n6\n15432, 4981, 2198, 684, 5246, 11987\n268\nThe model therefore learns from every position in the sequence, not only from the final token.\nCalculating the Loss\nWe use\ncross-entropy loss\nto measure the quality of the model’s predictions. At each position, the model produces one logit for every token in the vocabulary. Cross-entropy compares these scores with the actual next token and assigns a higher loss when the model gives the correct token a low probability.\nThe token-level losses are then averaged across all sequence positions and samples in the batch. A lower loss means the model is becoming better at predicting the next token.\nConsider a simplified vocabulary of four tokens, where the correct next token is\n“cat”\n:\nToken\nLogit (z_i)\n(e^{z_i})\nProbability\nCorrect token?\nLoss contribution\ndog\n1.2\n3.32\n(3.32/16.30 = 0.204)\nNo\n0\ncat\n2.4\n11.02\n(11.02/16.30 = 0.676)\nYes\n(-\\log(0.676)=0.39)\nhouse\n0.3\n1.35\n(1.35/16.30 = 0.083)\nNo\n0\nran\n-0.5\n0.61\n(0.61/16.30 = 0.037)\nNo\n0\nTotal\n—\n16.30\n1.000\n—\nLoss = 0.39\nThe model assigned a probability of approximately\n67.6%\nto the correct token, resulting in a cross-entropy loss of approximately\n0.39\n.\nThe incorrect tokens do not contribute directly to the final cross-entropy value because the target identifies “cat” as the correct token. However, their logits still affect the softmax denominator and therefore influence the probability assigned to “cat.”\nCross-entropy is useful because it considers not only whether the prediction was correct, but also how much probability the model assigned to the correct token. It gives a small loss to confident correct predictions and heavily penalizes confident incorrect ones. By minimizing this loss across many sequences, the model gradually learns to assign higher probabilities to tokens that are more likely to appear next.\nTraining the Model\nWith the data pipeline and loss function in place, we are ready to train the model:\ndef train_model_simple(\n    model,train_loader, val_loader, optimizer, device, num_epochs, eval_freq,eval_iter,start_context,tokenizer\n):\n    train_losses = []\n    val_losses = []\n    track_tokens_seen = []\n\n    tokens_seen = 0\n    global_step = -1\n\n    for epoch in range(num_epochs):\n        model.train()\n\n        for input_batch, target_batch in train_loader:\n            optimizer.zero_grad()\n\n            loss = calc_loss_batch(\n                input_batch,\n                target_batch,\n                model,\n                device\n            )\n\n            loss.backward()\n            optimizer.step()\n\n            tokens_seen += input_batch.numel()\n            global_step += 1\n\n            if global_step % eval_freq == 0:\n                train_loss, val_loss = evaluate_model(\n                    model,\n                    train_loader,\n                    val_loader,\n                    device,\n                    eval_iter\n                )\n\n                train_losses.append(train_loss)\n                val_losses.append(val_loss)\n                track_tokens_seen.append(tokens_seen)\n\n                print(\n                    f\"Epoch: {epoch + 1}, \"\n                    f\"Global Step: {global_step}, \"\n                    f\"Train Loss: {train_loss:.4f}, \"\n                    f\"Val Loss: {val_loss:.4f}\"\n                )\n\n        generate_and_print_sample(model,tokenizer,device,\n            start_context)\n\n    return train_losses, val_losses, track_tokens_seen\nMost of the training loop’s work happens in three lines:\nloss = calc_loss_batch(input_batch, target_batch,model,\n    device)\n\nloss.backward()\noptimizer.step()\nStep 1: Calculate the Loss\nThe model’s predictions are compared with the target tokens using the cross-entropy loss discussed above.\nStep 2: Run Backpropagation\nStarting from the output layer, backpropagation calculates how much each trainable parameter contributed to the loss. These gradients indicate the direction and extent to which the weights should change to reduce it.\nStep 3: Update the Weights\nThe optimizer uses these gradients to adjust the model’s parameters. We use AdamW, which adapts the update size for different parameters and applies weight decay to support stable training.\nLet us run the training loop:\ntorch.manual_seed(123)\n\nmodel = GPTModel(GPT_CONFIG_124M)\nmodel.to(device)\n\noptimizer = torch.optim.AdamW(model.parameters(),lr=3e-4,\n    weight_decay=0.1)\n\nnum_epochs = 10\n\ntrain_losses, val_losses, track_tokens_seen = train_model_simple(model,train_loader,val_loader,optimizer,\n    device,num_epochs=num_epochs,eval_freq=5,eval_iter=5,\n    start_context=\"Every effort moves you\",      tokenizer=tokenizer)\nA shortened version of the output is shown below:\nEpoch: 1, Global Step: 0, Train Loss: 9.9989, Val Loss: 10.1392\nEpoch: 1, Global Step: 5, Train Loss: 8.3282, Val Loss: 8.6033\n....\n\nEpoch: 3, Global Step: 20, Train Loss: 5.8362, Val Loss: 6.5150\nEpoch: 3, Global Step: 25, Train Loss: 5.3558, Val Loss: 6.3479\n\nEvery effort moves you. \" \"I I--I. \"I. \"\"I, and I...\n...\n\nEpoch: 9, Global Step: 75, Train Loss: 1.7470, Val Loss: 6.1943\nEpoch: 9, Global Step: 80, Train Loss: 1.4977, Val Loss: 6.2237\n\nEvery effort moves you?\" \"Yes--quite insensible to the fact\nwith equanimity. Gisburn's an awful...\nThe training loss decreases sharply, and the generated text begins to develop some structure. However, the validation loss eventually stops improving while the training loss continues to fall. This widening gap indicates overfitting, which is expected when training a large model on such a small text sample.\nLet us conclude this section with the mandatory visualization of losses over time:\nGeneration and Decoding Strategies\nDuring training, the model receives both input and target sequences. During inference, it receives only the input, so we use the logits from the final sequence position to select the next token.\nCommon decoding strategies include:\nTemperature scaling:\nAdjusts how concentrated or spread out the token probabilities are. Lower values make the output more predictable, while higher values increase randomness.\nTop-k sampling:\nLimits the choice to the\nk\nhighest-scoring tokens.\nGreedy decoding:\nAlways selects the highest-scoring token.\nMultinomial sampling:\nSamples from the full probability distribution.\nTop-p sampling:\nSamples from the smallest group of tokens whose combined probability exceeds a chosen threshold.\nIn our GPT model, we combine\ntemperature scaling and top-k sampling\n. Let us return to our earlier example and apply:\ntemperature = 1.2\ntop_k = 3\nWe first divide each logit by the temperature and then retain only the three highest-scoring tokens. Softmax converts the remaining logits into probabilities, from which the next token is sampled.\nToken\nOriginal logit\nAfter temperature\nAfter top-k\nProbability\ndog\n3.4\n2.83\n2.83\n14.0%\ncat\n4.4\n3.67\n3.67\n32.5%\nhouse\n2.0\n1.67\nRemoved\n0%\nran\n-4.0\n-3.33\nRemoved\n0%\nover\n-1.0\n-0.83\nRemoved\n0%\nthe\n5.0\n4.17\n4.17\n53.5%\nOnly\ndog\n,\ncat\n, and\nthe\nremain eligible. The model samples from these probabilities, so\n“the”\nis the most likely next token, although it is not guaranteed to be selected.\nLoading Pretrained Weights\nSo far, we have seen how a GPT model is trained and how decoding strategies convert its logits into generated text. However, training a model of this size on an internet-scale dataset would require substantial data, computing power, and time.\nInstead of doing that from scratch, we will load the publicly available GPT-2 weights released by OpenAI into the architecture we have already built.\nOpenAI released the GPT-2 checkpoints through its official\nGPT-2 GitHub repository\n. Its download script retrieves the trained weights, model configuration, and tokenizer files from OpenAI’s public storage.\nFor example, the\n124M\ncheckpoint corresponds to GPT-2 Small. These files contain the pretrained model parameters and supporting files—not the original WebText training dataset.\nGPT-2 was released in four configurations:\nModel parameter\nGPT-2 Small\nGPT-2 Medium\nGPT-2 Large\nGPT-2 XL\nApprox. parameters\n124 million\n355 million\n774 million\n1.558 billion\nTransformer blocks\n12\n24\n36\n48\nAttention heads\n12\n16\n20\n25\nEmbedding dimension\n768\n1,024\n1,280\n1,600\nContext length\n1,024\n1,024\n1,024\n1,024\nVocabulary size\n50,257\n50,257\n50,257\n50,257\nWe use the following code to download the GPT-2 Small weights and load them into our model:\nfrom gpt_download import download_and_load_gpt2\n\nsettings, params = download_and_load_gpt2(\n    model_size=\"124M\",\n    models_dir=\"gpt2\"\n)\n\ngpt = GPTModel(NEW_CONFIG)\n\nload_weights_into_gpt(gpt, params)\n\ngpt.to(device)\ngpt.eval()\nThe model configuration must match the selected checkpoint. For example, the\n124M\nweights require a model with 12 transformer blocks, 12 attention heads, and an embedding dimension of 768.\ndownload_and_load_gpt2()\nreads OpenAI’s TensorFlow checkpoint and organizes its parameters.\nload_weights_into_gpt()\nthen copies them into the corresponding components of our PyTorch model, including the token and positional embeddings, attention projections, activation layers, normalization layers, and output layer.\nOnce this mapping is complete, the architecture is no longer operating with random weights. It is now powered by the language patterns learned during OpenAI’s original GPT-2 pretraining.\nLet us test the pretrained model:\ntorch.manual_seed(123)\n\ntoken_ids = generate(\n    model=gpt,\n    idx=text_to_token_ids(\n        \"today's hit movies are\",\n        tokenizer\n    ).to(device),\n    max_new_tokens=50,\n    context_size=NEW_CONFIG[\"context_length\"],\n    top_k=50,\n    temperature=1.5\n)\n\nprint(\n    \"Output text:\",\n    token_ids_to_text(token_ids, tokenizer)\n)\ntoday's hit movies are not only awesome, it's actually kinda cool!\n\nWhat makes them special: Their unique, magical feel (along with the\nbest scenes, humor, comedy, a lot of that humor from a film noir-type\nangle) is the main...\nThe output is not perfect, but it is a clear improvement over the random text generated by the untrained model. The architecture has not changed—the difference comes entirely from loading weights that have already learned patterns from a large collection of text.\nConclusion\nIn this article, we moved from a randomly initialized GPT architecture to a working language model. We looked at how training data is prepared, how cross-entropy loss and backpropagation help the model learn, how decoding strategies turn logits into generated text, and how pretrained GPT-2 weights can be loaded into our custom architecture.\nIn the next two articles, we will build on this foundation and explore two practical applications: fine-tuning the model for text classification and instruction following.\nReferences\nSebastian Raschka —\nBuild a Large Language Model (From Scratch)\nThe primary reference for this project, covering tokenization, attention, GPT-style model construction, pretraining, and fine-tuning.\nBook on Manning\nOfficial book website\nOfficial companion code repository\nAman Jain — LLMs From Scratch Project\nThe complete code accompanying this article series, including the baseline GPT architecture, GPT-2 weight loading, classification fine-tuning, and instruction fine-tuning.\nView the project on GitHub\nOpenAI — GPT-2 Code and Model Checkpoints\nOpenAI’s official repository containing the GPT-2 code, model configurations, tokenizer files, and released model checkpoints.\nOpenAI GPT-2 repository\nOpenAI — GPT-2 Model Download Script\nThe official script used to download the\n124M\n,\n355M\n,\n774M\n, and\n1558M\nGPT-2 checkpoints from OpenAI’s public storage.\nView\ndownload_model.py\nRadford et al. —\nLanguage Models are Unsupervised Multitask Learners\nThe original GPT-2 paper describing the model architecture, WebText training corpus, model configurations, and zero-shot performance.\nRead the GPT-2 paper\nVaswani et al. —\nAttention Is All You Need\nThe original paper introducing the transformer architecture and scaled dot-product attention.\nRead the paper on arXiv\nBa, Kiros, and Hinton —\nLayer Normalization\nThe foundational paper introducing layer normalization.\nRead the paper on arXiv\nXiong et al. —\nOn Layer Normalization in the Transformer Architecture\nAn analysis of where layer normalization is placed within transformer blocks and how that placement affects training.\nRead the paper on arXiv\nHendrycks and Gimpel —\nGaussian Error Linear Units (GELUs)\nThe paper introducing GELU, the activation function used in GPT-2.\nRead the paper on arXiv\n#\nlarge-language-models\n#\ngpt\n#\ndeeplearning\n23\nviews",
  "metadata": {
    "title": "Train a GPT Model and Load Pretrained GPT-2 Weights",
    "description": "Learn how a GPT model is trained using cross-entropy loss and backpropagation, how decoding strategies work, and how to load pretrained GPT-2 weights.",
    "keywords": null,
    "author": null,
    "og:title": "Train a GPT Model and Load Pretrained GPT-2 Weights",
    "og:description": "Learn how a GPT model is trained using cross-entropy loss and backpropagation, how decoding strategies work, and how to load pretrained GPT-2 weights.",
    "og:url": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-4-training-and-pretrained-weights",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5b36e9e1-d451-4144-bfb9-7fc10cf2718f.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Train a GPT Model and Load Pretrained GPT-2 Weights",
    "twitter:description": "Learn how a GPT model is trained using cross-entropy loss and backpropagation, how decoding strategies work, and how to load pretrained GPT-2 weights.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5b36e9e1-d451-4144-bfb9-7fc10cf2718f.png",
    "article:published_time": "2026-07-16T04:01:01.349Z",
    "article:modified_time": "2026-07-16T04:01:01.349Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/7ec878e7a673/code/body-000.txt",
      "chars": 282,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-001.txt",
      "chars": 45,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-002.txt",
      "chars": 90,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-003.txt",
      "chars": 1488,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-004.txt",
      "chars": 102,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-005.txt",
      "chars": 408,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-006.txt",
      "chars": 548,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-007.txt",
      "chars": 28,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-008.txt",
      "chars": 230,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-009.txt",
      "chars": 338,
      "language": "text"
    },
    {
      "file": "attachments/7ec878e7a673/code/body-010.txt",
      "chars": 228,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5b36e9e1-d451-4144-bfb9-7fc10cf2718f.png",
      "bytes": 1146417,
      "reused": true,
      "local_path": "attachments/7ec878e7a673/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/5b36e9e1-d451-4144-bfb9-7fc10cf2718f.png",
      "alt": "Building a GPT Model from Scratch (Part 4): Training and Loading Pretrained Weights",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/c6e9b38d-91ca-42a7-9dc8-5643fefb6350.png",
      "bytes": 23450,
      "reused": true,
      "local_path": "attachments/7ec878e7a673/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/c6e9b38d-91ca-42a7-9dc8-5643fefb6350.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/7820cc78-8069-421f-9021-e6ecc34b9068.png",
      "bytes": 46444,
      "reused": true,
      "local_path": "attachments/7ec878e7a673/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/7820cc78-8069-421f-9021-e6ecc34b9068.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/c2524266-c069-46f0-8c81-099acb1da9f3.png",
      "bytes": 76234,
      "reused": true,
      "local_path": "attachments/7ec878e7a673/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/c2524266-c069-46f0-8c81-099acb1da9f3.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-4-training-and-pretrained-weights.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/building-gpt-from-scratch",
      "label": "Part of series Building a GPT Model From Scratch",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/gpt-2",
      "label": "GPT-2 GitHub repository",
      "type": "a"
    },
    {
      "href": "https://www.manning.com/books/build-a-large-language-model-from-scratch",
      "label": "Book on Manning",
      "type": "a"
    },
    {
      "href": "https://sebastianraschka.com/llms-from-scratch/",
      "label": "Official book website",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch",
      "label": "Official companion code repository",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work",
      "label": "View the project on GitHub",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/gpt-2/blob/master/download_model.py",
      "label": "View download_model.py",
      "type": "a"
    },
    {
      "href": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "label": "Read the GPT-2 paper",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03762",
      "label": "Read the paper on arXiv",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1607.06450",
      "label": "Read the paper on arXiv",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2002.04745",
      "label": "Read the paper on arXiv",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1606.08415",
      "label": "Read the paper on arXiv",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/large-language-models",
      "label": "# large-language-models",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/gpt",
      "label": "# gpt",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/deeplearning",
      "label": "# deeplearning",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:20",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:56Z",
  "source_url": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-4-training-and-pretrained-weights",
  "papers": [
    {
      "status": "ok",
      "url": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "bytes": 582775,
      "reused": true,
      "local_path": "papers/repair/3c62359064baa28a7743.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1607.06450",
      "bytes": 612446,
      "reused": false,
      "local_path": "papers/repair/52a2360d9b658baa64ff.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2002.04745",
      "bytes": 779713,
      "reused": true,
      "local_path": "papers/repair/eeee73d530f66e641d46.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1606.08415",
      "bytes": 2767669,
      "reused": false,
      "local_path": "papers/repair/74f293f3c925776e7545.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 16366,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/7ec878e7a673/pages/source.html",
  "local_html": "attachments/7ec878e7a673/pages/article.html",
  "body_base_path": "attachments/7ec878e7a673/pages/",
  "body_markdown_path": "attachments/7ec878e7a673/pages/article.md"
}