{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "Building a GPT Model from Scratch (Part 5): Fine-Tuning GPT-2 for Text Classification",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/building-a-gpt-model-from-scratch-part-5-fine-tuning-gpt-2-for-text-classification",
  "publish_date": "2026-07-16",
  "corpus_stage": "",
  "notes": "",
  "id": "be6f794c0539",
  "collected_at": "2026-10-02T14:11:45",
  "title": "Fine-Tuning GPT-2 for Text Classification",
  "body_markdown": "Building a GPT Model from Scratch (Part 5): Fine-Tuning GPT-2 for Text Classification\nHow I adapted a pretrained GPT-2 model to classify SMS messages as spam or not spam.\nUpdated\nJuly 16, 2026\n•\n10\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nBuilding a GPT Model From Scratch\nIn the\nprevious article\n, we loaded OpenAI’s pretrained GPT-2 weights into the model we had built from scratch. Now, we will adapt that model for text classification—specifically, identifying whether an SMS message is spam or not spam.\nThis requires three changes:\nData preparation:\nWe need to standardize messages of different lengths using padding and truncation, and assign each message a binary label:\n0\nfor not spam and\n1\nfor spam.\nModel architecture:\nWe need to replace the original output layer, which produces\n50,257\nvocabulary logits, with a smaller layer that produces two class logits.\nTraining and evaluation:\nWe need to calculate cross-entropy loss using the final sequence position and track classification accuracy separately.\nOnce these changes are in place, we will fine-tune the model on a public SMS spam dataset and test it on a few new messages.\nPreparing the Data for Classification\nThe first part of the process remains familiar. We begin with a text message, tokenize it, and convert the tokens into token IDs that can be passed through the model.\nClassification introduces two differences. First, messages can have very different lengths. Second, each message now has one label rather than a sequence of next-token targets.\nConsider two messages that produce the following token IDs:\nMessage 1: [1212, 318, 257, 1332]\nMessage 2: [1026, 338, 257, 2392, 286, 2420, 290, 467]\nThe first message contains four token IDs, while the second contains eight. To combine them into a single training batch, they must have the same length. We therefore add padding to the shorter message:\nMessage 1: [1212, 318, 257, 1332, 50256, 50256, 50256, 50256]\nMessage 2: [1026, 338, 257, 2392, 286, 2420, 290, 467]\nIn this implementation, we reuse GPT-2’s end-of-text token, which has the token ID\n50256\n, as the padding token. Shorter messages are padded, while messages longer than the selected maximum length are truncated.\nThe core logic inside the dataset class looks like this:\ndef __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):\n    self.data = pd.read_csv(csv_file)\n    self.encoded_texts = [tokenizer.encode(text) for text in self.data[\"Text\"]]\n    self.max_length = max_length or max(map(len, self.encoded_texts))\n\n    self.encoded_texts = [tokens[:self.max_length] for tokens in self.encoded_texts]\n    self.encoded_texts = [\n        tokens + [pad_token_id] * (self.max_length - len(tokens))\n        for tokens in self.encoded_texts\n    ]\nYou can think of padding tokens as placeholders that fill the unused positions in shorter inputs. Technically, however, they are still token IDs that the model processes; they are not automatically ignored.\nThe second change is the training target. During next-token training, the target was another sequence of token IDs. For classification, every message receives one label:\n0 → not spam\n1 → spam\nThe model receives the tokenized message as input and learns to predict this single class.\nAdapting the Output Layer for Classification\nThe second major change is in the model’s final output layer.\nOur GPT-2 Small model produces a 768-dimensional representation at every token position. The original output layer converts each representation into\n50,257\nlogits—one for every token in the GPT-2 vocabulary. These logits are used to predict which token should come next.\nFor classification, we no longer need an output for the entire vocabulary. We only need two class logits:\nNext-token prediction:\n768 dimensions → 50,257 vocabulary logits\n\nBinary classification:\n768 dimensions → 2 class logits\nWe first freeze the pretrained model parameters and then replace the original output layer with a new classification head:\nfor param in model.parameters():\n    param.requires_grad = False\n\ntorch.manual_seed(123)\nmodel.out_head = torch.nn.Linear(BASE_CONFIG[\"emb_dim\"], 2)\n\nfor layer in (model.trf_blocks[-1], model.final_norm):\n    for param in layer.parameters():\n        param.requires_grad = True\nThe new output head is trainable by default. We also make the final transformer block and final normalization layer trainable.\nThe earlier layers retain the language patterns learned during pretraining, while the later layers can adapt those representations for spam classification. This is much more efficient than retraining the entire GPT-2 model from scratch.\nUpdating the Loss and Evaluation\nThe final change is in how we train and evaluate the model.\nUltimately, we care about classification accuracy: the percentage of messages correctly identified as spam or not spam. Accuracy itself, however, depends on selecting the class with the highest score. This is a discrete decision and does not provide the gradients needed to update the model.\nWe therefore train the model using cross-entropy loss and calculate accuracy separately. Cross-entropy compares the model’s class logits with the correct label and gives us a differentiable value that can be minimized during training.\nThere is also a difference in which outputs contribute to the loss:\nNext-token prediction:\nLoss is calculated across every token position\n\nText classification:\nLoss is calculated using only the final sequence position\nThe model still produces two class logits at every position, but we select only the logits from the final position:\nlogits = model(input_batch)[:, -1, :]\nloss = torch.nn.functional.cross_entropy(logits, target_batch)\nBecause GPT uses causal attention, the representation at the final position can use information from all the positions that came before it. It therefore contains information accumulated across the sequence.\nFor a padded input, this final position will usually contain a padding token. Its representation can still draw on the actual message tokens before it, although it may also be influenced by the padding positions.\nIt is worth clarifying that the final token does not literally contain the answer. The actual label—\n0\nor\n1\n—is stored separately. We simply compare that label with the two class logits produced at the final sequence position.\nDuring evaluation, we select the class with the larger logit and check whether it matches the actual label.\nTraining on an SMS Spam Dataset\nWith these changes in place, we are ready to fine-tune the model.\nI used the public SMS Spam Collection dataset, which contains messages labelled as either\nham\n, meaning not spam, or\nspam\n.\ndf = pd.read_csv(\n    data_file_path, sep=\"\\t\", header=None, names=[\"Label\", \"Text\"]\n)\nA small preview of the data looks like this:\nIndex\nLabel\nText\n0\nham\nGo until Jurong Point, crazy… Available only…\n1\nham\nOk lar… Joking wif u oni…\n2\nspam\nFree entry in 2 a wkly comp to win FA Cup fina…\n…\n…\n…\n5570\nham\nThe guy did some bitching, but I acted like I'd…\n5571\nham\nRofl. It's true to its name.\nFile used in this experiment:\n5,572 messages\nThe current UCI dataset page lists 5,574 instances, while the downloaded file used in this notebook contains 5,572 rows. Dataset counts can therefore vary slightly depending on the distributed version of the file.\nThe data used here is heavily imbalanced:\nNot spam: 4,825\nSpam:       747\nA classifier trained directly on this data could achieve high accuracy by predicting\nnot spam\nmost of the time. To create a more balanced experiment, I retained all 747 spam messages and randomly selected the same number of non-spam messages:\nspam = df[df[\"Label\"] == \"spam\"]\nham = df[df[\"Label\"] == \"ham\"].sample(len(spam), random_state=123)\n\ndf = pd.concat([ham, spam]).sample(frac=1, random_state=123)\ndf[\"Label\"] = df[\"Label\"].map({\"ham\": 0, \"spam\": 1})\nThis gives us 1,494 examples, evenly divided between the two classes. I then split them into training, validation, and test sets using a\n70% / 10% / 20%\nsplit.\nThe training set updates the model’s trainable weights. The validation set helps us monitor performance during training, while the test set gives us a final measure using data the model did not see during training.\nFine-Tuning the Model\nI trained the model for five epochs using AdamW:\ntorch.manual_seed(123)\noptimizer = torch.optim.AdamW(\n    (p for p in model.parameters() if p.requires_grad),\n    lr=1e-5, weight_decay=0.1\n)\n\nresults = train_classifier_simple(\n    model, train_loader, val_loader, optimizer, device,\n    num_epochs=5, eval_freq=50, eval_iter=5\n)\nA shortened version of the training output is shown below:\nEpoch 1, Step 0:   Train Loss 3.0852, Val Loss 2.4835\nEpoch 1, Step 50:  Train Loss 0.5300, Val Loss 0.4591\n...\nEpoch 5, Step 550: Train Loss 0.0246, Val Loss 0.0727\nEpoch 5, Step 600: Train Loss 0.1635, Val Loss 0.1263\n\nCheckpoint train accuracy: 95.0%\nCheckpoint val accuracy:   97.5%\nTraining time: 641.97 minutes\nOn my local setup, completing five epochs took approximately 642 minutes—or a little over ten hours. GPT-2 Small may be small compared with modern commercial models, but fine-tuning it locally can still be computationally expensive.\nTrying the Classifier\nOnce training was complete, I tested the model using two custom messages:\ntexts = [\n    \"Congratulations! You've won a free ticket to Bahamas. Call now to claim your prize.\",\n    \"Hey, are we still on for dinner tonight?\"\n]\n\nfor text in texts:\n    print(classify_review(\n        text, model, tokenizer, device,\n        max_length=train_dataset.max_length\n    ))\nThe predictions were:\nspam\nnot spam\nThe model correctly classified both examples.\nFinally, I saved the trained weights so that I could reuse the classifier without running the full training process again:\ntorch.save(model.state_dict(), \"spam_classifier.pth\")\nOne Interesting Observation\nWhile experimenting with custom messages, I noticed that the classifier generally performed better on longer, clearly spam-like messages than on very short inputs.\nOne possible reason is the way the inputs are prepared. Every message is padded to the same maximum length, and the classifier uses the representation at the final sequence position. A short message therefore contains many more padding tokens than a longer message.\nThis does not mean padding is necessarily the only reason for an incorrect prediction. However, it shows how preprocessing decisions can affect real-world model behaviour. A model may achieve reasonable test accuracy while still behaving differently on inputs whose length or writing style differs from the training data.\nThis is also an important product lesson: a single overall accuracy number does not tell us how a model will behave across every type of real-world input.\nConclusion\nIn this article, we converted our pretrained GPT-2 model from a next-token predictor into a binary text classifier.\nWe standardized inputs using padding and truncation, replaced the vocabulary output layer with a two-class classification head, froze most of the pretrained model, and updated the loss calculation to use the class logits from the final sequence position. We then fine-tuned the model on an SMS spam dataset and used it to classify new messages as spam or not spam.\nIn the next article, we will explore another application of the same model: instruction fine-tuning. Instead of selecting between a fixed set of classes, the model will learn to generate a useful response to an instruction.\nAfter that, we will move into the experimentation phase and start changing different parts of the model to understand how architecture and training choices affect its behaviour.\nReferences\nSebastian Raschka —\nBuild a Large Language Model (From Scratch)\nThe primary reference for this project, covering tokenization, attention, GPT-style model construction, pretraining, classification fine-tuning, and instruction fine-tuning.\nBook on Manning\n·\nOfficial companion repository\nAman Jain — LLMs From Scratch Project\nThe code accompanying this article series, including the baseline model, GPT-2 weight loading, classification fine-tuning, and instruction fine-tuning.\nView the project on GitHub\nUCI Machine Learning Repository — SMS Spam Collection\nThe public SMS dataset used for the classification experiment.\nView the dataset\nOpenAI — GPT-2\nOpenAI’s official GPT-2 code, model configurations, tokenizer files, and released checkpoints.\nGPT-2 repository\n·\nGPT-2 paper\n#\nmachine-learning\n#\ngpt\n#\nclassification\n2\nviews",
  "metadata": {
    "title": "Fine-Tuning GPT-2 for Text Classification",
    "description": "Learn how to adapt a pretrained GPT-2 model for SMS spam classification using padding, a two-class output head, and classification fine-tuning.",
    "keywords": null,
    "author": null,
    "og:title": "Fine-Tuning GPT-2 for Text Classification",
    "og:description": "Learn how to adapt a pretrained GPT-2 model for SMS spam classification using padding, a two-class output head, and classification fine-tuning.",
    "og:url": "https://curious-pm.hashnode.dev/building-a-gpt-model-from-scratch-part-5-fine-tuning-gpt-2-for-text-classification",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/09cc2f2e-a631-4d80-bc64-6b0e0e3ad3cf.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Fine-Tuning GPT-2 for Text Classification",
    "twitter:description": "Learn how to adapt a pretrained GPT-2 model for SMS spam classification using padding, a two-class output head, and classification fine-tuning.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/09cc2f2e-a631-4d80-bc64-6b0e0e3ad3cf.png",
    "article:published_time": "2026-07-16T04:48:47.151Z",
    "article:modified_time": "2026-07-16T04:48:47.151Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/be6f794c0539/code/body-000.txt",
      "chars": 89,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-001.txt",
      "chars": 117,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-002.txt",
      "chars": 492,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-003.txt",
      "chars": 22,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-004.txt",
      "chars": 121,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-005.txt",
      "chars": 277,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-006.txt",
      "chars": 150,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-007.txt",
      "chars": 101,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-008.txt",
      "chars": 87,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-009.txt",
      "chars": 32,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-010.txt",
      "chars": 215,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-011.txt",
      "chars": 280,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-012.txt",
      "chars": 317,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-013.txt",
      "chars": 288,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-014.txt",
      "chars": 14,
      "language": "text"
    },
    {
      "file": "attachments/be6f794c0539/code/body-015.txt",
      "chars": 54,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/09cc2f2e-a631-4d80-bc64-6b0e0e3ad3cf.png",
      "bytes": 1259174,
      "reused": true,
      "local_path": "attachments/be6f794c0539/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/09cc2f2e-a631-4d80-bc64-6b0e0e3ad3cf.png",
      "alt": "Building a GPT Model from Scratch (Part 5): Fine-Tuning GPT-2 for Text Classification",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/f391b199-ccff-4194-acfc-deefd0898407.png",
      "bytes": 22710,
      "reused": true,
      "local_path": "attachments/be6f794c0539/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/f391b199-ccff-4194-acfc-deefd0898407.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/4a9a3ff9-e1a5-4264-ba06-dd9ce5636f3c.png",
      "bytes": 53493,
      "reused": true,
      "local_path": "attachments/be6f794c0539/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/4a9a3ff9-e1a5-4264-ba06-dd9ce5636f3c.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/93db215c-059d-44ab-b942-484dc37667c9.png",
      "bytes": 56148,
      "reused": true,
      "local_path": "attachments/be6f794c0539/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/93db215c-059d-44ab-b942-484dc37667c9.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/building-a-gpt-model-from-scratch-part-5-fine-tuning-gpt-2-for-text-classification.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/building-gpt-from-scratch",
      "label": "Part of series Building a GPT Model From Scratch",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/building-gpt-model-from-scratch-part-4-training-and-pretrained-weights",
      "label": "previous article",
      "type": "a"
    },
    {
      "href": "https://www.manning.com/books/build-a-large-language-model-from-scratch",
      "label": "Book on Manning",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch",
      "label": "Official companion repository",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work",
      "label": "View the project on GitHub",
      "type": "a"
    },
    {
      "href": "https://archive.ics.uci.edu/dataset/228/sms%2Bspam%2Bcollection",
      "label": "View the dataset",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/gpt-2",
      "label": "GPT-2 repository",
      "type": "a"
    },
    {
      "href": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "label": "GPT-2 paper",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/machine-learning",
      "label": "# machine-learning",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/gpt",
      "label": "# gpt",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/classification",
      "label": "# classification",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:17",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:55Z",
  "source_url": "https://curious-pm.hashnode.dev/building-a-gpt-model-from-scratch-part-5-fine-tuning-gpt-2-for-text-classification",
  "papers": [
    {
      "status": "ok",
      "url": "https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf",
      "bytes": 582775,
      "reused": true,
      "local_path": "papers/repair/3c62359064baa28a7743.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 12589,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/be6f794c0539/pages/source.html",
  "local_html": "attachments/be6f794c0539/pages/article.html",
  "body_base_path": "attachments/be6f794c0539/pages/",
  "body_markdown_path": "attachments/be6f794c0539/pages/article.md"
}