{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "Comparing Fine-Tuning Strategies: Which Parts of a GPT Model Should We Update?",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/comparing-fine-tuning-strategies-which-parts-of-a-gpt-model-should-we-update",
  "publish_date": "2026-07-21",
  "corpus_stage": "",
  "notes": "",
  "id": "09dfc2129936",
  "collected_at": "2026-10-02T14:11:44",
  "title": "Comparing GPT Fine-Tuning Strategies: Head / Partial / Full",
  "body_markdown": "Comparing Fine-Tuning Strategies: Which Parts of a GPT Model Should We Update?\nA controlled experiment comparing head-only, partial-layer, and full-backbone fine-tuning across accuracy, efficiency, and preservation.\nUpdated\nJuly 21, 2026\n•\n7\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nInside GPT: Experiments and Trade-offs\nTL;DR\nI compared four ways of fine-tuning a small GPT model for SMS spam classification:\nTrain only the classification head\nTrain the final transformer block\nTrain the final half of the transformer blocks\nTrain the full backbone\nTraining only the head reached\n72.33% test accuracy\n. Unfreezing the final transformer block increased accuracy to\n84.67%\nwhile updating only\n1.45% of the model\n.\nFull-backbone fine-tuning achieved the highest test accuracy at\n87.33%\n, but it took more than twice as long as last-block fine-tuning and changed the model’s original language behavior the most.\nThe main takeaway:\nupdating more of the model improved peak task performance, but most of the useful adaptation came from unfreezing just one transformer block.\nThe complete experiment notebook and supporting code are available in my\nGitHub repository\n.\nWhy Compare Fine-Tuning Strategies?\nFine-tuning is often presented as a simple final step: load a pretrained model and continue training it on a new task.\nBut what exactly should continue training?\nThe most flexible option is to update the entire model. The cheapest is to freeze the pretrained backbone and train only a new output head. Between those extremes, we can unfreeze only the later transformer blocks.\nThat creates a three-way trade-off:\nAdaptation:\nHow well does the model learn the new task?\nEfficiency:\nHow much time, memory, and trainable capacity does it require?\nPreservation:\nHow much of the original language-model behavior remains?\nThe goal was not to find a universal freezing rule. It was to compare these choices in one controlled small-model setup.\nExperiment Setup\nI started each run from the same pretrained four-block GPT checkpoint and adapted it to balanced SMS spam classification.\nI compared four strategies:\nStrategy\nComponents updated\nTrainable parameters\nHead only\nClassification head\n258\nLast block\nFinal block, final norm, and classification head\n198,274\nLast half\nFinal two blocks, final norm, and classification head\n396,034\nFull backbone\nEmbeddings, all four blocks, final norm, and classification head\n7,240,834\nThe language-model output head remained frozen in every run so that I could use it as a consistent probe of the model’s original behavior. Therefore, “full backbone” does not mean that every stored parameter was trainable.\nEach strategy used the same:\nInitial checkpoint and classification-head initialization\nTraining data and sample order\nOptimizer and learning-rate schedule\nRandom seed\nFixed budget of\n200,192 padded token positions\nFor adaptation, I measured classification validation loss, validation accuracy, and test accuracy.\nFor efficiency, I measured runtime, throughput, trainable parameters, and peak memory increase.\nFor preservation, I evaluated the model on held-out WikiText before and after fine-tuning: A larger positive value means that the model became worse at its original language-model objective.\nResults\nStrategy\nTest accuracy\nRuntime\nPeak memory increase\nLM degradation\nHead only\n72.33%\n36.6s\n9.3 MB\n0.0000\nLast block\n84.67%\n30.9s\n21.1 MB\n0.0058\nLast half\n85.00%\n46.5s\n15.9 MB\n0.0156\nFull backbone\n87.33%\n71.1s\n118.5 MB\n0.0262\nThe CPU runtime and memory measurements can be noisy, so I would not conclude that head-only training is inherently slower than last-block training. The larger differences between selective and full-backbone fine-tuning are more meaningful.\nDimension\nMetric\nHead only\nLast block\nLast half\nFull model\nEfficiency\nTrainable parameters\n258\n198,274\n396,034\n7,240,834\nEfficiency\nTrainable share\n0.00%\n1.45%\n2.90%\n52.95%\nEfficiency\nRuntime\n36.6s\n30.9s\n46.5s\n71.1s\nEfficiency\nPeak memory increase\n9.3 MB\n21.1 MB\n15.9 MB\n118.5 MB\nAdaptation\nInitial validation accuracy\n48.99%\n48.99%\n48.99%\n48.99%\nAdaptation\nFinal validation accuracy\n75.17%\n83.22%\n84.56%\n81.21%\nAdaptation\nTest accuracy\n72.33%\n84.67%\n85.00%\n87.33%\nPreservation\nOriginal LM loss degradation\n0.0000\n0.0058\n0.0156\n0.0262\nWhat I Learned\nTraining Only the Head Was Too Restrictive\nThe head-only model improved from an initial validation accuracy of\n48.99%\nto\n75.17%\n, showing that the pretrained representations were useful.\nHowever, test accuracy stopped at\n72.33%\n. The new classifier could reorganize the features produced by the model, but it could not change the features themselves.\nIts main advantage was preservation. Because the backbone never changed, the original language-model loss remained identical.\nOne Transformer Block Produced the Biggest Gain\nUnfreezing the final transformer block increased the trainable share to only\n1.45%\n, but test accuracy jumped by\n12.34 percentage points\n:\n[ 72.33% \\rightarrow 84.67% ]\nIt also recorded the highest measured throughput and increased the original language-model loss by only\n0.0058\n.\nThis was the clearest improvement in the experiment: allowing a relatively small part of the pretrained model to adapt delivered most of the available task performance.\nThe Last Half Showed Diminishing Returns\nTraining the final two blocks doubled the number of trainable transformer parameters, but test accuracy improved by only\n0.33 percentage points\nover training one block.\nValidation performance was slightly better, but runtime increased and language-model degradation rose from\n0.0058\nto\n0.0156\n.\nThe second unfrozen block helped—but much less than the first.\nFull Fine-Tuning Maximized Accuracy, Not Efficiency\nUpdating the full backbone produced the highest test accuracy at\n87.33%\n.\nCompared with last-block fine-tuning, however, it:\nTrained about\n36 times more parameters\nTook approximately\n2.3 times as long\nCaused around\n4.5 times more language-model degradation\nIt also finished with worse validation accuracy than the two selective fine-tuning strategies. Since this was a single run with relatively small validation and test splits, some of that difference could reflect sampling or training variation.\nThere Was No Universal Winner\nAll four strategies were Pareto-efficient across test accuracy, runtime, and language-model preservation. Each retained at least one advantage:\nHead-only fine-tuning preserved the original model perfectly.\nLast-block fine-tuning delivered the largest practical accuracy jump.\nLast-half fine-tuning achieved the best validation result.\nFull-backbone fine-tuning achieved the best test accuracy.\nStill, the\nlast-block strategy looked like the practical knee point\n. It captured most of the achievable accuracy while updating a very small part of the model and causing little degradation.\nFinal Takeaway\nFine-tuning more parameters improved the best task result—but the gains were not proportional to the cost.\nIn this setup, moving from a trainable classification head to one trainable transformer block mattered enormously. Moving from one block to the full backbone mattered much less.\nThe broader lesson is not that the final block will always be enough. It is that\nwhich parameters we update should be treated as an experimental choice, not a default setting\n.\nAppendix: Research References\nUniversal Language Model Fine-tuning for Text Classification\n— Howard and Ruder introduced techniques such as gradual unfreezing for adapting pretrained language models to downstream classification tasks.\nBitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models\n— Ben Zaken, Goldberg, and Ravfogel showed that updating only bias parameters can remain competitive with full fine-tuning in some small- and medium-data settings.\nSurgical Fine-Tuning Improves Adaptation to Distribution Shifts\n— Lee and colleagues found that selectively updating subsets of layers can match or outperform full fine-tuning under certain distribution shifts.\nFreezeOut: Accelerate Training by Progressively Freezing Layers\n— Brock and colleagues explored progressively freezing layers to reduce backward-pass computation during training.\n#\ndeeplearning\n#\nllm\n#\nfinetuning\n11\nviews",
  "metadata": {
    "title": "Comparing GPT Fine-Tuning Strategies: Head / Partial / Full",
    "description": "I compare four GPT fine-tuning strategies—to measure accuracy, runtime, memory use, and preservation of the model’s original language behavior.",
    "keywords": null,
    "author": null,
    "og:title": "Comparing GPT Fine-Tuning Strategies: Head / Partial / Full",
    "og:description": "I compare four GPT fine-tuning strategies—to measure accuracy, runtime, memory use, and preservation of the model’s original language behavior.",
    "og:url": "https://curious-pm.hashnode.dev/comparing-fine-tuning-strategies-which-parts-of-a-gpt-model-should-we-update",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/db38534c-8075-44dc-8f5e-4892f400e573.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Comparing GPT Fine-Tuning Strategies: Head / Partial / Full",
    "twitter:description": "I compare four GPT fine-tuning strategies—to measure accuracy, runtime, memory use, and preservation of the model’s original language behavior.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/db38534c-8075-44dc-8f5e-4892f400e573.png",
    "article:published_time": "2026-07-21T04:19:13.079Z",
    "article:modified_time": "2026-07-21T04:19:13.079Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/db38534c-8075-44dc-8f5e-4892f400e573.png",
      "bytes": 985658,
      "reused": true,
      "local_path": "attachments/09dfc2129936/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/db38534c-8075-44dc-8f5e-4892f400e573.png",
      "alt": "Comparing Fine-Tuning Strategies: Which Parts of a GPT Model Should We Update?",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/83ebae97-f219-457d-be64-e1b40ce2b9c6.png",
      "bytes": 19088,
      "reused": true,
      "local_path": "attachments/09dfc2129936/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/83ebae97-f219-457d-be64-e1b40ce2b9c6.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/901aec62-5cc6-48df-b181-e8111d4f4de8.png",
      "bytes": 1131996,
      "reused": true,
      "local_path": "attachments/09dfc2129936/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/901aec62-5cc6-48df-b181-e8111d4f4de8.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/ffa6db17-8037-4895-8872-ffbd9a637d38.png",
      "bytes": 57046,
      "reused": true,
      "local_path": "attachments/09dfc2129936/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/ffa6db17-8037-4895-8872-ffbd9a637d38.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/dc07f7e3-9990-4c44-9022-eb7a131eb80b.png",
      "bytes": 28113,
      "reused": true,
      "local_path": "attachments/09dfc2129936/images/004.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/dc07f7e3-9990-4c44-9022-eb7a131eb80b.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/comparing-fine-tuning-strategies-which-parts-of-a-gpt-model-should-we-update.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/inside-gpt-experiments-and-trade-offs",
      "label": "Part of series Inside GPT: Experiments and Trade-offs",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work",
      "label": "GitHub repository",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/P18-1031/",
      "label": "Universal Language Model Fine-tuning for Text Classification",
      "type": "a"
    },
    {
      "href": "https://aclanthology.org/2022.acl-short.1/",
      "label": "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2210.11466",
      "label": "Surgical Fine-Tuning Improves Adaptation to Distribution Shifts",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.04983",
      "label": "FreezeOut: Accelerate Training by Progressively Freezing Layers",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/deeplearning",
      "label": "# deeplearning",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/llm",
      "label": "# llm",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/finetuning",
      "label": "# finetuning",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:08:55",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:54Z",
  "source_url": "https://curious-pm.hashnode.dev/comparing-fine-tuning-strategies-which-parts-of-a-gpt-model-should-we-update",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.11466",
      "bytes": 2103113,
      "reused": false,
      "local_path": "papers/repair/039681f5774538e63801.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.04983",
      "bytes": 418920,
      "reused": false,
      "local_path": "papers/repair/5a698785e61df0f10a37.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 8325,
  "expected_body_images": 5,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/09dfc2129936/pages/source.html",
  "local_html": "attachments/09dfc2129936/pages/article.html",
  "body_base_path": "attachments/09dfc2129936/pages/",
  "body_markdown_path": "attachments/09dfc2129936/pages/article.md"
}