A controlled experiment comparing head-only, partial-layer, and full-backbone fine-tuning across accuracy, efficiency, and preservation.
TL;DR
I compared four ways of fine-tuning a small GPT model for SMS spam classification:
Train only the classification head
Train the final transformer block
Train the final half of the transformer blocks
Train the full backbone
Training only the head reached 72.33% test accuracy. Unfreezing the final transformer block increased accuracy to 84.67% while updating only 1.45% of the model.
Full-backbone fine-tuning achieved the highest test accuracy at 87.33%, but it took more than twice as long as last-block fine-tuning and changed the model’s original language behavior the most.
The main takeaway: updating more of the model improved peak task performance, but most of the useful adaptation came from unfreezing just one transformer block.
The complete experiment notebook and supporting code are available in my GitHub repository.
Fine-tuning is often presented as a simple final step: load a pretrained model and continue training it on a new task.
But what exactly should continue training?
The most flexible option is to update the entire model. The cheapest is to freeze the pretrained backbone and train only a new output head. Between those extremes, we can unfreeze only the later transformer blocks.
That creates a three-way trade-off:
Adaptation: How well does the model learn the new task?
Efficiency: How much time, memory, and trainable capacity does it require?
Preservation: How much of the original language-model behavior remains?
The goal was not to find a universal freezing rule. It was to compare these choices in one controlled small-model setup.
I started each run from the same pretrained four-block GPT checkpoint and adapted it to balanced SMS spam classification.
I compared four strategies:
| Strategy | Components updated | Trainable parameters |
|---|---|---|
| Head only | Classification head | 258 |
| Last block | Final block, final norm, and classification head | 198,274 |
| Last half | Final two blocks, final norm, and classification head | 396,034 |
| Full backbone | Embeddings, all four blocks, final norm, and classification head | 7,240,834 |
The language-model output head remained frozen in every run so that I could use it as a consistent probe of the model’s original behavior. Therefore, “full backbone” does not mean that every stored parameter was trainable.
Each strategy used the same:
Initial checkpoint and classification-head initialization
Training data and sample order
Optimizer and learning-rate schedule
Random seed
Fixed budget of 200,192 padded token positions
For adaptation, I measured classification validation loss, validation accuracy, and test accuracy.
For efficiency, I measured runtime, throughput, trainable parameters, and peak memory increase.
For preservation, I evaluated the model on held-out WikiText before and after fine-tuning: A larger positive value means that the model became worse at its original language-model objective.
| Strategy | Test accuracy | Runtime | Peak memory increase | LM degradation |
|---|---|---|---|---|
| Head only | 72.33% | 36.6s | 9.3 MB | 0.0000 |
| Last block | 84.67% | 30.9s | 21.1 MB | 0.0058 |
| Last half | 85.00% | 46.5s | 15.9 MB | 0.0156 |
| Full backbone | 87.33% | 71.1s | 118.5 MB | 0.0262 |
The CPU runtime and memory measurements can be noisy, so I would not conclude that head-only training is inherently slower than last-block training. The larger differences between selective and full-backbone fine-tuning are more meaningful.
| Dimension | Metric | Head only | Last block | Last half | Full model |
|---|---|---|---|---|---|
| Efficiency | Trainable parameters | 258 | 198,274 | 396,034 | 7,240,834 |
| Efficiency | Trainable share | 0.00% | 1.45% | 2.90% | 52.95% |
| Efficiency | Runtime | 36.6s | 30.9s | 46.5s | 71.1s |
| Efficiency | Peak memory increase | 9.3 MB | 21.1 MB | 15.9 MB | 118.5 MB |
| Adaptation | Initial validation accuracy | 48.99% | 48.99% | 48.99% | 48.99% |
| Adaptation | Final validation accuracy | 75.17% | 83.22% | 84.56% | 81.21% |
| Adaptation | Test accuracy | 72.33% | 84.67% | 85.00% | 87.33% |
| Preservation | Original LM loss degradation | 0.0000 | 0.0058 | 0.0156 | 0.0262 |
The head-only model improved from an initial validation accuracy of 48.99% to 75.17%, showing that the pretrained representations were useful.
However, test accuracy stopped at 72.33%. The new classifier could reorganize the features produced by the model, but it could not change the features themselves.
Its main advantage was preservation. Because the backbone never changed, the original language-model loss remained identical.
Unfreezing the final transformer block increased the trainable share to only 1.45%, but test accuracy jumped by 12.34 percentage points:
[ 72.33% \rightarrow 84.67% ]
It also recorded the highest measured throughput and increased the original language-model loss by only 0.0058.
This was the clearest improvement in the experiment: allowing a relatively small part of the pretrained model to adapt delivered most of the available task performance.
Training the final two blocks doubled the number of trainable transformer parameters, but test accuracy improved by only 0.33 percentage points over training one block.
Validation performance was slightly better, but runtime increased and language-model degradation rose from 0.0058 to 0.0156.
The second unfrozen block helped—but much less than the first.
Updating the full backbone produced the highest test accuracy at 87.33%.
Compared with last-block fine-tuning, however, it:
Trained about 36 times more parameters
Took approximately 2.3 times as long
Caused around 4.5 times more language-model degradation
It also finished with worse validation accuracy than the two selective fine-tuning strategies. Since this was a single run with relatively small validation and test splits, some of that difference could reflect sampling or training variation.
All four strategies were Pareto-efficient across test accuracy, runtime, and language-model preservation. Each retained at least one advantage:
Head-only fine-tuning preserved the original model perfectly.
Last-block fine-tuning delivered the largest practical accuracy jump.
Last-half fine-tuning achieved the best validation result.
Full-backbone fine-tuning achieved the best test accuracy.
Still, the last-block strategy looked like the practical knee point. It captured most of the achievable accuracy while updating a very small part of the model and causing little degradation.
Fine-tuning more parameters improved the best task result—but the gains were not proportional to the cost.
In this setup, moving from a trainable classification head to one trainable transformer block mattered enormously. Moving from one block to the full backbone mattered much less.
The broader lesson is not that the final block will always be enough. It is that which parameters we update should be treated as an experimental choice, not a default setting.
Universal Language Model Fine-tuning for Text Classification — Howard and Ruder introduced techniques such as gradual unfreezing for adapting pretrained language models to downstream classification tasks.
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models — Ben Zaken, Goldberg, and Ravfogel showed that updating only bias parameters can remain competitive with full fine-tuning in some small- and medium-data settings.
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts — Lee and colleagues found that selectively updating subsets of layers can match or outperform full fine-tuning under certain distribution shifts.
FreezeOut: Accelerate Training by Progressively Freezing Layers — Brock and colleagues explored progressively freezing layers to reduce backward-pass computation during training.
