Comparing Fine-Tuning Strategies: Which Parts of a GPT Model Should We Update?

TL;DR

I compared four ways of fine-tuning a small GPT model for SMS spam classification:

• Train only the classification head

• Train the final transformer block

• Train the final half of the transformer blocks

• Train the full backbone

Training only the head reached 72.33% test accuracy. Unfreezing the final transformer block increased accuracy to 84.67% while updating only 1.45% of the model.

Full-backbone fine-tuning achieved the highest test accuracy at 87.33%, but it took more than twice as long as last-block fine-tuning and changed the model’s original language behavior the most.

The main takeaway: updating more of the model improved peak task performance, but most of the useful adaptation came from unfreezing just one transformer block.

The complete experiment notebook and supporting code are available in my GitHub repository.

Why Compare Fine-Tuning Strategies?

Fine-tuning is often presented as a simple final step: load a pretrained model and continue training it on a new task.

But what exactly should continue training?

The most flexible option is to update the entire model. The cheapest is to freeze the pretrained backbone and train only a new output head. Between those extremes, we can unfreeze only the later transformer blocks.

That creates a three-way trade-off:

1. Adaptation: How well does the model learn the new task?

2. Efficiency: How much time, memory, and trainable capacity does it require?

3. Preservation: How much of the original language-model behavior remains?

The goal was not to find a universal freezing rule. It was to compare these choices in one controlled small-model setup.

Experiment Setup

I started each run from the same pretrained four-block GPT checkpoint and adapted it to balanced SMS spam classification.

I compared four strategies:

The language-model output head remained frozen in every run so that I could use it as a consistent probe of the model’s original behavior. Therefore, “full backbone” does not mean that every stored parameter was trainable.

Each strategy used the same:

• Initial checkpoint and classification-head initialization

• Training data and sample order

• Optimizer and learning-rate schedule

• Random seed

• Fixed budget of 200,192 padded token positions

For adaptation, I measured classification validation loss, validation accuracy, and test accuracy.

For efficiency, I measured runtime, throughput, trainable parameters, and peak memory increase.

For preservation, I evaluated the model on held-out WikiText before and after fine-tuning: A larger positive value means that the model became worse at its original language-model objective.

Results

The CPU runtime and memory measurements can be noisy, so I would not conclude that head-only training is inherently slower than last-block training. The larger differences between selective and full-backbone fine-tuning are more meaningful.

What I Learned

Training Only the Head Was Too Restrictive

The head-only model improved from an initial validation accuracy of 48.99% to 75.17%, showing that the pretrained representations were useful.

However, test accuracy stopped at 72.33%. The new classifier could reorganize the features produced by the model, but it could not change the features themselves.

Its main advantage was preservation. Because the backbone never changed, the original language-model loss remained identical.

One Transformer Block Produced the Biggest Gain

Unfreezing the final transformer block increased the trainable share to only 1.45%, but test accuracy jumped by 12.34 percentage points:

[ 72.33% \rightarrow 84.67% ]

It also recorded the highest measured throughput and increased the original language-model loss by only 0.0058.

This was the clearest improvement in the experiment: allowing a relatively small part of the pretrained model to adapt delivered most of the available task performance.

The Last Half Showed Diminishing Returns

Training the final two blocks doubled the number of trainable transformer parameters, but test accuracy improved by only 0.33 percentage points over training one block.

Validation performance was slightly better, but runtime increased and language-model degradation rose from 0.0058 to 0.0156.

The second unfrozen block helped—but much less than the first.

Full Fine-Tuning Maximized Accuracy, Not Efficiency

Updating the full backbone produced the highest test accuracy at 87.33%.

Compared with last-block fine-tuning, however, it:

• Trained about 36 times more parameters

• Took approximately 2.3 times as long

• Caused around 4.5 times more language-model degradation

It also finished with worse validation accuracy than the two selective fine-tuning strategies. Since this was a single run with relatively small validation and test splits, some of that difference could reflect sampling or training variation.

There Was No Universal Winner

All four strategies were Pareto-efficient across test accuracy, runtime, and language-model preservation. Each retained at least one advantage:

• Head-only fine-tuning preserved the original model perfectly.

• Last-block fine-tuning delivered the largest practical accuracy jump.

• Last-half fine-tuning achieved the best validation result.

• Full-backbone fine-tuning achieved the best test accuracy.

Still, the last-block strategy looked like the practical knee point. It captured most of the achievable accuracy while updating a very small part of the model and causing little degradation.

Final Takeaway

Fine-tuning more parameters improved the best task result—but the gains were not proportional to the cost.

In this setup, moving from a trainable classification head to one trainable transformer block mattered enormously. Moving from one block to the full backbone mattered much less.

The broader lesson is not that the final block will always be enough. It is that which parameters we update should be treated as an experimental choice, not a default setting.

Appendix: Research References

1. Universal Language Model Fine-tuning for Text Classification — Howard and Ruder introduced techniques such as gradual unfreezing for adapting pretrained language models to downstream classification tasks.

2. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models — Ben Zaken, Goldberg, and Ravfogel showed that updating only bias parameters can remain competitive with full fine-tuning in some small- and medium-data settings.

3. Surgical Fine-Tuning Improves Adaptation to Distribution Shifts — Lee and colleagues found that selectively updating subsets of layers can match or outperform full fine-tuning under certain distribution shifts.

4. FreezeOut: Accelerate Training by Progressively Freezing Layers — Brock and colleagues explored progressively freezing layers to reduce backward-pass computation during training.