From Building a GPT Model to Experimenting With One

Over the last few articles, I built the transformer block, assembled the complete GPT architecture, loaded pretrained GPT-2 weights, fine-tuned the model for classification and instruction tasks, and even used another LLM as a judge.

At this point, the model was beginning to feel less like a mysterious black box and more like a toy I had assembled myself—which naturally led to the most responsible next step: start changing the parts and see what breaks.

This article begins a new series: Inside GPT: Experiments and Trade-offs.

Why move into experiments?

Building the model helped me understand how its pieces fit together, but it also created a new set of questions.

Once you understand how a toy works, it is difficult to resist opening it again, replacing a few parts, turning one dial too far, and checking whether it still runs. That curiosity is the motivation behind this next phase.

I want to change individual training and architecture decisions, compare the results, and see whether the model behaves the way my intuition says it should. Sometimes that intuition will be right. Sometimes the experiment will expose something I had not considered.

Either outcome is useful.

What I plan to experiment with

I am organizing this phase around six broader areas. Each area can support its own article in this series.

• Learning rate and optimization: Explore different learning-rate ranges, how learning rate interacts with batch size, whether it should decay during training, and whether every layer should use the same rate.

• Context length: Compare shorter and longer input windows and examine the trade-offs between available context, training speed, memory usage, and model quality.

• Model size: Change the embedding dimensions, number of transformer blocks, and overall parameter count to see how additional capacity affects learning and compute requirements.

• Architecture choices: Experiment with attention heads, normalization placement, activation choices, and positional embeddings while keeping the rest of the model stable.

• Fine-tuning strategy: Compare which parts of the model are updated during fine-tuning, including freezing early layers, training only later layers, or updating the full model.

• Generation settings: Explore temperature, top-k sampling, and related decoding choices to understand how they affect repetition, coherence, predictability, and creativity.

The purpose is not to produce a leaderboard or claim that one configuration is universally correct.

I want to understand how these decisions interact, why a particular choice works in one setup, and what trade-offs it creates elsewhere.

The experimentation framework

The GPT-2 Small configuration I used earlier has roughly 124 million trainable parameters in my implementation.

That is manageable for loading pretrained weights and running inference. Repeatedly training it across dozens of configurations, however, would be slow on my local machine.

For this experiment phase, I created a much smaller GPT-style model with approximately 13 million parameters (Github):

BASE_MODEL_CONFIG = {
    "vocab_size": 50_257,
    "context_length": 128,
    "emb_dim": 128,
    "n_heads": 4,
    "n_layers": 4,
    "drop_rate": 0.1,
    "qkv_bias": False,
}

Compared with the GPT-2 Small configuration, I reduced the context length, embedding dimensions, number of attention heads, and number of transformer blocks.

This model is not intended to compete with GPT-2. It is an experimental sandbox.

A smaller model allows me to run several controlled training jobs locally, catch implementation issues quickly, and spend more time understanding the results instead of waiting for one large run to finish.

The exact findings may not transfer directly to a much larger model. However, the smaller model can still help reveal relationships, unexpected behavior, and questions worth investigating further.

Insert notebook image: Compact model configuration and parameter count.

Separating the configuration

I also wanted the experiment code to be reusable.

Without a common framework, every experiment could slowly develop its own training loop, data split, evaluation logic, and result format. Over time, those small differences would make the runs difficult to compare.

The framework separates the settings into three groups:

BASE_MODEL_CONFIG
BASE_TRAINING_CONFIG
BASE_DATA_CONFIG

The model configuration defines the architecture.

The training configuration contains settings such as the batch size, learning rate, number of epochs, evaluation frequency, and random seed:

BASE_TRAINING_CONFIG = {
    "batch_size": 2,
    "learning_rate": 3e-4,
    "weight_decay": 0.1,
    "num_epochs": 1,
    "eval_freq": 5,
    "eval_iter": 2,
    "train_ratio": 0.90,
    "seed": 123,
}

The data configuration identifies the dataset and its tokenization settings:

BASE_DATA_CONFIG = {
    "file_name": "wikitext-2-500k.txt",
    "allowed_special": {"<|endoftext|>"},
}

Separating these settings makes it easier to change one variable without unintentionally modifying something else.

For example, a learning-rate experiment should change the learning rate while keeping the model architecture, dataset, batch size, random seed, and evaluation approach stable.

It is similar to testing two recipes by changing only the amount of salt. If I also change the cooking time, temperature, and ingredients, I may get a different result—but I will not know which change caused it.

Describing each experiment

Each run is represented using an experiment specification:

@dataclass
class ExperimentSpec:
    name: str
    question: str
    hypothesis: str
    independent_variable: str
    variable_value: Any
    controlled_variables: list[str]

    model_overrides: dict[str, Any] = field(default_factory=dict)
    training_overrides: dict[str, Any] = field(default_factory=dict)
    data_overrides: dict[str, Any] = field(default_factory=dict)

An experiment specification records:

• the question I am trying to answer;

• the hypothesis before seeing the result;

• the variable being changed;

• the variables that should remain controlled;

• any changes to the model, training, or data configuration.

For example, the first baseline run asks whether the framework itself can complete a training run:

BASELINE_SPEC = ExperimentSpec(
    name="baseline",
    question="Can the reusable pipeline complete a training run?",
    hypothesis="The compact model should train and reduce its loss.",
    independent_variable="none",
    variable_value="baseline",
    controlled_variables=[
        "dataset",
        "tokenizer",
        "random seed",
        "model architecture",
        "optimizer",
    ],
)

Writing the question and hypothesis before running the model is important.

Otherwise, it becomes very easy to look at the final graph and create an explanation that happens to fit the result.

Running and saving the experiment

The reusable runner handles the common work:

model, history, result = run_experiment(
    spec=BASELINE_SPEC,
    save_checkpoint=False,
)

For every run, the framework records information such as:

• model and training configuration;

• initial and final loss;

• best validation loss;

• number of tokens processed;

• optimizer steps;

• parameter count;

• runtime;

• training history.

The detailed result is stored in a JSON file, while a shared CSV file makes it easier to compare multiple runs in a table.

Each run also receives a unique identifier. Running a new experiment therefore adds another record instead of overwriting the previous result. The complete implementation is available in the experiment notebook in my GitHub repository:

View the experiment notebook on GitHub

The framework does not make the experiments automatically correct. I still need to decide what should remain fixed, what metric matters, and whether the comparison is actually fair.

It simply gives every experiment the same basic foundation.

A trial experiment

Before designing more ambitious experiments, I wanted to make sure the pipeline itself worked.

Could I define several configurations, train each one, save the results, and compare them without manually rebuilding the notebook every time?

To test that, I ran a simple learning-rate sweep:

LEARNING_RATES = [
    1e-5,
    1e-4,
    3e-4,
    1e-3,
    1e-2,
    1e-1,
]

For each value, the framework created a new experiment specification and changed only the learning rate:

training_overrides={
    "learning_rate": learning_rate,
    "num_epochs": 2,
    "seed": 123,
}

Everything else remained fixed:

• batch size;

• model configuration;

• dataset and train-validation split;

• random seed;

• weight decay;

• optimizer;

• number of epochs;

• evaluation method.

The point of this sweep was not to conduct a complete investigation into learning rates. It was a trial run for the experimentation pipeline.

The different learning rates produced clearly different training behavior.

Very conservative values changed the model slowly. More aggressive values allowed the loss to fall faster, but increasing the rate too far also made training less stable.

That was already enough to show why a single experiment value rarely tells the complete story. A value may be too small, somewhere in a useful range, or aggressive enough to push the model beyond that range.

More importantly, the pipeline worked.

I could define multiple runs, keep the controlled settings stable, capture the results consistently, and compare their validation-loss curves without rewriting the training code.

That gave me enough confidence to move from testing the framework to using it for an actual investigation.

Where this leads next

The trial experiment answered the pipeline question, but it immediately created several new model questions.

How far could I increase the learning rate before training became unstable? Would the useful range change with batch size? Should the learning rate remain fixed throughout training or gradually decrease? And does it make sense to apply the same learning rate to the embeddings, transformer blocks, and output layer?

Those questions became the first real investigation in this series. In the next article, I will explore learning rate not as one isolated value in a configuration file, but as a decision that interacts with batch size, training progress, and different parts of the model.

The experiment framework is ready. It is time to start turning the dials.