Why I Built a GPT Model From Scratch — and What I Actually Built

“To exist is to change, to change is to mature, to mature is to go on creating oneself endlessly.” - Henri Bergson

Let me start by saying a little bit about myself, just to put this whole exercise in perspective. I am a product manager by profession. So, almost instinctively, before getting into how to build a large language model, I found myself stepping back and asking two questions: Why am I doing this? And what exactly am I trying to build?. I think this context is worth sharing because I hope this series motivates a few other people like me to get their hands dirty. Part of my own motivation came from reading about other people's learning journeys, so it feels only fair to share mine.

I am a fairly late adopter of generative AI

I was not one of those people who immediately looked at Chat GPT and declared that everything was about to change. My journey was more gradual. A few things started to pull me in. Seeing what Sora could do was one of them. Another was working with an LLM-powered data analytics tool at work and watching its capabilities improve over time. I also started using Claude more frequently at work. What surprised me was not just what these tools could do at a particular point in time. It was the step change in their capabilities over relatively short periods. 

To be really honest, my initial motivation for learning more about AI was not entirely intellectual curiosity, but was simply future-proofing. AI clearly represented a major shift in the technology industry, and I believed that a significant part of the next wave of technological growth would come from this space. As someone working in technology, ignoring that shift did not seem like a particularly good career strategy.

But something changed as I went deeper.

At some point, curiosity took over from career planning. I am still not sure whether understanding this level of technical detail will create a direct step change in how I perform as a product manager. Maybe it will. Maybe some of what I am learning will never be directly useful in my day-to-day role. But by then, that had stopped being the point. I was simply curious.

I have always had a tendency to ask: “How do things actually work?” It could be software. A bowling machine. A government office. Once something catches my attention, I want to understand what is happening underneath the surface. Naturally, that curiosity eventually landed on large language models. Three things, in particular, pushed me deeper.

• Trigger 1: Watching an AI product improve in front of me. The first trigger was an LLM-powered data analytics tool that I use at work. This was probably my first sustained exposure to an AI product in a real working environment - not just a demo or something I experimented with for a few minutes. Over time, I watched the quality of its output improve significantly. The experience gave me conviction that AI was not simply another technology trend I could casually follow from the sidelines. The capabilities were improving too quickly.

• Trigger 2: 3Blue1Brown made the internals feel understandable. The second trigger was 3Blue1Brown’s series on neural networks, transformers, and LLMs. Its visual explanations of attention and next-token prediction made the technology feel much more understandable. Then something clicked: LLMs are still neural networks - just far larger and more sophisticated ones. Since I had worked with simpler neural networks in graduate school, I decided to explore the technical foundations before moving to AI applications or agent frameworks. 

• Trigger 3: The book that turned curiosity into a project: The third trigger was Sebastian Raschka’s book, Build a Large Language Model (From Scratch). It walks through building a GPT-style model in Python and PyTorch, from tokenization to fine-tuning. I expected to read only a few chapters, but quickly got hooked. Watching the training loss fall brought back the excitement, confusion, and satisfaction of my graduate-school projects.

At that point, I was hooked.

But should a product manager really understand all of this?

This is probably the obvious next question. Unless you are directly building AI models, does a product manager really need to understand tokenizers, attention mechanisms, embeddings, activation functions, or training loops? My honest answer is: Probably not. At least not at this level of detail.

This is similar to asking whether a product manager needs to know how to code. Not necessarily—but understanding how software is built helps you follow technical discussions, evaluate trade-offs, and make better decisions. The same applies to AI: most product roles may only require a high-level understanding of LLMs. 

Understanding multi-head attention may not transform how I perform as a product manager, but it gives me a better mental model. Concepts like context length, embeddings, inference, and fine-tuning are no longer abstract. I have seen the pieces, written some of them, broken them, and spent far too long debugging tensor dimensions. 

It makes the technology feel little less like magic - and, for me, that alone has been worth it.

So, what are we going to build—and not build—in this series?

This series is about going underneath the applications and understanding the fundamentals of a GPT-style large language model. We will look at things such as tokenization, embeddings, self-attention and multi-head attention, transformer blocks, loading pretrained and publicly available weights, and fine-tuning models for different applications. In other words, this series is going to get fairly technical.

But it is not about building the next Chat GPT.

The models I am working with are small by modern standards. I primarily work with GPT-2-sized architectures, including the 124-million-parameter and 355-million-parameter configurations. On my Windows setup, those were the sizes I could reasonably experiment with before hardware constraints started becoming much more noticeable. And that is completely fine.

The goal is not to reproduce a frontier-scale commercial model. The goal is to understand the machinery underneath one. For learning purposes, a smaller model lets us inspect the architecture, run experiments locally, make changes, break things, and understand what is happening. That is what I care about in this series.

What's next in this series?

SoI have now built my first GPT-style model and organized the work into four reusable notebooks, largely based on Sebastian Raschka’s Build a Large Language Model (From Scratch). After completing the book chapter by chapter, I restructured the code into a cleaner implementation I could experiment with.

1. Building a baseline GPT model: This notebook covers the full path from raw text to a working GPT architecture, including tokenization, data loaders, embeddings, attention, transformer blocks, training, and loss calculation.

2. Loading pretrained GPT-2 weights: I load publicly available GPT-2 weights into my custom architecture and use the model to generate text.

3. Fine-tuning for classification: This notebook adapts a next-token prediction model to classify an entire piece of text.

4. Fine-tuning for instruction following: Finally, I train the model to respond to instructions, bringing it closer to the interaction pattern used by tools such as Chat GPT.

I started this project because I thought understanding AI would be useful. I kept going because I became curious about how it worked. And somewhere along the way, I rediscovered how much fun it is to build something you do not fully understand yet—and then slowly make it less mysterious. I'll see you in the next article.

References

1. 3Blue1Brown — Transformers, the tech behind LLMs and the accompanying lessons on attention and how LLMs might store facts.

2. Sebastian Raschka — Build a Large Language Model (From Scratch). The book and companion materials cover tokenization, attention, GPT-style model construction, pretraining, and fine-tuning.

3. OpenAI — GPT-2 publications and release materials. GPT-2 was released in 124M, 355M, 774M, and 1.5B parameter configurations.

4. Github repo