{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "From Building a GPT Model to Experimenting With One",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/from-building-a-gpt-model-to-experimenting-with-one",
  "publish_date": "2026-07-17",
  "corpus_stage": "",
  "notes": "",
  "id": "76ec8c281144",
  "collected_at": "2026-10-02T14:11:45",
  "title": "From Building a GPT Model to Experimenting With One",
  "body_markdown": "From Building a GPT Model to Experimenting With One\nCreating a reusable framework to explore the training and architecture decisions inside a small GPT model\nUpdated\nJuly 20, 2026\n•\n8\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nInside GPT: Experiments and Trade-offs\nOver the last few articles, I built the transformer block, assembled the complete GPT architecture, loaded pretrained GPT-2 weights, fine-tuned the model for classification and instruction tasks, and even used another LLM as a judge.\nAt this point, the model was beginning to feel less like a mysterious black box and more like a toy I had assembled myself—which naturally led to the most responsible next step: start changing the parts and see what breaks.\nThis article begins a new series:\nInside GPT: Experiments and Trade-offs\n.\nWhy move into experiments?\nBuilding the model helped me understand how its pieces fit together, but it also created a new set of questions.\nOnce you understand how a toy works, it is difficult to resist opening it again, replacing a few parts, turning one dial too far, and checking whether it still runs. That curiosity is the motivation behind this next phase.\nI want to change individual training and architecture decisions, compare the results, and see whether the model behaves the way my intuition says it should. Sometimes that intuition will be right. Sometimes the experiment will expose something I had not considered.\nEither outcome is useful.\nWhat I plan to experiment with\nI am organizing this phase around six broader areas. Each area can support its own article in this series.\nLearning rate and optimization:\nExplore different learning-rate ranges, how learning rate interacts with batch size, whether it should decay during training, and whether every layer should use the same rate.\nContext length:\nCompare shorter and longer input windows and examine the trade-offs between available context, training speed, memory usage, and model quality.\nModel size:\nChange the embedding dimensions, number of transformer blocks, and overall parameter count to see how additional capacity affects learning and compute requirements.\nArchitecture choices:\nExperiment with attention heads, normalization placement, activation choices, and positional embeddings while keeping the rest of the model stable.\nFine-tuning strategy:\nCompare which parts of the model are updated during fine-tuning, including freezing early layers, training only later layers, or updating the full model.\nGeneration settings:\nExplore temperature, top-k sampling, and related decoding choices to understand how they affect repetition, coherence, predictability, and creativity.\nThe purpose is not to produce a leaderboard or claim that one configuration is universally correct.\nI want to understand how these decisions interact, why a particular choice works in one setup, and what trade-offs it creates elsewhere.\nThe experimentation framework\nThe GPT-2 Small configuration I used earlier has roughly 124 million trainable parameters in my implementation.\nThat is manageable for loading pretrained weights and running inference. Repeatedly training it across dozens of configurations, however, would be slow on my local machine.\nFor this experiment phase, I created a much smaller GPT-style model with approximately 13 million parameters (\nGithub\n):\nBASE_MODEL_CONFIG = {\n    \"vocab_size\": 50_257,\n    \"context_length\": 128,\n    \"emb_dim\": 128,\n    \"n_heads\": 4,\n    \"n_layers\": 4,\n    \"drop_rate\": 0.1,\n    \"qkv_bias\": False,\n}\nCompared with the GPT-2 Small configuration, I reduced the context length, embedding dimensions, number of attention heads, and number of transformer blocks.\nThis model is not intended to compete with GPT-2. It is an experimental sandbox.\nA smaller model allows me to run several controlled training jobs locally, catch implementation issues quickly, and spend more time understanding the results instead of waiting for one large run to finish.\nThe exact findings may not transfer directly to a much larger model. However, the smaller model can still help reveal relationships, unexpected behavior, and questions worth investigating further.\nInsert notebook image:\nCompact model configuration and parameter count.\nSeparating the configuration\nI also wanted the experiment code to be reusable.\nWithout a common framework, every experiment could slowly develop its own training loop, data split, evaluation logic, and result format. Over time, those small differences would make the runs difficult to compare.\nThe framework separates the settings into three groups:\nBASE_MODEL_CONFIG\nBASE_TRAINING_CONFIG\nBASE_DATA_CONFIG\nThe model configuration defines the architecture.\nThe training configuration contains settings such as the batch size, learning rate, number of epochs, evaluation frequency, and random seed:\nBASE_TRAINING_CONFIG = {\n    \"batch_size\": 2,\n    \"learning_rate\": 3e-4,\n    \"weight_decay\": 0.1,\n    \"num_epochs\": 1,\n    \"eval_freq\": 5,\n    \"eval_iter\": 2,\n    \"train_ratio\": 0.90,\n    \"seed\": 123,\n}\nThe data configuration identifies the dataset and its tokenization settings:\nBASE_DATA_CONFIG = {\n    \"file_name\": \"wikitext-2-500k.txt\",\n    \"allowed_special\": {\"<|endoftext|>\"},\n}\nSeparating these settings makes it easier to change one variable without unintentionally modifying something else.\nFor example, a learning-rate experiment should change the learning rate while keeping the model architecture, dataset, batch size, random seed, and evaluation approach stable.\nIt is similar to testing two recipes by changing only the amount of salt. If I also change the cooking time, temperature, and ingredients, I may get a different result—but I will not know which change caused it.\nDescribing each experiment\nEach run is represented using an experiment specification:\n@dataclass\nclass ExperimentSpec:\n    name: str\n    question: str\n    hypothesis: str\n    independent_variable: str\n    variable_value: Any\n    controlled_variables: list[str]\n\n    model_overrides: dict[str, Any] = field(default_factory=dict)\n    training_overrides: dict[str, Any] = field(default_factory=dict)\n    data_overrides: dict[str, Any] = field(default_factory=dict)\nAn experiment specification records:\nthe question I am trying to answer;\nthe hypothesis before seeing the result;\nthe variable being changed;\nthe variables that should remain controlled;\nany changes to the model, training, or data configuration.\nFor example, the first baseline run asks whether the framework itself can complete a training run:\nBASELINE_SPEC = ExperimentSpec(\n    name=\"baseline\",\n    question=\"Can the reusable pipeline complete a training run?\",\n    hypothesis=\"The compact model should train and reduce its loss.\",\n    independent_variable=\"none\",\n    variable_value=\"baseline\",\n    controlled_variables=[\n        \"dataset\",\n        \"tokenizer\",\n        \"random seed\",\n        \"model architecture\",\n        \"optimizer\",\n    ],\n)\nWriting the question and hypothesis before running the model is important.\nOtherwise, it becomes very easy to look at the final graph and create an explanation that happens to fit the result.\nRunning and saving the experiment\nThe reusable runner handles the common work:\nmodel, history, result = run_experiment(\n    spec=BASELINE_SPEC,\n    save_checkpoint=False,\n)\nFor every run, the framework records information such as:\nmodel and training configuration;\ninitial and final loss;\nbest validation loss;\nnumber of tokens processed;\noptimizer steps;\nparameter count;\nruntime;\ntraining history.\nThe detailed result is stored in a JSON file, while a shared CSV file makes it easier to compare multiple runs in a table.\nEach run also receives a unique identifier. Running a new experiment therefore adds another record instead of overwriting the previous result. The complete implementation is available in the experiment notebook in my GitHub repository:\nView the experiment notebook on GitHub\nThe framework does not make the experiments automatically correct. I still need to decide what should remain fixed, what metric matters, and whether the comparison is actually fair.\nIt simply gives every experiment the same basic foundation.\nA trial experiment\nBefore designing more ambitious experiments, I wanted to make sure the pipeline itself worked.\nCould I define several configurations, train each one, save the results, and compare them without manually rebuilding the notebook every time?\nTo test that, I ran a simple learning-rate sweep:\nLEARNING_RATES = [\n    1e-5,\n    1e-4,\n    3e-4,\n    1e-3,\n    1e-2,\n    1e-1,\n]\nFor each value, the framework created a new experiment specification and changed only the learning rate:\ntraining_overrides={\n    \"learning_rate\": learning_rate,\n    \"num_epochs\": 2,\n    \"seed\": 123,\n}\nEverything else remained fixed:\nbatch size;\nmodel configuration;\ndataset and train-validation split;\nrandom seed;\nweight decay;\noptimizer;\nnumber of epochs;\nevaluation method.\nThe point of this sweep was not to conduct a complete investigation into learning rates. It was a trial run for the experimentation pipeline.\nThe different learning rates produced clearly different training behavior.\nVery conservative values changed the model slowly. More aggressive values allowed the loss to fall faster, but increasing the rate too far also made training less stable.\nThat was already enough to show why a single experiment value rarely tells the complete story. A value may be too small, somewhere in a useful range, or aggressive enough to push the model beyond that range.\nMore importantly, the pipeline worked.\nI could define multiple runs, keep the controlled settings stable, capture the results consistently, and compare their validation-loss curves without rewriting the training code.\nThat gave me enough confidence to move from testing the framework to using it for an actual investigation.\nWhere this leads next\nThe trial experiment answered the pipeline question, but it immediately created several new model questions.\nHow far could I increase the learning rate before training became unstable? Would the useful range change with batch size? Should the learning rate remain fixed throughout training or gradually decrease? And does it make sense to apply the same learning rate to the embeddings, transformer blocks, and output layer?\nThose questions became the first real investigation in this series. In the next article, I will explore learning rate not as one isolated value in a configuration file, but as a decision that interacts with batch size, training progress, and different parts of the model.\nThe experiment framework is ready. It is time to start turning the dials.\n#\nartificial-intelligence\n#\nmachine-learning\n#\ngpt\n#\nexperimentation\n2\nviews",
  "metadata": {
    "title": "From Building a GPT Model to Experimenting With One",
    "description": "This article explains the smaller model, reusable experiment framework, configuration tracking, and first learning-rate sweep.",
    "keywords": null,
    "author": null,
    "og:title": "From Building a GPT Model to Experimenting With One",
    "og:description": "This article explains the smaller model, reusable experiment framework, configuration tracking, and first learning-rate sweep.",
    "og:url": "https://curious-pm.hashnode.dev/from-building-a-gpt-model-to-experimenting-with-one",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/7f83b918-0937-4ba9-ad14-fbe04c789b1f.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "From Building a GPT Model to Experimenting With One",
    "twitter:description": "This article explains the smaller model, reusable experiment framework, configuration tracking, and first learning-rate sweep.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/7f83b918-0937-4ba9-ad14-fbe04c789b1f.png",
    "article:published_time": "2026-07-17T21:08:52.904Z",
    "article:modified_time": "2026-07-20T04:46:43.113Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/76ec8c281144/code/body-000.txt",
      "chars": 179,
      "language": "text"
    },
    {
      "file": "attachments/76ec8c281144/code/body-001.txt",
      "chars": 56,
      "language": "text"
    },
    {
      "file": "attachments/76ec8c281144/code/body-002.txt",
      "chars": 203,
      "language": "text"
    },
    {
      "file": "attachments/76ec8c281144/code/body-003.txt",
      "chars": 105,
      "language": "text"
    },
    {
      "file": "attachments/76ec8c281144/code/body-004.txt",
      "chars": 376,
      "language": "text"
    },
    {
      "file": "attachments/76ec8c281144/code/body-005.txt",
      "chars": 404,
      "language": "text"
    },
    {
      "file": "attachments/76ec8c281144/code/body-006.txt",
      "chars": 94,
      "language": "text"
    },
    {
      "file": "attachments/76ec8c281144/code/body-007.txt",
      "chars": 81,
      "language": "text"
    },
    {
      "file": "attachments/76ec8c281144/code/body-008.txt",
      "chars": 97,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/7f83b918-0937-4ba9-ad14-fbe04c789b1f.png",
      "bytes": 1410364,
      "reused": true,
      "local_path": "attachments/76ec8c281144/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/7f83b918-0937-4ba9-ad14-fbe04c789b1f.png",
      "alt": "From Building a GPT Model to Experimenting With One",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/320655ac-cdd3-49ab-b396-d4ebbcd60afc.png",
      "bytes": 18483,
      "reused": true,
      "local_path": "attachments/76ec8c281144/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/320655ac-cdd3-49ab-b396-d4ebbcd60afc.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/c4ac7081-ea4c-4379-8035-6541d9dc31f1.png",
      "bytes": 135895,
      "reused": true,
      "local_path": "attachments/76ec8c281144/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/c4ac7081-ea4c-4379-8035-6541d9dc31f1.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/from-building-a-gpt-model-to-experimenting-with-one.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/inside-gpt-experiments-and-trade-offs",
      "label": "Part of series Inside GPT: Experiments and Trade-offs",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work/tree/main/notebooks",
      "label": "Github",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work/blob/main/notebooks/05_experiment_template.ipynb",
      "label": "View the experiment notebook on GitHub",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/artificial-intelligence",
      "label": "# artificial-intelligence",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/machine-learning",
      "label": "# machine-learning",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/gpt",
      "label": "# gpt",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/experimentation",
      "label": "# experimentation",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:12",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:55Z",
  "source_url": "https://curious-pm.hashnode.dev/from-building-a-gpt-model-to-experimenting-with-one",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 10823,
  "expected_body_images": 3,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/76ec8c281144/pages/source.html",
  "local_html": "attachments/76ec8c281144/pages/article.html",
  "body_base_path": "attachments/76ec8c281144/pages/",
  "body_markdown_path": "attachments/76ec8c281144/pages/article.md"
}