{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "Learning Rate Is More Than Just a Number",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/learning-rate-is-more-than-just-a-number",
  "publish_date": "2026-07-17",
  "corpus_stage": "",
  "notes": "",
  "id": "096dab38f6f8",
  "collected_at": "2026-10-02T14:11:44",
  "title": "Learning Rate Is More Than Just a Number | GPT Experiments",
  "body_markdown": "Learning Rate Is More Than Just a Number\nWhat batch size, decay schedules, and layer wise updates taught me about training a small GPT model\nUpdated\nJuly 20, 2026\n•\n11\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nInside GPT: Experiments and Trade-offs\nIn the previous article, I tested a reusable experimentation framework with a small learning-rate sweep. Within the range I tried, higher learning rates produced lower validation loss—but the best result sat at the edge of the range.\nThat is usually the experiment’s polite way of saying, “Nice try, but you have not explored enough yet.”\nLearning rate looks like a single configuration value: smaller values make cautious updates, while larger values make more aggressive ones. But recent research shows that its effect depends on much more—batch size, model size, training duration, decay strategy, optimizer choice, fine-tuning goals, and even the part of the model being updated.\nI could not reproduce billion-parameter experiments on my local machine, so I used my smaller GPT model to explore three questions:\nDoes the useful learning-rate range change with batch size?\nShould the learning rate remain constant or decay during training?\nShould every part of the model use the same learning rate?\nThese experiments are not intended to validate the papers that inspired them. The goal is to use their questions to build a more practical intuition for what learning rate is actually doing.\nExperiment 1: Learning rate across batch sizes\nImagine a basketball coach helping a player correct a shot.\nThe learning rate is similar to the size of the correction after each attempt. A small correction might be, “Move your elbow slightly inward.” A large correction might be, “Change your entire shooting motion.”\nBatch size determines how many shots the coach watches before providing feedback. With a batch size of two, the coach watches two shots and then suggests a change. With a batch size of eight, the coach watches eight shots before changing anything.\nThe larger batch provides a more stable view of the player’s shooting pattern, but the feedback arrives less frequently.\nThat creates an immediate question: should the size of each correction change when the model receives fewer updates?\nSetting up a fair comparison\nI tested three batch sizes:\nBATCH_SIZES = [2, 4, 8]\nI kept the model architecture, dataset, random seed, optimizer, and evaluation approach constant.\nI also gave every run approximately the same token budget. Each model processed roughly 100,000 training tokens, regardless of batch size.\nThis sounds fair, but it creates an important difference:\nBatch size\nApproximate optimizer steps\n2\n391\n4\n196\n8\n98\nThe batch-size-two model updated its parameters approximately four times as often as the batch-size-eight model. That difference became important when interpreting the results.\nExpanding the learning-rate range\nI explored the learning rate in three rounds.\nThe first round used relatively cautious values. The second moved higher, and the third deliberately entered the unstable region.\nLEARNING_RATES = [\n    1e-4, 3e-4, 1e-3,\n    5e-3, 1e-2,\n    5e-2, 1e-1, 5e-1,\n]\nThe full experiment code creates one run for every batch-size and learning-rate combination:\nexperiment_grid = [\n    (batch_size, learning_rate)\n    for batch_size in BATCH_SIZES\n    for learning_rate in LEARNING_RATES\n]\nHere are the final validation losses:\nThe overall pattern was surprisingly clean:\nToo cautious → useful range → diminishing gains → instability\nIncreasing the learning rate helped at first. The improvement continued through\n1e-3\nand\n5e-3\n, before beginning to flatten around\n1e-2\n.\nAfter that, the pattern reversed. At\n5e-2\n, validation loss became worse. At\n1e-1\n, training showed signs of instability. At\n5e-1\n, the smaller-batch runs effectively broke down.\nThe best value I tested was\n1e-2\n, but I would not describe it as a universally optimal learning rate. It was simply the strongest value for this model, dataset, optimizer, and training budget. Change those conditions, and the answer may also change.\nWhy did the smaller batch perform better?\nAcross the useful learning-rate range, batch size two frequently produced the lowest validation loss. Under the fixed token budget, the batch-size-two model received 391 opportunities to adjust its parameters, while the batch-size-eight model received only 98.\nThis does not prove that smaller batches are always better. It tells us that fixed tokens and fixed optimizer steps answer different questions.\nA fixed-token experiment asks:\nWhich configuration learns more effectively from the same amount of data?\nA fixed-step experiment asks:\nWhich configuration learns more effectively after the same number of parameter updates?\nUnder fixed steps, larger batches would process more total tokens. Under fixed tokens, smaller batches receive more updates.\nNeither comparison is automatically more correct. The right comparison depends on which resource we are trying to hold constant.\nExperiment 2: Learning rate across training time\nThe first experiment used a fixed learning rate, but training does not need equally large updates from beginning to end.\nEarly on, the model starts with mostly random parameters, so larger corrections can help. Later, smaller updates may help it settle instead of overshooting.\nIt is similar to coaching a basketball shot: first you fix the overall form, then you refine the wrist, release angle, and foot position.\nThat is the idea behind a learning-rate schedule. I compared three approaches:\nConstant:\nAfter a short warmup, the learning rate stays at its peak value. It is simple, but updates may remain too aggressive near the end.\nCosine decay:\nThe learning rate decreases smoothly after warmup, moving gradually from larger updates to smaller refinements.\nWarmup–Stable–Decay:\nThe rate warms up, stays stable through most of training, and decays only near the end.\nAll three runs used the same model initialization, batch size, peak learning rate, and 200,000-token budget:\nSCHEDULES = [\"constant\", \"cosine\", \"wsd\"]\n\nBATCH_SIZE = 2\nPEAK_LR = 5e-3\nTOKEN_BUDGET = 200_000\nThe only variable was the schedule.'\nBoth decay schedules achieved slightly lower validation loss than the constant schedule, supporting the idea that smaller late-stage updates can help the model settle.\nBut the differences were modest.\nThe bigger lesson was:\nChoosing a good peak learning rate mattered more than choosing between the decay schedules.\nThere is also an important caveat. A schedule that produces the lowest pretraining loss may not produce the model that fine-tunes best later.\nMy experiment measured pretraining validation loss only, so it cannot answer that question. The “best” schedule therefore depends partly on what we plan to do with the model next.\nExperiment 3: Learning rate across the model\nSo far, every part of the model used the same learning rate.\nBut a transformer contains embeddings, multiple transformer blocks, normalization layers, and an output projection. These components perform different jobs and may not respond equally to the same update size.\nIt is like giving the goalkeeper, striker, and defender exactly the same coaching instruction. Simple, but not always appropriate.\nSupporting different learning rates across the model required changes to how the optimizer grouped and updated parameters. I will skip most of that implementation detail here, but the full code is available in the notebook:\nView the full experiment notebook on GitHub\nI tested three strategies.\nUniform learning rate\nEvery parameter group used the same rate.\nThis is simple and common, but equal learning rates do not guarantee equal parameter movement.\nDepth-increasing learning rate\nEarlier layers received smaller rates, while deeper transformer blocks received progressively larger ones.\nI treated this as a practical heuristic, not a rule that deeper layers should always learn faster.\nUpdate-balanced proxy\nInstead of assigning rates based on depth, I first ran a short 20-step calibration using a uniform learning rate.\nParameter groups that changed very little received a larger multiplier, while groups that changed aggressively received a smaller one. I capped the multipliers between\n0.5\nand\n2.0\nto avoid extreme adjustments.\nHere is how the three strategies distributed the learning rate:\nParameter group\nUniform\nDepth-increasing\nUpdate-balanced\nEmbeddings\n1.000\n0.500\n2.000\nBlock 0\n1.000\n0.600\n0.990\nBlock 1\n1.000\n0.800\n1.003\nBlock 2\n1.000\n1.000\n0.999\nBlock 3\n1.000\n1.200\n1.001\nOutput\n1.000\n1.000\n0.500\nInterestingly, the calibration did not recommend meaningfully different rates for the four transformer blocks.\nThe biggest corrections were elsewhere:\nincrease the embedding learning rate;\nreduce the output learning rate.\nWhich strategy performed best?\nThe update-balanced proxy performed best.\nUnlike the depth heuristic, it focused its largest adjustments on the embeddings and output projection—the groups showing the biggest movement imbalance during calibration.\nIn this run, assigning learning rates based only on layer depth did not improve validation loss. The data-driven proxy performed better by responding to observed parameter movement instead of assuming that deeper layers always need larger updates.\nThat does not establish the proxy as a universally better strategy. It simply made the strongest recommendation for this particular run.\nWhat actually changed inside the model?\nTo compare the strategies, I measured each parameter group’s cumulative relative change:\n||final parameters - initial parameters||\n-----------------------------------------\n       ||initial parameters||\nA value of\n1.0\nmeans the group’s overall change norm was equal to its initial parameter norm. It does not mean that every individual parameter changed by 100%.\nParameter group\nUniform\nDepth-increasing\nUpdate-balanced\nEmbeddings\n0.332\n0.182\n0.556\nBlock 0\n1.859\n1.290\n1.864\nBlock 1\n1.170\n1.206\n1.212\nBlock 2\n1.076\n1.144\n1.182\nBlock 3\n1.039\n1.156\n1.200\nOutput\n5.255\n5.309\n3.763\nThe depth-increasing strategy made movement across the transformer blocks more balanced, but that did not translate into lower validation loss.\nThe update-balanced proxy did something different. It increased movement in the embeddings and reduced the unusually large movement in the output group.\nThis was a useful reminder that learning rate is only one factor affecting parameter movement. Gradient magnitude, parameter scale, AdamW’s adaptive updates, and weight decay also matter.\nMore balanced movement may help explain what is happening inside the model, but balance itself is not the objective. The optimizer is still trying to reduce loss—not make every layer change by a visually pleasing amount.\nProduct and experimentation lessons\nAlthough these were model-training experiments, five lessons felt familiar from product experimentation:\nStart broad before optimizing narrowly.\nThe difference between\n1e-4\nand\n1e-2\nwas much larger than the difference between cosine and WSD. First find the region that works, then refine within it.\nDefine what a fair comparison means.\nGiving every batch size the same token budget also gave them different numbers of optimizer steps. A controlled experiment should match the question being asked, not simply make every number equal.\nInspect the mechanism, not only the headline metric.\nValidation loss showed which run performed better, while parameter movement explained how different parts of the model responded. Product metrics can similarly hide important differences in user behavior.\nDo not overstate small differences.\nWSD performed best in this run, but its advantage over cosine was tiny. One experiment supports a local observation, not a universal rule.\nUse part of the budget to reduce uncertainty.\nShort sweeps, proxy models, and even deliberately unstable runs can prevent expensive mistakes later. Sometimes the cheapest experiment is the one that tells us which costly experiment not to run.\nReferences\nhttps://github.com/amanjain454/llms-from-scratch-work/blob/main/notebooks/05\\_experiment\\_template.ipynb\nBenchmarking Optimizers for Large Language Model Pretraining\n— Andrei Semenov, Matteo Pagliardini, and Martin Jaggi.\nScaling and Transferability of Annealing Strategies in Large Language Model Training\n— Siqi Wang et al.\nAnytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging\n— Alexandru Meterez et al.\nPre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning\n— Kazuki Yano et al.\nScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models\n— Aaron Defazio.\nOne LR Doesn’t Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs\n— Di He et al.\nOn the Nonlinearity of Learning Rate Scaling for LLM Training\n— Zaiwen Yang et al.\n#\ndeeplearning\n#\ngpt\n#\nmachine-learning\n18\nviews",
  "metadata": {
    "title": "Learning Rate Is More Than Just a Number | GPT Experiments",
    "description": "I tested how batch size, decay schedules, and layerwise updates affect learning rate behavior while pretraining a small GPT model.",
    "keywords": null,
    "author": null,
    "og:title": "Learning Rate Is More Than Just a Number | GPT Experiments",
    "og:description": "I tested how batch size, decay schedules, and layerwise updates affect learning rate behavior while pretraining a small GPT model.",
    "og:url": "https://curious-pm.hashnode.dev/learning-rate-is-more-than-just-a-number",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/6b37d9c5-ecbc-47ad-ab56-5090b4a31f22.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Learning Rate Is More Than Just a Number | GPT Experiments",
    "twitter:description": "I tested how batch size, decay schedules, and layerwise updates affect learning rate behavior while pretraining a small GPT model.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/6b37d9c5-ecbc-47ad-ab56-5090b4a31f22.png",
    "article:published_time": "2026-07-17T22:40:31.310Z",
    "article:modified_time": "2026-07-20T04:45:42.595Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/096dab38f6f8/code/body-000.txt",
      "chars": 24,
      "language": "text"
    },
    {
      "file": "attachments/096dab38f6f8/code/body-001.txt",
      "chars": 81,
      "language": "text"
    },
    {
      "file": "attachments/096dab38f6f8/code/body-002.txt",
      "chars": 128,
      "language": "text"
    },
    {
      "file": "attachments/096dab38f6f8/code/body-003.txt",
      "chars": 96,
      "language": "text"
    },
    {
      "file": "attachments/096dab38f6f8/code/body-004.txt",
      "chars": 114,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/6b37d9c5-ecbc-47ad-ab56-5090b4a31f22.png",
      "bytes": 1294335,
      "reused": true,
      "local_path": "attachments/096dab38f6f8/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/6b37d9c5-ecbc-47ad-ab56-5090b4a31f22.png",
      "alt": "Learning Rate Is More Than Just a Number",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/3c889813-c02e-4f55-8e92-15cf79bc9108.png",
      "bytes": 18791,
      "reused": true,
      "local_path": "attachments/096dab38f6f8/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/3c889813-c02e-4f55-8e92-15cf79bc9108.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/91f4d011-c69e-4b63-bbf9-bf2cd960a1dd.png",
      "bytes": 59869,
      "reused": true,
      "local_path": "attachments/096dab38f6f8/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/91f4d011-c69e-4b63-bbf9-bf2cd960a1dd.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/430c4742-bd55-4205-8242-98dce7a33ae6.png",
      "bytes": 40228,
      "reused": true,
      "local_path": "attachments/096dab38f6f8/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/430c4742-bd55-4205-8242-98dce7a33ae6.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/516f3fb4-8b55-4a5a-8e04-5106c258f31b.png",
      "bytes": 79400,
      "reused": true,
      "local_path": "attachments/096dab38f6f8/images/004.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/516f3fb4-8b55-4a5a-8e04-5106c258f31b.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/4340432d-19e6-4f57-9b6f-477bcc8155e0.png",
      "bytes": 90426,
      "reused": true,
      "local_path": "attachments/096dab38f6f8/images/005.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/4340432d-19e6-4f57-9b6f-477bcc8155e0.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/learning-rate-is-more-than-just-a-number.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/inside-gpt-experiments-and-trade-offs",
      "label": "Part of series Inside GPT: Experiments and Trade-offs",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work/blob/main/notebooks/05_experiment_template.ipynb",
      "label": "View the full experiment notebook on GitHub",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work/blob/main/notebooks/05%5C_experiment%5C_template.ipynb",
      "label": "https://github.com/amanjain454/llms-from-scratch-work/blob/main/notebooks/05\\_experiment\\_template.ipynb",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/deeplearning",
      "label": "# deeplearning",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/gpt",
      "label": "# gpt",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/machine-learning",
      "label": "# machine-learning",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:09:09",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:55Z",
  "source_url": "https://curious-pm.hashnode.dev/learning-rate-is-more-than-just-a-number",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 12979,
  "expected_body_images": 6,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/096dab38f6f8/pages/source.html",
  "local_html": "attachments/096dab38f6f8/pages/article.html",
  "body_base_path": "attachments/096dab38f6f8/pages/",
  "body_markdown_path": "attachments/096dab38f6f8/pages/article.md"
}