{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Aman Jain",
  "title_listed": "Temperature, Top-k, and Top-p: How Decoding Changes What a GPT Model Says\n",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://curious-pm.hashnode.dev/temperature-top-k-and-top-p-how-decoding-changes-what-a-gpt-model-says",
  "publish_date": "2026-07-22",
  "corpus_stage": "",
  "notes": "",
  "id": "08763d32b268",
  "collected_at": "2026-10-02T14:11:43",
  "title": "temperature-top-k-top-p-decoding-strategies",
  "body_markdown": "Temperature, Top-k, and Top-p: How Decoding Changes What a GPT Model Says\nExperiments with temperature, greedy decoding, top-k, and top-p to understand the trade-offs between predictability, repetition, coherence, and creativity.\nUpdated\nJuly 22, 2026\n•\n10\nmin read\n•\nView as Markdown\nA\nAman Jain\nProduct manager learning to build AI from the inside out — from LLM architecture to product applications.\nPart of series\nInside GPT: Experiments and Trade-offs\nTL;DR\nA GPT model does not directly choose its next word. More precisely, it assigns probabilities to thousands of possible\nnext tokens\n.\nDecoding is the step that converts those probabilities into an actual token.\nTemperature is one of the controls used during decoding. A lower temperature concentrates probability around the model’s favorite choices—the output becomes more “frozen.” A higher temperature spreads probability across more alternatives, increasing uncertainty and potential variety.\nHowever, temperature does not always change the generated text. If we use greedy decoding, which always selects the highest-scoring token, every positive temperature produces the same choice. Temperature matters when the decoding strategy actually samples from the probability distribution.\nI explored this through two experiments:\nTemperature and test behavior:\nDoes temperature affect predictable and diverse datasets differently? I measured test loss, sampled next-token accuracy, and entropy.\nGeneration strategies:\nHow do greedy decoding, unrestricted sampling, top-k, and top-p change repetition, diversity, coherence, and creativity?\nIn the first experiment, both low- and high-diversity test sets achieved their lowest test loss at\nT = 1.0\n. Lower temperatures made sampling more predictable, but they also made the model too confident when it was wrong. Higher temperatures increased uncertainty while sampled accuracy fell sharply.\nIn the second experiment, greedy decoding was the most predictable but also the most repetitive. Unrestricted sampling produced the widest variation but was less reliable. Top-k and top-p offered more controlled exploration, with top-p showing the strongest overall balance in this setup.\nThe code is available in my\nGitHub repository\n.\nFrom Probabilities to Text\nSuppose the model is continuing:\nThe capital of France is\nIt may assign the highest probability to\nParis\n, while still keeping other tokens in consideration:\nPossible token\nProbability\nParis\n90%\nin\n5%\ncity\n5%\nA decoding strategy decides what happens next:\nGreedy:\nalways choose the highest-probability token.\nUnrestricted sampling:\nrandomly sample from the complete distribution.\nTop-k:\nremove all but the\nk\nmost likely tokens, then sample.\nTop-p:\nkeep the smallest set of tokens whose combined probability crosses a chosen threshold, then sample.\nSampling is random, but it is not uniformly random. A token with 30% probability is still much more likely to be selected than one with 2%.\nThese choices are repeated for every generated token. A small difference at one step can therefore send the continuation in a completely different direction.\nWhat Temperature Actually Changes\nTemperature rescales the model’s logits before converting them into probabilities:\nscaled_logits = logits / temperature\nConsider three possible tokens with logits of\n2\n,\n1\n, and\n0\n.\nTemperature\nParis\nin\ncity\n0.5\n86.7%\n11.7%\n1.6%\n1.0\n66.5%\n24.5%\n9.0%\n2.0\n50.6%\n30.7%\n18.6%\nAt\nT = 0.5\n,\nParis\ndominates. At\nT = 2.0\n, the alternatives receive much more probability.\nHowever,\nParis\nremains the most likely token at every temperature.\nThat leads to an important distinction:\nTemperature does not change positive-temperature greedy decoding because the ranking of tokens remains the same.\nIt does affect probability-based strategies:\nWith unrestricted sampling, it changes each token’s chance of being selected.\nWith top-k, it changes probabilities within the retained candidate set.\nWith top-p, it can also change which tokens enter the candidate set because cumulative probabilities change.\nExperiment 6.1 — Does Different Data Prefer Different Temperatures?\nFor this experiment, I trained a fresh small GPT model on a balanced mixture of two datasets:\nLow-diversity text:\nnarrower topics, repeated patterns, and fewer likely continuations.\nHigh-diversity text:\nbroader topics and vocabulary, with more plausible next-token alternatives.\nMy intuition was that predictable text might benefit from a lower temperature. If one token is usually clearly correct, sharpening the distribution could help.\nFor diverse text, several next tokens may be reasonable. I expected it to prefer a temperature closer to\n1.0\n, or possibly slightly above it, so that probability remained distributed across alternatives.\nI tested:\ntemperatures = [0.3, 0.5, 0.7, 0.9, 1.0, 1.2, 1.5, 2.0]\nWhat I Measured\nTest loss\nTest loss measures how well the complete probability distribution matches the actual next tokens.\nLower is better.\nUnlike accuracy, it also considers confidence. A model that is confidently wrong receives a much larger penalty than one that leaves some probability on the correct token.\nSampled next-token accuracy\nI sampled a token from the temperature-adjusted distribution and measured how often it matched the actual next token.\nHigher is better when predictability is the objective.\nHowever, this metric naturally favors conservative distributions. It does not tell us whether alternative generations are interesting or coherent.\nPredictive entropy\nEntropy measures how spread out the probability distribution is.\nLow entropy:\nprobability is concentrated on a few tokens.\nHigh entropy:\nprobability is spread across many alternatives.\nNeither is universally better. Low entropy is useful when predictable output matters. Higher entropy enables exploration, but too much can make the output feel random.\nArgmax accuracy\nArgmax accuracy measures how often the model’s highest-scoring token is correct.\nHigher is better.\nI used it as a control rather than a primary metric. Temperature changes confidence, but not token ranking, so argmax accuracy should remain constant across positive temperatures.\nTest Loss Was Lowest at Temperature 1.0\nBoth datasets produced almost identical U-shaped loss curves.\nAt very low temperatures, the model became too confident. When its preferred token was wrong, the correct token received almost no probability, causing test loss to rise sharply.\nAt high temperatures, probability was spread too widely across unlikely tokens.\nThe original distribution at\nT = 1.0\nprovided the best balance for both datasets.\nMy hypothesis that low-diversity text would prefer a lower temperature was therefore not supported. The two datasets behaved much more similarly than I expected.\nPredictability Fell as Exploration Increased\nAs temperature increased, entropy rose and sampled accuracy fell.\nLower temperature made sampling more predictable. Higher temperature created more possible paths.\nExperiment 6.2 — Which Decoding Strategy Works When?\nFor the generation experiment, I used official pretrained GPT-2 124M weights.\nThe freshly trained model was sufficient for probability-level analysis, but it had not been trained long enough to produce meaningful text. Using pretrained weights allowed me to compare decoding strategies on more realistic continuations.\nI compared four strategies:\nStrategy\nHow it works\nUseful when\nGreedy\nAlways selects the most likely token\nConsistency and deterministic output matter\nTemperature 1.0\nSamples from the complete distribution\nMaximum variety matters more than reliability\nTop-k 10\nSamples from the ten most likely tokens\nControlled, relatively safe variation\nTop-p 0.8\nSamples from the smallest set covering 80% probability\nThe candidate set should adapt to model confidence\nWhy Three Prompt Types?\nI tested three kinds of prompts because decoding may behave differently depending on the task.\nStructured prompts\nhave a relatively narrow expected continuation. I expected greedy or top-k to work best.\nNarrative prompts\nrequire coherence but still benefit from variation. I expected top-k or top-p to perform well.\nCreative prompts\nreward novelty. I expected unrestricted sampling or top-p to gain an advantage.\nMeasuring Repetition and Diversity\nA\nbigram\nis a pair of consecutive tokens.\nFor example:\nthe cat sat on the cat\ncontains bigrams such as\nthe cat\n,\ncat sat\n, and\nsat on\n. Because\nthe cat\nappears twice, the text contains a repeated bigram.\nThe\nrepeated-bigram rate\nmeasures how much of the output reuses token pairs.\nLower is better\nwhen we want to avoid repetitive text.\nI also measured\nDistinct-2\n, the percentage of unique bigrams.\nHigher generally means more variety\n, although extremely high diversity can also appear in incoherent text.\nAverage token log probability measured how probable the selected tokens were under the model. Values closer to zero indicate more likely choices.\nFinally, I used a local Phi-3 model as an LLM judge. It scored coherence, relevance, fluency, creativity, and repetition. I treated these scores as directional because the lightweight judge occasionally disagreed with obvious repetition in the text.\nGreedy Was Probable but Repetitive\nGreedy chose the most probable tokens, but repetition was high:\nPrompt type\nRepeated bigrams\nStructured\n50.4%\nNarrative\n36.8%\nCreative\n57.3%\nOne creative continuation started reasonably but quickly became repetitive:\n“The moon is a very small object, about the size of a small car. It is about the size of a small house. It is about the size of a small town…”\nThe text was fluent, but it followed a narrow path and repeated the same structure.\nUnrestricted Sampling Maximized Variety\nTemperature sampling at\nT = 1.0\nproduced almost entirely unique outputs, with repeated-bigram rates below 5%.\nHowever, it also selected much lower-probability tokens and was less consistent on coherence and relevance.\nFor example, a continuation about the moon disappearing jumped between a book, mirrors, researchers, and a colder moon. It contained variety, but the ideas did not connect cleanly.\nUnrestricted sampling was more useful for creative prompts than for structured ones.\nTop-k and Top-p Provided Controlled Exploration\nTop-k stayed closer to the model’s likely choices while reducing repetition substantially.\nTop-p dynamically adjusted its candidate set. When the model was confident, fewer tokens were eligible. When probability was spread across several alternatives, the candidate set became larger.\nStrategy\nAverage judge score\nTop-p 0.8\n3.04\nTop-k 10\n2.96\nGreedy\n2.87\nTemperature 1.0\n2.78\nThe differences were small, but the programmatic metrics and generated examples pointed in the same general direction: top-k and top-p offered the most useful middle ground.\n[IMAGE: REPEATED BIGRAM RATE BY STRATEGY AND PROMPT TYPE]\n[IMAGE: CREATIVITY AND JUDGE SCORES BY STRATEGY]\nWhat I Learned\nTemperature is not simply a creativity slider.\nIt changes how concentrated the model’s probabilities are, but its visible effect depends on the decoding strategy using those probabilities.\nThere was also no universal best strategy:\nGreedy\nmaximized predictability but often repeated itself.\nUnrestricted sampling\nmaximized exploration but could lose coherence.\nTop-k\nprovided conservative variation.\nTop-p\noffered the strongest overall quality-diversity balance in this setup.\nThe model defines what is possible.\nThe decoding strategy decides which possibility we actually see.\nClosing This Series—and What Comes Next\nThis article also brings the current\nInside GPT: Experiments and Trade-offs\nseries to a close. Across these experiments, I moved from building a GPT model to testing how choices such as learning rate, context length, model size, fine-tuning strategy, architecture, and decoding affect its behavior. The next series will move closer to real applications. I plan to explore more application-heavy topics such as agents, tool use, retrieval, memory, evaluation, and the practical trade-offs involved in turning a language model into a useful system.\nReferences\nThe Effect of Sampling Temperature on Problem Solving in Large Language Models\nA Thorough Examination of Decoding Methods in the Era of LLMs\nTurning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs\n#\ntemperature\n#\nllm\n#\ndeeplearning\n26\nviews",
  "metadata": {
    "title": "temperature-top-k-top-p-decoding-strategies",
    "description": "I tested temperature, greedy decoding, top-k, and top-p with GPT models to measure test loss, predictability, repetition, coherence, and creativity.",
    "keywords": null,
    "author": null,
    "og:title": "temperature-top-k-top-p-decoding-strategies",
    "og:description": "I tested temperature, greedy decoding, top-k, and top-p with GPT models to measure test loss, predictability, repetition, coherence, and creativity.",
    "og:url": "https://curious-pm.hashnode.dev/temperature-top-k-and-top-p-how-decoding-changes-what-a-gpt-model-says",
    "og:site_name": "Curious PM - Product to Parameters",
    "og:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d256b70d-f6f7-4a50-bbcd-76ea11eb54cb.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "temperature-top-k-top-p-decoding-strategies",
    "twitter:description": "I tested temperature, greedy decoding, top-k, and top-p with GPT models to measure test loss, predictability, repetition, coherence, and creativity.",
    "twitter:image": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d256b70d-f6f7-4a50-bbcd-76ea11eb54cb.png",
    "article:published_time": "2026-07-22T01:11:07.440Z",
    "article:modified_time": "2026-07-22T01:11:07.440Z",
    "article:author": "https://curious-pm.hashnode.dev/@aman454"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/08763d32b268/code/body-000.txt",
      "chars": 25,
      "language": "text"
    },
    {
      "file": "attachments/08763d32b268/code/body-001.txt",
      "chars": 37,
      "language": "text"
    },
    {
      "file": "attachments/08763d32b268/code/body-002.txt",
      "chars": 56,
      "language": "text"
    },
    {
      "file": "attachments/08763d32b268/code/body-003.txt",
      "chars": 23,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d256b70d-f6f7-4a50-bbcd-76ea11eb54cb.png",
      "bytes": 1300683,
      "reused": true,
      "local_path": "attachments/08763d32b268/images/000.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/d256b70d-f6f7-4a50-bbcd-76ea11eb54cb.png",
      "alt": "Temperature, Top-k, and Top-p: How Decoding Changes What a GPT Model Says\n",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/19c9e428-c24b-4287-8ad4-4acd083937d4.png",
      "bytes": 40611,
      "reused": true,
      "local_path": "attachments/08763d32b268/images/001.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/19c9e428-c24b-4287-8ad4-4acd083937d4.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/625061d4-6123-4545-a056-a1cc72544285.png",
      "bytes": 898793,
      "reused": true,
      "local_path": "attachments/08763d32b268/images/002.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/625061d4-6123-4545-a056-a1cc72544285.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/36ec7a63-177a-4dd2-8ba7-3179e5727d7d.png",
      "bytes": 29884,
      "reused": true,
      "local_path": "attachments/08763d32b268/images/003.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/36ec7a63-177a-4dd2-8ba7-3179e5727d7d.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/51c2682b-b48e-48e0-968b-aa5551572a2c.png",
      "bytes": 28526,
      "reused": true,
      "local_path": "attachments/08763d32b268/images/004.png",
      "src": "https://cdn.hashnode.com/uploads/covers/6a4f47d631bb27227f782535/51c2682b-b48e-48e0-968b-aa5551572a2c.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://curious-pm.hashnode.dev/temperature-top-k-and-top-p-how-decoding-changes-what-a-gpt-model-says.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@aman454",
      "label": "A",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/aman-jain-1049188/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/series/inside-gpt-experiments-and-trade-offs",
      "label": "Part of series Inside GPT: Experiments and Trade-offs",
      "type": "a"
    },
    {
      "href": "https://github.com/amanjain454/llms-from-scratch-work",
      "label": "GitHub repository",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/temperature",
      "label": "# temperature",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/llm",
      "label": "# llm",
      "type": "a"
    },
    {
      "href": "https://curious-pm.hashnode.dev/tag/deeplearning",
      "label": "# deeplearning",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:08:53",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:54Z",
  "source_url": "https://curious-pm.hashnode.dev/temperature-top-k-and-top-p-how-decoding-changes-what-a-gpt-model-says",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 12307,
  "expected_body_images": 5,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/08763d32b268/pages/source.html",
  "local_html": "attachments/08763d32b268/pages/article.html",
  "body_base_path": "attachments/08763d32b268/pages/",
  "body_markdown_path": "attachments/08763d32b268/pages/article.md"
}