{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Maxime Labonne",
  "title_listed": "Decoding Strategies in Large Language Models",
  "keywords": "解码策略,Beam Search,Nucleus Sampling",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/decoding-strategies-in-large-language-models-9733a8f70539/",
  "publish_date": "2023-06",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "bc3c38f1a02d",
  "collected_at": "2026-10-01T21:56:46",
  "title": "Decoding Strategies in Large Language Models | Towards Data Science",
  "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n  * Cookie\nBCTempID\n  * Duration\n10 minutes\n  * Description\nNo description available.\n\n\n  * Cookie\n__cf_bm\n  * Duration\n1 hour\n  * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n  * Cookie\nAWSALBCORS\n  * Duration\n7 days\n  * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n  * Cookie\n_cfuvid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n  * Cookie\nli_gc\n  * Duration\n6 months\n  * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n  * Cookie\n__hssrc\n  * Duration\nsession\n  * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n  * Cookie\n__hssc\n  * Duration\n1 hour\n  * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n  * Cookie\nwpEmojiSettingsSupports\n  * Duration\nsession\n  * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n  * Cookie\nBCSessionID\n  * Duration\n1 year 1 month 4 days\n  * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n  * Cookie\n_octo\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nlogged_in\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\n__Secure-YEC\n  * Duration\npast\n  * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n  * Cookie\n__eoi\n  * Duration\n6 months\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTGCORS\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\nlogin-status-p\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nAWSALBTG\n  * Duration\n7 days\n  * Description\nNo description available.\n\n\n  * Cookie\ncsrf_token\n  * Duration\nsession\n  * Description\nNo description available.\n\n\n  * Cookie\ntoken_v2\n  * Duration\n1 day\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nD\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nPHPSESSID\n  * Duration\nsession\n  * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n  * Cookie\nVISITOR_PRIVACY_METADATA\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n  * Cookie\ncookietest\n  * Duration\nsession\n  * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n  * Cookie\n__Host-airtable-session\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\n__Host-airtable-session.sig\n  * Duration\n1 year\n  * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n  * Cookie\nm\n  * Duration\n1 year 1 month 4 days\n  * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n  * Cookie\nBIGipServer*\n  * Duration\nsession\n  * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n  * Cookie\n__cfruid\n  * Duration\nsession\n  * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n  * Cookie\n_GRECAPTCHA\n  * Duration\n6 months\n  * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n  * Cookie\n__Secure-YNID\n  * Duration\n6 months\n  * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n  * Cookie\ncookieyes-consent\n  * Duration\n1 year\n  * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n  * Cookie\nlidc\n  * Duration\n1 day\n  * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n  * Cookie\nbrw\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nbrwConsent\n  * Duration\n5 minutes\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nWMF-Uniq\n  * Duration\n1 year\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nloom_anon_comment\n  * Duration\n1 year\n  * Description\nNo description available.\n\n\n  * Cookie\nloom_referral_video\n  * Duration\nsession\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nVISITOR_INFO1_LIVE\n  * Duration\n6 months\n  * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n  * Cookie\nyt-remote-connected-devices\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n  * Duration\nNever Expires\n  * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n  * Cookie\nyt-remote-device-id\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-name\n  * Duration\nsession\n  * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\nyt-remote-fast-check-period\n  * Duration\nsession\n  * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n  * Cookie\nyt-remote-session-app\n  * Duration\nsession\n  * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-available\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n  * Cookie\nyt-remote-cast-installed\n  * Duration\nsession\n  * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n  * Cookie\ncp_session\n  * Duration\n3 months\n  * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n  * Cookie\nloid\n  * Duration\n1 year 1 month 4 days\n  * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n  * Cookie\n__hstc\n  * Duration\n6 months\n  * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n  * Cookie\nhubspotutk\n  * Duration\n6 months\n  * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n  * Cookie\n_ga\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n  * Cookie\n_ga_*\n  * Duration\n1 year 1 month 4 days\n  * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n  * Cookie\n__Host-psifi.analyticsTrace\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n__Host-psifi.analyticsTraceV2\n  * Duration\n6 hours\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\n_gh_sess\n  * Duration\nsession\n  * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n  * Cookie\nYSC\n  * Duration\nsession\n  * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n  * Cookie\najs_anonymous_id\n  * Duration\n1 year\n  * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n  * Cookie\nvuid\n  * Duration\n1 year 1 month 4 days\n  * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n  * Cookie\nAWSALB\n  * Duration\n7 days\n  * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n  * Cookie\nacq\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nacq.sig\n  * Duration\npast\n  * Description\nDescription is currently not available.\n\n\n  * Cookie\nptc\n  * Duration\n2 years\n  * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n  * Cookie\nmuc_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n  * Cookie\nguest_id_marketing\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\nguest_id_ads\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n  * Cookie\npersonalization_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n  * Cookie\nguest_id\n  * Duration\n1 year 1 month 4 days\n  * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n  * Cookie\nbcookie\n  * Duration\n1 year\n  * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n  * Cookie\n__Secure-ROLLOUT_TOKEN\n  * Duration\n6 months\n  * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n  * Cookie\nyt.innertube::nextId\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nyt.innertube::requests\n  * Duration\nNever Expires\n  * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n  * Cookie\nsession_tracker\n  * Duration\nsession\n  * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n  * Cookie\nedgebucket\n  * Duration\nsession\n  * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n  * Cookie\ndid\n  * Duration\n1 year\n  * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\n# Decoding Strategies in Large Language Models\nA Guide to Text Generation From Beam Search to Nucleus Sampling\n[Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\nJune 4, 202315 min read\n![Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1SSYwvJCdjB4llSm1P1XzOg.jpeg)\nImage by author.\nIn the fascinating world of large language models (LLMs), much attention is given to model architectures, data processing, and optimization. However, decoding strategies like beam search, which play a crucial role in text generation, are often overlooked. In this article, we will explore how LLMs generate text by delving into the mechanics of greedy search and beam search, as well as sampling techniques with top-k and nucleus sampling.\nBy the conclusion of this article, you'll not only understand these decoding strategies thoroughly but also be familiar with how to handle important hyperparameters like temperature, num_beams, top_k, and top_p.\nThe code for this article can be found on [GitHub](https://github.com/mlabonne/llm-course/blob/main/Decoding_Strategies_in_Large_Language%C2%A0Models.ipynb) and [Google Colab](https://colab.research.google.com/drive/19CJlOS5lI29g-B3dziNn93Enez1yiHk2?usp=sharing) for reference and further exploration.\nLearn this step by step with the interactive [AI Engineer](https://roadmap.sh/ai-engineer) roadmap.\n## 📚 Background\nTo kick things off, let's start with an example. We'll feed the text \"I have a dream\" to a GPT-2 model and ask it to generate the next five tokens (words or subwords).\npython\n\n```\nfrom transformers import GPT2LMHeadModel, GPT2Tokenizer\nimport torch\n\ndevice = 'cuda' if torch.cuda.is_available() else 'cpu'\nmodel = GPT2LMHeadModel.from_pretrained('gpt2').to(device)\ntokenizer = GPT2Tokenizer.from_pretrained('gpt2')\nmodel.eval()\n\ntext = \"I have a dream\"\ninput_ids = tokenizer.encode(text, return_tensors='pt').to(device)\n\noutputs = model.generate(input_ids, max_length=len(input_ids.squeeze())+5)\ngenerated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)\nprint(f\"Generated text: {generated_text}\")\n```\n\nplaintext\n\n```\nGenerated text: I have a dream of being a doctor.\n```\n\nThe sentence \"I have a dream of being a doctor\" appears to have been generated by GPT-2. However, GPT-2 didn't _exactly_ produce this sentence.\nThere's a common misconception that LLMs like GPT-2 **directly produce text**. This isn't the case. Instead, LLMs calculate logits, which are scores assigned to every possible token in their vocabulary. To simplify, here's an illustrative breakdown of the process:\n![Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1-1v4rvYrR4RPfIK22tvetQ.png)\nImage by author.\nThe tokenizer, [Byte-Pair Encoding](https://en.wikipedia.org/wiki/Byte_pair_encoding) in this instance, translates each token in the input text into a corresponding token ID. Then, GPT-2 uses these token IDs as input and tries to predict the next most likely token. Finally, the model generates logits, which are converted into probabilities using a softmax function.\nFor example, the model assigns a probability of 17% to the token for \"of\" being the next token after \"I have a dream\". This output essentially represents a ranked list of potential next tokens in the sequence. More formally, we denote this probability as _P(of | I have a dream) = 17%_.\nAutoregressive models like GPT predict the next token in a sequence based on the preceding tokens. Consider a sequence of tokens _w = (w _₁_ , w _₂_ , ..., w _ₜ_)_. The joint probability of this sequence _P(w)_ can be broken down as:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1YxUBg9dqe4BtStEj2SkRjw.png)\nFor each token _wᵢ_ in the sequence, _P(wᵢ | w₁, w₂, ..., wᵢ₋₁)_ represents the conditional probability of _wᵢ_ given all the preceding tokens (_w₁, w₂, ..., wᵢ₋₁_). GPT-2 calculates this conditional probability for each of the 50,257 tokens in its vocabulary.\nThis leads to the question: how do we use these probabilities to generate text? This is where decoding strategies, such as greedy search and beam search, come into play.\n## 🏃 ‍♂️ Greedy Search\nGreedy search is a decoding method that takes the most probable token at each step as the next token in the sequence. To put it simply, it only retains the most likely token at each stage, discarding all other potential options. Using our example:\n  * **Step 1** : Input: \"I have a dream\" → Most likely token: \" of\"\n  * **Step 2** : Input: \"I have a dream of\" → Most likely token: \" being\"\n  * **Step 3** : Input: \"I have a dream of being\" → Most likely token: \" a\"\n  * **Step 4** : Input: \"I have a dream of being a\" → Most likely token: \" doctor\"\n  * **Step 5** : Input: \"I have a dream of being a doctor\" → Most likely token: \".\"\n\n\nWhile this approach might sound intuitive, it's important to note that the greedy search is short-sighted: it only considers the most probable token at each step without considering the overall effect on the sequence. This property makes it fast and efficient as it doesn't need to keep track of multiple sequences, but it also means that it can miss out on better sequences that might have appeared with slightly less probable next tokens.\nNext, let's illustrate the greedy search implementation using graphviz and networkx. We select the ID with the highest score, compute its log probability (we take the log to simplify calculations), and add it to the tree. We'll repeat this process for five tokens.\npython\n\n```\nimport matplotlib.pyplot as plt\nimport networkx as nx\nimport numpy as np\nimport time\n\ndef get_log_prob(logits, token_id):\n    # Compute the softmax of the logits\n    probabilities = torch.nn.functional.softmax(logits, dim=-1)\n    log_probabilities = torch.log(probabilities)\n\n    # Get the log probability of the token\n    token_log_probability = log_probabilities[token_id].item()\n    return token_log_probability\n\ndef greedy_search(input_ids, node, length=5):\n    if length == 0:\n        return input_ids\n\n    outputs = model(input_ids)\n    predictions = outputs.logits\n\n    # Get the predicted next sub-word (here we use top-k search)\n    logits = predictions[0, -1, :]\n    token_id = torch.argmax(logits).unsqueeze(0)\n\n    # Compute the score of the predicted token\n    token_score = get_log_prob(logits, token_id)\n\n    # Add the predicted token to the list of input ids\n    new_input_ids = torch.cat([input_ids, token_id.unsqueeze(0)], dim=-1)\n\n    # Add node and edge to graph\n    next_token = tokenizer.decode(token_id, skip_special_tokens=True)\n    current_node = list(graph.successors(node))[0]\n    graph.nodes[current_node]['tokenscore'] = np.exp(token_score) * 100\n    graph.nodes[current_node]['token'] = next_token + f\"_{length}\"\n\n    # Recursive call\n    input_ids = greedy_search(new_input_ids, current_node, length-1)\n\n    return input_ids\n\n# Parameters\nlength = 5\nbeams = 1\n\n# Create a balanced tree with height 'length'\ngraph = nx.balanced_tree(1, length, create_using=nx.DiGraph())\n\n# Add 'tokenscore', 'cumscore', and 'token' attributes to each node\nfor node in graph.nodes:\n    graph.nodes[node]['tokenscore'] = 100\n    graph.nodes[node]['token'] = text\n\n# Start generating text\noutput_ids = greedy_search(input_ids, 0, length=length)\noutput = tokenizer.decode(output_ids.squeeze().tolist(), skip_special_tokens=True)\nprint(f\"Generated text: {output}\")\n```\n\nplaintext\n\n```\nGenerated text: I have a dream of being a doctor.\n```\n\nOur greedy search generates the same text as the one from the transformers library: \"I have a dream of being a doctor.\" Let's visualize the tree we created.\npython\n\n```\nimport matplotlib.pyplot as plt\nimport networkx as nx\nimport matplotlib.colors as mcolors\nfrom matplotlib.colors import LinearSegmentedColormap\n\ndef plot_graph(graph, length, beams, score):\n    fig, ax = plt.subplots(figsize=(3+1.2*beams**length, max(5, 2+length)), dpi=300, facecolor='white')\n\n    # Create positions for each node\n    pos = nx.nx_agraph.graphviz_layout(graph, prog=\"dot\")\n\n    # Normalize the colors along the range of token scores\n    if score == 'token':\n        scores = [data['tokenscore'] for _, data in graph.nodes(data=True) if data['token'] is not None]\n    elif score == 'sequence':\n        scores = [data['sequencescore'] for _, data in graph.nodes(data=True) if data['token'] is not None]\n    vmin = min(scores)\n    vmax = max(scores)\n    norm = mcolors.Normalize(vmin=vmin, vmax=vmax)\n    cmap = LinearSegmentedColormap.from_list('rg', [\"r\", \"y\", \"g\"], N=256) \n\n    # Draw the nodes\n    nx.draw_networkx_nodes(graph, pos, node_size=2000, node_shape='o', alpha=1, linewidths=4, \n                          node_color=scores, cmap=cmap)\n\n    # Draw the edges\n    nx.draw_networkx_edges(graph, pos)\n\n    # Draw the labels\n    if score == 'token':\n        labels = {node: data['token'].split('_')[0] + f\"n{data['tokenscore']:.2f}%\" for node, data in graph.nodes(data=True) if data['token'] is not None}\n    elif score == 'sequence':\n        labels = {node: data['token'].split('_')[0] + f\"n{data['sequencescore']:.2f}\" for node, data in graph.nodes(data=True) if data['token'] is not None}\n    nx.draw_networkx_labels(graph, pos, labels=labels, font_size=10)\n    plt.box(False)\n\n    # Add a colorbar\n    sm = plt.cm.ScalarMappable(cmap=cmap, norm=norm)\n    sm.set_array([])\n    if score == 'token':\n        fig.colorbar(sm, ax=ax, orientation='vertical', pad=0, label='Token probability (%)')\n    elif score == 'sequence':\n        fig.colorbar(sm, ax=ax, orientation='vertical', pad=0, label='Sequence score')\n    plt.show()\n\n# Plot graph\nplot_graph(graph, length, 1.5, 'token')\n```\n\n![Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1E3552Q01cDUt1UDKzduuYw.png)\nImage by author.\nIn this graph, the top node stores the input token (thus with a 100% probability), while all other nodes represent generated tokens. Although each token in this sequence was the most likely at the time of prediction, \"being\" and \"doctor\" were assigned relatively low probabilities of 9.68% and 2.86%, respectively. This suggests that \"of\", our first predicted token, may not have been the most suitable choice as it led to \"being\", which is quite unlikely.\nIn the following section, we'll explore how beam search can address this problem.\n## ⚖️ Beam Search\nUnlike greedy search, which only considers the next most probable token, beam search takes into account the _n_ most likely tokens, where _n_ represents the number of beams. This procedure is repeated until a predefined maximum length is reached or an end-of-sequence token appears. At this point, the sequence (or \"beam\") with the highest overall score is chosen as the output.\nWe can adapt the previous function to consider the _n_ most probable tokens instead of just one. Here, we'll maintain the sequence score log _P(w)_ , which is the cumulative sum of the log probability of every token in the beam. We normalize this score by the sequence length to prevent bias towards longer sequences (this factor can be adjusted). Once again, we'll generate five additional tokens to complete the sentence \"I have a dream.\"\npython\n\n```\nfrom tqdm.notebook import tqdm\n\ndef greedy_sampling(logits, beams):\n    return torch.topk(logits, beams).indices\n\ndef beam_search(input_ids, node, bar, length, beams, sampling, temperature=0.1):\n    if length == 0:\n        return None\n\n    outputs = model(input_ids)\n    predictions = outputs.logits\n\n    # Get the predicted next sub-word (here we use top-k search)\n    logits = predictions[0, -1, :]\n\n    if sampling == 'greedy':\n        top_token_ids = greedy_sampling(logits, beams)\n    elif sampling == 'top_k':\n        top_token_ids = top_k_sampling(logits, temperature, 20, beams)\n    elif sampling == 'nucleus':\n        top_token_ids = nucleus_sampling(logits, temperature, 0.5, beams)\n\n    for j, token_id in enumerate(top_token_ids):\n        bar.update(1)\n\n        # Compute the score of the predicted token\n        token_score = get_log_prob(logits, token_id)\n        cumulative_score = graph.nodes[node]['cumscore'] + token_score\n\n        # Add the predicted token to the list of input ids\n        new_input_ids = torch.cat([input_ids, token_id.unsqueeze(0).unsqueeze(0)], dim=-1)\n\n        # Add node and edge to graph\n        token = tokenizer.decode(token_id, skip_special_tokens=True)\n        current_node = list(graph.successors(node))[j]\n        graph.nodes[current_node]['tokenscore'] = np.exp(token_score) * 100\n        graph.nodes[current_node]['cumscore'] = cumulative_score\n        graph.nodes[current_node]['sequencescore'] = 1/(len(new_input_ids.squeeze())) * cumulative_score\n        graph.nodes[current_node]['token'] = token + f\"_{length}_{j}\"\n\n        # Recursive call\n        beam_search(new_input_ids, current_node, bar, length-1, beams, sampling, 1)\n\n# Parameters\nlength = 5\nbeams = 2\n\n# Create a balanced tree with height 'length' and branching factor 'k'\ngraph = nx.balanced_tree(beams, length, create_using=nx.DiGraph())\nbar = tqdm(total=len(graph.nodes))\n\n# Add 'tokenscore', 'cumscore', and 'token' attributes to each node\nfor node in graph.nodes:\n    graph.nodes[node]['tokenscore'] = 100\n    graph.nodes[node]['cumscore'] = 0\n    graph.nodes[node]['sequencescore'] = 0\n    graph.nodes[node]['token'] = text\n\n# Start generating text\nbeam_search(input_ids, 0, bar, length, beams, 'greedy', 1)\n```\n\nThe function computes the scores for 63 tokens and beams^length = 5² = 25 possible sequences. In our implementation, all the information is stored in the graph. Our next step is to extract the best sequence.\nFirst, we identify the leaf node with the highest sequence score. Next, we find the shortest path from the root to this leaf. Every node along this path contains a token from the optimal sequence. Here's how we can implement it:\npython\n\n```\ndef get_best_sequence(G):\n    # Create a list of leaf nodes\n    leaf_nodes = [node for node in G.nodes() if G.out_degree(node)==0]\n\n    # Get the leaf node with the highest cumscore\n    max_score_node = None\n    max_score = float('-inf')\n    for node in leaf_nodes:\n        if G.nodes[node]['sequencescore'] > max_score:\n            max_score = G.nodes[node]['sequencescore']\n            max_score_node = node\n\n    # Retrieve the sequence of nodes from this leaf node to the root node in a list\n    path = nx.shortest_path(G, source=0, target=max_score_node)\n\n    # Return the string of token attributes of this sequence\n    sequence = \"\".join([G.nodes[node]['token'].split('_')[0] for node in path])\n\n    return sequence, max_score\n\nsequence, max_score = get_best_sequence(graph)\nprint(f\"Generated text: {sequence}\")\n```\n\nplaintext\n\n```\nGenerated text: I have a dream. I have a dream\n```\n\nThe best sequence seems to be \"I have a dream. I have a dream,\" which is a common response from GPT-2, even though it may be surprising. To verify this, let's plot the graph.\nIn this visualization, we'll display the sequence score for each node, which represents the score of the sequence up to that point. If the function get_best_sequence() is correct, the \"dream\" node in the sequence \"I have a dream. I have a dream\" should have the highest score among all the leaf nodes.\npython\n\n```\n# Plot graph\nplot_graph(graph, length, beams, 'sequence')\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1aKWWxjqnv79hvhrDLpfw3A.png)\nIndeed, the \"dream\" token has the **highest sequence score** with a value of -0.69. Interestingly, we can see the score of the greedy sequence \"I have a dream of being a doctor.\" on the left with a value of -1.16.\nAs expected, the greedy search leads to suboptimal results. But, to be honest, our new outcome is not particularly compelling either. To generate more varied sequences, we'll implement two sampling algorithms: top-k and nucleus.\n## 🎲 Top-k sampling\nTop-k sampling is a technique that leverages the probability distribution generated by the language model to **select a token randomly from the** _**k**_**most likely options**.\nTo illustrate, suppose we have _k = 3_ and four tokens: A, B, C, and D, with respective probabilities: _P(A) = 30%_ , _P(B) = 15%_ , _P(C) = 5%_ , and _P(D) = 1%_. In top-k sampling, token D is disregarded, and the algorithm will output A 60% of the time, B 30% of the time, and C 10% of the time. This approach ensures that we prioritize the most probable tokens while introducing an element of randomness in the selection process.\nAnother way of introducing randomness is the concept of temperature. The temperature _T_ is a parameter that ranges from 0 to 1, which affects the probabilities generated by the softmax function, making the most likely tokens more influential. In practice, it simply consists of dividing the input logits by a value we call temperature:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1XaEAvo08OCmNsfoyMXOFrA.png)\nHere is a chart that demonstrates the impact of temperature on the probabilities generated for a given set of input logits [1.5, -1.8, 0.9, -3.2]. We've plotted three different temperature values to observe the differences.\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1qAEEn6ylVTMZcSZRsFEqsw.png)\nA temperature of 1.0 is equivalent to a default softmax with no temperature at all. On the other hand, a low temperature setting (0.1) significantly alters the probability distribution. This is commonly used in text generation to control the level of \"creativity\" in the generated output. By adjusting the temperature, we can influence the extent to which the model produces more diverse or predictable responses.\nLet's now implement the top k sampling algorithm. We'll use it in the beam_search() function by providing the \"top_k\" argument. To illustrate how the algorithm works, we will also plot the probability distributions for top_k = 20.\npython\n\n```\ndef plot_prob_distribution(probabilities, next_tokens, sampling, potential_nb, total_nb=50):\n    # Get top k tokens\n    top_k_prob, top_k_indices = torch.topk(probabilities, total_nb)\n    top_k_tokens = [tokenizer.decode([idx]) for idx in top_k_indices.tolist()]\n\n    # Get next tokens and their probabilities\n    next_tokens_list = [tokenizer.decode([idx]) for idx in next_tokens.tolist()]\n    next_token_prob = probabilities[next_tokens].tolist()\n\n    # Create figure\n    plt.figure(figsize=(0.4*total_nb, 5), dpi=300, facecolor='white')\n    plt.rc('axes', axisbelow=True)\n    plt.grid(axis='y', linestyle='-', alpha=0.5)\n    if potential_nb < total_nb:\n        plt.axvline(x=potential_nb-0.5, ls=':', color='grey', label='Sampled tokens')\n    plt.bar(top_k_tokens, top_k_prob.tolist(), color='blue')\n    plt.bar(next_tokens_list, next_token_prob, color='red', label='Selected tokens')\n    plt.xticks(rotation=45, ha='right', va='top')\n    plt.gca().spines['top'].set_visible(False)\n    plt.gca().spines['right'].set_visible(False)\n    if sampling == 'top_k':\n        plt.title('Probability distribution of predicted tokens with top-k sampling')\n    elif sampling == 'nucleus':\n        plt.title('Probability distribution of predicted tokens with nucleus sampling')\n    plt.legend()\n    plt.savefig(f'{sampling}_{time.time()}.png', dpi=300)\n    plt.close()\n\ndef top_k_sampling(logits, temperature, top_k, beams, plot=True):\n    assert top_k >= 1\n    assert beams <= top_k\n\n    indices_to_remove = logits < torch.topk(logits, top_k)[0][..., -1, None]\n    new_logits = torch.clone(logits)\n    new_logits[indices_to_remove] = float('-inf')\n\n    # Convert logits to probabilities\n    probabilities = torch.nn.functional.softmax(new_logits / temperature, dim=-1)\n\n    # Sample n tokens from the resulting distribution\n    next_tokens = torch.multinomial(probabilities, beams)\n\n    # Plot distribution\n    if plot:\n        total_prob = torch.nn.functional.softmax(logits / temperature, dim=-1)\n        plot_prob_distribution(total_prob, next_tokens, 'top_k', top_k)\n\n    return next_tokens\n\n# Start generating text\nbeam_search(input_ids, 0, bar, length, beams, 'top_k', 1)\n```\n\n![Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/162P0A93C5dqQDqFg0BXK6Q.gif)\nImage by author.\nThese plots give a good intuition of how top-k sampling works, with all the potentially selected tokens on the left of the horizontal bar. While the most probable tokens are selected (in red) most of the time, it also allows less likely tokens to be chosen. This offers an interesting tradeoff that can steer a sequence towards a less predictable but more natural-sounding sentence. Now let's print the text it generated.\npython\n\n```\nsequence, max_score = get_best_sequence(graph)\nprint(f\"Generated text: {sequence}\")\n```\n\nplaintext\n\n```\nGenerated text: I have a dream job and I want to\n```\n\nThe top-k sampling found a new sequence: \"I have a dream job and I want to\", which feels significantly more natural than \"I have a dream. I have a dream\". We're making progress!\nLet's see how this decision tree differs from the previous one.\npython\n\n```\n# Plot graph\nplot_graph(graph, length, beams, 'sequence')\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1mnlTo893vrn59u-GUuuABA.png)\nYou can see how the nodes differ significantly from the previous iteration, making more diverse choices. Although the sequence score of this new outcome might not be the highest (-1.01 instead of -0.69 previously), it's important to remember that higher scores do not always lead to more realistic or meaningful sequences.\nNow that we've introduced top-k sampling, we have to present the other most popular sampling technique: nucleus sampling.\n## 🔬 Nucleus sampling\nNucleus sampling, also known as top-p sampling, takes a different approach from top-k sampling. Rather than selecting the top _k_ most probable tokens, nucleus sampling chooses a cutoff value _p_ such that the **sum of the probabilities of the selected tokens exceeds** _**p**_. This forms a \"nucleus\" of tokens from which to randomly choose the next token.\nIn other words, the model examines its top probable tokens in descending order and keeps adding them to the list until the total probability surpasses the threshold _p_. Unlike top-k sampling, the number of tokens included in the nucleus can vary from step to step. This variability often results in a more diverse and creative output, making nucleus sampling popular for tasks such as text generation.\nTo implement the nucleus sampling method, we can use the \"nucleus\" parameter in the beam_search() function. In this example, we'll set the value of _p_ to 0.5. To make it easier, we'll include a minimum number of tokens equal to the number of beams. We'll also consider tokens with cumulative probabilities lower than _p_ , rather than higher. It's worth noting that while the details may differ, the core idea of nucleus sampling remains the same.\npython\n\n```\ndef nucleus_sampling(logits, temperature, p, beams, plot=True):\n    assert p > 0\n    assert p <= 1\n\n    # Sort the probabilities in descending order and compute cumulative probabilities\n    sorted_logits, sorted_indices = torch.sort(logits, descending=True)\n    probabilities = torch.nn.functional.softmax(sorted_logits / temperature, dim=-1)\n    cumulative_probabilities = torch.cumsum(probabilities, dim=-1)\n\n    # Create a mask for probabilities that are in the top-p\n    mask = cumulative_probabilities < p\n\n    # If there's not n index where cumulative_probabilities < p, we use the top n tokens instead\n    if mask.sum() > beams:\n        top_p_index_to_keep = torch.where(mask)[0][-1].detach().cpu().tolist()\n    else:\n        top_p_index_to_keep = beams\n\n    # Only keep top-p indices\n    indices_to_remove = sorted_indices[top_p_index_to_keep:]\n    sorted_logits[indices_to_remove] = float('-inf')\n\n    # Sample n tokens from the resulting distribution\n    probabilities = torch.nn.functional.softmax(sorted_logits / temperature, dim=-1)\n    next_tokens = torch.multinomial(probabilities, beams)\n\n    # Plot distribution\n    if plot:\n        total_prob = torch.nn.functional.softmax(logits / temperature, dim=-1)\n        plot_prob_distribution(total_prob, next_tokens, 'nucleus', top_p_index_to_keep)\n\n    return next_tokens\n\n# Start generating text\nbeam_search(input_ids, 0, bar, length, beams, 'nucleus', 1)\n```\n\n![Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1pIvIyaLxQ_b_M7ZaZncY8A.gif)\nImage by author.\nIn this plot, you can see that the number of tokens included in the nucleus (left of the vertical bar) fluctuates a lot. The generated probability distributions vary considerably, leading to the selection of tokens that are not always among the most probable ones. This opens the door to the generation of unique and varied sequences. Now, let's observe the text it generated.\npython\n\n```\nsequence, max_score = get_best_sequence(graph)\nprint(f\"Generated text: {sequence}\")\n```\n\nplaintext\n\n```\nGenerated text: I have a dream. I'm going to\n```\n\nThe nucleus sampling algorithm produces the sequence: \"I have a dream. I'm going to\", which shows a notable enhancement in semantic coherence compared to greedy sampling.\nTo compare the decision paths, let's visualize the new tree nucleus sampling generated.\npython\n\n```\n# Plot graph\nplot_graph(graph, length, beams, 'sequence')\n```\n\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1sFl6hMRhlBZP0At4_VbHKg.png)\nAs with top-k sampling, this tree is very different from the one generated with greedy sampling, displaying more variety. Both top-k and nucleus sampling offer unique advantages when generating text, enhancing diversity, and introducing creativity into the output. Your choice between the two methods (or even greedy search) will depend on the specific requirements and constraints of your project.\n## Conclusion\nIn this article, we have delved deep into various decoding methods used by LLMs, specifically GPT-2. We started with a simply **greedy search** and its immediate (yet often suboptimal) selection of the most probable next token. Next, we introduced the **beam search** technique, which considers several of the most likely tokens at each step. Although it offers more nuanced results, beam search can sometimes fall short in generating diverse and creative sequences.\nTo bring more variability into the process, we then moved on to **top-k sampling** and **nucleus sampling**. Top-k sampling diversifies the text generation by randomly selecting among the _k_ most probable tokens, while nucleus sampling takes a different path by dynamically forming a nucleus of tokens based on cumulative probability. Each of these methods brings unique strengths and potential drawbacks to the table, and the specific requirements of your project will largely dictate the choice among them.\nUltimately, understanding these techniques and their trade-offs will equip you to better guide the LLMs towards producing increasingly realistic, nuanced, and compelling textual output.\nIf you're interested in more technical content around LLMs, you can follow me on Twitter [@maximelabonne](https://twitter.com/maximelabonne).\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657499317_sxy3ci.png)\nWritten By\n## [Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[See all from Maxime Labonne](https://towardsdatascience.com/author/mlabonne/)\n[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Artificial Intelligence](https://towardsdatascience.com/tag/artificial-intelligence/)[Large Language Models](https://towardsdatascience.com/tag/large-language-models/)[Neural Networks](https://towardsdatascience.com/tag/neural-networks/)[Text Generation](https://towardsdatascience.com/tag/text-generation/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fdecoding-strategies-in-large-language-models-9733a8f70539%2F&title=Decoding%20Strategies%20in%20Large%20Language%20Models)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fdecoding-strategies-in-large-language-models-9733a8f70539%2F&title=Decoding%20Strategies%20in%20Large%20Language%20Models)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fdecoding-strategies-in-large-language-models-9733a8f70539%2F&text=Decoding%20Strategies%20in%20Large%20Language%20Models)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1TXszGQtWh7PPHb_4BlEbIQ.png)](https://towardsdatascience.com/styleformer-convert-casual-text-to-formal-text-and-vice-versa-9cdc52abeaf5/)\n## [Styleformer: Convert Casual Text to Formal Text and Vice Versa](https://towardsdatascience.com/styleformer-convert-casual-text-to-formal-text-and-vice-versa-9cdc52abeaf5/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nHow to leverage a powerful T5 Transformer model to change the formality of text\n[Eric Fillion](https://towardsdatascience.com/author/ericfillion/)\nAugust 2, 20214 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/0alp8vdiV9Lka3oCE-scaled.jpg)](https://towardsdatascience.com/unsupervized-image-clusterization-7ff7fadcf358/)\n## [Unsupervised image mapping](https://towardsdatascience.com/unsupervized-image-clusterization-7ff7fadcf358/)\n[](https://towardsdatascience.com/decoding-strategies-in-large-language-models-9733a8f70539/)\nA simple and efficient way to explore a large quantity of images\n[Jordan Thieyre](https://towardsdatascience.com/author/jordan.thieyre/)\nSeptember 24, 20209 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1iYCCXCfq9iEf3-TLHGwWDg.png)](https://towardsdatascience.com/the-aim-of-scikit-learn-1-0-985642811f27/)\n## [The Aim of Scikit-Learn 1.0](https://towardsdatascience.com/the-aim-of-scikit-learn-1-0-985642811f27/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhat are the new features and why are they so important to be the first major release?\n[DelgadoPanadero](https://towardsdatascience.com/author/delgadopanadero/)\nOctober 12, 20215 min read\n  * [![](https://towardsdatascience.com/images/fallback-image.jpg)](https://towardsdatascience.com/when-i-was-told-to-lead-the-dataops-initiatives-at-work-i-didnt-know-where-to-begin-489b6aa751ca/)\n## [Getting to Know DataOps](https://towardsdatascience.com/when-i-was-told-to-lead-the-dataops-initiatives-at-work-i-didnt-know-where-to-begin-489b6aa751ca/)\n[Data Science](https://towardsdatascience.com/category/data-science/)\nWhen I was told to lead the DataOps initiatives at work, I didn't know where to begin. So I started from where it was the easiest, by...\n[Loh Meng Xin](https://towardsdatascience.com/author/lohmengxin/)\nAugust 1, 202112 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/0EFgJ2IaxCk8HPxMG-scaled.jpg)](https://towardsdatascience.com/balancing-innovation-with-safety-privacy-in-the-era-of-large-language-models-llm-a63570e4a24a/)\n## [Balancing Innovation With Safety & Privacy in the Era of Large Language Models (LLM)](https://towardsdatascience.com/balancing-innovation-with-safety-privacy-in-the-era-of-large-language-models-llm-a63570e4a24a/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nA Guide to Implement Safety, and Privacy Mechanisms for your Generative AI applications\n[Anjan Biswas](https://towardsdatascience.com/author/anjanava.biswas/)\nSeptember 20, 202312 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/0Wxyy1CmFCURJ0VRM-scaled.jpg)](https://towardsdatascience.com/c%e1%b5%a5-topic-coherence-explained-fc70e2a85227/)\n## [Cᵥ Topic Coherence Explained](https://towardsdatascience.com/c%e1%b5%a5-topic-coherence-explained-fc70e2a85227/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUnderstanding the metric that correlates the highest with humans.\n[Emil Rijcken](https://towardsdatascience.com/author/emilrijcken/)\nJanuary 12, 20236 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/ChatGPT-Image-May-27-2026-03_24_29-PM.jpg)](https://towardsdatascience.com/what-ai-agents-should-never-do-on-their-own/)\n## [What AI Agents Should Never Do on Their Own](https://towardsdatascience.com/what-ai-agents-should-never-do-on-their-own/)\n[Agentic AI](https://towardsdatascience.com/category/artificial-intelligence/agentic-ai/)\nHow to set the rules that keep agents effective and out of trouble\n[Sara Nobrega](https://towardsdatascience.com/author/saranobregafn/)\nJune 3, 20268 min read\n  * [![](https://towardsdatascience.com/images/fallback-image.jpg)](https://towardsdatascience.com/why-learn-data-science-in-2020-d3f54123b2e4/)\n## [Why learn Data Science in 2020?](https://towardsdatascience.com/why-learn-data-science-in-2020-d3f54123b2e4/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nWhy now? Can I really build a career in this field?\n[James Thorn](https://towardsdatascience.com/author/james-thorn/)\nJanuary 16, 20206 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1ZWSCJWpX5iDvZDEqCbx1gg.png)](https://towardsdatascience.com/v-net-u-nets-big-brother-in-image-segmentation-906e393968f7/)\n## [V-Net, U-Net's big brother in Image Segmentation](https://towardsdatascience.com/v-net-u-nets-big-brother-in-image-segmentation-906e393968f7/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nWelcome to this guide about the V-Net, the cousin of the well known U-Net, for 3D images segmentations. You will know it inside out!\n[François Porcher](https://towardsdatascience.com/author/francoisporcher/)\nJuly 28, 20237 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "Decoding Strategies in Large Language Models | Towards Data Science",
    "description": "A Guide to Text Generation From Beam Search to Nucleus Sampling",
    "keywords": null,
    "author": "Maxime Labonne",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Decoding Strategies in Large Language Models",
    "og:description": "A Guide to Text Generation From Beam Search to Nucleus Sampling",
    "og:url": "https://towardsdatascience.com/decoding-strategies-in-large-language-models-9733a8f70539/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1SSYwvJCdjB4llSm1P1XzOg.jpeg",
    "og:image:width": "1820",
    "og:image:height": "1024",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Maxime Labonne",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "15 min read",
    "article:published_time": "2023-06-04T14:41:49.000Z",
    "article:modified_time": "2023-06-04T14:41:49.000Z",
    "article:author": "https://towardsdatascience.com/author/mlabonne/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 202,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_000.txt",
      "chars": 531
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_001.txt",
      "chars": 50
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_002.txt",
      "chars": 1874
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_003.txt",
      "chars": 50
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_004.txt",
      "chars": 2004
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_005.txt",
      "chars": 2228
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_006.txt",
      "chars": 818
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_007.txt",
      "chars": 47
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_008.txt",
      "chars": 58
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_009.txt",
      "chars": 2169
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_010.txt",
      "chars": 84
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_011.txt",
      "chars": 49
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_012.txt",
      "chars": 58
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_013.txt",
      "chars": 1418
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_014.txt",
      "chars": 84
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_015.txt",
      "chars": 45
    },
    {
      "language": "txt",
      "file": "attachments/bc3c38f1a02d/code/block_016.txt",
      "chars": 58
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1SSYwvJCdjB4llSm1P1XzOg.jpeg",
      "alt": "Image by author.",
      "local_path": "attachments/bc3c38f1a02d/images/000.jpeg",
      "bytes": 120437,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1-1v4rvYrR4RPfIK22tvetQ.png",
      "alt": "Image by author.",
      "local_path": "attachments/bc3c38f1a02d/images/001.png",
      "bytes": 521412,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1YxUBg9dqe4BtStEj2SkRjw.png",
      "alt": "",
      "local_path": "attachments/bc3c38f1a02d/images/002.png",
      "bytes": 65172,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1E3552Q01cDUt1UDKzduuYw.png",
      "alt": "Image by author.",
      "local_path": "attachments/bc3c38f1a02d/images/003.png",
      "bytes": 88698,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/162P0A93C5dqQDqFg0BXK6Q.gif",
      "alt": "Image by author.",
      "local_path": "attachments/bc3c38f1a02d/images/004.gif",
      "bytes": 464537,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/06/1pIvIyaLxQ_b_M7ZaZncY8A.gif",
      "alt": "Image by author.",
      "local_path": "attachments/bc3c38f1a02d/images/005.gif",
      "bytes": 514284,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/08/1TXszGQtWh7PPHb_4BlEbIQ.png",
      "alt": "",
      "local_path": "attachments/bc3c38f1a02d/images/006.png",
      "bytes": 232619,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/09/0alp8vdiV9Lka3oCE-scaled.jpg",
      "alt": "",
      "local_path": "attachments/bc3c38f1a02d/images/007.jpg",
      "bytes": 791580,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1iYCCXCfq9iEf3-TLHGwWDg.png",
      "alt": "",
      "local_path": "attachments/bc3c38f1a02d/images/008.png",
      "bytes": 2275545,
      "downloaded": true
    },
    {
      "src": "/images/fallback-image.jpg",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/09/0EFgJ2IaxCk8HPxMG-scaled.jpg",
      "alt": "",
      "local_path": "attachments/bc3c38f1a02d/images/010.jpg",
      "bytes": 175623,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/0Wxyy1CmFCURJ0VRM-scaled.jpg",
      "alt": "",
      "local_path": "attachments/bc3c38f1a02d/images/011.jpg",
      "bytes": 644540,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2026/06/ChatGPT-Image-May-27-2026-03_24_29-PM.jpg",
      "alt": "",
      "local_path": "attachments/bc3c38f1a02d/images/012.jpg",
      "bytes": 254687,
      "downloaded": true
    },
    {
      "src": "/images/fallback-image.jpg",
      "alt": "",
      "local_path": "",
      "bytes": 0,
      "downloaded": false
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/07/1ZWSCJWpX5iDvZDEqCbx1gg.png",
      "alt": "",
      "local_path": "attachments/bc3c38f1a02d/images/014.png",
      "bytes": 209783,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://www.youtube.com/c/TowardsDataScience",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/mlabonne/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/large-language-models/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/neural-networks/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/text-generation/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/styleformer-convert-casual-text-to-formal-text-and-vice-versa-9cdc52abeaf5/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/ericfillion/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/unsupervized-image-clusterization-7ff7fadcf358/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/decoding-strategies-in-large-language-models-9733a8f70539/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/jordan.thieyre/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/the-aim-of-scikit-learn-1-0-985642811f27/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/delgadopanadero/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/when-i-was-told-to-lead-the-dataops-initiatives-at-work-i-didnt-know-where-to-begin-489b6aa751ca/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://github.com/mlabonne/llm-course/blob/main/Decoding_Strategies_in_Large_Language%C2%A0Models.ipynb"
    },
    {
      "type": "external",
      "href": "https://colab.research.google.com/drive/19CJlOS5lI29g-B3dziNn93Enez1yiHk2?usp=sharing"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/ai-engineer"
    },
    {
      "type": "external",
      "href": "https://en.wikipedia.org/wiki/Byte_pair_encoding"
    },
    {
      "type": "external",
      "href": "https://twitter.com/maximelabonne"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Decoding+Strategies+in+Large+Language+Models&u=https%3A%2F%2Ftowardsdatascience.com%2Fdecoding-strategies-in-large-language-models-9733a8f70539%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Decoding+Strategies+in+Large+Language+Models&url=https%3A%2F%2Ftowardsdatascience.com%2Fdecoding-strategies-in-large-language-models-9733a8f70539%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Decoding+Strategies+in+Large+Language+Models&url=https%3A%2F%2Ftowardsdatascience.com%2Fdecoding-strategies-in-large-language-models-9733a8f70539%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh"
    },
    {
      "type": "external",
      "href": "https://insightmediagroup.io/creator-network"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:48Z",
  "source_url": "https://towardsdatascience.com/decoding-strategies-in-large-language-models-9733a8f70539/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}