{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Jay Alammar",
  "title_listed": "Finding the Words to Say: Hidden State Visualizations for Language Models",
  "keywords": "隐藏状态,可视化,语言模型",
  "domain": "文本处理",
  "content_type": "图解教程",
  "is_paid_flag": "否",
  "url": "https://jalammar.github.io/hidden-states/",
  "publish_date": "2021-04",
  "corpus_stage": "预训练/SFT",
  "notes": "语言模型隐藏状态可视化",
  "id": "546e941916bb",
  "collected_at": "2026-10-02T14:24:17",
  "title": "Finding the Words to Say: Hidden State Visualizations for Language Models – Jay Alammar – Visualizing machine learning one concept at a time.",
  "body_markdown": "Finding the Words to Say: Hidden State Visualizations for Language Models\nBy visualizing the hidden state between a model's layers, we can get some clues as to the model's \"thought process\".\nFigure: Finding the words to say\nAfter a language model generates a sentence, we can visualize a view of how the model came by each word (column).  Each row is a model layer. The value and color indicate the ranking of the output token at that layer. The darker the color, the higher the ranking. Layer 0 is at the top. Layer 47 is at the bottom.\nModel:\nGPT2-XL\nPart 2: Continuing the pursuit of making Transformer language models more transparent, this article showcases a collection of visualizations to uncover mechanics of language generation inside a pre-trained language model. These visualizations are all created using\nEcco\n, the open-source package we're releasing\nIn the first part of this series,\nInterfaces for Explaining Transformer Language Models\n, we showcased interactive interfaces for input saliency and neuron activations. In this article, we will focus on the hidden state as it evolves from model layer to the next. By looking at the hidden states produced by every transformer decoder block, we aim to gleam information about how a language model arrived at a specific output token. This method is explored by Voita et al.\n. Nostalgebraist\npresents compelling visual treatments showcasing the evolution of token rankings, logit scores, and softmax\n        probabilities for the evolving hidden state through the various layers of the model.\nRecap: Transformer Hidden States\nThe following figure recaps how a transformer language model works. How the layers result in a final hidden state. And how that final state is then projected to the output vocabulary which results in a score assigned to each token in\n        the model's vocabulary. We can see here the top scoring tokens when DistilGPT2 is fed the input sequence \" 1, 1,\n        \":\nFigure: Recap of transformer language models.\nThis figure shows how the model arrives at the top five output token candidates and their probability scores. This shows us that at the final layer, the\n            model is 59% sure the next token is ' 1', and that would be chosen as the output token by greedy decoding.\n            Other probable outputs include ' 2' with 18% probability (maybe we are counting) and ' 0' with 5%\n            probability (maybe we are counting down).\nEcco provides a view of the model's top scoring tokens and their probability scores.\n# Generate one token to complete this input string\noutput\n=\nlm\n.\ngenerate\n(\n\" 1, 1, 1,\"\n,\ngenerate\n=\n1\n)\n# Visualize\noutput\n.\nlayer_predictions\n(\nposition\n=\n6\n,\nlayer\n=\n5\n)\nWhich would show the following breakdown of candidate output tokens and their probability scores:\nFigure: Ten tokens with highest probabilities at the final layer of the model.\nScores after each layer\nApplying the same projection to internal hidden states of the model gives us a view of how the model's conviction\n        for the output scoring developed over the processing of the inputs. This projection of internal hidden states\n        gives us a sense of which layer contributed the most to elevating the scores (and hence ranking) of a certain\n        potential output token.\nFigure: projecting inner hidden states to the model's vocabulary reveals cues of processing between layers.\nViewing the evolution of the hidden states means that instead of looking only at the candidates output tokens from\n        projecting the final model state, we can look at the top scoring tokens after projecting the hidden state\n        resulting from each of the model's six layers.\nThis visualization is created using the same method above with omitting the 'layer' argument (which we set to the final layer in the previous example, layer #5):\n# Visualize the top scoring tokens after each layer\noutput\n.\nlayer_predictions\n(\nposition\n=\n6\n)\nResulting in:\nFigure: Top scoring tokens after each of the model's six layers.\nEach row shows the top ten predicted tokens obtained by projecting each hidden state to the output\n            vocabulary. The probability scores are shown in pink (obtained by passing logit scores through softmax). We\n            can see that\nLayer 0\nhas no digits in its top ten predictions.\nLayer 1\ngives the token ' 1' a 0.03%, probability which, while low, still ranks the token as the seventh highest\n            ranking token. Subsequent layers keep elevating the probability and ranking of ' 1', until\nthe final\n            layer\ninjects a bit more caution by reducing the probability from 100% to ~60%, still retaining the\n            token as the highest ranked in the model's output.\nNote:\nThis figure is incorrect in showing 0 probability assigned to some tokens due to rounding. The current version of Ecco fixes this by showing '<0.01%'.\nYou can experiment with these visualizations and experiment with them on your own input sentences at the following colab link:\nEvolution of the selected token\nThe ranking of the token ' 1' after each layer\nLayer 0\nelevated the token ' 1' to be the 31st highest scored token in the hidden state it\n            produced.\nLayers 1 and 2\nkept increasing the ranking (to 7 then 5 respectively). All the\nfollowing layers\nwere sure this is the best token and gave it the top ranking spot.\nAnother visual perspective on the evolving hidden states is to re-examine the hidden states after selecting an output\n        token to see how the hidden state after each layer ranked that token. This is one of the many perspectives\n        explored by Nostalgebraist\nand the one we think is a great first approach. In the figure on the side, we can see the ranking (out of\n        +50,0000 tokens in the model's vocabulary) of the token ' 1' where each row\n        indicates a layer's output.\nThe same visualization can then be plotted for an entire generated sequence, where each column indicates a\n        generation step (and its output token), and each row the ranking of\n        the output token at each layer:\nEvolution of the rankings of the output sequence ' 1 , 1'\nWe can see that\nLayer 3\nis the point at which the model started to be certain of\n            the digit ' 1' as the output.\nWhen the output is to be a comma\n, Layer 0 usually ranks\n            the comma as 5.\nWhen the output is to be a ' 1'\n, Layer 0 is less certain, but still ranks the ' 1' token at\n            31 or 32.\n            Notice that every output token is ranked #1 after Layer 5. That is the definition of\ngreedy\n            sampling\n-- the reason we selected this token is because it was ranked first.\nLet us demonstrate this visualization by presenting the following input to GPT2-Large:\ninput:\n0\nThe\n1\ncountries\n2\nof\n3\nthe\n4\nEuropean\n5\nUnion\n6\nare\n7\n:\n8\n\\n\n9\n1\n10\n.\n11\nAustria\n12\n\\n\n13\n2\n14\n.\n15\nBelgium\n16\n\\n\n17\n3\n18\n.\n19\nBulgaria\n20\n\\n\n21\n4\n22\n.\nVisualizaing the evolution of the hidden states sheds light on how various layers contribute to generating this sequence as we can see in the following figure:\nFigure: Hidden state evolution of an output sequence\nClick to open image in full resolution. The figure reveals:\nColumns of solid pink corresponding to newlines and periods. Starting from Layer #0 and onwards, the model is certain early on of these tokens, indicating Layer #0's awareness of certain syntactic properties (and that later layers raise no objections).\nColumns where country names are predicted are very bright at the top and it's up to the last five layers to really come up with the appropriate token.\nColumns tracking the incrementing number tend to be resolved at layer #9.\nThe model erroneously lists Chile in the list, not a EU country. But notice that the ranking of that token is 43 -- indicating the error is better attributed to our token sampling method rather than to the model itself. In the case of all other countries they were correct and in the top 3.\nAside from Chile, the rest of the countries are correct, but also follow the alphabetical order followed in the input sequence.\nRankings of Other Tokens\nFigure: Rankings of which token should go in the blank\nWhile the final output succeeds in assigning the correct number, the first five layers surprisingly fail at\n            identifying the correct number (by giving \" is\" a higher ranking than \" are\", which is the correct answer).\n            Examining attention or inner-layer saliency could reveal clues as to the reason.\nWe are not limited to watching the evolution of only one (the selected) token for a specific position. There are\n        cases where we want to compare the rankings of multiple tokens\nin the same position\nregardless if the model selected them or not.\nOne such case is the number prediction task described by Linzen et al.\nwhich arises from the English language phenomenon of subject-verb agreement. In that task, we want to analyze the\n        model's capacity to encode\nsyntactic number\n(whether the subject we're addressing is singular or plural)\n        and\nsyntactic subjecthood\n(which subject in the sentence we're addressing).\nPut simply, fill-in the blank. The only acceptable answers are 1)\nis\n2)\nare\n:\nThe key\ns\nto the cabinet ______\nTo answer correctly, one has to first determine whether we're describing the keys (possible subject #1) or the\n        cabinet (possible subject #2). Having decided it is the keys, the second determination would be whether it is\n        singular or plural.\nThe model is able to assign a higher ranking to\nis\n, which is the correct token. Every layer\n            in the model managed to rank \" is\" higher than \" are\". The ranking of \" are\" remains high, however, as far as\n            rankings are concerned (the delta in probability scores might indicate otherwise, however).\nContrast your answer for the first question with the following variation:\nThe key to the cabinet\ns\n______\nThe figures in this section visualize the hidden-state evolution of the tokens \" is\" and \" are\". The numbers\n        in the cells are their ranking in the position of the blank (Both columns address the same position in the\n        sequence, they're not subsequent positions as was the case in the previous visualization).\nThe first figure (showing the rankings for the sequence \"The keys to the cabinet\") raises the question of why do five layers fail the task and only the final layer sets the record\n        straight. This is likely a similar effect to that observed in BERT of the final layer being the most\n        task-specific\n. It is also worth investigating whether that capability of succeeding at the task is predominantly localized in\n        Layer 5, or if the Layer is only the final expression in a circuit\nspanning multiple layers which is especially sensitive to subject-verb agreement.\nProbing for bias\nThis method can shed light on questions of bias and where they might emerge in a model. The following figures, for example, probe for the model's gender expectation associated with different professions:\nFigure: Probing bias in the model's association of gender with professions - Doctor and nurse\nThe first five layers all rank \" man\" higher than \" woman\" for both professions. For the nursing profession, the final layer decisively elevates \" woman\" to a higher ranking than \" man\".\nMore systemaic and nuanced examination of bias in contextualized word embeddings (another term for the vectors we've been referring to as \"hidden states\") can be found in\n.\nYour turn!\nYou can proceed to do your own experiments using Ecco and the three notebooks in this article:\nOutput Token Scores\nEvolution of Selected Token\nComparing Token Rankings\nYou can report issues you run into at the Ecco's Github page. Feel free to share any interesting findings at the Ecco\nDiscussion\nboard. I invite you again to read\nInterpreting GPT the Logit Lens\nand see the various ways the author examines such a visualization. I leave you with a small gallery of examples showcasing the responses of different models to different input prompts.\nGallery\nInput:\n\"Heathrow airport is located in the city of\"\nModel:\nDistilGPT2\nInput:\n\"Some of the most glorious historical attractions in Spain date from the period of Muslim rule, including The Mezquita, built as the Great Mosque of Cordoba and the Medina Azahara, also in Cordoba and now in ruins but still visitable as such and built as the Madinat al-Zahra, the Palace of al-Andalus; and the Alhambra in Granada, a splendid, intact palace. There are also two synagogues still standing that were built during the era of Muslim Spain: Santa Maria la Blanca in Toledo and the Synagogue of Cordoba, in the Old City. Reconquista and Imperial era\"\nModel:\nDistilGPT2\nModel:\nGPT2-Large\nModel:\nGPT2-XL\nInput:\n\"The countires of the European Union are:\\n1. Austria\\n2. Belgium\\n3.\n            Bulgaria\\n4.\"\nModel:\nDistilGPT2\nModel:\nGPT2-Large\nModel:\nGPT2-XL\nAcknowledgements\nThis article was vastly improved thanks to feedback on earlier drafts provided by\n        Abdullah Almaatouq,\n        Anfal Alatawi,\n        Fahd Alhazmi,\n        Hadeel Al-Negheimish,\n        Isabelle Augenstein,\n        Jasmijn Bastings,\n        Najwa Alghamdi,\n        Pepa Atanasova, and\n        Sebastian Gehrmann.\nReferences\nCitation\nIf you found this work helpful for your research, please cite it as following:\nAlammar, J. (2021). Finding the Words to Say: Hidden State Visualizations for Language Models [Blog post]. Retrieved from https://jalammar.github.io/hidden-states/\nBibTex:\n@misc{alammar2021hiddenstates, \n  title={Finding the Words to Say: Hidden State Visualizations for Language Models},\n  author={Alammar, J},\n  year={2021},\n  url={https://jalammar.github.io/hidden-states/}\n}\nWritten on January 19, 2021",
  "metadata": {
    "title": "Finding the Words to Say: Hidden State Visualizations for Language Models – Jay Alammar – Visualizing machine learning one concept at a time.",
    "description": "By visualizing the hidden state between a model's layers, we can get some clues as to the model's",
    "keywords": null,
    "author": "Jay Alammar",
    "og:description": "By visualizing the hidden state between a model's layers, we can get some clues as to the model's",
    "og:title": "Finding the Words to Say: Hidden State Visualizations for Language Models"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/546e941916bb/code/body-000.txt",
      "chars": 155,
      "language": "text"
    },
    {
      "file": "attachments/546e941916bb/code/body-001.txt",
      "chars": 88,
      "language": "text"
    },
    {
      "file": "attachments/546e941916bb/code/body-002.txt",
      "chars": 164,
      "language": "text"
    },
    {
      "file": "attachments/546e941916bb/code/body-003.txt",
      "chars": 207,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/rankings-gpt2xl.png",
      "bytes": 906526,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-0c19ae3988b14e5df32d.png",
      "src": "https://jalammar.github.io/images/explaining/rankings-gpt2xl.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/transformer-language-model-steps.png",
      "bytes": 138500,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-a842a1fa1d7250cda4de.png",
      "src": "https://jalammar.github.io/images/explaining/transformer-language-model-steps.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/prediction_scores.PNG",
      "bytes": 41525,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-375a4bbe56b5a593e0bd.png",
      "src": "https://jalammar.github.io/images/explaining/prediction_scores.PNG",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/predictions.PNG",
      "bytes": 198515,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-4450d15467b3caa144b4.png",
      "src": "https://jalammar.github.io/images/explaining/predictions.PNG",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/predictions%20all%20layers.PNG",
      "bytes": 199065,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-7ba39ced10cea7bd5b20.png",
      "src": "https://jalammar.github.io/images/explaining/predictions%20all%20layers.PNG",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/colab-badge.svg",
      "bytes": 2369,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-dcc12c5aed7f6af360e9.svg",
      "src": "https://jalammar.github.io/images/explaining/colab-badge.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/logit_ranking_1.png",
      "bytes": 10029,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-811d8596264f828a0348.png",
      "src": "https://jalammar.github.io/images/explaining/logit_ranking_1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/sequence_111_rankings.PNG",
      "bytes": 44484,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-199923d02224eb2b1f78.png",
      "src": "https://jalammar.github.io/images/explaining/sequence_111_rankings.PNG",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/ranking-eu-gpt2-thumb.png",
      "bytes": 161551,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-e58912f68d1fe34d9c05.png",
      "src": "https://jalammar.github.io/images/explaining/ranking-eu-gpt2-thumb.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/watch_keys_cabinet.png",
      "bytes": 20772,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-7f98cf3eff8c9502e3e9.png",
      "src": "https://jalammar.github.io/images/explaining/watch_keys_cabinet.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/watch_key_cabinets.png",
      "bytes": 20329,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-449a8b692c8a1c2b0d62.png",
      "src": "https://jalammar.github.io/images/explaining/watch_key_cabinets.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/doctor.png",
      "bytes": 20205,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-82572ae256c4ce770b98.png",
      "src": "https://jalammar.github.io/images/explaining/doctor.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/nurse.png",
      "bytes": 19110,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-996d0e2cd717279fdf1b.png",
      "src": "https://jalammar.github.io/images/explaining/nurse.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/london_rankings.png",
      "bytes": 14578,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-c3511d76bfc32333c0d7.png",
      "src": "https://jalammar.github.io/images/explaining/london_rankings.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/ranking-cordoba.png",
      "bytes": 194350,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-5c25641ccac5c18ccd21.png",
      "src": "https://jalammar.github.io/images/explaining/ranking-cordoba.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/cordoba-gpt2.png",
      "bytes": 213300,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-f62f416f859455490cf2.png",
      "src": "https://jalammar.github.io/images/explaining/cordoba-gpt2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/cordoba-gpt2xl.png",
      "bytes": 889643,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-ec31b78458cd83835f61.png",
      "src": "https://jalammar.github.io/images/explaining/cordoba-gpt2xl.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/ranking-eu.png",
      "bytes": 56182,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-c3fd77600196b64c460a.png",
      "src": "https://jalammar.github.io/images/explaining/ranking-eu.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/ranking-eu-gpt2.png",
      "bytes": 139113,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-08d526a3205a637b4d58.png",
      "src": "https://jalammar.github.io/images/explaining/ranking-eu-gpt2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/explaining/ranking-eu-gpt2xl.png",
      "bytes": 551360,
      "reused": false,
      "local_path": "attachments/546e941916bb/images/body-1da8bc7c7e745785ff71.png",
      "src": "https://jalammar.github.io/images/explaining/ranking-eu-gpt2xl.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.eccox.io/",
      "label": "Ecco",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/explaining-transformers/",
      "label": "Interfaces for Explaining Transformer Language Models",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/github/jalammar/ecco/blob/main/notebooks/Ecco_Output_Token_Scores.ipynb",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/images/explaining/ranking-eu-gpt2.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/github/jalammar/ecco/blob/main/notebooks/Ecco_Evolution_of_Selected_Token.ipynb",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/github/jalammar/ecco/blob/main/notebooks/Ecco_Comparing_Token_Rankings.ipynb",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/jalammar/ecco/discussions",
      "label": "Discussion",
      "type": "a"
    },
    {
      "href": "https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens",
      "label": "Interpreting GPT the Logit Lens",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/hidden-states/london_rankings.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/images/explaining/ranking-cordoba.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/images/explaining/cordoba-gpt2.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/images/explaining/cordoba-gpt2xl.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/images/explaining/ranking-eu.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/images/explaining/ranking-eu-gpt2xl.png",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:32Z",
  "source_url": "https://jalammar.github.io/hidden-states/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 13707,
  "expected_body_images": 20,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/546e941916bb/pages/source.html",
  "local_html": "attachments/546e941916bb/pages/article.html",
  "body_base_path": "attachments/546e941916bb/pages/",
  "body_markdown_path": "attachments/546e941916bb/pages/article.md"
}