{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Jay Alammar",
  "title_listed": "How GPT3 Works - Visualizations and Animations",
  "keywords": "GPT-3,可视化,大模型",
  "domain": "大模型算法",
  "content_type": "图解教程",
  "is_paid_flag": "否",
  "url": "https://jalammar.github.io/how-gpt3-works-visualizations-animations/",
  "publish_date": "2020-07",
  "corpus_stage": "预训练/SFT",
  "notes": "GPT-3工作原理图解",
  "id": "0e729c0ee4b6",
  "collected_at": "2026-10-02T14:17:14",
  "title": "How GPT3 Works - Visualizations and Animations – Jay Alammar – Visualizing machine learning one concept at a time.",
  "body_markdown": "How GPT3 Works - Visualizations and Animations\nDiscussions:\nHacker News (397 points, 97 comments)\n,\nReddit r/MachineLearning (247 points, 27 comments)\nTranslations:\nGerman\n,\nKorean\n,\nChinese (Simplified)\n,\nRussian\n,\nTurkish\nThe tech world is\nabuzz\nwith GPT3 hype. Massive language models (like GPT3) are starting to surprise us with their abilities. While not yet completely reliable for most businesses to put in front of their customers, these models are showing sparks of cleverness that are sure to accelerate the march of automation and the possibilities of intelligent computer systems. Let’s remove the aura of mystery around GPT3 and learn how it’s trained and how it works.\nA trained language model generates text.\nWe can optionally pass it some text as input, which influences its output.\nThe output is generated from what the model “learned” during its training period where it scanned vast amounts of text.\nTraining is the process of exposing the model to lots of text. That process has been completed. All the experiments you see now are from that one trained model. It was estimated to cost 355 GPU years and cost $4.6m.\nThe dataset of 300 billion tokens of text is used to generate training examples for the model. For example, these are three training examples generated from the one sentence at the top.\nYou can see how you can slide a window across all the text and make lots of examples.\nThe model is presented with an example. We only show it the features and ask it to predict the next word.\nThe model’s prediction will be wrong. We calculate the error in its prediction and update the model so next time it makes a better prediction.\nRepeat millions of times\nNow let’s look at these same steps with a bit more detail.\nGPT3 actually generates output one token at a time (let’s assume a token is a word for now).\nPlease note: This is a description of how GPT-3 works and not a discussion of what is novel about it (which is mainly the ridiculously large scale). The architecture is a transformer decoder model based on this paper https://arxiv.org/pdf/1801.10198.pdf\nGPT3 is MASSIVE. It encodes what it learns from training in 175 billion numbers (called parameters). These numbers are used to calculate which token to generate at each run.\nThe untrained model starts with random parameters. Training finds values that lead to better predictions.\nThese numbers are part of hundreds of matrices inside the model. Prediction is mostly a lot of matrix multiplication.\nIn my\nIntro to AI on YouTube\n, I showed a simple ML model with one parameter. A good start to unpack this 175B monstrosity.\nTo shed light on how these parameters are distributed and used, we’ll need to open the model and look inside.\nGPT3 is 2048 tokens wide. That is its “context window”. That means it has 2048 tracks along which tokens are processed.\nLet’s follow the purple track. How does a system process the word “robotics” and produce “A”?\nHigh-level steps:\nConvert the word to\na vector (list of numbers) representing the word\nCompute prediction\nConvert resulting vector to word\nThe important calculations of the GPT3 occur inside its stack of 96 transformer decoder layers.\nSee all these layers? This is the “depth” in “deep learning”.\nEach of these layers has its own 1.8B parameter to make its calculations. That is where the “magic” happens. This is a high-level view of that process:\nYou can see a detailed explanation of everything inside the decoder in my blog post\nThe Illustrated GPT2\n.\nThe difference with GPT3 is the alternating dense and\nsparse self-attention layers\n.\nThis is an X-ray of an input and response (“Okay human”) within GPT3. Notice how every token flows through the entire layer stack. We don’t care about the output of the first words. When the input is done, we start caring about the output. We feed every word back into the model.\nIn the\nReact code generation example\n, the description would be the input prompt (in green), in addition to a couple of examples of description=>code, I believe. And the react code would be generated like the pink tokens here token after token.\nMy assumption is that the priming examples and the description are appended as input, with specific tokens separating examples and the results. Then fed into the model.\nIt’s impressive that this works like this. Because you just wait until fine-tuning is rolled out for the GPT3. The possibilities will be even more amazing.\nFine-tuning actually updates the model’s weights to make the model better at a certain task.\nWritten on July 27, 2020",
  "metadata": {
    "title": "How GPT3 Works - Visualizations and Animations – Jay Alammar – Visualizing machine learning one concept at a time.",
    "description": "Discussions:\nHacker News (397 points, 97 comments), Reddit r/MachineLearning (247 points, 27 comments)\n\n\nTranslations: German, Korean, Chinese (Simplified), Russian, Turkish\n\n\nThe tech world is abuzz with GPT3 hype. Massive language models (like GPT3) are starting to surprise us with their abilities. While not yet completely reliable for most businesses to put in front of their customers, these models are showing sparks of cleverness that are sure to accelerate the march of automation and the possibilities of intelligent computer systems. Let’s remove the aura of mystery around GPT3 and learn how it’s trained and how it works.\n\n\n\n\n\nA trained language model generates text.\n\nWe can optionally pass it some text as input, which influences its output.\n\nThe output is generated from what the model “learned” during its training period where it scanned vast amounts of text.",
    "keywords": null,
    "author": "Jay Alammar",
    "og:description": "Discussions:\nHacker News (397 points, 97 comments), Reddit r/MachineLearning (247 points, 27 comments)\n\n\nTranslations: German, Korean, Chinese (Simplified), Russian, Turkish\n\n\nThe tech world is abuzz with GPT3 hype. Massive language models (like GPT3) are starting to surprise us with their abilities. While not yet completely reliable for most businesses to put in front of their customers, these models are showing sparks of cleverness that are sure to accelerate the march of automation and the possibilities of intelligent computer systems. Let’s remove the aura of mystery around GPT3 and learn how it’s trained and how it works.\n\n\n\n\n\nA trained language model generates text.\n\nWe can optionally pass it some text as input, which influences its output.\n\nThe output is generated from what the model “learned” during its training period where it scanned vast amounts of text.",
    "og:title": "How GPT3 Works - Visualizations and Animations"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/01-gpt3-language-model-overview.gif",
      "bytes": 2266096,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-0148f15fb4cd6435a977.gif",
      "src": "https://jalammar.github.io/images/gpt3/01-gpt3-language-model-overview.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/02-gpt3-training-language-model.gif",
      "bytes": 2810443,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-7e74af4261fe1d1bf9d6.gif",
      "src": "https://jalammar.github.io/images/gpt3/02-gpt3-training-language-model.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/gpt3-training-examples-sliding-window.png",
      "bytes": 105078,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-96feb1c622c0f5b2c3e7.png",
      "src": "https://jalammar.github.io/images/gpt3/gpt3-training-examples-sliding-window.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/03-gpt3-training-step-back-prop.gif",
      "bytes": 7927803,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-badd177047033d7bba3f.gif",
      "src": "https://jalammar.github.io/images/gpt3/03-gpt3-training-step-back-prop.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/04-gpt3-generate-tokens-output.gif",
      "bytes": 14684406,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-141d7d5964a2a028b4bc.gif",
      "src": "https://jalammar.github.io/images/gpt3/04-gpt3-generate-tokens-output.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/gpt3-parameters-weights.png",
      "bytes": 265247,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-e042c99c41ef4c85c89d.png",
      "src": "https://jalammar.github.io/images/gpt3/gpt3-parameters-weights.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/05-gpt3-generate-output-context-window.gif",
      "bytes": 25183764,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-da0a1d29d414ba420fc4.gif",
      "src": "https://jalammar.github.io/images/gpt3/05-gpt3-generate-output-context-window.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/06-gpt3-embedding.gif",
      "bytes": 4263258,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-b71d4b68b0ee735dab69.gif",
      "src": "https://jalammar.github.io/images/gpt3/06-gpt3-embedding.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/07-gpt3-processing-transformer-blocks.gif",
      "bytes": 7334278,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-bb38a1dbc72d263537b1.gif",
      "src": "https://jalammar.github.io/images/gpt3/07-gpt3-processing-transformer-blocks.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/08-gpt3-tokens-transformer-blocks.gif",
      "src": "https://jalammar.github.io/images/gpt3/08-gpt3-tokens-transformer-blocks.gif",
      "alt": "",
      "downloaded": false,
      "bytes": 36177486,
      "reused": false,
      "content_type": "image/gif",
      "local_path": "attachments/0e729c0ee4b6/retry-20261019/images/875262221de7fe1726ed.gif",
      "retried_at": "2026-10-18T17:17:33Z"
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/09-gpt3-generating-react-code-example.gif",
      "bytes": 26539756,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-441d859717cbb910fe8f.gif",
      "src": "https://jalammar.github.io/images/gpt3/09-gpt3-generating-react-code-example.gif",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://jalammar.github.io/images/gpt3/10-gpt3-fine-tuning.gif",
      "bytes": 4014710,
      "reused": false,
      "local_path": "attachments/0e729c0ee4b6/images/body-834225a2872526f00164.gif",
      "src": "https://jalammar.github.io/images/gpt3/10-gpt3-fine-tuning.gif",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=MQnJZuBGmSQ",
      "src": "https://www.youtube.com/embed/MQnJZuBGmSQ",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/0e729c0ee4b6/videos/repair-MQnJZuBGmSQ.mp4",
      "bytes": 104967409,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "821.461000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=mSTCzNgDJy4",
      "src": "https://youtube.com/watch?v=mSTCzNgDJy4",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/0e729c0ee4b6/videos/repair-mSTCzNgDJy4.mp4",
      "bytes": 77671042,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "569.061000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://news.ycombinator.com/item?id=23967887",
      "label": "Hacker News (397 points, 97 comments)",
      "type": "a"
    },
    {
      "href": "https://www.reddit.com/r/MachineLearning/comments/hwxn26/p_how_gpt3_works_visuals_and_animations/",
      "label": "Reddit r/MachineLearning (247 points, 27 comments)",
      "type": "a"
    },
    {
      "href": "https://www.arnevogel.com/wie-gpt3-funktioniert/",
      "label": "German",
      "type": "a"
    },
    {
      "href": "https://chloamme.github.io/2021/12/18/how-gpt3-works-visualizations-animations-korean.html",
      "label": "Korean",
      "type": "a"
    },
    {
      "href": "https://blogcn.acacess.com/how-gpt3-works-visualizations-and-animations-zhong-yi",
      "label": "Chinese (Simplified)",
      "type": "a"
    },
    {
      "href": "https://habr.com/ru/post/514698/",
      "label": "Russian",
      "type": "a"
    },
    {
      "href": "https://devrimdanyal.medium.com/g%C3%B6rselle%C5%9Ftirmeler-ve-animasyonlar-ile-gpt3-nas%C4%B1l-%C3%A7al%C4%B1%C5%9F%C4%B1r-e7891ed3fa88",
      "label": "Turkish",
      "type": "a"
    },
    {
      "href": "https://www.theverge.com/21346343/gpt-3-explainer-openai-examples-errors-agi-potential",
      "label": "abuzz",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/embed/MQnJZuBGmSQ",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://youtube.com/watch?v=mSTCzNgDJy4",
      "label": "Intro to AI on YouTube",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/illustrated-word2vec/",
      "label": "a vector (list of numbers) representing the word",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/illustrated-gpt2/",
      "label": "The Illustrated GPT2",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/1904.10509.pdf",
      "label": "sparse self-attention layers",
      "type": "a"
    },
    {
      "href": "https://twitter.com/sharifshameem/status/1284421499915403264",
      "label": "React code generation example",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:26Z",
  "source_url": "https://jalammar.github.io/how-gpt3-works-visualizations-animations/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.10509.pdf",
      "bytes": 4436935,
      "reused": false,
      "local_path": "papers/repair/e59165d159df0614c4b5.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 4541,
  "expected_body_images": 12,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/0e729c0ee4b6/pages/source.html",
  "local_html": "attachments/0e729c0ee4b6/pages/article.html",
  "body_base_path": "attachments/0e729c0ee4b6/pages/",
  "body_markdown_path": "attachments/0e729c0ee4b6/pages/article.md"
}