{ "source_file": "0930", "source_platform": "个人博客", "author": "Jay Alammar", "title_listed": "The Illustrated Retrieval Transformer", "keywords": "检索,Transformer,RAG", "domain": "大模型算法", "content_type": "图解教程", "is_paid_flag": "否", "url": "https://jalammar.github.io/illustrated-retrieval-transformer/", "publish_date": "2022-01", "corpus_stage": "推理部署", "notes": "检索增强Transformer图解", "id": "c62c94113743", "collected_at": "2026-10-02T14:20:47", "title": "The Illustrated Retrieval Transformer – Jay Alammar – Visualizing machine learning one concept at a time.", "body_markdown": "The Illustrated Retrieval Transformer\nDiscussion:\nDiscussion Thread\nfor comments, corrections, or any feedback.\nTranslations:\nKorean\n,\nRussian\nSummary\n: The latest batch of language models can be much smaller yet achieve GPT-3 like performance by being able to query a database or search the web for information. A key indication is that building larger and larger models is not the only way to improve performance.\nVideo\nThe last few years saw the rise of Large Language Models (LLMs) – machine learning models that rapidly improve how machines process and generate language. Some of the highlights since 2017 include:\nThe original\nTransformer\nbreaks previous performance records for machine translation.\nBERT\npopularizes the pre-training then finetuning process, as well as Transformer-based contextualized word embeddings. It then rapidly starts to power\nGoogle Search\nand\nBing Search\n.\nGPT-2\ndemonstrates the machine’s ability to write as well as humans do.\nFirst\nT5\n, then\nT0\npush the boundaries of transfer learning (training a model on one task, and then having it do well on other adjacent tasks) and posing a lot of different tasks as text-to-text tasks.\nGPT-3\nshowed that massive scaling of generative models can lead to shocking emergent applications (the industry continues to train larger models like\nGopher\n,\nMT-NLG\n…etc).\nFor a while, it seemed like scaling larger and larger models is the main way to improve performance. Recent developments in the field, like DeepMind’s\nRETRO Transformer\nand OpenAI’s\nWebGPT\n, reverse this trend by showing that smaller generative language models can perform on par with massive models if we augment them with a way to search/query for information.\nThis article breaks down DeepMind’s RETRO (\nR\netrieval-\nE\nnhanced\nTR\nansf\nO\nrmer) and how it works. The model performs on par with GPT-3 despite being 4% its size (7.5 billion parameters vs. 185 billion for GPT-3 Da Vinci).\nRETRO incorporates information retrieved from a database to free its parameters from being an expensive store of facts and world knowledge.\nRETRO was presented in the paper\nImproving Language Models by Retrieving from Trillions of Tokens\n. It continues and builds on a wide variety of retrieval\nwork\nin\nthe\nresearch\ncommunity\n. This article explains the model and not what is especially novel about it.\nWhy This is Important: Separating Language Information from World Knowledge Information\nLanguage modeling trains models to predict the next word–to fill-in-the-blank at the end of the sentence, essentially.\nFilling the blank sometimes requires knowledge of factual information (e.g. names or dates). For example:\nInput prompt: The Dune film was released in ....\nOther times, familiarity with the language is enough to guess what goes in the blank. For example:\nInput prompt: its popularity spread by word-of-mouth to allow Herbert to start working full ....\nThis distinction is important because LLMs encoded everything they know in their model parameters. While this makes sense for language information, it is inefficient for factual and world-knowledge information.\nBy including a retrieval method in the language model, the model can be much smaller. A neural database aids it with retrieving factual information it needs during text generation.\nAiding language models with retrieval methods allows us to reduce the amount of information a language model needs to encode in its parameters to perform well at text generation.\nTraining becomes fast with small language models, as training data memorization is reduced. Anyone can deploy these models on smaller and more affordable GPUs and tweak them as per need.\nMechanically, RETRO is an encoder-decoder model just like the original transformer. However, it augments the input sequence with the help of a retrieval database. The model finds the most probable sequences in the database and adds them to the input. RETRO works its magic to generate the output prediction.\nRETRO utilizes a database to augment its input prompt. The prompt is used to retrieve relevant information from the database.\nBefore we explore the model architecture, let’s dig deeper into the retrieval database.\nInspecting RETRO’s Retrieval Database\nThe database is a key-value store.\nThe key is a standard BERT sentence embedding.\nThe value is text in two parts:\nNeighbor, which is used to compute the key\nCompletion, the continuation of the text in the original document.\nRETRO’s database contains 2 trillion multi-lingual tokens based on the\nMassiveText\ndataset. Both the neighbor and completion chunks are at most 64 tokens long.\nA look inside RETRO's database shows examples of key-value pairs in the RETRO database. The value contains a neighbor chunk and a completion chunk.\nRETRO breaks the input prompt into multiple chunks. For simplicity, we’ll focus on how one chunk is augmented with retrieved text. The model, however, does this process for each chunk (except the first) in the input prompt.\nThe Database Lookup\nBefore hitting RETRO, the input prompt goes into BERT. The output contextualized vectors are then averaged to construct a sentence embedding vector. That vector is then used to query the database.\nProcessing the input prompt with BERT produces contextualized token embeddings. Averaging them produces a sentence embedding.\nThat sentence embedding is then used in an approximate nearest neighbor search (\nhttps://github.com/google-research/google-research/tree/master/scann\n).\nThe two nearest neighbors are retrieved, and their text becomes a part of the input into RETRO.\nThe BERT sentence embedding is used to retrieve the nearest neighbors from RETRO's neural database. These are then added to the input of the language model.\nThis is now the input to RETRO. The input prompt and its two nearest neighbors from the database (and their continuations).\nFrom here, the Transformer and RETRO Blocks incorporate the information into their processing.\nThe retrieved neighbors are added to the input of the language model. They're treated a little differently inside the model, however.\nRETRO Architecture at a High Level\nRETRO’s architecture is an encoder stack and a decoder stack.\nA RETRO transformer consists of an encoder stack (to process the neighbors) and a decoder stack (to process the input)\nThe encoder is made up of standard Transformer encoder blocks (self-attention + FFNN). To my best understanding, Retro uses an encoder made up of two Transformer Encoder Blocks.\nThe decoder stack interleaves two kinds of decoder blocks:\nStandard transformer decoder block (ATTN + FFNN)\nRETRO decoder block (ATTN + Chunked cross attention (CCA) + FFNN)\nThe three types of Transformer blocks that make up RETRO\nLet’s start by looking at the encoder stack, which processes the retrieved neighbors, resulting in KEYS and VALUES matrices that will later be used for attention (see\nThe Illustrated Transformer\nfor a refresher).\nThe encoder stack processes the retrieved neighbors resulting in KEYS and VALUE matrices\nDecoder blocks process the input text just like a GPT would. It applies self-attention on the prompt token (causally, so only attending to previous tokens), then passes through a FFNN layer.\nInput prompt passes through standard decoder block containing self-attention and FFNN layers\nIt’s only when a RETRO decoder is reached do we start to incorporate the retrieved information. Every third block starting from 9 is a RETRO block (that allows its input to attend to the neighbors). So layers 9, 12, 15…32 are RETRO blocks. (The two smaller Retro models, and the Retrofit models have these layers starting from the 6th instead of the 9th layer).\nInput prompt reaches RETRO Decoder block to start information retrieval\nSo effectively, this is the step where the retrieved information can glance at the dates it needs to complete the prompt.\nRETRO Decoder block retrieving information from nearest neighbour chunks using Chunked Cross-Attention\nPrevious Work\nAiding language models with retrieval techniques has been an active area of research. Some of the previous work in the space includes:\nImproving Neural Language Models with a Continuous Cache\nGeneralization through Memorization: Nearest Neighbor Language Models\nRead the\nRetrieval Augmented Generation\nblog from Meta AI and go through Jackie Chi Kit Cheung’s lecture on\nLeveraging External Knowledge in Natural Language Understanding Systems\nSPALM:\nAdaptive Semiparametric Language Models\nDPR:\nDense Passage Retrieval for Open-Domain Question Answering\nREALM: Retrieval-Augmented Language Model Pre-Training\nFiD:\nLeveraging Passage Retrieval with Generative Models for Open Domain Question Answering\nEMDR:\nEnd-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering\nBlenderBot 2.0:\nInternet-Augmented Dialogue Generation\nPlease post in\nthis thread\nor reach out to me on\nTwitter\nfor any corrections or feedback.\nWritten on January 3, 2022", "metadata": { "title": "The Illustrated Retrieval Transformer – Jay Alammar – Visualizing machine learning one concept at a time.", "description": "Discussion: Discussion Thread for comments, corrections, or any feedback. \n\nTranslations: Korean, Russian\n\n\nSummary: The latest batch of language models can be much smaller yet achieve GPT-3 like performance by being able to query a database or search the web for information. A key indication is that building larger and larger models is not the only way to improve performance.\n\nVideo\n\n\n\n\nThe last few years saw the rise of Large Language Models (LLMs) – machine learning models that rapidly improve how machines process and generate language. Some of the highlights since 2017 include:\n\n\n The original Transformer breaks previous performance records for machine translation.\n BERT popularizes the pre-training then finetuning process, as well as Transformer-based contextualized word embeddings. It then rapidly starts to power Google Search and Bing Search.\n GPT-2 demonstrates the machine’s ability to write as well as humans do.\n First T5, then T0 push the boundaries of transfer learning (training a model on one task, and then having it do well on other adjacent tasks) and posing a lot of different tasks as text-to-text tasks.\n GPT-3 showed that massive scaling of generative models can lead to shocking emergent applications (the industry continues to train larger models like Gopher, MT-NLG…etc).\n\n\nFor a while, it seemed like scaling larger and larger models is the main way to improve performance. Recent developments in the field, like DeepMind’s RETRO Transformer and OpenAI’s WebGPT, reverse this trend by showing that smaller generative language models can perform on par with massive models if we augment them with a way to search/query for information.\n\nThis article breaks down DeepMind’s RETRO (Retrieval-Enhanced TRansfOrmer) and how it works. The model performs on par with GPT-3 despite being 4% its size (7.5 billion parameters vs. 185 billion for GPT-3 Da Vinci).\n\n\n \n \n RETRO incorporates information retrieved from a database to free its parameters from being an expensive store of facts and world knowledge.\n\n\nRETRO was presented in the paper Improving Language Models by Retrieving from Trillions of Tokens. It continues and builds on a wide variety of retrieval work in the research community. This article explains the model and not what is especially novel about it.", "keywords": null, "author": "Jay Alammar", "og:description": "Discussion: Discussion Thread for comments, corrections, or any feedback. \n\nTranslations: Korean, Russian\n\n\nSummary: The latest batch of language models can be much smaller yet achieve GPT-3 like performance by being able to query a database or search the web for information. A key indication is that building larger and larger models is not the only way to improve performance.\n\nVideo\n\n\n\n\nThe last few years saw the rise of Large Language Models (LLMs) – machine learning models that rapidly improve how machines process and generate language. Some of the highlights since 2017 include:\n\n\n The original Transformer breaks previous performance records for machine translation.\n BERT popularizes the pre-training then finetuning process, as well as Transformer-based contextualized word embeddings. It then rapidly starts to power Google Search and Bing Search.\n GPT-2 demonstrates the machine’s ability to write as well as humans do.\n First T5, then T0 push the boundaries of transfer learning (training a model on one task, and then having it do well on other adjacent tasks) and posing a lot of different tasks as text-to-text tasks.\n GPT-3 showed that massive scaling of generative models can lead to shocking emergent applications (the industry continues to train larger models like Gopher, MT-NLG…etc).\n\n\nFor a while, it seemed like scaling larger and larger models is the main way to improve performance. Recent developments in the field, like DeepMind’s RETRO Transformer and OpenAI’s WebGPT, reverse this trend by showing that smaller generative language models can perform on par with massive models if we augment them with a way to search/query for information.\n\nThis article breaks down DeepMind’s RETRO (Retrieval-Enhanced TRansfOrmer) and how it works. The model performs on par with GPT-3 despite being 4% its size (7.5 billion parameters vs. 185 billion for GPT-3 Da Vinci).\n\n\n \n \n RETRO incorporates information retrieved from a database to free its parameters from being an expensive store of facts and world knowledge.\n\n\nRETRO was presented in the paper Improving Language Models by Retrieving from Trillions of Tokens. It continues and builds on a wide variety of retrieval work in the research community. This article explains the model and not what is especially novel about it.", "og:title": "The Illustrated Retrieval Transformer" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://jalammar.github.io/images/retro/deepmind-retro-retrieval-transformer.png", "bytes": 160618, "reused": false, "local_path": "attachments/c62c94113743/images/body-9704fe703ad65ba0540c.png", "src": "https://jalammar.github.io/images/retro/deepmind-retro-retrieval-transformer.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/prompt-1.png", "bytes": 47133, "reused": false, "local_path": "attachments/c62c94113743/images/body-7b62685a3f8eeb31e45f.png", "src": "https://jalammar.github.io/images/retro/prompt-1.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/prompt-2.png", "bytes": 94100, "reused": false, "local_path": "attachments/c62c94113743/images/body-b8a79118eadc84f5e946.png", "src": "https://jalammar.github.io/images/retro/prompt-2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/Large-GPT-vs-Retro-transformer-world-knowledge-information.png", "bytes": 805731, "reused": false, "local_path": "attachments/c62c94113743/images/body-b5608b742f5362dc79df.png", "src": "https://jalammar.github.io/images/retro/Large-GPT-vs-Retro-transformer-world-knowledge-information.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/dune-prompt-into-retro-transformer-4.png", "bytes": 206677, "reused": false, "local_path": "attachments/c62c94113743/images/body-8c4984c0469d860763cf.png", "src": "https://jalammar.github.io/images/retro/dune-prompt-into-retro-transformer-4.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/database-key-value-examples.png", "bytes": 365507, "reused": false, "local_path": "attachments/c62c94113743/images/body-c6f7e433ba90c1fe7ab5.png", "src": "https://jalammar.github.io/images/retro/database-key-value-examples.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/bert-sentence-embedding.png", "bytes": 195998, "reused": false, "local_path": "attachments/c62c94113743/images/body-0488fd17344ae060aca1.png", "src": "https://jalammar.github.io/images/retro/bert-sentence-embedding.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/neighbor-retrieval-from-retro-neural-database-with-bert-embeddings.png", "bytes": 1600927, "reused": false, "local_path": "attachments/c62c94113743/images/body-3d0203cc7da4531ac33a.png", "src": "https://jalammar.github.io/images/retro/neighbor-retrieval-from-retro-neural-database-with-bert-embeddings.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/input-prompt-and-retrieved-text-retro-transformer.png", "bytes": 1295134, "reused": false, "local_path": "attachments/c62c94113743/images/body-9508b38f35c09dd4d6f4.png", "src": "https://jalammar.github.io/images/retro/input-prompt-and-retrieved-text-retro-transformer.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/Retro-transformer-encoder-decoder-stacks-2.png", "bytes": 727247, "reused": false, "local_path": "attachments/c62c94113743/images/body-a655716cc697dc3e33a7.png", "src": "https://jalammar.github.io/images/retro/Retro-transformer-encoder-decoder-stacks-2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/retro-transformer-blocks-4.png", "bytes": 286760, "reused": false, "local_path": "attachments/c62c94113743/images/body-e5a0fd30b161018a5865.png", "src": "https://jalammar.github.io/images/retro/retro-transformer-blocks-4.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/retro-encoder-block-keys-values-2.png", "bytes": 278691, "reused": false, "local_path": "attachments/c62c94113743/images/body-b3f59749afe0f012d163.png", "src": "https://jalammar.github.io/images/retro/retro-encoder-block-keys-values-2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/retro-transformer-decoders-2.png", "bytes": 279213, "reused": false, "local_path": "attachments/c62c94113743/images/body-ade14f7ade6cd90fc609.png", "src": "https://jalammar.github.io/images/retro/retro-transformer-decoders-2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/retro-decoder-attention-2.png", "bytes": 352304, "reused": false, "local_path": "attachments/c62c94113743/images/body-199b8821679166353f78.png", "src": "https://jalammar.github.io/images/retro/retro-decoder-attention-2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/retro/retro-decoder-chunked-cross-attention.png", "bytes": 397307, "reused": false, "local_path": "attachments/c62c94113743/images/body-29c821f7cf7d24453719.png", "src": "https://jalammar.github.io/images/retro/retro-decoder-chunked-cross-attention.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=sMPq4cVS4kg", "src": "https://www.youtube.com/embed/sMPq4cVS4kg", "status": "ok", "kind": "single_video", "local_path": "attachments/c62c94113743/videos/repair-sMPq4cVS4kg.mp4", "bytes": 68324393, "valid": true, "height": 1080, "width": 1920, "duration": "433.241000", "reused": false } ], "links": [ { "href": "https://github.com/jalammar/jalammar.github.io/discussions/21", "label": "Discussion Thread", "type": "a" }, { "href": "https://chloamme.github.io/2022/01/08/illustrated-retrieval-transformer-korean.html", "label": "Korean", "type": "a" }, { "href": "https://habr.com/ru/post/648705/", "label": "Russian", "type": "a" }, { "href": "https://www.youtube.com/embed/sMPq4cVS4kg", "label": "", "type": "iframe" }, { "href": "https://jalammar.github.io/illustrated-transformer/", "label": "Transformer", "type": "a" }, { "href": "https://jalammar.github.io/illustrated-bert/", "label": "BERT", "type": "a" }, { "href": "https://blog.google/products/search/search-language-understanding-bert/", "label": "Google Search", "type": "a" }, { "href": "https://azure.microsoft.com/en-us/blog/bing-delivers-its-largest-improvement-in-search-experience-using-azure-gpus/", "label": "Bing Search", "type": "a" }, { "href": "https://jalammar.github.io/illustrated-gpt2/", "label": "GPT-2", "type": "a" }, { "href": "https://arxiv.org/abs/1910.10683", "label": "T5", "type": "a" }, { "href": "https://huggingface.co/bigscience/T0pp", "label": "T0", "type": "a" }, { "href": "https://jalammar.github.io/how-gpt3-works-visualizations-animations/", "label": "GPT-3", "type": "a" }, { "href": "https://deepmind.com/research/publications/2021/scaling-language-models-methods-analysis-insights-from-training-gopher", "label": "Gopher", "type": "a" }, { "href": "https://www.microsoft.com/en-us/research/blog/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model/", "label": "MT-NLG", "type": "a" }, { "href": "https://deepmind.com/research/publications/2021/improving-language-models-by-retrieving-from-trillions-of-tokens", "label": "RETRO Transformer", "type": "a" }, { "href": "https://openai.com/blog/improving-factual-accuracy/", "label": "WebGPT", "type": "a" }, { "href": "https://arxiv.org/abs/2112.04426", "label": "Improving Language Models by Retrieving from Trillions of Tokens", "type": "a" }, { "href": "http://www.crm.umontreal.ca/2018/Langue18/pdf/Cheung.pdf", "label": "work", "type": "a" }, { "href": "https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/", "label": "in", "type": "a" }, { "href": "https://openreview.net/forum?id=HklBjCEKvH", "label": "the", "type": "a" }, { "href": "https://arxiv.org/abs/2102.02557", "label": "research", "type": "a" }, { "href": "https://openreview.net/forum?id=B184E5qee", "label": "community", "type": "a" }, { "href": "https://github.com/google-research/google-research/tree/master/scann", "label": "https://github.com/google-research/google-research/tree/master/scann", "type": "a" }, { "href": "https://aclanthology.org/2020.emnlp-main.550/", "label": "Dense Passage Retrieval for Open-Domain Question Answering", "type": "a" }, { "href": "https://arxiv.org/abs/2002.08909", "label": "REALM: Retrieval-Augmented Language Model Pre-Training", "type": "a" }, { "href": "https://aclanthology.org/2021.eacl-main.74/", "label": "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering", "type": "a" }, { "href": "https://arxiv.org/abs/2106.05346", "label": "End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering", "type": "a" }, { "href": "https://arxiv.org/abs/2107.07566", "label": "Internet-Augmented Dialogue Generation", "type": "a" }, { "href": "https://twitter.com/JayAlammar", "label": "Twitter", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:30Z", "source_url": "https://jalammar.github.io/illustrated-retrieval-transformer/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/1910.10683", "bytes": 1190021, "reused": true, "local_path": "papers/repair/4e0745eb290124ed9e02.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2112.04426", "bytes": 1566448, "reused": true, "local_path": "papers/repair/e7c8f0139def992ffc25.pdf" }, { "status": "download_error", "url": "http://www.crm.umontreal.ca/2018/Langue18/pdf/Cheung.pdf", "reason": "ConnectionError: HTTPConnectionPool(host='www.crm.umontreal.ca', port=80): Max retries exceeded with url: /2018/Langue18/pdf/Cheung.pdf (Caused by NameResolutionError(\"HTTPConnection(host='www.crm.", "retried_at": "2026-10-20T22:23:40Z" }, { "status": "ok", "url": "https://arxiv.org/pdf/2102.02557", "bytes": 1018474, "reused": true, "local_path": "papers/repair/b5c9e91bd79c1b9ef250.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.08909", "bytes": 419037, "reused": true, "local_path": "papers/repair/9325ba1ac58281154385.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2106.05346", "bytes": 779198, "reused": true, "local_path": "papers/repair/332b4c30f6fed486a583.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2107.07566", "bytes": 5175833, "reused": false, "local_path": "papers/repair/da6b79dfd7ebe38cbf84.pdf" } ], "actual_video_files": [], "body_chars": 8917, "expected_body_images": 15, "discovery_status": "ok", "delivery_status": "partial", "missing_reason": "download_error", "raw_html": "attachments/c62c94113743/pages/source.html", "local_html": "attachments/c62c94113743/pages/article.html", "body_base_path": "attachments/c62c94113743/pages/", "body_markdown_path": "attachments/c62c94113743/pages/article.md" }